data pre processing for k means clustering
play

Data pre-processing for k- means clustering Karolis Urbonas Head - PowerPoint PPT Presentation

Mar 14, 2024 •217 likes •481 views

DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Data pre-processing for k- means clustering Karolis Urbonas Head of Data Science, Amazon DataCamp Customer Segmentation in Python Advantages of k-means clustering One

  1. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Data pre-processing for k- means clustering Karolis Urbonas Head of Data Science, Amazon

  2. DataCamp Customer Segmentation in Python Advantages of k-means clustering One of the most popular unsupervised learning method Simple and fast Works well* * with certain assumptions about the data

  3. DataCamp Customer Segmentation in Python Key k-means assumptions Symmetric distribution of variables (not skewed) Variables with same average values Variables with same variance

  4. DataCamp Customer Segmentation in Python Skewed variables Left-skewed Right-skewed

  5. DataCamp Customer Segmentation in Python Skewed variables Skew removed with logarithmic transformation

  6. DataCamp Customer Segmentation in Python Variables on the same scale datamart_rfm.describe() K-means assumes equal mean And equal variance It's not the case with RFM data

  7. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's review the concepts

  8. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Managing skewed variables Karolis Urbonas Head of Data Science, Amazon

  9. DataCamp Customer Segmentation in Python Identifying skewness Visual analysis of the distribution If it has a tail - it's skewed

  10. DataCamp Customer Segmentation in Python Exploring distribution of Recency import seaborn as sns from matplotlib import pyplot as plt sns.distplot(datamart['Recency']) plt.show()

  11. DataCamp Customer Segmentation in Python Exploring distribution of Frequency sns.distplot(datamart['Frequency']) plt.show()

  12. DataCamp Customer Segmentation in Python Data transformations to manage skewness Logarithmic transformation (positive values only) import numpy as np frequency_log= np.log(datamart['Frequency']) sns.distplot(frequency_log) plt.show()

  13. DataCamp Customer Segmentation in Python Dealing with negative values Adding a constant before log transformation Cube root transformation

  14. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Let's practice how to identify and manage skewed variables!

  15. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Centering and scaling variables Karolis Urbonas Head of Data Science, Amazon

  16. DataCamp Customer Segmentation in Python Identifying an issue datamart_rfm.describe() Analyze key statistics of the dataset Compare mean and standard deviation

  17. DataCamp Customer Segmentation in Python Centering variables with different means K-means works well on variables with the same mean Centering variables is done by subtracting average value from each observation datamart_centered = datamart_rfm - datamart_rfm.mean() datamart_centered.describe().round(2)

  18. DataCamp Customer Segmentation in Python Scaling variables with different variance K-means works better on variables with the same variance / standard deviation Scaling variables is done by dividing them by standard deviation of each datamart_scaled = datamart_rfm / datamart_rfm.std() datamart_scaled.describe().round(2)

  19. DataCamp Customer Segmentation in Python Combining centering and scaling Subtract mean and divide by standard deviation manually Or use a scaler from scikit-learn library (returns numpy.ndarray object) from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(datamart_rfm) datamart_normalized = scaler.transform(datamart_rfm) print('mean: ', datamart_normalized.mean(axis=0).round(2)) print('std: ', datamart_normalized.std(axis=0).round(2)) mean: [-0. -0. 0.] std: [1. 1. 1.]

  20. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Test different approaches by yourself!

  21. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Sequence of structuring pre-processing steps Karolis Urbonas Head of Data Science, Amazon

  22. DataCamp Customer Segmentation in Python Why the sequence matters? Log transformation only works with positive data Normalization forces data to have negative values and log will not work

  23. DataCamp Customer Segmentation in Python Sequence 1. Unskew the data - log transformation 2. Standardize to the same average values 3. Scale to the same standard deviation 4. Store as a separate array to be used for clustering

  24. DataCamp Customer Segmentation in Python Coding the sequence Unskew the data with log transformation import numpy as np datamart_log = np.log(datamart_rfm) Normalize the variables with StandardScaler from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(datamart_log) Store it separately for clustering datamart_normalized = scaler.transform(datamart_log)

  25. DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Practice on RFM data!

Recommend

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 18 notes: K-means and Factor Analysis Tues, 4.17 1 K-means clustering The K-means clustering algorithm can be seen as

547 views • 3 slides
k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering

k -means clustering Method to automatically separate data sets into distinct groups. Clustering example Clustering example k -means clustering algorithm 1. Start with k randomly chosen means 2. Color data points by the shortest distance to any

1.56k views • 19 slides
Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years

Data Clustering: Data Clustering: 50 Years Beyond K means 50 Years Beyond K means 50 Years Beyond K-means 50 Years Beyond K-means Anil K. Jain Department of Computer Science Michigan State University King-Sun Fu King-Sun Fu King-Sun

1.03k views • 39 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views • 110 slides
Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K

Multi-variable Optimization K-means clustering K-means clustering on points is finding K central locations that reduce the distance of each point to the nearest central location (summed over all points) K=3 K-means clustering For

383 views • 10 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data points. We want to find underlying structure in the data. Examples: Identify groups of similar data points. Clustering Find a better basis

831 views • 14 slides
Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business

Basics of k-means clustering CLUS TERIN G METH ODS W ITH S CIP Y Shaumik Daityari Business Analyst Why k-means clustering? A critical drawback of hierarchical clustering: runtime K means runs signicantly faster on large datasets

691 views • 25 slides
Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization

Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 1 Introduction to Machine Learning CMU-10701 Clustering and EM Barnabs Pczos & Aarti Singh Clustering 3 K- means

1.04k views • 67 slides
Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad,

Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad,

Asynchronous K-Means Clustering of Multiple Data Sets Marek Fiser, Illia Ziamtsov, Ariful Azad, Bedrich Benes, Alex Pothen Motivation Clustering bottleneck in Flow Cytometry research 3,000 data sets 25,000 points in 7D per data set 19 separate

406 views • 19 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides
DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu

DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu

Welcome to DS504/CS586: Big Data Analytics Big Data Clustering II Prof. Yanhua Li Time: 6pm 8:50pm Thu Location: AK 232 Fall 2016 More Discussions, Limitations v Center based clustering K-means BFR algorithm v Hierarchical clustering

873 views • 52 slides
Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means clustering Hierarchical clustering Incremental clustering Probability-based clustering Self-Organising Maps Classification vs. Clustering

1k views • 67 slides
Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering

Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering

Evolutionary Clustering Presenter: Lei Tang Evolutionary Clustering Evolutionary Clustering Processing time stamped data to produce Processing time stamped data to produce a sequence of clustering. Each clustering should be similar to

431 views • 32 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW

K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW

K-MEANS++ OPTIMAL INITIALIZATION ALGORITHM An Improved K-means Clustering Method OVERVIEW K-means Clustering Algorithm K-means++ Initialization Algorithm Experiment Datasets Conclusion K-MEANS CLUSTERING ALGORITHM A

313 views • 14 slides
Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability Density Estimation Gaussian Mixture Models M. Magdon-Ismail CSCI 4100/6100 recap: Radial Basis Functions Nonparametric RBF Parametric k -RBF-Network

253 views • 23 slides
Practical implementation of k-means clustering Karolis Urbonas Head of Data Science, Amazon

Practical implementation of k-means clustering Karolis Urbonas Head of Data Science, Amazon

DataCamp Customer Segmentation in Python CUSTOMER SEGMENTATION IN PYTHON Practical implementation of k-means clustering Karolis Urbonas Head of Data Science, Amazon DataCamp Customer Segmentation in Python Key steps Data pre-processing

862 views • 37 slides
K -means Clustering Ke Chen Reading: [7.3, EA], [9.1, CMB] COMP24111 Machine Learning Outline

K -means Clustering Ke Chen Reading: [7.3, EA], [9.1, CMB] COMP24111 Machine Learning Outline

K -means Clustering Ke Chen Reading: [7.3, EA], [9.1, CMB] COMP24111 Machine Learning Outline Introduction K -means Algorithm Example How K -means partitions? K -means Demo Relevant Issues Application:

416 views • 22 slides
EM, K-Means, GNG 4-6-16 Reading Quiz Which of the following can be considered an instance of the

EM, K-Means, GNG 4-6-16 Reading Quiz Which of the following can be considered an instance of the

EM, K-Means, GNG 4-6-16 Reading Quiz Which of the following can be considered an instance of the EM algorithm? a) Agglomerative clustering b) Divisive clustering c) K-means clustering d) Growing neural gas EM algorithm E step:

445 views • 17 slides
Clustering Reference:http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/ Dr Ahmed

Clustering Reference:http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/ Dr Ahmed

Clustering Reference:http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/ Dr Ahmed Rafea Outline Introduction Clustering Algorithms K-means Fuzzy C-means Hierarchical clustering Introduction(1)

394 views • 23 slides
1-means clustering and conductance Twan van Laarhoven Radboud University Nijmegen Open

1-means clustering and conductance Twan van Laarhoven Radboud University Nijmegen Open

1-means clustering and conductance Twan van Laarhoven Radboud University Nijmegen Open Universiteit Nederland November 27th, 2017 1 / 30 Outline Network community detection with conductance The relation to k -means clustering Algorithms

706 views • 52 slides

More recommend