lecture 12 clustering
play

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 - PowerPoint PPT Presentation

Jan 15, 2024 •475 likes •847 views

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma chine e Lea earn rning Paradigm Observe set of examples: training data Infer something about process that generated that data Use

  1. Lecture 12: Clustering 1 6.0002 LECTURE 12

  2. Re Reading § Chapter 23 6.0002 LECTURE 12 2

  3. Mach Ma chine e Lea earn rning Paradigm § Observe set of examples: training data § Infer something about process that generated that data § Use inference to make predictions about previously unseen data: test data § Supervised: given a set of feature/label pairs, find a rule that predicts the label associated with a previously unseen input § Unsupervised : given a set of feature vectors (without labels) group them into “natural clusters” 6.0002 LECTURE 12 3

  4. Clustering Cl g Is s an Op Optimization Pr Problem § Why not divide variability by size of cluster? ◦ Big and bad worse than small and bad § Is optimization problem finding a C that minimizes dissimilarity(C) ? ◦ No, otherwise could put each example in its own cluster § Need a constraint, e.g., ◦ Minimum distance between clusters ◦ Number of clusters 6.0002 LECTURE 12 4

  5. Tw Two Popular Methods § Hierarchical clustering § K-means clustering 6.0002 LECTURE 12 5

  6. Hi Hiea earchical Cl Clustering 1. Start by assigning each item to a cluster, so that if you have N items, you now have N clusters, each containing just one item. 2. Find the closest (most similar) pair of clusters and merge them into a single cluster, so that now you have one fewer cluster. 3. Continue the process until all items are clustered into a single cluster of size N. What does distance mean? 6.0002 LECTURE 12 6

  7. Link Linkag age Metr tric ics § S ingle-linkage: consider the distance between one cluster and another cluster to be equal to the shortest distance from any member of one cluster to any member of the other cluster § C omplete-linkage : consider the distance between one cluster and another cluster to be equal to the greatest distance from any member of one cluster to any member of the other cluster § A verage-linkage: consider the distance between one cluster and another cluster to be equal to the average distance from any member of one cluster to any member of the other cluster 6.0002 LECTURE 12 7

  8. Exampl Ex mple of Hierarchi hical Clus ustering ng BOS NY CHI DEN SF SEA BOS 0 206 963 1949 3095 2979 NY 0 802 1771 2934 2815 CHI 0 966 2142 2013 DEN 0 1235 1307 SF 0 808 SEA 0 {BOS} {NY} {CHI} {DEN} {SF} {SEA} {BOS, NY} {CHI} {DEN} {SF} {SEA} {BOS, NY, CHI} {DEN} {SF} {SEA} {BOS, NY, CHI} {DEN} {SF, SEA} Single linkage {BOS, NY, CHI, DEN} {SF, SEA} or {BOS, NY, CHI} {DEN, SF, SEA} Complete linkage 6.0002 LECTURE 12 8

  9. Cl Clustering Al g Algor orithms § Hierarchical clustering ◦ Can select number of clusters using dendogram ◦ Deterministic ◦ Flexible with respect to linkage criteria ◦ Slow ◦ Naïve algorithm n 3 ◦ n 2 algorithms exist for some linkage criteria § K-means a much faster greedy algorithm ◦ Most useful when you know how many clusters you want 6.0002 LECTURE 12 9

  10. K-me means ns Algorithm thm randomly chose k examples as initial centroids while true: create k clusters by assigning each example to closest centroid compute k new centroids by averaging examples in each cluster if centroids don’t change: break What is complexity of one iteration? k*n*d, where n is number of points and d time required to compute the distance between a pair of points 6.0002 LECTURE 12 10

  11. An An E Example 6.0002 LECTURE 12 11

  12. K = K = 4 4, In Initial Cen Centroi oids 6.0002 LECTURE 12 12

  13. It Iter eration 1 6.0002 LECTURE 12 13

  14. It Iter eration 2 6.0002 LECTURE 12 14

  15. It Iter eration 3 6.0002 LECTURE 12 15

  16. It Iter eration 4 6.0002 LECTURE 12 16

  17. It Iter eration 5 6.0002 LECTURE 12 17

  18. Is Issues es wi with k-me means ns § Choosing the “wrong” k can lead to strange results ◦ Consider k = 3 § Result can depend upon initial centroids ◦ Number of iterations ◦ Even final result ◦ Greedy algorithm can find different local optimas 6.0002 LECTURE 12 18

  19. Ho How w to Choose e K § A priori knowledge about application domain ◦ There are two kinds of people in the world: k = 2 ◦ There are five different types of bacteria: k = 5 § Search for a good k ◦ Try different values of k and evaluate quality of results ◦ Run hierarchical clustering on subset of data 6.0002 LECTURE 12 19

  20. Un Unlucky In Initial Cen Centroi oids 6.0002 LECTURE 12 20

  21. Con Converges O On 6.0002 LECTURE 12 21

  22. Mi Mitigating Dependence on Initial Centroids Try multiple sets of randomly chosen initial centroids Select “best” result best = kMeans(points) for t in range(numTrials): C = kMeans(points) if dissimilarity(C) < dissimilarity(best): best = C return best 6.0002 LECTURE 12 22

  23. An An Example E § Many patients with 4 features each ◦ Heart rate in beats per minute ◦ Number of past heart attacks ◦ Age ◦ ST elevation (binary) § Outcome (death) based on features ◦ Probabilistic, not deterministic ◦ E.g., older people with multiple heart attacks at higher risk § Cluster, and examine purity of clusters relative to outcomes 6.0002 LECTURE 12 23

  24. Da Data Sampl mple HR Att STE Age Outcome P000:[ 89. 1. 0. 66.]:1 P001:[ 59. 0. 0. 72.]:0 P002:[ 73. 0. 0. 73.]:0 P003:[ 56. 1. 0. 65.]:0 P004:[ 75. 1. 1. 68.]:1 P005:[ 68. 1. 0. 56.]:0 P006:[ 73. 1. 0. 75.]:1 P007:[ 72. 0. 0. 65.]:0 P008:[ 73. 1. 0. 64.]:1 P009:[ 73. 0. 0. 58.]:0 P010:[ 100. 0. 0. 75.]:0 P011:[ 79. 0. 0. 31.]:0 P012:[ 81. 0. 0. 58.]:0 P013:[ 89. 1. 0. 50.]:1 P014:[ 81. 0. 0. 70.]:0 6.0002 LECTURE 12 24

  25. Cl Class E Example 6.0002 LECTURE 12 25

  26. Cl Class Cl Cluster 6.0002 LECTURE 12 26

  27. Cl Class Cl Cluster, c con ont. 6.0002 LECTURE 12 27

  28. Ev Evaluating a Clustering 6.0002 LECTURE 12 28

  29. Pa Patients Z-Scaling Mean = ? Std = ? 6.0002 LECTURE 12 29

  30. km kmeans 6.0002 LECTURE 12 30

  31. Ex Exami mini ning ng Resul ults ts 6.0002 LECTURE 12 31

  32. Re Result of Running It Test k-means (k = 2) Cluster of size 118 with fraction of positives = 0.3305 Cluster of size 132 with fraction of positives = 0.3333 Like it? Try patients = getData(True) Test k-means (k = 2) Cluster of size 224 with fraction of positives = 0.2902 Cluster of size 26 with fraction of positives = 0.6923 Happy with sensitivity? 6.0002 LECTURE 12 32

  33. Ho How w Ma Many Positives es Ar Are e Ther ere? e? Total number of positive patients = 83 Test k-means (k = 2) Cluster of size 224 with fraction of positives = 0.2902 Cluster of size 26 with fraction of positives = 0.6923 6.0002 LECTURE 12 33

  34. A Hy A Hypot othes esis § Different subgroups of positive patients have different characteristics § How might we test this? § Try some other values of k 6.0002 LECTURE 12 34

  35. Te Testing Multiple Values of k Test k-means (k = 2) Cluster of size 224 with fraction of positives = 0.2902 Cluster of size 26 with fraction of positives = 0.6923 Test k-means (k = 4) Cluster of size 26 with fraction of positives = 0.6923 Cluster of size 86 with fraction of positives = 0.0814 Cluster of size 76 with fraction of positives = 0.7105 Cluster of size 62 with fraction of positives = 0.0645 Test k-means (k = 6) Cluster of size 49 with fraction of positives = 0.0204 Cluster of size 26 with fraction of positives = 0.6923 Cluster of size 45 with fraction of positives = 0.0889 Cluster of size 54 with fraction of positives = 0.0926 Cluster of size 36 with fraction of positives = 0.7778 Cluster of size 40 with fraction of positives = 0.675 Pick a k 6.0002 LECTURE 12 35

  36. MIT OpenCourseWare https://ocw.mit.edu 6.0002 Introduction to Computational Thinking and Data Science Fall 2016 For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.

Recommend

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture of Gaussians Underlying model for data: assumes a generative process: Each cluster follows a Gaussian distribution First choose which

302 views • 19 slides
Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document clustering Motivations Document representations Success criteria Clustering algorithms Partitional Hierarchical Ch. 16 What is

766 views • 61 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views • 110 slides
LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a

LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a

DATA MINING LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or related) to one another and

998 views • 78 slides
Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

559 views • 33 slides
Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

1.22k views • 56 slides
Spectral Clustering Lecture 16 David Sontag New York

Spectral Clustering Lecture 16 David Sontag New York

Spectral Clustering Lecture 16 David Sontag New York University Slides adapted from James Hays, Alan Fern, and Tommi Jaakkola Spectral clustering K-means Spectral clustering twocircles, 2

848 views • 19 slides
Hierarchical Clustering Lecture 15 David Sontag New York

Hierarchical Clustering Lecture 15 David Sontag New York

Hierarchical Clustering Lecture 15 David Sontag New York University Agglomerative Clustering Agglomerative clustering: First merge very similar instances Incrementally build larger clusters out

465 views • 7 slides
DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING

DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING

DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or related) to one another and

905 views • 79 slides
Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

833 views • 26 slides
Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

1.29k views • 69 slides
CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen

CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen

CSCE 478/878 Lecture 8: Clustering CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen Scott k -Means Clustering Hierarchical Clustering sscott@cse.unl.edu 1 / 19 Introduction CSCE 478/878 If

200 views • 19 slides
Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from

Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from

CSC321: Neural Networks Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from a number of different classes. The aim is to cluster data from the same class together. How do we decide the number of

529 views • 21 slides
Lecture: k-means &amp; mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford

Lecture: k-means &amp; mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford

Clustering Lecture: k-means &amp; mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 22-Oct-2019 1 St Stanfor ord University CS 131 Roadmap Clustering Pixels Segments Images Videos Web

769 views • 55 slides
Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is

Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is

Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is similar to each other, and dissimilar to other cluster data Finding structure in data Hard vs. Soft/Fuzzy vs. Exclusive Understanding and/or summarise

400 views • 26 slides
Hierarchical)&amp;)Spectral)clustering) Lecture)13) David&amp;Sontag&amp;

Hierarchical)&amp;)Spectral)clustering) Lecture)13) David&amp;Sontag&amp;

Hierarchical)&amp;)Spectral)clustering) Lecture)13) David&amp;Sontag&amp; New&amp;York&amp;University&amp; Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Agglomerative Clustering Agglomerative

496 views • 25 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views • 40 slides
1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 18 notes: K-means and Factor Analysis Tues, 4.17 1 K-means clustering The K-means clustering algorithm can be seen as

547 views • 3 slides
Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering overview Jan 31, 2016 Clustering Clustering (and unsupervised learning more generally) finds structure in data, using just X X = a set of

797 views • 57 slides
Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering overview Feb 3, 2016 Clustering Clustering (and unsupervised learning more generally) finds structure in data, using just X X = a set of

768 views • 59 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Administrivia Main feedback from last lecture. Lecture 8 Mud: k -means clustering. Lab 2 handed

Administrivia Main feedback from last lecture. Lecture 8 Mud: k -means clustering. Lab 2 handed

Administrivia Main feedback from last lecture. Lecture 8 Mud: k -means clustering. Lab 2 handed back today. LVCSR Decoding Answers: /user1/faculty/stanchen/e6870/lab2_ans/ . Bhuvana Ramabhadran, Michael Picheny, Stanley F. Chen Lab 3 due

616 views • 35 slides
Clustering Problem Given a set of points, with a

Clustering Problem Given a set of points, with a

Clustering CompSci 590.03 Instructor: Ashwin Machanavajjhala Lecture 17 : 590.02 Spring 13 1 Clustering Problem Given a set of points, with a

444 views • 40 slides

More recommend