a sober look at clustering stability
play

A Sober look at Clustering Stability Shai Ben-David 1 Ulrike von - PowerPoint PPT Presentation

Jun 14, 2023 •272 likes •812 views

A Sober look at Clustering Stability Shai Ben-David 1 Ulrike von Luxburg 2 Dvid Pl 1 1 School of Computer Science University of Waterloo 2 Fraunhofer IPSI, Darmstadt, Germany COLT 2006 Shai Ben-David, Ulrike von Luxburg, Dvid Pl A Sober

  1. A Sober look at Clustering Stability Shai Ben-David 1 Ulrike von Luxburg 2 Dávid Pál 1 1 School of Computer Science University of Waterloo 2 Fraunhofer IPSI, Darmstadt, Germany COLT 2006 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  2. What is clustering? By clustering we mean grouping data according to some distance/similarity measure. Data Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  3. What is clustering? By clustering we mean grouping data according to some distance/similarity measure. Clusters (Linkage algorithm) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  4. What is clustering? By clustering we mean grouping data according to some distance/similarity measure. Clusters (Center-based algorithm) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  5. Correctness of clustering Q: Clustering is not well defined problem. How do we know that we cluster correctly? A: Common solution – Stability. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  6. Correctness of clustering Q: Clustering is not well defined problem. How do we know that we cluster correctly? A: Common solution – Stability. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  7. Stability: Idea of our definition Pick your favorite clustering algorithm A . Generate two independent samples S 1 and S 2 . Stability How much will clusterings A ( S 1 ) and A ( S 2 ) differ? If for large sample sizes clusterings A ( S 1 ) and A ( S 2 ) are almost identical, we say that A is stable . Otherwise unstable . Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  8. Stability: Idea of our definition Pick your favorite clustering algorithm A . Generate two independent samples S 1 and S 2 . Stability How much will clusterings A ( S 1 ) and A ( S 2 ) differ? If for large sample sizes clusterings A ( S 1 ) and A ( S 2 ) are almost identical, we say that A is stable . Otherwise unstable . Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  9. Stability: Idea of our definition Pick your favorite clustering algorithm A . Generate two independent samples S 1 and S 2 . Stability How much will clusterings A ( S 1 ) and A ( S 2 ) differ? If for large sample sizes clusterings A ( S 1 ) and A ( S 2 ) are almost identical, we say that A is stable . Otherwise unstable . Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  10. Example of stability Probability distribution Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  11. Example of stability Sample S 1 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  12. Example of stability Clustering A ( S 1 ) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  13. Example of stability Sample S 2 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  14. Example of stability Clustering A ( S 2 ) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  15. Example of stability Clusterings A ( S 1 ) and A ( S 2 ) are equivalent. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  16. Example of instability Probability distribution Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  17. Example of instability Sample S 1 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  18. Example of instability Clustering A ( S 1 ) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  19. Example of instability Sample S 2 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  20. Example of instability Clustering A ( S 2 ) Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  21. Example of instability Clusterings A ( S 1 ) and A ( S 2 ) are different Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  22. Motivation Why do people think stability is important? For tuning parameters of clusterings algorithms, such as number of clusters To verify meaningfulness of clustering outputted by algorithm. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  23. Motivation Why do people think stability is important? For tuning parameters of clusterings algorithms, such as number of clusters To verify meaningfulness of clustering outputted by algorithm. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  24. Motivation Our intention: Provide theoretical justification. We discovered: The popular belief is false. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  25. Motivation Our intention: Provide theoretical justification. We discovered: The popular belief is false. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  26. First example 1D probability distribution Probability density 50% 50% x Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  27. First example 2 centers – stable Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  28. First example 3 centers – solution #1 Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  29. First example 3 centers – solution #2 = ⇒ unstable Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  30. First example slightly asymmetric distribution Probability density (50 + ǫ )% (50 − ǫ )% x Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  31. First example 2 centers – stable Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  32. First example 3 centers – stable x Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  33. Second example 1D probability distribution Probability density ∼ 90% ∼ 10% x Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  34. Second example 2 centers – unstable ∼ 90% ∼ 10% Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  35. Second example 3 centers – stable ∼ 90% ∼ 10% x Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  36. Our results Theorem For a cost based algorithm (e.g. k-means, k-medians): If the optimization problem has unique optimum, then the algorithm is stable. If the underlying probability distribution is symmetric and the optimization problem has multiple symmetric optima, then the algorithm is unstable. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  37. Our results Theorem For a cost based algorithm (e.g. k-means, k-medians): If the optimization problem has unique optimum, then the algorithm is stable. If the underlying probability distribution is symmetric and the optimization problem has multiple symmetric optima, then the algorithm is unstable. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  38. Our results Theorem For a cost based algorithm (e.g. k-means, k-medians): If the optimization problem has unique optimum, then the algorithm is stable. If the underlying probability distribution is symmetric and the optimization problem has multiple symmetric optima, then the algorithm is unstable. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  39. Conclusion Stability, contrary to common belief, does not measure validity of a clustering or meaningfulness of choice of number of clusters. Instead, it measures the number of solutions to the clustering optimization problem for the underlying probability distribution. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  40. Open problems Q: Is symmetry really needed for instability? A: No! (Work in progress, together with Shai Ben-David & Hans Ulrich Simon) Analyze finite sample sizes, and give explicit bounds. Analyze other types of algorithms e.g. linkage algorithms. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  41. Open problems Q: Is symmetry really needed for instability? A: No! (Work in progress, together with Shai Ben-David & Hans Ulrich Simon) Analyze finite sample sizes, and give explicit bounds. Analyze other types of algorithms e.g. linkage algorithms. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

  42. Open problems Q: Is symmetry really needed for instability? A: No! (Work in progress, together with Shai Ben-David & Hans Ulrich Simon) Analyze finite sample sizes, and give explicit bounds. Analyze other types of algorithms e.g. linkage algorithms. Shai Ben-David, Ulrike von Luxburg, Dávid Pál A Sober look at Clustering Stability

Recommend

Stability of Clustering Methods Sasha Rakhlin Ph.D. candidate, MIT 1 A procedure is stable if P

Stability of Clustering Methods Sasha Rakhlin Ph.D. candidate, MIT 1 A procedure is stable if P

Stability of Clustering Methods Sasha Rakhlin Ph.D. candidate, MIT 1 A procedure is stable if P ( solution perturbed solution > ) 0 2 This talk: A tool for theoretical analysis of stability of clustering algorithms. Idea:

346 views • 24 slides
Improved Correlation Attacks on SOSEMANUK and SOBER-128 Joo Yeon Cho Helsinki University of

Improved Correlation Attacks on SOSEMANUK and SOBER-128 Joo Yeon Cho Helsinki University of

Improved Correlation Attacks on SOSEMANUK and SOBER-128 Joo Yeon Cho Helsinki University of Technology Department of Information and Computer Science, Espoo, Finland 24th March 2009 1 / 35 SOSEMANUK Attack Approximations SOBER-128

365 views • 35 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views • 40 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Stability Analysis For Unsupervised Learning Dr. Derek Greene Insight @ UCD April 2014

Stability Analysis For Unsupervised Learning Dr. Derek Greene Insight @ UCD April 2014

Stability Analysis For Unsupervised Learning Dr. Derek Greene Insight @ UCD April 2014 derek.greene@ucd.ie Introduction Cluster Validation : A quantitative means of assessing the quality of a clustering. Feedback Cluster Clustering Pre-

475 views • 12 slides
Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity and high separation Many

768 views • 43 slides
Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space. Finding meaningful

396 views • 36 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering What is Clustering? Types of Data in Cluster Analysis Clustering A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods 1 2 What is Clustering? What is Clustering? Typical

647 views • 18 slides
SOBER LIVING ENVIRONMENT Presentation WELCOME! TREATMENT Longer Treatment = Improves Outcome

SOBER LIVING ENVIRONMENT Presentation WELCOME! TREATMENT Longer Treatment = Improves Outcome

SOBER LIVING ENVIRONMENT Presentation WELCOME! TREATMENT Longer Treatment = Improves Outcome WORKS. More Support = Improves Outcome RECOVERY IS SOBER LIVING = IMPROVES OUTCOME! POSSIBLE! FO RM S FORMS PRECOVERY RESIDENCE

640 views • 26 slides
Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

LUDWIG- MAXIMILIANS- INSTITUTE DATABASE UNIVERSITT FOR SYSTEMS MNCHEN MNCHEN INFORMATICS INFORMATICS GROUP GROUP Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative Clustering, Multiview

513 views • 9 slides
Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering Clustering by Partitioning, e.g. k-Means Density Based Clustering, e.g. DBScan Grid Based Clustering Compendium slides for Guide to Intelligent

1.32k views • 82 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides
1 Peter 5:8, Be sober [stable in your thinking], be vigilant; because your adversary the devil

1 Peter 5:8, Be sober [stable in your thinking], be vigilant; because your adversary the devil

1 Peter 5:8, Be sober [stable in your thinking], be vigilant; because your adversary the devil walks about like a roaring lion, seeking whom he may devour. 1 Peter 5:9, Resist him, steadfast in the faith [doctrine], Job 1:1,

452 views • 26 slides
PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin

PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin

PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin Motivation Example Clustering cannot be analyzed without specifying what it will be used for! be used for! Example Cluster then pack

293 views • 28 slides
Beware of the THIEF Be alert and of sober mind. Your enemy the devil prowls around like a

Beware of the THIEF Be alert and of sober mind. Your enemy the devil prowls around like a

Beware of the THIEF Be alert and of sober mind. Your enemy the devil prowls around like a roaring lion looking for someone to devour. 1 Peter 5:8 God's Plans for us are good! We need to KNOW it and BELIEVE it! Jeremiah 29:11 Satan also has

815 views • 11 slides
Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means clustering Hierarchical clustering Incremental clustering Probability-based clustering Self-Organising Maps Classification vs. Clustering

1k views • 67 slides
Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat Clustering 7.3 Embedding into Vector Space for Visualization 7.4 Applications Clustering: unsupervised grouping (partitioning) of objects into

763 views • 49 slides
Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as

Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as

Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as any other clustering Identification of communities in social networks Webpage clustering for better data management of web data Outline

607 views • 41 slides
Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document clustering Motivations Document representations Success criteria Clustering algorithms Partitional Hierarchical Ch. 16 What is

766 views • 61 slides
Clustering: Models and Algorithms Shikui Tu 2019-02-28 1 Outline Clustering K-mean

Clustering: Models and Algorithms Shikui Tu 2019-02-28 1 Outline Clustering K-mean

Clustering: Models and Algorithms Shikui Tu 2019-02-28 1 Outline Clustering K-mean clustering, hierarchical clustering Adaptive learning (online learning) CL, FSCL, RPCL Gaussian Mixture Models (GMM)

805 views • 34 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views • 110 slides

More recommend