Lecture 13 Jan-Willem van de Meent Four Types of Clustering 1. - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 13 Jan-Willem van de Meent

Four Types of Clustering 1. Centroid-based (K-means, K-medoids) Notion of Clusters: Voronoi tesselation

Four Types of Clustering (not on midterm) 4. Distribution-based (Mixture Models) Notion of Clusters: Distributions on features

<latexit sha1_base64="tsSLORHBRSCrRBz1S0p20bai2bU=">AGLHicfZRLb9QwEIBd6EJZXi0cuUTspUhLlZSq7aVSBaoAIaHSt7TZRk4yuxut4wTb6WNd/yV+AeJXcEGIG+IGvwEnm4pNHOFIycTzcOesf2URFzY9re5GzfnW7duL9xp3713/8HDxaVHRzJWACHQUISduJjDiSicCgiQeAkZYBjn8CxP36V64/PgPEoQfiMoV+jIc0GkQBFnrKWzxBVwIub+/o5bdMzlRXUt/3DhTz6wty+VZ7Em65ajT9+XPOP95Z73tTy6Ne5bVxq/UB61nl8beuOr01XLW+zYK3YxLFNwSqGzjaZj1ua/+yGSZDFQEVAMOc9x05FX2ImoCAarsZhxQHYzyEnhYpjoH3ZbEFyqpoD5y+HCRUA0qZhLHPMZiZEzmMK/OBiMdGFg1bDnZl7mXEHg0pFUrP1bthvCQJejyEyGPslAyb3XL5W0u+svus7qhqohDMKScDbtrn7qwJAB0BLZXOs65smk2YsJfAPsnMsz4YBhfMgiWNMQ+meQaB6en9coDxjkC9Eun4sO45SyoCnqLYp9G13VnmhpHQrCRatUMcuZ7B8oRq6NKBJk6+JgX1swlwxAoEbshfNM4a2MxgMxNiBsTqGUJjTEh5RBJqrGcwQxd9MjCDkhmrHukugzHmLDYzpqxtNRZLB7tcrsqbxdZgnMhjHWdXaTFBgWCcsP3XkRiSKI8FlqVemVUT/b6X19WA7qtqU+dv35Y4yMAnRWNW987s0ICFVS5fZQM2ZFVsWrgGMK2B5QbnpL7vnPrtZgpHqyuOlj+sdbavb74F9AQ9RcvIQRtoG71Bu+gQBegL+ol+oz+tT62vre+tH1P0xlxp8xhVRuvX8E/ROY=</latexit> <latexit sha1_base64="tsSLORHBRSCrRBz1S0p20bai2bU=">AGLHicfZRLb9QwEIBd6EJZXi0cuUTspUhLlZSq7aVSBaoAIaHSt7TZRk4yuxut4wTb6WNd/yV+AeJXcEGIG+IGvwEnm4pNHOFIycTzcOesf2URFzY9re5GzfnW7duL9xp3713/8HDxaVHRzJWACHQUISduJjDiSicCgiQeAkZYBjn8CxP36V64/PgPEoQfiMoV+jIc0GkQBFnrKWzxBVwIub+/o5bdMzlRXUt/3DhTz6wty+VZ7Em65ajT9+XPOP95Z73tTy6Ne5bVxq/UB61nl8beuOr01XLW+zYK3YxLFNwSqGzjaZj1ua/+yGSZDFQEVAMOc9x05FX2ImoCAarsZhxQHYzyEnhYpjoH3ZbEFyqpoD5y+HCRUA0qZhLHPMZiZEzmMK/OBiMdGFg1bDnZl7mXEHg0pFUrP1bthvCQJejyEyGPslAyb3XL5W0u+svus7qhqohDMKScDbtrn7qwJAB0BLZXOs65smk2YsJfAPsnMsz4YBhfMgiWNMQ+meQaB6en9coDxjkC9Eun4sO45SyoCnqLYp9G13VnmhpHQrCRatUMcuZ7B8oRq6NKBJk6+JgX1swlwxAoEbshfNM4a2MxgMxNiBsTqGUJjTEh5RBJqrGcwQxd9MjCDkhmrHukugzHmLDYzpqxtNRZLB7tcrsqbxdZgnMhjHWdXaTFBgWCcsP3XkRiSKI8FlqVemVUT/b6X19WA7qtqU+dv35Y4yMAnRWNW987s0ICFVS5fZQM2ZFVsWrgGMK2B5QbnpL7vnPrtZgpHqyuOlj+sdbavb74F9AQ9RcvIQRtoG71Bu+gQBegL+ol+oz+tT62vre+tH1P0xlxp8xhVRuvX8E/ROY=</latexit> <latexit sha1_base64="tsSLORHBRSCrRBz1S0p20bai2bU=">AGLHicfZRLb9QwEIBd6EJZXi0cuUTspUhLlZSq7aVSBaoAIaHSt7TZRk4yuxut4wTb6WNd/yV+AeJXcEGIG+IGvwEnm4pNHOFIycTzcOesf2URFzY9re5GzfnW7duL9xp3713/8HDxaVHRzJWACHQUISduJjDiSicCgiQeAkZYBjn8CxP36V64/PgPEoQfiMoV+jIc0GkQBFnrKWzxBVwIub+/o5bdMzlRXUt/3DhTz6wty+VZ7Em65ajT9+XPOP95Z73tTy6Ne5bVxq/UB61nl8beuOr01XLW+zYK3YxLFNwSqGzjaZj1ua/+yGSZDFQEVAMOc9x05FX2ImoCAarsZhxQHYzyEnhYpjoH3ZbEFyqpoD5y+HCRUA0qZhLHPMZiZEzmMK/OBiMdGFg1bDnZl7mXEHg0pFUrP1bthvCQJejyEyGPslAyb3XL5W0u+svus7qhqohDMKScDbtrn7qwJAB0BLZXOs65smk2YsJfAPsnMsz4YBhfMgiWNMQ+meQaB6en9coDxjkC9Eun4sO45SyoCnqLYp9G13VnmhpHQrCRatUMcuZ7B8oRq6NKBJk6+JgX1swlwxAoEbshfNM4a2MxgMxNiBsTqGUJjTEh5RBJqrGcwQxd9MjCDkhmrHukugzHmLDYzpqxtNRZLB7tcrsqbxdZgnMhjHWdXaTFBgWCcsP3XkRiSKI8FlqVemVUT/b6X19WA7qtqU+dv35Y4yMAnRWNW987s0ICFVS5fZQM2ZFVsWrgGMK2B5QbnpL7vnPrtZgpHqyuOlj+sdbavb74F9AQ9RcvIQRtoG71Bu+gQBegL+ol+oz+tT62vre+tH1P0xlxp8xhVRuvX8E/ROY=</latexit> <latexit sha1_base64="khI/Vf/F3cDU/2igGVct8pkf8ZQ=">AGLHicfZRLb9QwEIDd0oWyvFo4conYS5GWKlmqtpdKFagChIRK39J6GznJ7G60iRNsp491/Zf4BYhfwQUhbogb/AacNIhNHOFIycTzcOesb0Crmw7a9z8zcWjdvLd5u37l7/6DpeWHRzJmA+HfhIl7MQjHKQwqEIRQnKQMSexEce5OXuf74DBgPE3ogLlMYxGREw2HoE6Gn3KUTLOBCyP39HbWCz+RUdS39wXGmnlpbFuZ7Eq65ajTd+XPJP95a73pT126NRlYVxq/UC61nv01dCdXpz3LXerYq3YxLFNwSqGDyrHrLi98wkHiZzFQ4UeE875jp2IgCROhH4Fq4xDSvwJGUFfi5TEwAey2AJlVbQHzkAOEyqA+hUzSWIeEzE2JnOYV2f9sQ4MrBq2nBzI3EsAPBzRqpUXq3YbBzDU5Sgyk4EXZaDk3qsXStrd9edp7ehagiDoCScTburnzowYgC0RDbXus76psmkGUsj+AfZOZnw4DCuZ/EMaGBxGfgq7eHwyUZwzyhUjsxbLjKUM+BrVNoW+jWeVF0pKXEmwaIU6djmD5QvV0KUBTZt8TQ3sQxOGxRgEacheNMka2Azg81MiBkQq2cIjTEh5WGUGM9wxm6JOhGTSaYcoa5y4jfcYDYnhMx814Og4Ndq9WmT2Vt8sQdgoJrOEmBEZGw/NCdh2IchXEouCz1yrQK6f+tL4ebEdVmzJ/e57cUQbpe1HRmNW9MzvUZ0GVy1fZgI1YFbsuXAOY1sByg3NS3dO/XYzhaPeqPl92ud7e3y5ltEj9ETtIctIG20Wu0iw6Rjz6jH+gX+t362PrS+tb6fo3Oz5U2j1BltH7+AW9wRKY=</latexit> 
 Review: K-means Clustering Objective: Sum of Squares μ 1 N K X X I [ z n = k ] | x n − µ k | 2 SSE ( z , µ ) = μ 2 n = 1 k = 1 One-hot assignment Center for cluster k µ k Alternate between two steps 
 μ 3 1. Minimize SSE w.r.t. z n 2. Minimize SSE w.r.t. μ k

K-means Clustering 5 4 μ 1 3 μ 2 2 1 μ 3 0 0 1 2 3 4 5 Assign each point to closest centroid, then update centroids to average of points

K-means Clustering 5 4 μ 1 3 2 μ 3 μ 2 1 0 0 1 2 3 4 5 Assign each point to closest centroid, then update centroids to average of points

K-means Clustering 5 4 μ 1 3 2 μ 3 μ 2 1 0 0 1 2 3 4 5 Repeat until convergence 
 (no points reassigned, means unchanged)

K-means Clustering 5 4 μ 1 3 2 μ 2 μ 3 1 0 0 1 2 3 4 5 Repeat until convergence 
 (no points reassigned, means unchanged)

K-Means vs Gaussian Mixture Models Idea1: Learn both means μ k and covariances Σ k μ 3 Σ 3 μ 2 Σ 2 μ 2 μ 1 μ 3 μ 1 Σ 1 Don’t just learn where the center of the cluster is, but also how big it is , and what shape it has .