Scalable Laplacian K-modes Imtiaz Masud Ziko, Eric Granger and - PowerPoint PPT Presentation

Scalable Laplacian K-modes Imtiaz Masud Ziko, Eric Granger and Ismail Ben Ayed

Laplacian K-modes (LK) [ Wang and Carreira-Perpiñán 2014 ] 2

Laplacian K-modes (LK) [ Wang and Carreira-Perpiñán 2014 ] K-modes Mode ( ) 2

Laplacian K-modes (LK) [ Wang and Carreira-Perpiñán 2014 ] K-modes Laplacian Zhu ‘02, Weston ‘08, Shi ‘00, Belkin ‘03, ‘06 etc Mode ( ) 2

Laplacian K-modes (LK) [ Wang and Carreira-Perpiñán 2014 ] K-modes Laplacian 3

Laplacian K-modes (LK) [ Wang and Carreira-Perpiñán 2014 ] K-modes Laplacian Simplex constraint Discrete 3

Why Laplacian K-modes? K-means LK ★ Handles non convex (manifold structured) clusters. 4

Why Laplacian K-modes? K-means LK ★ Handles non convex (manifold structured) clusters. ★ Mean or Mode ? 4

Why Laplacian K-modes? ★ Handles non convex (manifold structured) clusters. ☑ Prototypes from input set ★ Mean or Mode ? 4

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😓 simplex/integer constraint. 😓 Dependance of modes on 😓 Laplacian over discrete variable! 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 😓 Eigen-decomposition of Laplacian ( N x N ). 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 😓 Eigen-decomposition of Laplacian ( N x N ). 😋 Convex relaxation (relax integer constraint) [Wang and Carreira-Perpiñán ‘14] : 😓 Solve over N x L variables altogether. 😓 Projection to L -dimensional simplex. Not applicable in large scale clustering 😓 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 👊 😓 Eigen-decomposition of Laplacian ( N x N ). Concave Relaxation 😋 Convex relaxation (relax integer constraint) [Wang and Carreira-Perpiñán ‘14] : 😓 Solve over N x L variables altogether. We Tackle 😓 Projection to L -dimensional simplex. 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 👊 😓 Eigen-decomposition of Laplacian ( N x N ). 😋 Convex relaxation (relax integer constraint) [Wang and Carreira-Perpiñán ‘14] : 👊 😓 Solve over N x L variables altogether. Parallel structure We Tackle 😓 Projection to L -dimensional simplex. 5

Laplacian k-modes: challenges 😓 Challenging Optimization problem : 😋 Well-known Spectral relaxation [Shi & Malik ‘00] : 👊 😓 Eigen-decomposition of Laplacian ( N x N ). 😋 Convex relaxation (relax integer constraint) [Wang and Carreira-Perpiñán ‘14] : 👊 😓 Solve over N x L variables altogether. We Tackle 👊 😓 Projection to L -dimensional simplex. avoid 5

SLK Concave-Convex Relaxation Laplacian 6

SLK Concave-Convex Relaxation Laplacian Direct convex relaxaton Concave relaxation (ours) = = When 6

SLK Concave-Convex Relaxation Laplacian Direct convex relaxaton Concave relaxation (ours) = = When = When 6

SLK Concave-Convex Relaxation Laplacian Direct convex relaxaton Concave relaxation (ours) = = When = When 6

SLK Concave-Convex Relaxation 7

SLK Concave-Convex Relaxation Laplacian 7

SLK Concave-Convex Relaxation Laplacian concave 7

SLK Concave-Convex Relaxation Laplacian concave Linear bound 7

SLK Concave-Convex Relaxation Laplacian concave Linear bound 7

SLK Concave-Convex Relaxation K-modes Laplacian concave Linear bound 7

SLK Concave-Convex Relaxation K-modes Laplacian concave 7

SLK Concave-Convex Relaxation K-modes Laplacian convex concave 7

SLK Concave-Convex Relaxation K-modes Laplacian convex concave 👊 Avoids extra dual variables for constraints: 👊 Closed- form update duel : 7

SLK Proposed bound Iterative bound: Where, 8

SLK Proposed bound Iterative bound: Sum of independent function Where, 8

SLK Proposed bound Independent Iterative bound: 9

SLK Proposed bound Independent Iterative bound: 👊 KKT conditions get closed form solution : 10

SLK-BO Modes as byproducts of the formulated z-updates: 👊 In z - updates: 11

SLK-BO Modes as byproducts of the formulated z-updates: 👊 In z - updates: 👊 take the form of soft approximation of hard max as: 11

SLK-BO Modes as byproducts of the formulated z-updates: 👊 take the form of soft approximation of hard max as: Linear in N Unlike Mean-shift : No gradient ascent iterates ☑ ☑ Independent of feature dimensions Arbitrary kernels ☑ 11

SLK Result NMI/Accuracy Time (seconds) 12

SLK Result NMI/Accuracy Time (seconds) 12

SLK Result Comparison of optimization quality w.r.t LK [Wang and Carreira-Perpiñán 2014] MNIST (small) LabelMe (Alexnet) 13

Thank you Code on: https://github.com/imtiazziko/SLK More at poster session: Room 210 & 230 AB #96 40