Applying bootstrap AMG in spectral clustering Luisa Cutillo School of Mathematics, University of Leeds l.cutillo@leeds.ac.uk Visiting academic, University of Sheffield University Parthenope of Naples, IT joint work with: P. D’Ambra - Institute for Applied Computing, National Research Council of Italy (CNR) and P.S. Vassilevski - Portland State Univ. and CASC-LLNL, USA Sheffield, 6th of September , 2018 L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 1 / 28
Wait a minute...did I say S P E C T R A L? L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 2 / 28
Wait a minute...did I say S P E C T R A L? Ouch! L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 2 / 28
Clustering techniques: two categories Clustering Hierarchical Par11oning L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 3 / 28
nonlinear separating hypersurfaces What if we consider non linear clusters? need of clustering methods that produce nonlinear separating hypersurfaces among clusters two big families: kernel and spectral clustering methods. L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 4 / 28
Kernel and Spectral methods Kernel clustering Spectral clustering Kernels allow to map implicitly Construct a weighted graph data into a high dimensional from the initial data set; feature space; eigenvalue decomposition computing a linear partitioning (spectrum) of the Laplacian in this feature space results in matrix for dimensionality a nonlinear partitioning in the reduction − > clustering in input space. fewer dimensions. L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 5 / 28
A unified view of Spectral and Kernel methods Hint: the adjacency between patterns in the spectral approach is the analogous of the kernel functions in kernel methods. explicit mathematical proof in A survey of kernel and spectral methods for clustering by M. Filippone et. al. In particular Kernel K-Means and Spectral clustering, with the ratio association as the objective function, are perfectly equivalent (shown by Dhillon et al. ) L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 6 / 28
A unified view of Spectral and Kernel methods Hint: the adjacency between patterns in the spectral approach is the analogous of the kernel functions in kernel methods. explicit mathematical proof in A survey of kernel and spectral methods for clustering by M. Filippone et. al. In particular Kernel K-Means and Spectral clustering, with the ratio association as the objective function, are perfectly equivalent (shown by Dhillon et al. ) OMdays L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 6 / 28
Complex Networks Representation Let X = { x 1 , . . . , x n } and W = ( w ij ≥ 0) i , j =1 ,..., n be a set of data and a matrix of similarities between pairs of vertices Similarity Graph G = ( V , E , W ), a weighted undirected graph with V = { 1 , 2 , . . . , n } the vertex set, E = { ( i , j ) = ( j , i ) | w ij > 0 } the edge set, and W the edge weight matrix L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 7 / 28
Complex Networks Representation Let X = { x 1 , . . . , x n } and W = ( w ij ≥ 0) i , j =1 ,..., n be a set of data and a matrix of similarities between pairs of vertices Similarity Graph G = ( V , E , W ), a weighted undirected graph with V = { 1 , 2 , . . . , n } the vertex set, E = { ( i , j ) = ( j , i ) | w ij > 0 } the edge set, and W the edge weight matrix Graph Laplacian The Laplacian matrix of the graph G is: L = D − W ∈ R n × n , where D = diag ( d i = � n j =1 w ij ) i =1 ,..., n is the diagonal matrix of weighted vertex degrees L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 7 / 28
Community detection Communities/Clusters Vertices groups with dense connections within groups and only sparser connections between them functional units such as cycles or pathways in metabolic networks collections of pages on a single topic on the web individuals contacts in social networks L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 8 / 28
Community detection Communities/Clusters Vertices groups with dense connections within groups and only sparser connections between them functional units such as cycles or pathways in metabolic networks collections of pages on a single topic on the web individuals contacts in social networks Community detection as mincut problem Find a graph partition V 1 , . . . , V K minimizing: K RatioCut ( V 1 , . . . , V K ) = 1 W ( V k , V k ) � , 2 | V k | k =1 where W ( V k , V k ) = � i ∈ V k , j ∈ V k w ij and V k complement of V k in V L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 8 / 28
Mincut as trace minimization problem Given a partition V 1 , . . . , V K , let h k = ( h 1 k , . . . , h nk ) T and H = ( h k ) k =1 ,..., K ∈ R n × K be, where: � � 1 / | V k | if x i ∈ V k h ik = i = 1 , . . . , n ; k = 1 , . . . , K . 0 otherwise It holds: K � ( H T LH ) kk = Tr ( H T LH ) , with H T H = I RatioCut ( V 1 , . . . , V K ) = k =1 L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 9 / 28
Mincut as trace minimization problem Given a partition V 1 , . . . , V K , let h k = ( h 1 k , . . . , h nk ) T and H = ( h k ) k =1 ,..., K ∈ R n × K be, where: � � 1 / | V k | if x i ∈ V k h ik = i = 1 , . . . , n ; k = 1 , . . . , K . 0 otherwise It holds: K � ( H T LH ) kk = Tr ( H T LH ) , with H T H = I RatioCut ( V 1 , . . . , V K ) = k =1 trace minimization problem for graph Laplacian Tr ( H T LH ) , subject to H T H = I min V 1 ,..., V K ( h k ) k =1 ,..., K first K eigenvectors of L are the solution (Rayleigh-Ritz theorem) L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 9 / 28
Spectral Clustering Using the first K eigenvectors of graph Laplacian as low-dimension graph embedding (Euclidean) space and applying a spatial clutering in the new space Peng et al., Partitioning Well-Clustered Graphs: Spectral Clustering Works! JMLR, 2015 L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 10 / 28
Our Proposal We propose to use as graph embedding, the space spanned by the algebraically smooth vectors of the graph Laplacian, associated to an adaptive algebraic multigrid method for solving linear systems. Algebraic MultiGrid (AMG) 1 AMG are scalable iterative methods for solving large and sparse linear systems arising from modern applications 2 apply recursively a two-grid process: smoother iterations and a coarse-grid correction L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 11 / 28
Smooth Vectors of Graph Laplacian � L x = b , b subject to b i = 0 i Algebraic MultiGrid (AMG) 1 Pre-smoothing: x = x + M − 1 ( b − L x ) 2 Residual restriction: r c = P T ( b − L x ) 3 Solution on coarse grid: L c e = r c , applying recursion 4 Error interpolation and solution update: x = x + P e 5 Post-smoothing: x = x + ( M T ) − 1 ( b − L x ) L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 12 / 28
Estimating smooth vectors Laplacian graph L can be transformed to s.p.d matrix by rank-1 update: L S = L + α qq T , α > 0 with q having non-zero entries q i = q j = 1 for an arbitrary edge ( i , j ) ∈ E Smooth vectors can be estimated by applying iterative methods to the homogeneous system L S x = 0 , starting from arbitrary x 0 : x ℓ := ( I − B − 1 L S ) x ℓ − 1 ℓ = 1 , . . . , ℓ max L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 13 / 28
Smooth Vectors as effective embedding space Effective embedding algebraically smooth vectors of L S computed by (good convergent) bootstrap AMG well capture the global connectivity of a graph L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 14 / 28
BootCMatch Software Framework AMG Solver Krylov Solvers Bootstrap AMG Apply SuperLU Bootstrap AMG Build HSL−MC64 Single AMG Matching Auction Hierarchy Build Half−approximate Matrix/Vector BootCMatch Software Framework. Available at github.com/bootcmatch/BootCMatch/ D’Ambra et al., BootCMatch: a Software package for Bootstrap AMG based on Graph Weighted Matching , ACM TOMS, 2018 L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 15 / 28
Quality Metrics for Clustering Modularity Function Graphs with strong community structure has large values of: Q = 1 ( A ij − k i k j � 2 m ) δ V i V j 2 m ij defined as the fraction of the edges that fall within the groups minus the expected such fraction if edges were distributed at random. L. Cutillo (Univ. of Leeds) Bootstrap AMG in spectral clutering GPSS 2108 16 / 28
Recommend
More recommend
