Multiresol olution M on Method hods f for Large-scale L Learni - PowerPoint PPT Presentation

Multiresol olution M on Method hods f for Large-scale L Learni ning ng @ @ NIPS 2 S 2015 Ily lya Safro ro Clemso son U Universi sity Multigrid-inspired Methods for Large-scale Networks Fast Response to Infection Spread (with S. Leyffer) • Support Vector Machines (with T. Razzaghi) • Network Generation (with A. Gutfraind and L.A. Meyers) •

Algebraic Multigrid in 3 Slides: Relaxation, Smoothness Observation A suitable relaxation can reduce the information content of the error (by smoothing it), and quickly make it approximable by far fewer variables (which are related to the smooth error modes). Example: Solve Ax=b with initial random guess x (0) ( A is s.p.d.) by stationary iterative relaxation (such as Gauss-Seidel) x (k+1) = T x (k) + v Error = x * - x (k) Initial error After 5 iterations After 10 iterations After 500 iterations Multiscale Methods for Networks 2 Ilya Safro, Clemson University

Algebraic Multigrid in 3 Slides: Optimization Problem Multiscale Methods for Networks 3 Ilya Safro, Clemson University

Algebraic Multigrid in 3 Slides: Coarsening, Correction Scheme Multiscale Methods for Networks 4 Ilya Safro, Clemson University

Multigrid Framework Examples: VLSI placement • Graph partitioning • Eigensolvers • Clustering • Linear arrangement • Community detection • Modularity • Traveling salesman • Visualization • Compression-friendly • ordering Coloring • Spectral problems • Multiscale Methods for Networks 5 Ilya Safro, Clemson University

Algebraic Distance lower upper diagonal triangular triangular Slow convergence but very fast stabilization. Extendible to hypergraphs. See [1] [1] Chen, S “Algebraic Distance on Graphs”,SISC, 2012 [2] Ron, S, Brandt “Relaxation-based coarsening and multiscale graph organization”, MMS, 2011 [3] Bolten, Brandt, Brannick, Frommer, Kahl, Livshits “BAMG for Markov chains”, SISC, 2011 [4] Livne, Brandt “Lean algebraic multigrid (LAMG): Fast graph Laplacian linear solver”, SISC 2012 Multiscale Methods for Networks 6 Ilya Safro, Clemson University

Weighted Aggregation of Graphs (inspired by Algebraic Multigrid) Examples S, Ron, Brandt “Graph minimum linear arrangement by multilevel weighted edge • contractions”, 2006 Ron, S, Brandt “Relaxation-based coarsening and multiscale graph organization”, 2011 • S, Sanders, Schultz “Advanced coarsening schemes for graph partitioning”, 2013 • Multiscale Methods for Networks 7 Ilya Safro, Clemson University

Coarse nodes seeds Multiscale Methods for Networks 8 Ilya Safro, Clemson University

Interpolation weights -1 -1 • Define the interpolation weights for all F-nodes • Intuitively, these weights are the probabilities for a vertex to share a common property (such as the partition in the partitioning problem) with the aggregates it belongs to Multiscale Methods for Networks 9 Ilya Safro, Clemson University

Coarse Graph by AMG weighted aggregation its density is a major computational issue, so effective kernels are important Note that well known matching-based multilevel solvers such as Metis, Scotch, KAHIP, Jostle, etc. can be formulated as restricted cases of AMG Multiscale Methods for Networks 10 Ilya Safro, Clemson University

Compression, Linear Arrangement, Bandwidth, 2-sum, Wavefront Partitioning, Clustering, Vertex Separator Computational Optimization Nonlinear Dimensionality Reduction Problems Response to Epidemics and Cyber Attacks Multiscale Visualization Methods for Network Generation Network Networks Modeling Graph Sparsification Support Vector Machines Machine Text Analysis and Hypothesis Modeling Learning Segmentation More examples of non-matching coarsening: Eigensolvers (Livne, Brandt, Sanders, Henson,…), Random- walk ranking (Sanders, Henson, Sterck,…), Segmentation (Basri,Galun,…), Wavefront(Hu, …) and more Multiscale Methods for Networks 11 Ilya Safro, Clemson University

Response to Epidemics and Cyber Attacks Open Science Grid: collaboration network example Goldberg, Leyffer, S “Optimal Response to Epidemics and Cyber Attacks on Networks”, 2015 Multiscale Methods for Networks 12 Ilya Safro, Clemson University

connections between open sites infection at node I is less than some constant Multiscale Methods for Networks 13 Ilya Safro, Clemson University

Leyffer, S “Fast Response to Infection Spread and Cyber Attacks on Large-scale Networks”, 2013 Multiscale Methods for Networks 14 Ilya Safro, Clemson University

Coarsening sum-of-degrees matrix adjacency matrix Jacobi over-relaxation Algebraic distance is a strength of connection Used as sparsification for Galerkin and in detection of coarse variables Ron, S, Brandt “Relaxation-based coarsening and multiscale graph organization”, 2011 Chen, S “Algebraic distance on graphs”, 2012 New linear term Links between accumulated nodes Multiscale Methods for Networks 15 Ilya Safro, Clemson University

Uncoarsening Local refinement Boundary conditions Multiscale Methods for Networks 16 Ilya Safro, Clemson University

Small random graphs, |V|<100 nodes Erdos-Renyi, Barabasi-Albert, and R-MAT models Multiscale Methods for Networks 17 Ilya Safro, Clemson University

Large-scale networks, 10K<|V|<100M Multilevel algorithm is approximately 200-300 times faster than iterative combination of several solvers. Quality of the objective: Heavy-tail degree Ratios between distribution graphs Multilevel Alg and best combination of several solvers Sources: SNAP and UFL collections Multiscale Methods for Networks 18 Ilya Safro, Clemson University

Classification problems: Weighted SVM Multiscale Methods for Networks 19 Ilya Safro, Clemson University

Multilevel SVM and Weighted SVM Razzaghi, S “Scalable Multilevel Support Vector Machines”, ICCS 2015 minority majority Create approximate k-NN graph for each class or for their mixture Main ideas: Inherit support • vectors from the coarse level as training set Add some of their • neighborhood Inherit parameters for • model selection from the coarse level Multiscale Methods for Networks 20 Ilya Safro, Clemson University

Coarse Variables 1. Iterative selection of (several) independent set(s) Types of separate of nodes (similar to [Sakellaridi et al 2008]) coarsening 2. Strict coarsening (merging pairs of variables for two classes based on some distance function) 3. AMG coarsening (Galerkin for separate classes) Mutual 1. Yes. Formulate as fuzzy SVM coarsening 2. No. Formulate as regular (W)SVM for two classes? Multiscale Methods for Networks 21 Ilya Safro, Clemson University

Merging two classes: Probabilistic Support Vector Machines Multiscale Methods for Networks 22 Ilya Safro, Clemson University

Merging two classes: Probabilistic Support Vector Machine 0.5 + - 0.85 ( 0.54,0.46 ) 0.9 1 0.80 Multiscale Methods for Networks 23 Ilya Safro, Clemson University

Uncoarsening: Strict, AMG (separate classes) and Probabilistic WSVM (merged classes) Set of support vectors is relatively small Refinement: Training is performed by pairs of clusters of support vectors using libSVM Separating hyperplane Multiscale Methods for Networks 24 Ilya Safro, Clemson University

Matlab+ libSVM libSVM Time in seconds Model selection is applied • Comparable quality except Advertisement and Forest in which • AMG WSVM is better by 20% in G-mean Multiscale Methods for Networks 25 Ilya Safro, Clemson University

Without Algebraic Distance AMG SVM AMG WSVM Multiscale Methods for Networks 26 Ilya Safro, Clemson University

Network Generation and Modeling Practical task • Simulate and verify algorithms, policies, and scenarios on networks that can be created by similar processes Original network Artificial networks Artificial network This network has similar degrees, some eigenvalues, diameter but … is it really similar to the original network? US Western States Power Grid Watts, Strogatz 1998 Multiscale Methods for Networks 27 Ilya Safro, Clemson University

Properties that are preserved by most of the existing network generators (such as Chung-Lu, Stochastic Kronecker Graph and Block Two-Level Erdös–Rényi): • degree distribution • clustering coefficient • some eigenvalues • diameter, etc. Common algorithm: 1) start with empty or small graph 2) add some components at random, at the end preserving several properties. What makes the resulting graphs non-realistic? • These properties are different at different resolutions • Too many operations such as randomization and replication take us away from the realistic structure Multiscale Methods for Networks 28 Ilya Safro, Clemson University