gossip algorithms for solving laplacian systems
play

Gossip algorithms for solving Laplacian systems Anastasios Zouzias - PowerPoint PPT Presentation

Gossip algorithms for solving Laplacian systems Anastasios Zouzias University of Toronto joint work with Nikolaos Freris (EPFL) Based on : 1.Fast Distributed Smoothing for Clock Synchronization (CDC 12) 2.Randomized Extended Kaczmarz for


  1. Gossip algorithms for solving Laplacian systems Anastasios Zouzias University of Toronto joint work with Nikolaos Freris (EPFL) Based on : 1.Fast Distributed Smoothing for Clock Synchronization (CDC ‘12) 2.Randomized Extended Kaczmarz for Solving Least-Squares (Arxiv, May ‘12)

  2. Outline I. Problems: Solving Laplacian & edge-vertex systems II. Motivation: Clock Synchronization over WSNs III. Randomized Gossip Model & Averaging Problem IV. Gossip Solvers via Randomized (Extended) Kaczmarz

  3. Outline I. Problems: Solving Laplacian & edge-vertex systems II. Motivation: Clock Synchronization over WSNs III. Randomized Gossip Model & Averaging Problem IV. Gossip Solvers via Randomized (Extended) Kaczmarz

  4. Distributed solver : Laplacian system b 1 b 2 Laplacian System b 4 b 5 x 1 x 2 x 5 x 4 b 6 b 3 x 6 b 8 x 3 L b 9 b x = b 7 x 7 x 8 b 13 b 12 x 9 x 13 b 11 x 12 b 10 x 11 x 10 G =( V , E ): n nodes, m edges Model of computation • Each node is aware of its neighbors; Problem I exchanges packets with them only Input : Each node i gets b i • Static network; no communication Goal : Each node i computes errors; ignore numerical issues i th coordinate of x LS := L † b • Synchronous, asynchronous & Gossip

  5. Distributed solver : edge-vertex system Edge-vertex System 7 2 x 1 x 2 -5 x 5 7 8 x 4 3 -8 x 6 4 x 3 9 x 1 -2 1 -12 x 7 x 8 -1 x 9 3 -2 6 B x 13 -4 y x 12 = 2 x 11 x 10 G=(V,E): n nodes, m edges m × n Problem II  − 1 , if k = i ; Input : Each edge ( i , j ) gets y (i,j)  B ek := 1 , if k = j ; Goal : Each node i computes i th 0 , otherwise .  Normal equation of Bx = y is coordinate of x LS = B † y Laplacian system ( L = B ⊤ B )

  6. Outline I. Problems: Solving Laplacian & edge-vertex Systems II. Motivation: Clock Synchronization over WSNs III. Randomized Gossip Model & Averaging Problem IV. Gossip Solvers via Randomized (Extended) Kaczmarz

  7. Case Study: Clock Synchronization over WSNs Assumptions v • Each node has clock; same speed Neigh( v ) o v ( t ) = t + o v , o v ∈ R • Node v does not know o v Nodes can approx. relative offsets o uv = o v − o u for every u ∈ Neigh( v ) G=(V,E): n nodes, m edges Clock Synchronization Problem over all pairs Input: Estimates for all o uv = o uv + N (0 , 1) ( u, v ) ∈ E ˆ of nodes Goal: Compute offsets that min. (˜ o u ) u ∈ V o uv − o uv | 2 u,v E | ˜ max

  8. Tree-based Approach Idea : Build a spanning tree u Path u ~ v : diam(G) Every edge: normal error � o uv = ˆ ˆ o u ′ v ′ v ( u ′ ,v ′ ) ∈ P Sync error between u & v grows like ≈ O ( � diam(G)) In general, no hope for better accuracy ...but wireless networks are ``well-connected’’

  9. Modeling Wireless Networks ...as Random Geometric Graphs • n nodes uniform over square �� � log n • Connectivity [GK00] : r = O n �� � • Diameter: n O log n • Tree-based approach: error � O ( n 1 / 4 ) Square: unit area Q : Can we do better on Random Geometric Graphs? Yes ! Spatial Smoothing [KEES03,GK06]

  10. Spatial Smoothing Observation : Every loop in G : sum of relative offsets equals zero Idea: Incorporate the loop constraints How ? x Encode constraints in linear system: B Relative offset = B x = o of ( i, j ) o ( i,j ) m × n

  11. Properties of Least-Squares Gaussian error: compute LS solution of B x = ˆ o Thm [KEES03] Replace each edge by unit resistor. Then error variance between any pair of nodes u and v is : o uv − o uv | 2 ∼ R eff ( u, v ) E | ˜ Effective resistances of RGG bounded by O(1) [GK06] Tree-based vs Smoothing vs O ( n 1 / 4 ) O (1) Q : How to compute the LS solution?

  12. The Model Matters... ∂ Use coordinate descent: � B x − o � 2 = 0 ∂x u Synchronous Jacobi Asynchronous Jacobi o v = 0 , ∀ v ∈ V ˆ Each node regularly: For k = 1 , 2 , . . . v ∈ V • estimates relative offsets with nghbrs ← 1 � � � o ( k +1) o ( k ) o uv ˜ ˜ + ˆ v u • broadcasts its current offset d v ˆ o v u ∈ Neigh( v ) • updates its estimate: o v ← 1 k ≥ 4 m 2 Thm [GK06] : After � ˜ (˜ o u + ˆ o uv ) β 2 ln( � x ∗ � /ε ) d v rounds, it holds that u ∈ Neigh( v ) � x ( k ) − x ∗ � � It converges [BT89] 2 ≤ ε � � � where is the min-cut value β

  13. The Model Matters... Randomized Gossip Model (a.k.a. asynchronous time model) [BTA86,BGPS06] Each node u (randomly) activates itself Synchronous Model Asynchronous Model w.p. p u & performs local computation o v = 0 , ∀ v ∈ V ˆ Each node regularly: For k = 1 , 2 , . . . v ∈ V • estimates relative offsets with nghbrs ← 1 � � � o ( k +1) o ( k ) o uv ˜ ˜ + ˆ v u • broadcasts its current offset d v ˆ o v u ∈ Neigh( v ) • updates its estimate: all nothing o v ← 1 k ≥ 4 m 2 Thm [GK06] : After � ˜ (˜ o u + ˆ o uv ) β 2 ln( � x ∗ � /ε ) d v rounds, it holds that u ∈ Neigh( v ) It converges [BT89] � x ( k ) − x ∗ � � 2 ≤ ε � � � where is the min-cut value β

  14. Outline I. Problems: Solving Laplacian & edge-vertex systems II. Motivation: Clock Synchronization over WSNs III. Randomized Gossip Model & Averaging Problem IV. Gossip Solvers via Randomized (Extended) Kaczmarz

  15. Distributed Averaging How many rounds required to approx. 2.5 3 within ε ? Distributed Averaging: Input : Every node u gets w u 4 1 Goal: Every node want access 2.5 2 to global average 7 4 Gossip averaging algorithm 5 1.Every node u activates uniformly at random 2.Picks random neighbor v and averages their Basic primitive for current values w u , w v other functions Averaging can solve Problems I and II n [BGPS06] proved that rounds are [BDFSV10,XBL05,XBL06] O ( λ 2 ( G ) log( n/ε )) sufficient whp Special cases, complete graph [KSSV00,KDG03,KDN + 06]

  16. Gossip Model Assumptions • Each node u has independent Poisson time process: rate γ u • Each node activates when its arrival occurs • Equivalently * : single global Poisson process: rate � γ u • Arrivals correspond to rounds u ∈ V Claim : Non-uniform sampling of nodes is feasible with zero communication under gossip model (given ‘s) γ u Goal : Design and analyze gossip algorithms for Problem I and II *minimum of ind. Poisson is equivalent to single Poisson with sum of their rates

  17. Outline I. Problems: Solving Laplacian & edge-vertex systems II. Motivation: Clock Synchronization over WSNs III. Randomized Gossip Model & Averaging Problem IV. Gossip Solvers via Randomized (Extended) Kaczmarz

  18. Kaczmarz Method x · A ·· y = · · x ( m ) H m . . . · x (0) x (2) · x (1) · Kaczmarz Method (K) H 2 Initialize : � � � � A (1) , x x (0) = 0 H 1 = x | = y 1 Repeat: It convergences [K37] Set i k = k mod m + 1 A ( i k ) , x ( k ) � � x ( k +1) = x ( k ) + y i k − A ( i k ) � 2 Huge literature; many extensions; � � A ( i k ) � k = k + 1 rediscovered many times (Assumption: Ax = y has solution)

  19. Randomized Kaczmarz Method x Pick rows randomly · A ·· = y · · x ( m ) H m . . . · x (0) x (2) · x (1) · Randomized Kaczmarz(RK) [SV06] H 2 Initialize : x (0) = 0 � � � � A (1) , x H 1 = x | = y 1 Repeat: Exponential convergence 2 � A ( i ) � � Pick w.p. i k ∈ [ m ] � � p i ∝ � k 2 � 1 � � � x ( k ) − x ∗ � � x ∗ � 2 A ( i k ) , x ( k ) � � ≤ 1 − E � � x ( k +1) = x ( k ) + y i k − A ( i k ) κ 2 F ( A ) � � 2 � � A ( i k ) � � A � 2 where κ 2 F F ( A ) := k = k + 1 σ 2 min ( A ) (Assumption: Ax = y has solution)

  20. Let’s apply RK on Problems I and II

  21. RK Laplacian Solver b 1 b 2 x 1 x 2 b 5 Laplacian System x 5 b 4 x 4 b 6 b 3 x 6 x 3 b 8 b 9 b 7 x 7 x 8 L b 13 b x 9 x = b 12 x 13 b 11 x 12 x 11 b 10 x 10 Gossip Laplacian Solver Randomized Kaczmarz(RK) [SV06] Each node u : Initialize : x u = 0 x (0) = 0 Repeat: Repeat: 2 Node u activates w.p. d 2 � � L ( i ) � Pick w.p. u + d u i k ∈ [ n ] � � p i ∝ � • broadcasts θ = x u − 1 � L ( i k ) , x ( k ) � � sparsity x ( k +1) = x ( k ) + b i k − x ℓ L ( i k ) d u • sets x u ← x u + b u − d u θ � 2 of L ℓ ∈ N u � L ( i k ) � � 1 + d u k = k + 1 Every : x v ← x v + θ − b u /d u RK analysis & diag. preconditioning : v ∈ Neigh( u ) 1 + d u � rounds whp (Assumption: Lx = b has solution) O ( n/λ 2 2 ( G ))

  22. RK Edge-vertex Solver 7 2 x 1 x 2 -5 x 5 7 8 x 4 3 -8 x x 6 4 x 3 9 B 1 -2 y 1 -12 = x 7 x 8 -1 x 9 3 -2 6 x 13 -4 x 12 2 x 11 m × n x 10 Consistency assumption (limitation of RK) Gossip Edge-Vertex Solver Randomized Kaczmarz(RK) [SV06] Every node u : x u = 0 How to handle the general case? Repeat: Initialize : x (0) = 0 Node u activates w.p. d u & selects Repeat: random neighbor v Pick uniformly e = ( i, j ) ∈ E • sends x u & receives x v sparsity � B ( e ) , x ( k ) � x ( k +1) = x ( k ) + y e − • Performs: B ( e ) x u ← ( x u + x v + y ( u,v ) ) / 2 of B 2 k = k + 1 Similarly for node v RK analysis & diag. preconditioning : (Assumption: Bx = y has solution) � rounds whp O ( n/λ 2 ( G ))

Recommend


More recommend