3.26pt Randomized Projections for Corrupted Linear Systems Jamie - PowerPoint PPT Presentation

3.26pt

Randomized Projections for Corrupted Linear Systems Jamie Haddock 1 , Deanna Needell 2 1 Graduate Group in Applied Mathematics, University of California, Davis 2 Department of Mathematics, University of California, Los Angeles jhaddock@math.ucdavis.edu January 28, 2018 0

Problem Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. 1

Problem Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Ax = b + e , A ∈ R m × n , m >> n Problem: (Corrupted) Error ( e ): sparse, arbitrarily large entries x ∗ ∈ { x : Ax = b } Solution ( x ∗ ): 1

Problem Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Ax = b + e , A ∈ R m × n , m >> n Problem: (Corrupted) Error ( e ): sparse, arbitrarily large entries x ∗ ∈ { x : Ax = b } Solution ( x ∗ ): Applications: logic programming, error correction in telecommunications 1

Problem Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Ax = b + e , A ∈ R m × n , m >> n Problem: (Corrupted) Error ( e ): sparse, arbitrarily large entries x ∗ ∈ { x : Ax = b } Solution ( x ∗ ): Applications: logic programming, error correction in telecommunications Ax = b + e , A ∈ R m × n , m >> n Problem: (Noisy) Error ( e ): small, evenly distributed entries x LS ∈ argmin � Ax − b − e � 2 Solution ( x LS ): 1

Why not least-squares? x ∗ x LS 2

Randomized Kaczmarz RK 1. Start with initial guess x 0 b ik − a T ik x k 2. x k +1 = x k + � a ik � 2 a i k where i k ∈ [ m ] is chosen randomly 3. Repeat (2) x x 0 3

Randomized Kaczmarz RK 1. Start with initial guess x 0 b ik − a T ik x k 2. x k +1 = x k + � a ik � 2 a i k where i k ∈ [ m ] is chosen randomly 3. Repeat (2) x 1 x x 0 3

Randomized Kaczmarz RK 1. Start with initial guess x 0 b ik − a T ik x k 2. x k +1 = x k + � a ik � 2 a i k where i k ∈ [ m ] is chosen randomly 3. Repeat (2) x 2 x 1 x x 0 3

Randomized Kaczmarz RK 1. Start with initial guess x 0 b ik − a T ik x k 2. x k +1 = x k + � a ik � 2 a i k where i k ∈ [ m ] is chosen randomly 3. Repeat (2) x 2 x 1 x 3 x x 0 3

Randomized Kaczmarz RK 1. Start with initial guess x 0 b ik − a T ik x k 2. x k +1 = x k + � a ik � 2 a i k where i k ∈ [ m ] is chosen randomly 3. Repeat (2) Theorem (Strohmer-Vershynin, 2008) If Ax = b is consistent and RK is used with P [ i k = j ] = � a j � 2 / � A � 2 F then iterates converge linearly in expectation with � k � 1 E � x k − x � 2 ≤ � x 0 − x � 2 . 1 − � A � 2 F � A − 1 � 2 4

Proposed Method Goal: Use RK to detect the corrupted equations with high probability. 5

Proposed Method Goal: Use RK to detect the corrupted equations with high probability. Lemma (H.-Needell) Let ǫ ∗ = min i ∈ [ m ] | Ax ∗ − b | i = | e i | and suppose | supp ( e ) | = s. If 2 ǫ ∗ we have that the d ≤ s indices || a i || = 1 for i ∈ [ m ] and || x − x ∗ || < 1 of largest magnitude residual entries are contained in supp ( e ) . That is, we have D ⊂ supp ( e ) , where � D = argmax | Ax − b | i . D ⊂ [ A ] , | D | = d i ∈ D 5

Proposed Method Goal: Use RK to detect the corrupted equations with high probability. x k x ∗ 5

Proposed Method Goal: Use RK to detect the corrupted equations with high probability. x k x ∗ We call ǫ ∗ / 2 the detection horizon . 5

Proposed Method Method 1 Windowed Kaczmarz 1: procedure WK ( A , b , k , W , d ) S = ∅ 2: for i = 1 , 2 , ... W do 3: x i k = k th iterate produced by RK with x 0 = 0, A , b . 4: D = d indices of the largest entries of the residual, | Ax i k − b | . 5: S = S ∪ D 6: return x , where A S C x = b S C 7: 6

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 1, S = ∅ x ∗ H 1 H 2 H 3 x 1 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 1, S = ∅ x 1 1 x ∗ H 1 H 2 H 3 x 1 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 2, i = 1, S = { 7 } x 1 2 x 1 1 x ∗ H 1 H 2 H 3 x 1 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 2, S = { 7 } x ∗ H 1 H 2 H 3 x 2 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 2, S = { 7 } x 2 1 x ∗ H 1 H 2 H 3 x 2 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 2, i = 2, S = { 7 , 5 } x 2 1 x ∗ x 2 2 H 1 H 2 H 3 x 2 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 3, S = { 7 , 5 } x ∗ H 1 H 2 H 3 x 3 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 1, i = 3, S = { 7 , 5 } x ∗ H 1 x 3 H 2 1 H 3 x 3 0 H 4 H 5 H 6 H 7 7

Example WK( A , b , k = 2 , W = 3 , d = 1 ): j = 2, i = 3, S = { 7 , 5 , 6 } x 3 2 x ∗ H 1 x 3 H 2 1 H 3 x 3 0 H 4 H 5 H 6 H 7 7

Example Solve A S C x = b S C . x ∗ H 1 H 2 H 3 H 4 7

Theoretical Guarantees Lemma (H.-Needell) Let ǫ ∗ = min i ∈ [ m ] | Ax ∗ − b | i = | e i | and suppose | supp ( e ) | = s. Assume that || a i || = 1 for all i ∈ [ m ] and let 0 < δ < 1 . Define � δ ( ǫ ∗ ) 2 � � log � 4 || x ∗ || 2 k ∗ = . σ 2 � min ( A supp ( e ) C ) � log 1 − m − s Then in window i of the Windowed Kaczmarz method, the iterate produced by the RK iterations, x i k ∗ satisfies � k ∗ k ∗ − x ∗ || ≤ 1 � m − s � 2 ǫ ∗ � || x i ≥ p := (1 − δ ) P . m 8

Theoretical Guarantee Values (Gaussian A ∈ R 50000 × 100 ) � � δ ( ǫ ∗ )2 � � log k ∗ = 4 || x ∗|| 2 σ 2 � � min( A supp( e ) C ) log 1 − m − s 9

Theoretical Guarantee Values (Correlated A ∈ R 50000 × 100 ) � � δ ( ǫ ∗ )2 � � log k ∗ = 4 || x ∗|| 2 σ 2 � � min( A supp( e ) C ) log 1 − m − s 10

Theoretical Guarantee Values (Gaussian A ∈ R 50000 × 100 ) � k ∗ � 2 ǫ ∗ � � k ∗ − x ∗ || ≤ 1 m − s || x i ≥ p := (1 − δ ) P m 1 s = 1 s = 10 s = 50 0.8 s = 100 s = 200 s = 300 s = 400 0.6 p 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 11

Theoretical Guarantee Values (Correlated A ∈ R 50000 × 100 ) � k ∗ � 2 ǫ ∗ � � k ∗ − x ∗ || ≤ 1 m − s || x i ≥ p := (1 − δ ) P m 12

Theoretical Guarantees Theorem (H.-Needell) Assume that || a i || = 1 for all i ∈ [ m ] and let 0 < δ < 1 . Suppose d ⌋ and k ∗ is as given in lemma 2. Then the d ≥ s = | supp ( e ) | , W ≤ ⌊ m − n Windowed Kaczmarz method on A , b will detect the corrupted equations (supp ( e ) ⊂ S) and the remaining equations given by A [ m ] − S , b [ m ] − S will have solution x ∗ with probability at least � k ∗ � W � � m − s p W := 1 − 1 − (1 − δ ) . m 13

Theoretical Guarantee Values (Gaussian A ∈ R 50000 × 100 ) � k ∗ � W � � m − s p W := 1 − 1 − (1 − δ ) m 1 s = 1 s = 10 s = 50 0.8 s = 100 s = 200 s = 300 s = 400 0.6 p W 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 14

Experimental Values (Gaussian A ∈ R 50000 × 100 ) 1 s = 100 s = 200 s = 500 0.9 s = 750 s = 1000 0.8 Success ratio 0.7 0.6 0.5 0.4 0 0.2 0.4 0.6 0.8 1 15

Experimental Values (Gaussian A ∈ R 50000 × 100 ) 1 s = 100 s = 200 s = 500 0.8 s = 750 s = 1000 Success ratio 0.6 0.4 0.2 0 0 500 1000 1500 2000 k 16

Experimental Values (Gaussian A ∈ R 50000 × 100 ) 17

Experimental Values (Gaussian A ∈ R 50000 × 100 ) 18

Theoretical Guarantee Values (Correlated A ∈ R 50000 × 100 ) � k ∗ � W � � m − s p W := 1 − 1 − (1 − δ ) m 19

Experimental Values (Correlated A ∈ R 50000 × 100 ) 20

Experimental Values (Correlated A ∈ R 50000 × 100 ) 21

Conclusions and Future Work • randomized projection methods are able to detect corruption • often experimental results far outperform theoretical guarantees • performance on real data • reduce dependence on artificial parameters 22

The End Thanks! Questions? E. Amaldi, P. Belotti, and R. Hauser. Randomized relaxation methods for the maximum feasible subsystem problem. In International Conference on Integer Programming and Combinatorial Optimization, pages 249–264. Springer, 2005. N. Jamil, X. Chen, and A. Cloninger. Hildreths algorithm with applications to soft constraints for user interface layout. Journal of Computational and Applied Mathematics, 288:193–202, 2015. S. Kaczmarz. Angen¨ aherte aufl¨ osung von systemen linearer gleichungen. Bull.Internat.Acad.Polon.Sci.Lettres A, pages 335–357, 1937. T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. Journal of Fourier Analysis and Applications, 15(2):262–278, 2009. 23