Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms Message Passing Algorithms for Compressed Sensing DLD, Arian Maleki, Andrea Montanari Stanford September 2, 2009 DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms Outline DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms Compressed Sensing – the heuristic ◮ Real images and signals are compressible ◮ Equivalently: Few large coefficients, eg in Wavelet basis ◮ Fewer than nominal degrees of freedom not 10 6 pixels, just 10 4 wavelet coeffs, + positions of those coefficients ◮ Standard sampling: 10 6 measurements ◮ “Morally” c · 10 4 measurements should suffice DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms Compressed Sensing MRI ◮ ”Sparse MRI – The Application of Compressed Sensing in Magnetic Resonance Imaging” – Michael Lustig, DLD, John Pauly 2007, Magnetic Resonance in Medicine ◮ “Compressed Sensing MRI” – Michael Lustig, John Pauly, Juan Santos, DLD IEEE Signal Processing Special Issue on CS, March 2008 ◮ Inspired by CS theory ◮ Rapid Contrast-Enhanced 3D Angiography, ◮ Whole-Heart Coronary Imaging, ◮ Brain Imaging, and ◮ Dynamic Heart Imaging. DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms A Theoretical Formalization of Compressed Sensing Ingredients ◮ Sparsity. An object x 0 ∈ R N with k ≪ N nonzero coefficients in a fixed basis ◮ Random Undersampled Measurements Measure y = Ax 0 with random n by N matrix A (eg iid Gaussian). ◮ Nonlinear Reconstruction Attempt reconstruction with x 1 solving ( P 1 ) min � x � 1 subject to y = Ax AKA: Minimum ℓ 1 , Basis Pursuit. ◮ Computationally Feasible; compare NP-Hard: ( P 0 ) min � x � 0 subject to y = Ax ◮ Surprise: often in the n < N underdetermined case ( P 0 ) and ( P 1 ) will have the same, unique solution. ◮ Voluminous literature IEEE 2001-today. DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Phase Transition for ± Theorem . There is a function ρ CG ( δ, ± ) with the following characteristics: Fix δ > 0 . If n / N > ρ CG ( k / n ; ± )(1 + δ ) then with overwhelming probabilty for large n , N, x 1 = x 0 If n / N < ρ CG ( k / n , ± )(1 − δ ) , then with overwhelming probabilty for large n , N, x 1 � = x 0 . DLD (Discr. & Comput. Geom., 2006); DLD and Jared Tanner (JAMS, 2009) fully rigorous , explicit calculation of ρ CG using special functions. Methods: Combinatorial Geometry [Affentranger and Schneider (1992), Vershik and Sporyshev (1992)]. DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Empirical Results: ± Random Matrix A , δ = k / n , ρ = n / N , N = 400. 1 0.9 0.8 0.7 0.6 0.90 ρ =k/n 0.5 0.4 0.50 0.3 0.10 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 δ =n/N DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Nonnegative coefficients ◮ Underdetermined system of equations: y = Ax , x ≥ 0 . ◮ Sparsest Solution: ( NP + ) x 0 = argmin � x � 0 s.t. y = Ax , x ≥ 0 ◮ Problem: NP-hard in general. ◮ Relaxation: x 1 = argmin 1 ′ x s.t. y = Ax , ( LP + ) x ≥ 0 . Standard linear program DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Phase Transition for + Theorem There is a function ρ CG ( · , +) : (0 , 1] �→ (0 , 1] with the following characteristics: Fix ǫ > 0 . If n / N > ρ CG ( k / n , +)(1 + ǫ ) then with overwhelming probabilty for large n , N, x 1 = x 0 If n / N < ρ CG ( k / n , +)(1 − ǫ ) , then x 1 � = x 0 . DLD and Tanner (PNAS, 2005[a,b]) fully rigorous , explicit calculation of ρ CG using special functions. Methods: Combinatorial Geometry [Affentranger and Schneider (1992), Vershik and Sporyshev (1992)]. DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Empirical Results + Random Matrix A , δ = k / n , ρ = n / N . N = 400. 1 0.9 0.8 0.7 0.6 0.90 ρ =k/n 0.5 0.4 0.50 0.3 0.10 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 δ =n/N DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Object with k-Simple, Bounded Coefficients ◮ Underdetermined system of equations: y = Ax , − 1 ≤ x ( i ) ≤ 1 , 1 ≤ i ≤ N . ◮ Simplicity: # { i : | x ( i ) | � = 1 } small. ◮ Simplest Solution: ( NP � ) x 0 = argmin# { i : | x ( i ) | � = 1 } s.t. y = Ax , x ≥ 0 Problem: NP-hard in general. ◮ Relaxation: x 1 = argmin 1 ′ x s.t. y = Ax , ( LP � ) x ( i ) ∈ [ − 1 , 1] Standard linear program (feasibility problem) DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Phase Transition for � Set ρ CG ( δ, � ) = (2 − δ − 1 ) + . Fix ǫ > 0 . Theorem. If n / N > ρ CG ( k / n , � )(1 + ǫ ) then, with overwhelming probabilty for large n , N, x 1 = x 0 If n / N < ρ CG ( k / n , � )(1 − ǫ ) , then , with overwhelming probabilty for large n , N, x 1 � = x 0 . DLD and Jared Tanner (2008, in press D&CG) Exact finite n identities in Geometric Probability Methods: Wendel’s Theorem, Winder’s Theorem, Oriented Matroids. DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms Empirical Results � Random Matrix A , δ = k / n , ρ = n / N . N = 400. 1 0.9 0.90 0.8 0.7 0.6 0.50 ρ =k/n 0.5 0.4 0.3 0.2 0.10 0.1 0 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 δ =n/N DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Compressed Sensing Phase Transitions Object With k-Sparse Coefficients Simple Iterative Algorithms Object With k-sparse, Nonnegative Coefficients Heuristics Object with k-Simple, Bounded Coefficients Message Passing Algorithms The Three Theoretical PT’s 1 T C 0.9 I 0.8 0.7 0.6 ρ =k/n 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 δ =n/N DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing Green: +, Red ± , Blue � .
Compressed Sensing Phase Transitions Simple Iterative Algorithms Heuristics Message Passing Algorithms Fast Iterative Algorithms ◮ Problem: interesting problem sizes for MRI application: N ≈ 10 6 , n ≈ 10 4 . ◮ Generic LP at least n 2 · N complexity. ◮ Alternative: iterative algorithms involving C applications 10 < C < 50 of A and A ∗ to appropriate vectors ◮ Complexity: CnN in general eg A iid Gaussian; ◮ Complexity: CN log( N ) for FFT-based matrices. ◮ Heavily used for large-scale applications: Jean-Luc Starck (2003-), Miki Elad (2004-), .... DLD, Arian Maleki, Andrea Montanari Message Passing Algorithms for Compressed Sensing
Recommend
More recommend
