Information-Theoretically Optimal Compressed Sensing via Spatial - PowerPoint PPT Presentation

Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing Adel Javanmard with David Donoho and Andrea Montanari Stanford University November 10, 2012 Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 1 / 28

General problem y ❂ Ax ✰ noise ❀ m y ❂ A x ✰ w n n ◮ x high-dimensional but highly structured ◮ How many linear measurements are needed? Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 2 / 28

Normalization ✦ w ✘ N ✭ 0 ❀ ✛ 2 I m ✂ m ✮ ✦ m ❀ n ✦ ✶ , m ❂ n ❂ ✍ ✦ A ❂ ❬ A 1 ❥ ✁ ✁ ✁ ❥ A n ❪ ❦ A i ❦ 2 ❂ ✂✭ 1 ✮ Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 3 / 28

Compressed sensing: Basic insights Donoho, Candés, Romberg, Tao, Indyk, Gilbert, . . . [2005-. . . ] Structure ✦ ❦ x ❦ 0 ✔ k adversarial ✦ m ❂ C k log ✭ n ❂ k ✮ Rate Reconstruction ✦ Convex optimization ✦ Random isotropic vectors Measurements MSE ✔ C ✛ 2 Robustness ✦ Is this the optimal compression rate? Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 4 / 28

Compressed sensing: Basic insights Donoho, Candés, Romberg, Tao, Indyk, Gilbert, . . . [2005-. . . ] Structure ✦ ❦ x ❦ 0 ✔ k adversarial ✦ m ❂ C k log ✭ n ❂ k ✮ Rate Reconstruction ✦ Convex optimization ✦ Random isotropic vectors Measurements MSE ✔ C ✛ 2 Robustness ✦ Is this the optimal compression rate? Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 4 / 28

This paper Structure ✦ x ❂ x discr ✰ x other ; ❦ x other ❦ 0 ✔ k oblivious Rate ✦ m ❂ k ✰ o ✭ n ✮ Reconstruction ✦ Bayesian AMP Measurements ✦ Spatially coupled matrices MSE ✔ C ✭ x ✮ ✛ 2 Robustness ✦ Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 5 / 28

This paper Structure ✦ x ❂ x discr ✰ x other ; ❦ x other ❦ 0 ✔ k oblivious Rate ✦ m ❂ k ✰ o ✭ n ✮ Reconstruction ✦ Bayesian AMP Measurements ✦ Spatially coupled matrices MSE ✔ C ✭ x ✮ ✛ 2 Robustness ✦ Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 5 / 28

Outline ◮ A toy example (random signal). ◮ Results. ◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP ◮ Proof technique. ◮ State evolution ◮ Supercooling. Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

Outline ◮ A toy example (random signal). ◮ Results. ◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP ◮ Proof technique. ◮ State evolution ◮ Supercooling. Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

Outline ◮ A toy example (random signal). ◮ Results. ◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP ◮ Proof technique. ◮ State evolution ◮ Supercooling. Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

Outline ◮ A toy example (random signal). ◮ Results. ◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP ◮ Proof technique. ◮ State evolution ◮ Supercooling. Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

A toy example (random signal) Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 7 / 28

A toy example (random signal) x ❂ ✭ x 1 ❀ ✿ ✿ ✿ ❀ x n ✮ ❀ x i ✘ i ✿ i ✿ d ✿ p X ❀ y ✷ R m ❀ y ❂ Ax ❀ p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ p X is known! Non-universal! Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

A toy example (random signal) x ❂ ✭ x 1 ❀ ✿ ✿ ✿ ❀ x n ✮ ❀ x i ✘ i ✿ i ✿ d ✿ p X ❀ y ✷ R m ❀ y ❂ Ax ❀ p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ p X is known! Non-universal! Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

A toy example (random signal) x ❂ ✭ x 1 ❀ ✿ ✿ ✿ ❀ x n ✮ ❀ x i ✘ i ✿ i ✿ d ✿ p X ❀ y ✷ R m ❀ y ❂ Ax ❀ p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ p X is known! Non-universal! Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

How many measurements are needed? p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ ◮ Classical compressed sensing: m ❂ 0 ✿ 97 n ✰ o ✭ n ✮ (Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust) ◮ This talk: m ❂ 0 ✿ 1 n ✰ o ✭ n ✮ (non-universal, robust) Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

How many measurements are needed? p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ ◮ Classical compressed sensing: m ❂ 0 ✿ 97 n ✰ o ✭ n ✮ (Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust) ◮ This talk: m ❂ 0 ✿ 1 n ✰ o ✭ n ✮ (non-universal, robust) Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

How many measurements are needed? p X ❂ 0 ✿ 2 ✍ 0 ✰ 0 ✿ 3 ✍ 1 ✰ 0 ✿ 2 ✍ � 1 ✰ 0 ✿ 2 ✍ 3 ✰ 0 ✿ 1 Uniform ✭ � 2 ❀ 2 ✮ ✿ ◮ Classical compressed sensing: m ❂ 0 ✿ 97 n ✰ o ✭ n ✮ (Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust) ◮ This talk: m ❂ 0 ✿ 1 n ✰ o ✭ n ✮ (non-universal, robust) Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

What is 0 ✿ 1 here? Definition (Renyi’s Information Dimension) For X ✘ p X , let ❤ X ✐ m ❂ ❜ 2 m X ❝ ❂ 2 m be an m -digits rounding of X H ✭ ❤ X ✐ m ✮ d ✭ X ✮ ✑ lim sup ✿ m m ✦✶ Alternative characterization: ✎ If p X ❂ ✭ 1 � ✧ ✮ ✁ discrete ✰ ✧ ✁ abs. continuous, then d ✭ X ✮ ❂ ✧ . Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 10 / 28

What is 0 ✿ 1 here? Definition (Renyi’s Information Dimension) For X ✘ p X , let ❤ X ✐ m ❂ ❜ 2 m X ❝ ❂ 2 m be an m -digits rounding of X H ✭ ❤ X ✐ m ✮ d ✭ X ✮ ✑ lim sup ✿ m m ✦✶ Alternative characterization: ✎ If p X ❂ ✭ 1 � ✧ ✮ ✁ discrete ✰ ✧ ✁ abs. continuous, then d ✭ X ✮ ❂ ✧ . Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 10 / 28

Why is this important? Theorem (Verdú, Wu, 2010) Under mild regularity hypotheses, non-adaptive compressed sensing is possible if and only if m ❃ d ✭ X ✮ n ✰ o ✭ n ✮ ✿ (equivalently, ✍ ❃ d ✭ X ✮ ✰ o ✭ 1 ✮ ). Shannon-theoretic argument. Exhaustive-search reconstruction :-( Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 11 / 28

Results Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 12 / 28

Two tricks ◮ ‘Spatially coupled’ sensing matrix. [Kudekar, Pfister, 2010] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011] ◮ AMP reconstruction, Posterior-expectation denoiser [Donoho, Maleki, Montanari 2009] ◮ Spatial coupling + MP reconstruction [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [no proof :-(] Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 13 / 28

Two tricks ◮ ‘Spatially coupled’ sensing matrix. [Kudekar, Pfister, 2010] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011] ◮ AMP reconstruction, Posterior-expectation denoiser [Donoho, Maleki, Montanari 2009] ◮ Spatial coupling + MP reconstruction [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [no proof :-(] Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 13 / 28

Our contributions ◮ Construction ◮ A rigorous proof ◮ Beyond random signals ◮ Robustness Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 14 / 28