mmse approximation for the sparse prior using stochastic
play

MMSE Approximation for the Sparse Prior Using Stochastic Resonance - PowerPoint PPT Presentation

Feb 05, 2024 •6 likes •986 views

MMSE Approximation for the Sparse Prior Using Stochastic Resonance Dror Simon Computer Science, Technion, Israel January 15, 2020 Joint work with Jeremias Sulam, Yaniv Romano, Yue M. Lu and Michael Elad Dror Simon (Technion) MMSE for Sparse

  1. MMSE Approximation for the Sparse Prior Using Stochastic Resonance Dror Simon Computer Science, Technion, Israel January 15, 2020 Joint work with Jeremias Sulam, Yaniv Romano, Yue M. Lu and Michael Elad Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 1 / 46

  2. Noise Removal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  3. Noise Removal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  4. Noise Removal Why denoising? Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  5. Noise Removal Why denoising? A simple testing ground for novel concepts in signal processing. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  6. Noise Removal Why denoising? A simple testing ground for novel concepts in signal processing. Can be generalized to other, more complicated applications. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  7. Noise Removal Noisy Signal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  8. Noise Removal Noisy Signal Clean Estimate Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  9. Noise Removal Noisy Signal Signal Clean Restoration Estimate Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  10. Noise Removal Noisy Signal Signal Clean Restoration Estimate Prior Knowledge Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  11. Noise Removal – Bayesian Standpoint Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  12. Noise Removal – Bayesian Standpoint Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  13. Noise Removal – Bayesian Standpoint Signal Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  14. Noise Removal – Bayesian Standpoint Signal Approx. Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  15. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  16. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Optimal Signal Estimation Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  17. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Optimal Signal Approx. Estimation Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  18. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 5 / 46

  19. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 6 / 46

  20. The Generative Model D ∈ R n × m is a dictionary with normalized columns. m n D Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  21. The Generative Model Each element i in α is non zero with probability p i ≪ 1. m n D α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  22. The Generative Model The non-zero elements of the sparse representation, denoted by α s , are � � 0 , σ 2 sampled from a Gaussian distribution α s | s ∼ N . α I | s | m n D α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  23. The Generative Model The product D α leads to a signal x . m = n D x α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  24. The Generative Model We are given noisy measurements y = D α + ν , where ν is a white � � 0 , σ 2 Gaussian noise ν ∼ N . ν I n m = n D x α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  25. The Generative Model – Results 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  26. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  27. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  28. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  29. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). � � D s α s , σ 2 y | α s , s is Gaussian: y | α s , s ∼ N ν I n . 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  30. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). � � D s α s , σ 2 y | α s , s is Gaussian: y | α s , s ∼ N ν I n . � � 1 ν Q − 1 s D T s y , Q − 1 α s | y , s is Gaussian: α s | y , s ∼ N . σ 2 s Q s = 1 I | s | + 1 C s = σ 2 α D s D T s + σ 2 D T ν I n , s D s σ 2 σ ν α 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  31. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 9 / 46

  32. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  33. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  34. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  35. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. 2 The Maximum A-posteriori Probability (MAP) estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  36. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. 2 The Maximum A-posteriori Probability (MAP) estimator. 3 The Minimum Mean Square Error (MMSE) estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  37. The Bayesian Estimators Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

  38. The Bayesian Estimators Oracle Estimator = E { α s | s , y } = 1 α Oracle Q − 1 s D T � s y s σ 2 ν Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

  39. The Bayesian Estimators MAP Support Estimator � s MAP = arg max p ( s | y ) s Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

Recommend

Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse

Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse

Two-Level Approximation Error-Aware Trust-Region Model Management Numerical Experiments Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse Grids and Model Reduction Matthew J. Zahr Advisor: Charbel

612 views • 46 slides
Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS

Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS

Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS Paris, INRIA 17 november 2017 Joint work with Francis Bach and Alain Durmus. 1 Outline Introduction to Stochastic Approximation for Machine

725 views • 56 slides
Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop

Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop

Sparse Approximation Phase Transitions Matrix completion Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop on Sparsity, Compressed Sensing and Applications University of Oxford

655 views • 34 slides
Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH

Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH

Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH September 28, 2017 0 Outline 1. Introduction: Supervised Machine Learning Stochastic Approximation 2. Finite dimensional results 3. Infinite

1.25k views • 121 slides
Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA

Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA

Introduction Analysis of the SA scheme Multi-level stochastic approximation algorithms Numerical results Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA 28th October, 2013. Stochastic

746 views • 35 slides
Calmness in stochastic programming exact penalization and sample approximation techniques

Calmness in stochastic programming exact penalization and sample approximation techniques

Calmness in stochastic programming exact penalization and sample approximation techniques Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Stochastic

940 views • 69 slides
Recent Theoretical Advances in Sparse Approximation Joel A. Tropp

Recent Theoretical Advances in Sparse Approximation Joel A. Tropp

Recent Theoretical Advances in Sparse Approximation Joel A. Tropp <jtropp@ices.utexas.edu> Institute for Computational Engineering and Sciences The University of Texas at Austin Includes joint work with A. C. Gilbert, S.

1.17k views • 80 slides
Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI /

Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI /

Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI / CNRS - TELECOM ParisTech, France Stochastic approximation for adaptive

448 views • 44 slides
A regularized least-squares method for sparse low-rank approximation of multivariate functions

A regularized least-squares method for sparse low-rank approximation of multivariate functions

Workshop Numerical methods for high-dimensional problems April 18, 2014 A regularized least-squares method for sparse low-rank approximation of multivariate functions Mathilde Chevreuil joint work with Prashant Rai, Loic Giraldi, Anthony

1k views • 53 slides
On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse

On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse

On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse (TU Berlin) (joint work with Adam Andersson und Stig Larsson) Berlin-Padova Young Researchers Meeting Stochastic Analysis and Applications in

677 views • 32 slides
Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Stochastic Approximation and Stochastic Gradient Optimization Theory and Large Scale Systems Dual Effect and Discretization Stochastic Programming and Stochastic Optimal Control Some References P. Carpentier Master MMMEF Cours MNOS

385 views • 9 slides
Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation (approximation theory, learning Fourier coeffs, linear sketching, finite rate of innovation, compressed sensing...) Setup: Data/signal in n-dimensional

418 views • 25 slides
Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied

Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied

Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied Mathematics University of G ottingen in collaboration with Dennis Heinen, Armin Iske (Hamburg), Thomas Peter, Daniela Ro sca (Cluj-Napoca),

704 views • 54 slides
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

715 views • 68 slides
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

884 views • 71 slides
Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of

Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of

Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of Computer Science UT Austin MLSLP Symposium Portland, Oregon Sept 14, 2012 Joint work with C. Hsieh, M. Sustik and P. Ravikumar Inderjit S.

393 views • 28 slides
Approximation Algorithms for Geometric Proximity Problems: Introduction Background Approximation

Approximation Algorithms for Geometric Proximity Problems: Introduction Background Approximation

Approximation Algorithms for Geometric Proximity Problems: Introduction Background Approximation Part I: Approximating Euclidean MSTs Prior Work Preliminaries WSPD+MST Lower Bound In Theory David M. Mount Simple+Slow Smart+Sloppy

1.41k views • 126 slides
Gradient-based Sparse Approximation for Computed Tomography Elham Sakhaee , Manuel Arreola and

Gradient-based Sparse Approximation for Computed Tomography Elham Sakhaee , Manuel Arreola and

Gradient-based Sparse Approximation for Computed Tomography Elham Sakhaee , Manuel Arreola and and Alireza Entezari University of Florida esakhaee@cise.ufl.edu Tomographic Reconstruction Recover the image given X-ray measurements X-ray

731 views • 29 slides
Just Relax Convex Programming Methods for Subset Selection and Sparse Approximation Joel A.

Just Relax Convex Programming Methods for Subset Selection and Sparse Approximation Joel A.

Just Relax Convex Programming Methods for Subset Selection and Sparse Approximation Joel A. Tropp <jtropp@ices.utexas.edu> The University of Texas at Austin 1 Subset Selection Work in finite-dimensional inner-product space

511 views • 15 slides
Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende

Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende

Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende Fort Institut de Math ematiques de Toulouse CNRS Toulouse,

559 views • 29 slides
Stochastic approximation for speeding up LSTD/LSPI (and least squares regression/LinUCB) Prashanth

Stochastic approximation for speeding up LSTD/LSPI (and least squares regression/LinUCB) Prashanth

Stochastic approximation for speeding up LSTD/LSPI (and least squares regression/LinUCB) Prashanth L A Joint work with Nathaniel Korda and Rmi Munos INRIA Lille - Team SequeL MLRG - Oxford University November 24, 2014

1.13k views • 73 slides
Stochastic approximation based methods for computing the optimal thresholds in remote-state

Stochastic approximation based methods for computing the optimal thresholds in remote-state

Stochastic approximation based methods for computing the optimal thresholds in remote-state estimation with packet drops Jhelum Chakravorty Joint work with Jayakumar Subramanian and Aditya Mahajan McGill University American Control Conference

717 views • 46 slides
Bayesian Batch Active Learning as Sparse Subset Approximation Robert Pinsler Jonathan Gordon

Bayesian Batch Active Learning as Sparse Subset Approximation Robert Pinsler Jonathan Gordon

Bayesian Batch Active Learning as Sparse Subset Approximation Robert Pinsler Jonathan Gordon Eric Nalisnick Jos Miguel Hernndez-Lobato October 2019 Research Talk Introduction Acquiring labels for supervised learning can be costly

599 views • 40 slides
Stochastic Combinatorial Optimization via Poisson Approximation Jian Li, Wen Yuan Institute of

Stochastic Combinatorial Optimization via Poisson Approximation Jian Li, Wen Yuan Institute of

Stochastic Combinatorial Optimization via Poisson Approximation Jian Li, Wen Yuan Institute of Interdisiplinary Information Sciences Tsinghua University STOC 2013 lijian83@mail.tsinghua.edu.cn Outline Threshold Probability

480 views • 30 slides

More recommend