Compressive Imaging EE367/CS448I: Computational Imaging and Display - PowerPoint PPT Presentation

Compressive Imaging � EE367/CS448I: Computational Imaging and Display � stanford.edu/class/ee367 � Lecture 11 � Gordon Wetzstein � Stanford University �

Motivation � • whiteboard derivations of square matrix, over-determined, under- determined � • least-norm solution �

Single Pixel Camera � Wakin et al. 2006

Single Pixel Camera � original � 10% � 5% � 2% � Wakin et al. 2006

Single Pixel Camera – Image Formation � = , b A = , … … measurements � measurement matrix � = ,

Beyond Shannon Nyquist � • Shannon: highest frequency in image with NxN pixels is N/2 � • compressive sensing: only need M<N measurements! � • magic? à no, relies on sparsity of sampled image � • what’s sparsity? number of non-zero elements in image � • a signal is k-sparse if there are k non-zero elements �

Natural Images are Sparse (in some transform domain) � Haar wavelets � discrete cosine transform � wikipedia cnx.org

Natural Images are Sparse (in some transform domain) � ∇ y x ∇ x x x

Natural Images are Sparse (in some transform domain) � better: isotropic � easier: anisotropic � 2 + ∇ y x 2 + ( ) ( ) ( ) ( ) 2 2 ∇ x x ∇ x x ∇ y x x

Compressive Sensing – Synthesis Problem � b = Ax = A Ψ s • general idea: � optics � computation � Ψ b s A

Compressive Sensing – Analysis Problem � we will use “K” as the matrix transforming the signal into some (sparse) domain � minimize Kx 1 { } x b = Ax subject to

Compressive Sensing – Analysis Problem � minimize Kx 1 { } x 2 ≤ ε b = Ax b − Ax 2 subject to or � basis pursuit � basis pursuit denoise �

Basis Pursuit Denoise / LASSO � 2 + λ Kx 1 1 2 b − Ax 2 minimize Kx 1 minimize { } { } x x 2 ≤ ε b − Ax 2 subject to λ • there is a that makes both problems equivalent, but we don’t know what that is à manually tune parameters �

Regularized Image Reconstruction � 2 + λ Γ x 1 ( ) 2 b − Ax 2 minimize { } x data fidelity some image prior, such as ℓ 1 term � norm or others �

Regularized Image Reconstruction � 1 ( ) 2 b − Ax 2 2 + λ Γ z minimize % { } ! " # # $ x g ( z ) f ( x ) Kx − z = 0 subject to • split into two parts à mathematically equivalent �

Regularized Image Reconstruction � ) + ρ ( ) = f ( x ) + g ( z ) + y T Kx − z ( 2 Kx − z 2 2 L ρ x , z , y • Augmented Lagrangian �

Regularized Image Reconstruction � repeat until converged � 2 + ρ 1 ( ) = argmin ( ) = argmin x ← 2 Ax − b 2 2 Kx − v , v = z − u prox ⋅ 2 , ρ v L ρ x , z , y { } { } x x ( ) + ρ ( ) = argmin ( ) = argmin z ← λ Γ z 2 v − z , v = Kx + u prox Γ , ρ v L ρ x , z , y { } { } z z u ← u + Kx − z • iterative updates - ADMM �

Regularized Image Reconstruction � repeat until converged � 2 + ρ 1 ( ) = argmin ( ) = argmin x ← 2 Ax − b 2 2 Kx − v , v = z − u prox ⋅ 2 , ρ v L ρ x , z , y { } { } x x ( ) + ρ ( ) = argmin ( ) = argmin z ← λ Γ z 2 v − z , v = Kx + u prox Γ , ρ v L ρ x , z , y { } { } z z u ← u + Kx − z • iterative updates - ADMM �

Regularized Image Reconstruction � 2 + ρ 1 ( ) = argmin 2 Ax − b 2 2 Kx − v prox ⋅ 2 , ρ v { } x … see whiteboard / notes ... �

Regularized Image Reconstruction � 2 + ρ 1 ( ) = argmin 2 Ax − b 2 2 Kx − v prox ⋅ 2 , ρ v { } x … see whiteboard / notes ... � − 1 ⎛ ⎞ ⎛ ⎞ ( ) = A T A + ρ K T K A T b + ρ K T v ⎜ ⎟ ⎜ ⎟ prox ⋅ 2 , ρ v " $ $ # $$ % " $ # $ % ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ! & A b ! x = b " • x-update: solve � A • symmetric, positive definite matrix à conjugate gradient method � �

Regularized Image Reconstruction � ( ) + ρ ( ) = argmin λ Γ z 2 v − z prox Γ , ρ v { } z … depends on prior ... �

Regularized Image Reconstruction � ( ) + ρ ( ) = argmin λ Γ z 2 v − z prox Γ , ρ v { } z TV prior � λ z 1 + ρ ( ) = argmin ( ) 2 v − z = S λ / ρ v prox ⋅ 1 , ρ v { } z ⎡ ⎤ D x ⎢ ⎥ K = D = • also: � ⎢ ⎥ D y � ⎣ ⎦

Regularized Image Reconstruction � ( ) + ρ ( ) = argmin λ Γ z 2 v − z prox Γ , ρ v { } z NLM prior � ⎛ ⎞ ( ) = NLM v , λ prox NLM, ρ v ⎜ ⎟ ρ ⎝ ⎠ K = I • also: � variance variance of denoising � �

Applied to Single Pixel Camera �

Applied to Single Pixel Camera �

Applied to Single Pixel Camera – TV Prior � 1x � 2x � 4x � 8x �

Applied to Single Pixel Camera – NLM Prior � 1x � 2x � 4x � 8x �

Applications of Compressive Imaging �

Compressive Medical Imaging � • reduce acquisition time, radiation exposure, or allow for more patients in same time, … � • examples: x-ray computed tomography and MRI �

Computed Tomography (CT) � This slide has a 16:9 media window Image: Wikipedia � x-ray sensor � 3D Reconstruction � x-ray source � Reconstructed 2D Slices �

Computed Tomography – Fourier Slice Theorem � This slide has a 16:9 media window • measurements = Fourier slices � • compressive CT: e.g. fewer slices � primal domain � frequency domain �

Magnetic Resonance Imaging � • measurements = (random) Fourier coefficients � • compressive MRI: fewer Fourier coefficients � wikipedia � frequency domain �

Compressive Imaging: CT & MRI � • people in bio-medical imaging often hesitant about priors: � • few guarantees for success � • if reconstruction breaks, not clear how exactly � • is that feature a reconstruction artifact or the thing I’m looking for? �

Compressive Hyperspectral Imaging � λ • y motivation: � x • conventional: either scan over xy or over lambda! � • idea: capture hyperspectral datacube with a single, coded image – use compressive sensing to reconstruct � • first approach: CASSI (coded aperture snapshot spectral imager), Wagadarikar 2008 �

Compressive Hyperspectral Imaging � Arce et al. 2014

Compressive Hyperspectral Imaging � Arce et al. 2014

Compressive Hyperspectral Imaging � • moderate quality for snapshot, but good quality for coded multi-shot � • applications: remote sensing, cultural heritage, … � Arce et al. 2014

Compressive Light Field Imaging �

Integral Imaging – Lippmann 1908 � lenslet � f sensor � Fixed trade-off between spatial and angular resolution! �

� � � � Scene from � Integral Imaging � Above � # Sensor Pixels Y � Lenslet Array � # Sensor Pixels X � [Lippman 1908], [Adelson and Wang 1992], [Ng et al. 2005] �

� � Scene from � Integral Imaging � Above �

� � � Scene from � Integral Imaging: Spatio-Angular Resolution Tradeoff! � Spatio-Angular Resolution Tradeoff! � Above � # Sensor Pixels Y / # Views Y # Sensor Pixels X / # Views X

� � Scene from � Key Insight: Light Field is Redundant! � Above �

Compressive Light Field Photography � ACM SIGGRAPH 2013 � Exploit Redundancy � Sparse Representation Optimal Optical Setup Computationally � Sparse � Overcomplete Dictionaries � Mask-based � Reconstruction Light Field Atoms Light Field Coding

Optical Preservation of Light Field = Image Light field Dictionary Coefficient vector Coded projection � = Overcomplete dictionary Light field atoms We need to be able to distinguish atoms from their projections �

� Scene from Previous Mask-Coded Light Field Projection � Above Parallax Barriers � Sum of Sinusoids or MURA � [Ives 1903] � [Veeraraghavan 2007, Lanman 2008] � • Multiplexing + linear reconstruction � • Low resolution light fields similar to the lenslets design � “On Plenoptic Multiplexing and Reconstruction”, IJCV, Wetzstein et al. 2013 �

� Scene from Proposed Technology: Optimized Mask w.r.t. the Dictionary � Above = Image Dictionary Coefficients Coded projection � = Overcomplete dictionary Light field atoms • Multiplexing + nonlinear reconstruction � • Higher spatial resolution �

Coded Attenuation Mask � Polarizing � Imaging � Beamsplitter Virtual sensor Lens LCoS Camera Image sensor

Compressive Light Field Photography � ACM SIGGRAPH 2013 � Captured 2D Image � Mask-Coded Projection � = 4D Light Field � b = Φ x = ΦΨ s Sparse Coefficients! � 4D Reconstruction � Basis Pursuit Denoise: � minimize s 1 s 2 ≤ ε b − ΦΨ s 2 subject to

Compressive Light Field Photography � ACM SIGGRAPH 2013 � Captured 2D Image � • massively parallel à move to cloud?! � • better: replace with convolutional sparse coding! �

Compressive Imaging Everywhere � • metamaterials � • THz imaging � • x-ray imaging � • thermal IR � • ultra-fast imaging � • not as much on compressive coherent imaging (could be interesting for course projects: OCT, holography, …) � • … �