Statistical Models of Images, with Application to Denoising and - PowerPoint PPT Presentation

Statistical Models of Images, with Application to Denoising and Texture Synthesis Eero Simoncelli Center for Neural Science, and Courant Institute of Mathematical Sciences New York University http://www.cns.nyu.edu/ ∼ eero

Statistical Image Models Applications in Image Processing / Graphics: • Compression • Restoration • Enhancement • Synthesis Theoretical Neurobiology: • Ecological optimality principle for early visual processing • Adaptation / plasticity / learning SIAM, 3/02 1

Model I: Gaussian 6 10 5 10 4 10 Power 3 10 2 10 1 10 0 10 0 1 2 3 10 10 10 10 Spatial−frequency (cycles/image) Power spectra of natural images fall as 1 / f α , α ∼ 2. [Field ’87, Ruderman & Bialek ’94, etc] SIAM, 3/02 2

Bandpass Filters Reveal non-Gaussian Behaviors 0 10 Response histogram Gaussian density Probability -2 10 -4 10 500 0 500 Filter Response Marginal densities of bandpass filtered images are non-Gaussian. [Field87, Mallat89] . SIAM, 3/02 3

Optimizing non-Gaussianity Linear operators with maximally independent (or maximally non-Gaussian) re- sponses are oriented bandpass filters [Bell/Sejnowski ’97; Olshausen/Field ’96] SIAM, 3/02 4

Sample Kurtosis vs. Filter Bandwidth 16 14 12 Sample Kurtosis 10 8 6 4 0 0.5 1 1.5 2 2.5 3 Filter Bandwidth (octaves) For most images, maximum is near one octave [after Field, 1987] . SIAM, 3/02 5

Separable Wavelets • Basis functions are bandpass filters, related by translation, dilation, modu- lation. • Orthogonal. • Lacking translation- and rotation-invariance. SIAM, 3/02 6

Steerable Pyramid • Basis functions are oriented bandpass filters, related by translation, dilation, rotation (directional derivatives, order K − 1). • Tight frame, 4 K / 3 overcompleteness for K orientations. • Translation-invariant, rotation-invariant. [Freeman & Adelson, ’90; Simoncelli et.al., ’91; Freeman & Simoncelli ’95] SIAM, 3/02 7

Steerable Pyramid: Block Diagram H 0 (- ω ) H 0 ( ω ) L 0 (- ω ) B 0 (- ω ) L 0 ( ω ) B 0 ( ω ) B 1 ( ω ) B 1 (- ω ) B K-1 (- ω ) B K-1 ( ω ) L(- ω ) 2 ↓ 2 ↑ L( ω ) Lowpass band is recursively split using central diagram (gray box). SIAM, 3/02 8

Model II: Wavelet Marginals Boats Lena Toys Goldhill −100 −50 0 50 100 −100 −50 0 50 100 −100 −50 0 50 100 −100 −50 0 50 100 p = 0 . 62 p = 0 . 56 p = 0 . 52 p = 0 . 60 ∆ H = 0 . 014 ∆ H = 0 . 013 ∆ H = 0 . 021 ∆ H = 0 . 0019 • Coefficient densities well fit by generalized Gaussian [Mallat ’89; Simoncelli/Adelson ’96] : f ( c ) ∝ e −| c / s | p , p ∈ [ 0 . 5 , 0 . 8 ] . • Non-Gaussianity due to both image content and choice of basis. SIAM, 3/02 9

Coefficient Dependency Large-magnitude subband coefficients are found at neighboring positions, ori- entations, and scales. SIAM, 3/02 10

Wavelet Conditional Histogram 1 1 0.6 0.6 0.2 0.2 -40 0 40 -40 0 40 50 40 0 -40 0 -40 40 • Conditional mean is zero • But, conditional variance grows with amplitude of L 2 SIAM, 3/02 11

Conditional Histograms Strength of dependency is different for each pair of filters: But the form of dependency is highly consistent across a wide range of images. SIAM, 3/02 12

Model III: Local GSM model Model generalized neighborhood of coefficients as a Gaussian Scale Mixture (GSM) [Andrews & Mallows ’74] : x = √ z � � u , where - z and � u are independent - � x | z is Gaussian, with covariance zC u - marginals are always leptokurtotic - we choose a flat (non-informative) prior on log ( z ) [Wainwright & Simoncelli, ’99] SIAM, 3/02 13

GSM Simulations 500 500 500 500 0 0 0 0 image data −500 −500 −500 −500 −500 0 500 −500 0 500 −500 0 500 −500 0 500 500 500 500 500 0 0 0 0 model sim −500 −500 −500 −500 −500 0 500 −500 0 500 −500 0 500 −500 0 500 Conditional Histograms of pairs of coefficients with different spatial separa- tions. SIAM, 3/02 14

Denoising I (Gaussian model) y = x + w , where w is Gaussian, white. y is an observed transform coefficient. Bayes least squares solution: σ 2 x IE ( x | y ) = y σ 2 x + σ 2 w SIAM, 3/02 15

Denoising II (marginal model) p = 0 . 5 p = 1 . 0 p = 2 . 0 Bayes least squares solution: � dx P ( y | x ) P ( x ) x IE ( x | y ) = � dx P ( y | x ) P ( x ) No closed-form expression with generalized Gaussian prior, but numerical com- putation is reasonably efficient. [Simoncelli & Adelson, ’96] SIAM, 3/02 16

Denoising III (GSM model) � IE ( x | � y ) = dz P ( z | � y ) IE ( x | � y , z ) � zC u ( zC u + C w ) − 1 � � � = dz P ( z | � y ) y ctr where y T ( zC u + C w ) − 1 � P ( � y | z ) P ( z ) y | z ) = exp ( − � y / 2 ) P ( z | � y ) = � dz P ( � y | z ) P ( z ) , P ( � ( 2 π ) N | zC u + C w | � Numerical computation of solution is reasonably efficient if one jointly diago- nalizes C u and C w ... [Portilla et.al., ’01] SIAM, 3/02 17

Denoising Simulation: Face noisy I-linear (4.8) (10.61) III-GSM II-marginal nbd: 5 × 5 + p (11.98) (13.60) - Semi-blind (all parameters estimated except for σ w ). - All methods use same steerable pyramid decomposition. - SNR (in dB) shown in parentheses. SIAM, 3/02 18

Denoising Simulation: Fingerprint noisy original (8.1) soft thresholding GSM (17.5) (21.2) - PSNR shown in parentheses. - Both methods use same steerable pyramid decomposition. - Joint statistics capture oriented structures. SIAM, 3/02 19

Denoising Comparison 16 14 12 10 8 6 4 2 0 5 10 15 20 25 30 35 40 45 50 PSNR improvement as a function of noise level, averaged over three images: - squares: GSM - triangles: MatLab wiener2 , optimized neighborhood [Lee, ’80] - circles: soft thresholding, optimized threshold [Donoho, ’95] SIAM, 3/02 20

Example Texture Types structured random periodic 2nd-order Can we derive a statistical model (and sampling technique) to represent all of these? SIAM, 3/02 21

Synthesis: Gaussian model Captures periodicity. SIAM, 3/02 22

Synthesis: Wavelet marginal model Captures some local structure. [Heeger & Bergen, ’95] SIAM, 3/02 23

Synthesis: GSM model [Portilla & Simoncelli, ’00] SIAM, 3/02 24

Credits Local Gaussian Scale mixtures: Martin Wainwright (MIT) Global Tree Model: Martin Wainwright & Allan Willsky (MIT) Denoising: Javier Portilla (U. Granada), Vasily Strela (Drexel U.), Martin Wainwright (MIT) Texture Analysis/Synthesis: Javier Portilla (U. Granada) Compression: Robert Buccigrossi (U Pennsylvania) Funding provided by the National Science Foundation, the Alfred P. Sloan Foundation, and the Howard Hughes Medical Institute. SIAM, 3/02 25