Statistical Image Models Eero Simoncelli Howard Hughes Medical - PowerPoint PPT Presentation

Joint densities adjacent near far other scale other ori 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0 � 50 � 50 � 50 � 50 � 50 � 100 � 100 � 100 � 100 � 100 � 150 � 150 � 150 � 150 � 150 � 100 0 100 � 100 0 100 � 100 0 100 � 500 0 500 � 100 0 100 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0 � 50 � 50 � 50 � 50 � 50 � 100 � 100 � 100 � 100 � 100 � 150 � 150 � 150 � 150 � 150 � 100 0 100 � 100 0 100 � 100 0 100 � 500 0 500 � 100 0 100 • Nearby: densities are approximately circular/elliptical • Distant: densities are approximately factorial [Simoncelli, ‘97; Wainwright&Simoncelli, ‘99]

ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10 kurtosis 8 8 8 6 6 6 4 4 4 0 ! /4 ! /2 3 ! /4 0 ! /4 ! /2 3 ! /4 0 ! /4 ! /2 3 ! /4 ! ! ! orientation data (ICA’d): sphericalized: factorialized:

ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10 kurtosis • Local densities are elliptical (but non-Gaussian) 8 8 8 6 6 6 • Distant densities are factorial 4 4 4 0 ! /4 ! /2 3 ! /4 0 ! /4 ! /2 3 ! /4 0 ! /4 ! /2 3 ! /4 ! ! ! orientation data (ICA’d): sphericalized: factorialized: [Wegmann&Zetzsche ‘90; Simoncelli ’97; + many recent models]

Spherical vs LTF 0.2 blk blk blk 0.4 blk size = 3x3 blk size = 7x7 blk size = 11x11 spherical 0.2 spherical spherical 0.35 factorial factorial factorial 0.15 0.3 0.15 0.25 0.1 0.2 0.1 0.15 0.05 0.1 0.05 0.05 0 0 0 3 6 9 12 15 18 20 3 6 9 12 15 18 20 3 6 9 12 15 18 20 kurtosis kurtosis kurtosis 3x3 7x7 15x15 data (ICA’d): sphericalized: factorialized: • Histograms, kurtosis of projections of image blocks onto random unit-norm basis functions. • These imply data are closer to spherical than factorial [Lyu & Simoncelli 08]

non-Gaussian elliptical observations and models of natural images: - Zetzsche & Krieger, 1999; - Huang & Mumford, 1999; - Wainwright & Simoncelli, 2000; - Hyvärinen and Hoyer, 2000; - Parra et al., 2001; - Srivastava et al., 2002; - Sendur & Selesnick, 2002; - Teh et al., 2003; - Gehler and Welling, 2006 - Lyu & Simoncelli, 2008 - etc.

Modeling heteroscedasticity Method 2: Hidden scaling variable for each patch Gaussian scale mixture (GSM) [Andrews & Mallows 74]: x = √ z � � u • is Gaussian, � z > 0 u � • and are independent z u • is elliptically symmetric, with covariance ∝ C u � x � • marginals of are leptokurtotic x [Wainwright&Simoncelli 99]

GSM - prior on z • Empirically, z is approximately lognormal [Portilla etal, icip-01] p z ( z ) = exp ( − (log z − µ l ) 2 / (2 σ 2 l )) l ) 1 / 2 z (2 πσ 2 • Alternatively, can use Jeffrey’s noninformative prior [Figueiredo&Nowak, ‘01; Portilla etal, ‘03] p z ( z ) ∝ 1 /z

GSM simulation GSM simulation Image data ! ! #" #" " " #" #" ! !" " !" ! !" " !" [Wainwright & Simoncelli, NIPS*99]

Model III (GSM) Coefficient density: Basis set: Image: � � � � ������� � � �

Original coefficients Normalized by √ z � 2 � 4 � 4 � 5 marginal Log probability Log probability � 6 [Ruderman&Bialek 94] � 6 � 7 � 8 � 8 � 9 � 10 � 500 0 500 � 10 � 5 0 5 100 8 6 50 joint [Schwartz&Simoncelli 01] 4 0 2 � 50 0 � 100 0 2 4 6 8 � 100 � 50 0 50 100 subband

6 Model Encoding Cost (bits/coeff) Model Encoding cost (bits/coeff) 5.5 5 5 4.5 4 4 3 3.5 3 2 First Order Ideal Gaussian Model Conditional Model Generalized Laplacian 2.5 1 1 2 3 4 5 6 3 4 5 Empirical Conditional Entropy Empirical First Order Entropy (bits/coeff) [Buccigrossi & Simoncelli 99]

Bayesian denoising • Additive Gaussian noise: y = x + w P ( y | x ) ∝ exp[ − ( y − x ) 2 / 2 σ 2 w ] • Bayes’ least squares solution is conditional mean: x ( y ) = I ˆ E( x | y ) � = dx P ( y | x ) P ( x ) x/ P ( y )

I. Classical If signal is Gaussian, BLS estimator is linear: x ) denoised (ˆ σ 2 x ( y ) = ˆ x · y σ 2 x + σ 2 n => suppress fine scales, noisy ( y ) retain coarse scales

Non-Gaussian coefficients " #" -*./01.*,6'.)07+48 94:..'41,;*1.')5,, 2+0343'(')5 ! % #" ! $ #" ! !"" " !"" &'()*+,-*./01.* [Burt&Adelson ‘81; Field ‘87; Mallat ‘89; Daugman ‘89; etc]

II. BLS for non-Gaussian prior • Assume marginal distribution [Mallat ‘89] : P ( x ) ∝ exp −| x/s | p • Then Bayes estimator is generally nonlinear: p = 1.0 p = 0.5 p = 2.0 [Simoncelli & Adelson, ‘96]

MAP shrinkage p=2.0 p=1.0 p=0.5 [Simoncelli 99]

Denoising: Joint � dz P ( z | � I E( x | � y ) = y ) I E( x | � y, z ) � dz P ( z | �  zC u ( zC u + C w ) − 1 �   = y ) y  ctr where y T ( zC u + C w ) − 1 � y ) = P ( � y | z ) P ( z ) y | z ) = exp( − � y/ 2) P ( z | � , P ( � � (2 π ) N | zC u + C w | P � y Numerical computation of solution is reasonably efficient if one jointly diagonalizes C u and C w ... [Portilla, Strela, Wainwright, Simoncelli, ’03] IPAM, 9/04 20

Example estimators ESTIMATED COEFF. !" +'1&2/1+3)*%+,,- " ! w ! !" #" !" " " NOISY COEFF. ! #" ! !" $%&'()./0+$1 $%&'()*%+,,- Estimators for the scalar and single-neighbor cases [Portilla etal 03]

Comparison to other methods "'& "'& " " ,456748+91:;<= /456,(74-.)/-0123 ! "'& ! "'& :>6965#8*>?6< ! ! ! ! ! !'& ! !'& ! # ! # ! #'& ! #'& )89 ! :1:;<=>? ! $ ! $ .=9@A?-9?=@BC8D ! $'& ! $'& !" #" $" %" &" !" #" $" %" &" ()*+,-*.)/-0123 ()*+,-*.)/-0123 Results averaged over 3 images [Portilla etal 03]

Noisy Original (22.1 dB) Matlab’s BLS-GSM wiener2 (30.5 dB) (28 dB)

Noisy Original (8.1 dB) UndWvlt BLS-GSM Thresh (21.2 dB) (19.0 dB)

Real sensor noise 400 ISO denoised

GSM summary • GSM captures local variance • Underlying Gaussian leads to simple computation • Excellent denoising results • What’s missing? • Global model of z variables [Wainwright etal 99; Romberg etal ‘99; Hyvarinen/Hoyer ‘02; Karklin/ Lewicki ‘02; Lyu/Simoncelli 08] • Explicit geometry: phase and orientation

Global models for z • Non-overlapping neighborhoods, tree-structured z [Wainwright etal 99; Romberg etal ’99] z � u Coarse scale Fine scale • Field of GSMs: z is an exponentiated GMRF, is � u a GMRF, subband is the product [Lyu&Simoncelli 08]

State-of-the-art denoising Lena Boats " " # # ! " ! " ∆ ()*+, ∆ ()*+, ! $ ! $ ! ' ! ' ! & ! & ! "#"! $! !# %! "## ! "#"! $! !# %! "## σ σ FoGSM BM3D kSVD GSM FoE [Lyu&Simoncelli, PAMI 08]

Measuring Orientation 2-band steerable pyramid: Image decomposition in terms of multi-scale gradient measurements [Simoncelli et.al., 1992; Simoncelli & Freeman 1995]

Multi-scale gradient basis

Multi-scale gradient basis • Multi-scale bases: efficient representation

Multi-scale gradient basis • Multi-scale bases: efficient representation • Derivatives: good for analysis • Local Taylor expansion of image structures • Explicit geometry (orientation)

Multi-scale gradient basis • Multi-scale bases: efficient representation • Derivatives: good for analysis • Local Taylor expansion of image structures • Explicit geometry (orientation) • Combination: • Explicit incorporation of geometry in basis • Bridge between PDE / harmonic analysis approaches

orientation magnitude orientation [Hammond&Simoncelli 06; cf. Oppenheim and Lim 81]

Importance of local orientation Randomized orientation Randomized magnitude [Hammond&Simoncelli 05]

Reconstruction from orientation Original Quantized to 2 bits • Reconstruction by projections onto convex sets • Resilient to quantization [Hammond&Simoncelli 06]

Image patches related by rotation two-band steerable pyramid coefficients [Hammond&Simoncelli 06]

raw rotated patches patches PCA of normalized gradient patches --- Raw Patches Rotated Patches [Hammond&Simoncelli 06]

Orientation-Adaptive GSM model Model a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variables [Hammond&Simoncelli 06]

Orientation-Adaptive GSM model Model a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variables Conditioned on ; is zero mean gaussian with covariance [Hammond&Simoncelli 06]

Estimation of C( θ ) from noisy data noisy patch unknown, approximate by measured from noisy data. Assuming independent and noise rotationally invariant (assuming w.l.o.g. E[z] =1 ) [Hammond&Simoncelli 06]

Bayesian MMSE Estimator [Hammond&Simoncelli 06]

Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]

Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]

Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate [Hammond&Simoncelli 06]

Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate has covariance separable prior for hidden variables [Hammond&Simoncelli 06]