Hyperprior bayesian approach for inverse problems in imaging. Application to single shot HDR. Julie Delon with Cecilia Aguerrebere, Andr´ es Almansa, Yann Gousseau and Pablo Mus´ e 1 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? Capture a scene containing a large range of intensity levels... 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? ... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? ... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? ... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? ... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? ... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Teaser : what is High Dynamic Range Imaging (HDR) ? Limited dynamic range of the camera ! loss of details in bright and/or dark areas. 2 / 32
Motivation : High Dynamic Range Imaging (HDR) Usual approach for HDR image generation : fusion of mutiple exposures. HDR generation Irradiance Map (number of photons reaching each pixel per unit time) 3 / 32
Challenges of HDR imaging in dynamic scenes moving objects noise camera motion 4 / 32
Challenges of HDR imaging in dynamic scenes camera + object motion ghosting effect 5 / 32
Would it be possible to create a HDR image from a single shot ? 6 / 32
Would it be possible to create a HDR image from a single shot ? First, let’s focus on a very generic inverse problem... 6 / 32
A generic inverse problem Original image 7 / 32
A generic inverse problem Noise 7 / 32
A generic inverse problem Missing pixels 7 / 32
Inverse Problem Degradation model u = Au + n ˜ u reference image A is a diagonal operator Additive noise n may depend on u : Exemple RAW data (shot noise and readout noise) n ( x ) ∼ N (0 , α ( x ) u ( x ) + β ( x )) 8 / 32
Inverse Problem Degradation model u = Au + n ˜ Notation for patches Z i = p i (˜ u ) (degraded patch of size d = f ⇥ f centered at i ) C i = p i ( u ) (unknown reference patch) N i = p i ( n ) (additive noise patch) D i restriction of A to p i ( u ) Degradation model for a patch centered at pixel i Z i = D i C i + N i 8 / 32
Patch degradation Model Observed patch = x + 9 / 32
Patch degradation Model Observed patch = x + Assumptions : D is known N ⇠ N (0 , Σ N ), eventually depends on C but Cov ( N , C ) = 0 9 / 32
Patch degradation Model Observed Patch we seek to estimate patch = x + Gaussian prior for patches Assumptions : D is known N ⇠ N (0 , Σ N ), eventually depends on C but Cov ( N , C ) = 0 C ⇠ N ( µ, Σ ) with µ and Σ unknown 9 / 32
How to set Gaussian prior parameters µ and Σ ? Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] 1 Set of similar patches Z 1 , . . . , Z M , such that all the (unknown) C i follow the same law N ( µ, Σ ). X M X M µ = 1 1 b µ ] T and Σ = [ Z i � b µ ][ Z i � b b Z i M � 1 M i =1 i =1 10 / 32
How to set Gaussian prior parameters µ and Σ ? Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] 1 Set of similar patches Z 1 , . . . , Z M , such that all the (unknown) C i follow the same law N ( µ, Σ ). X M X M µ = 1 1 b µ ] T and Σ = [ Z i � b µ ][ Z i � b b Z i M � 1 M i =1 i =1 Not reliable when pixels are missing ! 10 / 32
How to set Gaussian prior parameters µ and Σ ? Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] 1 Set of similar patches Z 1 , . . . , Z M , such that all the (unknown) C i follow the same law N ( µ, Σ ). X M X M µ = 1 1 b µ ] T and Σ = [ Z i � b µ ][ Z i � b b Z i M � 1 M i =1 i =1 Not reliable when pixels are missing ! Gaussian Mixture Model prior on patches [EPLL - Zoran and Weiss 2 2011 ; PLE - Yu et al. 2012] 10 / 32
How to set Gaussian prior parameters µ and Σ ? MAP with an hyperprior on ( µ, Σ ) argmax p ( { C i } i , µ, Σ | { Z i } i ) = { C i } ,µ, Σ argmax p ( { Z i } | { C i } , µ, Σ ) . p ( { C i } | µ, Σ ) . p ( µ, Σ ) . { C i } ,µ, Σ Rappel Z i | C i , µ i , Σ i N ( D i C i , Σ N i ) ⇠ C i | µ i , Σ i N ( µ, Σ ) ⇠ ( µ, Σ ) ? Inclusion of hyperprior information compensates for missing pixels. 11 / 32
Hyperprior on ( µ, Σ ) Conjugate prior for a multivariate normal distribution Normal prior on the mean (conditionnal on the covariance) ⇣ ⌘ � κ µ | Σ ⇠ N ( µ 0 , Σ / κ ) / | Σ | − 1 2 exp 2 ( µ � µ 0 ) T Σ − 1 ( µ � µ 0 ) inverse Wishart prior on the covariance matrix ✓ ◆ � 1 Σ ⇠ IW ( ν Σ 0 , ν ) / | Σ | − ν + d +1 2 trace [ ν Σ 0 Σ − 1 ] exp 2 12 / 32
Hyperprior on ( µ, Λ ) with Λ = Σ � 1 (precision matrix) Conjugate prior for a multivariate normal distribution Normal prior on the mean (conditionnal on the covariance) ⇣ ⌘ � κ 1 µ | Λ ⇠ N ( µ 0 , Λ − 1 / κ ) / | Λ | 2 exp 2 ( µ � µ 0 ) T Λ ( µ � µ 0 ) Wishart prior on the inverse covariance matrix ✓ ◆ � 1 ν − d − 1 Λ ⇠ W (( ν Σ 0 ) − 1 , ν ) / | Λ | exp 2 trace [ ν Σ 0 Λ ] 2 13 / 32
Minimization with respect to { C i } i { Z i } set of similar patches; µ, Λ fixed. p ( { Z i } | { C i } ) . p ( { C i } | µ, Λ ) . p ( µ, Λ ) argmax { C i } Π M = argmax i =1 ( p ( Z i | C i ) . p ( C i | µ, Λ )) { C i } ⇣ ⌘ Π M = argmax g 0 , Σ Ni ( Z i � D i C i ) . g µ, Λ − 1 ( C i ) . i =1 { C i } Solution given by Wiener estimator for each i separately b C i = Λ − 1 D T ( D i Λ − 1 D T ) − 1 i + Σ N i ( Z i � D i µ ) + µ i | {z } | {z } E ( C i Z T E ( Z i Z T i ) i ) | {z } W i 14 / 32
Minimization with respect to µ, Λ { C i } i fixed. argmax p ( { Z i } | { C i } , µ, Λ ) . p ( { C i } | µ, Λ ) . p ( µ, Λ ) | {z } µ, Λ HYP : independent of µ, Λ p ( { C i } | µ, Λ ) . p ( µ, Λ ) ' argmax µ, Λ Π M = argmax i =1 g µ, Λ − 1 ( C i ) g µ 0 , Λ − 1 / κ ( µ ) w Λ 0 / ν , ν ( Λ ) . µ, Λ Explicit solution ( µ = MC + κ µ 0 b M + κ µ − µ 0 ) T + P M i =1 ( b µ )( b µ ) T Λ − 1 = ν Σ 0 + κ ( b b µ − µ 0 )( b C i − b C i − b ν + M − d 15 / 32
Loop in ( µ, Λ ) In the previous formula, ˆ µ and ˆ Λ depend on each other. Replacing ˆ C i , ˆ C i by its µ, ˆ expression in ( µ, Λ ) and reinjecting this in the formula of (ˆ Λ ), we get 0 1 − 1 0 1 M M X X @ κ Id + A @ A µ = b W j Z j + κ µ 0 W j D j j =1 j =1 X M ( ν + M � d ) b Λ − 1 ( W j ( Z j � D j µ ))( W j ( Z j � D j µ )) T = j =1 κ ( µ � µ 0 )( µ � µ 0 ) T + ν Σ 0 + with W j = Λ − 1 D T j ( D j Λ − 1 D T j + Σ N j ) − 1 . 16 / 32
Algorithm Initialization : compute Oracle image C oracle For k = 1 : maxit For each Patch Z Find patches similar to Z in C oracle 1 Compute µ 0 and Σ 0 from this set of similar patches in C oracle 2 µ , b Σ with a small loop and then ˆ Compute first b C . 3 Restore image from restored patches and update C oracle = restored image. 17 / 32
Initialization From PLE [Yu et al., 2012]: DCT (K-1) edges with K predefined models : + for isotropic different orientations patterns 18 / 32
Results Synthetic data, 70% missing pixels. 19 / 32
Results Synthetic data, 70% missing pixels. 19 / 32
Results Synthetic data, 70% missing pixels, gaussian noise σ = 10. 19 / 32
Results Synthetic data, 70% missing pixels, gaussian noise σ = 10. 19 / 32
Results Zoom on Real data. Left to right : Input low-resolution image, HBE, PLE, bicubic. 19 / 32
Now, how can we do HDR imaging from a single shot ? 20 / 32
Spatially Varying pixel Exposures (SVE) [Nayar and Mitsunaga, 2000] 21 / 32
1 image = N exposures 21 / 32
SVE Single-image HDR X No need for image alignment. X No need for motion detection. X No ghosting problems. X No large saturated regions to fill. ⇥ Resolution loss : unknown pixels to be restored (over and under exposed pixels). ⇥ Noise. ⇥ Need to modify the standard camera. I Alternative without camera modification [Hirakawa and Simon, 2011]. 22 / 32
SVE: Regular or Random? Random pattern to avoid aliasing [ Sch¨ oberl et al., 2012] 23 / 32
Inverse Problem for HDR Degradation model for a patch centered at pixel i Z i = D i C i + N i D i is a diagonal operator I D ii = 0 ⇒ over- or under-exposed pixel (ignored) I D ii = 1 ⇒ well-exposed pixel (kept) C i irradiance at pixel i (reference image) Noise model for RAW data (shot noise and readout noise) N i ⇠ N (0 , Σ N i ) with diagonal covariance matrix Σ N i such that ( Σ N i ) k = α k C k + β k , with α and β known. 24 / 32
Results HDR - Synthetic data 25 / 32
Recommend
More recommend
