On The Foundations of Computational Photography Yohann Tendero - PowerPoint PPT Presentation

On The Foundations of Computational Photography Yohann Tendero Joint work with: Jean-Michel Morel Bernard Roug´ e January,  th 

Photographing moving scenes could only be done using short exposure times, until...

Amit Agrawal, one of the The flutter shutter inventors of the flutter camera. shutter method. Anat Levin, one of the The motion-invariant inventors of the photography camera. motion-invariant photography method.

Input Photo Deblurred Result Agrawal et al. ''Resolving Objects at Higher Resolution from a Single Motion-Blurred Image'', CVPR, 2007.

Traditional Camera : Shutter is OPEN

Flutter Shutter Camera : the Shutter OPENS /CLOSES

Short Exposure Long Exposure Coded Exposure Our result Matlab Lucy Ground Truth

Gain Time Standard camera Standard camera Gain Time Alternate Alternate Gain Time Random Random Random Gain Time Agrawal et al. Agrawal et al. Code Code

➓ Optimal code: J. Jelinek. “Designing the optimal shutter sequences for the flutter shutter imaging method.” 2010. ➓ Gain: The gain (RMSE) of a flutter shutter is bounded above ❜ 1 � σ 2 by J and “The gain for computational imaging is r significant only when the average signal level J is considerably smaller than the read noise variance σ 2 r ” O. Cossairt, M. Gupta, and S.K. Nayar. “When Does Computational Imaging Improve Performance?” 2012.

Overview ➓ What are the flutter shutter acquisition formulae? ➓ Main question: What is the MSE? ➓ Consequence 1) Optimal flutter shutter code for known velocity Byproduct: optimal temporal filter for blind motion blur deconvolution ➓ Consequence 2) Optimal snapshot theory ➓ Consequence 3) Paradox and its solution: An optimal (MSE) aperture theory for random velocity models

Image Model ➓ ∆ t length of a time interval ➓ u ✏ ✶ r✁ 1 2 s ✝ g ✝ l ideal 2 , 1 observable landscape. Assumption: r✁ π, π s band limited and u P L 1 ♣ ❘ q ❳ L 2 ♣ ❘ q A “∆ t snapshot” at a pixel at position n is a Poisson random variable ✂➺ ∆ t P l ♣r 0 , ∆ t s ✂ r n ✁ 1 2 , n � 1 ✡ 2 sq ✒ P u ♣ n q dt . 0 X ✒ P ♣ λ q , P ♣ X ✏ k q ✏ λ k e ✁ λ . k !

Image Model ➓ ∆ t length of a time interval ➓ v relative velocity (unit: pixels per second) ➓ u ✏ ✶ r✁ 1 2 s ✝ g ✝ l ideal 2 , 1 observable landscape. Assumption: r✁ π, π s band limited and u P L 1 ♣ ❘ q ❳ L 2 ♣ ❘ q A “∆ t snapshot” at a pixel at position n is a Poisson random variable ✂➺ ∆ t P l ♣r 0 , ∆ t s ✂ r n ✁ 1 2 , n � 1 ✡ 2 sq ✒ P u ♣ n ✁ vt q dt . 0 X ✒ P ♣ λ q , P ♣ X ✏ k q ✏ λ k e ✁ λ . k !

The Numerical Flutter Shutter Setup 1. The camera takes a burst of L images using exposure time ∆ t ; 2. The k -th elementary image is assigned a numerical weight α k P ❘ ; 3. All images are added together to get one observed image. 9 1.5 8 1 7 6 0.5 5 0 4 −0.5 3 2 −1 1 −1.5 0 5 10 15 20 25 30 35 40 45 50 −4 −3 −2 −1 0 1 2 3 4 A non positive flutter The modulus of its Fourier shutter function . transform.

The Agrawal et al. code. The binary flutter shutter function for the optimized Agrawal et al. code. L ✁ 1 Code: ♣ α 0 , ..., α L ✁ 1 q P ❘ L ô Flutter shutter function : α ♣ t q ✏ ➳ α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q . k ✏ 0

L ✁ 1 Code: ♣ α 0 , ..., α L ✁ 1 q P ❘ L ô Flutter shutter function : α ♣ t q ✏ ➳ α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q . k ✏ 0 Definition ✁➩ ♣ k � 1 q ∆ t ✠ ➓ Numerical samples: obs ♣ n q ✒ ➦ L ✁ 1 u ♣ n ✁ vt q dt k ✏ 0 α k P . k ∆ t � 1 ➓ Analog samples ( α ♣ t q P r 0 , 1 s ): obs ♣ n q ✒ P ✟ v ♣ α ♣ . v q ✝ u q♣ n q . ➓ Band limited interpolate: obs ♣ x q ✒ ➦ n P ❩ obs ♣ n q sinc ♣ x ✁ n q . Continuous numerical flutter shutter : any function α P L 2 ♣ ❘ q . Velocity v : unit in pixel(s) per ∆ t .

Observed Images Varying the Code Agrawal et al. code. Random uniform on The motion-invariant r✁ 1 , 1 s code. photography code.

Inverse Filter Design Flutter shutter Numerical Analog α ♣ t q ✏ ➦ L ✁ 1 Flutter shutter k ✏ 0 α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q α ♣ t q P r 0 , 1 s function α ♣ t q (with α k P ❘ and ∆ t → 0) � 1 � . � . 1 ✟ ✟ ✟ ❊ ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v α v v (observed) � 1 v α 2 � . � . ✟ ✟ 1 ✟ var ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v v (observed) ✶ r✁ π,π s ♣ ξ q ✶ r✁ π,π s ♣ ξ q Inverse filter ˆ γ ♣ ξ q α ♣ ξ v q ˆ α ♣ ξ v q ˆ

Deconvolved Varying the Code Code: Agrawal et al. . Code: random Code: RMSE ✏ 2 . 54 uniform on r✁ 1 , 1 s . motion-invariant RMSE ✏ 2 . 25 photography . RMSE ✏ 2 . 31

Usual Codes: Agrawal et al. 30 25 20 15 10 5 0 −4 −3 −2 −1 0 1 2 3 4 The binary flutter shutter function The Fourier transform (modulus) of for the optimized Agrawal et al. the flutter shutter function with the code. Agrawal et al. code. ✁ ξ ∆ t ✁ i ξ ∆ t α ♣ t q ✏ ➦ L ✁ 1 ✠ ➦ L ✁ 1 k ✏ 0 α k e ✁ ik ξ ∆ t . k ✏ 0 α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q α ♣ ξ q ✏ sinc ˆ e 2 2 π

Poisson noise. Deconvolved Poisson noise using the Agrawal et al. code.

Flutter Shutter Formalism Summary Flutter shutter type Numerical Analog α ♣ t q ✏ ➦ L ✁ 1 k ✏ 0 α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q α ♣ t q P r 0 , 1 s Flutter shutter function α ♣ t q (with α k P ❘ and ∆ t → 0) ✁ ✁ ✠ ✠ ✁ ✠ 1 . 1 . ❊ ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v α v v (observed) ✁ v α 2 ✁ 1 ✠ ✠ 1 ✁ ✠ . . var ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v v (observed) ✶ r✁ π,π s♣ ξ q ✶ r✁ π,π s♣ ξ q Inverse filter ˆ γ ♣ ξ q α ♣ ξ v q ˆ α ♣ ξ v q ˆ ❊ ♣ ˆ ✉ est ♣ ξ qq ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q (deconvolved) ⑥ α ⑥ 2 L 2 ♣ ❘ q⑥ u ⑥ L 1 ⑥ α ⑥ L 1 ⑥ u ⑥ L 1 var ♣ ˆ ✉ est ♣ ξ qq ✶ r✁ π,π s ♣ ξ q ✶ r✁ π,π s♣ ξ q α ♣ ξ v q⑤ 2 α ⑤ 2 ♣ ξ v q ⑤ ˆ ⑤ ˆ (deconvolved) ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 2 ♣ ❘ q ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 1 ♣ ❘ q 1 1 ➩ π ➩ π MSE d ξ d ξ 2 π ✁ π α ♣ ξ v q⑤ 2 2 π ✁ π α ♣ ξ v q⑤ 2 ⑤ ˆ ⑤ ˆ

Flutter Shutter Formalism Summary Flutter shutter type Numerical Analog α ♣ t q ✏ ➦ L ✁ 1 Flutter shutter k ✏ 0 α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q α ♣ t q P r 0 , 1 s function α ♣ t q (with α k P ❘ and ∆ t → 0) � 1 � . � . ✟ ✟ 1 ✟ ❊ ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v α v v (observed) ✁ v α 2 ✁ 1 . ✠ ✠ 1 ✁ . ✠ var ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v v (observed) ✶ r✁ π,π s♣ ξ q ✶ r✁ π,π s♣ ξ q Inverse filter ˆ γ ♣ ξ q α ♣ ξ v q ˆ α ♣ ξ v q ˆ ❊ ♣ ˆ ✉ est ♣ ξ qq ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q (deconvolved) ⑥ α ⑥ 2 L 2 ⑥ u ⑥ L 1 ⑥ α ⑥ L 1 ⑥ u ⑥ L 1 var ♣ ˆ ✉ est ♣ ξ qq ✶ r✁ π,π s♣ ξ q ✶ r✁ π,π s ♣ ξ q α ♣ ξ v q⑤ 2 α ⑤ 2 ♣ ξ v q ⑤ ˆ ⑤ ˆ (deconvolved) ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 2 ♣ ❘ q ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 1 ♣ ❘ q 1 ➩ π 1 ➩ π MSE d ξ d ξ 2 π ✁ π α ♣ ξ v q⑤ 2 2 π ✁ π α ♣ ξ v q⑤ 2 ⑤ ˆ ⑤ ˆ

Flutter Shutter Formalism Summary Flutter shutter type Numerical Analog α ♣ t q ✏ ➦ L ✁ 1 Flutter shutter k ✏ 0 α k ✶ r k ∆ t , ♣ k � 1 q ∆ t r ♣ t q α ♣ t q P r 0 , 1 s function α ♣ t q (with α k P ❘ and ∆ t → 0) ✁ ✁ ✠ ✠ ✁ ✠ 1 . 1 . ❊ ♣ obs ♣ n qq v α ✝ u ♣ n q v ♣ α ✝ u q♣ n q v v (observed) ✁ v α 2 ✁ ✠ ✠ ✁ ✠ 1 . 1 . var ♣ obs ♣ n qq ✝ u ♣ n q v ♣ α ✝ u q♣ n q v v (observed) ✶ r✁ π,π s ♣ ξ q ✶ r✁ π,π s ♣ ξ q Inverse filter ˆ γ ♣ ξ q α ♣ ξ v q ˆ α ♣ ξ v q ˆ ❊ ♣ ˆ ✉ est ♣ ξ qq ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q ˆ u ♣ ξ q ✶ r✁ π,π s ♣ ξ q (deconvolved) ⑥ α ⑥ 2 L 2 ⑥ u ⑥ L 1 ⑥ α ⑥ L 1 ⑥ u ⑥ L 1 var ♣ ˆ ✉ est ♣ ξ qq ✶ r✁ π,π s♣ ξ q ✶ r✁ π,π s ♣ ξ q α ♣ ξ v q⑤ 2 α ⑤ 2 ♣ ξ v q ⑤ ˆ ⑤ ˆ (deconvolved) ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 2 ♣ ❘ q ⑥ u ⑥ L 1 ♣ ❘ q⑥ α ⑥ L 1 ♣ ❘ q 1 ➩ π 1 ➩ π MSE d ξ d ξ ✁ π α ♣ ξ v q⑤ 2 ✁ π α ♣ ξ v q⑤ 2 2 π 2 π ⑤ ˆ ⑤ ˆ