An introduction to mathematical models in image processing with a - PowerPoint PPT Presentation

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system An introduction to mathematical models in image processing with a focus on PDEs and optimization techniques Yifei Lou Department of Mathematical Sciences, UTD October 10, 2014 1/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Outline ABCs of image processing 1 Image and PDEs 2 Forward/backward diffusion Imaging through turbulence Image and optimization 3 TV regularization Bregman iterations Optimization and dynamic differential system 4 Inverse scale space flow Forward-backward splitting 2/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Outline ABCs of image processing 1 Image and PDEs 2 Forward/backward diffusion Imaging through turbulence Image and optimization 3 TV regularization Bregman iterations Optimization and dynamic differential system 4 Inverse scale space flow Forward-backward splitting 3/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Image representation Grayscale digital image 8 bits per sampled pixel = 256 different intensities Value 255 = white, 0 = black, and in-between is a shade of gray Resolution: number of detectors/CCD sensors Figure: Decreasing resolution 4/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Some applications–denoising 5/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Some applications–deblurring 6/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Some applications–inpainting and segmentation 7/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system 1992 LA Riots 8/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Outline ABCs of image processing 1 Image and PDEs 2 Forward/backward diffusion Imaging through turbulence Image and optimization 3 TV regularization Bregman iterations Optimization and dynamic differential system 4 Inverse scale space flow Forward-backward splitting 9/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Forward diffusion methods Heat equation u ( 0 ) = u 0 , u t = △ u , t > 0 . 10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Forward diffusion methods Heat equation u ( 0 ) = u 0 , u t = △ u , t > 0 . 10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Forward diffusion methods Anisotropic diffusion (Rudin-Osher-Fatemi 1992) � ∇ u � u t = ∇ · |∇ u | Diffuse along edges ⇒ edge-preserving 10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Backward diffusion methods Backward heat equation is unstable u t = −△ u . 11/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Backward diffusion methods Backward heat equation is unstable u t = −△ u . A balanced diffusion-sharpening diffusion by Alvarez and Mazorra in 1994 u t = u ξξ − sign ( u ηη ) |∇ u | , for the gradient direction η , and normal direction ξ . 11/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Sobolev gradient flow In 2009, Calder-Mansouri-Yezzi propose Sobolev gradient diffusion. 12/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Sobolev gradient flow In 2009, Calder-Mansouri-Yezzi propose Sobolev gradient diffusion. The heat equation is the gradient descent on the functional E ( u ) = 1 � �∇ u � 2 , 2 Ω with respect to the L 2 metric. 12/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Sobolev gradient flow They consider an inner product on the Sobolev space H 1 0 (Ω) � v , w � − → g λ ( v , w ) = ( 1 − λ ) � v , w � L 2 + λ � v , w � H 1 , for any λ > 0. The gradient of E ( u ) w.r.t. g λ is given by ∇ g λ E | u = −△ ( Id − λ △ ) − 1 u , The PDE u t = ±∇ g λ E | u are stable for both forward and backward directions. 13/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion Sobolev gradient flow The backward direction can be used for image sharpening. They propose the following energy � 2 � � � Ω �∇ u � 2 E s ( u ) = 1 �� �∇ u 0 � 2 Ω �∇ u 0 � 2 − α , � 4 Ω where u 0 is the initial value and α is a scale. The gradient descent to minimize such energy is � � � Ω �∇ u � 2 △ ( Id − △ ) − 1 u . u t = Ω �∇ u 0 � 2 − α � 14/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion (a) input (b) ROF (c) AM (d) SOB 15/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Imaging through turbulence 1 Two effects of turbulence blurry image frames temporal oscillations 1 Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Imaging through turbulence 1 Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction 1 Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Imaging through turbulence 1 Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction Sharpen individual frame (backward diffusion): shock filter/Sobolev gradient flow. Stabilize temporal direction (forward diffusion): linear/anisotropic diffusion. 1 Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Our stabilization model Suppose u k ( x , y ) = u ( x , y , k ) , u 0 be the original video sequence. Then our model is u n + 1 � � − u n �∇ u k � 2 △ ( Id − λ △ ) − 1 u n k k = k � 2 − α k �∇ u 0 dt � � u n k + 1 + u n k − 1 − 2 u n + 1 + µ , k λ is a parameter in defining the Sobolev gradient. α > 1 controls spatial sharpness. µ balances between the spatial sharpening and the temporal smoothing. 17/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Analysis We prove the local and global existence and uniqueness of the solution. The frequency approach yields an efficient algorithm. This is a weakly conditionally stable method. The stability condition only depends on dt, not on spatial grid resolution. 18/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Figure: Raw data. 19/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Figure: SOB. 20/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Figure: SOB+LAP . 21/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Figure: SOB+LAP . 22/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence Turbulence motion analysis (a) (b) Figure: The positioning of the key points along a line. The key points are displayed as the blue dots on (a). (b) shows how these points are oscillating as time t changes. The wave movement of the turbulence happens in groups. 23/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Outline ABCs of image processing 1 Image and PDEs 2 Forward/backward diffusion Imaging through turbulence Image and optimization 3 TV regularization Bregman iterations Optimization and dynamic differential system 4 Inverse scale space flow Forward-backward splitting 24/39