Dense optical flow estimation in image sequences and disparity map - PowerPoint PPT Presentation

Dense optical flow estimation in image sequences and disparity map computation for stereo pairs applied to 3D reconstruction Javier Sánchez Pérez Universidad de Las Palmas de Gran Canaria Departamento de Informática y Sistemas Grupo de Análisis Matemático de Imágenes

Contents • Optical flow methods – Standard approach – Considering colour images – Making the method symmetric • Stereoscopic methods – Epipolar geometry – Standard approach – Colour method – Symmetrical method • Main contributions and conclusions

Optical flow Standard approach • Variational approach • Energy minimization • Large displacements r • Numerical scheme h ( x , y ) • Experimental results

Variational approach • Energy r r r ∫ ∫ = ⋅ + α Φ ⋅ E ( h ) L ( h ) dw ( h ) dw Ω Ω r ( ) t = h u ( x , y ), v ( x , y ) r ( ) 2 = − + + L ( h ) I ( x , y ) I ( x u ( x , y ), y v ( x , y ) ) 1 2 ( ) r r r Φ = ∇ ⋅ ∇ ⋅ ∇ t h h D ( I ) h 1

Regularizing term • Nagel-Enkelmann operator ⎧ ⎫ ⎛ ⎞ − 2 I I I ⎪ ⎪ 1 ( ) ⎜ ⎟ y x y ∇ = + γ 2 D I Id ⎨ ⎬ ⎜ ⎟ 2 − 2 I I I ∇ + γ 2 ⎪ ⎪ I 2 ⎝ ⎠ ⎩ ⎭ x y x

Energy minimization • Associated Euler-Lagrange equations ) ( ) r r ∂ ⎛ ⎞ I r r ( ( ) = ∇ ∇ + − + + ⎜ 2 ⎟ 0 C div D I u I I ( x h ) ( x h ) 1 1 2 ∂ x ⎝ ⎠ ) ( ) r r ⎛ ⎞ ∂ I r r ( ( ) ⎜ ⎟ = ∇ ∇ + − + + 0 C div D I v I I ( x h ) 2 ( x h ) ⎜ ⎟ 1 1 2 ∂ y ⎝ ⎠

Energy minimization • Gradient descent ( ) r r ∂ ∂ ⎛ ⎞ u I r r ( ( ) ) = ∇ ∇ + − + + ⎜ ⎟ C div D I u I I ( x h ) 2 ( x h ) 1 1 2 ∂ ∂ t ⎝ x ⎠ ( ) r ⎛ ⎞ r ∂ ∂ v I r r ( ( ) ) = ∇ ∇ + − + ⎜ ⎟ + C div D I v I I ( x h ) 2 ( x h ) ⎜ ⎟ 1 1 2 ∂ ∂ t y ⎝ ⎠ • Anisotropic diffusion – diffusion tensor ( ) ⎧ ∇ ↑ ∇ I div u ∂ ⎪ u ⊥ ∇ = ∇ ∇ ⇒ I div ( D ( I ) u ) ⎨ 1 ∇ ↓ ∇ ∂ I div ( u ) t ⎪ ⎩ 2

Large displacements • Pyramidal approach ) ( ) r ⎛ ⎞ r ∂ ∂ ( ( ) z u I r r ⎜ ⎟ = ∇ ∇ + − + + z z z z C div D I u I I ( x h ) 2 ( x h ) ⎜ ⎟ 1 z 1 2 z z ∂ ∂ t x ⎝ ⎠ ) ( ) r ⎛ ⎞ r ∂ ( ( ) ∂ z v I r r ⎜ ⎟ = ∇ ∇ + − + + z z z z C div D I v I I ( x h ) 2 ( x h ) ⎜ ⎟ 1 z 1 2 z z ∂ ∂ t y ⎝ ⎠ ( ) ( ) = z z z I x , y I 2 x , 2 y > > > → z z z 0 K 0 1 n ( ) ( ) ( ) ( ) → → → → u , v u , v u , v u , v K z z z z z z 0 0 1 1 n n

Large displacements

Large displacements • Scale-Space approach ) ( ) r ⎛ ⎞ r ∂ ∂ σ ( ( ) u I r r ⎜ ⎟ = ∇ σ ∇ + σ − σ + + σ C div D I u I I ( x h ) 2 ( x h ) ⎜ ⎟ σ σ σ 1 1 2 ∂ ∂ t x ⎝ ⎠ ) ( ) r ⎛ ⎞ r ∂ ∂ σ ( ( ) v I r r ⎜ ⎟ = ∇ σ ∇ + σ − σ + + σ C div D I v I I ( x h ) 2 ( x h ) ⎜ ⎟ σ σ σ 1 1 2 ∂ ∂ t y ⎝ ⎠ σ = I G * I σ σ > σ > > σ → .... 0 0 1 n → → → ( u , v ) ( u , v ) ..... ( u n v , ) σ σ σ σ σ σ 0 0 1 1 n

Large displacements

Invariance • Invariance under gray level shifts α = C 2 ∇ max I ( x , y ) ( x , y ) ν ∫ = s H ( z ) dz ∇ I 0

Numerical scheme ⎛ ⎞ + + + + + − + − k 1 k 1 k 1 k 1 d d u u d d u u ⎜ ⎟ + + − − i 1 , j i , j i 1 , j i , j + i 1 , j i , j i 1 , j i , j + ⎜ 2 2 ⎟ 2 h 2 h 1 1 ⎜ ⎟ + + + + + k 1 − k 1 + k 1 − k 1 f f u u f f u u ⎜ ⎟ + + − − i , j 1 i , j i , j 1 i , j + i , j 1 i , j i , j 1 i , j + ⎜ ⎟ + − k 1 k 2 2 u u 2 h 2 h ⎜ ⎟ i , j i , j 1 1 = + C ∆ t ⎜ + + − + + + − + ⎟ k 1 k 1 k 1 k 1 e e u u e e u u + + + + − − − − + i 1 , j 1 i , j i 1 , j 1 i , j + i 1 , j 1 i , j i 1 , j 1 i , j − ⎜ ⎟ 2 2 2 2 h 2 2 h ⎜ ⎟ 1 1 ⎜ ⎟ + + − + + + − + k 1 k 1 k 1 k 1 e e u u e e u u ⎜ ⎟ + − + − − + − + i 1 , j 1 i , j i 1 , j 1 i , j i 1 , j 1 i , j i 1 , j 1 i , j − − ⎜ ⎟ 2 2 2 2 h 2 2 h ⎠ ⎝ 1 1 ∂ ( ( ) ( ) ) I ( ) σ − + + 2 , + + k k k k I x , y I x u , y v x u , y v σ σ 1 , i , j i , j 2 , i , j i , j i , j i , j i , j i , j i , j i , j ∂ x

Experimental results • Squares sequence

Experimental results ( ) ( ) ⎛ ⎞ ~ ~ 2 2 σ σ = − + − Error Average u u v v ⎜ ⎟ i , j i , j i , j i , j ⎝ ⎠ ( ) ( ) ⎛ ⎞ ⎛ ⎞ ~ ~ σ σ t u , v , 1 u , v , 1 ⎜ ⎟ ⎜ ⎟ = i , j i , j i , j i , j Error Average arccos ( ) ( ) ( ) ( ) ⎜ ⎟ ⎜ ⎟ ~ ~ ⎜ ⎟ σ σ 2 + 2 + 2 + 2 + u v 1 u v 1 ⎝ ⎠ ⎝ ⎠ i , j i , j i , j i , j

Experimental results • Square sequence

Experimental results Comparison with other methods in Barron et al. 100% dense methods Method Ang. Error Stand. Dev. Horn y Schunk 32,81º 13,67º Nagel 34,57º 14,38º Anandan 31,46º 18,31º Singh 46,12º 18,64º Ours 10,97º 9,60º

Experimental results • Taxi sequence

Experimental results • Yosemite sequence

Experimental results Comparison with other methods in Barron et al. 100% dense methods Method Ang. Error Stand. Dev. Horn y Schunk 9,78º 16,19º Nagel 10,22º 16,51º Anandan 13,36º 15,64º Singh 10,03º 13,13º Ours 5,53º 7,40º

Optical flow for color images • Energy ( ) r r r r 3 ( ) r r ∑∫ ∫ = − + + α ∇ ∇ ∇ 2 t E ( h ) I ( x ) I ( x h ) h D I h 1 , i 2 , i 1 , max = i 1 { } ∇ = ∇ ∇ ≥ ∇ ∀ = I ( x , y ) I I I i 1 , 2 , 3 max i i i 0 0

Results • Cube sequence

Symmetrical optic flow r h 1 r h 2 • Composed energy r r r r r r = + E ( h , h ) E ( h , h ) E ( h , h ) 1 2 1 1 2 2 1 2

Variational approach • Energies ( ) r r r r r r r ⎛ ⎞ ∫ ∫ ∫ h = + α Φ + β ψ + 2 ⎜ ⎟ E ( h , h ) L ( h ) ( h ) h h 1 ⎝ ⎠ 1 1 2 1 1 1 2 Ω Ω Ω ( ) r r r r r r r ⎛ ⎞ ∫ ∫ ∫ h = + α Φ + β ψ + 2 ⎜ ⎟ E ( h , h ) L ( h ) ( h ) h h 2 ⎝ ⎠ 2 1 2 2 2 2 1 Ω Ω Ω

Energy minimization • Choosing the ψ function ⎧ s ⎪ s ψ = ( s ) ⎨ s − γ 1 e ⎪ γ ⎩ • Iterative scheme r r r r r r + + + + = + n 1 n 1 n 1 n n n 1 E ( h , h ) E ( h , h ) E ( h , h ) 1 2 1 1 2 2 1 2

Experimental results • Squares sequence r h Occlusions 1 Euclid. error Ang. error Non- 0.16 0.50º symmetric Symmetric 0.081 0.18º

Experimental results • Marble blocks (H. H. Nagel) r h 1 Euclid. error Ang. error Non- 0.37 9.4º symmetric Symmetric 0.17 5.2º

Stereoscopic vision • Epipolar geometry • Variational approach • Energy minimization • Experimental results

Epipolar geometry Fundamental matrix m t 2 F m 1 =0 Epipolar lines I 2 I 1 m 1 m 2 C 2 C 1 M

Variational approach • Embedding the epipolar geometry I 2 I 1 m 1 r m 1 h m 2 λ ⎛ ⎞ a ⎜ ⎟ λ r ⎛ ⎞ u ( ( x , y )) = ⋅ ⎜ ⎟ ⎜ b ⎟ F m = h ⎜ ⎟ 1 λ ⎜ ⎟ v ( ( x , y )) ⎝ ⎠ c ⎝ ⎠

Variational approach • Energy ( ) r r r ( ) ∫ ∫ λ = − + λ + α ∇ λ ⋅ ∇ ⋅ ∇ λ 2 E ( ) I ( x ) I ( x h ( )) D I 1 2 1 λ λ ⎛ ⎞ ∂ ∂ ⎛ ⎞ I I r r ⎜ ⎟ − a 2 ( x ) b ⎜ 2 ⎟ ( x ) ⎜ ⎟ ( ) ∂ ∂ ∂ λ y ⎝ x ⎠ r r ⎝ ⎠ ( ( ) ) ( ) λ = ∇ ∇ λ + − C div D I I ( x ) I ( x ) 2 1 2 ∂ t + 2 2 a b

Experimental results • Hervé’s face

Experimental results • Inria Library