Improving the Robustness of Variational Optical Flow through - PowerPoint PPT Presentation

Improving the Robustness of Variational Optical Flow through Tensor Voting by: Hatem A. Rashwan, Domenec Puig, Miguel Angel Garcia presented by: Merlin Lang langmerlin@stud.uni-sb.de Milestones And Advatages in Image Analysis M athematical I mage A nalysis Group Saarland University

Contents 1. Introduction 2. Complementary Optic Flow Model 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary

Contents 1. Introduction Motivation 2. Complementary Optic Flow Model 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary 3 / 26

Introduction Motivation • Variational methods outperform other methods • State of the art method: complementary optic flow • Improvement with tensor voting 4 / 26

Contents 1. Introduction 2. Complementary Optic Flow Model Data Term Smoothness Term Constraint Adaptive Regularizer (CAR) 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary 5 / 26

Complementary Optic Flow Model • Given and image sequence f ( x ) with x := ( x, y, t ) and displacement w = ( u, v, 1) • Energy functional formulation: � E ( w ) = ( M ( w , f ) + α V ( ∇ 2 u, ∇ 2 v, f ) dxdy � �� � � �� � Ω data term smoothness term • Minimization with Euler-Lagrange-Equations: 0 = ∂ u M − α ( ∂ x ( ∂ u x V ) + ∂ y ( ∂ u y V )) 0 = ∂ v M − α ( ∂ x ( ∂ v x V ) + ∂ y ( ∂ v y V )) 6 / 26

Complementary Optic Flow Model Data Term • Given grey value constancy f ( x + w ) = f ( x ) • can be linearized as f x u + f y v + f t = w T ∇ 3 f = 0 • Rewriting to a least squares data term M = ( w T ∇ 3 f ) 2 = w T ∇ 3 f ( ∇ 3 f ) T w = w T J 0 w • Where J 0 is called the motion tensor 7 / 26

Complementary Optic Flow Model • J 0 unsufficient since aperture problem present • Remedy: Gradient constancy ∇ 3 f ( x + w ) = ∇ 3 f ( x ) • One can use the final Motion Tensor J = ∇ 3 f ( ∇ 3 f ) T + γ ( ∇ 3 f x ( ∇ 3 f x ) T + ∇ 3 f y ( ∇ 3 f y ) T ) • With postponing the linearisation: M ( u, v ) =Ψ M (( f ( x + w ) − f ( x )) 2 ) + γ Ψ M (( ∇ 2 f ( x + w ) − ∇ 2 f ( x )) 2 ) • Using the robust penalizer � s 2 + ξ 2 Ψ M ( s 2 ) = 8 / 26

Complementary Optic Flow Model Smoothness Term • Classical homogenious regularisation V ( ∇ 2 u, ∇ 2 v ) = |∇ 2 u | 2 + |∇ 2 v 2 | = ( u 2 x + u 2 y ) + ( v 2 x + v 2 y ) • Compute eigenvectors of structure tensor S ρ = K ρ ∗ ( ∇ 2 f ∇ 2 f T ) • Results in joint image- and flow-driven regularisation V ( ∇ 2 u, ∇ 2 v ) = ( e 1 ∇ 2 u ) 2 + ( e 2 ∇ 2 u ) 2 + ( e 1 ∇ 2 v ) 2 + ( e 1 ∇ 2 v ) 2 • Yields the rubustified smoothness term V ( ∇ 2 u, ∇ 2 v ) = Ψ V (( s T 1 ∇ 2 u ) 2 ) + ( s T 1 ∇ 2 v ) 2 ) +Ψ V (( s T 2 ∇ 2 u ) 2 ) + ( s T 2 ∇ 2 v ) 2 ) 9 / 26

Complementary Optic Flow Model • Results in new Euler Lagrange Equations: 0 = ∂ u M − α ( divD u ( s 1 , s 2 , ∇ 2 u ) ∇ 2 u ) 0 = ∂ v M − α ( divD v ( s 1 , s 2 , ∇ 2 v ) ∇ 2 v ) • with � � � � V (( s T 1 ∇ 2 p ) 2 ) Ψ ′ 0 s 1 D p ( s 1 , s 2 , ∇ 2 p ) = ( s 1 , s 2 ) V (( s T 2 ∇ 2 p ) 2 ) 0 Ψ ′ s 2 • called the “diffusion tensor” 10 / 26

Complementary Optic Flow Model Constraint Adaptive Regularizer (CAR) • Regularisation Tensor. ∇ 2 f ( ∇ 2 f ) T + γ ∇ 2 f x ( ∇ 2 f x ) T + ∇ 2 f y ( ∇ 2 f y ) T �� � � R p = K ρ ∗ • Single Robust Penalisation. 1 ∇ 2 u ) 2 ) + ( r T 1 ∇ 2 v ) 2 ) V ( ∇ 2 u, ∇ 2 v ) = Ψ V (( r T 2 ∇ 2 u ) 2 ) + ( r T 2 ∇ 2 v ) 2 ) +( r T • gives final diffusion tensor � � � � 1 ∇ 2 u ) 2 ) + r T 1 ∇ 2 v ) 2 ) V (( r T Ψ ′ 0 r 1 D p ( s 1 , s 2 , ∇ 2 p ) = ( r 1 , r 2 ) 0 1 r 2 11 / 26

Contents 1. Introduction 2. Complementary Optic Flow Model 3. Proposed Model Pre-segmentation of image pixels Approach overview Tensor Voting Smoothing image gradients 4. Adapted optical flow model 5. Experiment and Results 6. Summary 12 / 26

Proposed Model Pre-segmentation of image pixels Homogeneous and textured regions • Compute signal to noise ratio SNR = 20 log 10 ( µ/σ ) • Classify as homogenious if SNR > τ and 1 cos ( β ) = ≈ 0 � 1 + ||∇ 3 f || • else classify as texture moving if SNR ≤ τ , above holds and f t cos ( δ ) = ||∇ 3 f || + ǫ ≈ 1 • else as non moving 13 / 26

Proposed Model Approach overview Overview of the model using tensor voting 14 / 26

Proposed Model Tensor Voting • Tensor Voting for pixel p : � TV ( p ) = SV ( v, S q ) + PV ( v, P q ) + BV ( v, B q ) q ∈ Θ( p ) • Where SV stick, PV plate and BV ball tensor votes • Stick voting by rotation oround surface normal and applying � � � l (Θ)+ bk (Θ) if Θ ≤ π exp σ 4 f (Θ) = 0 else • BV and PV obtained by integration 15 / 26

Proposed Model Smoothing image gradients • Apply Tensor Voting after segmentation to TM and HM pixels • Only applied to the same class of pixels • No voting for pixels with huge gradient difference 16 / 26

Contents 1. Introduction 2. Complementary Optic Flow Model 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary 17 / 26

Adapted optical flow model • Replace Gaussion Convolution with TV T = TV ( ∇ 3 f ) + γ ( TV ( ∇ 3 f x ) + TV ( ∇ 3 f y )) • Change CAR to: R = TV ( ∇ 2 f ) + γ ( TV ( ∇ 2 f x ) + TV ( ∇ 2 f y )) • With additional regularisation M ( w , f ) = w T T w V ( ∇ 2 u, ∇ 2 v ) = Ψ V ( R ) + R 18 / 26

Contents 1. Introduction 2. Complementary Optic Flow Model 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary 19 / 26

Experiment and Results (a)Frame at time t in sequence OPEN-HOTEL.(b) Frame at time t + dt. (c) Classified pixels: red pixels are textured-moving regions, green pixels are homogeneous- moving regions and blue pixels are stationary (not moving) regions. (d) Frame at time t in sequence STREET-CROSS. (e) Frame at time t + dt. (f) Classified pixels: red pixels are textured-moving regions, green pixels are homogeneous-moving regions and blue pixels are stationary (not moving) regions. 20 / 26

Experiment and Results Results for some Middlebury sequences with corresponding ground-truth. (1st column and 2nd column) Frames 10 and 11. (3rd column) Ground-truths (black points correspond to pixels without available ground-truth). (4th column) Optical flow fields obtained with the proposed approach. 21 / 26

Experiment and Results Results for some Middlebury and MIT sequences with associated ground-truths. (1st column and 2nd column) Two consecutive frames. (3rd column) Ground-truths. (4th Column) Optical flow fields obtained with the proposed approach. 22 / 26

Contents 1. Introduction 2. Complementary Optic Flow Model 3. Proposed Model 4. Adapted optical flow model 5. Experiment and Results 6. Summary 23 / 26

Summary • Proposed method enhances Complementary model with Tensor voting • Separately applied to homogeneous-moving and textured-moving regions • Proposed model yields flow fields with lower quantitative errors • Drawback: Computational Complexity 24 / 26

References • R ASHWAN Hatem A., P UIG Domenec , G ARCIA Miguel A.: Improving the Robustness of Variational Optical Flow Through Tensor Voting. In: Computer Vision and Image Understanding (2012) • Z IMMER Henning, B RUHN Andr´ es, W EICKERT Joachim, V ALGAERTS Levi, S ALGADO Agust’ın, R OSENHAHN Bodo , S EIDEL Hans P .: Complementary Optic Flow. In: Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition . Berlin, Heidelberg : Springer-Verlag, 2009 (EMMCVPR ’09), p. 207–220 25 / 26

Thank you for your attention! 26 / 26