Aggregation Methods for Optical Flow Computation Denis Fortun, Patrick Bouthemy, Charles Kervrann SERPICO Team Inria Rennes - Bretagne Atlantique Campus universitaire de Beaulieu 35 042 Rennes Cedex, France SIAM Conference on Imaging Science, Philadelphia, 2012
Framework : patch-wise approach and aggregation method for large and small displacement optical flow Aggregation Methods for Optical Flow Computation 2
Outline 1. Local and global methods for computing optical flow ⊲ data conservation ⊲ local smoothing ⊲ global approach and spatial regularization 2. Semi-local estimation and global aggregation ⊲ semi-local estimation of flow fields ⊲ global aggregation 3. Experimental results and application ⊲ comparisons of aggregation procedures ⊲ comparison of parametric and variational methods ⊲ algorithm parameters Discussion and conclusion Aggregation Methods for Optical Flow Computation 3
1. Local and global methods for computing optical flow 1. Local and global methods for computing optical flow Aggregation Methods for Optical Flow Computation 4
1. Local and global methods for computing optical flow ⊲ data conservation Data conservation constraint Fundamental assumption to find correspondences : conservation of brightness or image gradient over time I 2 ( x + w ( x )) = I 1 ( x ) , x ∈ Ω Insufficient constraint : ◮ ”Aperture problem” : 1 equation for 2 unknowns w ( x ) = ( u ( x ) , v ( x )) T ◮ Uniform regions : no image gradient ◮ Assumption violations : occlusion, brightness changes, . . . → Adding a spatial constraint to make the problem well-posed ! Aggregation Methods for Optical Flow Computation 5
1. Local and global methods for computing optical flow ⊲ local smoothing Local smoothing and spatial neighborhoods Assume a coherent motion in a neighborhood V ( x 0 ) ⊂ Ω Coherent motion : parametric motion model w θ ( x ) ◮ translation : w θ ( x ) = ( θ 1 , θ 4 ) T � θ 1 � � 1 � θ 2 θ 3 ◮ Affine motion : w θ ( x ) = x T θ 4 θ 5 θ 6 Local motion estimation of θ over V ( x 0 ) (Lucas & Kanade, 1981) : � E LK ( w θ ( x 0 )) = ρ data ( x , w θ ) dx v ( x 0 ) where ρ data ( x , w θ ) = ψ ( I 2 ( x + w θ ( x )) − I 1 ( x )) denotes the data potential (penalizes deviations from the data constraint) Aggregation Methods for Optical Flow Computation 6
1. Local and global methods for computing optical flow ⊲ local smoothing Limitations of the local smoothing Problem : choice of the region V V must fulfill two criteria : ◮ Delineating a coherently moving region to ensure the validity of the parametric assumption ◮ Collecting enough trustful gradient information for reliable optical flow computation Aggregation Methods for Optical Flow Computation 7
1. Local and global methods for computing optical flow ⊲ local smoothing Limitations of the local smoothing square fixed-size windows regions with variable shapes ◮ joint motion estimation and ◮ parametric assumption violated image segmentation problem over motion boundaries → minimization of non-convex ◮ ”aperture problem” for small functionals (e.g. ”motion windows competition” , Cremers & → fast optimization at each pixel Soatto, 2005) Aggregation Methods for Optical Flow Computation 8
1. Local and global methods for computing optical flow ⊲ global approach and spatial regularization Global approach and spatial regularization Spatial regularization : penalization of high gradients |∇ w | Minimization of a global energy (Brox et al., 2004) : � E G ( w ) = ρ data ( x , w ) + λρ reg ( x , w ) dx Ω � φ ( | I 2 ( x + w ( x )) − I 1 ( x ) | 2 )) + φ ( �∇ w ( x ) � 2 ) dx = Ω √ z 2 + ǫ 2 with ǫ = 0 . 001 where φ ( z 2 ) = Optimization method : resolution of Euler-Lagrange equations → state-of-the-art methods Aggregation Methods for Optical Flow Computation 9
1. Local and global methods for computing optical flow ⊲ global approach and spatial regularization Limitation of the global approach Variational optimization : ◮ Difficulty to handle non-convex functionals : ⊲ restriction to convex penalty functions ⊲ over-smoothing of discontinuities ◮ Coarse-to-fine warping to cope with large displacements : ⊲ over-smoothing of fast moving details Discrete optimization : ◮ Discretization of the search space and multi-label assignment ◮ Quantization of the flow field range : ⊲ compromise between computational cost and accuracy Aggregation Methods for Optical Flow Computation 10
2. Semi-local estimation and global aggregation 2. Semi-local estimation and global aggregation Aggregation Methods for Optical Flow Computation 11
2. Semi-local estimation and global aggregation ⊲ semi-local estimation of flow fields Semi-local estimations of flow fields Decomposition of I 1 into overlapping square patches with several sizes : Set of patches containing x : V s 2 ( x ) = { V 1 , . . . , V 4 } � V ( x ) = V s i ( x ) i = 1 ... 4 ◮ Computation of flow fields over overlapping patches ◮ Combining adaptively the patches of V ( x ) for pointwise motion estimation Aggregation Methods for Optical Flow Computation 12
2. Semi-local estimation and global aggregation ⊲ semi-local estimation of flow fields Block matching to cope with large displacements For each patch of I 1 , find the N most similar patches M V = { M 1 , . . . , M N } in I 2 (normalized cross-correlation) : ◮ Crude estimation of translation (i.e. pixel accuracy) ◮ Handle large displacements of small structures (small patch sizes) Aggregation Methods for Optical Flow Computation 13
2. Semi-local estimation and global aggregation ⊲ semi-local estimation of flow fields Hierarchical motion estimation For each pair of registered patches ( V , M i ) , compute optical flow w 0 V , M i with a variational (Brox et al., 2004) or parametric (Odobez & Odobez, 1995) method (sub-pixel accuracy, optional coarse-to-fine warping) x V − x M 1 w V , M 1 V M 1 w 0 V , M 1 At each pixel x , we collect several hundreds of candidates : + w 0 W ( x ) = { w V , M ( x ) = ( x M − x V ) V , M ( x ) : M ∈ M V , V ∈ V ( x ) } � �� � � �� � translation optical flow Aggregation Methods for Optical Flow Computation 14
2. Semi-local estimation and global aggregation ⊲ global aggregation Global aggregation Motivations : ◮ Optimization of a variational energy or parametric estimation for each patch ◮ “Selection of the best” competing estimators at each location Aggregation principle : minimization of a global energy to compute a unique flow field w : Ω → R 2 from the set W = � x ∈ Ω W ( x ) : ◮ Discrete optimization ( W considered as a discrete space) w = arg min w ∈W E DA ( w ) ◮ Continuous optimization ( W used to derive a new data term) w = arg w ∈ R 2 | Ω | E CA ( w , W ) min Aggregation Methods for Optical Flow Computation 15
2. Semi-local estimation and global aggregation ⊲ global aggregation Global aggregation : discrete optimization Finite set of candidates W ( x ) at each pixel : quantization of the space of motion vectors Adaptive selection by global energy minimization � Ψ DA data ( x , w ) + β Ψ DA E DA ( w ) = reg ( x , w ) x ∈ Ω Data term : � ρ data ( x , w ) Ψ DA data ( x , w ) = 1 − NCC σ ( x , w ) (normalized cross-correlation) Regularization term : ( N ( x ) : spatial neighborhood of pixel x ) � Ψ DA φ ( � w ( x ) − w ( y ) � 2 ) reg ( x , w ) = y ∈N ( x ) Aggregation Methods for Optical Flow Computation 16
2. Semi-local estimation and global aggregation ⊲ global aggregation Global aggregation : discrete optimization (contd’) Discrete optimization : no restriction on the form of E DA ( w ) (differentiability, convexity) “Fusion-Move” transforms a multi-label problem into a succession of binary labeled problems (Lempitsky et al., 2008) : application to the fusion of independent pre-computed motion fields (e.g. Horn & Schunk, Lucas & Kanade, ...) Aggregation Methods for Optical Flow Computation 17
2. Semi-local estimation and global aggregation ⊲ global aggregation Global aggregation : continuous optimization Limitations of discrete optimization : minimizer is found in the finite set of candidates Continuous aggregation : aggregated motion field is allowed to deviate from the local candidates to ensure global smoothness � Ψ CA data ( x , W , w ) + β φ ( �∇ w ( x ) � 2 ) dx E CA ( w ) x ∈ Ω A new data term to compare the candidates w 0 ( x ) ∈ W ( x ) : � α ( x , w 0 ) � w ( x ) − w 0 ( x ) � p Ψ CA data ( x , W , w ) = w 0 ( x ) ∈W ( x ) → variational minimization Aggregation Methods for Optical Flow Computation 18
2. Semi-local estimation and global aggregation ⊲ global aggregation Global aggregation : continuous optimization Remark : if β = 0, the aggregated pointwise solution is easily found if the weights α ( x , w 0 ) are uniform : : α ( x , w 0 ) = |W ( x ) | − 1 , p = 2 ◮ Mean ◮ Median : α ( x , w 0 ) = |W ( x ) | − 1 , p = 1 → experimentally insufficient Weighting according to a “confidence measure” : 1 1 α ( x , w 0 ) = ρ data ( x , w 0 ) = φ ( | I 2 ( x + w 0 ( x )) − I 1 ( x ) | 2 ) → α ( x , w 0 ) as a function of Var ( w 0 ) is possible in the parametric case ( Odobez & Bouthemy, 1995) Aggregation Methods for Optical Flow Computation 19
3. Experimental results and application 3. Experimental results and application Aggregation Methods for Optical Flow Computation 20
Recommend
More recommend
