Optical Flow: Constant Flow Computer Vision 16-385 Carnegie Mellon University (Kris Kitani)
Optical Flow (a.k.a., Video Stabilization, Tracking, Stereo Matching, Registration) Given a pair of images { I t , I t +1 } Estimate the optical flow field { v ( p i ) , u ( p i ) }
I x u + I y v + I t = 0 I y = ∂ I I x = ∂ I I t = ∂ I u = dx v = dy ∂ y ∂ x ∂ t dt dt spatial derivative optical flow temporal derivative How can we use the brightness constancy equation to estimate the optical flow?
unknown I x u + I y v + I t = 0 known We need at least ____ equations to solve for 2 unknowns.
unknown I x u + I y v + I t = 0 known Where do we get more equations (constraints)?
Where do we get more equations (constraints)? I x u + I y v + I t = 0 Assume that the surrounding patch (say 5x5) has ‘ constant flow’
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations I x ( p 1 ) u + I y ( p 1 ) v = − I t ( p 1 ) I x ( p 2 ) u + I y ( p 2 ) v = − I t ( p 2 ) . . . I x ( p 25 ) u + I y ( p 25 ) v = − I t ( p 25 )
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations 2 3 2 3 I x ( p 1 ) I y ( p 1 ) I t ( p 1 ) u I x ( p 2 ) I y ( p 2 ) I t ( p 2 ) � 6 7 6 7 = − . . . 6 7 6 7 . . . v 6 7 6 7 . . . 4 5 4 5 I x ( p 25 ) I y ( p 25 ) I t ( p 25 ) Matrix form
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations 2 3 2 3 I x ( p 1 ) I y ( p 1 ) I t ( p 1 ) u I x ( p 2 ) I y ( p 2 ) I t ( p 2 ) � 6 7 6 7 = − . . . 6 7 6 7 . . . v 6 7 6 7 . . . 4 5 4 5 I x ( p 25 ) I y ( p 25 ) I t ( p 25 ) A b x 25 × 2 2 × 1 25 × 1 How many equations? How many unknowns? How do we solve this?
Least squares approximation || Ax − b || 2 A > A ˆ x = A > b x = arg min ˆ is equivalent to solving x
Least squares approximation || Ax − b || 2 A > A ˆ x = A > b x = arg min ˆ is equivalent to solving x To obtain the least squares solution solve: A > A A > b ˆ x P P P u 2 I x I x I x I y 3 2 I x I t 3 � p ∈ P p ∈ P p ∈ P = − 4 5 4 5 P P P I y I x I y I y v I y I t p ∈ P p ∈ P p ∈ P where the summation is over each pixel p in patch P x = ( A > A ) � 1 A > b
Least squares approximation || Ax − b || 2 A > A ˆ x = A > b x = arg min ˆ is equivalent to solving x To obtain the least squares solution solve: A > A A > b ˆ x P P P u 2 I x I x I x I y 3 2 I x I t 3 � p ∈ P p ∈ P p ∈ P = − 4 5 4 5 P P P I y I x I y I y v I y I t p ∈ P p ∈ P p ∈ P where the summation is over each pixel p in patch P Sometimes called ‘Lucas-Kanade Optical Flow’ (can be interpreted to be a special case of the LK method with a translational warp model)
When is this solvable? A > A ˆ x = A > b A T A should be invertible A T A should not be too small λ 1 and λ 2 should not be too small A T A should be well conditioned λ 1 / λ 2 should not be too large ( λ 1 =larger eigenvalue)
Where have you seen this before? P P 2 I x I x I x I y 3 p ∈ P p ∈ P A > A = 4 5 P P I y I x I y I y p ∈ P p ∈ P
Where have you seen this before? P P 2 I x I x I x I y 3 p ∈ P p ∈ P 4 5 P P I y I x I y I y p ∈ P p ∈ P Harris Corner Detector!
Implications • Corners are when λ 1, λ 2 are big; this is also when Lucas-Kanade optical flow works best • Corners are regions with two different directions of gradient (at least) • Corners are good places to compute flow! What happens when you have no ‘corners’?
You want to compute optical flow. What happens if the image patch contains only a line?
Aperture�Problem small visible image patch In which direction is the line moving?
Aperture�Problem small visible image patch In which direction is the line moving?
Aperture�Problem
Aperture�Problem
Aperture�Problem
Aperture�Problem
Want patches with different gradients to the avoid aperture problem
Want patches with different gradients to the avoid aperture problem
H(x,y) = y I(x,y) x x 1 1 1 1 - - - - optical flow: (1,1) 2 2 2 2 - 1 1 1 3 3 3 3 - 2 2 2 4 4 4 4 - 3 3 3 5 5 5 5 - 4 4 4 y y I x u + I y v + I t = 0 Compute gradients Solution: I x (3 , 3) = 0 v = 1 I y (3 , 3) = 1 I t (3 , 3) = I (3 , 3) − H (3 , 3) = − 1 We recover the v of the optical flow but not the u. This is the aperture problem.
Recommend
More recommend
