Optical flow Subhransu Maji CMPSCI 670: Computer Vision October - PowerPoint PPT Presentation

Visual motion Optical flow Subhransu Maji CMPSCI 670: Computer Vision October 20, 2016 Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys 2 CMPSCI 670 Subhransu Maji (UMass, Fall 16) Motion and perceptual organization Motion and perceptual organization Sometimes, motion is the only cue Sometimes, motion is the only cue CMPSCI 670 Subhransu Maji (UMass, Fall 16) 3 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 4

Motion and perceptual organization Uses of motion Even “impoverished” motion data can evoke a strong percept Segmenting objects based on motion cues Estimating the 3D structure Learning and tracking dynamical models Recognizing events and activities G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. 5 6 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Motion field Optical flow The motion field is the projection of the 3D scene motion into the Definition : optical flow is the apparent motion of brightness patterns image in the image Ideally, optical flow would be the same as the motion field Have to be careful: apparent motion can be caused by lighting changes without any actual motion ‣ Think of a uniform rotating sphere under fixed lighting 
 vs. a stationary sphere under moving illumination CMPSCI 670 Subhransu Maji (UMass, Fall 16) 7 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 8

Estimating optical flow The brightness constancy constraint Given two subsequent frames, estimate the apparent motion field u ( x , y ) and v ( x , y ) between them I ( x , y , t ) I ( x , y , t –1) Brightness Constancy Equation: I ( x , y , t 1 ) I ( x u , y v ) − = + ( x , y ) + ( x , y ), t I ( x , y , t –1) I ( x , y , t ) Linearizing the right side using Taylor expansion: • Key assumptions • Brightness constancy: projection of the same point looks the I ( x , y , t 1 ) I ( x , y , t ) I u ( x , y ) I v ( x , y ) − ≈ + + x y same in every frame • Small motion: points do not move very far I u I v I 0 • Spatial coherence: points move like their neighbors + + ≈ Hence, x y t 9 10 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) The brightness constancy constraint The brightness constancy constraint - How many equations and unknowns per pixel? - How many equations and unknowns per pixel? ‣ One equation, two unknowns ‣ One equation, two unknowns I u I v I 0 I u I v I 0 + + = + + = x y t x y t What does this constraint mean? What does this constraint mean? • • I ( u , v ) I 0 I ( u , v ) I 0 ∇ ⋅ + = ∇ ⋅ + = t t • The component of the flow perpendicular to the • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown gradient (i.e., parallel to the edge) is unknown gradient ( u , v ) If ( u , v ) satisfies the equation, 
 so does ( u+u’ , v+v’ ) if I ( u ' , v ' ) 0 ( u ’, v ’) ∇ ⋅ = ( u + u ’, v + v ’) edge CMPSCI 670 Subhransu Maji (UMass, Fall 16) 11 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 12

The aperture problem The aperture problem Actual motion Perceived motion The aperture problem The barber pole illusion What direction is the motion? http://en.wikipedia.org/wiki/Barberpole_illusion

Solving the aperture problem Solving the aperture problem How to get more equations for a pixel? Least squares problem: Spatial coherence constraint: pretend the pixel’s I ( x ) I ( x ) I ( x ) & # & # neighbors have the same (u,v) x 1 y 1 t 1 $ ! $ ! ‣ I ( x ) I ( x ) E.g., if we use a 5x5 window, that gives us 25 equations per pixel u I ( x ) & # $ x 2 y 2 ! t 2 $ ! I ( x ) [ u , v ] I ( x ) 0 = − ∇ ⋅ + = $ ! ! ! v ! i t i $ ! $ ! % " $ ! $ ! I ( x ) I ( x ) I ( x ) I ( x ) I ( x ) I ( x ) & # & # $ ! % " x 1 y 1 % x n y n " t 1 t n $ ! $ ! I ( x ) I ( x ) u I ( x ) & # $ x 2 y 2 ! t 2 $ ! • When is this system solvable? = − $ ! ! ! v ! $ ! $ ! • What if the window contains just a single straight edge? % " $ ! $ ! I ( x ) I ( x ) I ( x ) $ ! % " % x n y n " t n B. Lucas and T. Kanade. An iterative image registration technique with an application to B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence , pp. stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence , pp. 674–679, 1981. 674–679, 1981. 17 18 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Conditions for solvability Conditions for solvability “Bad” case: single straight edge “Good” case: corner CMPSCI 670 Subhransu Maji (UMass, Fall 16) 19 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 20

Lucas-Kanade flow Lucas-Kanade flow Linear least squares problem I I I I u I I ∑ ∑ ∑ & # & # & # x x x y x t I ( x ) I ( x ) I ( x ) & # = − & # $ ! $ ! x 1 y 1 $ ! t 1 A d b I I I I v I I ∑ ∑ ∑ $ ! $ ! = I ( x ) I ( x ) u I ( x ) $ ! $ ! % " % " % " & # x y y y y t x 2 y 2 $ ! $ t 2 ! = − $ ! n 2 ! ! ! 2 1 n 1 $ ! v × $ ! × × % " • Recall the Harris corner detector: M = A T A is the $ ! $ ! I ( x ) I ( x ) I ( x ) $ ! % " % x n y n " t n second moment matrix T T ( A A)d A b Solution given by • We can figure out whether the system is = solvable by looking at the eigenvalues of the ∑ I I ∑ I I u ∑ I I & # & # & # second moment matrix x x x y x t = − $ ! $ ! $ ! ∑ I I ∑ I I v ∑ I I • The eigenvectors and eigenvalues of M relate to edge direction $ ! $ ! % " % " % " x y y y y t and magnitude • The eigenvector associated with the larger eigenvalue points in The summations are over all pixels in the window the direction of fastest intensity change, and the other eigenvector is orthogonal to it B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence , pp. 674–679, 1981. 21 22 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Visualization of second moment matrices Visualization of second moment matrices CMPSCI 670 Subhransu Maji (UMass, Fall 16) 23 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 24

Interpreting the eigenvalues Example Classification of image points using eigenvalues of the second moment matrix: λ 2 “Edge” 
 λ 2 >> λ 1 “Corner” 
 λ 1 and λ 2 are large, 
 λ 1 ~ λ 2 λ 1 and λ 2 are small “Edge” 
 “Flat” λ 1 >> λ 2 region λ 1 25 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 26 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Uniform region Edge – gradients have small magnitude – gradients have one dominant direction – small λ 1 , small λ 2 – large λ 1 , small λ 2 – system is ill-conditioned – system is ill-conditioned CMPSCI 670 Subhransu Maji (UMass, Fall 16) 27 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 28

High-texture or corner region Optical Flow Results – gradients have different directions, large magnitudes – large λ 1 , large λ 2 – system is well-conditioned 29 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 30 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Errors in Lucas-Kanade Multi-resolution registration The motion is large (larger than a pixel) ‣ Iterative refinement ‣ Coarse-to-fine estimation ‣ Exhaustive neighborhood search (feature matching) A point does not move like its neighbors ‣ Motion segmentation Brightness constancy does not hold ‣ Exhaustive neighborhood search with normalized correlation * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 31 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 32

Optical flow results Optical flow results * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 33 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 34 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) State-of-the-art optical flow State-of-the-art optical flow Start with something similar to Lucas-Kanade Epic Flow: Feature matching + edge preserving flow interpolation + gradient constancy + energy minimization with smoothing term + region matching + keypoint matching (long-range) +Pixel-based +Keypoint-based Region-based EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow, Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui and Cordelia Schmid, CVPR 2015. Large displacement optical flow, Brox et al., CVPR 2009 Source: J. Hays CMPSCI 670 Subhransu Maji (UMass, Fall 16) 35 CMPSCI 670 Subhransu Maji (UMass, Fall 16) 36