CS 4495 Computer Vision Motion Models Aaron Bobick School of - PowerPoint PPT Presentation

Motion models CS 4495 Computer Vision – A. Bobick CS 4495 Computer Vision Motion Models Aaron Bobick School of Interactive Computing

Motion models CS 4495 Computer Vision – A. Bobick Outline • Last time: dense motion: optic flow • Brightness constancy constraint equation • Lucas-Kanade • 2D Motion models • Bergen, ’92 • Pyramids • Layers • Motion fields from 3D motions • Parametric motion

Motion models CS 4495 Computer Vision – A. Bobick Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys, K. Grauman and others…

Motion models CS 4495 Computer Vision – A. Bobick Motion estimation: Optical flow Will start by estimating motion of each pixel separately Then will consider motion of entire image

Motion models CS 4495 Computer Vision – A. Bobick Problem definition: optical flow I x y t + I x y t ( , , ) ( , , 1) How to estimate pixel motion from image I(x,y,t) to I(x,y,t) ? • Solve pixel correspondence problem – given a pixel in I(x,y,t), look for nearby pixels of the same color in I(x,y,t+1) Key assumptions • color constancy: a point in I(x,y, looks the same in I(x,y,t+1) – For grayscale images, this is brightness constancy • small motion: points do not move very far This is called the optical flow problem

Motion models CS 4495 Computer Vision – A. Bobick Optical flow equation • Combining these two equations = + + + − 0 I x ( u y , v t , 1) I x y t ( , , ) ≈ + + + − I x y t ( , , 1) I u I v I x y t ( , , ) x y ≈ + − + + [ ( , , I x y t 1) I x y ( , )] I u I v x y ≈ + + I I u I v t x y ≈ + ∇ ⋅ < > , I I u v t In the limit as u and v go to zero, this becomes exact = + ∇ ⋅ < > 0 I I u v , t Brightness constancy constraint equation + + = I u I v I 0 x y t

Motion models CS 4495 Computer Vision – A. Bobick Optical flow equation + + = = + ∇ ⋅ < > I u I v I 0 0 I I u v , or x y t t • Q: how many unknowns and equations per pixel? 2 unknowns, one equation Intuitively, what does this constraint mean? • The component of the flow in the gradient direction is determined • The component of the flow parallel to an edge is unknown gradient ( u , v ) ( u + u ’, v + v ’) ( u ’, v ’) edge

Motion models CS 4495 Computer Vision – A. Bobick Aperture problem

Motion models CS 4495 Computer Vision – A. Bobick Solving the aperture problem • How to get more equations for a pixel? • Basic idea: impose additional constraints • most common is to assume that the flow field is smooth locally • one method: pretend the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25 equations per pixel!

Motion models CS 4495 Computer Vision – A. Bobick Lukas-Kanade flow • Prob: we have more equations than unknowns Solution: solve least squares problem • minimum least squares solution given by solution (in d) of: • The summations are over all pixels in the K x K window • This technique was first proposed by Lukas & Kanade (1981)

Motion models CS 4495 Computer Vision – A. Bobick T A Eigenvectors of A • Recall the Harris corner detector: M = A T A is the second moment matrix • The eigenvectors and eigenvalues of M relate to edge direction and magnitude • The eigenvector associated with the larger eigenvalue points in the direction of fastest intensity change • The other eigenvector is orthogonal to it

Motion models CS 4495 Computer Vision – A. Bobick Violating assumptions in Lucas-Kanade • The motion is large (larger than a pixel) • Not-linear: Iterative refinement • Local minima: coarse-to-fine estimation • A point does not move like its neighbors • Motion segmentation • Brightness constancy does not hold • Do exhaustive neighborhood search with normalized correlation - tracking features – maybe SIFT – more later….

Motion models CS 4495 Computer Vision – A. Bobick Violating assumptions in Lucas-Kanade • The motion is large (larger than a pixel) • Not-linear: Iterative refinement • Local minima: coarse-to-fine estimation • A point does not move like its neighbors • Motion segmentation • Brightness constancy does not hold • Do exhaustive neighborhood search with normalized correlation - tracking features – maybe SIFT – more later….

Motion models CS 4495 Computer Vision – A. Bobick Not tangent: Iterative Refinement Iterative Lukas-Kanade Algorithm Estimate velocity at each pixel by solving Lucas-Kanade equations 1. Warp I t towards I t+1 using the estimated flow field 2. - use image warping techniques • Repeat until convergence 3.

Motion models CS 4495 Computer Vision – A. Bobick Optical Flow: Iterative Estimation estimate update Initial guess: Estimate: x x 0 (using d for displacement here instead of u )

Motion models CS 4495 Computer Vision – A. Bobick Optical Flow: Iterative Estimation estimate Initial guess: update Estimate: x x 0

Motion models CS 4495 Computer Vision – A. Bobick Optical Flow: Iterative Estimation estimate Initial guess: Initial guess: update Estimate: Estimate: x x 0

Motion models CS 4495 Computer Vision – A. Bobick Optical Flow: Iterative Estimation x x 0

Motion models CS 4495 Computer Vision – A. Bobick Revisiting the small motion assumption • Is this motion small enough? • Probably not—it’s much larger than one pixel (2 nd order terms dominate) • How might we solve this problem?

Motion models CS 4495 Computer Vision – A. Bobick Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? actual shift estimated shift nearest match is correct nearest match is incorrect (no aliasing) (aliasing) To overcome aliasing: coarse-to-fine estimation.

Motion models CS 4495 Computer Vision – A. Bobick Reduce the resolution!

Motion models CS 4495 Computer Vision – A. Bobick Coarse-to-fine optical flow estimation u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image 1 image 1 image 2 image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2

Motion models CS 4495 Computer Vision – A. Bobick Coarse-to-fine optical flow estimation run iterative L-K Upsample flow warp run iterative L-K . . . image J image 1 image I image 2 warp & upsample Gaussian pyramid of image 1 Gaussian pyramid of image 2

Motion models CS 4495 Computer Vision – A. Bobick Multi-scale

Motion models CS 4495 Computer Vision – A. Bobick Remember: Image sub-sampling 1/8 1/4 Throw away every other row and column to create a 1/2 size image - called image sub-sampling S. Seitz

Motion models CS 4495 Computer Vision – A. Bobick Bad image sub-sampling 1/2 1/4 (2x zoom) 1/8 (4x zoom) Aliasing! What do we do? S. Seitz

Motion models CS 4495 Computer Vision – A. Bobick Gaussian (lowpass) pre-filtering G 1/8 G 1/4 Gaussian 1/2 Solution: filter the image, then subsample S. Seitz

Motion models CS 4495 Computer Vision – A. Bobick Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 S. Seitz

Motion models CS 4495 Computer Vision – A. Bobick Band-pass filtering Gaussian Pyramid (low-pass images) Laplacian Pyramid (subband images) These are “bandpass” images (almost).

Motion models CS 4495 Computer Vision – A. Bobick Laplacian Pyramid Need this! Original image • How can we reconstruct (collapse) this pyramid into the original image?

Motion models CS 4495 Computer Vision – A. Bobick Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] S. Seitz

Motion models CS 4495 Computer Vision – A. Bobick Computing the Laplacian Pyramid Expand Reduce Need G k to reconstruct

Motion models CS 4495 Computer Vision – A. Bobick Reduce and Expand Reduce Apply “5-tap” separable filter to make reduced image.

Motion models CS 4495 Computer Vision – A. Bobick Reduce and Expand Reduce Expand Apply “5-tap” separable Apply different “3-tap” filter to make reduced separable filters for even image. and odd pixels to make expanded image...

Motion models CS 4495 Computer Vision – A. Bobick Just Expand Apply different “3-tap” separable filters for even and odd pixels to make expanded image. Even Odd Coarser Finer

Motion models CS 4495 Computer Vision – A. Bobick What can you do with band limited imaged?

Motion models CS 4495 Computer Vision – A. Bobick Apples and Oranges in bandpass L 0 L 2 L 4 Reconstructed

Motion models CS 4495 Computer Vision – A. Bobick Applying pyramids to LK

Motion models CS 4495 Computer Vision – A. Bobick Coarse-to-fine global motion estimation LK Warp Expand x2 Reduce Reduce LK Warp Expand x2 Reduce Reduce LK Warp Expand x2 Final <u(x,y), v(x,y)>