11/20/2007 • Review Problem set 3 – Dense stereo matching Lecture 19: Motion – Sparse stereo matching – Indexing scenes Indexing scenes Tuesday, Nov 20 Sources of error in Effect of window size correspondences • Low-contrast / textureless image regions • Occlusions • Camera calibration errors Camera calibration errors • Poor image resolution • Violations of brightness constancy W = 3 W = 20 (specular reflections) Want window large enough to have sufficient intensity variation, yet small enough to contain only pixels with • Large motions about the same disparity. Figures from Li Zhang Sparse matching Indexing scenes 1
11/20/2007 So far • Features and filters • Grouping, segmentation, fitting • Multiple views, stereo, matching • Recognition and learning So far: Grouping So far: Features and filters [fig from Shi et al] Clustering, segmentation, Transforming and fitting; what parts describing images; belong together? textures and colors So far: Multiple views So far: Recognition and learning Lowe Shape matching, Hartley and Zisserman recognizing objects Multi-view geometry and and categories, matching, stereo learning techniques Tomasi and Kanade 2
11/20/2007 Motion and tracking Tracking objects, video analysis, low level motion Tomas Izo Outline Uses of motion • Motion field and parallax • Analyzing motion can be useful for – Estimating 3d structure • Optical flow, brightness constancy – Segmentation of moving objects • Aperture problem – Tracking objects, features over time Tracking objects features over time • Constraints on image motion Image sequences Types of motion in video • Considering rigid objects – they can rotate and A digital video is a translate in the scene. sequence of images (frames) • Motion may be due to captured over captured over – Movement in scene time. – Movement of camera (ego motion) Now we consider • Geometrically equivalent, however illumination image as a effects can make one appear different than the function of both other. position and time . Figure by Martial Hebert, CMU 3
11/20/2007 Motion field and apparent motion Motion field equations f P Point in the scene p = (Big V) Z Velocity vector = V [ V , V , V ] x y z T k Take the time derivative th ti d i ti of both sides: − V z P Z V v = p Projection of scene point v v f (Little v) p Apparent 2 Z velocity Goal: estimate apparent motion, the u and v values at each pixel x,y, i.e., u(x,y), v(x,y) Figure by Martial Hebert, CMU Figure by Martial Hebert, CMU = V Motion V V V [ , , ] x y z [ ] Motion field equations = ω ω ω ω , , Translational Angular x y z [ ] motion velocity = P X , Y , Z f P = − − × Velocity of scene V T ω P p = point described as (Big V) Z V = V V V [ , , ] x y z = − − ω ω + + ω ω V V T T Z Z Y Y T k Take the time derivative th ti d i ti x x y z of both sides: = − − ω + ω V T X Z − y y z x V z P Z V = v = − − ω + ω v f (Little v) V T Y X p 2 z z x y Z Using this and the motion field equation, can give expressions for the components of the image velocity v... Figure by Martial Hebert, CMU Motion field equations Motion field equations = − − ω + ω V T Z Y x x y z • Translational part of image motion depends on − V z P = − − ω + ω Z V V T X Z = v (unknown) depth of the point f y y z x 2 Z = − − ω + ω V T Y X • Motion parallax : image motion is a function of z z x y both motion in space and depth of each point. p p p ω ω − ω − ω 2 2 x x T x T f xy T x T f xy = − ω + ω + − = − ω + ω + − y y z x x z x x v f y v f y x y z x y z Z f f Z f f − ω − ω ω ω 2 2 T y T f xy T y T f xy y y = − ω + ω + − = − ω + ω + − z y y z y y x x v f x v f x y x z y x z Z f f Z f f Translational Translational Rotational Rotational components components components components Trucco & Verri Section 8.2.1 Trucco & Verri Section 8.2.1 4
11/20/2007 Translational motion Motion parallax • http://psych.hanover.edu/KRANTZ/Motion Radial motion field if T z Parallax/MotionParallax.html nonzero. Length of flow Length of flow vectors inversely proportional to depth of 3d point points closer to the camera move more quickly across the image plane Figure from Michael Black, Ph.D. Thesis Translational motion Translational motion Radial motion Radial motion field if T z field if T z nonzero. nonzero. Length of flow Length of flow Length of flow Length of flow vectors inversely vectors inversely proportional to proportional to depth of 3d point depth of 3d point Figure from Michael Black, Ph.D. Thesis Figure from Michael Black, Ph.D. Thesis Motion vs. Stereo: Similarities Motion vs. Stereo: Differences • Motion: • Both involve solving – Uses velocity: consecutive frames must be – Correspondence: disparities, motion vectors close to get good approximate time derivative – Reconstruction – 3d movement between camera and scene not necessarily single 3d rigid transformation • Whereas with stereo: – Could have any disparity value – View pair separated by a single 3d transformation 5
11/20/2007 Optical flow problem Optical flow problem How to estimate pixel motion from image H to image I? • Solve pixel correspondence problem – given a pixel in H, look for nearby pixels of the same color in I Goal: estimate apparent motion, the u and v values at each pixel x,y, i.e., u(x,y), v(x,y) Adapted from Steve Seitz, UW Brightness constancy • What might make it difficult to estimate apparent motion? Figure by Michael Black Spatial coherence Temporal smoothness Figure by Michael Black Figure by Michael Black 6
11/20/2007 Brightness constancy equation Motion constraints • To recover optical flow, we need some constraints (assumptions) – Brightness constancy : in spite of motion, image g y p g measurement in small region will remain the same – Spatial coherence : assume nearby points belong to dI the same surface, thus have similar motions, so = Total derivative: x and y are 0 estimated motion should vary smoothly. also functions of time t dt – Temporal smoothness : motion of a surface patch ∂ ∂ ∂ I dx I dy I = + + changes gradually over time. ∂ ∂ ∂ x dt y dt t temporal spatial image derivatives, gradients u and v Brightness constancy equation Brightness constancy equation ∂ ∂ ∂ ∂ ∂ ∂ I dx I dy I I dx I dy I + + = + + = 0 0 ∂ ∂ ∂ ∂ ∂ ∂ x dt y dt t x dt y dt t u v u v Rewritten as: Rewritten as: This is exactly true in the limit as u and v go to 0, Which terms are measurable from images? for infinitesimal motions. How many unknowns in this equation? Aperture problem Aperture problem • Brightness constancy equation: single equation, two unknowns; infinitely many solutions. • Can only compute projection of actual flow vector [ u , v ] in the direction of the image gradient, that is, in the direction normal to the image edge. According to brightness constancy constraint, – Flow component in gradient direction determined motions that satisfy the optical flow equation are – Flow component parallel to edge unknown. only constrained to lie along a line in u,v space. Figure from Michael Black’s Ph.D. Thesis 7
11/20/2007 Aperture problem Aperture problem Slide by Steve Seitz, UW Slide by Steve Seitz, UW Solving the aperture problem Aperture problem How to get more equations for a pixel? • Basic idea: impose additional constraints – most common is to assume that the flow field is smooth locally – one method: pretend the pixel’s neighbors have the same (u,v) » If we use a 5x5 window, that gives us 25 equations per pixel! • http://www.psychologie.tu- dresden.de/i1/kaw/diverses%20Material/www.illusionworks.com/html/barber _pole.html Slide by Steve Seitz, UW RGB version Lucas-Kanade flow How to get more equations for a pixel? Prob: we have more equations than unknowns • Basic idea: impose additional constraints – most common is to assume that the flow field is smooth locally – one method: pretend the pixel’s neighbors have the same (u,v) Solution: solve least squares problem » If we use a 5x5 window, that gives us 25*3 equations per pixel! • minimum least squares solution given by solution (in d) of: • The summations are over all pixels in the K x K window • This technique was first proposed by Lucas & Kanade (1981) Slide by Steve Seitz, UW Slide by Steve Seitz, UW 8
Recommend
More recommend
