Image Motion COMPSCI 527 — Computer Vision COMPSCI 527 — Computer Vision Image Motion 1 / 12
Outline 1 Image Motion 2 Occlusion, Correspondence, Motion Boundaries 3 Constancy of Appearance 4 Motion Field and Optical Flow 5 The Aperture Problem 6 Estimating the Motion Field COMPSCI 527 — Computer Vision Image Motion 2 / 12
Image Motion Sensor Irradiance → Pixel Values x y t T e T lens aperture sensor P s P • Irradiance is the patterns of colors on the image sensor, compressed as an ( r , g , b ) triple e ( x , t ) • A pixel value is a noisy and quantized version of the integral of irradiance over a volume of size P s × P s × T e : � ´ nT + T e / 2 � ˜ i P + P s / 2 � � f ( i , n ) = Q i P − P s / 2 e ( x , t ) d x dt + ν ( i , n ) nT − T e / 2 COMPSCI 527 — Computer Vision Image Motion 3 / 12
Image Motion Motion Field and Displacement • We are interested in real-valued positions x and times t • We have to infer those from integer-valued positions i and times n • Motion field at x and at time t is the instantaneous velocity of the image point that is visible at x at time t • The true image velocity of a world point • Displacement at x between times s and t is the difference between the position of a point at time t and the position of the same point at time s < t • Displacement is the integral of image velocity between two times • We typically approximate image velocity with displacement between consecutive image frames COMPSCI 527 — Computer Vision Image Motion 4 / 12
Occlusion, Correspondence, Motion Boundaries The Displacement Field is not a 1 − 1 Map • Point visible at x at time s that becomes hidden at time t (with s < t ) forms an occlusion • When s > t , this is called a disocclusion • If points x at time s and y at time t do not form an occlusion and are projections of the same point in the world, they correspond to each other • The displacement field is generally not integer-valued, so we cannot compute a 1 − 1 map between image pixels even if no occlusions or disocclusions exist • A displacement field is typically given as a map Z 2 → R 2 , undefined at occlusions COMPSCI 527 — Computer Vision Image Motion 5 / 12
Constancy of Appearance Constancy of Appearance • What is assumed to remain constant across images? • Motion estimation is impossible without such an assumption • Most generic assumption: The appearance of a point does not change with time or viewpoint • If two image points in two images correspond, they look the same • If x at time s and y at time t correspond, then e ( x , s ) = e ( y , t ) (finite-displacement formulation) d e ( x ( t ) , t ) • Equivalently, = 0 (differential formulation) dt • This is the key constraint for motion estimation COMPSCI 527 — Computer Vision Image Motion 6 / 12
Motion Field and Optical Flow Motion Field and Optical Flow • Extreme violations of constancy of appearance: B. K. P . Horn, Robot Vision , MIT Press, 1986 • Ill-defined distinction: • Motion field ≈ true motion • Optical flow ≈ locally observed motion COMPSCI 527 — Computer Vision Image Motion 7 / 12
Motion Field and Optical Flow The Optical Flow Constraint Equation • The appearance of a point does not change with time or d e ( x ( t ) , t ) viewpoint: = 0 dt • Total derivative, not partial: def d e ( x ( t ) , t ) e ( x ( t +∆ t ) , t +∆ t ) − e ( x ( t ) , t ) = lim ∆ t → 0 dt ∆ t d e ( x ( t ) , t ) • Use chain rule on = 0 to obtain the dt Optical Flow Constraint Equation (OFCE) ∂ e d x dt + ∂ e ∂ t = 0 ∂ x T def • v d x = dt is the unknown motion field • This is the key constraint for motion estimation COMPSCI 527 — Computer Vision Image Motion 8 / 12
The Aperture Problem The Aperture Problem • Issues arise even when the appearance is constant ∂ x T v + ∂ e ∂ e OFCE: ∂ t = 0 • Three equations in two unknowns • However, changes in irradiance are often caused by shading or shadows, which affects r , g , b similarly ∂ r ∂ r ∂ x 1 ∂ x 2 def ∂ e ∂ g ∂ g • The Jacobian has often rank close to 1 = ∂ x 1 ∂ x 2 ∂ x T ∂ b ∂ b ∂ x 1 ∂ x 2 and the components of ∂ e ∂ t are close to each other • This degeneracy is called the aperture problem COMPSCI 527 — Computer Vision Image Motion 9 / 12
The Aperture Problem The Aperture Problem for Black-and-White Video • The aperture problem is extreme for black-and-white images, for which e ∈ R : ∂ x T v + ∂ e ∂ e ∂ t = 0 (OFCE is one scalar equation in the two unknowns in v ) • We cannot recover motion based on local measurements alone • Only recover the normal component along the gradient ∂ e ∇ e ( x ) = ∂ x T (if the gradient is nonzero): = �∇ e ( x ) � − 1 [ ∇ e ( x )] T v ( x ) def v ( x ) • In practice, this is very often the case also with color video COMPSCI 527 — Computer Vision Image Motion 10 / 12
Estimating the Motion Field Smoothness and Motion Boundaries • The assumption of constancy of appearance yields about one equation in two unknowns at every point in the image • To solve for v , we need further assumptions • The motion field v : R 2 → R 2 is usually modeled as piecewise smooth (in space) • OFCE is solved in the LSE sense, and an additional regularization term is added to penalize deviations from smoothness • Smoothness holds almost everywhere, but not everywhere • Motion discontinuities are smooth image curves called motion boundaries COMPSCI 527 — Computer Vision Image Motion 11 / 12
Estimating the Motion Field Estimating the Motion Field • Because of the aperture problem, we can only estimate several displacement vectors d or motion field vectors v simultaneously, not each individually • Local methods • The image displacement d in a small window around a pixel x is assumed to be constant (extreme local smoothness) • Write one constancy of appearance equation for every pixel in the window • Solve for the one displacement that satisfies all these equations as much as possible (in the LSE sense) • Global methods • A data term measures deviations from constancy of appearance at every pixel in the image • A smoothness term measures deviations of the motion field v ( x ) from smoothness • Minimize a linear combination of the two types of terms COMPSCI 527 — Computer Vision Image Motion 12 / 12
Recommend
More recommend
