In the name of Allah In the name of Allah the compassionate, the merciful
Digital Video Processing S. Kasaei S. Kasaei R Room: CE 307 CE 307 Department of Computer Engineering Sharif University of Technology E M il E-Mail: skasaei@sharif.edu k i@ h if d Webpage: http://sharif.edu/~skasaei Lab. Website: http://ipl.ce.sharif.edu
Acknowledgment Acknowledgment Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: based on the book: Video Processing & Communications written by: Yao Wang Jom Ostermann & Ya Oin Zhang written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001]
Chapter 6 Chapter 6 2-D Motion Estimation 2 D Motion Estimation Part I: Fundamentals & Basic Techniques
Outline � 2-D motion vs. optical flow � Optical flow equation & ambiguity in motion estimation estimation � General methodologies in motion estimation � Motion representation � Motion estimation criterion � Optimization methods � Gradient descent methods � Pixel-based motion estimation � Block-based motion estimation � EBMA algorithm 6 Kasaei
2 D Motion Estimation 2-D Motion Estimation � Motion estimation (ME) is an important part of many video processing tasks. � ME main applications are video compression, ME i li ti id i sampling rate conversion, filtering, … � For computer vision, motion vectors (MV) are used to � For computer vision motion vectors (MV) are used to deduce 3-D structure & motion parameters (sparse but accurate set of MVs are required). � For video coding MVs are used to produce motion � For video coding, MVs are used to produce motion- compensated predicted frame to reduce required bitrate for coding MVs & prediction errors (tense & accurate set of MVs are required) accurate set of MVs are required). 7 Kasaei
2-D Motion Estimation 2 D Motion Estimation � An ME problem is converted to an optimization problem that involves key p p y components of: � Parameterization of motion field. � Parameterization of motion field. � Formulation of optimization criterion. � Searching for optimal parameters � Searching for optimal parameters. Input Frames Optimal Motion Parameters Optimization Criteria Motion Field 8 Kasaei
2-D Motion vs Optical Flow 2 D Motion vs. Optical Flow � 2-D Motion: Projection of 3-D motion Depends on 3-D object motion & 2-D Motion: Projection of 3-D motion. Depends on 3-D object motion & projection operator (physical aspects). � Optical flow: “Perceived” 2-D motion based on changes in image pattern, also depends on illumination & object surface texture. also depends on illumination & object surface texture (a) A sphere is rotating under a ( ) A h i t ti d constant ambient illumination, but observed image does not change. g (b) A point light source is rotating around a stationary sphere, causing highlight point on causing highlight point on sphere to rotate. (a) (b) 9 Kasaei
Correspondence & Optical Flow p & p � 2-D displacement & velocity fields are projections of respective 3-D fields into image plane. � Correspondence field & optical flow field are displacement & velocity functions “perceived” from the time-varying image intensity pattern. h i i i i i � Correspondence field & optical flow field are also � Correspondence field & optical flow field are also called “apparent 2-D displacement” field & “apparent 2-D velocity” field. 10 Kasaei
Correspondence & Optical Flow p & p � Since we can only observe correspondence & optical flow fields, we assume that they are the same as the 2 D motion field 2-D motion field. � When illumination condition is unknown, the best one can do is to estimate the optical flow can do is to estimate the optical flow. � Constant intensity assumption : The image of the same object point at different time intervals have the same bj t i t t diff t ti i t l h th luminance value. � Constant intensity assumption (CIA) Optical flow C i i i (CIA) O i l fl (OF) equation. 11 Kasaei
Optical Flow Equation p q Under " constant intensity assumption " : ψ + + + = ψ [(x,y,t) � (x+dx, y+dy, t+dt)] ( x d , y d , t d ) ( x , y , t ) x y t But, using Taylor' s expansion : ∂ ψ ∂ ψ ∂ ψ ψ + + + = ψ + + + ( x d , y d , t d ) ( x , y , t ) d d d x x y y t t x x y y t t ∂ ∂ ∂ ∂ ∂ ∂ x y y Compare the above two, we have the optical flow equation : ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ T + + = + + = ∇ ψ + = d d d 0 or v v 0 or v 0 x y t x y ∂ ∂ ∂ ∂ ∂ ∂ ∂ x y t x y t t spatial [The velocity vector ( flow vector ) , v, is the unknown parameter. gradient One equation with two unknowns.] vector 12 Kasaei
Ambiguities in Motion Estimation Ambiguities in Motion Estimation � Optical flow equation only constrains the flow vector in aperture the gradient direction ( ) the gradient direction ( ). v v ? ? problem problem n � The flow vector in the tangent v direction ( ) is under- ( ) ? t t determined (aperture problem). � Also, in regions with constant v brightness ( brightness ( ), the flow is ∇ ψ ∇ ψ = ) the flow is 0 0 indeterminate Motion = + If: v v e v e n n t t estimation is unreliable in no ∂ ∂ ψ ψ vt ! t ! regions with flat texture, but ∇ ψ + = v n 0 gradient ∂ t more reliable near edges. vector magnitude 13 Kasaei
Ambiguities in Motion Estimation Ambiguities in Motion Estimation � To solve the undetermined component problem ( ) of OFE, one must impose additional v t constraints constraints. � The most common constraints is that the flow vectors should vary smoothly spatially (to estimate the motion vector). ok ok ? 14 Kasaei
General Considerations for ME General Considerations for ME � Two categories of approaches: � Feature-based: More often used in object tracking & 3-D reconstruction from 2-D (least-squares fitting of features, good for global motions). � Intensity-based: Based on CIA (no simple model) More � Intensity-based: Based on CIA (no simple model). More often used for motion compensated prediction (required in video coding), frame interpolation Our focus. � Three important questions: � How to represent (parameterize) the motion field? p (p ) � What criteria to use to estimate motion parameters? � How to search for optimal motion parameters? 15 Kasaei
Motion Representation Motion Representation Pixel-based: Global: One MV at each pixel, Entire motion field is with some smoothness represented by a few constraint between global parameters adjacent MVs (very (global motion time consuming). time consuming) representation; camera representation; camera motion). Block-based: Region-based: E ti Entire frame is divided f i di id d Entire frame is divided into non-overlapping into regions, then each blocks, then motion in region corresponding to each block is an object (or sub- characterized by a few y object) with consistent object) with consistent parameters (good motion, is represented compromise between by a few parameters accuracy & complexity, (requires region discontinuous across segmentation map, blocks no multiple blocks, no multiple which pels have similar objects, scale, or motions?). rotation). 16 Kasaei
Motion Representation Motion Representation � Other representation: mesh-based representation. � Underlying image frame is partitioned into non-overlapping polygonal elements. p yg � Mvs at the corners of polygonal elements determine the entire motion field. � Mvs at the interior points of an element are interpolated from the nodal MVs. h d l MV � Induced motion field is continuous everywhere. � Adaptive methods allow discontinuities when necessary (on object boundaries). 17 Kasaei
Notations Notations ψ 1 x ( ) Anchor frame: ψ 2 x ( ) Target frame: Motion parameters: M ti t a Motion vector at a pixel in the anchor f frame: d d ( ( x ) ) Motion field: ∈ Λ d ( x ; a ), x Mapping function: = + ∈ Λ w ( x ; a ) x d ( x ; a ), x reference current frame frame [in video [in video coding] coding] 18 Kasaei
Regularization Theory Regularization Theory Ill-posed problems. � Regularization methods Regularization methods. � � Stochastic regularization methods. � R l Relaxation labeling. ti l b li � � Discrete relaxation labeling. � Stochastic relaxation labeling. 19 Kasaei
Well-Posed Problems Well Posed Problems A mathematical problem is well-posed � when its solution Exists, 1. is unique, and 2. is robust to noise. 3. Physical simulation problems are well- Physical simulation problems are well- � � posed, but “inverse” problems are usually ill-posed. p 20 Kasaei
Recommend
More recommend
