In the name of Allah the compassionate, the merciful
Digital Video Systems S. Kasaei S. Kasaei Room: CE 307 Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei Lab. Website: http://ipl.ce.sharif.edu
Acknowledgment Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001]
Chapter 6 2-D Motion Estimation Part I: Fundamentals & Basic Techniques
Outline � 2-D motion vs. optical flow � Optical flow equation & ambiguity in motion estimation � General methodologies in motion estimation � Motion representation � Motion estimation criterion � Optimization methods � Gradient descent methods � Pixel-based motion estimation � Block-based motion estimation � EBMA algorithm 6 Kasaei
2-D Motion Estimation � Motion estimation (ME) is an important part of many video processing tasks. � ME main applications are video compression, sampling rate conversion, filtering, … � For computer vision, motion vectors (MV) are used to deduce 3-D structure & motion parameters (sparse but accurate set of MVs are required). � For video coding, MVs are used to produce motion- compensated predicted frame to reduce required bitrate for coding MVs & prediction errors (tense and accurate set of MVs are required). 7 Kasaei
2-D Motion Estimation � An ME problem is converted to an optimization problem that involves key components of: � Parameterization of motion field. � Formulation of optimization criterion. � Searching for optimal parameters. Input Frames Optimal Motion Parameters Optimization Criteria Motion Field 8 Kasaei
2-D Motion vs. Optical Flow � 2-D Motion: Projection of 3-D motion. Depends on 3-D object motion & projection operator (physical aspects). � Optical flow: “Perceived” 2-D motion based on changes in image pattern, also depends on illumination & object surface texture. (a) A sphere is rotating under a constant ambient illumination, but observed image does not change. (b) A point light source is rotating around a stationary sphere, causing highlight point on sphere to rotate. (a) (b) 9 Kasaei
Correspondence & Optical Flow � 2-D displacement & velocity fields are projections of respective 3-D fields into image plane. � Correspondence field & optical flow field are displacement & velocity functions “perceived” from the time-varying image intensity pattern. � Correspondence field & optical flow field are also called “apparent 2-D displacement” field & “apparent 2-D velocity” field. 10 Kasaei
Correspondence & Optical Flow � Since we can only observe correspondence & optical flow fields, we assume that they are the same as the 2-D motion field. � When illumination condition is unknown, the best one can do is to estimate the optical flow. � Constant intensity assumption : The image of the same object point at different time intervals have the same luminance value. � Constant intensity assumption (CIA) Optical flow (OF) equation. 11 Kasaei
Optical Flow Equation Under " constant intensity assumption " : ψ + + + = ψ [(x,y,t) � (x+dx, y+dy, t+dt)] ( x d , y d , t d ) ( x , y , t ) x y t But, using Taylor' s expansion : ∂ ψ ∂ ψ ∂ ψ ψ + + + = ψ + + + ( x d , y d , t d ) ( x , y , t ) d d d x y t x y t ∂ ∂ ∂ x y y Compare the above two, we have the optical flow equation : ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ ∂ ψ T + + = + + = ∇ ψ + = d d d 0 or v v 0 or v 0 x y t x y ∂ ∂ ∂ ∂ ∂ ∂ ∂ x y t x y t t spatial [The velocity vector ( flow vector ) , v, is the unknown parameter. gradient One equation with two unknowns.] vector 12 Kasaei
Ambiguities in Motion Estimation � Optical flow equation only constrains the flow vector in aperture the gradient direction ( ). v ? problem n � The flow vector in the tangent v direction ( ) is under- ? t determined (aperture problem). � Also, in regions with constant v brightness ( ), the flow is ∇ ψ = 0 indeterminate Motion = + If: v v e v e estimation is unreliable in n n t t no ∂ ψ vt ! regions with flat texture, but ∇ ψ + = v 0 gradient n ∂ t vector more reliable near edges. magnitude 13 Kasaei
Ambiguities in Motion Estimation � To solve the undetermined component problem ( ) of OFE, one must impose additional v t constraints. � The most common constraints is that the flow vectors should vary smoothly spatially (to estimate the motion vector). ok ? 14 Kasaei
General Considerations for ME � Two categories of approaches: � Feature-based: More often used in object tracking & 3-D reconstruction from 2-D (least-squares fitting of features, good for global motions). � Intensity-based: Based on CIA (no simple model). More often used for motion compensated prediction (required in video coding), frame interpolation Our focus. � Three important questions: � How to represent (parameterize) the motion field? � What criteria to use to estimate motion parameters? � How to search for optimal motion parameters? 15 Kasaei
Motion Representation Pixel-based: Global: One MV at each pixel, Entire motion field is with some smoothness represented by a few constraint between global parameters adjacent MVs (very (global motion time consuming). representation; camera motion). Region-based: Block-based: Entire frame is divided Entire frame is divided into regions, then each into non-overlapping region corresponding to blocks, then motion in an object (or sub- each block is object) with consistent characterized by a few motion, is represented parameters (good by a few parameters compromise between (requires region accuracy & complexity, segmentation map, discontinuous across which pels have similar blocks, no multiple motions?). objects, scale, or 16 Kasaei rotation).
Motion Representation � Other representation: mesh-based representation. � Underlying image frame is partitioned into nonoverlapping polygonal elements. � Mvs at the corners of polygonal elements determine the entire motion field. � Mvs at the interior points of an element are interpolated from the nodal MVs. � Induces a motion field that is continuous everywhere. � Adaptive methods allow discontinuities when necessary (on object boundaries). 17 Kasaei
Notations ψ 1 x ( ) Anchor frame: ψ 2 x ( ) Target frame: Motion parameters: a Motion vector at a pixel in the anchor frame: d ( x ) Motion field: ∈ Λ d ( x ; a ), x Mapping function: = + ∈ Λ w ( x ; a ) x d ( x ; a ), x current reference frame in frame in video video coding 18 Kasaei coding
Regularization Theory Ill-posed problems. � Regularization methods. � Stochastic regularization methods. � Relaxation labeling. � � Discrete relaxation labeling. � Stochastic relaxation labeling. 19 Kasaei
Well-Posed Problems A mathematical problem is well-posed � when its solution: Exists. 1. Is unique. 2. Is robust to noise. 3. Physical simulation problems are well- � posed, but “inverse” problems are usually ill-posed. 20 Kasaei
Regularization Methods Basis idea behind regularization is: � to restrict the space of acceptable solutions, 1. and by choosing the function that minimizes an 2. appropriate functional. For solving an ill-posed problem, regularization � theory provides the mathematical function for choosing the norm and stabilizing functional that together characterize the global constraints for the problem. i.e., finding x that satisfies: || G ( x )||< C & min || F ( x ) -y ||. � 21 Kasaei
Stochastic Regularization Methods � Instead of regularization theory, a Bayesian formulation can be used to transform ill-posed inverse problems into the functional optimization framework. � Looking for the most likely model given a set of data. � Likelihood: evaluates how well the model describes the data ( stabilizing functional || F ( x ) -y ||). � A priori : evaluates the model ( norm || G ( x )||). � Other modeling approaches: � Minimum description length also describe the model constraints. � Other stochastic or probabilistic optimization methods: � Simulated annealing, � Genetic algorithms/ evolutionary strategy, � Expectation-maximization. 22 Kasaei
Recommend
More recommend
