In the name of Allah f the compassionate, the merciful Digital - PowerPoint PPT Presentation

In the name of Allah f the compassionate, the merciful

Digital Video Processing Digital Video Processing S. S. Kasaei Kasaei Room: CE 307 Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu @ Webpage: http://sharif.edu/~skasaei Lab. Website: http://ipl.ce.sharif.edu

Acknowledgment g Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001].

Chapter 6 Chapter 6 2-D Motion Estimation Part II: Advanced Techniques

Outline � Problems with EBMA � Deformable block matching algorithm (DBMA): � Node-based motion model � Mesh-based motion estimation: � Mesh-based motion representation � Mesh-based motion estimation � Global motion estimation: G oba ot o est at o � Direct method � Indirect method � Region-based motion estimation � Region based motion estimation � Multiresolution motion estimation: � Hierarchical block matching algorithm (HBMA) � Summary � Summary Kasaei 6

Problems with Exhaustive Block-Matching Algorithm (EBMA) � Blocking artifact (discontinuity across block oc g a ac (d sco u y ac oss b oc boundaries) in the predicted image: � Because the block-wise translation model is not accurate. � Real motion in a block may be more complicated than a Real motion in a block may be more complicated than a pure translation (rotation, zooming, multiple objects, …). Fix: Deformable BMA: � � Uses a more sophisticated model: affine bilinear or perspective � Uses a more sophisticated model: affine, bilinear, or perspective mapping (to describe block motion). Kasaei 7

Problems with EBMA � There may be multiple objects with different � There may be multiple objects with different motions in a block. � Fix: � Region-based motion estimation. � Mesh-based motion estimation (using adaptive meshes). � Intensity changes may be due to illumination � Intensity changes may be due to illumination effect: � Should compensate for illumination effect before applying the “constant intensity assumption”. Kasaei 8

Problems with EBMA � Motion field is somewhat chaotic: � Because MVs are estimated independently from block to block. � Fix: � Fix: Imposing smoothness constraint explicitly. � Multiresolution approach. � Mesh-based motion estimation. Mesh-based motion estimation � � � Wrong MV in flat regions: � This is because motion is indeterminate when spatial gradient is near zero. di t i � Ideally, should use non-regular partitions. � Fix: region-based motion estimation. g Kasaei 9

Problems with EBMA � Requires tremendous computation! � Requires tremendous computation! � Fix: � Fast algorithms. Fast algorithms. � Multiresolution approaches. Kasaei 10

Deformable Block-Matching Algorithm (DBMA) Allowed block deformation depends on the used motion model. Kasaei 11

Overview of DBMA � Three steps: p � Partition the anchor frame into regular blocks. � Model the motion in each block by a more complex motion motion. � A 2-D motion caused by a flat surface patch undergoing a rigid 3-D motion can be approximated well by a projective mapping projective mapping. � Projective mapping can be approximated by: � affine mapping + bilinear mapping. � Various possible mappings can be described by a node- based motion model. Kasaei 12

Overview of DBMA � Estimate the motion parameters block by block p y independently. � Discontinuity problem cross block boundaries still remains remains. � Still cannot solve the problem of multiple motions within a block or changes due to illumination effect! g Kasaei 13

Problems with DBMA � There might be motion discontinuity across block g y boundaries (because nodal MVs are estimated independently from block to block): � Fix: mesh-based motion estimation � Fix: mesh-based motion estimation. � First apply EBMA to all blocks. Kasaei 14

Problems with DBMA � Cannot do well on blocks with multiple moving Cannot do well on blocks with multiple moving objects or changes due to illumination effect. � Three mode method: First, apply EBMA to all blocks. � Blocks with small EBMA errors have translational motion . � Blocks with large EBMA errors may have non-translational g y � motion . � First, apply DBMA to these blocks. � Blocks still having errors are non-motion compensable. g [Ref] O. Lee and Y. Wang, Motion compensated prediction � using nodal-based deformable block matching. J. Visual Communications and Image Representation (March 1995), 6 26 34 6:26-34 Kasaei 15

Affine & Bilinear Model � Affine (6 parameters): Affine (6 parameters): � Good for mapping triangles to triangles.  d    + + d ( ( x , y ) )   a a x a y   x 0 1 2 =     + + d ( x , y ) b b x b y     y 0 1 2 � Bilinear (8 parameters): � Good for mapping blocks to quadrangles. + + +  d ( x , y )   a a x a y a xy  x 0 1 2 3 =     + + + d ( x , y ) b b x b y b xy     y 0 1 2 3 Kasaei 16

Difficulties in Estimating Affine & Bilinear Motion Parameters � The coefficients need floating point precision. � The coefficients have different influence on the � The coefficients have different influence on the estimated motion. � 0 -th order coefficients ( a0,b0 ) represent the translation component. � Other coefficients’ influence depends on pixel coordinates. Kasaei 17

Node-Based Motion Model � Control nodes can move freely; in this example: block freely; in this example: block corners. � Motion in other points are i t interpolated from the nodal l t d f th d l MVs, d m,k. � Control node MVs can be described with integer- or half- pel accuracy, all have the same importance. displacement at any point in element m di l t t i t i l t � Translation (1-node), affine (3-nodes), & bilinear (4-nodes) are special cases of this p model. “interpolation kernel” associated with node k in element m Kasaei 18

Interpolation Kernels � To guarantee continuity across element boundary: g y y � Shape functions of standard triangular element: � Affine function. Kasaei 19

Estimation of Nodal Motions � Shape functions of standard quadrilateral p q element: � Bilinear function. � Objective DFD function: Difficult to calculate! Kasaei 20

Estimation of Nodal Motions � Search method: Search method: � Exhaustive search: Search K nodal MVs simultaneously in integer- or half-pel � accuracy (may not be feasible in practice) accuracy (may not be feasible in practice). � Gradient descent approach: See textbook for the Newton-Raphson update algorithm. � Solution depends on the initial solution. � � A good initial solution is the translation MV found using EBMA. � One can use the average of the motion vectors of the 4 blocks g attached to each node as the initial estimate of the MV for that node. It will then be updated. Kasaei 21

Mesh-Based Motion Estimation (An Overview) non-overlapping polygonal elements triangular mesh h quadrilateral mesh Kasaei 22

Mesh-Based vs. Block- Based Motion Estimation Based Motion Estimation block-based backward ME (blocking artifacts) mesh-based backward ME (continuous tracking better to (continuous tracking, better to have separate meshes for different objects) mesh-based forward ME

Mesh-Based Motion Model The motion in each element is interpolated from nodal MVs: � Mesh based vs node based model: Mesh-based vs. node-based model: � � Mesh-based : Each node has a single MV , which influences the � motion of all four adjacent elements. Node-based : Each node can have four different MVs depending p g � on within which element it is considered to be in. Kasaei 24

Mesh Generation & Motion Estimation � Two problems: p � Given a mesh in the anchor frame, determine nodal positions in the target frame – Motion estimation. � Set up the mesh in the anchor frame so that the mesh � Set up the mesh in the anchor frame, so that the mesh conforms with object boundaries – Mesh generation. Backward ME: can use either regular mesh or object adaptive � mesh at each new frame. � Motion estimation is easier with a regular mesh, but adaptive mesh can yield more accurate result. Forward ME: � � Only needs to establish a mesh for the initial frame. Meshes in the following frames depend on the nodal MVs between successive frames. � To accommodate appearing/disappearing objects, the mesh � To accommodate appearing/disappearing objects, the mesh geometry needs to be updated. � We only discuss motion estimation problem here. Kasaei 25

Estimation of Nodal Motion Unlike DBMA, all nodal MVs should be estimated simultaneously. � Unless the anchor frame uses a regular mesh, the interpolation � kernels are complicated kernels are complicated. To simplify, use a mapping to a master element: � * u * * mapping [ J (u): Jacobian] function Kasaei 26