Full-Frame Video Stabilization with Motion Inpainting Yasuyuki - PowerPoint PPT Presentation

Full-Frame Video Stabilization with Motion Inpainting Yasuyuki Matsushita, E Eyal Ofek l Of k Weina Ge Xiaoou Tang Heung-Yeung Shum g g IEEE Trans on PAMI, July 2006

Outline Outline • Introduction I t d ti • Proposed Method • Experimental results • Quantitative Evaluation Quantitative Evaluation • Computation Cost • Conclusion C l i 2

Introduction Introduction • Stabilization ： St bili ti – Remove undesirable motion caused by unintentional shake of a human hand. i t ti l h k f h h d • remove high frequency camera motion vs. completely remove camera motion completely remove camera motion. • full frame vs. trimming • motion inpainting vs. mosaicing p g g 3

4 Prior Work vs Now Prior Work vs. Now

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 5

Global Motion Estimation Global Motion Estimation • GM is estimated by aligning pair-wise GM i ti t d b li i i i adjacent frames. − min( min( ( ( ) ) ( ( )) )) – I Tp I Tp I I p p ' t t t t T • Hierarchical motion estimation – construct an image pyramid – construct an image pyramid – start from the coarsest level • By applying the parameter estimation for By applying the parameter estimation for every pair of adjacent frames, a global transformation chain is obtained. j T i i H.-Y. Shum and R. Szeliski, “ Construction of Panoramic Mosaics with Global and Local 6 Alignment, ” Int ’ l J. Computer Vision, vol. 36, no. 2, pp. 101-130, 2000.

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 7

Image Deblurring Image Deblurring • Transferring sharper image pixels T f i h i i l from neighboring frames. – evaluates the “relative blurriness” 1 b = = b ∑ ⊗ + ⊗ t 2 2 {(( )( )) (( )( )) } f I p f I p y x t t t t p t – evaluates the “alignment error” = − ' t t t ( ( ) ) | | ( ( ) ) ( ( ) | ) | E p p I T p p I p p ' ' t t t t t t t t t t t t 8

Image Deblurring Image Deblurring • Blurry pixel are replaced by interpolating Bl i l l d b i t l ti shaper pixels. + ∑ ' t t ( ) ( ) ( ) I p w p I T p t t t ' t t ' t t ฀ ฀ = ∈ ' ( ) t N + ∑ I p t t t 1 ( ) w p ' t t ∈ ' t N • w is the weight factor which consists of w is the eight factor hich consists of the pixel-wise alignment error and relative blurriness blurriness ⎧ < 0 b ⎪ 1 if t = ⎨ t b ( ) w p ' t α b ' t t t ⎪ ⎪ ⎩ ⎩ otherwise otherwise + + α α b b t ' ( ( ) ) E E p p ' ' t t t t 9

10 Image Deblurring Image Deblurring

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 11

12 k = σ Motion Smoothing Motion Smoothing , 2 σ / 2 2 k − e πσ 1 2 = ( ) G k }, k ≤ + t ( ) ( ) G k G k j ≤ k ⊗ ⊗ − i j t t T T { : ∑ ∑ t i N ∈ = = t t N S S

13 Motion Smoothing Motion Smoothing

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 14

Local Motion Estimation Local Motion Estimation • A pyramidal version of Lucas-Kanade A id l i f L K d optical flow computation is applied to obtain the local motion field. bt i th l l ti fi ld J. Bouguet, “ Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the 15 Algorithm, ” OpenCV Document, Intel, Microprocessor Research Labs, 2000.

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 16

Motion Inpainting Motion Inpainting • Mosaicing with consistency constraint. M i i ith i t t i t < ⎧ t ' ( ) ( ( )) if v p T midian I T p = ⎨ t t ⎨ ' ' t t t t ( ( ) ) I I p p t t ⎩ otherwise keep it as missing where 1 − ∑ 2 = − ' ' t t ( ) ( ( ) ( )) v p I T p I T p ' ' t t t t t t t t 1 n ∈ ' t M t 1 1 ∑ = ' ' t t ( ) ( ) I T p I T p ' ' t t t t t t n ∈ ' t M t 17

Motion Inpainting Motion Inpainting A. Telea, “ An Image Inpainting Technique Based on the Fast Marching Method, ” J. Graphics Tools, 18 vol. 9, no. 1, pp. 23-34, 2004.

Motion Inpainting Motion Inpainting • The motion value for pixel p t is Th ti l f i l i generated by a weighted average of the motion vectors of the pixels H(p t ) th ti t f th i l H( ) ∈ ∑ ( , ) ( | ) w p q F p q t t t t ∈ = q q H H ( ( p p ) ) ( ( ) ) F p F t t t t ∑ t ( , ) w p q t t ∈ ( ) q H p t t where ⎡ ∂ ∂ ⎤ Δ ( ) ( ) F q F q ⎡ ⎤ x t x t x ∂ ∂ ⎢ ⎥ x y = + ∇ − = + ( | ) ( ) ( )( ) ( ) ⎢ ⎥ F p q F q F q p q F q Δ ∂ ∂ t t t t t t t ⎢ ⎥ ⎣ ( ) ( ) F q F q ⎦ y y t y t ⎣ ⎦ ∂ ∂ x y 1 1 = ( , ) w p q − + − − + ε t t 19 || || || ( ) ( ) || p q I q p q I q t t t ' t ' t t t ' t '

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion Output Video 20

21 Summary of the Algorithm Summary of the Algorithm

Experimental Results Experimental Results • 30 video clips (about 80 minutes) 30 id li ( b t 80 i t ) with different types of scenes • k = 6 for motion smoothing • 5x5 filter for motion inpainting p g 22

23 Experimental Results (1) Experimental Results (1)

24 Experimental Results (2) Experimental Results (2)

25 Experimental Results (3) Experimental Results (3)

incorrect estimation of motion 26 Failure Cases –

27 abrupt changes of motion Failure Cases –

Quantitative Evaluation Quantitative Evaluation • Deviation from the Ground Truth. D i ti f th G d T th • MAD of intensity 28

Quantitative Evaluation Quantitative Evaluation • Evaluation of Spatio-Temporal Smoothness. E l ti f S ti T l S th – The normalized discontinuity measure D is defined as defined as 1 n 1 n ∑ ∑ = ∇ = ∇ ⋅∇ || || D I I I i i i n n i i ⎡ ⎡ ⎤ ⎤ + − − ∂ ∂ ⎡ ⎡ ⎤ ⎤ I I ( 1, , ) ( 1, , ) I x y t I x y t ∂ ⎢ ⎥ x ⎢ ⎥ ∇ = ≈ + − − ∂ I ( , 1, ) ( , 1, ) I ⎢ ⎥ I x y t I x y t ⎢ ⎥ ∂ y ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ + + − − ∂ ∂ ⎣ ⎣ ( , , ( 1) 1) ( , , ( 1) 1) ⎦ ⎦ I x y t I x y t I x y t I x y t I I ⎣ ⎣ ⎦ ⎦ ∂ t – The relative smoothness is evaluated by (D M -D O )/(D M -D A ) (D D )/(D D ) – 5.9%~23.5% smoother than mosaicing 29

Computation Cost Computation Cost • 2.2 frames/s for 720x486 video with 2 2 f / f 720 486 id ith P4 2.8GHz CPU 30

Conclusion Conclusion • Motion inpainting instead of cropping. Motion inpainting instead of cropping • Deblurring without estimating PSFs. • Spatial smoothness is indirectly guaranteed by the smoothness of the extrapolated motion extrapolated motion. • Temporal consistency on both static and dynamic areas is given by optical and dynamic areas is given by optical flow from the neighboring frames. 31

Thank You Hsiao, Chih-Chieh presented by 蕭志傑 蕭志傑