Its Moving! A Probabilistic Model for Causal Motion Segmentation in - PowerPoint PPT Presentation

It’s Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos Erik Learned-Miller with Manju Narayana Pia Bideau U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Do you see an insect? Uploaded by “Houssem” https://www.youtube.com/watch?v=adufPBDNCKo 2 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Walking Stick 3 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Other examples U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Other examples U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Problem Definition U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Prior work: 2 common approaches � Keypoint matching—”sparse” � Track points through multiple frames � Cluster tracks in subspaces � Typically non-causal � e.g. Vidal (2010), Fragkiadaki et al. (2012), Keuper et al. (2015) . � Optical Flow � Dense tracking information � enforces local smoothness and appearance constraints � slower � Narayana and Learned-Miller (ICCV13), Papazoglou and Ferrari (ICCV13), Zamalieva et al. (ECCV 2014) . 7 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Keypoint tracking 8 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Segmentation using Flow representation of flow optical flow segmented labels 9 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Complex optical flow original frame optical flow 10 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Difficulties � What makes flow complicated? � Camera motion along optical axis � Camera rotation � Dependence on scene depth � Noise in optical flow � Multiple moving objects 11 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Flow Angle Fields � Definition: Flow Angle Field The angle portion of each vector in an optical flow field. 12 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Flow angle fields optical flow magnitudes original scene (Sintel Optical Flow Database) optical flow angles 13 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Observation � For a translating camera and a rigid scene: The angle field depends ONLY on direction of motion. 14 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Observation � For a translating camera and a rigid scene: The angle field depends ONLY on direction of motion. 15 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Motion along y axis optical flow angle for Scene 1 for Scene 1 16 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Motion along y axis optical flow angle for Scene 2 for Scene 2 17 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Motion along z axis angle optical flow for Scene 1 for Scene 1 18 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Motion along z axis optical flow angle for Scene 2 for Scene 2 19 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translational “canonical” flow angle fields � Optical flow angle fields for different translations 20 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

z-motion example 21 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

z-motion example 22 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Segmentation using angle fields � Estimate which translations are present � Estimate which pixels belong to each translation optical flow observed FOF segmented labels 23 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translation only: An example Translation portion of optical flow Angle field of translation portion of optical flow 24 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translation only: An example Translation Angle Field 25 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translation only: An example Best fit to canonical Translation Translation Angle Field Angle Field 26 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translation only: An example Best fit to canonical Translation Translation Angle Field Angle Field closed form solution, by Bruss and Horn, 1983 27 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Translation only: An example Best fit to canonical Translation Translation Angle Field Angle Field Residual error 28 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Mixture of Canonical Angle Fields 29 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Generative model of flow orientations, Narayana et al. (ICCV 2013) segment pixel coordinates 3D-direction flow associated angle with each segment 30 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

So far... � Canonical angle fields result from translational motion. � But how do we � Deal with camera rotation? � Deal with noisy optical flow? 31 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Example with rotational camera motion Total optical flow Angle field of total optical flow 32 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Example with rotational camera motion Total optical flow Angle field of total optical flow Foreground tree does not fit canonical angle field 33 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Dealing with camera rotation flow angle original frame optical flow Does there exist a rotation field R, such that when R is subtracted from optical flow , the resulting flow angle field is close to a canonical translation angle field? 34 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Finding the camera rotation 35 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Finding the camera rotation translation optimization 36 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Finding the camera rotation translation optimization Our own implementation: simple optimization in Matlab, calling Bruss and Horn as inner loop. 37 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Subtracting off rotation Original Opt. Flow Estimated “Compensated flow” Rotation Angle field without rotation Angle field with rotation 38 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Pixels that don’t belong - = translation field ideal translation field error image 39 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Algorithm: main body 40 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Algorithm Details � Initialization: Segment first frame � Main body: Segment current frame given posteriors from previous frame 1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors 41 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Algorithm Details � Initialization: Segment first frame � Main body: Segment current frame given posteriors from previous frame 1. Compute optical flow 2. Compute motion component priors 3. Compute camera rotation, “subtract off” 4. Compute motion component likelihoods 5. Compute motion component posteriors 42 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences

Optical Flow � Sun, Roth, and Black. “Secrets of Optical Flow Estimation and their Principles.” CVPR 2010. 43 U NIVERSITY OF M ASSACHUSETTS , A MHERST • College of Computer and Information Sciences