Tracking Deformable Objects with Point Clouds John Schulman, - PowerPoint PPT Presentation

Tracking ¡Deformable ¡Objects ¡with ¡Point ¡Clouds John ¡Schulman, ¡Alex ¡Lee, ¡Jonathan ¡Ho, ¡Pieter ¡Abbeel UC ¡Berkeley, ¡EECS ¡Department Thursday, July 4, 13

Goal § Track ¡deformable ¡objects ¡from ¡point ¡cloud ¡data § AssumpHon: ¡we ¡have ¡a ¡physical ¡model ¡of ¡the ¡object Kinect ¡RGB Rendering ¡of ¡state ¡esHmate 2D 1D 3D Thursday, July 4, 13

Snakes 329 single stereo snake on the outline of a piece of paper. The surface is rendered from a very different viewpoint than the original to emphasize that a 3D model of the piece of paper has been computed rather than merely a 2.5D model. 4.2 Motion Once a snake finds a salient visual feature, it ‘locks on.’ If the feature then begins to move slowly, the snake will simply track the same local minimum. Movement that is too rapid can cause a snake to flip into a different local minimum, but for ordinary speeds and video-rate sampling, snakes do a good job of tracking motion. Figure 8 Fig. 7. Bottom: Stereogram of a bent piece of paper. Below: shows eight selected frames out of a two-second Surface reconstruction from the outline of the paper matched using stereo snakes. The surface model is rendered from a video sequence. Edge-attracted snakes were in- very different viewpoint than the original to emphasize that it itialized by hand on the speaker’s lips in the first Energy ¡MinimizaHon ¡Methods is a full 3D model, rather than a 2SD model. frame. After that, the snakes tracked the lip movements automatically. The motion tracking was done in this case without any interframe constraints. Introducing such constraints will doubtless make the tracking x : ¡state ¡esHmate ¡ ¡ ¡ ¡ ¡ ¡ ¡ y : ¡observaHon E total ( x , y ) = E internal ( x ) + E external ( x , y ) discourage bending encourage model to and stretching match up with image / r min x E total ( x , y ) Fig. 8. Selected frames from a 2-second video sequence show- the speaker’s lips in the first frame, the snakes automatically ing snakes used for motion tracking. After being initialized to track the lip movements with high accuracy. Kass, Terzopoulos, Witkin, 1988 Thursday, July 4, 13

Snakes 329 single stereo snake on the outline of a piece of paper. The surface is rendered from a very different viewpoint than the original to emphasize that a 3D model of the piece of paper has been computed rather than merely a 2.5D model. 4.2 Motion Once a snake finds a salient visual feature, it ‘locks on.’ If the feature then begins to move slowly, the snake will simply track the same local minimum. Movement that is too rapid can cause a snake to flip into a different local minimum, but for ordinary speeds and video-rate sampling, snakes do a good job of tracking motion. Figure 8 Fig. 7. Bottom: Stereogram of a bent piece of paper. Below: shows eight selected frames out of a two-second Surface reconstruction from the outline of the paper matched using stereo snakes. The surface model is rendered from a video sequence. Edge-attracted snakes were in- very different viewpoint than the original to emphasize that it itialized by hand on the speaker’s lips in the first Energy ¡MinimizaHon ¡Methods is a full 3D model, rather than a 2SD model. frame. After that, the snakes tracked the lip movements automatically. The motion tracking was done in this case without any interframe constraints. Introducing such constraints will doubtless make the tracking x : ¡state ¡esHmate ¡ ¡ ¡ ¡ ¡ ¡ ¡ y : ¡observaHon E total ( x , y ) = E internal ( x ) + E external ( x , y ) discourage bending encourage model to and stretching match up with image / r min x E total ( x , y ) Fig. 8. Selected frames from a 2-second video sequence show- the speaker’s lips in the first frame, the snakes automatically ing snakes used for motion tracking. After being initialized to track the lip movements with high accuracy. Kass, Terzopoulos, Witkin, 1988 Wuhrer, Lang, & Shu 2012 Saltzman et al. 2007 Padoy & Hager 2011 Thursday, July 4, 13

ProbabilisHc ¡Methods Thursday, July 4, 13

ProbabilisHc ¡Methods Energy minimization methods E total ( x , y ) = E internal ( x ) + E external ( x , y ) min x E total ( x , y ) Thursday, July 4, 13

ProbabilisHc ¡Methods x : ¡state ¡esHmate ¡ ¡ ¡ ¡ ¡ ¡ ¡ y : ¡observaHon Energy minimization methods X p ( x , y ) / e − E internal ( x ) e − E external ( x , y ) E total ( x , y ) = E internal ( x ) + E external ( x , y ) min x E total ( x , y ) (MAP estimation) max p ( x , y ) x Thursday, July 4, 13

ProbabilisHc ¡Methods x : ¡state ¡esHmate ¡ ¡ ¡ ¡ ¡ ¡ ¡ y : ¡observaHon Energy minimization methods X p ( x , y ) / e − E internal ( x ) e − E external ( x , y ) E total ( x , y ) = E internal ( x ) + E external ( x , y ) min x E total ( x , y ) (MAP estimation) max p ( x , y ) x X e − E ( x , y , z ) p ( x , y ) / z : ¡correspondences z Thursday, July 4, 13

ProbabilisHc ¡Methods x : ¡state ¡esHmate ¡ ¡ ¡ ¡ ¡ ¡ ¡ y : ¡observaHon Energy minimization methods X p ( x , y ) / e − E internal ( x ) e − E external ( x , y ) E total ( x , y ) = E internal ( x ) + E external ( x , y ) min x E total ( x , y ) (MAP estimation) max p ( x , y ) x X e − E ( x , y , z ) p ( x , y ) / z : ¡correspondences z Figure 4 : Example of a thinned graph superimposed to the origi- Myronenko ¡& ¡Song ¡2007 Hahnel, ¡Thrun ¡& ¡Burgard ¡2003 Cagniart, ¡Boyer, ¡& ¡Ilic ¡2010 Thursday, July 4, 13

Challenges Thursday, July 4, 13

Challenges § ObservaHon ¡modeling: § correspondence ¡problem § noise § occlusions Thursday, July 4, 13

Challenges § ObservaHon ¡modeling: § correspondence ¡problem § noise § occlusions § Physical ¡constraints: § non-‑penetraHon § hard ¡constraints ¡on ¡bending ¡ and ¡stretching Thursday, July 4, 13

Challenges ContribuHons § ObservaHon ¡modeling: § correspondence ¡problem § noise § occlusions § Physical ¡constraints: § non-‑penetraHon § hard ¡constraints ¡on ¡bending ¡ and ¡stretching Thursday, July 4, 13

Challenges ContribuHons § ObservaHon ¡modeling: § Modeling ¡contribuHon: § correspondence ¡problem § ProbabilisHc ¡model ¡that ¡captures ¡correspondence ¡ problem, ¡noise, ¡and ¡occlusions § noise § occlusions § Physical ¡constraints: § non-‑penetraHon § hard ¡constraints ¡on ¡bending ¡ and ¡stretching Thursday, July 4, 13

Challenges ContribuHons § ObservaHon ¡modeling: § Modeling ¡contribuHon: § correspondence ¡problem § ProbabilisHc ¡model ¡that ¡captures ¡correspondence ¡ problem, ¡noise, ¡and ¡occlusions § noise § Algorithmic ¡contribuHon: § occlusions § Physical ¡constraints: § ModificaHon ¡of ¡the ¡EM ¡algorithm ¡that ¡accounts ¡ for ¡physical ¡constraints § non-‑penetraHon § Operates ¡by ¡only ¡introducing ¡external ¡forces ¡into ¡ § hard ¡constraints ¡on ¡bending ¡ physics ¡simulaHon ¡engines and ¡stretching Thursday, July 4, 13

Preliminaries Thursday, July 4, 13

Preliminaries § Physical ¡models Thursday, July 4, 13

Preliminaries § Physical ¡models § Image ¡/ ¡point ¡cloud ¡processing § background ¡subtracHon ¡(may ¡have ¡false ¡posiHves ¡& ¡negaHves) Thursday, July 4, 13

ObservaHon ¡Model camera Graphical ¡model                       Thursday, July 4, 13

ObservaHon ¡Model camera Graphical ¡model                   y 1 z 1 = (1 , 0 , 0)   y 1 = N ( x 1 , σ 2 )   Thursday, July 4, 13

ObservaHon ¡Model camera Graphical ¡model                   y 1 y 2 z 1 = (1 , 0 , 0)     z 2 = (0 , 0 , 1) y 1 = N ( x 1 , σ 2 ) y 2 = N ( x 3 , σ 2 )     Thursday, July 4, 13

EM ¡Algorithm Thursday, July 4, 13

EM ¡Algorithm X § MAP ¡inference ¡problem: X arg max log p ( x , y ) = arg max log p ( x , y , z ) x x z Thursday, July 4, 13

EM ¡Algorithm X § MAP ¡inference ¡problem: X arg max log p ( x , y ) = arg max log p ( x , y , z ) x x z § For ¡i=1,2,3,... Thursday, July 4, 13

EM ¡Algorithm X § MAP ¡inference ¡problem: X arg max log p ( x , y ) = arg max log p ( x , y , z ) x x z § For ¡i=1,2,3,... x Calculate posterior of E Step: q ( i ) ( z ) = p ( z | x ( i − 1) , y ) latent variables X Thursday, July 4, 13

Tracking Deformable Objects with Point Clouds John Schulman, - PowerPoint PPT Presentation

Tracking Deformable Objects with Point Clouds John Schulman, Alex Lee, Jonathan Ho, Pieter Abbeel UC Berkeley, EECS Department Thursday, July 4, 13 Goal Track deformable

Tracking deformable objects with WiSARD networks: a preliminary work INNOROBO 2014 European

Tracking Articulated Objects Alexander (Sasha) Lambert CS7495 Fall 2014 Tracking From Depth

Manipulation of 1D and 2D Deformable Objects Without Modeling Deformation Dmitry Berenson

Sof oft-Bod ody y Dy Dyna namics cs Deformable Objects A difficult problem Lots of

Path Planning and Execution For Deformable Objects Using a Voxel-Based Representation Calder

Video Object Tracking Real-time tracking of objects in video is an important problem in various

Simulation Deformable Objects High-quality deformable simulation is important Well-known

3D Shape Completion from Sparse Point Clouds Christian Diller 26 th July 2019 Motivation 2

Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild

and Deformable Objects in Hand-object Interactions Hao Zhang, Zi-Hao Bo, Jun-Hai Yong, Feng Xu *

Towards Real-time Simulation of Deformable Objects From mass-spring system to general

Potree - Rendering Point Clouds in Web Browsers Markus Schtz, www.potree.org/ Potree Potree

When you look up into the sky, you will often see clouds. No two clouds are the same, and there

Perception with Point Clouds Robert Platt Northeastern University Topics depth sensors

Effj fjcient Collision Detection While Rendering Dynamic Point Clouds M. Radwan, S. Ohrhallinger

Visualization of 3D data: - 3D objects Is there an alternative to N x 2D slices ? Clouds,

Finally: Motion and tracking Tracking objects, video analysis, low level motion Motion Wed,

Rigblocks: Player- Deformable Objects Lydia Choy, Ryan Ingram, Ocean Quigley, Brian Sharp,

Learning to Optimally Segment Point Clouds Peiyun Hu, David Held, Deva Ramanan Carnegie Mellon

Dynamic Graph CNN for learning on point clouds Wang Yue, et al. Otakar Jaek March 25, 2019

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Francis Engelmann*

Matching Deformable Objects in Clutter Emanuele Rodol` a USI Lugano Joint work with L. Cosmo

VALSE VA ON ONLINE Tr Tracking Mu Multiple Ob Objects in in Im Image Se Sequences

Geometric Registration for Deformable Shapes 2.2 Deformable Registration Variational Model