4DVideo & Dynamic Scenes Torsten Sattler and Martin Oswald Spring 2018 1 Institute of Visual Computing
3D Reconstruction over time? 2 Institute of Visual Computing
Motivation: Dynamic Scenes 3 Institute of Visual Computing
Applications Free Viewpoint TV Markerless Motion Capture Motion Analysis for Sports Content Creation for Movies and Games / VR / AR Special Effects for Movies e.g. „Bullet Time“ Effect from „The Matrix“ The making of: 4 Institute of Visual Computing
Overview Calibrate and 3D Video-based synchronize reconstruction Rendering 3D representation Free viewpoint video of the scene content Input videos 5 Institute of Visual Computing
Surface Representations Explicit Representation Implicit Representation embedding volume is labeled (indicator surface is directly parametrized function / signed distance to surface) -> (n-1)-dimensional space -> n-dimensional space e.g. mesh, spline function/NURBS e.g. voxel grid, octree, tetrahedra + efficient storage, small data amounts + topology changes are trivial - topology changes are difficult + watertight, closed manifold - mathematical operations (e.g. set + mathematical operations (e.g. set union/intersection, average) are complex union/intersection, average) are simple - expensive storage, large data amounts 6 Institute of Visual Computing
Overview • Model-based Approaches • Spatio-temporal 3D Reconstruction • Scene Flow Approaches • Hybrid Approaches 7 Institute of Visual Computing
Marker-less Motion Capture (Ballan and Cortelazzo, 3DPVT‘08) 8 Institute of Visual Computing
Obtaining a 3D Model Captured Images Home-made 3D Silhouette + Photoconsistency body scanner [Ballan et al. ‘06] wavelet based texture mapping 9 Textured model Wireframe model Institute of Visual Computing
Recording Studio § 4 cameras § 21 fps § 0.8 Mpixels 10 Institute of Visual Computing
Marker-less Motion Capture § Start with known pose Track pose over time with all cameras § Optical flow & silhouette constraints § 11 Institute of Visual Computing
Interactions Close contacts 2 people + an object (83 DOF) More challenging situations Fails 12 Institute of Visual Computing
Model-based Tracking (Vlasic et al., TOG‘08) 13 Institute of Visual Computing
Face Tracking (Model-based) (Cao et al., SIGGRAPH‘15) 14 Institute of Visual Computing
A More Challenging Problem << Much more challenging Tracking closely Motion Capture of interacting people Interacting Hands § Collisions § Occlusions § Self-similarities, similar color 15 Institute of Visual Computing
Motion Capture of Hands (Ballan et al., ECCV‘12) + Template Model + Kinematic Structure Scene recorded from multiple viewing angles 16 Institute of Visual Computing
Motion Capture of Hands Input: Output: Scene Motion (angles and positions) 17 Institute of Visual Computing
Motion Capture of Hands Input: Output: Full 3D Geometry of the Scene 18 Institute of Visual Computing
Collisions and Self-Intersections § Lack of information § Overfitting Collisions Self-Intersections Colliding faces Local Distance Field 19 Institute of Visual Computing
Results 20 Institute of Visual Computing
How to handle additional Objects Scan the object 3D models Unique Use the algorithm object as it is!! Virtual scene § Collisions • bone Handled transparently § Occlusions • 21 Institute of Visual Computing
Results 22 Institute of Visual Computing
Results 23 Institute of Visual Computing
Overview • Model-based Approaches • Spatio-temporal 3D Reconstruction • Scene Flow Approaches • Hybrid Approaches 24 Institute of Visual Computing
Shape from Silhouettes (Franco and Boyer, ICCV‘05) Unreliable silhouettes: do not make decision about their location • Do sensor fusion: use all image information simultaneously • 25 Institute of Visual Computing
Bayesian Formulation (Franco and Boyer, ICCV‘05) Idea: find the content of the scene from images, as a • probability grid Modeling forward problem - explaining images given • grid state - is easy è sensor model . Bayesian inference: formulation of initial inverse • problem from the sensor model Simplification for tractability: independent analysis and • processing of voxels 26 Institute of Visual Computing
Modeling (Franco and Boyer, ICCV‘05) I : color information in images • Sensor model: B : background color models • F : silhouette detection variable (0 or 1): hidden • G X : occupancy at voxel X (0 or 1) • Silhouette Image likelihood likelihood Inference: Grid G x 27 Institute of Visual Computing
Visualization (Franco and Boyer, ICCV‘05) 28 Institute of Visual Computing
Variational Volumetric Reconstruction (Oswald et al., 4DMOD‘13) 3D Reconstruction Interior/exterior labeling weighted TV + data term [Kolev et al. IJCV’09] 4D Reconstruction Interior/exterior labeling spatial temporal data term regularization term 29 Institute of Visual Computing
Variational Volumetric Reconstruction (Oswald et al., BMVC‘14) 30 Institute of Visual Computing
Advanced: Handling Thin Structures (Oswald et al., ECCV‘14) input Jancosek and Pajdla CVPR‘11 Furukawa et al. PAMI‘11 no connectivity Oswald and Cremers 4DMOD‘13 Oswald et al., ECCV‘14 generalized connectivity 31 Institute of Visual Computing
Advanced: Handling Thin Structures (Oswald et al., ECCV‘14) 32 Institute of Visual Computing
Overview • Model-based Approaches • Spatio-temporal 3D Reconstruction • Scene Flow Approaches • Hybrid Approaches 33 Institute of Visual Computing
Sceneflow Approaches (Guan et al., CVPR‘10) 34 Institute of Visual Computing
Sceneflow Approaches (Joo et al., CVPR‘14) 35 Institute of Visual Computing
Sceneflow Approaches 36 Institute of Visual Computing
Overview • Model-based Approaches • Spatio-temporal 3D Reconstruction • Scene Flow Approaches • Hybrid Approaches 37 Institute of Visual Computing
Implicit Surface Representation Signed Distance Function 38 Institute of Visual Computing
Static Case: KinectFusion (and variants) (Whelan et al., RSS‘15) 39 Institute of Visual Computing
DynamicFusion (Newcombe et al., CVPR‘15) 40 Institute of Visual Computing
DynamicFusion (Newcombe et al., CVPR‘15) 41 Institute of Visual Computing
VolumeDeform (Innmann et al., ECCV‘16) 42 Institute of Visual Computing
KillingFusion (Slavcheva et al., CVPR‘17) 43 Institute of Visual Computing
Fusion4D (Dou et al., SIGGRAPH‘16) 44 Institute of Visual Computing
Next Lecture: Final Presentations! 45 Institute of Visual Computing
Recommend
More recommend
