3D Reconstruction with Tango Ivan Dryanovski, Google Inc. Contents - PowerPoint PPT Presentation

3D Reconstruction with Tango Ivan Dryanovski, Google Inc.

Contents Problem statement and motivation The Tango SDK 3D reconstruction - data structures & algorithms Applications Developer tools

Problem formulation Goal: Create accurate, textured 3D models of indoor spaces • real-time, on mobile device • offline, in the cloud Motivation • immersive VR / AR - gaming, professional apps • large scale maps, floorpans - navigation

Problem formulation Challenges: • Partial, noisy depth data • Memory and computational constraints • Platform diversity

Tango software stack Motion Depth Tracking Area Learning perception

Tango 3D reconstruction Real-time, online process 100Hz 100Hz 6DoF Pose Application Volumetric Mesh tasks fusion extraction 10Hz 10Hz 10Hz • rendering Color images • using two-tier • using • occlusions TSDF voxel Marching • path planning grid Cubes 10Hz • collision Depth images detection Klingensmith, Matthew, et al. "Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields." Robotics: Science and Systems XI (2015).

Tango 3D reconstruction Offline process 6DoF Pose 6DoF Pose (IMU + features) 6DoF Pose Volumetric Mesh Mesh Mesh (IMU + features) Application fusion extraction simplification texturing tasks Color images Color images • using two-tier • using • vertex • texture atlas • rendering Color images TSDF voxel Marching decimation generation • occlusions grid Cubes • projective • path planning Depth images image • collision Depth images Depth images texturing detection

Signed Distance Function (SDF) Represent space using voxel grid 3 3 3 3 3 3 3 3 2 2 2 2 2 2 Voxel contains signed distance 3 2 1 1 1 1 1 function to nearest surface ( SDF ) 3 2 1 0 0 0 0 Only update near surface - 3 2 1 0 -1 -1 -1 estimate truncated function ( TSDF ) 3 2 1 0 -1 -2 -2 Grid is updated by fusing depth 3 2 1 0 -1 -2 -3 readings (using running weighted average filter) 3 2 1 0 -1 -2 -3 Curless, Brian, and Marc Levoy. "A volumetric method for building complex models from range images." Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 1996.

Signed Distance Function (SDF) Surface of objects is represented 3 3 3 3 3 3 3 by the zero-isosurface in voxel 3 2 2 2 2 2 2 grid 3 2 1 1 1 1 1 Traversing the isosurface is done 3 2 1 0 0 0 0 using Marching Cubes 3 2 1 0 -1 -1 -1 Output is a triangle mesh 3 2 1 0 -1 -2 -2 3 2 1 0 -1 -2 -3 3 2 1 0 -1 -2 -3 Curless, Brian, and Marc Levoy. "A volumetric method for building complex models from range images." Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 1996.

Voxel memory layout 64-bit per voxel (includes TSDF value, color value, filter weights)

Dynamic Two-Tier TSDF voxel grid Single TSDF grid for the entire scene would require too much memory Hierarchical grid - larger volumes contain sub-grids Volumes are sparsely allocated, and indexed by a hash map Nießner, Matthias, et al. "Real-time 3D reconstruction at scale using voxel hashing." ACM Transactions on Graphics (TOG) 32.6 (2013): 169.

Local frustum culling When adding new observation, do intersect the depth camera frustum with the coarse volume grid New volumes are allocated on- demand Klingensmith, Matthew, et al. "Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields." Robotics: Science and Systems XI (2015).

Local frustum culling Only intersected volumes receive TSDF updates Only intersected volumes have their mesh segments re-extracted Allows building of large models while updating and streaming only local segments Klingensmith, Matthew, et al. "Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields." Robotics: Science and Systems XI (2015).

Large-scale reconstruction 25 individual trajectories (~3 hrs) Final 3D map (false color by height) Co-located using area learning

Large-scale reconstruction Ivan sits here

Developer tools Tango Client API • C / Java / Unity • 6DoF pose, images, depth

Developer tools Tango 3D Tango Client API Reconstruction API 
 • C / Java / Unity •C / Java / Unity • 6DoF pose, images, •provides textured 3D depth meshes

Developer tools Constructor 
 •Android app for building and sharing 3D models Tango 3D Tango Client API Reconstruction API 
 • C / Java / Unity •C / Java / Unity • 6DoF pose, images, •provides textured 3D depth meshes

GPU implementation If no shared memory between CPU & GPU, copying data is a bottleneck 100Hz 100Hz 6DoF Pose Application Volumetric Mesh tasks fusion extraction 10Hz 10Hz 10Hz • rendering Color images • using two-tier • using • occlusions TSDF voxel Marching COPY • path planning grid Cubes 10Hz • collision Depth images detection COPY Depth images as graphics texture? Collision calculations on graphics hardware?

Thank you! Q&A Ivan Dryanovski, Google Inc.