Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust - PowerPoint PPT Presentation

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust Robotics Group, MIT CSAIL LPM Workshop IROS 2017 September 28, 2017 with Nicholas Roy 1

We want Autonomous Vision-Based Navigation https://commons.wikimedia.org/wiki/File%3AQuadcopter_Drone.png 2 https://commons.wikimedia.org/wiki/File%3ACanon_90-300mm_camera_lens.jpg

But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Efficiency Density https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 3 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Efficiency Density https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 4 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

But Dense Monocular SLAM is Expensive Sparse Methods Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Efficiency Dense Methods DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014) Density https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 5 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

But Dense Monocular SLAM is Expensive https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://commons.wikimedia.org/wiki/File%3AQuadcopter_Drone.png 6 https://commons.wikimedia.org/wiki/File%3ACanon_90-300mm_camera_lens.jpg

But Dense Monocular SLAM is Expensive Sparse Methods ? Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014) Semi-Dense Methods Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016 Efficiency Dense Methods DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014) Density Can we more efficiently estimate dense geometry? https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 7 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

What Would a Dense Method Do? Image 8

Estimate Depth for Every Pixel Depthmap 9

Dense Methods Oversample Geometry Depthmap One depth estimate per pixel 100k - 1M pixels per image 10

Dense Methods Make Regularization Hard(er) Depthmap One depth estimate per pixel 100k - 1M pixels per image 11

Meshes Encode Geometry with Fewer Depths Depthmap 12

Meshes Encode Geometry with Fewer Depths Depthmap 13

Meshes Encode Geometry with Fewer Depths Depthmap 14

Meshes Encode Geometry with Fewer Depths Depthmap One depth estimate per vertex 100 - 10k vertices per mesh 15

Meshes Make Regularization Easy(er) Depthmap One depth estimate per vertex 100 - 10k vertices per mesh 16

FLaME: Fast Lightweight Mesh Estimation Fast (< 5 ms/frame) Lightweight (< 1 Intel i7 CPU core) Runs Onboard MAV 17 Greene and Roy, ICCV2017

FLaME: Fast Lightweight Mesh Estimation Image and pose Inverse Depth Feature Selection Estimation Depthmap interpolation Delaunay Triangulation Variational Regularizer Output (NLTGV 2 -L1) 18

FLaME: Fast Lightweight Mesh Estimation Image and pose Inverse Depth Feature Selection Estimation Depthmap interpolation Delaunay Triangulation Variational Regularizer Output (NLTGV 2 -L1) 19

Select Easily Trackable Features - At each frame we sample trackable pixels or features over the image - These features will serve as potential vertices to add to the mesh 20

Select Easily Trackable Features - At each frame we sample trackable pixels or features over the image - These features will serve as potential vertices to add to the mesh 21

Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 22

Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 23

Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh 24

Select Easily Trackable Features - At each frame we sample trackable pixels over the image domain - These features will serve as potential vertices to add to the mesh Select pixel in each grid cell with maximum score: 25

FLaME: Fast Lightweight Mesh Estimation Image and pose Inverse Depth Feature Selection Estimation Depthmap interpolation Delaunay Triangulation Variational Regularizer Output (NLTGV 2 -L1) 26

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 27

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 28

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 29

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 30

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons 31

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons - Each successful match generates an inverse depth measurement 32

Estimate Inverse Depths for Features - We track features across new images using direct epipolar stereo comparisons - Each successful match generates an inverse depth measurement - Measurements are fused over time using Bayesian update 33

FLaME: Fast Lightweight Mesh Estimation Image and pose Inverse Depth Feature Selection Estimation Depthmap interpolation Delaunay Triangulation Variational Regularizer Output (NLTGV 2 -L1) 34

Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 35

Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 36

Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations 37

Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations - We can project 2D mesh into 3D using the inverse depth for each vertex 38

Triangulate Features into Mesh - Once a features inverse depth variance drops below threshold, we insert it into the mesh using Delaunay triangulations - We can project 2D mesh into 3D using the inverse depth for each vertex 39

FLaME: Fast Lightweight Mesh Estimation Image and pose Inverse Depth Feature Selection Estimation Depthmap interpolation Delaunay Triangulation Variational Regularizer Output (NLTGV 2 -L1) 40

Smooth Mesh Using Variational Regularization - Mesh inverse depths are noisy and prone to outliers - Need to spatially regularize/smooth inverse depths Raw 41

Smooth Mesh Using Variational Regularization - Mesh inverse depths are noisy and prone to outliers - Need to spatially regularize/smooth inverse depths - Will exploit graph structure to make optimization fast and incremental Raw Smoothed 42

Define Variational Cost over Inverse Depthmap Raw idepthmap 43

Define Variational Cost over Inverse Depthmap Smoothed idepthmap Raw idepthmap 44

Define Variational Cost over Inverse Depthmap Smoothed idepthmap Raw idepthmap NLTGV 2 stands for Non-Local Total Generalized Variation (Second Order) Ranftl et al. 2014, Pinies et al. 2015 45

Approximate Cost over Mesh - Reinterpret mesh as graph 46

Approximate Cost over Mesh - Reinterpret mesh as graph 47