PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D - PowerPoint PPT Presentation

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz, Thomas Funkhouser Princeton University National University of Defense Technology Technical University of Munich Google

RGB-D Reconstruction Microsoft Kinect Structure Sensor Xtion

RGB-D Reconstruction Bundle Fusion [Dai et al. 17]

RGB-D Reconstruction KinectFusion VoxelHashing [Newcombe/Izadi et al. 2011] [Niessner et al. 2013] BundleFusion [Dai et al. 2017] ElasticFusion Robust Recon. [Whelan et al. 2016] [Choi et al. 2015]

VoxelHashing Loop Closure BundleFusion

Loop Closure -> Feature Descriptor RGB Features: - SIFT, SURF, ORB, Freak, … - LIFT, MatchNet , … Keypoint-based Geometric Features: Are there additional primitives? - SHOT, FPFH, SpinImages , … - 3DMatch, …

Our Id Idea: Planar Feature Descriptors Coplanar Surface Patches

Existing Planar Matching is Local Point-to-Plane ICP [Chen & Medioni 91] Online Structure Analysis [Zhang et al. 2015] Fine-to-Coarse Registration [Halber and Funkhouser 2017]

Long-Range Constraints for SLAM Coplanar Surface Patches

Task: Co-planarity Matching? … …

PlaneMatch: Learning Co-planarity Features ➢ Color ➢ Depth ➢ Normals ➢ Plane Segmentation (Mask) ➢ … Learn from … 3D data!

Siamese Network Architecture 256D 256D 256D

Siamese Network Architecture 256D 256D 256D

Siamese Network Architecture 256D 256D 256D

Siamese Network Architecture 256D 256D 256D

Step 1: : Ext xtract Planar Patches RGB Planar Patches Depth

Step 2: : Ext xtract Global Rep. / Patch Depth RGB Normals Patch Mask

Step 3: : Ext xtract Local Rep. / Patch Depth RGB Normals Patch Mask

Local / Global Representations Global Local Representation Representation

Siamese Network Architecture 256D 256D 256D

Training: Self-Supervised Learning Positive Negative Anchor Anchor Positive Negative ScanNet [Dai et al. 2017] … 10 million triplets

Triplets for Training Negative Positive Anchor

Benchmark for Task of f Co-planarity Matching Positive pair (6k) Negative pair (6k) By patch size By pair distance

PlaneMatch Evaluation

PlaneMatch Ablation Study

PlaneMatch Registration : indicator variables ( ∈ [0,1] ) : transformation matrix

PlaneMatch Registration : indicator variables ( ∈ [0,1] ) : transformation matrix cop cop Pairs : plane pair set : plane-to-plane distance cop cop predicted by : confidence weight π coplanarity network

PlaneMatch Registration : indicator variables ( ∈ [0,1] ) : transformation matrix kp kp Pairs : point pair set : point-to-point distance kp kp predicted by : confidence weight π SIFT keypoints

PlaneMatch Registration : indicator variables ( ∈ [0,1] ) : transformation matrix : threshold for error (0.01 m) If > , = 0 If < , = 1 Robust optimization following [Choi et al. 15] / [Zollhoefer et al. 14] / [Zach et al. 14]

PlaneMatch Registration Results .

PlaneMatch Registration Results BundleFusion [Dai et al.17] PlaneMatch (Ours)

PlaneMatch Registration Results BundleFusion [Dai et al.17] PlaneMatch (Ours)

Evaluation on TUM-RGBD RMSE in cm (lower is better)

Ablation on TUM-RGBD RMSE in cm (lower is better)

Effect of f Long-range Co-planar Pairs 1-5m 1-5m 1-5m 1-5m 0% deduction 50% deduction 100% deduction

Effect of f Long-range Co-planar Pairs 1-5m 1-5m 1-5m 1-5m 0% deduction 50% deduction 100% deduction

Effect of f Long-range Co-planar Pairs 1-5m 1-5m 1-5m 1-5m 0% deduction 50% deduction 100% deduction

Conclusion 1. New task: co-planarity matching 2. Feature learning using self-supervision 3. Integration with robust optimization into SLAM Thank You! Yifei Shi Kai Xu Szymon Rusinkiewicz Tom Funkhouser