Increasing the Convergence Domain of RGB-D Direct Registration - PowerPoint PPT Presentation

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Increasing the Convergence Domain of RGB-D Direct Registration Methods for Vision-based Localization in Large Scale Environments Renato Martins and Patrick Rives Inria Sophia Antipolis, France http://team.inria.fr/lagadic ITSC PPNIV’16 Workshop, Rio de Janeiro, Brazil 01 November 2016 Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Outline Introduction & Motivation 1 RGB-D Registration & Large Motion 2 Adaptive Formulation 3 Results 4 Conclusions & Perspectives 5 Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Outline Introduction & Motivation 1 RGB-D Registration & Large Motion 2 3 Adaptive Formulation Results 4 Conclusions & Perspectives 5 Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Introduction and Motivation Main objective : the design of a robust/efficient direct RGB-D registration technique for large motions. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Introduction and Motivation Main objective : the design of a robust/efficient direct RGB-D registration technique for large motions. Multiple applications: Visual odometry, mapping and SLAM; Navigation and visual servoing; Augmented reality. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Introduction and Motivation Main objective : the design of a robust/efficient direct RGB-D registration technique for large motions. Multiple applications: Visual odometry, mapping and SLAM; Navigation and visual servoing; Augmented reality. ? ? Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Outline Introduction & Motivation 1 RGB-D Registration & Large Motion 2 3 Adaptive Formulation Results 4 Conclusions & Perspectives 5 Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Direct RGB-D Registration Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Direct RGB-D Registration F ∗ F ∗ p p n ∗ n ∗ T ( x ) T ( x ) w ( p , T ) w ( p , T ) F F Minimisation of intensity and depth costs: p ρ ( e I ( p , T ( x ))) + λ 2 � C ( x ) = � p ρ ( e D ( p , T ( x ))) with e I ( p , T ( x )) = I ( w ( p , T ( x ))) − I ∗ ( p ) e D ( p , T ( x ))= ( Rn ∗ ( p )) T ( g ( w ( p , T ( x ))) − T ( x ) g ∗ ( p )) g ( • ) is a 3D point (the inverse sensor projection) and n the normal vector. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives RGB-D Registration & Large Motion Direct Registration: Strengths of direct methods: Accurate (precision similar to expensive IMU’s) [A. Howard, IROS 2008]; More robust to outliers than feature based registration (essential/fundamental matrix). Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives RGB-D Registration & Large Motion Direct Registration: Strengths of direct methods: Accurate (precision similar to expensive IMU’s) [A. Howard, IROS 2008]; More robust to outliers than feature based registration (essential/fundamental matrix). But... Direct methods are applied mostly for small displacements (small convergence domain); Real-time constraint: cannot process all frames (low frame rate or gaps); Navigation does not follow exactly the acquired 3D model (learned map). Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Direct Reg. Convergence Domain Strategies to increase the basin of convergence In the motion estimate (prediction): Hypothetical constraints of the movement (locally 2D, non-holonomic); Prediction (assumed motion model + online estimates); cost cost initial initial prediction prediction solution solution x x Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Direct Reg. Convergence Domain Strategies to increase the basin of convergence In the motion estimate (prediction): Hypothetical constraints of the movement (locally 2D, non-holonomic); Prediction (assumed motion model + online estimates); Acting in the sensor’s measurements: Multi-resolution (Gaussian pyramid); Correlation (often done simultaneously with assumptions in the motion); Extraction and matching of stable features (SIFT, SURF, ...). **Choice of scaling factor affects the shape of the cost . cost cost initial initial prediction prediction solution solution x x Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Outline Introduction & Motivation 1 RGB-D Registration & Large Motion 2 3 Adaptive Formulation Results 4 Conclusions & Perspectives 5 Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Intensity and Geometric Cost Shape/Convergence Considered data terms and modelling: Classic RGB-D formulation: C ( x ) = C I ( x ) + λ 2 C D ( x ) Scaling factor λ : Heuristically set; λ based on covariance of each point [C. Kerl & D. Cremers, ICRA 2013]; λ scaling pixels to meters [T.Tykkala & A.Comport, ICCV 2011]. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Intensity and Geometric Cost Shape/Convergence Considered data terms and modelling: Classic RGB-D formulation: C ( x ) = C I ( x ) + λ 2 C D ( x ) Scaling factor λ : Heuristically set; λ based on covariance of each point [C. Kerl & D. Cremers, ICRA 2013]; λ scaling pixels to meters [T.Tykkala & A.Comport, ICCV 2011]. In fixed pyramidal resolution: RGB and geometric costs have different convergence properties [M. Levoy et al., 3DIM 2003] [S. Bonnabel et al., ACC 2016]; RGB term dominates when combined as in [T. Tykkala & A. Comport, ICCV 2011] and [C. Kerl & D. Cremers, ICRA 2013]. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Intensity and Geometric Cost Shape/Convergence Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives Intensity and Geometric Cost Shape/Convergence Wrong idea: set high λ during all the optimization 1) Geom. cost is flatter than RGB in the neighbourhood of the solution; 2) More sensible/unstable than RGB registration (visibility constraint); 3) Do not guarantee sub-pixel precision from intensity only cost term. Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods