Nonparametric Directional Perception Julian Straub Collaborators: - PowerPoint PPT Presentation

Nonparametric Directional Perception Julian Straub Collaborators: Oren Freifeld, Jason Chang, Guy Rosman, Trevor Campbell, Randi Cabezas, Nishchal Bhandari, Jonathan P. How, John J. Leonard, John W. Fisher III. This talk is entirely based on my PhD thesis at MIT. October 5, 2017

Perception is Key

Perception Localization Mapping Scene Understanding

Perception Localization Simultaneous Localization and Mapping (SLAM) Mapping Scene Understanding

Perception Localization Simultaneous Localization and Mapping (SLAM) Semantic Mapping SLAM Scene Understanding

Perception Localization Simultaneous Localization and Mapping (SLAM) Semantic Mapping SLAM Scene Understanding Directional Scene Understanding

Outlinep Background 1.

Outlinep Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric

Outlinep Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric 3. Nonparametric Directional SLAM

Outlinep Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric Nonparametric Directional Perception capture and use regularities of man- made environments revealed in their surface normal distribution 3. Nonparametric Directional SLAM

Outlinep Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric Nonparametric Directional Perception capture and use regularities of man- made environments revealed in their surface normal distribution 3. Nonparametric Directional SLAM

RGB and Depth Image

RGB and Point Cloud and Depth Image Surface Normals

RGB and Point Cloud and Depth Image Surface Normals True Surface

RGB and Point Cloud and Depth Image Surface Normals Sensing True Surface Point Cloud

RGB and Point Cloud and Depth Image Surface Normals Sensing Normal Extraction True Surface Point Cloud Surface Normals

RGB and Point Cloud and Surface Normal Space: Sphere S 2 Depth Image Surface Normals Sensing Normal Extraction True Surface Point Cloud Surface Normals

RGB and Point Cloud and Surface Normal Space: Sphere S 2 Depth Image Surface Normals Sensing Normal Extraction True Surface Point Cloud Surface Normals

Scene Structure and Distribution of Normals Large Scale Small Scale

Scene Structure and Distribution of Normals Large Scale Small Scale

Scene Structure and Distribution of Normals Large Scale Small Scale Surface normal clusters capture environment regularities.

Directional Clustering and Segmentation Scene Surface Normals

Directional Clustering and Segmentation Scene Surface Normals Directional Clustering Bayesian directional mixture models for directional clustering.

Directional Clustering and Segmentation Scene Surface Normals Directional Directional Clustering Segmentation Bayesian directional mixture models for directional clustering.

Outline Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric Nonparametric Directional Perception capture and use regularities of man- made environments revealed in their surface normal distribution 3. Nonparametric Directional SLAM

Scene Representations Manhattan World (MW) [Coughlan 1999] Real World ≈ MW

Scene Representations Atlanta World (AW) Mixture of Manhattan Frames (MMF) Manhattan World (MW) [Schindler 2004] [Straub 2014] [Coughlan 1999] Real World ≈ MW AW MMF

Scene Representations Atlanta World (AW) Mixture of Manhattan Frames (MMF) Manhattan World (MW) [Schindler 2004] [Straub 2014] [Coughlan 1999] Real World ≈ MW AW MMF Manhattan Constrained Directional Models

Manhattan World R 3

Orth. Vanishing Points Manhattan World Projection R 2 R 3 [Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

Orth. Vanishing Points Manhattan World Projection R 2 R 3 [Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ] sparse line observations ⇒ fragile

Orth. Vanishing Points Manhattan World Manhattan Frame Surface Projection Normal Extraction R 2 R 3 S 2 [Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ] sparse line observations ⇒ fragile

Orth. Vanishing Points Manhattan World Manhattan Frame Surface Projection Normal Extraction R 2 R 3 S 2 [Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ] sparse line observations ⇒ fragile

Orth. Vanishing Points Manhattan World Manhattan Frame Surface Projection Normal Extraction R 2 R 3 S 2 [Caprile 1990, Coughlan 1999, Bose 2003, [ Straub 2014, Straub 2015 , Ghanem Lee 2009, Neverova 2013, Liu 2015, . . . ] 2015, Joo 2016, Straub 2017 ] sparse line observations dense surface normal observations ⇒ fragile ⇒ accurate, robust

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017]

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017]

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017] MF 1 MF 2 MF 3

Manhattan Frame: Mixture over Axes Distributions MF 1 MF 2 MF 3

Manhattan Frame: Mixture over Axes Distributions MF 1 MF 2 MF 3 MF Axes Assignments

Manhattan Frame: Mixture over Axes Distributions MF 1 MF 2 MF 3 MF Axes Assignments Sampling-based algorithm allows inference of number of MFs.

Scene Representations Atlanta World (AW) Mixture of Manhattan Frames (MMF) Manhattan World (MW) [Schindler 2004] [Straub 2014] [Coughlan 1999] Real World ≈ MW AW MMF SCW Stata Center World (SCW) [Straub 2015] Nonparametric Unconstrained Directional Model

Scene Representations Atlanta World (AW) Mixture of Manhattan Frames (MMF) Manhattan World (MW) [Schindler 2004] [Straub 2014] [Coughlan 1999] Real World ≈ MW AW MMF SCW Planes Stata Center World (SCW) Planes [Straub 2015] Nonparametric Unconstrained Directional Model

Stata Center World Stata Center World R 3

Stata Center World Vanishing Points Stata Center World Projection R 2 R 3

Stata Center World Vanishing Points Stata Center World Projection R 2 R 3 [Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ] sparse observations, no MW constraints ⇒ even more fragile

Stata Center World Dir. Clusters Vanishing Points Stata Center World Surface Projection Normal Extraction S 2 R 2 R 3 [Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ] sparse observations, no MW constraints ⇒ even more fragile

Stata Center World Dir. Clusters Vanishing Points Stata Center World Surface Projection Normal Extraction S 2 R 2 R 3 [Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ] sparse observations, no MW constraints ⇒ even more fragile

Stata Center World Dir. Clusters Vanishing Points Stata Center World Surface Projection Normal Extraction S 2 R 2 R 3 [Collins 1990, Antone 2000, Tardif 2009, nonparametric surface normal clustering Barinova 2010, Xu 2013, Lezama 2014, [Triebel 2005, Straub 2015, Straub Kroeger 2015, . . . ] 2015 , Zhou 2016] sparse observations, no MW dense observations constraints ⇒ even more fragile ⇒ accurate, robust

Outlinep Directional Scene Background 1. 2. Understanding Manhattan Manhattan Manhattan Nonparametric Nonparametric Nonparametric Directional Perception capture and use regularities of man- made environments revealed in their surface normal distribution 3. Nonparametric Directional SLAM

Nonparametric Directional SLAM Localization Nonparametric Mapping Directional SLAM Directional Scene Understanding

Related Geometry-based Semantic SLAM Systems Planes [Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Related Geometry-based Semantic SLAM Systems Planes Manhattan World (MW) [Castle 2007, Taguchi [Peasley 2012, 2013, Salas-Moreno Furukawa 2014, Kaess 2015, 2009, Le 2017] Ma 2016, Hsiao 2017]

Related Geometry-based Semantic SLAM Systems Planes Manhattan Vanishing World (MW) Points (VPs) [Castle 2007, Taguchi [Peasley 2012, [Bosse 2003] 2013, Salas-Moreno Furukawa 2014, Kaess 2015, 2009, Le 2017] Ma 2016, Hsiao 2017]

Related Geometry-based Semantic SLAM Systems Planes Manhattan Vanishing Stata Center World (MW) Points (VPs) World (SCW) [Castle 2007, Taguchi [Peasley 2012, [Bosse 2003] [ Straub 2017 ] 2013, Salas-Moreno Furukawa 2014, Kaess 2015, 2009, Le 2017] Ma 2016, Hsiao 2017]

Related Geometry-based Semantic SLAM Systems Planes Manhattan Vanishing Stata Center World (MW) Points (VPs) World (SCW) [Castle 2007, Taguchi [Peasley 2012, [Bosse 2003] [ Straub 2017 ] 2013, Salas-Moreno Furukawa 2014, Kaess 2015, 2009, Le 2017] Ma 2016, Hsiao 2017] Directional Scene Understanding