Regularity of Man-Made Environments Danping Zou VALSE SE online - PowerPoint PPT Presentation

StructVIO: Visual-Inertial Odometry with Structural Regularity of Man-Made Environments ▪ Danping Zou VALSE SE online ne semina nar 2019 2019 年 7 月 10 10 日

Visual SLAM ▪ Visual ual SLAM M techniques iques have e been widel ely appl plie ied d to unmanned ed vehicle cles. s. Fishey eye e camer era Stereo eo camera mera Stereo eo camera mera

Visual SLAM ▪ Augment ented ed reali lity ty (AR) (Holol ololen ens Glass ss ， Project Project Tango go Tabl blet et ） Holole lens uses s four Tango use one camer eras as for visual al SLAM fisheye eye camera mera for visua ual SLAM

Visual SLAM ▪ Operat ration ion syst stem m on cellph phones ones ▪ Google and Apple integrate visual SLAM into their OS (iOS, Android).

Visual SLAM ▪ A lot of algorithms have been proposed for visual SLAM in the past 15 years. ▪ MonoSLAM AM (2003), 3), Struc uctSL tSLAM(20 2014 14) ▪ PTAM( M(2007) 2007), , ORB-SL SLAM(20 M(2015) 5) ▪ SVO(201 2014) 4), , LSD-SL SLAM( M(20 2014 14), , DSO(2016) 2016) ▪ Pure visual SLAM system is not robust in practical applications. ▪ Visual-inertial systems become predominant for real applications. ▪ MSCKF F (2007) 7), , ROVIO (2009) 9) ▪ OKVIS (2015) 5), , VINS(2017) 2017), ICE-BA( BA(20 2018 18)

Features in v/vi-SLAM systems ▪ Most st visual ual-SLA LAM or visual ual-inertia inertial l syst stems ms choose ose points ts as the landma dmarks. rks.

Features in v/vi-SLAM systems ▪ Man made environments exhibit stron rong g regul ulari rity ty on geome metr try. Street eet Natural ural scenes nes Indoor r Under ergroun round d parki king ng

Structural regularity - Manhattan word 1. 1. Rich of line featu ture res 2. 2. Three known wn directions ctions (x, y, z)

Visual SLAM with Manhattan world model ▪ StructSLAM ctSLAM (Pr Present esented ed VALSE SE online e seminar, nar, 2016, 30 th th ,Mar) ,Mar) ▪ Point + structural lines (lines aligned with x, y, z directions) ▪ The direction of lines improves the observability of camera orientation Zhou, , Huizho izhong, , Danpin ing, , Zou, , et al. . "StructSL ctSLAM: : Visual al SLAM with th build ildin ing struct cture lines es." ." Vehicu icula lar Tech chnolo logy, y, IEEE Tran ansact actio ions on 64.4 (2015): ): 1364-13 1375. . - Specia ial l session for indoor loca calizat lizatio ion

Real word is full of diversity ▪ A lot of man made de environm onment ents can not be well describe cribed d by Manh nhatt ttan n worl rld d model. l. ▪ Obliqu ique/ e/cur curvy y structur ctures. es.

StructVIO ▪ A novel l visual ual-inertial inertial odom ometry etry method od is presen sented ted ▪ Use Atlanta nta world model to better describe irregular scenes. ▪ Made sever eral al improvements ovements to existing VIO approach. ▪ A VIO dataset that can be used evaluate different methods. Zou, Danping, et al. “ StructVIO: Visual-inertial Odometry with Structural Regularity of Man- made Environments .” IEEE Trans. on Robotics, 2019 Executable, tools & dataset : http://d /dro rone.sjt sjtu.edu edu.cn cn/d /dpzo zou/p /pro roje ject ct/s /stru ruct ctvi vio. o.html ml

Key idea – Atlanta world model ▪ We can approximate an irregular world by a group up of local cal Manh nhatt ttan an worl rlds ds. ▪ Each one of them can be represented by a heading direction ∶ 𝜚 . One Manhattan attan world Two Manhat attan tan worlds ds Three ee Manhatt hattan an worlds ds

Key idea – Atlanta world model ▪ Locally, the world is a Manhattan world. We can still use ▪ Three ee direc ectio ions ns ▪ Struc uctu tural al line features s ▪ to improve the performance of the VIO system. X,Y direc ection ions s – Render er the Yaw angle le observabl vable e (locally) ally) Three directio ions ns Z d direc ectio ion n – Render er the gravit vity y directi ction on observable vable A g good complementary plementary to point t features ures in textur ture- Line e features es less scenes. nes.

The framework of StructVIO ▪ We adop opt t the multi-state tate EKF KF filter er based sed fram amew ework. k. ▪ Compar mparin ing g with class ssic c EKF KF filter er Clas assic ic EKF filter ter ▪ Much faster since the features are not included in the state vector. ▪ Compar mparin ing g with key-frame frame optim imiz ization ation ▪ Short feature trajectories are fully explored. Key-fra frame me opti timi mizat ation ion ▪ State update using a single feature trajectory. ▪ Efficient but without losing much accuracy. Multi lti-sta tate te EKF KF filter lter

The framework of StructVIO ▪ The pipeline of StructVIO is as the following:

State definition of StructVIO ▪ The state vector consists of the current ent IMU U state te, historical orical IMU U poses, ses, calibra ibration tion para rameters eters , and the headin ding g direc ections tions of local Manhattan worlds Current ent IMU state ate Camera-IMU IMU cali libr brati ation on Manhatt hattan an world lds Historic ical al IMU poses

StructVIO – Technical details ▪ Inside ide of the filter er ▪ Other details ▪ Paramet ameter eriz izat ation ion ▪ Outlier rejection ▪ Measu surement nt equation ion ▪ Outsid ide of the filter ter ▪ Struc ructur tural al line e relat ated ed tasks: sks: ▪ Line e detec tectio tion n & tracking acking ▪ Classif sific icati ation n of struc uctural tural lines ▪ initia tializat lization ion & triangulat iangulation ion ▪ Hand ndling ling long ng feature ture tracks acks ▪ Manha hatta ttan world rld : ▪ Detect tection ion ▪ Merg rging ing

Coordinate frames ▪ World ld frame e : ▪ Z axis aligned with gravity ▪ Starting point as the origin ▪ Local cal Manhat hattan tan frame me: ▪ Camera era frame me : ▪ Z axis aligned with the optical axis toward the viewing direction. ▪ X, Y axes aligned with x,y axes of the image ▪ Startin rting g frame: me: - Movin ing Manha hattan an frame me ▪ The origin is located at the camera center. ▪ Three axes aligned with those of local Manhattan frame

Representation of a structural line ▪ We use a camera mera-cent centric ric representation. World d frame Starting ing frame Paramete ameter space ce Camera a frame ▪ Para ramete eter r spa pace ce : - use for line represent esentation ation

Structural line parameter space ▪ In parameter space {𝑀} ， a structural line can be represented by a point and a vertical direction. ▪ To achieve better linearization, the intersection point can be represented using inverse-depth approach. We have

Line space -> Starting frame ▪ The structural line can be transformed into three axes of the starting frame by the Line e space ce 𝑇 𝑆 . rotation 𝑀 Starting ing frame World d frame Camera a frame

Starting frame -> World frame ▪ The structural line can be further transformed into the world frame by using the heading Line e space ce direction ( 𝜚 𝑗 ) of the local Manhattan world. Starting ing frame World d ▪ The structural line is then transformed to the frame current camera frame by. Camera a frame

Line projection on the image ▪ Apply the transformations to both the point 𝑚 𝑞 and the vertical direction 𝑎 Starting ing World d frame frame Line e Camera a Paramete ameter space ce equati ation on frame

Line projection on the image ▪ Line projection can be written as the following functions (unknown camera-IMU 𝐽 𝜐 ) calibration 𝐷 𝑇 𝑆 are known constants after line direction classification. , where 𝑀 ▪ Hence we further write 𝑗𝑛 𝑚 = Π(𝑚, 𝜚 𝑗 , 𝐷 𝑋 𝜐) 𝐽 𝜐 , 𝐷 𝑗𝑛 𝑚 = Π(𝑚, 𝜚 𝑗 , 𝐷 𝑋 𝜐) ▪ We can use the above functions to derive the measurement equations.

Measurement equations ▪ Measur surem ement nt equation tion by re-pro project jection ion errors rs ▪ The line projection at time 𝑙 is given by: 𝐽 𝜐, 𝐷 𝑗𝑛 𝑚 𝑙 = Π(𝑚, 𝜚 𝑗 , 𝐷 𝑋 𝜐) ▪ The line segment detected on the image is denoted by : 𝑡 𝑏 ↔ 𝑡 𝑐 ▪ Hence the re-projection error can be computed as the signed distances between the line projection and the two end points:

Measurement equations ▪ After local linearization, we have ▪ By stacking all observations from time 1 to time 𝑁 Camera- Line Headi ding ng of Historic ical al IMU IMU paramete ameters Manhatt hattan an poses calibr ibratio ation world

Measurement equations ▪ Project the residual to the left null space of 𝐼 𝑚 , we can get rid of the line parameters ： ▪ The measurement equation involves ▪ 1. Heading ing directio ion n of the local al Manhatta attan n world ▪ 2. IMU-camer camera a relati ative ve pose ▪ 3. Histor oric ical al IMU poses

Outside of the filter ▪ Structu ctura ral line related ted tasks: sks: ▪ Line e detec ection ion & classi ssific icatio ation n of structural uctural lines ▪ initiali alizat ation ion & & triang angulati ulation ▪ Line track acking ing ▪ Handling ling long featur ure e trac acks