See Through Smoke : Robust Indoor Mapping with Low-cost mmWave Radar - PowerPoint PPT Presentation

Cyber Pysical Systems Group See Through Smoke : Robust Indoor Mapping with Low-cost mmWave Radar (Chris) Xiaoxuan Lu * , Stefano Rosa * , Peijun Zhao * , Bing Wang * , Changhao Chen * , John.A.Stankovic + , Niki Trigoni * , Andrew Markham * * University of Oxford, UK + University of Virginia, USA ACM MobiSys 2020 Toronto, Canada 1 1

Motivation ACM MobiSys 2020 Toronto, Canada 2

Spatial Awareness ✧ Why - lack of spatial awareness • Spatial awareness : An as-comprehensive-as-possible map • How about : employ a mobile robot to fast map the env. first? ‘ Blind’ : Is it an exit?? Use HANDs to perceive obstacles Mobile robot to survey first! ACM MobiSys 2020 Toronto, Canada 3

Limitation of traditional sensors Smoke ✧ Optical Sensors cannot see through airborne particles ‣ RGB camera ‣ Depth Imaging ‣ Lidar Dust Fog ACM MobiSys 2020 Toronto, Canada 4

Single-Chip CMOS mmWave radar Automobile Manufactory ACM MobiSys 2020 Toronto, Canada 5

Primer ✧ Working Principles - A transceiver device operating in the spectrum between 30 mmWave Radar GHz - 300 GHz Object - Use a frequency modulated continuous wave (FMCW) approach to measure object distance and orientation Object mmWave Radar ACM MobiSys 2020 Toronto, Canada 6

Primer ✧ Pros - Sub-mm range accuracy - Impervious to environmental conditions, e.g., fog, smoke dust … - Small footprint - Cheap ACM MobiSys 2020 Toronto, Canada 7

milliMap Use a mobile-mounted single-chip mmWave radar for metric and semantic indoor mapping ACM MobiSys 2020 Toronto, Canada 8

Challenge I ✧ Very sparse point cloud ‣ Fundamental specularity of mmWave signals ‣ 4 x 3 antennas for cost reason ‣ CFAR (Constant False Alarm Rate) on-chip pre-processing < 100 points per scan, 100-fold sparser than a lidar ACM MobiSys 2020 Toronto, Canada 9

Challenge II Multi-path noise • Reflected signals arriving at a receiver antenna can from two or more paths • Leading to `ghost points' in a mmWave point cloud Points outside the black line (i.e. walls) are ghost points ACM MobiSys 2020 Toronto, Canada 10

Metric Map Reconstruction Formulate metric mapping to a reconstruction problem • Basic Model : conditional Generative Adversarial Network (cGAN) • Cross-modal Supervision : a co-located lidar providing labels for mmWave Lidar works fine in benigh (non-smoke) situations! ACM MobiSys 2020 Toronto, Canada 11

Metric Map Reconstruction Formulate metric mapping to a reconstruction problem • Basic Model : conditional Generative Adversarial Network (cGAN) • Online inference: independently predict a good map without the help of lidar Independently work for online inference! ACM MobiSys 2020 Toronto, Canada 12

Metric Map Reconstruction ✧ Network Input ➡ Single-frame SCAN? ➡ Too much information absence ➡ Probably Overfitting if learn in Lidar Scan mmWave Scan brute force Single mmWave scan misses lots of information! Almost learn sth. from nothing… ACM MobiSys 2020 Toronto, Canada 13

Metric Map Reconstruction mmWave Patch ✧ Network Input Odom ✓ Stitch scans into a patch assisted by odometry ✓ Use patches as inputs Series of Scans ✓ KEY : Odometry drifts in short-term is negligible lidar patch ACM MobiSys 2020 Toronto, Canada 14

Metric Map Reconstruction ✧ Network Input ✓ Stitch scans into a patch assisted by odometry ✓ Use patches as inputs ✓ KEY : Odometry drifts in short-term is negligible What loss? ACM MobiSys 2020 Toronto, Canada 15

Metric Map Reconstruction prior likelihood ✧ Bayes Perspective ✓ Goal: maximize posterior ✓ Likelihood (e.g., Pix2PixHD) - cGAN loss (appearance) - Feature Matching - Low-level geometry ✓ Prior ➡ Map of structure (lines etc.) ACM MobiSys 2020 Toronto, Canada 16

Metric Map Reconstruction ✧ Map Prior (MP) Loss ✓ ‘Manhattan World’ Model ✓ Geometric regularities in indoor environment, e.g., following rectilinear outlines Indoor Floor Plan ACM MobiSys 2020 Toronto, Canada 17

Metric Map Reconstruction Four line detection kernels which respond maximally to horizontal, vertical and oblique ✧ Map Prior (MP) Loss ✓ ‘Manhattan World’ Constraints ✓ Geometric regularities in indoor environment, e.g., following rectilinear outlines ✓ Realised through shape detector conv. masks (e.g., line detector) ACM MobiSys 2020 Toronto, Canada 18

milliMap Use a mobile-mounted single-chip mmWave radar for metric and semantic indoor mapping ACM MobiSys 2020 Toronto, Canada 19

4 Key Access Objects (AO) Horizontal AO - Door Vertical AO - Lift Alternative AO - Window Non AO - Wall ACM MobiSys 2020 Toronto, Canada 20

Challenge III Complex interior construction objects • Indoor construction objects are made by di ff erent layers of materials ➡ Multiple reflections from internal layers, di ff usion of mmWave on rough surfaces Hard to Model! Complicated multi-path e ff ects Interior Wall made by multiple layers ACM MobiSys 2020 Toronto, Canada 21

Semantic Recogniser Key Observation • Range FFT profile can capture the object-related mmWave propagation patterns • A Segment of Interest (SOI) is decided by profile peak point and its neighbours Robot rotates get a perpendicular obs. angle for mmWave radar ACM MobiSys 2020 Toronto, Canada 22

Semantic Recogniser In a example of 5-point SOI feature • ‘Average’ SOI of three key objects aggregated from 27, 952 training samples • Distinct shape patterns observed for di ff erent objects SOI - Door SOI - Lift SOI - Glass ACM MobiSys 2020 Toronto, Canada 23

Semantic Recogniser Semantic Mapping • Employ a NN classifier to output softmax probability logit • Use the probability distribution to determine semantic label and alien obj. ACM MobiSys 2020 Toronto, Canada 24

Implementation ✧ Multi-modal Robotic Sensing Platform Lidar • OS : ROS Melodic • Robot Platform : Turtle Bot 2 mmWave XSENS IMU • mmWave radar : TI AWR1443 radar • Lidar : Velodyne VLP-16 • Odometry provision : wheel Turtlebot 2 odom + XSENS IMU ACM MobiSys 2020 Toronto, Canada 25

Evaluation Experiment Sites • Training: 1st, 2nd and 3rd floor of Building A 1st floor, Building A 2nd floor, Building A 3rd floor, Building A ACM MobiSys 2020 Toronto, Canada 26

Evaluation Experiment Sites • Cross-site Test: 4th floor of Building A, Building B and smoke-filled arcade 4th floor, Building A Building B Arcade (smoke) ACM MobiSys 2020 Toronto, Canada 27

Evaluation X - prediction Y - truth ✧ Metrics p - pixel index ✓ Mean absolute error (L 1 ) ✓ Mean Intersection of Union IoU = | X ∩ Y | (IoU) | X ∪ Y | ✓ NOTE : sometimes, manual qualitative inspection is also L 1 = 1 N ∑ needed in our context | X ( p ) − Y ( p ) | p ∈ P ACM MobiSys 2020 Toronto, Canada 28

Evaluation w.o. stitch stitch 4 3 Pix2Pix 2 ✧ Order of Densification 1 ➡ Fix the model by using two 0 Cross-floor Cross-Build. established baseline generators ✓ Stitching-first consistently w.o. stitch stitch 4 yields smaller mean L 1 3 2 Pix2PixHD 1 0 Cross-floor Cross-Build. ACM MobiSys 2020 Toronto, Canada 29

Evaluation w.o. stitch stitch 0.4 0.3 Pix2Pix 0.2 ✧ Order of Densification 0.1 ➡ Fix the model by using two 0 Cross-floor Cross-Build. established baseline generators ✓ Stitching-first consistently w.o. stitch stitch 0.4 yields larger mean IoU 0.3 0.2 Pix2PixHD 0.1 0 Cross-floor Cross-Build. ACM MobiSys 2020 Toronto, Canada 30

Evaluation Method comparison • Outperform 5 grid map reconstruction methods in both L 1 and IoU Linefitting CVAE BiCycGAN Linefitting CVAE BiCycGAN 6 Pix2Pix Pix2PixHD Ours Pix2Pix Pix2PixHD Ours 0.5 4 0.3 IoU 2 0.2 0 0.0 Cross-floor Cross-Build. Cross-floor Cross-Build. Mean L 1 Mean IoU ACM MobiSys 2020 Toronto, Canada 31

Evaluation Method comparison • ‘Ghost’ area mis-generated. ACM MobiSys 2020 Toronto, Canada 32

Evaluation Where is `ghost' area coming from? • Incorrect lidar supervision due to presence of glass objects in training data ACM MobiSys 2020 Toronto, Canada 33

Evaluation E ff ectiveness for downstream navigation • Cross-building: 0.285 m trans. error and 0.142 rad orientation error • Cross-building: 0.178 m trans. error and 0.140 rad orientation error Translation Orientation ACM MobiSys 2020 Toronto, Canada 34

Evaluation Generalise to handheld case • Imperfect yet odom i can largely recover the basic shape raw mmWave Generated ACM MobiSys 2020 Toronto, Canada 35

Evaluation Semantic Recognition • Over 0.9 F 1 score for cross-floor, ~ 0.88 F 1 score for cross-building • ‘Alien’ objects outside target classes Cross-floor Cross-build. ACM MobiSys 2020 Toronto, Canada 36

Evaluation Semantic Recognition • Impact of SOI length: 6 points (~20cm) yields best performance Cross-build. Cross-floor ACM MobiSys 2020 Toronto, Canada 37

Demo ACM MobiSys 2020 Toronto, Canada 38

Smoke-filled Test ACM MobiSys 2020 Toronto, Canada 39

Conclusion ✧ Limitation & Next • More diverse and di ff erent places for testing • Odometry drifts in long run -> mmWave Odom (our milliEgo on arxiv already) • Aerial drones • Real disaster situation ACM MobiSys 2020 Toronto, Canada 40