Following Dirt Roads at Night-Time Sensors and Features for Lane - PowerPoint PPT Presentation

Following Dirt Roads at Night-Time Sensors and Features for Lane Recognition and Tracking Sebastian F. X. Bayerl Thorsten Luettel Hans-Joachim Wuensche Autonomous Systems Technology (TAS) Department of Aerospace Engineering University of the Bundeswehr Munich 2015-09-28

Motivation  Recognition of ego lane is prerequisite for many ADAS • Camera based methods usually  Most methods valid for well marked roads at night Color Color gradient  Little work done for unmarked rural roads at night Color gradient Color 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 2

Hardware  Stock color camera • Color of surface Integration time: 30ms Integration time: 100ms 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 3

Hardware  Stock color camera  Color Night Vision (CNV) camera • Color of Surface Integration time: 50ms 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 4

Hardware  Stock color camera  Color Night Vision (CNV) camera  Near Infrared (NIR) camera • Reflectivity Paved road Forest road 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 5

Hardware  Stock color camera  Color Night Vision (CNV) camera  Near Infrared (NIR) camera  Far Infrared (FIR) camera • Temperature (12.5°C - 15.0°C) 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 6

Hardware  Velodyne LiDAR • 3D measurements • NIR reflectivity 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 7

Fusion and Accumulation  Fusion and accumulation into a Local Terrain Map  Multiple layers • Heights • Slopes • Obstacles • NIR Reflectivity • Color • Temperature • … update step 2. Update Velodyne 3. Update camera layers 1. Update robot position 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 8

Features  Color Features • Gradient at lane boundary • Saturation channel • Ratio of green color channel • … Color gradient Color with obstacles (red) Green color ratio g / ( r+g+b ) Color saturation 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 9

Features  Temperature • Transitions at lane boundary • Temperature back projection 15°C – 17°C Surface temperature Temperature gradient Temperature back projection 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 10

Features  3D / LiDAR • Obstacle probability cross section of a hill • Heights • Slopes cross section of a valley Obstacles (red) Heights 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 11

Features - Evaluation  10.000 positive and negative samples for each feature • Different road scenes: paved, unpaved, dirt, forest, … • Different seasons: summer, winter, … • Different weather conditions: sun, rain, snow • Different day times!  Receiver Operating Characteristic (ROC) True positive rate Color gradient Color gradient False positive rate 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 12

Features - Evaluation  Color Color green ratio Color gradient True positive rate True positive rate False positive rate False positive rate  Degradation of color from day to night! 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 13

Features - Evaluation  Temperature • E.g. temperature gradient  Temperature more informative without illumination! 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 14

Features - Evaluation  Thermal limitations ROC of temperature back projection True positive rate False positive rate road covered by leaves CNV Camera FIR Camera 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 15

Features - Evaluation  3D/LiDAR • e.g. obstacle probability PDF ROC  No dependency to illumination  No „stand alone“ feature 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 16

Features - Evaluation  Benefit of night sensors: • Classifier with full feature capability ( 𝐷 𝐵 ) • Classifier with reduced feature capability ( 𝑫 𝑪 ) True positive rate False positive rate 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 17

Tracking  Geometry • Clothoid(s) for modelling road net w 𝛀 d T 𝐲 cross = p 𝑦 p 𝑧 𝐲 brach1 𝐲 brach2 … 𝐲 lane = d Ψ c0 c1 w T 𝛺 c0 c1 w T 𝐲 branch = 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 18

Tracking  Particle Filter • Correction step: ▫ Project particles into Local Terrain Map ▫ Calculate mean feature values 𝐺 𝑔 for all particles (state vector 𝑦 𝑞 ) ▫ Naive Bayes Classification result as particle weight 𝑥 𝑞 𝐺 1 , … , 𝐺 𝑜 = 𝑞 𝑔 (𝐺 𝑔 |𝑦 𝑞 ) 𝑔=1,…,𝑜 ▫ State Vector and Covariance from weighted mean • Prediction : model road as static object moving with inverse robot motion 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 19

Movie 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 20

Thank you for your attention! Questions? 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 21

 Motivation  Sensors • Hardware • Fusion and Accumulation  Features • Road Features • Evaluation  Perception • Particle Filter • Limitations  Movie 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 22

Hardware  Robot: Mucar-3 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 23

Features  Color Features • Edges at lane boundary • Saturation channel • Ratio of green color channel • … Color edge intensity Color with obstacles (red) Green color ratio Color saturation 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 24

Limitation  Limitations 1. Sun scene (14.5 °C – 17.0 °C) 2. Rain scene (14.5 °C – 16.0 °C) 3. Forest scene (14.5 °C – 17.0 °C) 4. Forest scene (15.0 °C – 17.0 °C) 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 25

Limitation  Limitations 5. Foggy winter scene (10.0 °C – 10.5 °C) 6. Snow scene (3.0 °C – 4.0 °C) 7. Winter scene (8.0 °C – 10.0 °C) 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 26

Tracking  Thermal limitations ROC of thermal edge direction  Set of features provides robustness: • At least one significant feature necessary 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 27

Tracking  Geometry • Clothoid(s) for modelling road net w 𝛀 d T 𝐲 lane = d Ψ c0 c1 w T 𝐲 cross = p 𝑦 p 𝑧 𝐲 brach1 𝐲 brach2 … 𝛺 c0 c1 w T 𝐲 branch = • Using rough information of road map to switch between road and crossroad ▫ Distance to crossroad ▫ Direction of outgoing branch 2015-09-28 PPNIV 2015 - Sebastian Bayerl: Following Dirt Roads at Night Time 28