Connected Automated Driving Overview, design, and technical - PowerPoint PPT Presentation

PROTECTED Connected Automated Driving Overview, design, and technical challenges Gaurav Bansal Toyota InfoTechnology Center, USA March 28, 2018

Automated Driving PROTECTED HIGHWAY AUTOMATED DRIVING URBAN AUTOMATED DRIVING 1 Sensors are at the heart of Automated Driving Technology

Challenges in Automated Driving Design PROTECTED ACCURACY/ URB RBAN/ NLOS WEATHER COST MAI AIN CHALLENGES RANGE RAN SC SCEN ENARI RIO Mig ight no not exis xist for or all all US S roa oadways. MAPP APPING ∆ O ∆ O Errors Er s will ill exi xist. GPS S (inc (including futu future GPS3) S3) has has ch challenges s in in lan lane le level localization for loc or urb urban can anyon. X O O X LOCALIZATION LiDa LiDar base based loc localization can an be be cos ostly and and do does s no not work ork wel ell in in bad bad wea eather. O X ∆ O Lim Limited range (ar (around 50 50-80 80 m). ). Majo ajor ch chal allenges s in in NLOS and bad and bad wea eather PER ERCEP EPTION ∆ X O X Challenges in Ch in Mer ergi ging, g, NLOS S Urban dr driving, Pla latooning PATH PLA LANNING ∆ X O ∆ ∆ - room for LEGEND O – good performance X - challenges improvement 2

What if we add Connected Intelligence to Automated Vehicles? PROTECTED ACCURACY/ URB RBAN/ NLOS WEATHER COST SO SOLVING CURRE RENT CHALLENGES RANGE RAN SC SCEN ENARI RIO Da Data fr from lea lead vehicles s can an be be use used for or real eal-time map ap MAPP APPING updates up s and and correction of of err errors. s. Hig ighly im improved loc localization by le leveragi ging GPS, S, LiD LiDar of of the ne neighboring con onnected vehicles. s. LOCALIZATION O - good Non on-line of of sigh sight vie iew, with ith robustness s for or an any wea eather performance sce cenario. . Ra Rang nges up up to o 300-500 meters. PER ERCEP EPTION Ultr ltra-low la latency Com ommunication with ith vehicles s whil hile mer erging, pla platooning, g, in intersection cr crossing PATH PLA LANNING Connected Intelligence will help provide long range sensing, NLOS operations in a cost-effective manner with robustness for any weather condition 3

What has been done so far on Connected Vehicles? PROTECTED Basic Safety Message being transmitted on Safety Channel using DSRC DSRC: Dedicated Short- Item Range Communication Time 3D Position Basic Safety Message Position Accuracy Ad hoc networking technology that allows Speed Heading vehicles to communicate with each other, Steering Wheel Angle roadside devices, pedestrians, bicycles, Acceleration Brake Status trains, … Vehicle Size Event Flags Path History Path Prediction CURRENT STATUS: Other optional fields - Moving toward deployment - Many stakeholders in US, China, Japan, Europe and elsewhere 4

System design PROTECTED Cooperative Automated Driving with highly improved mapping & localization, perception, and path planning Local Sensors Local Sensors + V2X data Cooperative Perception Perception Cooperative Cooperative Path Mapping Controller Controller Path Mapping Planner Planner Cooperative Localization Localization 5

Cooperative Mapping Overview PROTECTED Main Idea behind Measuring error in estimating road geometry using Camera, V2X and GPS Cooperative Mapping: Building real-time Automated Driving Data collected in Ann Arbor maps using the long-range of V2X Lane-level accuracy in road estimation at 200m Significant improvement in road estimation as compared to using only local sensor data (Radar + Camera) 6

Cooperative Mapping: Remote Vehicle Lane Change Detection PROTECTED Detecting state of the Remote Vehicle (keeping its lane or changing lanes) is crucial for Cooperative Mapping When the vehicle is not changing lanes, its trajectory is a good indicator of lane geometry. In this scenario, V2V data from the remote vehicle can be used to augment accuracy of lane geometry estimation. High detection accuracy achieved using Machine Learning Classifiers 7

Cooperative Localization Overview PROTECTED Main Idea behind Cooperative Localization: Improving positioning by leveraging GPS receivers of neighboring vehicles Flow of data between host Sensor Fusion block Measuring error in Vehicle vehicle and neighbors Positioning 43.8% Error Reduction 8

Cooperative Perception Overview PROTECTED Main Idea behind Cooperative Perception: Enhancing sensing range by sharing data (incl. camera, LIDAR, radar, etc.) with neighboring vehicles Car and Bus communicate to let the car know about the pedestrian 3 Pedestrian will be on the same section of the road as the car in the next few 1 seconds 2 Car will not see the the pedestrian for few more seconds 9

Cooperative Perception: Dynamic Object Detection PROTECTED PROBLEM: Neural networks are unable to recognize all objects in certain conditions APPROACH : Combine multiple perspectives from neighboring cars of the same scenario Car / biker not detected by Neural Network Designed novel 3D Vision Fusion Algorithm to allow bounding boxes to be updated with greater precision and occluded/distant objects to be shared between vehicles 10

Skateboarder PROTECTED Corolla Camera Prius Camera neural network does not detect skateboarder neural network detects skateboarder Other Skateboarder Skateboarder pedestrians detected not detected detected 11

Skateboarder PROTECTED Corolla Detection Prius Detection Other Other Skateboarder Skateboarder pedestrians pedestrians CONFIDENCE LEVEL IN DETECTION No detection by Prius Detected by Corolla High confidence in FUSED VIEW 12

Cooperative Path Planning Overview PROTECTED Main Idea behind Cooperative Path Planning: Highway merging using V2X handshakes Exchange V2V messages before merge to determine the Reducing the wait time on-ramp before merge merge order between on-ramp and highway vehicles 25% Improvement in wait-time 13

Cooperative End-to-end Driving Overview PROTECTED Main Idea behind Cooperative What is End-to-end Driving? Deep Learning to predict steering angles, required force End-to-end Driving: on brake and gas pedals using sensor data Achieve near 100% accuracy your vehicle’s end -to-end driving by utilizing data from neighboring connected vehicles V2X Camera data, V2X data Steering angle 14 Convolutional Neural Network

Cooperative End-to-end Driving: Video-based Network PROTECTED Baseline scenario: Ego vehicle predicts steering angle using forward facing camera data Ego 1 Input to neural network: Video frames Steering Angle output of video neural network 10 layer Video Neural Network vid 15

Cooperative End-to-end Driving: V2V-based neural network - An alternative? PROTECTED This is what the leading This is what the Ego vehicle Ego vehicle broadcasts: measures independently: Latitude: 𝑀𝑏𝑢 1 Latitude: 𝑀𝑏𝑢 0 Longitude: 𝑀𝑝𝑜 1 Longitude: 𝑀𝑝𝑜 0 Heading: 𝜚 1 Heading: 𝜚 0 V2V Neural Network predicts the steering angle for Ego vehicle 2 Steering Angle output of V2V neural network 2 layer Latitude difference x = V2V Neural Longitude difference Network Sine of heading difference V2V 𝑀𝑏𝑢 1 − 𝑀𝑏𝑢 0 𝑀𝑝𝑜 1 − 𝑀𝑝𝑜 0 sin 𝜚 1 − sin⁡ (𝜚 0 )

Cooperative End-to-end Driving: Combining video-based and V2V-based networks PROTECTED Video-based Weighing network prediction Final prediction vid + 𝐷 1 𝐷 2 = + vid V2V F F V2V Weights V2V-based prediction

Cooperative End-to-end Driving: Steering Angle Prediction PROTECTED Good steering angle prediction using a neural network trained on integrated video and V2V data Measured vs predicted steering angle Steering angle [rad] Trip Time [s] 18

New Communication Requirements.. PROTECTED Total data rate per link: 50 Mbps. For multiple links: Application Data Rate Details Requirement #1: Total system data rate target 10 Gbps Cooperative Processed – detected 0.5 Mbps 60 byte/object, 100 Perception objects objects, 10 Hz (Link data rate @50 Mbps x Target to support 200 links) Cooperative H.265/HEVC HD Camera 10 Mbps 640x360 resolution, Perception - Raw compressed Cooperative LIDAR - Raw 35 Mbps 6 verticle angels, 64 Requirement #2: Perception elements, 10 Hz horizontal rotation Latency target1 = 20 msec Cooperative Path Coarse driving intention 0.05 Mbps Few 100 bytes (e.g. Ultra-low latency to support applications such as platooning Planning 500 bytes) / msg, 10 Hz Latency target2 = 100 msec Cooperative Path Planned Trajectory 2.5 Mbps 32 byte / For applications such as long range sensing, driving intention Planning coordination, 10 msec resolution, 10 latency requirements would be relaxed (100 msec target) sec trajectory, 10 Hz 19

mmWave Communications PROTECTED Automotive radar 22-29 GHz DSRC (75 MHz) 3 GHz 30 GHz Log-scale 60 GHz WiFi Automotive radar 57-64 GHz 71-86 GHz 30 GHz 300 GHz 38-49 GHz Huge spectrum available at mmWave bands Advancement in CMOS technology enables low cost mmWave devices** 60 GHz WiFi (WirelessHD, IEEE 802.11ad) is already on the market 20 * United States radio spectrum frequency allocation chart as of 2011 ** T. S. Rappaport, J. N. Murdock, and F. Gutierrez, “State of the art in 60 -GHz integrated circuits and systems for wireless co mmunications,” Proceedings of the IEEE, vol. 99, no. 8, pp. 1390– 1436, Aug 2011