Teaching a Car to Drive: An application of End-to-End Deep Learning - PowerPoint PPT Presentation

Teaching a Car to Drive: An application of End-to-End Deep Learning Larry Jackel NVIDIA, Holmdel NJ 07733 ljackel@nvidia.com arXiv.org > cs > arXiv:1604.07316

Teaching a Car to Drive: An application of End-to-End Deep Learning Larry Jackel NVIDIA, Holmdel NJ 07733 ljackel@nvidia.com arXiv.org > cs > arXiv:1604.07316

Status • Mean Autonomous Distance > 50 km on highways • Good enough that passengers doze off • Need to get Mean Autonomous Distance > 1,000,000 km

DriveAV: NVIDIA Self-Driving Software Mapping Localization Route Planning Actuator Controllers . . . Lane Following Lane Following

Lane Following Path Steering LaneNet Camera Estimation Controller

Lane Following Path LaneNet Estimation Steering Camera Fusion Controller PilotNet Use diverse systems with different strengths and weakness to get required performance Best case: error rate = error rate(LaneNet) * error rate (PilotNet)

How we got to PilotNet: A long journey in Neural Nets

Bell Labs, Holmdel NJ 6000 employees in Holmdel ~300 in Research ~30 in Machine Learning

1986-Early Results on Recurrent Nets • Smallest Associative Memory • Extremely high density • Months to make • Computer Experiments revealed problems • Inefficient use of resources for pattern storage • Not as good at matched filters • No Learning 144 “synapses” in 6x6 micron cell Initial Results: Recurrent nets were disappointing

1986 – The Hype Begins The Neural Net Fad • Lots of coverage in the press • Minimal science • No practical results “ Bell Labs breakthrough: Chips that work like the brain”

1988 : First Applications • Graf builds 54 neural programmable chip at Holmdel • Good results on handwritten character recognition with hand-crafted Hubel-Wiesel like features (ConvNet without learning) Analog hardware Feature extraction • LeCun Joins Group • Builds first “ LeNet ” OCR engine with learned features • Early benchmarks indicate excellent performance

LeNet’s Feature extraction kernels are learned. Combines the architecture of Fukushima’s Neocognitron (1981) with Gradient Descent learning of weights

1989 -1990: OCR: excellent results But no systematic understanding of what governed the learning process

1989 -1990: OCR: excellent results But no systematic understanding of what governed the learning process Then came Vladimir Vapnik

How we came to view learning (Largely Vladimir Vapnik’s Influence) • Choosing the right structure • “Structural Risk Minimization” • Bring prior knowledge into the learning machine • Capacity control • Matching learning machine complexity to available data • Examine learning curves

With Vladimir Vapnik’s help, by the early 1990s we had a much better understanding of the learning process and its applications • AT&T OCR has become a product • Holmdel group ~ 30 people • Applications include pen- computing, finance, network fault prediction • SVM soft margin and kernel classifiers invented (This time not hype!!)

17

The 1995 Bets In 2000 there was a fancy dinner for four paid for by Jackel (We couldn’t prove why Yann’s multi -layer nets work so well) In 2005 there was a fancy dinner for four paid for by Vapnik ( Yann’s nets are still in use and keep getting better) Yann and Leon didn’t bet, but they got to eat too. 18

Autonomous Driving

The usual approach for self-driving cars Sensors Feature Object Extraction Recognition Now often combined using Convolutional Nets Cost Detailed Localization Map Maps Path Actuation Planning

Learning a complete control loop DARPA seedling project – Yann LeCun and Urs Muller “DAVE” Steering Camera Deep Convolutional Nets Actuation • Learn to steer by observing a human • Minimal use of hand-crafted rules • Minimal need for hand labeling

Early examples • ALVINN • Pomerlau – CMU ~1990 low-res image, fully-connected nets • DAVE • Muller, LeCun – 2003 • Higher res images • ConvNets Show Dave Video

Example: Road Following Good quality lane markers and good driving conditions Traditional lane detection-based systems work well Poor quality lane markers Lane detection-based systems struggle End-to-end learning empowers the network to use additional cues

Training the network Camera

NVIDIA PilotNet ~250,000 distinct weights ~27,000,000 connections

Neural Network Driving the Car in New Jersey

Visualization What the network pays attention to

ATYPICAL VEHICLE CLASS

Integrating PilotNet: The Rail Pilot was modified and trained to have 41 outputs 45 meters specifying the center of the path to be The rail Computed at 30Hz followed in 1 meter increments To drive, the 10 most recent rails (the last 1/3 sec) are averaged to get a target position at a set distance in front of the vehicle

Driving the rail from Holmdel to Woodbridge Note performance when lane markings are missing

Summary • Good progress in driving • A lot more work needed to get to required safety • Looks like diversity of parallel methods is the key ljackel@nvidia.com