Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1 - PowerPoint PPT Presentation

S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1

Ford ’ s 12 Year History in Autonomous Driving Today: examples from • Stereo image processing • Object detection • Using RNN’s • Motorsports 2

Stereo Matching Problem • Determining the correspondences in stereo images • Calculating the disparities • But what is the correct correspondence? • Basic stereo matching algorithm − Compare pixels on the same epipolar line in two images − Choose the best match 3

Deep neural networks for stereo matching • The brain can estimate the distance of an object using the visual information from two eyes. • We can use deep neural networks Left Stereo Camera Deep Convolutional Distance Map Post-Processing Neural Networks Estimation Right Stereo Camera 4

Proposed deep convolutional neural network • AV driving requires an intelligent distance map estimation, which filters out the objects not of interest. • Network I − General network − Encoding and decoding layers − Retain objects of interest in the training data sets Encoder Decoder Deconv10 Deconv9 Deconv7 Deconv8 Conv10 Deconv6 Conv9 Conv1 Conv2 Conv3 Conv4 Conv7 Conv8 Conv6 Loss Function Conv5 5

Proposed deep convolutional neural network II − Specialized network − Encoding and decoding layers − The cross correlation layers force the network to look for correspondence on the epipolar line − The weights in the encoding layers are shared Encoder Decoder Conv1L Conv2L Conv3L Conv4L Deconv8 Deconv9 Function Deconv7 Deconv6 Conv6 Conv7 Conv8 Conv9 Conv5 Loss CC5 CC6 CC7 Conv1R Conv2R Conv3R Conv4R 6

Proposed deep convolutional neural network • Cross correlation ( CC ) layer − Computes CC values between each pairs of patches − Outputs the CC values for each pair of patches − Does not lose any information • Loss function − In AV driving, closer objects are more important than distant ones 1 − Assigns more weight to the closer objects 0.8 − The closer object distance is estimated more α accurately 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 d 7

Performance on synthetic and real stereo data • Synthetic data generation − Generate 14,000 pairs of RGB stereo images − Synthetic distance maps are only generated for the objects of interest, e.g. cars or pedestrians − Gaussian noise added to the stereo images 8

Performance on synthetic and real stereo data • Fine tuning with LIDAR data sets − Project LIDAR point clouds onto the camera images − The baseline and optic axes are not the same as the synthetic data Left camera Right camera Network I Network II 9

1/2x 10

Comparing Manual Annotation to DNN Model 11

Detection Result Original Image Enhanced Contrast Network’s detection outperforms human labeler in low-contrast areas Pedestrian detection Pedestrian misdetection Detected, but not labeled 12

Introducing Recurrence in Detection and Tracking • Use RNN’s to detect occluded objects • Remember location of static objects • Predict location of non-static objects Detector Detector Detector RNN Conv RNN Conv RNN Conv Feature Feature Feature Map Map Map Image 0 Image 2 Image 1 13

Orange = ground truth; Green = model prediction 14

Classifying NASCAR images The Ford team reviews pictures during the race 15

Classifying NASCAR images Gap Looking for damage and other performance indicators 16

Results – Boxing the Cars Using ~2k images labeled with boxes around the vehicles, the model does well detecting cars 17

Results – Boxing the Cars 18

Classifying NASCAR images Next – determine car number: labeled ~30k images

Classifying NASCAR images Outliers easy to find in review

Classifying NASCAR images Human: ??? Model: 78 Confidence: 0.999

Classifying NASCAR images Human: ??? Model: 42 Confidence: 0.985

Inspecting the Neural Network Activated Filter Input Image The Model is not a black box. We can see that it is detecting the numbers – important for robustness when the paint changes 23

Argo AI • Argo AI is an artificial intelligence company, established to tackle one of the most challenging applications in computer science, robotics and artificial intelligence: self-driving vehicles • Engineering hubs in Pittsburgh, Southeastern Michigan and the Bay Area of California • For more information regarding Argo AI and its work, please talk to me at GTC or visit: www.argo.ai 24