New Perspective on Perception and Prediction Pipeline for Autonomous - PowerPoint PPT Presentation

New Perspective on Perception and Prediction Pipeline for Autonomous Driving Xinshuo Weng, Kris Kitani Robotics Institute, Carnegie Mellon University August 28, 2020 1

Perception and prediction are important components in the autonomous driving stack 3

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection Perception 3D Multi-Object Tracking Trajectory Prediction Forecasting 4

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection LiDAR RGB 3D Multi-Object Tracking Trajectory Forecasting 5

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection Detection results 3D Multi-Object Tracking Trajectory Forecasting 6

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection 3D Multi-Object Tracking Tracking results Trajectory Forecasting 7

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection 3D Multi-Object Tracking Trajectory Forecasting Forecasting results 8

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection 3D Multi-Object Tracking Is this really the best Trajectory place to perform Forecasting prediction? 9

Standard Perception and Prediction Pipeline Sensor Data 3D Object Detection Can we do prediction here? 3D Multi-Object Tracking Trajectory Forecasting 10

Standard Perception and Prediction Pipeline Sensor Data Can we do prediction here? 3D Object Detection 3D Multi-Object Tracking Trajectory Forecasting 11

What is the state of the art for trajectory forecasting? 1. Datasets: Bigger and multi-modal 12

State of the Art: Datasets Sensor Data Better models from bigger datasets! * (Waymo) 3D Object Detection 150x increase! 3D Multi-Object Tracking Trajectory * Mined trajectory data not counted for the Argo dataset Forecasting Sun et al. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. CVPR 2020 13

State of the Art: Datasets Dataset with multi-modal ground truth Sensor Data 3D Object Detection Green : multi-modal ground truth future Yellow: past observations 3D Multi-Object Tracking Each modality of the future is generated by setting a different goal in the simulator In contrast to prior dataset with single future ground truth and make multi-future evaluation possible Trajectory Forecasting What are the right metrics for evaluation? J. Liang, L. Jiang, K. Murphy, T. Yu, A. Hauptmann. 14 The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction. CVPR 2020

What is the state of the art for trajectory forecasting? 2. Model: more side information 15

State of the Art: Trajectory Forecasting Models Sensor Data Multi-agent interaction modeling with Graph Neural Networks (GNNs) 3D Object Detection 3D Multi-Object Tracking Trajectory Contextual features are encoded to take into account of nearby agents’ motion during prediction Forecasting J. Sun, Q. Jiang and C. Lu. Recursive Social Behavior Graph for Trajectory Prediction. CVPR 2020 16

State of the Art: Trajectory Forecasting Models Sensor Data Road context / physical constraint helps 3D Object Detection 3D Multi-Object Tracking Using road structure semantics as inputs eliminates physically impossible trajectories Trajectory Forecasting T. Phan-Minh, E. Grigore, F. Boulton. O. Beijbom, E. Wolff. 17 CoverNet: Multimodal Behavior Prediction using Trajectory Sets. CVPR 2020

State of the Art: Trajectory Forecasting Models Goal-conditioned forecasting Sensor Data 3D Object Detection 3D Multi-Object Tracking Different goals could lead to different forecasts Trajectory Forecasting N. Rhinehart, R. Mcallister, K. Kitani and S. Levine. 18 PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings. ICCV 2019

State of the Art: Trajectory Forecasting Models End-to-end perception and prediction pipeline Sensor Data 3D Object Detection Gradients Separately optimized 3D Multi-Object 1. Suboptimal performance Tracking 2. Slow inference speed Gradients All modules are optimized for the end goal: trajectory prediction Trajectory Forecasting Jointly optimized M. Liang, B. Yang, W. Zeng, Y. Chen, R. Hu, S. Casas, R. Urtasun. 19 PnPNet: End-to-End Perception and Prediction with Tracking in the Loop. CVPR 2020

State of the Art Sensor Data Lots of progress on (1) building better/larger 3D Object datasets and (2) improving forecasting models Detection The pipeline stays the same! 3D Multi-Object Tracking Any possible improvement at the pipeline level? Trajectory Forecasting 20

Our recent work on new perception and prediction pipeline 1. Parallelized tracking and forecasting 2. SPF2: Sequential Pose forecasting by Sequential Pointcloud Forecasting 21

Limitation of the Standard Pipeline • Pipeline in a sequential order Sensor Data • Downstream module takes the outputs of its upstream module as inputs 3D Object • Any limitation? Detection • Errors from the upstream module cannot be corrected and will degrade performance of the downstream module • Can we go beyond the sequential pipeline? 3D Multi-Object Tracking Data association Trajectory error in tracking Forecasting GT past trajectories Tracking results Predicted trajectories Predicted trajectories 22

Parallelized Tracking and Forecasting Sensor Data Sensor Data 3D Object 3D Object Detection Detection 3D Multi-Object Tracking Shared Feature Feature Extraction Learning Similar components, which aims Matching to encode object features from past information Trajectory Forecasting Trajectory 3D Multi-Object Module-specific components Feature Extraction Forecasting Tracking Matching Trajectory Decoder Trajectory Decoder Sequential Pipeline Parallelized Tracking and Forecasting Pipeline X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 23

Parallelized Tracking and Forecasting Sensor Data 3D Object Detection • Advantages Shared Feature • Forecasting does not explicitly depend on the tracking results but implicitly use Learning the association information in the current frame • Improve computational efficiency by sharing the feature learning process 3D Multi-Object Trajectory Forecasting Tracking • Overview 3D MOT Shared Feature Learning Edge features 3D MOT Feature head extraction Last frame Current frame Objects trajectories in Forecasting GNN for feature past H frames interaction Feature Trajectory Diversity extraction Detected objects in forecasting Node features sampling current frame head Predicted trajectories Joint 3D Tracking and Forecasting in future T frames X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 24

Parallelized Tracking and Forecasting Sensor Data 3D Object Detection • Shared feature learning Shared Feature • Use LSTM/MLP to learn motion features from objects’ box trajectories Learning • Encode contextual / relative features from nearby objects by modeling interaction with GNNs 3D Multi-Object Trajectory Forecasting Tracking X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 25

Parallelized Tracking and Forecasting Sensor Data 3D Object Detection • 3D multi-object tracking Shared Feature • MLP takes edge features as inputs to regress the similarity scores Learning between every pair of objects • During training, estimated affinity matrix is supervised with GT 3D Multi-Object Trajectory Forecasting Tracking • During testing, estimated affinity matrix is fed to Hungarian algorithm X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 26

Parallelized Tracking and Forecasting Sensor Data 3D Object Detection • Trajectory forecasting Shared Feature • A diversity sampling function that maps each object feature to a set of Learning latent code covering various modes • A conditional VAE is used to predict future trajectories from diverse 3D Multi-Object Trajectory Forecasting Tracking latent codes X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 27

Quantitative Results 28

Parallelized Tracking and Forecasting Sensor Data 3D Object Detection • Is the parallel pipeline effective? Can two modules benefit one another? Shared Feature • How does adding 3D MOT affect performance of forecasting? Learning • Add 3D MOT branch improves performance on forecasting 3D Multi-Object Trajectory Forecasting Tracking Performance improved after adding MOT! Forecasting evaluation without 3D MOT X. Weng, Y. Ye, K. Kitani. Joint 3D Tracking and Forecasting with Graph Neural Network and Diversity Sampling. arXiv 2020 29