Impact of Traffic Lights on Trajectory Forecasting of Human-driven - PowerPoint PPT Presentation

12th Workshop on Planning, Perception, Navigation for Intelligent Vehicle @ IROS2020 Impact of Traffic Lights on Trajectory Forecasting of Human-driven Vehicles Near Signalized Intersections Geunseob (GS) Oh , Huei Peng University of Michigan

Video Reference: Waymo

Video Reference: Waymo We tackle this challenging vehicle forecasting problem near traffic lights (TLs) 𝐼𝑊 . Goal: Trajectory forecasts for the host vehicle, 𝑌 0:𝑈 𝐔𝐬𝐛𝐠𝐠𝐣𝐝 𝐦𝐣𝐡𝐢𝐮 Rear/Side Vehicle Traffic Flow (Present) Host Vehicle Front Vehicle 𝐈𝐩𝐭𝐮 𝐖𝐟𝐢𝐣𝐝𝐦𝐟 (Present) (Present) (𝐆𝐯𝐮𝐯𝐬𝐟) Well addressed in literatures Core elements of prediction near TL include: 𝐼𝑊 1. History of the host vehicle (HV), 𝑌 −𝜐:0 𝐺𝑊 , 𝑌 −𝜐:0 𝑆𝑊 , 𝑌 −𝜐:0 𝑇𝑊 2. Interactions with other vehicles, 𝑌 −𝜐:0 Has received much less attention, 𝑈𝑀 3. Rule imposed by traffic light (TL), 𝑌 𝑢 despite the importance.

Video Reference: Waymo We tackle this challenging vehicle forecasting problem near traffic lights (TLs) Goal: Trajectory forecasts for the host vehicle 𝐔𝐬𝐛𝐠𝐠𝐣𝐝 𝐦𝐣𝐡𝐢𝐮 Rear/Side Vehicle Traffic Flow (Present) Host Vehicle Front Vehicle 𝐈𝐩𝐭𝐮 𝐖𝐟𝐢𝐣𝐝𝐦𝐟 (Present) (Present) (𝐆𝐯𝐮𝐯𝐬𝐟) Our contribution: 1. Identification of the impacts of traffic lights on prediction; qualitative and quantitative 2. A novel prediction approach that is mindful of the impacts which utilizes vehicle-to- infrastructure (V2I) communications.

Video Reference: Waymo How does traffic light impact the prediction? 𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟐 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑢 −𝜐 𝑢 0 𝑢 𝑈 𝑄ℎ𝑏𝑡𝑓 time 𝑈𝑀 Possible predictions from methods that do not consider 𝑌 𝑢 𝐼𝑊 , 𝑌 −𝜐:0 𝐺𝑊 Given the phase (Red) at t=0, 𝑌 −𝜐:0 Existing methods would predict HV to stay put

Video Reference: Waymo How does traffic light impact the prediction? 𝐔𝐢𝐟 𝐮𝐬𝐯𝐮𝐢 𝐣𝐭 … 𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟐 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑢 −𝜐 𝑢 0 𝑢 𝑈 𝑢 −𝜐 𝑢 0 𝑢 𝑈 𝑄ℎ𝑏𝑡𝑓 time time Ground-truth Possible predictions from existing methods 𝐼𝑊 , 𝑌 −𝜐:0 𝐺𝑊 Actually, the phase changed to Green shortly after. Given the phase (Red) at t=0, 𝑌 −𝜐:0 The ground-truth trajectory started accelerating. Existing methods would predict HV to stay put

Video Reference: Waymo How does traffic light impact the prediction? 𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟑 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑢 −𝜐 𝑢 0 𝑢 𝑈 time 𝑈𝑀 Possible predictions from methods that do not consider 𝑌 𝑢 𝐼𝑊 , 𝑌 −𝜐:0 𝐺𝑊 Given the phase (Yellow) at t=0, 𝑌 −𝜐:0 Existing methods would predict HV to keep the speed

Video Reference: Waymo How does traffic light impact the prediction? 𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟑 𝐔𝐢𝐟 𝐮𝐬𝐯𝐮𝐢 𝐣𝐭 … 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑢 −𝜐 𝑢 0 𝑢 −𝜐 𝑢 0 𝑢 𝑈 𝑢 𝑈 time time 𝐼𝑊 , 𝑌 −𝜐:0 𝐺𝑊 Actually, the phase changed to Red shortly, Given the phase (Yellow) at t=0, 𝑌 −𝜐:0 The ground-truth trajectory started decelerating. Existing methods would predict HV to keep the speed

Video Reference: Waymo We propose a solution to the problem we identified Idea: Utilizing vehicle communications to infrastructures (V2I), obtain the future profiles of TL states ahead of time Image Reference: USDOT Future phase and timing can be shared through V2I

Video Reference: Waymo A sneak peek of the results 𝑈𝑀 ), When we leverages the future profiles of TL ( 𝑌 0:5𝑡 the predictions are so much better! 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑢 −𝜐 𝑢 0 𝑢 −𝜐 𝑢 0 𝑢 𝑈 𝑢 𝑈 𝑈𝑀 Pink: methods that do not leverage 𝑌 0:5𝑡 Blues and Reds (Fig. 4) are trajectories forecasted from our methods

Video Reference: Waymo Prediction model - setup HV Front Vehicle (FV) A data-driven approach A mapping function 𝑔 from states to actions 𝑌 𝑢 : state of the host vehicle + environment at time t 𝐼𝑊 𝑔 𝑌 𝑢−𝜐:𝑢 = 𝑏 𝑢 𝐼𝑊 : action of the host vehicle (acceleration) 𝑏 𝑢 We simplify the problem: longitudinal prediction with the presence of a preceding vehicle Dataset limitation: rear/side vehicles were not modeled.

Video Reference: Waymo Prediction model - setup 𝐼𝑊 𝑔 𝑌 𝑢−𝜐:𝑢 = 𝑏 𝑢 HV Front Vehicle (FV) 𝐼𝑊 , 𝑌 𝑢 𝐺𝑊 , 𝑌 𝑢 𝑈𝑀 , 𝑈𝑃𝐸 𝑢 In detail, a state is defined as: 𝑌 𝑢 ≔ 𝑌 𝑢 Host vehicle state ( 𝑌 𝐼𝑊 ): Longitudinal position (i.e., distance to the intersection) & speed Context ( 𝐷 ≔ [𝑌 𝐺𝑊 , 𝑌 𝑈𝑀 , 𝑈𝑃𝐸] ): 𝑌 𝐺𝑊 ≔ [𝐺𝑊 𝑢 , 𝑠 𝑢 , ሶ 𝑠 𝑢 ] FV state: captures interactions with the front vehicle (binary flag for presence of FV, relative pos, speed) 𝑌 𝑈𝑀 ≔ [𝑄 𝑢 , 𝑈 𝑢 ] TL state: captures interactions with traffic light ( phase (G,Y,R) and timing (time elapsed since the phase change)) 𝑈𝑃𝐸 Time of the day (0-24): macro-scopic traffic characteristics Output: Action taken by HV (longitudinal acceleration)

Image Reference: Google Map, UMTRI Video Reference: Waymo Dataset We used real-world driving records & traffic light states from SPMD: Naturalistic Driving Records of 3,000 vehicles over 2 years Host vehicle (GPS, kinematics, time information) Traffic light (TL state profile) Front Camera (post-processed information on FV) SPMD is a dataset established by USDOT & UMTRI A signalized intersection (Plymouth-Huron Pkwy, Ann Arbor) was used for a study The study includes 50 cars passed through the intersection Total 502,253 sample trips made during 03/2015 – 05/2017 (27 months)

Video Reference: Waymo Prediction model Modeling Intuition Deterministic Policy ( 𝑔 𝑒 ) Learning: RNN . . . + MLP RNN(LSTM) models LSTM temporal dependencies 𝐼𝑊 𝐼𝑊 𝑌 𝑢 𝑌 𝑢−𝜐 𝐷 𝑢 𝐼𝑊 𝑏 𝑢 𝐼𝑊 , 𝐷 𝑢 = 𝑏 𝑢 (a) 𝐼𝑊 𝑔 𝑒 𝑌 𝑢−𝜐:𝑢 Probabilistic Policy ( 𝑔 𝑞 ) Learning: RNN-MDN MDN captures M 𝑎 𝑢 . . . + D competing policies N LSTM And allows 𝐼𝑊 , 𝐷 𝑢 ; 𝑎 𝑢 ) probabilistic interpretation 𝐼𝑊 |𝑌 𝑢−𝜐:𝑢 𝐼𝑊 𝑞(𝑏 𝑢 𝐼𝑊 𝑌 𝑢 𝑌 𝑢−𝜐 𝐷 𝑢 𝐼𝑊 , 𝐷 𝑢 = 𝑎 𝑢 (b) 𝑔 𝑞 𝑌 𝑢−𝜐:𝑢

Video Reference: Waymo Prediction framework Autoregressive prediction using the learned policies to obtain the roll-outs

Video Reference: Waymo Qualitative evaluation Problem 𝐼𝑊(𝑄𝑠𝑓𝑡𝑓𝑜𝑢) 𝐺𝑊(𝑄𝑠𝑓𝑡𝑓𝑜𝑢) 𝑰𝑾(𝑮𝒗𝒖𝒗𝒔𝒇) [𝑌 𝐼𝑊 , 𝑌 𝐺𝑊 , 𝑌 𝑈𝑀 , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑔 𝑒 The impact of TL: The deterministic 𝑂𝑝𝐺𝑊 [𝑌 𝐼𝑊 , , 𝑌 𝑈𝑀 , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑔 𝑒 𝑂𝑝𝑈𝑀 policies 𝑔 𝑒 vs 𝑔 𝑒 𝑂𝑝𝑈𝑀 [𝑌 𝐼𝑊 , 𝑌 𝐺𝑊 , , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑔 𝑒 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑢 𝑈 𝑢 0 𝑢 0 𝑢 0 𝑢 𝑈 𝑢 𝑈 𝑥𝑗𝑜𝑒𝑝𝑥 Scenarios 𝑢𝑗𝑛𝑓 𝑢𝑗𝑛𝑓 𝑢𝑗𝑛𝑓 Scenario GYR Scenario G Scenario YR

Video Reference: Waymo Qualitative evaluation with 3 sample episodes: TL vs NoTL Given ground-truth (Black) trajectories, the trajectory forecasts from the following 3 models are compared: 𝑒 , Blue) : a model which uses both 𝒀 𝑮𝑩 and future 𝒀 𝑼𝑴 All ( 𝑔 Our approach 𝑂𝑝𝐺𝑊 , Red) : a model which doesn’t use 𝒀 𝑮𝑩 No FV ( 𝑔 𝑒 𝑂𝑝𝐺𝑊 , Pink) : a model which doesn’t use future 𝒀 𝑼𝑴 Benchmarking purpose No TL ( 𝑔 𝑒 Our models (blue and red) produce more accurate predictions than the model (pink ) that doesn’t utilize future 𝒀 𝑼𝑴 The results demonstrate how the utilization of the future 𝒀 𝑼𝑴 can improve the predictions

Video Reference: Waymo Quantitative evaluation with 3111 test samples & ablation study Models for the ablation study: [𝑌 𝐼𝑊 , 𝑌 𝐺𝑊 , 𝑌 𝑈𝑀 , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑔 𝑒 The impact of TL: [𝑌 𝐼𝑊 , , 𝑌 𝑈𝑀 , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑂𝑝𝐺𝑊 𝑔 𝑒 𝑂𝑝𝑈𝑀 𝑔 𝑒 vs 𝑔 𝑒 [𝑌 𝐼𝑊 , 𝑌 𝐺𝑊 , , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑂𝑝𝑈𝑀 𝑔 𝑒 [𝑌 𝐼𝑊 , , 𝑈𝑃𝐸] → 𝑏 𝑈𝑀 𝑂𝑝𝐺𝑊𝑈𝑀 𝑔 𝑒 Evaluation metrics: Scenarios: G, R, GY, YR, RG, GYR