Pe Pedestria ian n Tra Trajectory jectory Predi redicti ction - PowerPoint PPT Presentation

Pe Pedestria ian n Tra Trajectory jectory Predi redicti ction on

Ov Overv rview • What is pedestrian trajectory prediction good for? • What is the task of pedestrian trajectory prediction? • What makes trajectory prediction challenging? • What kind models are used for trajectory prediction? – Deterministic models – Stochastic Models • How to model social interactions? • How to model agent-scene interactions? CV3DST | Prof. Leal-Taixé 2

In Introduction Autonomous vehicles must predict the future movements of • pedestrians in order to avoid fatal collisions CV3DST | Prof. Leal-Taixé 3

In Introduction Autonomous vehicles must predict the future movements of • pedestrians in order to avoid fatal collisions CV3DST | Prof. Leal-Taixé 4

In Introduction Social robots that move autonomously through crowded scenes • and interact with moving humans CV3DST | Prof. Leal-Taixé 5

In Introduction Motion models improve the performance of Multi-Object Trackers • www.motchallenge.net CV3DST | Prof. Leal-Taixé 6

Ta Task of pe pedestri rian n tra rajectory ry pre prediction Scene • Sequence of observations: " = " for 𝑢 = 1, … , 𝑈 #$% " , 𝑧 ! 𝑌 ! 𝑦 ! • Ground Truth: " = " , 𝑧 ! " 𝑍 𝑦 ! ! for 𝑢 = 𝑈 #$%&' , … , 𝑈 ()*+ • Prediction: " = 𝑔 + " , 𝑧 ! " 𝑍 , 𝑌 ! = 𝑦 ! ! for 𝑢 = 𝑈 #$%&' , … , 𝑈 ()*+ CV3DST | Prof. Leal-Taixé 7

In Introduction Human motion behavior is influenced by a variety of different factors: • 2. Personal 1. Destination Preferences 3. Human – Space 4. Human – Human Interactions Interactions CV3DST | Prof. Leal-Taixé 8

So Soci cial l Force ce Model Contribution: Mathematical model of dynamics a: Pedestrian act in force field 𝐺 like particles e.g. in • Idea: an electric field +- ! . " 𝑦 = +" ! = • Second Newton’s law: ̈ / • Trajectory x 𝑢 is solution of differential equation [Helbing et al. 98] Social Force Model CV3DST | Prof. Leal-Taixé 9

So Soci cial l Force ce Model Obstacle B Destination and personal ⃗ preferences s 𝐺 direction of desired velocity !$ destination # = ⃗ # ⃗ ⃗ # ( ⃗ 𝐺 𝐺 𝑤 " 𝑢 , 𝑤 " 𝑓 " ) " " ⃗ " 𝐺 ! Human – Human interactions α distance betw. peds Destination of 𝛽 ⃗ 1 𝐺 "$ (⃗ 𝑓 " , ⃗ 𝑠 "$ ) Pedestrian 𝛽 $ ⃗ 𝐺 Human – Space interactions !# 𝛾 ⃗ " ) 1 𝐺 "! (⃗ 𝑓 " , ⃗ 𝑠 ! ! CV3DST | Prof. Leal-Taixé 10

̈ So Soci cial l Force ce Model • Force resultant determines motion of pedestrian 𝛽 • The respective trajectory 𝑦 𝑢 is the solution of a differential equation: 𝑦(𝑢) = 𝑒𝑦 < 𝑒𝑢 < = 𝐺 = (𝑢) CV3DST | Prof. Leal-Taixé 11

So Soci cial l Force ce Model • The force between pedestrians is described by the gradient of a repulsive potential 𝑊 => : ‖%"#‖ 𝑠 => = 𝑊 ? 𝑓 @ ⃗ 𝐺 => ⃗ 𝑠 => = − ∇ ⃗ ) "# 𝑊 => (⃗ 𝑠 => ) with 𝑊 => ⃗ & • Parameters 𝑊 ? and 𝜏 determine the shape of this potential • Parameters effect strength of interaction between pedestrians CV3DST | Prof. Leal-Taixé 12

So Soci cial l Force ce Model • Exponential potential shaped by 𝑊 ? and 𝜏: • Interaction Potential: ‖%"#‖ 𝑠 => = 𝑊 ? 𝑓 @ • 𝑊 => ⃗ & CV3DST | Prof. Leal-Taixé 13

So Soci cial l Force ce Model • Exponential potential shaped by 𝑊 ? and 𝜏: • Interaction Potential: ‖%"#‖ 𝑠 => = 𝑊 ? 𝑓 @ • 𝑊 => ⃗ & CV3DST | Prof. Leal-Taixé 14

So Soci cial l Force ce Model • Exponential potential shaped by 𝑊 ? and 𝜏: • Interaction Potential: ‖%"#‖ 𝑠 => = 𝑊 ? 𝑓 @ • 𝑊 => ⃗ & CV3DST | Prof. Leal-Taixé 15

So Soci cial l Force ce Model • Exponential potential shaped by 𝑊 ? and 𝜏: • Interaction Potential: ‖%"#‖ 𝑠 => = 𝑊 ? 𝑓 @ • 𝑊 => ⃗ & CV3DST | Prof. Leal-Taixé 16

So Soci cial l Force ce Model l Si Simula lati tion CV3DST | Prof. Leal-Taixé 17

Dr Drawba back ck of So Soci cial l Force ce Model • Model requires many parameters for each agent and all agent-agent and agent-environment pairs • Shape of interaction functions are handcrafted CV3DST | Prof. Leal-Taixé 18

Dr Drawba back ck of So Soci cial l Force ce Model l Handcrafted Learned Functions Functions CV3DST | Prof. Leal-Taixé 19

Ne Neural Ne Network k fo for Trajectory Prediction • Building a naïve prediction model with FC layers • FC Layers do not account for sequential and temporal behavior of trajectories Recurrent Neural Networks CV3DST | Prof. Leal-Taixé 20

Rec Recurren ent Neu eural al Net etwor orks https://colah.github.io/posts/2015-08-Understanding-LSTMs/ CV3DST | Prof. Leal-Taixé 21

Si Simple le RNN CV3DST | Prof. Leal-Taixé 22

LSTM (Lo LSTM Long ng Sho Short-te term Memory) Cell Forget Output State Gate Hidden Output Input State Gate Gate [Hochreiter et al. 97] Long Short-term Memory CV3DST | Prof. Leal-Taixé 23

LSTM LSTM for Tr Trajecto ctory Predicti ction • Using a sinlge LSTM does not work well Encoder-Decoder Architecture CV3DST | Prof. Leal-Taixé 24

LS LSTM Encode der –De Decode der Ar Archit itecture Input Output LSTM LSTM ()*,…,- !"# z ()- !"#$% ,…,- &'() & X ' Y ' Encoder Decoder Compressed (latent) representation # Training Objective: ℒ = Y − Y < CV3DST | Prof. Leal-Taixé 25

Van Vanilla a LSTM • Vanilla LSTM is not able to predict trajectories of interacting pedestrians [Zhang et al. 19] SR-LSTM CV3DST | Prof. Leal-Taixé 26

So Soci cial l LSTM LSTM Contribution: Modeling social interactions • LSTM encoder-decoder architectures • Social pooling between neighboring pedestrians in each time step [Alahi et al. 16] Social LSTM CV3DST | Prof. Leal-Taixé 27

So Soci cial l LSTM LSTM – Po Pooli ling g Module le • Interaction Module pools hidden states of LSTM of pedestrian in vicinity • Pooled hidden states are passed to decoder for next step prediction [Alahi et al. 16] Social LSTM CV3DST | Prof. Leal-Taixé 28

So Soci cial l LSTM LSTM – Res Result Comparison of Models with and without social pooling Social Pooling can resolve social interactions Plots by Philipp Mondorf 20, BA CV3DST | Prof. Leal-Taixé 29

Pe Pede destria ian Trajectorie ies are multim imoda dal Same past trajectory can have multiple realistic future trajectories CV3DST | Prof. Leal-Taixé 30

De Dete terministi tic c vs vs. Sto Stoch chasti tic c Models ls Deterministic Stochastic One-to-one mapping One-to-many mapping Learning distribution of future trajectories instead of deterministic mapping Distribution of 2d distribution of Scenario final end final end CV3DST | Prof. Leal-Taixé 31

To Towa wards Gene nerati tive ve Models ls Deterministic Generative Models Models CV3DST | Prof. Leal-Taixé 32

Ge Genera rative M Models Figure from Ian Goodfellow, Tutorial on Generative Adversarial /networks, 2017 CV3DST | Prof. Leal-Taixé 33

Rec Recap ap: Gen ener erat ative e Model Models Figure from Ian Goodfellow, Tutorial on Generative Adversarial /networks, 2017 CV3DST | Prof. Leal-Taixé 34

Var Variat ation onal al Autoen oencoder oder https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 35 variational-autoencoders-vaes-f70510919f73

Var Variat ation onal al Autoen oencoder oder https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 36 variational-autoencoders-vaes-f70510919f73

Var Variat ation onal al Autoen oencoder oder https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 37 variational-autoencoders-vaes-f70510919f73

Var Variat ation onal al Autoen oencoder oder https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 38 variational-autoencoders-vaes-f70510919f73

Variat Var ation onal al Autoen oencoder oder Ku Kulbac ack- Le Leibler (K (KL) diverg di rgence https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 39 variational-autoencoders-vaes-f70510919f73

Var Variat ation onal al Autoen oencoder oder https://towardsdatascience.com/understanding- [Kingma et al. 16] Variational Bayes CV3DST | Prof. Leal-Taixé 40 variational-autoencoders-vaes-f70510919f73

Var Variat ation onal al Autoen oencoder oder Each element of z encodes a different feature CV3DST | Prof. Leal-Taixé 41

Var Variat ation onal al Autoen oencoder oder Degree of smile Head pose CV3DST | Prof. Leal-Taixé 42