Real-Time Pedestrian Tracking, Prediction & Navigation Dinesh - PowerPoint PPT Presentation

Real-Time Pedestrian Tracking, Prediction & Navigation Dinesh Manocha Univ. of North Carolina dm@cs.unc.edu http://gamma.cs.unc.edu

Collabo Collaborators • Aniket Bera • Jur van den Berg (UNC/Utah/Google/Otto/Uber) • Andrew Best • Sean Curtis (UNC/Boeing/TRI) • Ernest Cheung • Stephen Guy (UNC/Minnesota) • Davik Kasik (Boeing) • Sujeong Kim (UNC/SRI) • Ming C Lin • Rahul Narain (UNC/Berkeley/Minnesota) • Sahil Narang • Chonhyon Park (UNC/Zoox) • Sachin Patil (UNC/Berkeley/Ottp/Uber) • Ari Shapiro (ICT/USC) • Jamie Snape (UNC/Kitware) • Basim Zafar (Hajj Research Institute) 2

Pe Pedestrian Ve Vehicle In Interactio ions

Pe Pedestrian/Cro rowd Sim Simula lation tion & Pre Prediction 4

AC ACM New News: Avoi oiding ding th the Crush Crush (May 2016) By Keith Kirkpatrick ACM NEWS May 3, 2016 Avoiding the Crush Researchers have been developing models that mimic how people move in large groups. Earthquakes, floods, and hurricanes take lives around the world, due to their unpredictable nature and massive power. Humans are responsible for similar levels of carnage through human crushes or stampedes; for example, more than 2,200 Muslim pilgrims were killed at the Ratangarh Mata Temple in India killed 115 people and injured more than 100. To address such public safety issues, researchers have been developing computer models that mimic how people actually move when in large groups. Their ultimate goal: to develop reliable algorithms that can feed into models to accurately predict how crowds move, then design physical spaces to safely accommodate and control that movement to prevent deadly crushes. 5

Pe Pedestrian Mo Motion on Sim Simula lation tion • Non ‐ Velocity Based • Rule ‐ based • Boids • Force ‐ based • Cellular Automata (CA) • Velocity Based • Plan control based on velocity ‐ space considerations • … 6

RVO: “R RVO “Reactiv eactive” e” Obs Obstacl acles • Reciprocity Assumption Y Vy Vx X Workspace Velocity Space A – v A  VO A • RVO A B (v B , v A ) = {v’ A | 2v’ B (v B )} [Berg et al. 2008] 7

Bene Benefit fit of of re reciprocity: Pe Pedestrian mot motion on model model Velocity Obstacle (VO) Reciprocal Velocity Obstacle (RVO) 8

Me Menge: nge: Open Open Sour Source ce pedes pedestrian rian sim simula lation tion fr framework http://gamma.cs.unc.edu/Menge 9

Pedestrian Tracking and Prediction Where? • Anomaly Detection • Behavior Analysis • Crowd Counting • Driverless Cars • Robotics

Pedestrian Tracking and Prediction: Pipeline Input Crowd Video Crowd Video

Pedestrian Tracking and Prediction: Pipeline Motion-model Driven Sensor Capture Input Trajectory Trajectory Crowd Video Crowd Video Extraction Extraction EnKF State EnKF State Estimation Estimation

Pedestrian Tracking and Prediction: Pipeline Motion-model Driven Sensor Capture Input Trajectory Trajectory Crowd Video Crowd Video Extraction Extraction Compute Compute Pedestrian Pedestrian Clusters Clusters EnKF State EnKF State Estimation Estimation

Pedestrian Tracking and Prediction: Pipeline Crowd Flow Learning Motion-model Driven Sensor Capture Input Trajectory Trajectory Global Global Crowd Video Crowd Video Movement Flow Movement Flow Extraction Extraction Movement Movement Compute Compute Learning Learning Patterns Patterns Pedestrian Pedestrian Clusters Clusters EnKF State EnKF State Estimation Estimation

Pedestrian Tracking and Prediction: Pipeline Motion-model Driven Sensor Capture Crowd Flow Learning Input Trajectory Trajectory Global Global Crowd Video Crowd Video Movement Flow Movement Flow Extraction Extraction Movement Movement Compute Compute Learning Learning Patterns Patterns Pedestrian Pedestrian Predicted State Predicted State Learning Learning Clusters Clusters Local Local EnKF State EnKF State Microscopic and Microscopic and Movement Movement Estimation Estimation Macroscopic Macroscopic Patterns Patterns Motion Models Motion Models

Pedestrian Tracking and Prediction: Pipeline Prediction Feedback Crowd Flow Learning Motion-model Driven Sensor Capture Input Trajectory Trajectory Global Global Crowd Video Crowd Video Movement Flow Movement Flow Extraction Extraction Movement Movement Compute Compute Learning Learning Patterns Patterns Pedestrian Pedestrian Predicted State Predicted State Learning Learning Clusters Clusters Local Local EnKF State EnKF State Microscopic and Microscopic and Movement Movement Estimation Estimation Macroscopic Macroscopic Patterns Patterns Motion Models Motion Models

State Estimation Pedestrian Pedestrian Simulation Simulation Predicted states f( x ) Noisy observation Maximum Maximum z Q Sensor Sensor EnKF EnKF Likelihood Likelihood Error Estimation Estimation distribution x Estimated state 17

Local Features Pedestrian dynamics feature: Position, Average velocity Goal velocity For each time step, for each pedestrian Entry Points: - Popular entry points (temporal variance) - Gaussian Mixture Model Entry points learning 18 (Gaussian Mixture Model)

Global Features Movement Flow Learning (K-means clustering) 19

What more do we need? • Behavior Learning • Culture sensitive • Scene specific

Pedestrian/Crowd Classification

Synthetic Labeled Datasets for learning http:/// gamma.cs.unc.edu/RCrowdT/Dataset http://arxiv.org/abs/1606.08998

LCrowdV: Synthesized Labeled datasets for pedestrian video analysis

Synthetic Labeled Data for Learning: pedestrian detection + behavior classification

Pedestrian Behavior Learning:Pipeline Live Live Linear Regression Fitting Linear Regression Fitting Video Stream Video Stream Based on Eysenck Model Based on Eysenck Model Predict Future state based Predict Future state based State Estimation State Estimation on Restricted Behavior on Restricted Behavior State Space State Space Labeled Crowd Dataset Labeled Crowd Dataset Prediction/Navigation Prediction/Navigation

Pedestrian State Pedestrian State Labeled Dataset Impulsive Shy Pedestrian Path Prediction Pedestrian Path Prediction

Real ‐ time Pedestrian Behavior Classification Video : International Trade Fair, New Delhi 2016

Improved Tracking and Prediction using Behavior Classification

Panic Simulation

Robot Navigation

Robot Navigation Kinematic model of the robot Robot’s position R(t, u) at time t given the control u can be derived as follows

Robot Navigation: Improvements GVO : Generalized velocity obstacles for non-holonomic constraints

Ongoing Work: Vehicle Navigation Simulator

Pedestrian Modeling: Conclusions • Improved motion models for dense scenarios • Pedestrian dynamics learning using Bayesian Inference • Simulation + perception + learning • Use of synthetic labeled datasets for deep learning • Results: • Real-time pedestrian tracking and prediction in dense scenes • Pedestrian behavior prediction • Improved accuracy

Ongoing and Future Work • Predict pedestrian’s mood and body language • Integrate with autonomous vehicles • Evaluate different motion planning algorithms • Develop robust pedestrian navigation algorithms • Model other behaviors: bicycles

Acknowledgments • Collaborators: • Boeing, Hajj Research Institute, Julich SuperComputing Center, Intel, Relic, Willow Garage • Funding: • Army Research Office, National Science Foundation, Intel, HajjCORE, Boeing, KAUST, Willow Garage 38