Human Gesture Recognition for Drone Control Drones are cool - - PowerPoint PPT Presentation

Human Gesture Recognition for Drone Control

Drones are cool - Flying is hard 2

Drone Controllers 3

Proposed Solution: Gesture Control ● Drones already have cameras ● No additional HW required - Use human body as the controller ● Visible from long distances ● Requires little or no training ● Control commands that are universal and standardized 4

Our Solution 5

Our Solution (Skeleton Extraction) 6

Joint Based Feature Extraction - Raw Features Sample i ... Frame t Frame 1 (F 1 ) F 2 F 3 x 1 y 1 x 2 y 2 x 3 y 3 ... ... ... ... ... ... ... ... ... ... x n- y n- x n y n 1 1 Keypoints (Upper body 8x2 + left palm 21x2 + right palm 21x2 = 100) 7

Clustering Sample 17 C 1 C 2 C 1 C 2 C 1 C 2 Sample 18 9 9 9 F 1 F 2 F 3 ... F 4 F 5 F 6 ... C 19 C 2 8

Gesture Graphs Sample 1 Sample 2 G 1 G 7 C 2 C 7 C 1 C 1 C 1 C 2 C 3 ... C 2 C 1 C 1 C 2 ... 9 9 9 F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 1 F 2 F 3 F 4 ... ... 9

Maximum Entropy Markov Model Discriminative model 10

Training: Compute Probabilities Sample 2 G 7 P(C t 19 | F 1 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 2 | F 2 ) C 2 C 1 C 1 C 2 ... P(C t 19 | C t-1 2 , G 7 ) P(C t 19 | F 3 ) 7 9 P(C t 2 | C t-1 19 , G 7 ) P(C t 2 | F 4 ) F 1 F 2 F 3 F 4 ... 11

Trained Probabilities Inference P(C t 2 | C t-1 19 , G 7 ) P(C t 19 | F 1 ) P(C t 19 | F 1 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 19 | F 1 ) P(C t 19 | C t-1 2 , G 7 ) P(C t 2 | F 2 ) P(C t 2 | F 2 ) P(C t 19 | C t-1 2 , G 7 ) P(C t 19 | C t-1 2 , G 7 ) P(C t 2 | F 2 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 19 | F 3 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 19 | F 3 ) P(C t 2 | C t-1 19 , G 7 ) P(C t 19 | F 3 ) P(C t 2 | F 4 ) P(C t 2 | F 4 ) P(C t 2 | F 4 ) Sample test G n P(C t 17 | F 1 ) C 2 C 1 C 1 C 2 ... P(C t 2 | C t-1 17 , G n ) P(C t 2 | F 2 ) 9 7 P(C t 19 | C t-1 2 , G n ) P(C t 19 | F 3 ) F 1 F 2 F 3 F 4 P(C t 2 | C t-1 19 , G n ) ... P(C t 2 | F 4 ) 12

Baseline and Comparison 14 votes i=1 13

Dataset ● Source: Isolated Gesture Recognition (ICPR '16) ● RGB-D gesture videos = 47,933 (1 video = 1 gesture) ● Gestures labels 249 ● Different individuals 21 ● Contains 9 Air Marshalling gestures (along with others) ● Data Samples Split: Train 1399, Valid 200, Test 300 ● Used OpenPose to extract 2D skeleton data from RGB videos Move backward 14

Results Model Precision Recall Accuracy MEMM 0.80 0.80 0.80 Multilayer Perceptron 0.77 0.77 0.74 Eigen Joints 0.80 0.79 0.76 HMM Tuned 0.67 0.71 0.66 HMM Baseline 0.32 0.37 0.38 15

Challenges and Error Analysis MEMM HMM 16

Conclusion - We proposed a real-time gesture recognition for drone control using structured prediction. - Our proposed model achieved an improvement in accuracy of ~15% over the baseline (tuned). - Future Work - Combining multiple graphical probabilistic models. - Adding the hand joints for the HMM baseline. 17

Thanks :) 18

Appendix 19

Challenges and Our approach ● Detecting Human Body Pose ● OpenPose to extract skeleton data from RGB camera ● Maintaining Visibility ● Drone will rotate (yaw) to always face the commander ● Gestures Selection ● Use Aircraft Marshalling gestures 20

Drone Dynamics 21

For Paper s i=12 z g=1 y k= y k= y k= y k= y k= y k=19 y k= ... 1 1 2 2 7 3 x t=4 x t=5 x t=6 x t=7 x t=1 x t=2 x t=3 ... 22