GESTURE RECOGNITION: USING A MULTI SENSOR APPROACH SHALINI GUPTA, - PowerPoint PPT Presentation

GESTURE RECOGNITION: USING A MULTI SENSOR APPROACH SHALINI GUPTA, PAVLO MOLCHANOV, KIHWAN KIM, KARI PULLI, JAN KAUTZ NVIDIA RESEARCH

DRIVER DISTRACTION

GESTURE INTERFACE (http://www.softkinetic.com)

DAY

DAY NIGHT

NO SUNLIGHT

NO SUNLIGHT SUNLIGHT

MULTI-SENSOR SOLUTION COLOR + DEPTH RADAR sensors

MULTI-SENSOR SOLUTION COLOR + DEPTH RADAR gesture UI sensors

3.2% INCREASED ACCURACY +1.5m/s 0 m/s -1.5m/s 3D shape color velocity

16X POWER EFFICIENCY velocity power v 0.15W t

16X POWER EFFICIENCY velocity power v 2.5 W v T 0.15W gesture gesture t

SHORT RANGE FMCW RADAR radar prototype

SHORT RANGE FMCW RADAR y +1.5m/s 0 m/s x v radar prototype 4D vector z -1.5m/s

POSITION RESULTS

VELOCITY RESULTS +1.5m/s 0 m/s -1.5m/s

GESTURE NETWORK fully connected NN logistic regression 60 3D convolutional frames layer subsampling layer Trained on GPU

10 GESTURES PALM up down left right SHAKE SWIPE right left ROTATION CALL clockwise counter-clockwise

INDOOR CAR SIMULATOR

OUTDOOR CAR

ERROR RATE 39.90% 10.90% 9.10% C D R D+C R+D R+C R+D+C D – depth C – color R - radar

ERROR RATE 39.90% 10.90% 9.10% 7.90% 8.30% 7.40% C D R D+C R+D R+C R+D+C D – depth C – color R - radar

ERROR RATE 39.90% 10.90% 9.10% 7.90% 8.30% 7.40% 5.90% C D R D+C R+D R+C R+D+C D – depth C – color R - radar

ERROR RATE 20.90% D+R (CNN) D+R+C (CNN) 9.70% 6.70% D+C (HOG) 3.00% Night Evening Day (shadow) Day (sunlight) D – depth C – color R - radar

ERROR RATE 20.90% D+R (CNN) D+R+C (CNN) 9.70% 6.70% 6.70% D+C (HOG) 8.30% 7.50% 3.00% 1.50% Night Evening Day (shadow) Day (sunlight) D – depth C – color R - radar

ERROR RATE 22.20% 20.90% D+R (CNN) 13.00% D+R+C (CNN) 6.70% D+C (HOG*) 8.30% 7.50% 2.45% 1.50% Night Evening Day (shadow) Day (sunlight) D – depth C – color * Ohn-Bar and Trivedi, IEEE Trans. on R - radar Intelligent Transportation Systems , 2014.

DEMO 52ms on Quadro 6000

CONCLUSION COLOR + DEPTH RADAR 1 GESTURE UI 2 1 Multi- sensor System for Driver’s Hand -Gesture Recognition , IEEE Automatic Face and Gesture Recognition, May 2015. 2 Short-Range FMCW Monopulse Radar for Hand-Gesture Sensing , IEEE International Radar Conference, May 2015.

THANK YOU SHALINIG@NVIDIA.COM

QUESTIONS?