Humanoid Companion Embedded GPUs can make your robotic companion - PowerPoint PPT Presentation

Smarter Humanoid Companion Embedded GPUs can make your robotic companion more alive Hi Alex! Alexandre Mazel Innovation Software Director Oct 2017

Agenda Overview ● Innovation Team Presentation ○ Mummer Research Project ○ Problematics ● Proposed solution ● Live Demo ● Question ●

Team Presentation Innovation Software AI Lab Protolab Fundamental Research on Applied Research Developmental Robotics 3 Permanent 4 Permanent 3 PhD student 1 PhD student 3 Intern Part of the Innovation Department, which includes hardware, electronics, collaborative projects and design.

Goal: Prospection Enhance our Humanoid Robots for more natural Human-Robot Interaction ● Explore future uses ● Test and embed new algorithms ● Hardware improvement ● Provide versatile platforms for research

Goals: ● Make Pepper navigates in malls ● Entertain visitors/customers Experimentation field: ● Ideapark mall in Lempäälä (Finland) ● Huge: More than 150 stores, restaurants and cafes within 100.000 m2 ● Crowdy: 7 million visitors (2013)

University of Glasgow SoftBank Robotics Europe Heriot-Watt University VTT TechnicalResearch Center of Finland Idiap Research Institute Ideapark LAAS-CNRS

Challenges ● Obstacle avoidance ● Quick person detection (<1s) ● Self Localization ● Data confidentiality

Navigation Sensors RGB Camera (55°H, 44°V) Depth Camera (58°H, 45°V) RGB Camera (55°H, 44°V) Sonar Laser (45 points, up to 3m)

Current limitations

Current limitations

Proposed solution: case study RGB Fish Eye (100°H, 180°V)

Robot POV

ConvNet Learning for obstacle avoidance Passable Non passable ● pretrained AlexNet using the LSVRC-2010 ImageNet (1.3M Images) ● learning FC7, FC8 and binary classification

Collection of training data

Region of Interest

Results of the learning process 3400 images ● Learning rate: 0.001 ● dropout rate: 0.5 ● batches of size: 40 ● duration of one epoch: 70 sec (using a Geforce GTX 1070) ●

Proposed solution: case study DC-DC 29V-12V Jetson TM TX2 RGB Fish Eye (100°H, 180°V)

Embedding Jetson TM TX2 USB Ethernet (or wifi)

Video Technical Demo

Embedding Jetson TM TX2 Action Time (s) Acquire image 0.016 Computing difference 0.005 Undistort and rotation (numpy) 0.049 Inference 0.033 Total Time 0.103 (9.7 fps) NB: Using the trained tensorflow model as is

Battery Draining Measure Standard Pepper - no movement 11h32 Pepper with gpu processing and infering every 10h21 frame - no movement NB: based on one test only

Advantages ● Dodges obstacles ● Fully autonomous (no cloud, no wifi) ● Quick training - can be done multiple times ● Can be learned directly on site ● Confidentiality is preserved

To be continued Next steps: ● add more classes (left/right/center) ● optimisation (int8, tensorRT , …) ● autonomous & continuous learning on the fly Future work: ● Navigation: Localisation/VSlam ● Skeleton estimation (2D) (Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh) ● Face features extraction ● Speech Recognition (Caldi)

Live Demo

Conclusion

Acknowledgement Based on work from: ● Abdelhak Loukkal (2017) ● Michael Guerzhoy and Davi Frossard (2016) Reference: ● Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks (2015)

Questions Time More questions: Alexandre Mazel amazel@softbankrobotics.com