NATURAL, INTERACTIVE TRAINING OF SERVICE ROBOTS TO DETECT NOVEL - PowerPoint PPT Presentation

MUNICH 10-12 OCT 2017 NATURAL, INTERACTIVE TRAINING OF SERVICE ROBOTS TO DETECT NOVEL OBJECTS Elisa Maiettini and Dr. Giulia Pasquale Joint work with: Prof. Lorenzo Natale, Prof. Lorenzo Rosasco

iCub R1

What do we need? Target scenario WINDOW LIBRARY To close at Contains books PLANT night To dust every week To water every friday PLANT To water every monday SOFA Elisa’s favorite sofa

Step 1: Object Detection Step 0: Object Recognition  Library Window Plant Sofa Sofa Sofa Plant Are we done with Object Recognition? The R1 perspective. Giulia Pasquale, GTC 2017, San Jose, CA [ http://on-demand.gputechconf.com/gtc/2017/video/s7295-giulia-pasquale-are- we-done-with-object-recognition-the-r1-robot-perspective.PNG.mp4k ]

Approaches to Object Detection What is Where? Sliding window Region-based Grid-based No object … approach: approaches [3][4][5] : approaches [1][2] : No object No object It’s a sofa! 1. Slide a window on the image 1. Partition the image with a grid 1. Identify Regions of Interest (RoI) 2. Run a classifier for each window 2. Run a classifier for each grid’s cell 2. Run a classifier for each RoI We cannot run a classifier for [1] Redmond J. et al, 2016 [3] Girshick R et al., 2014 [5] Shaoqing R et al., 2015 [2] Liu W. et al, 2016 [4] Girshick R et al., 2015 [6] Uijlings J. R. R., 2013 each possible window!

Region-based approach: Faster R-CNN [5] ROI pooling Where to What? Classifier Layer fc6 fc7 Region look? Classification proposals scores RPN Where? Predicted CNN Bounding boxes Bounding Feature box regressor Map For each ROI  Modularity = Flexibility!!  RPN is faster and more efficient than external methods [5] Shaoqing R et al., NIPS, 2015

Object Detection task: Robotic setting Robotics brings new challenges… …but also more information! ? Time coherence • Open-set problem • Contextual information • Automatic self-supervision • from sensors (e.g. depth)

Object Detection for Robotics: our solution 1. Data acquisition [8] and 2. Deployment on R1 model training Detection system deployed on R1 Bounding boxes thanks to the on board Jetson Tx2 Labels [8] Pasquale et al., Frontiers 2016

Object Detection for Robotics: our solution iCubWorld Taransformations dataset [9] 1 2 3 4 5 6 7 iCubWorld 8 9 10 [9] Pasquale et al. IROS 2016 [https://robotology.github.io/iCubWorld]

Object Detection for Robotics: evaluating models Scenario 1: same HRI setting 2. Validate results: predictions compared with manual 1. Predictions compared with automatically Ground Truth (Mean Average Precision = 0.75) acquired Ground Truth (Mean Average Precision = 0.71) Even better! Scenario 2: different scenes New sequences acquisition and manual Promising results: annotation mAP floor =0.55, mAP table =0.66 mAP shelf =0,53

Deployment on R1

Object Detection on R1 Train details 2 Performed offline on: RPN Train: GPU : NVIDIA Tesla P100 Iterations: 81k Time: ~40 minutes using: 2 CNN: Zeiler and Fergus network [9] Detector Train: DATASET: iCubWorld Transformations Iterations: 54k Time: ~53 minutes with: Num images: ~27k Total train Time: ~3 hours [9] Zeiler M. D. and Fergus R., CoRR, 2013

Object Detection on R1 Deployment details Thanks to NVIDIA Jetson Tx2:  fast & easy  fully autonomous platform  easy systems integration CAFFE & YARP & Python Evaluation of Tensor RT framework Regions per Frame Frame per second 100 ~4 300 ~3 1000 ~2

Contribution: Future steps: 1. Pipeline to overcome lack of manual 1. Further exploit contextual annotation for robotic platforms information to improve precision (e.g. time coherence) 2. Deployment of detection system on NVIDIA Jetson Tx2 on board 2. Extend the system to open sets of R1 towards scene understanding task

Lorenzo Giorgio Lorenzo Natale Metta Rosasco Principal Research Team Leader, Investigator, Director, LCSL iCub Facility iCub Facility Tanis Vadim Ugo Marco Alberto Raffaello Alessandro Carlo Francesca Mar Tikhanoff Pattacini Randazzo Cardellino Camoriano Rudi Ciliberto Odone Technologist Technologist Senior Junior Postdoc, Postdoc, Postdoc, Assistant Research Researcher, Researcher, Technician, Technician, iCub Facility LCSL LCSL Professor, Associate, iCub Facility iCub Facility iCub Facility iCub Facility UCL IRIS Univ. of Genoa • … And all the team of the iCub Facility and the Laboratory for Computational and Statistical Learning

R1 is looking forward to meet you! Please come to see it at Booth number

Thank you! Any question? Giulia Pasquale Elisa Maiettini giulia.pasquale@iit.it elisa.maiettini@iit.it