The Curious Robot: Learning Visual Representa6ons via Physical - PowerPoint PPT Presentation

The Curious Robot: Learning Visual Representa6ons via Physical Interac6ons Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, Abhinav Gupta ECCV 2016 Presenter: Ginevra Gaudioso R B

Problem • Learning visual representa-ons of objects • By ac-vely interac-ng with the objects Image source: hOps:// youtu.be/oSqHc0nLkm8?t=49

Why do we care? • Robo6cs: robot needs to recognize objects • Vision: classifica6on Image source: hOps:// youtu.be/oSqHc0nLkm8?t=47

Related Work • Unsupervised Learning – Other approaches use passive data – Here the robot ac6vely plays with the objects • Robo6c Tasks – In robo6cs, we use vision to plan the best grasp – Here we use grasp data to classify the object

Approach • Use robot’s experiment results to label objects • Train Network to predict experiment results given the picture of the object as input Experiments: - Grasping - Pushing - Poking - View at different angles Image source: www.robo6csbusinessreview.com

Grasp Image source: paper

Grasp Image source: paper

Push Image source: paper

Push Image source: paper

Poke Image source: paper

Poke Image source: paper

Complete Network

Complete Network Image source: paper

Complete Network Image source: paper

Complete Network Image source: paper

Experiments – Root Network Which images generate similar ac6va6on paOerns in the Root network?

Experiments – Root Network Network learns high level features of objects, such as shape. Image source: paper

Experiments – Image Retrieval Recall@k : this approach leads to good retrieval levels Image source: paper

Experiments – Image Retrieval Nearest Neighbors relies mostly on shape Image source: paper

Discussion • Strengths – Robot learns all by itself – Able to learn meaningful features (shape) – Very good retrieval results • Weaknesses – Physical interac6on is expensive – Network is heavily handcraged – Limited to objects physically available in training

Extensions • Allow for passive data to enrich dataset • Ac6vely choose which training data to gather • Would this work with simpler network?