Deep Robotic Learning Sergey Levine UC Berkeley Google Brain - PowerPoint PPT Presentation

Deep Robotic Learning Sergey Levine UC Berkeley Google Brain

robotic state low-level modeling & control observations estimation planning controls control prediction (e.g. vision) pipeline

standard classifier features mid-level features computer (e.g. SVM) (e.g. HOG) (e.g. DPM) vision Felzenszwalb ‘08 end-to-end training deep learning robotic state low-level modeling & control controls observations estimation planning control prediction (e.g. vision) pipeline end-to-end training deep state low-level modeling & robotic controls observations estimation planning prediction control (e.g. vision) learning

no direct supervision actions have consequences

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

Chelsea Finn

96.3% success rate end-to-end training 0% success (trained on pose only) rate pose prediction L.*, Finn*, Darrell, Abbeel , ‘16

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

Deep Robotic Learning Applications manipulation dexterous hands soft hands with N. Wagener, P. Abbeel with V. Kumar, A. Gupta, E. Todorov with C. Eppner, A. Gupta, P. Abbeel locomotion aerial vehicles tensegrity robot with X. Geng, M. Zhang, J. Bruce, K. Caluwaerts, M. Vespignani, V. SunSpiral, P. Abbeel with G. Kahn, T. Zhang, P. Abbeel with V. Koltun

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

ingredients for success in learning: supervised learning: learning robotic skills: computation computation ~ data algorithms algorithms ? data

Grasping with Learned Hand-Eye Coordination monocular • monocular camera (no depth) RGB camera • no camera calibration either 7 DoF arm • 2-5 Hz update • continuous arm control 2-finger gripper • servo the gripper to target • fix mistakes object • no prior knowledge bin Alex Peter Pastor Krizhevsky Deirdre Quillen L., Pastor, Krizhevsky, Quillen ‘16

Grasping Experiments

Policy Learning with Multiple Robots Rollout execution Local policy optimization Global policy optimization Mrinal Ali Yahya Kalakrishnan Yevgen Chebotar Adrian Li

Yahya, Li, Kalakrishnan, Chebotar , L., ‘16

Policy Learning with Multiple Robots: Deep RL with NAF Ethan Holly Tim Lillicrap Shane Gu Gu*, Holly*, Lillicrap , L., ‘16

Learning a Predictive Model of Natural Images original video Chelsea Finn predictions

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

Safe Uncertainty-Aware Learning unknown environment Key idea: To learn about collisions, must experience collisions (but safely!) 1. Learn a collision prediction model raw image command velocities neural network ensemble 2. Speed-dependent, uncertainty-aware collision cost Greg Kahn 3. Iteratively train with on-policy samples Kahn, Pong, Abbeel , L. ‘16

Safe Uncertainty-Aware Learning Kahn, Pong, Abbeel , L. ‘16

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ?

Training in Simulation: CAD2RL Fereshteh Sadeghi Sadeghi , L. ‘16

Training in Simulation: CAD2RL Sadeghi , L. ‘16

Training in Simulation: CAD2RL Sadeghi , L. ‘16

Sadeghi , L. ‘16

Learning with Transfer in Mind: Ensemble Policy Optimization (EPOpt) training on single torso mass training on model ensemble train test unmodeled effects ensemble adaptation adapt Aravind Rajeswaran

1. Does end-to-end learning produce better sensorimotor skills? 2. Can we apply sensorimotor skill learning to a wide variety of robots & tasks? 3. Can we scale up deep robotic learning and produce skills that generalize ? 4. How can we learn safely and efficiently in safety-critical domains? 5. Can we transfer skills from simulation to the real world , and from one robot to another ? 6. How can we get sufficient supervision to learn in unstructured real-world environments?

Learning what Success Means can we learn the goal with visual features? Finn, Abbeel , L. ‘16

Learning what Success Means Sermanet , Xu, L. ‘16

ingredients for success in learning: supervised learning: learning robotic skills: computation computation ~ data algorithms algorithms ? data

Fereshteh Sadeghi Aravind Rajeswaran Chelsea Finn Greg Kahn Announcement: New Conference Conference on Robotic Learning (CoRL) www.robot-learning.org Goal: bring together robotics & machine learning in a focused conference format Alex Conference: November 2017 Peter Pastor Krizhevsky Deirdre Quillen Trevor Darrell Pieter Abbeel Papers deadline: late June 2017 Steering committee: Ken Goldberg (UC Berkeley), Sergey Levine (UC Berkeley), Vincent Vanhoucke (Google), Abhinav Gupta (CMU), Stefan Schaal (USC, MPI), Michael I. Jordan (UC Berkeley), Raia Hadsell (DeepMind), Dieter Fox (UW), Joelle Pineau (McGill), J. Andrew Bagnell (CMU), Aude Billard (EPFL), Stefanie Tellex (Brown), Minoru Asada (Osaka), Wolfram Burgard (Freiburg), Pieter Abbeel (UC Berkeley) Mrinal Shane Gu Ethan Holly Tim Lillicrap Kalakrishnan Yevgen Chebotar Ali Yahya Adrian Li