SOLAR: Deep Structured Representations for Model-Based Reinforcement - PowerPoint PPT Presentation

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google

Efficient reinforcement learning from images

Efficient reinforcement learning from images Model-free RL: 4 hours for image-based robotic task, 2 hours for block stacking from states https://sites.google.com/view/sac-and-applications

Efficient reinforcement learning from images Model-free RL: 20 hours for image-based robotic task, 2 hours for block stacking from states Model-based RL from images: relies on accurate forward prediction , which is difficult

Efficient reinforcement learning from images Model-free RL: 20 hours for image-based robotic task, 2 hours for block stacking from states Model-based RL from images: relies on accurate forward prediction , which is difficult Key idea: structured representation learning to enable accurate modeling with simple models; model-based method that does n o t use forward prediction

Preliminary: LQR-FLM Levine and Abbeel, “Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics”. NIPS 2014. Levine*, Finn*, Darrell, and Abbeel, “End-to-End Training of Deep Visuomotor Policies”. JMLR 2016. Chebotar*, Hausman*, Zhang*, Sukhatme, Schaal, and Levine, “Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning”. ICML 2017.

Preliminary: LQR-FLM LQR-FLM fits local models for policy improvement, not forward prediction Levine and Abbeel, “Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics”. NIPS 2014. Levine*, Finn*, Darrell, and Abbeel, “End-to-End Training of Deep Visuomotor Policies”. JMLR 2016. Chebotar*, Hausman*, Zhang*, Sukhatme, Schaal, and Levine, “Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning”. ICML 2017.

Preliminary: LQR-FLM LQR-FLM fits local models for policy improvement, not forward prediction LQR-FLM has worked on complex robotic systems from states Levine and Abbeel, “Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics”. NIPS 2014. Levine*, Finn*, Darrell, and Abbeel, “End-to-End Training of Deep Visuomotor Policies”. JMLR 2016. Chebotar*, Hausman*, Zhang*, Sukhatme, Schaal, and Levine, “Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning”. ICML 2017.

Preliminary: LQR-FLM LQR-FLM fits linear dynamics and quadratic cost models for policy improvement:

Preliminary: LQR-FLM LQR-FLM fits linear dynamics and quadratic cost models for policy improvement: This works well, even for complex systems, if the state is relatively simple, but this doesn’t work if the state is complex, such as images

Our method: SOLAR In this work, we enable LQR-FLM for images using structured representation learning

Real robot results Our method is more efficient than both prior model-free and model-based methods

Real robot results Our method is more efficient than both prior model-free and model-based methods Block stacking: we can transfer a representation and model to multiple initial arm positions

Real robot results Our method is more efficient than both prior model-free and model-based methods Mug pushing: We can solve this task from sparse reward using human key presses

Thank you Poster #34 Paper: https://arxiv.org/abs/1808.09105 Website: https://sites.google.com/view/icml19solar Blog post: https://bair.berkeley.edu/blog/2019/05/20/solar Code: https://github.com/sharadmv/parasol

SOLAR: Deep Structured Representations for Model-Based Reinforcement - PowerPoint PPT Presentation

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

CS 4803 / 7643: Deep Learning Topics: Structured representations with graph networks Zsolt

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Learning Structured Natural Language Representations for Semantic Parsing Jianpeng Cheng, Siva

Computing Sparse Representations in O(NlogN) time May 3, 2013 Tsung-Han Lin and H.T. Kung

Satellite-based solar resource data: Model validation statistics versus users uncertainty

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng

Domain-guided construction of semantic representations for model-based interpretation PARC |

3D Deep Learning on Geometric Forms Hao Su Many 3D representations are available Candidates:

3D Deep Learning on Geometric Forms Hao Su Many 3D representations are available Candidates:

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Advances in model-based predictions of decadal and seasonal solar activity Mausumi Dikpati ,

A STRUCTURED L IFE A STRUCTURED L IFE A STRUCTURED L IFE A STRUCTURED L IFE A STRUCTURED L IFE

Composing graphical models with neural networks for structured representations and fast inference

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Abstract rule representations in a Abstract rule representations in a bilinear model bilinear

Deep Learning for Geometry Processing 3D Representations View-Based and Volumetric CNNs 3D

Solar Car Racing and Model Solar Cars as a Teaching Tool for Science Gary Vliet, April 13, 2013

Investigating Model-Based Autonomy for Solar Probe Plus. Workshop on Spacecraft Flight Software.

The solar wind Gas dynamic model I 9/28/2016 2 Gas dynamic model II 9/28/2016 3 Gas dynamic

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional

A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief