cs 285
play

CS 285 Instructor: Sergey Levine UC Berkeley Recap: whats the - PowerPoint PPT Presentation

Jan 04, 2024 •480 likes •790 views

Exploration (Part 2) CS 285 Instructor: Sergey Levine UC Berkeley Recap: whats the problem? this is easy (mostly) this is impossible Why? Unsupervised learning of diverse behaviors What if we want to recover diverse behavior without any

  1. Exploration (Part 2) CS 285 Instructor: Sergey Levine UC Berkeley

  2. Recap: what’s the problem? this is easy (mostly) this is impossible Why?

  3. Unsupervised learning of diverse behaviors What if we want to recover diverse behavior without any reward function at all ? Why? ➢ Learn skills without supervision, then use them to accomplish goals ➢ Learn sub-skills to use with hierarchical reinforcement learning ➢ Explore the space of possible behaviors

  4. An Example Scenario How can you prepare for an unknown future goal? training time: unsupervised

  5. In this lecture… ➢ Definitions & concepts from information theory ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  6. In this lecture… ➢ Definitions & concepts from information theory ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  7. Some useful identities

  8. Some useful identities

  9. Information theoretic quantities in RL quantifies coverage can be viewed as quantifying “control authority” in an information -theoretic way

  10. In this lecture… ➢ Definitions & concepts from information theory ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  11. An Example Scenario How can you prepare for an unknown future goal? training time: unsupervised

  12. Learn without any rewards at all (but there are many other choices) Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 12 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  13. Learn without any rewards at all Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 13 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  14. Learn without any rewards at all Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 14 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  15. How do we get diverse goals? Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 15 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  16. How do we get diverse goals? Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 16 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  17. How do we get diverse goals? goals get higher entropy due to Skew-Fit goal final state Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 17 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  18. How do we get diverse goals? Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 18 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  19. Reinforcement learning with imagined goals imagined goal RL episode Nair*, Pong*, Bahl, Dalal, Lin, L. Visual Reinforcement Learning with Imagined Goals . ’18 19 Dalal*, Pong*, Lin*, Nair, Bahl, Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. ‘19

  20. In this lecture… ➢ Definitions & concepts from information theory ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  21. Aside: exploration with intrinsic motivation

  22. Can we use this for state marginal matching? Lee*, Eysenbach*, Parisotto*, Xing, Levine, Salakhutdinov. Efficient Exploration via State Marginal Matching See also: Hazan, Kakade, Singh, Van Soest. Provably Efficient Maximum Entropy Exploration

  23. State marginal matching for exploration much better coverage! MaxEnt on actions variants of SMM Lee*, Eysenbach*, Parisotto*, Xing, Levine, Salakhutdinov. Efficient Exploration via State Marginal Matching See also: Hazan, Kakade, Singh, Van Soest. Provably Efficient Maximum Entropy Exploration

  24. In this lecture… ➢ Definitions & concepts from information theory ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  25. Is state entropy really a good objective? more or less the same thing See also: Hazan, Kakade, Singh, Van Soest. Provably Efficient Maximum Entropy Exploration 25 Gupta, Eysenbach, Finn, Levine. Unsupervised Meta-Learning for Reinforcement Learning

  26. In this lecture… ➢ Definitions & concepts from information theory ➢ A distribution-matching formulation of reinforcement learning ➢ Learning without a reward function by reaching goals ➢ A state distribution-matching formulation of reinforcement learning ➢ Is coverage of valid states a good exploration objective? ➢ Beyond state covering: covering the space of skills

  27. Learning diverse skills task index Reaching diverse goals is not the same as performing diverse tasks not all behaviors can be captured by goal-reaching Intuition: different skills should visit different state-space regions Eysenbach, Gupta, Ibarz, Levine. Diversity is All You Need.

  28. Diversity-promoting reward function Environment Action State Discriminator(D) Policy(Agent) Skill (z) Predict Skill Eysenbach, Gupta, Ibarz, Levine. Diversity is All You Need.

  29. Examples of learned tasks Cheetah Ant Mountain car Eysenbach, Gupta, Ibarz, Levine. Diversity is All You Need.

  30. A connection to mutual information Eysenbach, Gupta, Ibarz, Levine. Diversity is All You Need. See also: Gregor et al. Variational Intrinsic Control. 2016

Recommend

More recommend