10703 Deep Reinforcement Learning Reinforcement Learning in Humans - PowerPoint PPT Presentation

Feb 09, 2024 •237 likes •448 views

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018 Outline RL in primates RL in humans Error signals and

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018
Outline • RL in primates • RL in humans • Error signals and predictive coding Tom Mitchell, October 2018
Reward based learning in primates Tom Mitchell, October 2018
Dopamine As Reward Signal t [Schultz et al., Science , 1997] Tom Mitchell, October 2018
Dopamine As Reward Signal t [Schultz et al., Science , 1997] Tom Mitchell, October 2018
Dopamine As Reward Signal t [Schultz et al., Science , 1997] error r γ V(s ) V(s ) = + − t t 1 t + � 6 Tom Mitchell, October 2018
Reward based learning in humans Tom Mitchell, October 2018
RL Models for Human Learning [Seymore et al., Nature 2004] Tom Mitchell, October 2018
[Seymore et al., Nature 2004] � 9 Tom Mitchell, October 2018
One Theory of RL in the Brain from [Nieuwenhuis et al.] • Basal ganglia monitor events, predict future rewards • When prediction revised upward (downward), causes increase (decrease) in activity of midbrain dopaminergic neurons, influencing ACC • This dopamine-based activation somehow results in revising the reward prediction function. Possibly through direct influence on Basal ganglia, and via prefrontal cortex Tom Mitchell, October 2018
Tom Mitchell, October 2018
Tom Mitchell, October 2018
Neuron Level Learning Mechanisms • Hebbian learning – fire together � wire together • Spike Timing Dependent Plasticity   (STDP) – if incoming neuron fires before outgoing   then strengthen connection – if incoming neuron fires after outgoing   then weaken connection • Reward modulated STDP – less understood – in some neurons, appears STDP occurs only if neuromodulator (e.g., dopamine) activity follows firing within time up to 10 sec Tom Mitchell, October 2018
Tom Mitchell, October 2018
Summary: Temporal Difference ML Model   Predicts Dopaminergic Neuron Acitivity during Learning • Evidence now of neural reward signals from – Direct neural recordings in monkeys – fMRI in humans (1 mm spatial resolution) – EEG in humans (1-10 msec temporal resolution) • Dopaminergic responses encode Temporal Difference error • Some differences, and efforts to refine the model – How/where is the value function encoded in the brain? – Study timing (e.g., basal ganglia learns faster than PFC ?) – Role of prior knowledge, rehearsal of experience, multi-task learning? Tom Mitchell, October 2018
Predictive Coding Tom Mitchell, October 2018
[Rao & Ballard, Nature, 1999] Tom Mitchell, October 2018
[Rao & Ballard, 1999] Tom Mitchell, October 2018
Tom Mitchell, October 2018
Tom Mitchell, October 2018

Recommend

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018 Recommended readings: Used Materials Much of the material and slides for this lecture were borrowed from Katerina Fragkiadaki, and Ruslan

1.25k views • 49 slides

10703 Deep Reinforcement Learning Policy Gradient Methods Part 3 Tom Mitchell October 8, 2018

10703 Deep Reinforcement Learning Policy Gradient Methods Part 3 Tom Mitchell October 8, 2018 Recommended readings: next slide. (not covered in Barto & Sutton) Used Materials Disclaimer : Much of the material and slides for this

781 views • 26 slides

10703 Deep Reinforcement Learning Exploration vs. Exploitation Tom Mitchell October 22, 2018

10703 Deep Reinforcement Learning Exploration vs. Exploitation Tom Mitchell October 22, 2018 Reading: Barto & Sutton, Chapter 2 Used Materials Some of the material and slides for this lecture were taken from Chapter 2 of Barto &

785 views • 37 slides

10703 Deep Reinforcement Learning Solving known MDPs Tom Mitchell September 10, 2018 Many

10703 Deep Reinforcement Learning Solving known MDPs Tom Mitchell September 10, 2018 Many slides borrowed from Katerina Fragkiadaki Russ Salakhutdinov Markov Decision Process (MDP) A Markov Decision Process is a tuple is a

1.05k views • 36 slides

Asynchronous RL CMU 10703 Katerina Fragkiadaki Non-stationary data problem for Deep RL

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Asynchronous RL CMU 10703 Katerina Fragkiadaki Non-stationary data problem for Deep RL Stability of training neural networks requires the gradient updates

933 views • 69 slides

10703 Deep Reinforcement Learning Tom Mitchell September 5, 2018 Solving known MDPs Many slides

9/7/18 10703 Deep Reinforcement Learning Tom Mitchell September 5, 2018 Solving known MDPs Many slides borrowed from Katerina Fragkiadaki Russ Salakhutdinov Markov Decision Process (MDP) A Markov Decision Process is a tuple is a

501 views • 14 slides

10703 Deep Reinforcement Learning Policy Gradient Methods Tom Mitchell October 1, 2018 Reading:

10703 Deep Reinforcement Learning Policy Gradient Methods Tom Mitchell October 1, 2018 Reading: Barto & Sutton, Chapter 13 Used Materials Much of the material and slides for this lecture were taken from Chapter 13 of Barto & Sutton

586 views • 35 slides

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides

Exploration and Function Approximation CMU 10703 Katerina Fragkiadaki This lecture Exploration

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Exploration and Function Approximation CMU 10703 Katerina Fragkiadaki This lecture Exploration in Large Continuous State Spaces Exploration-Exploitation

1.21k views • 86 slides

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides

Natural Policy Gradients, TRPO, PPO CMU 10703 Katerina Fragkiadaki Part of the slides adapted

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Natural Policy Gradients, TRPO, PPO CMU 10703 Katerina Fragkiadaki Part of the slides adapted from John Shulman and Joshua Achiam Stochastic policies

758 views • 44 slides

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides

Used Materials Acknowledgement : Much of the material and slides for this lecture were

10703 Deep Reinforcement Learning Tom Mitchell Machine Learning Department September 12, 2018 Monte Carlo Methods Used Materials Acknowledgement : Much of the material and slides for this lecture were borrowed from Ruslan Salakhutdinov, who

469 views • 32 slides

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

938 views • 63 slides