Lecture 5: Value Function Approximation Emma Brunskill CS234 - PowerPoint PPT Presentation

Lecture 5: Value Function Approximation Emma Brunskill CS234 Reinforcement Learning. Winter 2018 The value function approximation structure for today closely follows much of David Silver’s Lecture 6. For additional reading please see SB 2018 Sections 9.3, 9.6-9.7. The deep learning slides come almost exclusively from Ruslan Salakhutdinov’s class, and Hugo Larochelle’s class (and with thanks to Zico Kolter also for slide inspiration). The slides in my standard style format in the deep learning section are my own. Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 1 / 48

Important Information About Homework 2 Homework 2 will now be due on Saturday February 10 (instead of February 7) We are making this change to try to give some background on deep learning, give people enough time to do homework 2, and still give people time to study for the midterm on February 14 We will release the homework this week You will be able to start on some aspects of the homework this week, but we will be covering DQN which is the largest part, on Monday We will also be providing optional tutorial sessions on tensorflow Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 2 / 48

Table of Contents Introduction 1 VFA for Prediction 2 Control using Value Function Approximation 3 Deep Learning 4 Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 3 / 48

Class Structure Last time: Control (making decisions) without a model of how the world works This time: Value function approximation and deep learning Next time: Deep reinforcement learning Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 4 / 48

Last time: Model-Free Control Last time: how to learn a good policy from experience So far, have been assuming we can represent the value function or state-action value function as a vector Tabular representation Many real world problems have enormous state and/or action spaces Tabular representation is insu ffi cient Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 5 / 48

Recall: Reinforcement Learning Involves Optimization Delayed consequences Exploration Generalization Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 6 / 48

Today: Focus on Generalization Optimization Delayed consequences Exploration Generalization Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 7 / 48

Table of Contents Introduction 1 VFA for Prediction 2 Control using Value Function Approximation 3 Deep Learning 4 Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 8 / 48

Value Function Approximation (VFA) Represent a (state-action/state) value function with a parameterized function instead of a table Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 9 / 48

Motivation for VFA Don’t want to have to explicitly store or learn for every single state a Dynamics or reward model Value State-action value Policy Want more compact representation that generalizes across state or states and actions Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 10 / 48

Benefits of Generalization Reduce memory needed to store ( P , R )/ V / Q / ⇡ Reduce computation needed to compute ( P , R )/ V / Q / ⇡ Reduce experience needed to find a good P , R / V / Q / ⇡ Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 11 / 48

Value Function Approximation (VFA) Represent a (state-action/state) value function with a parameterized function instead of a table Which function approximator? Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 12 / 48

Function Approximators Many possible function approximators including Linear combinations of features Neural networks Decision trees Nearest neighbors Fourier / wavelet bases In this class we will focus on function approximators that are di ff erentiable (Why?) Two very popular classes of di ff erentiable function approximators Linear feature representations (Today) Neural networks (Today and next lecture) Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 13 / 48

Review: Gradient Descent Consider a function J ( w ) that is a di ff erentiable function of a parameter vector w Goal is to find parameter w that minimizes J The gradient of J ( w ) is Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 14 / 48

Table of Contents Introduction 1 VFA for Prediction 2 Control using Value Function Approximation 3 Deep Learning 4 Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 15 / 48

Value Function Approximation for Policy Evaluation with an Oracle First consider if could query any state s and an oracle would return the true value for v π ( s ) The objective was to find the best approximate representation of v π given a particular parameterized function Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 16 / 48

Stochastic Gradient Descent Goal: Find the parameter vector w that minimizes the loss between a true value function v π ( s ) and its approximation ˆ v as represented with a particular function class parameterized by w . Generally use mean squared error and define the loss as v ( S , w )) 2 ] J ( w ) = π [( v π ( S ) � ˆ (1) Can use gradient descent to find a local minimum � 1 = 2 ↵ 5 w J ( w ) (2) ∆ w Stochastic gradient descent (SGD) samples the gradient: Expected SGD is the same as the full gradient update Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 17 / 48

VFA Prediction Without An Oracle Don’t actually have access to an oracle to tell true v π ( S ) for any state s Now consider how to do value function approximation for prediction / evaluation / policy evaluation without a model Note: policy evaluation without a model is sometimes also called passive reinforcement learning with value function approximation ”passive” because not trying to learn the optimal decision policy Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 18 / 48

Model Free VFA Prediction / Policy Evaluation Recall model-free policy evaluation (Lecture 3) Following a fixed policy ⇡ (or had access to prior data) Goal is to estimate V π and/or Q π Maintained a look up table to store estimates V π and/or Q π Updated these estimates after each episode (Monte Carlo methods) or after each step (TD methods) Now: in value function approximation, change the estimate update step to include fitting the function approximator Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 19 / 48

Feature Vectors Use a feature vector to represent a state 0 1 x 1 ( s ) x 2 ( s ) B C x ( s ) = (3) B C . . . @ A x n ( s ) Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 20 / 48

Linear Value Function Approximation for Prediction With An Oracle Represent a value function (or state-action value function) for a particular policy with a weighted linear combination of features n X x j ( S ) w j = x ( S ) T ( w ) v ( S , w ) = ˆ j =1 Objective function is v ( S , w )) 2 ] J ( w ) = π [( v π ( S ) � ˆ Recall weight update is ∆ ( w ) = � 1 2 ↵ 5 w J ( w ) (4) Update is: Update = step-size ⇥ prediction error ⇥ feature value Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 21 / 48

Monte Carlo Value Function Approximation Return G t is an unbiased but noisy sample of the true expected return v π ( S t ) Therefore can reduce MC VFA to doing supervised learning on a set of (state,return) pairs: < S 1 , G 1 >, < S 2 , G 2 >, . . . , < S T , G T > Susbstituting G t ( S t ) for the true v π ( S t ) when fitting the function approximator Concretely when using linear VFA for policy evaluation ∆ w = ↵ ( G t � ˆ v ( S t , w )) 5 w ˆ v ( S t , w ) (5) = ↵ ( G t � ˆ v ( S t , w )) x ( S t ) (6) Note: G t may be a very noisy estimate of true return Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 22 / 48

MC Linear Value Function Approximation for Policy Evaluation 1: Initialize w = 0 , Returns ( s ) = 0 8 ( s , a ), k = 1 2: loop Sample k -th episode ( s k 1 , a k 1 , r k 1 , s k 2 , . . . , s k , L k ) given ⇡ 3: for t = 1 , . . . , L k do 4: if First visit to ( s ) in episode k then 5: Append P L k j = t r kj to Returns ( s t ) 6: Update weights 7: end if 8: end for 9: k = k + 1 10: 11: end loop Emma Brunskill (CS234 Reinforcement Learning. ) Lecture 5: Value Function Approximation Winter 2018 23 / 48