Discount Factor as a Regularizer in RL Ron Amit , Ron Meir - PowerPoint PPT Presentation

ICML 2020 Discount Factor as a Regularizer in RL Ron Amit , Ron Meir (Technion) , Kamil Ciosek (MSR) Microsoft Research, Cambridge UK

RL problems objectives • The expected 𝛿 𝑓 -discounted return (value function) Evaluation discount factor • Policy Evaluation • Policy Optimization How can we improve perfomance in the limited data regime?

Discount regularization • Discount regularization : “ guidance discount factor ” (Jiang ’ 15 ) Algorithm hyperparameter • Theoretical analysis: • Petrik and Scherrer ’ 09 – Approx. DP • Jiang ’ 15 – model based Better performance for limited data • Regularization effect: • ↑ Bias 𝑊 𝛿 − 𝑊 𝛿 𝑓 • ↓ Variance ෠ 𝑊 − 𝑊 𝛿 • Our work: • In TD learning, discount regularization == explicit added regularizer • When is discount regularization effective?

Temporal Difference (TD) Learning • Policy evaluation with value-function model • Batch TD(0) Discount factor hyperparameter Aim to minimize

Equivalent Form • Equivalent update steps Discount regularization (using 𝜹 < 𝜹 𝒇 ) ⇕ ⇕ Using 𝜹 𝒇 + regularization term Regularization term gradient Similar Equivalence • (expected) SARSA • LSTD Activation regularization

The Equivalent Regularizer • Activation regularization 𝑀 2 regularization • Tabular case: Discount regularization is sensitive to the empirical distribution

Tabular Experiments 4x4 GridWorld • Policy evaluation, 𝜌 𝑏 𝑡 uniform. 𝜌 ( 𝛿 𝑓 = 0.99 ) • Goal : find ෠ 𝑊 that estimates 𝑊 𝛿 𝑓 • Loss measures: 2 = σ 𝑡∈𝑇 ෠ 𝜌 2 ෠ • 𝑴 𝟑 loss: 𝜌 In each MDP Instance: 𝑊 − 𝑊 𝑊 − 𝑊 𝛿 𝑓 𝛿 𝑓 • 2 Draw 𝔽𝑆 𝑡 • Draw 𝑄(. |𝑡, 𝑏) • Ranking Loss: −Kandal`s_Tau ෠ 𝜌 𝑊, 𝑊 𝛿 𝑓 ( ~ number of order switches between state ranks) • Average over 1000 MDP instances • Data: trajectories of 50 time-steps

Discount Regularization 𝑀 2 Regularization TD(0) Results 𝑀 2 loss Ranking Loss (𝛿 𝑓 = 0.99)

Effect of the Empirical Distribution • Equivalent regularizer: • Tuples (𝑡, 𝑡 ′ , 𝑠) generation: 𝑡~𝑕(𝑡) , 𝑡 ′ ~𝑄 𝜌 𝑡 ′ 𝑡 , 𝑠~𝑆 𝜌 (𝑡) • For each MDP - draw distribution 𝑕(𝑡) at 𝑒 𝑈𝑊 from uniform 𝑴 𝟑 regularization Discount regularization Non-uniform Non-uniform Uniform Uniform (𝛿 𝑓 = 0.99)

Effect of the Mixing Time • Lower mixing time (slow mixing) → Higher estimation variance → more regularization is needed 𝑴 𝟑 regularization Discount regularization Slow mixing Slow mixing Fast mixing Fast mixing (𝛿 𝑓 = 0.99) (LSTD, 2 trajectories)

Policy Optimization 𝜌 − 𝑊 𝜌 ∗ Goal : min 𝑊 𝛿 𝑓 𝛿 𝑓 𝜌 1 Policy-iteration: • For episodes: • Get data • ෠ 𝑅 ← Policy evaluation (e.g, SARSA) • Improvement step (e.g., 𝜁 -epsilon-greedy) Activation regularization term:

Deep RL Experiments • Actor-critic algorithms: DDPG (Lillicrap ‘ 15), TD3 (Fujimoto ‘ 18) • Mujoco continuous control (Todorov ‘ 12) • Goal: undiscounted sum of rewards (𝛿 𝑓 = 1) • Limited number of time-steps (2e5 or less) • Tested cases: • Discount regularization (and no 𝑀 2 ) • 𝑀 2 regularization (and 𝛿 = 0.999 )

Discount Regularization L2 Regularization Discount Regularization L2 Regularization 2e5 steps 2e5 steps Fewer steps Fewer steps 𝛿 = 0.99 2.5e2 steps HalfCheetah-v2 𝛿 = 0.99 𝛿 = 0.8 1e5 steps Ant-v2 𝛿 = 0.99 5 5e4 steps Hopper-v2

Conclusions • Discount regularization in TD is equivalent to adding a regularizer term • Regularization effectiveness is closely related to the data distribution and mixing rate. • Generalization in deep RL is strongly affected by regularization • Future work – theory needed Thanks for listening