Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement - - PowerPoint PPT Presentation

Module 9 LAO*

CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo

CS886 (c) 2013 Pascal Poupart

2

Large State Space

• Value Iteration, Policy Iteration and Linear

Programming

– Complexity at least quadratic in |𝑇|

• Problem: |𝑇| may be very large

– Queuing problems: infinite state space – Factored problems: exponentially many states

CS886 (c) 2013 Pascal Poupart

3

Mitigate Size of State Space

• Two ideas:
• Exploit initial state

– Not all states are reachable

• Exploit heuristic ℎ

– approximation of optimal value function – usually an upper bound ℎ 𝑡 ≥ 𝑊∗ 𝑡 ∀𝑡

CS886 (c) 2013 Pascal Poupart

4

State Space

State space |𝑇|

𝑡0

Reachable states States reachable by 𝜌∗

CS886 (c) 2013 Pascal Poupart

5

State Space

State space |𝑇|

𝑡0

Reachable states States reachable by 𝜌∗

CS886 (c) 2013 Pascal Poupart

6

LAO* Algorithm

• Related to

– A*: path heuristic search – AO*: tree heuristic search – LAO*: cyclic graph heuristic search

• LAO* alternates between

– State space expansion – Policy optimization

• value iteration, policy iteration, linear programming

CS886 (c) 2013 Pascal Poupart

7

Terminology

• 𝑇: state space
• 𝑇𝐹 ⊆ 𝑇: envelope

– Growing set of states

• 𝑇𝑈 ⊆ 𝑇𝐹: terminal states

– States whose children are not in the envelope

• 𝑇𝑡0

𝜌 ⊆ 𝑇𝐹: states reachable from 𝑡0 by following 𝜌

• ℎ(𝑡): heuristic such that ℎ 𝑡 ≥ 𝑊∗ 𝑡 ∀𝑡

– E.g., ℎ 𝑡 = max

𝑡,𝑏 𝑆(𝑡, 𝑏)/(1 − 𝛿)

CS886 (c) 2013 Pascal Poupart

8

LAO* Algorithm

LAO*(MDP, heuristic ℎ)

𝑇𝐹 ← {𝑡0}, 𝑇𝑈 ← {𝑡0} Repeat Let 𝑆𝐹 𝑡, 𝑏 = ℎ(𝑡) 𝑡 ∈ 𝑇𝑈 𝑆(𝑡, 𝑏)

• therwise

Let 𝑈𝐹(𝑡′|𝑡, 𝑏) = 𝑡 ∈ 𝑇𝑈 Pr (𝑡′|𝑡, 𝑏)

• therwise

Find optimal policy 𝜌 for 𝑇𝐹, 𝑆𝐹, 𝑈𝐹 Find reachable states 𝑇𝑡0

𝜌

Select reachable terminal states s1, … , sk ⊆ 𝑇𝑡0

𝜌 ∩ 𝑇𝑈

𝑇𝑈 ← (𝑇𝑈 ∖ 𝑡1, … , 𝑡𝑙 ) ∪ (𝑑ℎ𝑗𝑚𝑒𝑠𝑓𝑜 𝑡1, … , 𝑡𝑙 ∖ 𝑇𝐹) 𝑇𝐹 ← 𝑇𝐹 ∪ 𝑑ℎ𝑗𝑚𝑒𝑠𝑓𝑜( 𝑡1, … , 𝑡𝑙 ) Until 𝑇𝑡0

𝜌 ∩ 𝑇𝑈 is empty

CS886 (c) 2013 Pascal Poupart

9

Efficiency

Efficiency influenced by

• 1. Choice of terminal states to add to envelope
• 2. Algorithm to find optimal policy

– Can use value iteration, policy iteration, modified policy iteration, linear programming – Key: reuse previous computation

• E.g., start with previous policy or value function at each

iteration

CS886 (c) 2013 Pascal Poupart

10

Convergence

• Theorem: LAO* converges to the optimal policy
• Proof:

– Fact: At each iteration, the value function 𝑊 is an upper bound on 𝑊∗ due to the heuristic function ℎ – Proof by contradiction: suppose the algorithm stops, but 𝜌 is not optimal.

• Since the algorithm stopped, all states reachable by 𝜌 are in

𝑇𝐹 ∖ 𝑇𝑈

• Hence, the value function 𝑊 is the value of 𝜌 and since 𝜌 is

suboptimal then 𝑊 < 𝑊∗, which contradicts the fact that 𝑊 is an upper bound on 𝑊∗

CS886 (c) 2013 Pascal Poupart

11

Summary

• LAO*

– Extension of basic solution algorithms (value iteration, policy iteration, linear programming) – Exploit initial state and heuristic function – Gradually grow an envelope of states – Complexity depends on # of reachable states instead

• f size of state space