Optimal Control, LQR, Trajectory Optimization Lecture 13 What will - PowerPoint PPT Presentation

Optimal Control, LQR, Trajectory Optimization Lecture 13

What will you take home today? Intro Optimal Control Principle of Optimality Bellman Equation Deriving LQR Trajectory Optimization Paper

Optimal Control and Reinforcement Learning from a unified point of view Optimal Control Problem

Principle of Optimality – Example: Graph Search problem

Quiz

Forward Search

Backward Search

Principle of Optimality

Problem setup System Dynamics Cost function

Goal

Formalize Cost-to-Go / Value function

Optimal Value function = V with lowest cost

Deriving the Bellman Equation

Comparing Optimal Bellman and Value Function

Infinite time horizon, deterministic system

Infinite time horizon, deterministic system

Infinite time horizon, deterministic system

Finite Horizon, Stochastic system Cost function Stochastic System Dynamics

Finite Horizon, Stochastic system Value function Optimal Value function Optimal Policy

Finite Horizon, Stochastic system Bellman Equation Optimal Bellman Equation

Infinite Horizon, Stochastic system - Combining formulation from infinite horizon - discrete system with stochastic system derivation - See lecture notes on webpage

Continuous time systems Hamilton-Jacobi-Bellman Equation

How do you solve these equations?

Sequential Quadratic Programming

Sequential Quadratic Programming

Example – Newton-Raphson Method

Sequential Linear Quadratic Programming

SLQ Algorithm

Iterative Linear Quadratic Controller: ILQC

ILQC – Linearizing Dynamics

ILQC - Quadratizing the cost

Deriving the Value function and Bellman Equation

Incremental Updates

Linear Dynamical Systems, Quadratic cost – L inear Q uadratic R egulator (LQR)

Linear Dynamical Systems, Quadratic cost – L inear Q uadratic R egulator (LQR)

Trajectory Optimization