Suboptimal feedback control of PDEs by solving Hamilton-Jacobi - PowerPoint PPT Presentation

Suboptimal feedback control of PDEs by solving Hamilton-Jacobi Bellman equations on sparse grids Jochen Garcke joint work with Axel Kr¨ oner, INRIA Saclay and CMAP, Ecole Polytechnique Ilja Kalmykov, Universit¨ at Bonn (who is looking for a PhD position) Department Numerical Data-Driven Prediction Institute for Numerical Simulation

Optimal control Sparse Grids Numerical Results Higher Order Methods Outline 1 Optimal control 2 Sparse Grids 3 Numerical Results 4 Higher Order Methods

Optimal control Sparse Grids Numerical Results Higher Order Methods Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Optimal control Sparse Grids Numerical Results Higher Order Methods Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Optimal control Sparse Grids Numerical Results Higher Order Methods Optimal Feedback Control optimal feedback control of a dynamical system � T  u ∈U ad J ( u ) = min l ( y ( t ) , u ( t )) d t , s.t.     0 y ( t ) = f ( y ( t ) , u ( t )) , ˙ t > 0     y (0) = y 0 state y ( t ) ∈ ❘ d , initial state y 0 ∈ ❘ d control u ( t ) ∈ U m ⊂ ❘ m (often called action) Lipschitz continuous dynamics f : ❘ d × ❘ m → ❘ d running cost with polynomial growth l : ❘ d × ❘ m → ❘ set of admissible controls U ad = { u ∈ L 2 ([0 , T ]; U m ) | U m ⊂ ❘ m compact } aim: feedback law u ∗ = K ( t , y ∗ ( t ))

Optimal control Sparse Grids Numerical Results Higher Order Methods Semi-Lagrangian scheme from DPP one derives semi-Lagrangian scheme � � v k ( x ) = min v k +1 ( y x (∆ t )) + ∆ t · l ( x , u ) (SL) u ∈ U for k = N − 1 , ..., 0 with v N ( x ) = 0, time step ∆ t > 0, x ∈ ❘ d y x (∆ t ) state obtained by time discretization scheme from x evaluation of the right hand side in (SL), either: (potentially expensive) comparison over a finite set U finite ⊂ U in linear quadratic case, i.e. A ∈ ❘ d × d , B ∈ ❘ d × m f ( x , u ) = Ax + Bu , l ( x , u ) = 1 � x T Mx + u T Ru � M ∈ ❘ d × d , R ∈ ❘ m × m , 2 the optional feedback control is given by (needing approx. for ∇ v ) − R − 1 B T ∇ v ( y ∗ ( t ) , t ) � � u ∗ ( t ) = P U

Optimal control Sparse Grids Numerical Results Higher Order Methods Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Optimal control Sparse Grids Numerical Results Higher Order Methods Application to a PDE Control Problem wave equation (following Kr¨ oner, Kunisch, Zidani (2015))  y tt − c ∆ˆ ˆ y = Bu in (0 , T ) × Ω ,   y (0) = ˆ ˆ y 0 , y t (0) = ˆ ˆ in Ω , y 1  y = 0 ˆ on (0 , T ) × ∂ Ω  y 0 ∈ H 1 (Ω) and ˆ y 1 ∈ L 2 (Ω) initial state and velocity ˆ control operator B := (sin( π x ) , . . . , sin( m π y )) (can be generalized) formulate as first order system in time with y 1 = y , y 2 = ˙ y y 1 t = y 2 , t − ∆ y 1 = Bu , y 2 y 1 (0) = ˆ y 2 (0) = ˆ y 0 , y 1 , which we can write as y t + A y = (0 , Bu ) T , y (0) = y 0 � � 0 Id y 1 ) T ∈ Y 1 , and A = with y = ( y 1 , y 2 ), y 0 = (ˆ y 0 , ˆ − c ∆ 0

Optimal control Sparse Grids Numerical Results Higher Order Methods Semi-Discrete Formulation of Control Problem semi-discrete formulation of control problem by method of lines for a given basis b := ( ϕ 1 , . . . , ϕ d ) , d ∈ N , with ϕ i : Ω → ❘ we define A := (( ∇ ϕ i ( x ) , ∇ ϕ j ( x )) i , j =1 ,..., d ) (stiffness matrix) M := (( ϕ i ( x ) , ϕ j ( x )) i , j =1 ,..., d ) (mass matrix) in our numerical examples we later choose ϕ i ( x ) := sin( i π x ) , i = 1 , . . . , d , and obtain A = diag((1 / 2( i π ) 2 ) i =1 ,..., d ) , M = diag((1 / 2) i =1 ,..., d )

Optimal control Sparse Grids Numerical Results Higher Order Methods Resulting Semi-Discrete PDE Control Problem this now fits into control formulation from beginning � T  u ∈U ad J ( u ) = min l ( y ( t ) , u ( t )) d t , s.t.     0 y ( t ) = f ( y ( t ) , u ( t )) , ˙ t > 0    y (0) = y 0  for initial value y 0 ∈ ❘ 2 d and t ∈ [0 , T ). dynamics f ( x , u ) = A x + B u with running cost l ( x , u ) := β x x T 1 Mx 1 + β u u T u if value function is differentiable in x , t optimal feedback control � − 1 � u ∗ ( y ∗ ( t ) , t ) = P U B T ∇ x v ( y ∗ ( t ) , t ) β u where P U projection on set of admissible controls Hamilton-Jacobi Bellman equation with dimension 2 d

Optimal control Sparse Grids Numerical Results Higher Order Methods Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete problem HJB-equation sparse grids feedback operator

Optimal control Sparse Grids Numerical Results Higher Order Methods Interpolation with Hierarchical Basis φ 3 , 4 φ 3 , 2 φ 3 , 6 φ 3 , 1 φ 3 , 3 φ 3 , 5 φ 3 , 7 nodal basis V 1 ⊂ V 2 ⊂ V 3 φ 1 , 1 φ 2 , 1 φ 2 , 3 φ 3 , 1 φ 3 , 3 φ 3 , 5 φ 3 , 7 � W 2 � V 1 hierarchical basis V 3 = W 3

Optimal control Sparse Grids Numerical Results Higher Order Methods Hier. Basis Functions in Higher Dimensions d -dimensional piecewise d -linear functions d � φ l , j ( x ) := φ l t , j t ( x t ) t =1 hierarchical difference space W l ( e t is t -th unit vector) d � W l := V l \ V l − e t , t =1 hier. diff. space represented by W l = span { φ l , j | j ∈ B l } with j t = 1 , . . . , 2 l t − 1 , � � � j t odd , t = 1 , . . . , d , if l t > 1 , j ∈ N d � B l := . � j t = 0 , 1 , 2 , t = 1 , . . . , d , if l t = 1 � full grid space in hierarchical basis � V s n := W l | l | ∞ ≤ n

Optimal control Sparse Grids Numerical Results Higher Order Methods Hierarchical Subspaces W l W 1 , 1 W 1 , 2 W 1 , 3 W 1 , 4 W 2 , 1 W 2 , 2 W 2 , 3 W 2 , 4 W 3 , 1 W 3 , 2 W 3 , 3 W 3 , 4 W 4 , 1 W 4 , 2 W 4 , 3 W 4 , 4

Optimal control Sparse Grids Numerical Results Higher Order Methods Sparse Grids we define the sparse grid function space V s n ⊂ V n as � V s n := W l | l | 1 ≤ n + d − 1 every f ∈ V s n can now be represented as � � f s n ( x ) = α l , j φ l , j ( x ) j ∈ B l | l | 1 ≤ n + d − 1 approximation property in H 2 mix || f − f s n || 2 = O ( h 2 n log( h − 1 n ) d − 1 ) sparse grid needs O ( h − 1 n (log( h − 1 n )) d − 1 ) points

Optimal control Sparse Grids Numerical Results Higher Order Methods Sparse Grids in two and three dimensions

Optimal control Sparse Grids Numerical Results Higher Order Methods Problems with Sparse Grids: Monotonicity interpolation of peaked Gaussian fct. with sparse grid n = 2 − 100( x 1 − 0 . 5) 2 � − 100( x 2 − 0 . 5) 2 � � � f ( x 1 , x 2 ) := exp ∗ exp 0-level set of interpolant is pink in the right picture sparse grid interpolation does not preserve positivity

Optimal control Sparse Grids Numerical Results Higher Order Methods Spatially Adaptive Sparse Grids to approximate functions which either do not fulfil smoothness condition at all or strongly vary due to finite but locally large derivatives adaptive refinement may be used start with a regular grid of level 2 (left) to populate index set I refine one grid point by creating all children (middle) to keep grid consistent, missing parents are created (right) usually hierarchical surplus α l , j is used as refinement indicator

Optimal control Sparse Grids Numerical Results Higher Order Methods Basic Sparse Grid SL Scheme evaluate for x ∈ Q I , ∆ t > 0, K = T / ∆ t , k = K − 1 , . . . , 0,  � � ∆ tl ( x , u ) + v k +1 ( y x (∆ t )) v k ( x ) = min ,  u ∈ U v k ( x ) = 0  y x (∆ t ) state obtained by time discretization scheme from x Algorithm 1: Adaptive SL-SG scheme Data: refinement constant ε , coarsening constant η Result: sequence of adaptive sparse grid solutions v k ∈ V I ( k ) initialize I ( K ) for k = K − 1 , . . . , 0 do ⊲ iterate in time with ∆ t = T / K initialize I ( k − 1) with I ( k ) � � adaptively interpolate min u ∈ U v k ( y x (∆ t )) + ∆ tl ( x , u ) ⊲ compute v k − 1 coarsen v k − 1 ∈ V I ( k − 1) see Bokanowski, G., Griebel, and Klompmaker (2013)

Optimal control Sparse Grids Numerical Results Higher Order Methods Further Discretization Aspects to determine minimizing control within the SL-scheme we use minimization by comparison over finite subset U σ ⊂ U or gradient of the value function � − 1 � u n ( x , s ) = P U B T ∇ h v n +1 ( x , s ) β u per finite differences i := v n +1 ( x + h · e i , s ) − v n +1 ( x − h · e i , s ) ∇ h v n +1 ( x , s ) � � 2 h for computation of y x (∆ t ) we use second order Heun scheme in our experiments we focus on discretization error in space, while using time resolution which is “good enough” reference solution v r in 2D computed with a higher order finite difference code on a uniform mesh by an ENO scheme compute reference trajectories y r in state space and u r in control space using a Riccati approach