Generalized finite differences for solving stochastic control - PowerPoint PPT Presentation

Generalized finite differences for solving stochastic control problems March 2005 F . Bonnans, INRIA Rocquencourt H. Zidani, ENSTA and INRIA Contributions from our Ph.D. students: E. Ottenwaelter and S. Maroso http://www-rocq.inria.fr/sydoco Generalized finite differencesfor solving stochastic control problems – p.1/37

Stochastic optimal control problem ( P x ) � ∞  ℓ ( y ( t ) , u ( t )) e − λt d t ; Min I E     0       �  d y ( t ) = f ( y ( t ) , u ( t ))d t + σ ( y ( t ) , u ( t ))d w ( t ) , y (0) = x,           u ( t ) ∈ U, t ∈ [0 , ∞ [ .  Generalized finite differencesfor solving stochastic control problems – p.2/37

Notation R n : state variable, • y ( t ) ∈ I Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, R n × I R m → I • ℓ : I R : distributed cost, Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, R n × I R m → I • ℓ : I R : distributed cost, R n × I R m → I R n : deterministic dynamics, • f : I Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, R n × I R m → I • ℓ : I R : distributed cost, R n × I R m → I R n : deterministic dynamics, • f : I R n × I R m → space of n × r matrices • σ ( · ) : I Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, R n × I R m → I • ℓ : I R : distributed cost, R n × I R m → I R n : deterministic dynamics, • f : I R n × I R m → space of n × r matrices • σ ( · ) : I • w :standard r dimensional Brownian motion. Generalized finite differencesfor solving stochastic control problems – p.3/37

Notation R n : state variable, • y ( t ) ∈ I R m : control variable, • u ( t ) ∈ I • λ ≥ 0 : discounting factor, R n × I R m → I • ℓ : I R : distributed cost, R n × I R m → I R n : deterministic dynamics, • f : I R n × I R m → space of n × r matrices • σ ( · ) : I • w :standard r dimensional Brownian motion. • Value function V of problem ( P x ) : finite and continuous Generalized finite differencesfor solving stochastic control problems – p.3/37

HJB equation λv ( x ) = u ∈ U { ℓ ( x, u ) + f ( x, u ) · v x ( x ) inf � n + 1 i,j =1 a ij ( x, u ) v x i x j ( x ) } , 2 R n . for all x ∈ I Generalized finite differencesfor solving stochastic control problems – p.4/37

HJB equation λv ( x ) = u ∈ U { ℓ ( x, u ) + f ( x, u ) · v x ( x ) inf � n + 1 i,j =1 a ij ( x, u ) v x i x j ( x ) } , 2 R n . for all x ∈ I • Covariance matrix: a ( x, u ) := R n × I σ ( x, u ) σ ( x, u ) T , ∀ ( x, u ) ∈ I R m . Generalized finite differencesfor solving stochastic control problems – p.4/37

HJB equation λv ( x ) = u ∈ U { ℓ ( x, u ) + f ( x, u ) · v x ( x ) inf � n + 1 i,j =1 a ij ( x, u ) v x i x j ( x ) } , 2 R n . for all x ∈ I • Covariance matrix: a ( x, u ) := R n × I σ ( x, u ) σ ( x, u ) T , ∀ ( x, u ) ∈ I R m . • All functions Lipschitz continuous and bounded: V unique bounded viscosity solution of HJB. Generalized finite differencesfor solving stochastic control problems – p.4/37

Simplified HJB equation Discretization already non-trivial in this case: Null drift f and data not depending on control variable u . Then the HJB equation reduces to: n � λv ( x ) = ℓ ( x ) + 1 R n a ij ( x ) v x i x j ( x ) , for all x ∈ I 2 i,j =1 Generalized finite differencesfor solving stochastic control problems – p.5/37

Generalized finite differences I ξ 2 ξ 1 O p Generalized finite differencesfor solving stochastic control problems – p.6/37

Generalized finite differences II • Discretization steps h 1 , . . . , h n , points x k := ( k 1 h 1 , . . . , k n h n ) . Generalized finite differencesfor solving stochastic control problems – p.7/37

Generalized finite differences II • Discretization steps h 1 , . . . , h n , points x k := ( k 1 h 1 , . . . , k n h n ) . • Given ϕ = { ϕ k } : real valued function over Z n . Generalized finite differencesfor solving stochastic control problems – p.7/37

Generalized finite differences II • Discretization steps h 1 , . . . , h n , points x k := ( k 1 h 1 , . . . , k n h n ) . • Given ϕ = { ϕ k } : real valued function over Z n . • Second order finite difference operator ∆ ξ ϕ k := ϕ k + ξ + ϕ k − ξ − 2 ϕ k = ϕ k + ξ − ϕ k − ( ϕ k − ϕ k − ξ ) . Generalized finite differencesfor solving stochastic control problems – p.7/37

Generalized finite differences II • Discretization steps h 1 , . . . , h n , points x k := ( k 1 h 1 , . . . , k n h n ) . • Given ϕ = { ϕ k } : real valued function over Z n . • Second order finite difference operator ∆ ξ ϕ k := ϕ k + ξ + ϕ k − ξ − 2 ϕ k = ϕ k + ξ − ϕ k − ( ϕ k − ϕ k − ξ ) . • If ϕ k = Φ( x k ) . Then ∆ ξ ϕ k = D 2 Φ( x k ) x ξ x ξ + o ( � x ξ � 2 ) = � n i,j =1 ξ i h i ξ j h j Φ x i x j ( x k ) + o ( h 2 ) Generalized finite differencesfor solving stochastic control problems – p.7/37

Approximation of second-order term • Scaled covariance: a h ij ( x ) = a ( x ) / ( h i h j ) . Generalized finite differencesfor solving stochastic control problems – p.8/37

Approximation of second-order term • Scaled covariance: a h ij ( x ) = a ( x ) / ( h i h j ) . • Strong consistency: ξ ∈S α k,ξ ∆ ξ φ k = � n � i,j =1 a ij ( x k )Φ x i x j ( x k ) Generalized finite differencesfor solving stochastic control problems – p.8/37

Approximation of second-order term • Scaled covariance: a h ij ( x ) = a ( x ) / ( h i h j ) . • Strong consistency: ξ ∈S α k,ξ ∆ ξ φ k = � n � i,j =1 a ij ( x k )Φ x i x j ( x k ) • Characterization: ξ ∈S α k,ξ ξξ T = a h ( x k ) , k ∈ Z n . � for all Generalized finite differencesfor solving stochastic control problems – p.8/37

Numerical scheme • Explicit schemes: ξ ∈S α k,ξ ∆ ξ v k , k ∈ Z n λv k = ℓ ( x k ) + � Generalized finite differencesfor solving stochastic control problems – p.9/37

Numerical scheme • Explicit schemes: ξ ∈S α k,ξ ∆ ξ v k , k ∈ Z n λv k = ℓ ( x k ) + � • Fictitious time step: h 0 > 0 . Equivalent scheme:     � v k := (1+ λh 0 ) − 1  v k + h 0 ℓ ( x k ) + h 0 α k,ξ ∆ ξ v k  ξ ∈S Generalized finite differencesfor solving stochastic control problems – p.9/37

Monotonicity condition • Nondecreasing mapping � v → v k + h 0 ℓ ( x k ) + h 0 ξ ∈S α k,ξ ∆ ξ v k Generalized finite differencesfor solving stochastic control problems – p.10/37

Monotonicity condition • Nondecreasing mapping � v → v k + h 0 ℓ ( x k ) + h 0 ξ ∈S α k,ξ ∆ ξ v k • Holds iff α k,ξ ≥ 0 , ∀ ( ξ, k ) ∈ S × Z n . � α k,ξ ≤ h − 1 0 , ∀ ( k ) ∈ Z n . 2 ξ ∈S Generalized finite differencesfor solving stochastic control problems – p.10/37

Monotonicity condition • Nondecreasing mapping � v → v k + h 0 ℓ ( x k ) + h 0 ξ ∈S α k,ξ ∆ ξ v k • Holds iff α k,ξ ≥ 0 , ∀ ( ξ, k ) ∈ S × Z n . � α k,ξ ≤ h − 1 0 , ∀ ( k ) ∈ Z n . 2 ξ ∈S • Second condition satisfied when h 0 small enough, once an estimate of � ξ ∈S α k,ξ is known (see below). Generalized finite differencesfor solving stochastic control problems – p.10/37

Convergence • Monotonicity and consistency imply convergence. Generalized finite differencesfor solving stochastic control problems – p.11/37

Convergence • Monotonicity and consistency imply convergence. • Strong consistency implies ξ ∈S α k,ξ ≤ trace a h ( x k ) � Generalized finite differencesfor solving stochastic control problems – p.11/37

Convergence • Monotonicity and consistency imply convergence. • Strong consistency implies ξ ∈S α k,ξ ≤ trace a h ( x k ) � • Condition for time step: h 0 = O (min i h 2 i ) Generalized finite differencesfor solving stochastic control problems – p.11/37

Explicit strong consistency conditions • Characterization: a h ( x k ) belongs to �� � R |S| ξ ∈S α ξ ξξ T ; α ∈ I C ( S ) := . + Generalized finite differencesfor solving stochastic control problems – p.12/37