Mean Field Games: Numerical Methods Y. Achdou October 24, 2011 - PowerPoint PPT Presentation

Mean Field Games: Numerical Methods Y. Achdou October 24, 2011

Content • A partial review on the theory of Lasry and Lions • Numerical schemes for the mean field games system – Description of the scheme – Existence, bounds, uniqueness • Numerical tests • The optimal planning problem – Description of the scheme – Existence of the solution via convex programming – Numerical results 1

I. Mean field games: some aspects of the theory of Lasry and Lions Consider N identical players whose dynamics are √ dX i 2 νdW i t − γ i dt, t = 0 = x i ∈ R d . X i t , . . . , W N t ) independent Brownian motions in R d , • ( W 1 • ν > 0 , • The control of the player i , i.e. γ i is a bounded process assumed to be t , . . . , W N adapted to ( W 1 t ) . For simplicity, all the functions used below are periodic with period 1 in every direction. Let T be the unit torus of R d . 2

The cost of the player i at time t is J i ( t ) = 0 0 2 3 1 2 3 1 Z T X X 1 1 @ L ( X i s , γ i 5 ( X i 5 ( X i @ 4 A ds + V 0 4 A s ) + V s ) T ) δ X j δ X j E N − 1 N − 1 s T t j � = i j � = i • V and V 0 are operators which continuously map the set of probability measures on T (endowed with the weak * topology) to a bounded subset of Lip ( T ) . • L is Lipschitz in x uniformly in γ bounded, and L ( x, γ ) | γ |→∞ inf lim = + ∞ . | γ | x Introduce the Hamiltonian x ∈ T , p ∈ R d . H ( x, p ) = sup γ ∈ R d ( p · γ − L ( x, γ )) , Assume that H is C 1 . 3

γ N ) is a Nash point, if ∀ i = 1 , . . . , N , γ 1 , . . . , ¯ Nash equilibria: (¯ J i ( t, ¯ γ i − 1 , γ i , ¯ γ i +1 , . . . , ¯ γ N ) ≥ J i ( t, ¯ γ i − 1 , ¯ γ i , ¯ γ i +1 , . . . , ¯ γ N ) γ 1 , . . . , ¯ γ 1 , . . . , ¯ Theorem There exist N functions ( u j ( t, x 1 , . . . , x N )) j =1 ,...,N such that 2 3 8 X X > 1 > ∂u i ∂H > 4 5 ( x i ) > ∂t + ν ∆ X u i − H ( x i , ∇ x i u i ) − ∂p ( x j , ∇ x j u j ) · ∇ x j u i = − V δ x j > > N − 1 < j � = i j � = i 2 3 > > X > 1 > > 4 5 ( x i ) u i ( T, x 1 , . . . , x N ) = V 0 δ x j > : N − 1 j � = i The feedbacks γ i = ∂H ¯ ∂p ( x i , ∇ x i u i ) yield a Nash point. In general, no uniqueness 4

Assuming that the players have the same distribution m 0 at t = 0 , and passing to the limit as N → ∞ , Lasry and Lions get the system of 2 PDEs  ∂u   ∂t + ν ∆ u − H ( x, ∇ u ) = − V [ m ] , in (0 , T ) × T , � � ∂m m∂H   ∂t − ν ∆ m − div ∂p ( x, ∇ u ) = 0 , in (0 , T ) × T , with the terminal and initial conditions u ( t = T ) = V 0 [ m ( t = T )] , m (0 , x ) = m 0 ( x ) , in T , and where m ( t, · ) is the density of players at time t : � m ≥ 0 , m ( t, x ) dx = 1 . T Remark The full justification of the passage to the limit is done in special cases only. 5

Some results on the MFG system  ∂u  ∂t − ν ∆ u + H ( x, ∇ u ) = V [ m ] , in (0 , T ] × T ,     � �    ∂m m∂H   ∂t + ν ∆ m + div ∂p ( x, ∇ u ) = 0 , in [0 , T ) × T , ( ∗ ) �     mdx = 1 , m > 0 in T ,     T   u ( t = 0) = V 0 [ m ( t = 0)] , m ( t = T ) = m ◦ , Remark Note the special structure of the system: 1. forward/backward w.r.t. time. 2. the operator in the Fokker-Planck equation is the adjoint of the linearized version of the operator in the HJB equation. 3. coupling: via V [ m ] in the HJB equation and ∂ p H ( t, x, ∇ u ) in the Fokker Planck equation, and possibly via the initial condition on u . 6

Theorem (Lasry-Lions) : Existence for (*) 1) If ν > 0 and • V and V 0 are operators which continuously map the set of probability measures on T (endowed with the weak * topology) to a bounded subset of Lip ( T ) , i.e. nonlocal smoothing operators, • H is smooth on T × R d and � � � � ∂H � � ∀ x ∈ T , ∀ p ∈ R d , ∂x ( x, p ) � ≤ C (1 + | p | ) , � • m 0 is a smooth probabilty density, then (*) has at least a classical solution. 2) Existence can also be proved if H is Lipschitz w.r.t. p uniformly in x and V [ m ]( x ) = F ( m ( x )) where F is a smooth function. 7

Uniqueness for (*) Theorem (Lasry-Lions) If the operators V and V 0 are monotone, i.e. � ( V [ m ] − V [ ˜ m ])( m − ˜ m ) ≤ 0 ⇒ V [ m ] = V [ ˜ m ] , T � ( V 0 [ m ] − V 0 [ ˜ m ])( m − ˜ m ) ≤ 0 ⇒ V 0 [ m ] = V 0 [ ˜ m ] , T then (*) has a unique solution. Remark This assumption on V has an economical interpretation if V is local: crowd aversion. 8

Proof Consider two solutions of (*): ( v 1 , m 1 ) and ( v 2 , m 2 ) : • multiply HJB 1 − HJB 2 by m 1 − m 2 Z T Z ( − ( v 1 − v 2 )( ∂ t m 1 − ∂ t m 2 ) + ν ∇ ( v 1 − v 2 ) · ∇ ( m 1 − m 2 )) 0 T Z T Z “ ” + H ( x, ∇ v 1 ) − H ( x, ∇ v 2 ) ( m 1 − m 2 ) T 0 Z T Z Z “ ” = ( V [ m 1 ] − V [ m 2 ])( m 1 − m 2 ) + V 0 [ m 1 (0)] − V 0 [ m 2 (0)])( m 1 (0) − m 2 (0) . 0 T T • multiply FP 1 − FP 2 by v 1 − v 2 Z T Z 0 = − ( v 1 − v 2 )( ∂ t m 1 − ∂ t m 2 ) + ν ∇ ( v 1 − v 2 ) · ∇ ( m 1 − m 2 ) 0 T Z T Z „ « m 1 ∂H ∂p ( x, ∇ v 1 ) − m 2 ∂H + ∂p ( x, ∇ v 2 ) · ∇ ( v 1 − v 2 ) . 0 T 9

• subtract: 8 „ « Z T Z H ( x, ∇ v 1 ) − H ( x, ∇ v 2 ) − ∂H > > > ∂p ( x, ∇ v 1 ) · ∇ ( v 1 − v 2 ) m 1 > > > „ « Z T Z > T 0 > > H ( x, ∇ v 2 ) − H ( x, ∇ v 1 ) − ∂H > < + ∂p ( x, ∇ v 2 ) · ∇ ( v 2 − v 1 ) m 2 0 = Z T Z 0 T > > > + ( V [ m 1 ] − V [ m 2 ])( m 1 − m 2 ) > > > Z > 0 T > > > : + ( V 0 [ m 1 ( t = 0)] − V 0 [ m 2 ( t = 0)])( m 1 ( t = 0) − m 2 ( t = 0)) T Since H is convex, V and V 0 are monotone, the 4 terms vanish. The strict monotonicity of V implies that V [ m 1 ] = V [ m 2 ] and v 1 ( t = 0) = v 2 ( t = 0) . The identity v 1 = v 2 comes from the uniqueness for the HJB equation. The identity m 1 = m 2 comes from the uniqueness for the Fokker-Planck equation. 10

II. Finite horizon: numerical methods (Y.A, I. Capuzzo Dolcetta, SIAM J. Numerical Analysis, 2010) 11

Finite difference schemes Goal: use a (semi-)implicit finite difference scheme, robust when ν → 0 , which guarantees existence, and possibly uniform bounds and uniqueness. Take d = 2 : • Let T h be a uniform grid on the torus with mesh step h , and x ij be a generic point in T h . • Uniform time grid: ∆ t = T/N T , t n = n ∆ t . • The values of u and m at ( x i,j , t n ) are resp. approximated by U n i,j and M n i,j . 12

Notation: • The discrete Laplace operator: (∆ h W ) i,j = − 1 h 2 (4 W i,j − W i +1 ,j − W i − 1 ,j − W i,j +1 − W i,j − 1 ) . • Right-sided finite difference formulas for ∂w ∂x 1 ( x i,j ) and ∂w ∂x 2 ( x i,j ) : 1 W ) i,j = W i +1 ,j − W i,j 2 W ) i,j = W i,j +1 − W i,j ( D + ( D + , . and h h • The set of 4 finite difference formulas at x i,j : � � ( D + 1 W ) i,j , ( D + 1 W ) i − 1 ,j , ( D + 2 W ) i,j , ( D + [ D h W ] i,j = 2 W ) i,j − 1 . 13

Discrete HJB equation ∂u ∂t − ν ∆ u + H ( x, ∇ u ) = V [ m ] ↓ U n +1 − U n i,j i,j − ν (∆ h U n +1 ) i,j + g ( x i,j , [ D h U n +1 ] i,j ) = ( V h [ M n ]) i,j ∆ t • g ( x i,j , [ D h U n +1 ] i,j ) “ ” 1 U n +1 ) i,j , ( D + 1 U n +1 ) i − 1 ,j , ( D + 2 U n +1 ) i,j , ( D + 2 U n +1 ) i,j − 1 x i,j , ( D + = g , • for instance, ( V h [ M ]) i,j = V [ m h ]( x i,j ) , calling m h the piecewise constant function on T taking the value M i,j in the square | x − x i,j | ∞ ≤ h/ 2 . 14

Classical assumptions on the discrete Hamiltonian g ( q 1 , q 2 , q 3 , q 4 ) → g ( x, q 1 , q 2 , q 3 , q 4 ) . • Monotonicity: g is nonincreasing with respect to q 1 and q 3 and nondecreasing with respect to to q 2 and q 4 . • Consistency: ∀ x ∈ T , ∀ q = ( q 1 , q 3 ) ∈ R 2 . g ( x, q 1 , q 1 , q 3 , q 3 ) = H ( x, q ) , • Differentiability: g is of class C 1 , and � �� � � � ∂g � � x, ( q 1 , q 2 , q 3 , q 4 ) � ≤ C (1 + | q 1 | + | q 2 | + | q 3 | + | q 4 | ) . � ∂x • Convexity: ( q 1 , q 2 , q 3 , q 4 ) → g ( x, q 1 , q 2 , q 3 , q 4 ) is convex. 15

The discrete version of � � ∂m m∂H ∂t + ν ∆ m + div ∂p ( x, ∇ v ) = 0 . ( † ) It is chosen so that • each time step leads to a linear system with a matrix – whose diagonal coefficients are negative, – whose off-diagonal coefficients are nonnegative, in order to hopefully use some discrete maximum principle. • The argument for uniqueness should hold in the discrete case, so the discrete Hamiltonian g should be used for ( † ) as well. 16