Mean Field Games on Unbounded Networks and the Graphon MFG Equations - PowerPoint PPT Presentation

Mean Field Games on Unbounded Networks and the Graphon MFG Equations Peter E. Caines McGill University Work with Shuang Gao and Minyi Huang CROWDS models and control CIRM, Marseille, France, June, 2019 Work supported by NSERC and ARL 1 / 55

Program Program Major-Minor Agent Systems and MFG Equilibria LQG PO Major-Minor Agent MFG Theory Populations of Agents Distributed on Networks: Motivation + Introduction to Graphon Theory Graphon Control Systems Graphon Mean Field Games LQG-GMFG Example 2 / 55

Basic Formulation of Nonlinear Major-Minor MFG Systems Problem Formulation: Notation: Subscript 0 for the major agent A 0 and an integer valued subscript for minor agents {A i : 1 ≤ i ≤ N } . The states of A 0 and A i are R n valued and denoted z N 0 ( t ) and z N i ( t ) . State Dynamics of the Major and N Minor Agents: N 0 ( t ) = 1 dz N � f 0 ( t, z N 0 ( t ) , u N 0 ( t ) , z N j ( t )) dt N j =1 N + 1 � σ 0 ( t, z N 0 ( t ) , z N z N j ( t )) dw 0 ( t ) , 0 (0) = z 0 (0) , 0 ≤ t ≤ T, N j =1 N i ( t ) = 1 dz N � f ( t, z N i ( t ) , z N 0 ( t ) , u N i ( t ) , z N j ( t )) dt N j =1 N + 1 � σ ( t, z N i ( t ) , z N z N j ( t )) dw i ( t ) , i (0) = z i (0) , 1 ≤ i ≤ N. N j =1 3 / 55

MFG Nonlinear Major-Minor Agent Formulation Performance Functions for Major and Minor Agents: � T � 1 N � J N 0 ( u N 0 ; u N � L 0 [ t, z N 0 ( t ) , u N 0 ( t ) , z N − 0 ) := E j ( t )] dt, N 0 j =1 � T � 1 N � J N i ( u N i ; u N � L [ t, z N i ( t ) , z N 0 ( t ) , u N i ( t ) , z N − i ) := E j ( t )] dt. N 0 j =1 The major agent has non-negligible influence on the mean field (mass) behaviour of the minor agents. (A consequence will be that the system mean field is no longer a deterministic function of time.) (Ω , F , {F t } N t ≥ 0 , P ) : a complete filtered probability space F N := σ { z j (0) , w j ( s ) : 0 ≤ j ≤ N, 0 ≤ s ≤ t } Mtlly. Ind. ICs, Ind. BMs. t F w 0 := σ { z 0 (0) , w 0 ( s ) : 0 ≤ s ≤ t } . t 4 / 55

Basic Formulation of Nonlinear MFG Systems Controlled McKean-Vlasov Equations: Infinite population limit dynamics: = f [ x t , u t , µ t ] dt + σdw t dx t � � f [ x, u, µ t ] f ( x, u, y ) µ t ( dy ) R Given ICs, a solution to the MKV SDE is a pair ( x t , µ t ( dx ); 0 ≤ t < T ) Infinite population limit cost: � T � u ∈U J ( u, µ ) inf inf L [ x t , u t , µ t ] dt u ∈U E 0 where µ t ( · ) = measure of the population state distribution 5 / 55

Information Patterns and Nash Equilibria Information Patterns: Local to Agent i : F i � σ ( x i ( τ ); τ ≤ t ) , 1 ≤ i ≤ N U loc,i : F i adapted control + system parameters Global with respect to the Population: F N � σ ( x j ( τ ); τ ≤ t, 1 ≤ j ≤ N ) U : F N adapted control + system parameters Definition: Nash Equilibrium: Unilateral Move Yields No Gain The set of controls U a = { u a i ; u a i adapted to U loc,i , 1 ≤ i ≤ N } generates a Nash Equilibrium w.r.t. the performance functions { J i ; 1 ≤ i ≤ N } if, for each i , J i ( u a i , u a u i ∈U J i ( u i , u a − i ) = inf − i ) 6 / 55

Saddle Point Nash Equilibrium Agent y is a maximizer Agent x is a minimizer 4 3 2 1 0 −1 −2 −3 −4 2 1 0 2 1 −1 0 −1 −2 −2 y x 7 / 55

ǫ -Nash Equilibrium ǫ -Nash Equilibria: Given ε > 0 , the set of controls U 0 = { u 0 i ; 1 ≤ i ≤ N } generates an ε -Nash Equilibrium w.r.t. the performance functions { J i ; 1 ≤ i ≤ N } if, for each i , J i ( u 0 i , u 0 u i ∈U J i ( u i , u 0 − i ) ≤ J i ( u 0 i , u 0 − i ) − ε ≤ inf − i ) 8 / 55

Fundamental Mean Field Game MV HJB-FPK Theory Mean Field Game Pair (HMC, 2006, LL, 2006-07): Assuming that for any given strategy (i.e. control law) the infinite population limits exist for the population dynamics and performance functions, then: (i) the generic agent best response (BR) is generated by an MKV-HJB equation and (ii) the corresponding generic agent state distribution is generated by an MV-FPK equation (equivalently MKV SDE): + σ 2 ∂ 2 V � � − ∂V f [ x, u, µ t ] ∂V [MF-HJB] ∂t = inf ∂x + L [ x, u, µ t ] 2 ∂x 2 u ∈ U V ( T, x ) = 0 , ( t, x ) ∈ [0 , T ) × R + σ 2 ∂ 2 p ( t, x ) ∂p ( t, x ) = − ∂ { f [ x, u, µ ] p ( t, x ) } [MF-FPK] ∂t ∂x 2 ∂x 2 ( [MF-MKV SDE ] dx t = f [ x t , ϕ ( t, x | µ t ) , µ t ] dt + σdw t ) [MF-BR] u t = ϕ ( t, x | µ t ) , ( t, x ) ∈ [0 , T ] × R 9 / 55

Basic Mean Field Game MV HJB-FPK Theory Theorem (Huang, Malham´ e, PEC, CIS’06) Subject to technical conditions (i.e. uniform cty.+ boundedness on all functions + their derivatives + Lipschitz cty. wrt. controls): (i) the MKV MFG Equations have a unique solution with the best response control generating a unique Nash equilibrium given by u 0 i = ϕ ( t, x | µ t ) , 1 ≤ i ≤ N. Furthermore, (ii) ∀ ǫ > 0 ∃ N ( ǫ ) s.t. ∀ N ≥ N ( ǫ ) i ( u 0 i , u 0 i ( u i , u 0 i ( u 0 i , u 0 J N u i ∈U J N − i ) ≤ J N − i ) − ǫ ≤ inf − i ) , where u i ∈ U is adapted to F N := { σ ( x j ( τ ); τ ≤ t, 1 ≤ j ≤ N ) } . 10 / 55

Outline of Proof of Basic Result Outline of Proof: Restrict Lipschitz constants so that a Banach contraction argument gives existence and uniqueness via an iterated closed loop from mean field measure to control (from HJB) to measure (from FPK). Major-Minor NL MFG theory: Mojtaba Nourian, PEC, SICON, 2013. 11 / 55

The Three Key Ideas of Mean Field Game Theory Three Key Ideas: Nash Equilibrium Non-Cooperative Game Theoretic Equilibrium given by the solution to a **stochastic control problem** (wrt the distribution of the mass of agents) Dynamic Regeneration of Equilibrium: Generic Agent Mean Field Equilbrium is **regenerated** when all agents use the MFG BR strategies) Drastic Simplification of Dynamic Games: Infinite Population Control Strategies Yield **simple** Approximate Nash Equilbria for Large Finite Populations 12 / 55

Next on the Program Major-Minor Agent State Estimation and MFG Equilibria Populations of Agents Distributed on Networks: Introduction to Graphon Theory Graphon Control Systems Graphon Mean Field Games LQG-MFG Example 13 / 55

Separated and Linked Populations Seek an MFG theory of flocking and swarming. 14 / 55

Motivation for Application of Graphon Theory in Systems and Control Networks are ubiguitous, and are often growing in size and complexity: Online Social Networks, Brain Networks, Grid Networks, Transportation Networks, IoT, etc. 15 / 55

Motivation for a Graphon Theory of Systems and Control A Common Feature of Networks of Dynamical Systems: Local nodes possess intrinsic states which evolve due to interactions with other nodes. Power grids (loads, generators and energy storage units) Epidemic networks Brain networks Social networks (opinions) and Fish Schooling Networks of computational devices Crowds? Range of System Networks Behaviours: freely evolving, or locally controlled, and (or) globally controlled. 16 / 55

Motivation for Application of Graphon Theory in Systems and Control Shall consider a class of complex networks characterized by: Large number of nodes (in principle millions/billions of nodes) Complex connections which are asymptotically dense at each node (but sparse case is important) Intrinsically capable of growth in size The recently developed mathematical theory of graphons provides a methodology for analyzing arbitrarily complex networks. (Sparse theory is developing.) 17 / 55

Introduction to Graphons Graphs, Adjacency Matrices and Pixel Pictures Graph Adjacency Matrix Pixel Picture 0 1 0 1 0 1 B C 1 0 1 0 B C − ! − ! @ A 0 1 0 1 1 0 1 0 Graph, Adjacency Matrix, Pixel Picture The whole pixel picture is presented in a unit square [0 , 1] × [0 , 1] , so the square elements have sides of length 1 N , where N is the number of nodes. 18 / 55

Introduction to Graphons 0 1 0 1 0 1 B 1 0 1 0 C B C − ! − ! @ A 0 1 0 1 1 0 1 0 Graph Sequence Converging to Graphon Graph Sequence Converging to its Limit Graphons: bounded symmetric Lebesgue measurable functions W : [0 , 1] 2 ! [0 , 1] interpreted as weighted graphs on the vertex set [0 , 1] . 0 := { W : [0 , 1] 2 ! [0 , 1] } 1 := { W : [0 , 1] 2 ! [ − 1 , 1] } G sp G sp R := { W : [0 , 1] 2 ! R } G sp L. Lov´asz, Large Networks and Graph Limits. American Mathematical Soc., 2012, vol. 60. 19 / 55

Introduction to Graphons Metric in Graphon Space Cut norm � � W � � := sup | W ( x, y ) dxdy | (1) M,T ⊂ [0 , 1] M × T Cut metric φ � W φ − V � � d � ( W , V ) := inf (2) L 2 metric φ � W φ − V � 2 d L 2 ( W , V ) := inf (3) where W φ ( x, y ) = W ( φ ( x ) , φ ( y )) . Since � W � � ≤ � W � L 2 for any graphon W , convergence in d L 2 implies convergence in d � . 20 / 55

Introduction to Graphons Compactness of Graphon Spaces Theorem The graphon spaces ( G sp 0 , d � ) , and hence the closed subsets of any ( G sp R , d � ) , are compact. 21 / 55

Introduction to Graphons Graphons as Operators Graphon W ∈ G sp 1 as an operator: W : L 2 [0 , 1] ! L 2 [0 , 1] Operation on v ∈ L 2 [0 , 1] : � 1 [ Wv ]( x ) = W ( x, α ) v ( α ) dα (4) 0 Operator product : � 1 [ UW ]( x, y ) = U ( x, z ) W ( z, y ) dz (5) 0 where U , W ∈ G sp 1 22 / 55