Mean field limit of controlled system: From discrete to continuous - PowerPoint PPT Presentation

Mean field limit of controlled system: From discrete to continuous problems. Nicolas Gast 1 EPFL LIG and INRIA January 25, 2011 1 Joint work with Bruno Gaujal (INRIA) and Jean-Yves Le Boudec (EPFL)

Introduction: mean field interacting objects The term “mean field” applies for system of interacting objects: communication networks, computing clusters. epidemic models, gossip. chemical reactions. Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 2/25

Introduction: mean field interacting objects The term “mean field” applies for system of interacting objects: communication networks, computing clusters. epidemic models, gossip. chemical reactions. Objectives Analyze and improve the performance of the system: Characterize the dynamics of the system Find good (or optimal) policies to control the system. Tools Build a stochastic (microscopic) model of interacting objects. Study macroscopic properties. Fight curse of dimensionality via continuous approximation. Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 2/25

Example of mean field approximation: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . Two main characteristics Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . Two main characteristics 1 Exchangeable objects – The important quantity is the number of objects in each state M N ( t ) 0.7 N=100 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . Two main characteristics 1 Exchangeable objects – The important quantity is the number of objects in each state M N ( t ) 2 The drift – average difference of M N ( t ) between t and t + dt . The drift is:   − γ IS + µ R 0.7 N=100 drift f ( m ) = γ IS − β I   0.6 β I − µ R 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . Two main characteristics 1 Exchangeable objects – The important quantity is the number of objects in each state M N ( t ) 2 The drift – average difference of M N ( t ) between t and t + dt . The drift is:   − γ IS + µ R 0.7 N=100 drift f ( m ) = γ IS − β I   0.6 mean field approximation β I − µ R 0.5 0.4 The mean field approximation is 0.3 the solution of ˙ m = f ( m ). 0.2 0.1 Question: what is the link 0 between M N ( t ) and m ( t )? 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Mean field for performance evaluation. Generic tool to study multiple properties: Transient behavior Starting from M N (0) = m (0), is M N ( t ) close to m ( t ) for all t ? Yes if f has some continuity properties ((Le Boudec, Bena¨ ım 08)). Steady state behavior If lim t →∞ m ( t ) = m ∗ , is the stationary distrib. of M N ( t ) close to m ∗ ? “Yes, but”. Ex: load balancing (Mitzenmacher 98, Gast et. al. 10). Propagation du chaos (Sznitman 91, Graham 00) : When the size of the system grows, objects become independent. Ex: 802.11 (Bianchi 00, Bordenave et al. 05) Stability Stability of the fluid implies stability of the stochastic system. True for some models (Ex: Dai 95, Bramson 08) Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 4/25

Mean field for performance evaluation. Generic tool to study multiple properties: Transient behavior Starting from M N (0) = m (0), is M N ( t ) close to m ( t ) for all t ? Yes if f has some continuity properties ((Le Boudec, Bena¨ ım 08)). Steady state behavior If lim t →∞ m ( t ) = m ∗ , is the stationary distrib. of M N ( t ) close to m ∗ ? “Yes, but”. Ex: load balancing (Mitzenmacher 98, Gast et. al. 10). Propagation du chaos (Sznitman 91, Graham 00) : When the size of the system grows, objects become independent. Ex: 802.11 (Bianchi 00, Bordenave et al. 05) Stability Stability of the fluid implies stability of the stochastic system. True for some models (Ex: Dai 95, Bramson 08) In this talk: Optimal control problems. extensions to controlled system. Non-smooth dynamics. Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 4/25

Outline 1. Introduction 2. Mean field via stochastic approximation ◮ Basic hypothesis and definition. 3. Optimal control of mean field models ◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability. 4. Going further: non-smooth dynamics ◮ if the drift f is not continuous: can we define an ODE dm dt = f ( m )? ◮ What is the limit of M N ? 5. Conclusion and references Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 5/25

Outline 1. Introduction 2. Mean field via stochastic approximation ◮ Basic hypothesis and definition. 3. Optimal control of mean field models ◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability. 4. Going further: non-smooth dynamics ◮ if the drift f is not continuous: can we define an ODE dm dt = f ( m )? ◮ What is the limit of M N ? 5. Conclusion and references Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 5/25

Mean field convergence via stochastic approximation Assumptions [Bena¨ ım, Le Boudec 08] A1 Objects are exchangeable ( i.e. M N ( t ) is Markovian.) A2 Number of objects doing a transition at a given time step is less than NI N (in average) and N 2 I 2 N (in second moment). Typically, there are at most K objects doing a transition. In that case, I N = K / N works. A3 The drift f is lipschitz-continuous. � � f ( m ) = 1 M N ( t + 1 N ) − M N ( t ) | M N ( t ) = m . N E (classical assumption but annoying.) Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 6/25

Mean field convergence via stochastic approximation A1 Objects are exchangeable Number of objects doing a transition at a given time step is less than NI N (in average) and N 2 I 2 A2 N (in second moment). A3 The drift f is lipschitz-continuous. Theorem (Bena¨ ım-Le Boudec 08) � � P � � M N ( t ) − m ( t ) � Under assumptions (A1,A2,A3), sup − → 0. � 0 ≤ t ≤ T where m is the solution of the ODE ˙ m = f ( m ). Theorem (BB 08) m = f ( m ) has a unique attractor m ∗ . If the ODE ˙ The stationary distribution π N of M N converges to δ m ∗ . If M N is picked according to π N , then any k objects are independent as N → ∞ . Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 6/25

Idea of the proof: π ( t + 1 By definition of the drift, M N N ) can be written: � � M N ( t + 1 N ) = M N ( t ) + 1 f ( M N ( t )) + noise . N � �� � � �� � Drift ( deterministic ) Random , E [ . ]=0 At time t = k 1 N , M N ( t ) is equal to: k − 1 k N f ( M N ( i 1 1 � � M N ( t ) = M N 0 + N )) + . noise N i =0 i =0 � �� � � �� � Euler discretization Converges to 0 Convergence of the noise: martingale argument. Convergence of the Euler discretization: by Gronwall’s lemma. Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 7/25

Application example: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . The drift is: This is a mean field model   Symmetry: Objects are indistinguishable. − γ IS + µ R γ IS − β I Intensity: each transition affects one   β I − µ R (recovery) or two objects (infection). Drift continuous: OK. Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 8/25

Application example: SIR model a Susceptible individual becomes infected with rate γ I an Infected recovers with rate β a Recovered becomes susceptible with rate µ . The drift is: This is a mean field model   Symmetry: Objects are indistinguishable. − γ IS + µ R γ IS − β I Intensity: each transition affects one   β I − µ R (recovery) or two objects (infection). Drift continuous: OK. 0.7 N=100 0.7 N=100 drift drift 0.6 mean field approximation 0.6 mean field approximation 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 8/25