On Ergodic Impulse Control with Constraint Maurice Robin Based on - PowerPoint PPT Presentation

On Ergodic Impulse Control with Constraint Maurice Robin Based on joint papers with J.L. Menaldi University Paris-Sanclay 91190 Saint-Aubin, France (e-mail: maurice.robin@polytechnique.edu) IMA, Minneapolis, MN May 7–11, 2018 MR (UP-S) Ergodic Impulse Control with Constraint 1 / 19

Content - Statement of the problem - HJB equation - Solution of the HJB equation - Existence of an optimal control - Extensions - References MR (UP-S) Ergodic Impulse Control with Constraint 2 / 19

Introduction Statement of the Problem Statement of the Problem (as in JL Menaldi’s talk except for the cost and ergodic assumptions) - The uncontrolled state is described by a Markov-Feller process x t (values in E metric compact) - impulse control ν = ( θ i , ξ i ) i ≥ 1 , θ i increasing sequence of stopping times, ξ i E valued random variable - constraint on impulse controls: θ i > 0 and θ i is a jump time of the signal process y t y τ n = 0 , y t = t − τ n for τ n ≤ t ≤ τ n +1 , n ≥ 1 , T n = τ n +1 − τ n , conditionally to x t as IID random variables with intensity λ ( x , y ) - ξ i ∈ Γ( x θ i ), Γ( x ) closed set of E and ∀ ξ ∈ Γ( x ), Γ( ξ ) ⊂ Γ( x ) MR (UP-S) Ergodic Impulse Control with Constraint 3 / 19

Introduction Statement of the Problem (2) Statement of the Problem (2) - running cost f ( x , y ) and impulse cost c ( x , ξ ), both positive bounded and continuous, c ( x , ξ ) + c ( ξ, ξ ′ ) ≥ c ( x , ξ ′ ) c ( x , ξ ) ≥ c 0 > 0 and - Mg ( x ) ≡ inf ξ ∈ Γ( x ) { c ( x , ξ ) + g ( x ) } is assumed to be continuous if g is continuous and there exists a measurable selector ˆ ξ ( x , g ). V will denote the set of admissible controls V = { ( θ i , ξ i ) , i ≥ 1 , θ 1 > 0 , y θ i = 0 } V 0 the set of admissible controls satisfying the constraint, but θ 1 = 0 is allowed MR (UP-S) Ergodic Impulse Control with Constraint 4 / 19

Introduction Statement of the Problem (3) Statement of the Problem (3) - Controlled process The controlled process for a control ν is defined on the product space Ω ∞ , Ω = D ( R + ; E × R + ) by a probability P ν xy , and ( x ν t , y ν t ) = ( x i t , y i t ) for θ i − 1 ≤ t < θ i evolves as the uncontrolled process between impulses instants. MR (UP-S) Ergodic Impulse Control with Constraint 5 / 19

Introduction Statement of the Problem (4) Statement of the Problem (4) - The average cost to be minimized: � � T 1 � T E ν f ( x ν s , y ν � ✶ θ i ≤ T c ( x i − 1 J ( x , y , ν ) = lim inf s ) d s + , ξ i ) xy θ i T →∞ 0 i µ ( x , y ) = inf { J ( x , y , ν ) : ν ∈ V} We will use an auxiliary problem � � τ n 1 � ˜ � E ν f ( x ν s , y ν ✶ θ i ≤ T c ( x i − 1 J ( x , y , ν ) = lim inf s ) d s + , ξ i ) xy θ i E ν xy τ n n →∞ 0 i µ 0 ( x , y ) = inf { J ( x , y , ν ) : ν ∈ V 0 } MR (UP-S) Ergodic Impulse Control with Constraint 6 / 19

Introduction Additional Assumptions Additional Assumptions λ ( x , y ) is ≥ 0 bounded and continuous and 0 < a 1 ≤ E x 0 ( τ 1 ) ≤ a 2 Ergodicity assumption: P ( x , B ) = E x 0 ✶ B ( x τ 1 ) ∀ B ∈ B ( E ) satisfies: there exists a positive measure m on E s.t. 0 < m ( E ) ≤ 1 and P ( x , B ) ≥ m ( B ) ∀ B ∈ B ( E ) Example: reflected diffusions and reflected diffusions with jumps for which the transition density satisfies p ( x , t , x ′ ) ≥ k ( ε ) on E × [ ε, ∞ [ × E MR (UP-S) Ergodic Impulse Control with Constraint 7 / 19

Solution HJB Equation HJB Equation Heuristic argument with the discounted problem � � τ 1 � �� u α Mu α e − α t f d t + e − ατ 1 u α 0 ( x , 0) = min 0 ( x , 0) , E x 0 0 ( x τ 1 , 0) 0 Let m α = inf u α 0 ( x , 0), w α 0 = u α 0 − m α , � � τ 1 � �� w α Mw α e − α t ( f − α m α ) d t + e − ατ 1 w α 0 ( x , 0) = min 0 ( x , 0) , E x 0 0 ( x τ 1 , 0) 0 Assuming w α 0 → w 0 a function, and α m α → µ 0 a constant, � � τ 1 � �� w 0 ( x , 0) = min Mw 0 ( x , 0) , E x 0 ( f − µ 0 ) d t + w 0 ( x τ 1 , 0) 0 One can also use heuristic argument on � � T − t � � u T 0 ( t , x , y ) = inf E ν ✶ θ i ≤ T − t c ( x i − 1 f d t + , ξ i ) . x 0 θ i 0 i MR (UP-S) Ergodic Impulse Control with Constraint 8 / 19

Solution HJB Equation HJB Equation (2) - For the auxiliary problem: Find ( w 0 , µ 0 ), µ 0 constant, such that � � τ 1 � �� w 0 ( x , 0) = min Mw 0 ( x , 0) , E x 0 [ f − µ 0 ] d s + w 0 ( x τ 1 , 0) 0 then w 0 ( x , y ), for y > 0, is given by � � τ 1 � w 0 ( x , y ) = E xy [ f − µ 0 ] d s + w 0 ( x τ 1 , 0) 0 - For the initial problem: ( w 0 , µ 0 ) gives w ( x , y ) as � � τ 1 � w ( x , y ) = E xy [ f − µ 0 ] d s + w 0 ( x τ 1 , 0) 0 MR (UP-S) Ergodic Impulse Control with Constraint 9 / 19

Solution Solution ( µ 0 , w 0 ) Solution ( µ 0 , w 0 ) � � - A discrete time HJB equation for µ 0 , w 0 ( x , 0) : define � � τ 1 � ℓ ( x ) = E x 0 f ( x s , y s ) d s , Pg ( x ) = E x 0 g ( x τ 1 ) 0 τ ( x ) = E x 0 τ 1 , w 0 ( x ) ≡ w 0 ( x , 0) , then � � � � w 0 ( x ) = min inf c ( x , ξ ) + w 0 ( ξ ) , ℓ ( x ) − µ 0 τ ( x ) + Pw 0 ( x ) ξ ∈ Γ( x ) is equivalent to the previous HJB equation Proposition There exists a solution ( µ 0 , w 0 ) in R + × C ( E ) of the HJB equation. Remark: If w 0 is solution, w 0 + constant is also solution. The uniqueness of µ 0 will come from the stochastic interpretation. MR (UP-S) Ergodic Impulse Control with Constraint 10 / 19

Solution Solution ( µ 0 , w 0 ) Arguments The proof uses the following equivalent form of the HJB equation � � w 0 ( x ) = inf ℓ ( ξ ) + ✶ ξ � = x c ( x , ξ ) − µ 0 τ ( ξ ) + Pw 0 ( ξ ) = Rw 0 ξ ∈ Γ( x ) ∪{ x } and the fact that P ( x , β ) ≥ τ ( x ) γ ( β ) for a positive measure γ on E satisfying γ ( E ) > 1 − β τ ( x ) , 0 < β < 1. Then R is a contraction on C ( E ). w 0 is the unique fixed point and � µ 0 = w 0 ( x ) γ ( d x ) E MR (UP-S) Ergodic Impulse Control with Constraint 11 / 19

Solution Existence of an Optimal Control Existence of an Optimal Control Additional assumptions: the (uncontrolled) Markov process ( x t , y t ) has a unique invariant measure ζ and there exists a continuous function h ( x , y ) s.t. � � τ � [ f ( x t , y t ) − ¯ E xy h ( x τ , y τ ) = h ( x , y ) − E xy f ] d t , 0 for any finite stopping time τ , with � ¯ f = E × R + f ( x , y ) d ζ. Remark: if f ( x , y ) = f ( x ), then it is sufficient to assume that the Poisson equation for x t , i.e. − A x h = f ( x ) − ¯ f has a continuous solution. MR (UP-S) Ergodic Impulse Control with Constraint 12 / 19

Solution Existence of an Optimal Control (2) Existence of an Optimal Control (2) Theorem With the additional assumption, we have � ˜ � µ 0 = inf J ( x , 0 , ν ) , ν ∈ V 0 and there exists an optimal control ˆ ν 0 of the auxiliary problem case 1. µ 0 = ¯ f : then it is optimal to “do nothing” case 2. µ 0 < ¯ f : one can rewrite the HJB equation � ˜ ψ ( x ) , ˜ � w ( x ) = min ˜ ℓ ( x ) + P ˜ w ( x ) w = w 0 − h ( x , 0) , ˜ ψ = Mw 0 − h ( x , 0) , ˜ ℓ ( x ) = (¯ with ˜ f − µ 0 ) E x 0 τ 1 . This is the HJB equation of a discrete optimal stopping problem which has an optimal control w ( x n ) = ˜ � � � � η = inf ˆ n ≥ 0 : ˜ ψ ( x n ) , i.e., ˆ η = inf n ≥ 0 : w 0 ( x n ) = Mw 0 ( x n ) where x n is the Markov chain x τ n . From this, we deduce an optimal control with � � θ 1 = τ ˆ η , and θ i = θ ˆ η i with ˆ η i = inf n ≥ ˆ η i : w 0 ( x n ) = Mw 0 ( x n ) . MR (UP-S) Ergodic Impulse Control with Constraint 13 / 19

Solution Existence of an Optimal Control (3) Existence of an Optimal Control (3) Corollary � ˜ � µ 0 = inf J ( x , y , ν ) , v ∈ V and the optimal control ˆ ν obtained by translation by τ 1 of the control ˆ ν 0 . The final result is given by Theorem µ 0 = inf { J ( x , y , ν ) : ν ∈ V} = J ( x , y , ˆ ν ) MR (UP-S) Ergodic Impulse Control with Constraint 14 / 19

Solution Existence of an Optimal Control (4) Existence of an Optimal Control (4) A first step is to prove: Proposition ( µ 0 , w 0 ) being the solution previously obtained and recalling that � � τ 1 � w ( x , y ) = E x 0 [ f − µ 0 ] d s + w 0 ( x τ 1 , 0) 0 ( µ 0 , w ) is solution of − A xy w ( x , y ) + λ ( x , y )[ w ( x , 0) − Mw ( x , 0)] + = f − µ 0 . where A xy is the (weak) infinitesimal generator of the uncontrolled process A xy ϕ = A x ϕ + ∂ϕ � � ∂ y + λ ( x , y ) ϕ ( x , 0) − ϕ ( x , y ) . MR (UP-S) Ergodic Impulse Control with Constraint 15 / 19

Solution Existence of an Optimal Control (5) Existence of an Optimal Control (5) To prove the proposition, one first shows w 0 ( x ) = min { w ( x , 0) , Mw ( x , 0) } which gives � � τ 1 [ f − µ 0 ] d t + w ( x τ 1 , 0) − [ w ( x τ 1 , 0) − Mw ] + � w ( x , y ) = E xy 0 from which we deduce that equation. ✷ Next, the proposition allows us to show that � T M T = [ f ( x t , y t ) − µ 0 ] d s + w ( x T , y T ) is a submartingale. 0 This gives µ 0 ≤ J ( x , y , ν ), ∀ ν ∈ V , and, from the first expression of w ( x , y ), on obtains µ 0 = J ( x , y , ν ). MR (UP-S) Ergodic Impulse Control with Constraint 16 / 19