Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Dual Effect in Stochastic Optimization February 10, 2015 P. Carpentier Master MMMEF — Cours MNOS 2014-2015 162 / 267
Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Introduction During the first part of the course, we have studied open-loop stochastic optimization problems, that is, problems in which the decisions correspond to deterministic variables which minimize a cost function defined as an expectation. � � min j ( u , W ) . u ∈ U ad E We now entre the realm of closed-loop stochastic optimization, that is, the case where on-line information is available to the decision maker. The decisions are thus functions of information and correspond to random variables. � � min j ( U , W ) . U ∈U ad E P. Carpentier Master MMMEF — Cours MNOS 2014-2015 163 / 267
Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Lecture Outline Closed Loop Stochastic Optimization Problems 1 Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample Dual Effect in Stochastic Optimization 2 Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems P. Carpentier Master MMMEF — Cours MNOS 2014-2015 164 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Closed Loop Stochastic Optimization Problems 1 Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample Dual Effect in Stochastic Optimization 2 Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems P. Carpentier Master MMMEF — Cours MNOS 2014-2015 165 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Closed Loop Stochastic Optimization Problems 1 Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample Dual Effect in Stochastic Optimization 2 Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems P. Carpentier Master MMMEF — Cours MNOS 2014-2015 166 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Variables and Constraints The decision variable U is now a random variable and belongs to a functional space U . A canonical example is: U = L 2 (Ω , A , P ; U ). The contraints on the r.v. U may be of different nature: point-wise constraints dealing with the possible values of U : U ∈ U po = � U ∈ U , U ( ω ) ∈ U ad P -a.s. � , risk constraints, such as expectation or probability constraints: U ∈ U ri = � � � � U ∈ U , P Θ( U ) ≤ θ ≥ π , measurability constraints which express the fact that a given amount of information Y is available to the decision maker: U ∈ U me = � � U ∈ U , U is measurable w.r.t. Y . We will mainly concentrate on the measurability constraints. P. Carpentier Master MMMEF — Cours MNOS 2014-2015 167 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Compact Formulation of a Closed-Loop Problem Given a probability space (Ω , A , P ), the essential ingredients of a stochastic optimization problem are noise W : r.v. with values in a measurable space ( W , W ), decision U : r.v. with values in a measurable space ( U , U ), information Y : r.v. with values in a measurable space ( Y , Y ), a cost function: measurable mapping j : U × W → R . The σ -field generated by W (resp. Y ) is denoted F (resp. G ). With all these elements at hand, the problem is set as follows: � � min E j ( U , W ) . U � Y The notation U � Y (or equivalently U � G ) is used to express that the r.v. U is measurable w.r.t. to the σ -field generated by Y . P. Carpentier Master MMMEF — Cours MNOS 2014-2015 168 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Representation of Measurability Constraints Consider the information structure of the stochastic optimization problem in a compact form, that is, the measurability constraints U � Y . This information structure may be interpreted in different ways. From the functional point of view, using Doob’s Theorem, the decision U is expressed as a measurable function of Y : U = ϕ ( Y ) . In this setting, the decision variable becomes the function ϕ . From the algebraic point of view, the constraints are expressed in terms of σ -field, that is, � � � � σ ⊂ σ . U Y Question : how to take all these representations into account? P. Carpentier Master MMMEF — Cours MNOS 2014-2015 169 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Static Information Structure (SIS) This is the case when G = σ ( Y ) is fixed, defined independently of the decision U . Therefore, the terminology “static” expresses that the σ -field G constraining the decision cannot be modified by the decision maker. It does not imply that no dynamics is present in the problem formulation. 11 If the information Y is defined as a function of the noise W , that is, Y = h ( W ), it generates a static information structure. Note that it may happen that Y does depend on U whereas the σ -field G it generates remains fixed. Remember from now that SIS will be the “easy” case. 11 If time is involved in the optimization problem, a decision U t has to be taken at each time t , based on an information Y t , so that a measurability constraint U t � Y t is written at each time stage t . P. Carpentier Master MMMEF — Cours MNOS 2014-2015 170 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Dynamic Information Structure (DIS) (1) This is the situation when G = σ ( Y ) depends on U . For example, in the case where Y = h ( U , W ), the constraint reads U � h ( U , W ) , which yields a (seemingly) implicit measurability constraint. This is a source of huge complexity for stochastic optimization problems, known under the name of the dual effect of control. Indeed, the decision maker has to take care of the following double effect: � � on the one hand, his decision affects the cost E j ( U , W ) , on the other hand, she makes the information more or less constrained, that is, a less or more large admissible set. P. Carpentier Master MMMEF — Cours MNOS 2014-2015 171 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Dynamic Information Structure (DIS) (2) It will be easier to imagine such problems by explicitly introducing several agents which take decisions based on observations which may depend on decisions of other agents. Those agents may be a priori ordered. Then the notion of causality (who is “upstream” and who is “downstream”) becomes relevant, and it turns out that two notions are paramount for the level of difficulty of the problem: 1 who influences the available information of whom? 2 who knows more than whom? We will illustrate these subtle notions and questions in the case of stochastic optimal control, for which an “agent” takes a decision � � 0 , . . . , T − 1 at each time stage t of the time horizon . P. Carpentier Master MMMEF — Cours MNOS 2014-2015 172 / 267
Stochastic Optimization Formulation Closed Loop Stochastic Optimization Problems Stochastic Optimal Control Problem Dual Effect in Stochastic Optimization Witsenhausen’s Counterexample Closed Loop Stochastic Optimization Problems 1 Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample Dual Effect in Stochastic Optimization 2 Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems P. Carpentier Master MMMEF — Cours MNOS 2014-2015 173 / 267
Recommend
More recommend
