Optimal decentralized control of coupled subsystems with control - PowerPoint PPT Presentation

Optimal decentralized control of coupled subsystems with control sharing Aditya Mahajan McGill University IEEE Conference on Decision and Control, 2011

A Mahajan (McGill) Control sharing info struc 1 Notation Random variables: 𝑌 , realizations: � , state spaces: 𝒴 . 𝑏 􀐊 􀐕 means that variable 𝑏 belongs to subsystem � at time � . 𝑏 􀍣:􀐕 = �𝑏 􀍣 , 𝑏 􀍤 , …, 𝑏 􀐕 � 𝐛 = �𝑏 􀍣 , 𝑏 􀍤 , …, 𝑏 􀐏 � .

A Mahajan (McGill) Control sharing info struc Controller with control sharing Objective Control-coupled subsystems 2 System Model � 􀍣 � 􀍤 ⋯ � 􀐏 􀐕 􀐕 􀐕 � 􀐏 � 􀍣 � 􀍤 ⋯ 􀐕 􀐕 􀐕 � 􀐏 � 􀍣 � 􀍤 􀐕 􀐕 􀐕 𝐯 􀐕􀍭􀍣 𝐯 􀐕􀍭􀍣 𝐯 􀐕􀍭􀍣 � 􀐊 􀐕􀍬􀍣 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝐯 􀐕 , � 􀐊 � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 � 􀍣:􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � 􀏻 min all policies 𝐡 𝔽 [ ∑ � 􀐕 �𝐲 􀐕 , 𝐯 􀐕 �] 􀐕􀍮􀍣

A Mahajan (McGill) Control sharing info struc 3 Some applications Feedback communication systems (physical layer) Point-to-point real-time source coding, multi-terminal source coding with feedback, some classes of multiple access channel with feedback Queueing networks (media access layer) Multi-access broadcast, some classes of decentralized scheduling and routing. Cellular networks Paging and registration in cellular networks

A Mahajan (McGill) Control sharing info struc 4 Conceptual difficulties The system has non-classical information structure Data at each controller is increasing with time Is part of this data redundant? Can part of this data be compressed to a sufficient statistic? Multi-stage decision making How does current control action affect future estimation? its control action? � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀍣:􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � What information does controller � communicate to controller � via

A Mahajan (McGill) Other non-classical info-structures with sharing Belief sharing: Yüksel, 2009 Periodic sharing: Ooi, Verbout, Ludwig, Wornell, 1997 Walrand, 1979, Nayyar, Mahajan, and Teneketzis, 2011 Witsenhausen 1971, Varaiya and Delayed (observation) sharing: Delayed state sharing: Aicadri, Davoli, and Minciardi, 1987 Reduces to one-step delayed sharing pattern Control sharing info struc embed the observations in control Exploit the fact that the action space is continuous and compact to Considered the LQG version of the problem Athans, 1974) Control sharing info-structure (Bismut, 1972, Sandell and Literature Overview 5 Partial history sharing: Mahajan, Nayyar, Teneketzis, 2008

A Mahajan (McGill) wlo, wlo, Control sharing info struc Second structural result (based on common info approach of MNT 2008) Dynamic programming decomposition First structural result (based on person-by-person opt.) 6 Outline of the results � 􀐊 􀍣:􀐕􀍭􀍣 is redundant for optimal performance. � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � Define Π 􀐊 􀐕 ��� = ℙ�𝑌 􀐊 􀐕 = � | 𝐕 􀍣:􀐕􀍭􀍣 � and 𝚸 􀐕 = �Π 􀍣 􀐕 , …, Π 􀐏 􀐕 � . 𝝆 􀐕 is a sufficient statistic of 𝐯 􀍣:􀐕􀍭􀍣 for optimal performance. � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝝆 􀐕 �

A Mahajan (McGill) 7 Structural result based on person-by-person optimality Main lemma The states processes are conditionally independent given the past control actions. Control sharing info struc Implications 􀐏 ℙ�𝑌 􀐊 􀍣:􀐕 = � 􀐊 ℙ�𝐘 􀍣:􀐕 = 𝐲 􀍣:􀐕 | 𝐕 􀍣:􀐕 � = ∏ 􀍣:􀐕 | 𝐕 􀍣:􀐕 � 􀐊􀍮􀍣 Fix � 􀍭􀐊 and consider optimal design of � 􀐊 . Let 𝑆 􀐊 􀐕 = �𝑌 􀐊 􀐕 , 𝐕 􀍣:􀐕􀍭􀍣 � . Then {𝑆 􀐊 􀐕 , � = �, …} is a controlled MDP with control action � 􀐊 􀐕 . ℙ�� 􀐊 􀐕􀍬􀍣 | � 􀐊 􀍣:􀐕 , � 􀐊 􀍣:􀐕 � = ℙ�� 􀐊 􀐕􀍬􀍣 | � 􀐊 􀐕 , � 􀐊 􀐕 � 𝔽[� 􀐕 �𝐲 􀐕 , 𝐯 􀐕 � | � 􀐊 􀍣:􀐕 , � 􀐊 􀍣:􀐕 ] = 𝔽[� 􀐕 �𝐲 􀐕 , 𝐯 􀐕 � | � 􀐊 􀐕 , � 􀐊 􀐕 ]

Structural result . . . (cont.) A Mahajan (McGill) Design difficulty Implication of person-by-person optimality argument Original model 8 Control sharing info struc Data at the controller is still increasing with time � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀍣:􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 � = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 �

A Mahajan (McGill) 9 A coordinator based on common information General idea proposed in (Mahajan, Nayyar, and Teneketzis 2008) Control sharing info struc � 􀍣 𝑌 􀍣 � 􀍣 � 􀍤 𝑌 􀍤 � 􀍤 􀐕 , 𝐕 􀍣:􀐕􀍭􀍣 􀐕 , 𝐕 􀍣:􀐕􀍭􀍣 􀐕 􀐕 􀐕 􀐕

A coordinator based on common information (cont.) Control sharing info struc 10 A Mahajan (McGill) � 􀍣 𝑌 􀍣 � 􀍣 � 􀍤 𝑌 􀍤 � 􀍤 􀐕 􀐕 􀐕 􀐕 􀐕 􀐕 �� 􀍣 􀐕 , � 􀍤 ℎ 􀐕 𝐕 􀍣:􀐕􀍭􀍣 􀐕 � where � 􀐊 􀐕 �⋅� = � 􀐊 􀐕 �⋅, 𝐯 􀍣:􀐕􀍭􀍣 �

A coordinator based on common information (cont.) A Mahajan (McGill) Control sharing info struc 11 Solution approach The coordinated system is a POMDP Identify the structure of optimal coordination strategies for the coordinated system Show that the coordinated system is equivalent to the original model Translate the structure of optimal coordination strategies to the original model

A Mahajan (McGill) 12 The coordinated system wlo, Structure of optimal coordination strategy Control sharing info struc State: 𝐲 􀐕 = �� 􀍣 􀐕 , …, � 􀐏 􀐕 � � 􀍣 𝑌 􀍣 � 􀍣 � 􀍤 𝑌 􀍤 � 􀍤 􀐕 􀐕 􀐕 􀐕 􀐕 􀐕 Observations: 𝐯 􀐕􀍭􀍣 = �� 􀍣 􀐕􀍭􀍣 , …, � 􀐏 􀐕􀍭􀍣 � �� 􀍣 􀐕 , � 􀍤 ℎ 􀐕 𝐕 􀍣:􀐕􀍭􀍣 􀐕 � Control actions: 𝐞 􀐕 = �� 􀍣 􀐕 , …, � 􀐏 􀐕 � , 􀐏 􀐕􀍭􀍣 : 􀐏 �𝒴 􀐊 → 𝒱 􀐊 𝒱 􀐊 ) Coordination rule: ℎ 􀐕 : ( ∏ ∏ 􀐕 � 􀐊􀍮􀍣 􀐊􀍮􀍣 𝐞 􀐕 = ℎ 􀐕 �𝐯 􀍣:􀐕􀍭􀍣 � Define Ξ 􀐕 = ℙ� state | history of observations � = ℙ�𝐲 | 𝐕 􀍣:􀐕􀍭􀍣 � . Then, 𝐞 􀐕 = ℎ 􀐕 �𝜊 􀐕 �

The coordinated system (cont.) A Mahajan (McGill) Control sharing info struc 13 Dynamic programming decomposition Salient features The optimization at each step is a functional optimization problem. (In our opinion) functional optimization at each step is the only way to circumvent the issue of signaling. � 􀐕 �𝜊� = min 𝐞 𝔽 [� 􀐕 �𝐘 􀐕 , 𝐕 􀐕 � + � 􀐕􀍬􀍣 �Ξ 􀐕􀍬􀍣 � | Ξ 􀐕 = 𝜊]

A Mahajan (McGill) Control sharing info struc Solve the DP for coordinated system. Dynamic programming decomposition Structural result wlo, system Translation of results back to the original 14 � 􀍣 𝑌 􀍣 � 􀍣 � 􀍤 𝑌 􀍤 � 􀍤 􀐕 􀐕 􀐕 􀐕 􀐕 􀐕 � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 � = ℎ 􀐊 􀐕 �𝜊 􀐕 ��� 􀐊 􀐕 � = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝜊 􀐕 � ℎ 􀐕 𝐕 􀍣:􀐕􀍭􀍣 �� 􀍣 􀐕 , � 􀍤 􀐕 � Choose � 􀐊 􀐕 �� 􀐊 􀐕 , 𝜊 􀐕 � = ℎ 􀐊 􀐕 �𝜊 􀐕 ��� 􀐊 􀐕 �

A Mahajan (McGill) Control sharing info struc 15 Further simplification of structural result Recall main lemma: The states processes are conditionally independent given the past control actions. Implication 􀐏 ℙ�𝑌 􀐊 􀍣:􀐕 = � 􀐊 ℙ�𝐘 􀍣:􀐕 = 𝐲 􀍣:􀐕 | 𝐕 􀍣:􀐕 � = ∏ 􀍣:􀐕 | 𝐕 􀍣:􀐕 � 􀐊􀍮􀍣 􀐏 𝜌 􀐊 􀐕 �� 􀐊 𝜊 􀐕 �𝐲� = ℙ�𝐘 􀐕 = 𝐲 | 𝐕 􀍣:􀐕􀍭􀍣 � = ∏ 􀐕 � 􀐊􀍮􀍣

Further simplification of structural result (cont.) while Control sharing info struc 16 Simplified structural result wlo, A Mahajan (McGill) Simplified dynamic programming decomposition Significant reduction is size. � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝜊 􀐕 � = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝝆 􀐕 � 𝜊 􀐕 ∈ Δ�𝒴 􀍣 × ⋯ × 𝒴 􀐏 � 𝝆 􀐕 ∈ Δ�𝒴 􀍣 � × ⋯ × Δ�𝒴 􀐏 � � 􀐕 �𝝆� = min 𝐞 𝔽 [� 􀐕 �𝐘 􀐕 , 𝐕 􀐕 � + � 􀐕􀍬􀍣 �𝚸 􀐕􀍬􀍣 � | 𝚸 􀐕 = 𝝆]

A Mahajan (McGill) Using person-by-person approach Using specific conditional independence due to the dynamics Control sharing info struc Using the common information approach of (NMT 2008, 2011) 17 Original: Recap of structural results � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀍣:􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝐯 􀍣:􀐕􀍭􀍣 � � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 􀐕 , 𝜊 􀐕 �, 𝜊 􀐕 = ℙ�𝐘 􀐕 | 𝐯 􀍣:􀐕􀍭􀍣 � � 􀐊 􀐕 = � 􀐊 􀐕 �� 􀐊 𝜌 􀐊 􀐕 = ℙ�𝑌 􀐊 􀐕 , 𝝆 􀐕 �, 􀐕 | 𝐯 􀍣:􀐕􀍭􀍣 �