Compositionality for Markov Reward Chains with Fast Transitions J. - PowerPoint PPT Presentation

Compositionality for Markov Reward Chains with Fast Transitions J. Markovski, A. Sokolova, N. Trˇ cka, E. P. de Vink presented by Elias Pschernig January 24, 2008

Outline Introduction Motivation Recapitulation: Markov Chains Aggregation methods Discontinuous Markov reward chains Ordinary lumping Reduction Markov reward chains with fast transitions τ -lumping τ -reduction Relational properties Parallel composition

Markov Reward Chains ◮ Among most important and wide-spread analytical performance models ◮ Ever growing complexity of Markov reward chain systems ◮ Compositional generation: Composing a big system from several small components ◮ State space explosion: Result size is product of sizes of components ◮ Need aggregation methods... ◮ ...and they should be compositional ◮ We consider special models of Markov reward chains: Discontinuous Markov reward chains and Markov reward chains with fast transitions

Markov Reward Chains ◮ Among most important and wide-spread analytical performance models ◮ Ever growing complexity of Markov reward chain systems ◮ Compositional generation: Composing a big system from several small components ◮ State space explosion: Result size is product of sizes of components ◮ Need aggregation methods... ◮ ...and they should be compositional ◮ We consider special models of Markov reward chains: Discontinuous Markov reward chains and Markov reward chains with fast transitions

� � � � � � � Recapitulation: Discrete time Markov chains Transition probability ma- ���� ���� trix: 0 . 4 1 2 3 1 0.4 0.3 0.3 1 2 0.1 0.9 0 ���� ���� 3 0.9 0.1 0 0 . 3 ◮ Graphs with nodes 0 . 1 representing states 0 . 3 2 ◮ Outgoing arrows 0 . 9 ���� ���� determine stochastic 0 . 9 behavior of each state 0 . 1 ◮ Probabilities only 3 depend on current state

� � � Continuous time Markov chains ���� ���� 1 2 1 ���� ���� ���� � ���� 1 2 3 2 1 2 3 1 -3 1 2 ◮ Generator matrix: = Q 2 0 -2 2 3 0 1 -1

� � � ���� ���� Continuous time Markov reward chains r 0 0 . 8 1 λ τ ���� ���� � ���� ���� ν r 1 r 2 0 . 1 0 . 1 2 3 µ ◮ P = ( σ, Q , ρ ) ◮ σ is a stochastic row initial probability vector (0 . 8 , 0 . 1 , 0 . 1) ◮ ρ is a state reward vector ( r 0 , r 1 , r 2 ) ◮ Transition probability matrix ∞ Q n t n � = e Qt P ( t ) = n ! n =0 ◮ Rewards are used to measure performance (application dependent).

� � � Discontinuous Markov reward chains ◮ Markov chains with instantaneous transitions → discontinuous Markov chains ◮ Discontinuous Markov reward chain: P = ( σ, Π , Q , ρ ) ◮ Intuition: Π[ i , j ] denotes probability that a process occupies two states via an instantaneous transition. ���� ���� ���� � ���� ���� � ���� ���� ���� ◮ Π = I leads to a standard Markov chain → generalization 1 2 3 4

Discontinuous Markov reward chains Aggregation for discontinuous Markov reward chains ◮ Ordinary lumping ◮ Reduction

Ordinary lumping ◮ We lump P = ( σ, Π , Q , ρ ) to P = ( σ, Π , Q , ρ ) ◮ Partition L is an ordinary lumping L ◮ P → P

Ordinary lumping L ◮ P → P ◮ Partition of the state space into classes ◮ States lumped together form a class ◮ Equivalent transition behavior to other classes (intuitively: probability of class is sum of probabilities of states) ◮ All states in a class have the same reward, total reward is preserved

� � � � � � � � � � � Example ���� ���� L ���� ���� ◮ P → P r 1 0 . 5 r 1 0 . 5 1 1 µ λ ���� ���� ���� ���� ���� ���� ρ ρ λ + µ r 2 r 2 r 2 0 . 5 0 . 2 0 . 3 2 , 3 2 3 {{ 1 } , { 2 , 3 } , { 4 , 5 }} ρ � � � � � � � ���� ���� � ���� ���� ���� ���� 1 � 2 λ � λ λ � � 1 � 2 λ � � r 3 � r 3 r 3 � 4 5 4 , 5

Reduction ◮ We reduce P = ( σ, Π , Q , ρ ) to P = ( σ, I , Q , ρ ) ◮ P → r P ◮ Result is unique up to state permutation. ◮ Canonical product decomposition of Π ◮ Reduced states are given by ergodic classes of the original process (ergodic = each state can be reached from each other state in finite time) ◮ Total reward is preserved

Markov reward chains with fast transitions Markov reward chains with fast transitions ◮ Definition ◮ Aggregation

Markov reward chains with fast transitions ◮ Adds parameterized (“fast”) transitions to a standard Markov reward chain. ◮ Uses two generator matrixes Q s and Q f , for slow and fast transitions. ◮ P = ( σ, Q s , Q f , ρ ) is a function... ◮ ...where to each τ > 0 a Markov reward chain P τ = ( σ, I , Q s + τ Q f , ρ ) is assigned ◮ The limit τ → ∞ makes fast transitions instantaneous, and we end up with a discontinuous Markov reward chain.

Markov reward chains with fast transitions Aggregation for Markov reward chains with fast transitions ◮ τ -lumping ◮ τ -reduction

τ -lumping ◮ We τ -lump P = ( σ, Q s , Q f , ρ ) to P = ( σ, Q s , Q f , ρ ) ◮ Can define it using the limiting discontinuous Markov reward chain. L ◮ P � P ◮ Not unique

� � � � � � � � � � � � � � � τ -lumping P P L (fast transitions) (lumped fast transitions) ∞ ∞ Q Q L (discontinuous) (lumped discontinuous)

τ -reduction ◮ We τ -reduce P = ( σ, Q s , Q f , ρ ) to R = ( σ, I , Q , ρ ) ◮ P � r R

� � � � � � Example ���� ���� ◮ P � r R ���� ���� 1 ���� ���� 1 λ � � � �������� a � b a + b λ a + b λ � ���� ���� ���� ���� � 2 � � � � � � �������� � � ���� ���� a τ ���� ���� b τ τ -reduction 2 , 3 2 , 4 � � � � � � � �������� � � ���� ���� 3 4 � µ ρ � � � � � � �������� � � ���� ���� � 5 µ � ρ � � 5

� � � � � � � � � � � � � � � τ -reduction P (fast transitions) ∞ r R = R ′ Q (discontinuous) (continuous) r ◮ if P � r R ◮ and P → ∞ Q → r R ′ ◮ then R = R ′

Relational properties of ordinary lumping and τ -lumping ◮ Reduction works in one step, so no need to look at details of its relational properties. Lumping: ◮ Need transitivity and strong confluence... ◮ ...to ensure that iterative application yields a uniquely determined process. ◮ Repeated application of ordinary lumping... ◮ ...can be replaced by single application of composition of individual lumpings. ◮ For τ -lumping, only the limit is uniquely determined.

Relational properties of ordinary lumping and τ -lumping ◮ Reduction works in one step, so no need to look at details of its relational properties. Lumping: ◮ Need transitivity and strong confluence... ◮ ...to ensure that iterative application yields a uniquely determined process. ◮ Repeated application of ordinary lumping... ◮ ...can be replaced by single application of composition of individual lumpings. ◮ For τ -lumping, only the limit is uniquely determined.

� � � � � � � � � � � � � Example ���� ���� r 1 1 ���� ���� 1 r 2 1 ���� ���� ���� ���� 1 , 2 a τ cr 2+ br 3 r 2 1 b + c ���� ���� 2 1 , 2 , 3 c τ b τ {{ 1 , 2 } , r 3 {{{ 1 , 2 } , { 3 }} , ���� ���� { 3 } , ���� ���� c τ b b τ 3 µ b + c λ {{ 4 }}} { 4 }} � r 3 r 4 ���� ���� 3 µ 4 λ r 4 ���� ���� µ λ 4 r 4 4 {{ 1 , 2 , 3 } , { 4 }}

Parallel composition ◮ P 1 ≥ P 1 , P 2 ≥ P 2 = ⇒ P 1 � P 2 ≥ P 1 � P 2 ◮ Aggregate smaller components first... ◮ ...then combine them into the aggregated complete system. ◮ ≥ is semantic preorder. ◮ P ≥ P means P is an aggregated version of P . ◮ � is a parallel composition.

Composing discontinuous Markov reward chains ◮ Kronecker sum ⊕ and Kronecker product ⊗ ◮ Parallel composition P 1 � P 2 = ( σ 1 ⊗ σ 2 , Π 1 ⊗ Π 2 , Q 1 ⊗ Π 2 + Π 1 ⊗ Q 2 , ρ 1 ⊗ 1 | ρ 2 | + 1 | ρ 1 | ⊗ ρ 2 ) ◮ If P 1 and P 2 are discontinuous Markov reward chains, then so is P 1 � P 2

Composing discontinuous Markov reward chains ◮ Both lumping and reduction are compositional with respect to the parallel composition of discontinuous Markov reward chains L 1 L 2 L 1 ⊗L 2 ◮ If P 1 → P 1 and P 2 → P 2 , then P 1 � P 2 → P 1 � P 2 . ◮ If P 1 → r P 1 and P 2 → r P 2 , then P 1 � P 2 → r P 1 � P 2