Relations between irreducible and absorbing Markov chains G. Rubino AMS Sectional Meeting Riverside, November 2019 G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 1 / 20
Introduction • For illustration purposes, or for reference, I will take as global application areas of the talk, the quantitative analysis of (complex) systems using Markov models. This can be decomposed into • performance evaluation of systems, • dependability evaluation of systems. • The typical model for performance evaluation is the queue. The Markov model is typically irreducible, it lives in continuous time, and the analysis is typically performed in equilibrium. Both finite and infinite processes appear frequently. • In dependability, the model is typically absorbing (sooner or later the system is dead), it is also in continuous time, and in general the process is finite. G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 2 / 20
A reference performance model • X : the M / M / 1 queue λ λ λ λ λ λ · · · n n + 1 · · · 0 1 2 µ µ µ µ µ µ Figure: The M / M / 1 basic model. • The queue is stable iff λ < µ . In this case, the stationary distribution is given by π n = ( 1 − ρ ) ρ n , n ∈ N , with ρ = λ/µ < 1. • Typical metric of interest: the mean # of units in the queue, � ρ E ( X ( ∞ )) = n π n = 1 − ρ. n ∈ N G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 3 / 20
A classic dependability model • 2 independent components with failure rate λ ; if one alive and one down, the latter repairs the former with rate µ ; if both down, system is dead. 2 λ λ 2 1 0 µ Figure: A basic dependability model. • Observe that π is trivial here, but useless: taking the states in the � � order ( 2 , 1 , 0 ) , we have, whatever the initial state, π = 0 0 1 . • Typical object of interest: the r.v. “system lifetime T ” . For instance, starting at state 2, E ( T ) = 3 λ + µ . 2 λ 2 G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 4 / 20
Solving • Computing the stationary distribution π of an irreducible chain, or computing E ( T ) with T the absorption time of an absorbing chain, are both linear problems: • in the first case, π is the unique solution to the equilibrium equations having unit norm; • in the second case, denote by P u , v the transition probability of moving from u to v ( P u , v = transition rate Q u , v of u → v divided by departure rate d u from u ); then, writing α x = P ( X ( 0 ) = x ) and denoting τ x = E ( T | X ( 0 ) = x ) , we have that ( τ x ) is the solution to the system � τ x = 1 + P x , y τ y , for all state x � = a . d x y : y � = x , a • Next result connects the two problems. G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 5 / 20
First result • Theorem 1: • Start from an absorbing chain X on the finite state space S , with a single absorbing state a and initial distribution α . • To avoid trivialities, assume that α a = 0 and that from every state x � = a there is a path to state a . This means that every state x � = a is transient. • Build an irreducible process Y from X by adding to X the transition ( a , x ) with rate r α x , for all x � = a , where r is an arbitrary real > 0. • Let π be the stationary distribution of Y and let T be the absorption time of X . Then, E ( T ) = 1 − π a , r π a or equivalently, 1 π a = 1 + r E ( T ) . G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 6 / 20
Example • Using previous example in dependability, the Y irreducible model is the following: 2 λ λ 2 1 0 µ r α 1 r α 2 Figure: Y : X with feedback, where X is previous 3-state dependability example; α 2 + α 1 = 1. • On Y , we have � � � � 1 r ( α 2 λ + µ ) r π = = π 2 π 1 π 0 . 1 r ( α 2 λ + µ ) + r 2 λ 2 λ λ + 1 2 λ 2 G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 7 / 20
• If we use α 2 = 1, α 1 = 0, we obtain � � 1 r ( λ + µ ) r π = 1 r ( λ + µ ) + r 2 λ 2 λ λ + 1 2 λ 2 and we easily check here the formulas of previous Theorem 1. G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 8 / 20
Proof • Based on the concept of pseudo-aggregation: • Let U be an irreducible Markov model and let C = ( C k ) k ∈K be a partition of its state space. • The aggregation of U with respect to C is the process V with values on K , defined by V ( t ) = k ⇐ ⇒ U ( t ) ∈ C k . • In general, V is not Markov, not homogenous, etc. • But we can always define a Markov process with values on K , the pseudo-aggregation of U w.r.t. C , using the stationary distribution π of U . • In continuous time, for instance, if Q is the rate matrix of U , the rate matrix � Q of V is defined by � x ∈ C m π x Q x , C ℓ � Q m ,ℓ = . π C m • Then, if � π denotes the stationary distribution of V , we have that for all class C k , π k = π C k . � G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 9 / 20
Cut Lemma • Consider an irreducible and ergodic Markov process X on the state space S , with stationary distribution π . • Let ( B , C ) be a partition of S . • Then, we have equilibrium or mean flow conservation between the two subsets of states B and C : � � π x Q x , C = π y Q y , B . x ∈ B y ∈ C • This is also called Kelly’s Lemma (Lemma 1.4 in Reversibility and Stochastic Networks , Frank Kelly, Cambridge University Press, 2011). • The Cut Lemma is a very useful tool in deriving equations necessarily satisfied by the stationary distribution of an irreducible Markov process, when such a distribution exists. G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 10 / 20
Cuts in absorbing chains • Consider an absorbing finite Markov chain X on S , with a single absorbing state a ; assume all states in S \ { a } are transient. • If we index the states putting a at the end, the limiting distribution � � of X is π = 0 0 · · · 0 1 , as stated before, trivial and useless. • Consider a partition ( B , C ) of S \ { a } . • Using Theorem 1, we can connect X to an irreducible process Y on S , then apply the Cut Lemma, and come back to X . • This idea leads to new properties now related to cuts in absorbing models. G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 11 / 20
Relevant variables in this context • The variables that appear following this idea are the σ i , x s and σ x s where � � ∞ � σ x = E 1 ( X ( t ) = x ) d t , 0 and � � ∞ � σ i , x = E 1 ( X ( t ) = x ) d t | X ( 0 ) = i . 0 • In words, σ x is the mean total time spent by X in state x until absorption, and σ i , x is the same average but conditional to starting at state i . � � • Matrix σ u , v u , v is also called the potential matrix of the chain. • How to compute these metrics? G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 12 / 20
• We have an absorbing chain X on the finite state space S , with a single absorbing state a and initial distribution α (with α a = 0). From every state x � = a there is a path to state a ; then, every state x � = a is transient. • The σ i , x s are the unique solution to the following linear system: fix i � = a ; for any x � = i , a , � σ i , x = P i , j σ j , x ; j : j � = a in the case of x = i , � σ x , x = 1 + P x , j σ j , x . d x j : j � = a • For the unconditional case, � σ x = α i σ i , x . i G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 13 / 20
“Conditional”Cut Lemma Using Theorem 1, we obtain • Theorem 2: In the context of previous setting, for any i ∈ B , � � � � = 1 + σ i , j Q j , C + Q j , a σ i , k Q k , B . j ∈ B k ∈ C Of course, if i ∈ C , we symmetrically write � � � � 1 + σ i , j Q j , C = σ i , k Q k , B + Q k , a . j ∈ B k ∈ C G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 14 / 20
• Equivalent formulation: Let N B , C be the r.v. “# of transitions from B to C before absorption”and analogously for N C , B , N B , a , etc. Observe that N B , a + N C , a = 1. We have � � • � j ∈ B σ i , j Q j , C = E i N B , C , � � • � k ∈ C σ i , k Q k , B = E i N C , B , � � • � j ∈ B σ i , j Q j , a = E i N B , a = P ( X gets absorbed from B ) . • After some algebra, this leads to E i ( N B , C ) = E i ( N C , B ) + P i ( X gets absorbed from C ) ( i ∈ B ) . • Again, we can write this in the case we called C the partition class containing the initial state i . G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 15 / 20
“Unconditional”Cut Lemma • Theorem 3: In the context of previous setting” � � � � Q j , C + Q i , a = α B + σ j σ k Q k , B . j ∈ B k ∈ C • Equivalent formulation: E ( N B , C ) + P ( X gets absorbed from B ) = E ( N C , B ) + α ( B ) . By symmetry, we also have E ( N B , C ) + α ( C ) = E ( N C , B ) + P ( X gets absorbed from C ) . G. Rubino (AMS Sectional Meeting) Irreducible & absorbing Riverside, November 2019 16 / 20
Recommend
More recommend
