Continuous Time Markov Chain Birth and Death Process IE 502: - PowerPoint PPT Presentation

Continuous Time Markov Chain Birth and Death Process IE 502: Probabilistic Models Jayendran Venkateswaran IE & OR

Continuous Time Markov Chain • { X ( t ), t ≥ 0} is a Continuous Time Markov Chain if, for all s , t ≥ 0 and for all i , j in the state space, P { X ( t+s ) = j | X ( s ) = i , X ( u ) = x ( u ), 0 ≤ u < s } = P { X ( t+s ) = j | X ( s ) = i } Markovian Property – X ( t+s ) is future state; X ( s ) is the current state; X ( u ) are past states – X ( t ) takes on discrete valued states, but change in state happens on continuous time! IE502: Probabilistic Models IEOR @ IITBombay

Waiting time in a state • Suppose a CTMC enters state i at some time. How long will the process be in state i before it transitions into a different state? That is, – After entering a state i , suppose the process has not left state i in the first 10 minutes. What is the probability the process will not leave i in the next 5 minutes? – T i : Time in state i before process transitions • In CTMC, the amount of time the process waits at a state has an exponential distribution. IE502: Probabilistic Models IEOR @ IITBombay

‘Hand on’ definition • In a CTMC, 1. T i , the amount of time process spends in each state i before transitioning into a different state is exponentially distributed with rate ν i . • T i ~ Expo( v i ) 2. Has a discrete-time Markov chain with transition matrix P (with no self-loops) • Why no self loops? IE502: Probabilistic Models IEOR @ IITBombay

Transition Probability Function • In DTMC , given the one-step transition probabilities p ij we can find p ij (n) the probability of being in various states after n timesteps . • In CTMC, transition probability function denotes the probability that the process presently in state i will be in state j a time t later. p i , j ( t ) = P{ X ( t+s ) = j | X ( s ) = i } • Chapman-Kolmogorov Equations for CTMC ∞ ∑ + = p ( t s ) p ( t ) p ( s ) ij ik kj = k 0 IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process • Let us identify by state X ( t )= n the condition of the system in which there are n objects . Given the system is in state n , new elements arrive at rate λ n , and leave at rate μ n . – λ n  Arrival rate or birth rate when process is in state n – μ n  Departure rate or death rate when in state n • In this CTMC, transitions can go from state n to either state n+ 1 or state n- 1 • State space representation IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process (contd..) • Relation between birth and death rates ( λ n & μ n ) state transition rates ( v i ) and transition probabilities ( p i , j ) • Scenarios – λ n = 0 → pure death process → only decrement – μ n = 0 → pure birth or Yule process → only increment – μ n = 0 , λ n = λ → Poisson process! • Now, let’s looks at the transition probability function IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process 1. Probability that process will be in state j in the next ∆t (or h ) time units • Three ways this can happen: • P { X ( t + h ) = j | X ( t ) = j - 1} = • P { X ( t + h ) = j | X ( t ) = j + 1} = • P { X ( t + h ) = j | X ( t ) = j } = 2. Now, probability that the process presently in state i will be in state j a time t later, p i , j ( t ), is … p i,j ( t ) = P { X ( t + s ) = j | X ( s ) = i } = P { X ( t ) = j | X (0) = i } • IE502: Probabilistic Models IEOR @ IITBombay

Birth and Death Process 3. Next, combining equation set 1 and 2, and taking the limit ∆t → 0, a set of linear differential equations can be derived for the transition probabilities ′ = µ − λ p ( t ) p ( t ) p ( t ) i , 0 1 i , 1 0 i , 0 ′ = λ + µ − λ + µ p ( t ) p ( t ) p ( t ) ( ) p ( t ) > , for j 0 − − + + i , j j 1 i , j 1 j 1 i , j 1 j j i , j • Note: Above equation is a flow balance equation • Variation of flow = inflow – outflow IE502: Probabilistic Models IEOR @ IITBombay

Limiting Probabilities • Probability that a CTMC will be in state j at time t converges to a limiting value ( P j ) which is independent of the initial state. = P lim p ( t ) j ij → ∞ t (Recall) Sufficient condition for limiting probabilities to exist: MC is irreducible, positive recurrent. • Now, if steady state solution exists, it is characterized by d p ( t ) ij ′ = = lim p ( t ) lim 0 ij d t → ∞ → ∞ t t IE502: Probabilistic Models IEOR @ IITBombay

Compute Limiting Probabilities 1. In steady state, rate of leaving = rate of entering (to see this, apply limits to the differential eqns) 2. General Equilibrium Results: Compute P n in terms of P n -1 – ∞ – Compute P n in terms of P 0 ∑ = P 1 n = n 0 3. The Normalizing condition must hold 4. Solve for P 0 and P n – What is the condition for the limiting probabilities to exist? IE502: Probabilistic Models IEOR @ IITBombay