Math 20, Fall 2017 Edgar Costa Week 9 Dartmouth College Edgar Costa Math 20, Fall 2017 Week 9 1 / 23
Absorbing Markov Chains • Markov chain is absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step). • In an absorbing Markov chain, a state which is not absorbing is called transient. Edgar Costa Math 20, Fall 2017 Week 9 2 / 23 • A state s i of a Markov chain is called absorbing if it is impossible to leave it (i.e., p ii = 1).
Absorbing Markov Chains • Markov chain is absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step). • In an absorbing Markov chain, a state which is not absorbing is called transient. Edgar Costa Math 20, Fall 2017 Week 9 2 / 23 • A state s i of a Markov chain is called absorbing if it is impossible to leave it (i.e., p ii = 1).
Absorbing Markov Chains • Markov chain is absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step). • In an absorbing Markov chain, a state which is not absorbing is called transient. Edgar Costa Math 20, Fall 2017 Week 9 2 / 23 • A state s i of a Markov chain is called absorbing if it is impossible to leave it (i.e., p ii = 1).
Example - Drunkard’s Walk he walks to the left or right with equal probability. He continues like this, until he reaches corner 4, which is a bar, or corner 0, which is his home. If he reaches either home or the bar, he stays there. Edgar Costa Math 20, Fall 2017 Week 9 3 / 23 A man walks along a four-block stretch of Park Avenue. If he is at corner 1 , 2 or 3,
Example - Drunkard’s Walk he walks to the left or right with equal probability. He continues like this, until he reaches corner 4, which is a bar, or corner 0, which is his home. If he reaches either home or the bar, he stays there. Edgar Costa Math 20, Fall 2017 Week 9 3 / 23 A man walks along a four-block stretch of Park Avenue. If he is at corner 1 , 2 or 3,
The Transition Matrix - Drunkard’s Walk 0 Week 9 Math 20, Fall 2017 Edgar Costa 4 3 2 1 4 / 23 0 1 2 3 4 P =
The Transition Matrix - Drunkard Walk 0 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 0 1 Edgar Costa Math 20, Fall 2017 Week 9 0 1 0 0 0 1 2 3 4 5 / 23 0 1 1 / 2 1 / 2 P = 1 / 2 1 / 2 1 / 2 1 / 2
Absorbing Markov Chain I Week 9 Math 20, Fall 2017 Edgar Costa t matrix. matrix, and Q is an t r t zero matrix, R is a nonzero t r identity matrix, 0 is an r • I is an r • The first t states are transient and the last r states are absorbing. 0 Canonical Form Absorb R Q Trans Absorb Trans P will have the following canonical form • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient 6 / 23
Absorbing Markov Chain I Week 9 Math 20, Fall 2017 Edgar Costa t matrix. matrix, and Q is an t r t zero matrix, R is a nonzero t r identity matrix, 0 is an r • I is an r • The first t states are transient and the last r states are absorbing. 0 Canonical Form Absorb R Q Trans Absorb Trans P will have the following canonical form • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient 6 / 23
Absorbing Markov Chain I Week 9 Math 20, Fall 2017 Edgar Costa t matrix. matrix, and Q is an t r t zero matrix, R is a nonzero t r identity matrix, 0 is an r • I is an r • The first t states are transient and the last r states are absorbing. 0 Canonical Form Absorb R Q Trans Absorb Trans P will have the following canonical form • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient 6 / 23
Absorbing Markov Chain 0 Week 9 Math 20, Fall 2017 Edgar Costa t matrix. matrix, and Q is an t r t zero matrix, R is a nonzero t r identity matrix, 0 is an r • I is an r • The first t states are transient and the last r states are absorbing. I 6 / 23 Canonical Form R Q Trans Trans will have the following canonical form • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient Absorb . ( ) P = Absorb .
Absorbing Markov Chain 0 Week 9 Math 20, Fall 2017 Edgar Costa t matrix. matrix, and Q is an t r t zero matrix, R is a nonzero t r identity matrix, 0 is an r • I is an r • The first t states are transient and the last r states are absorbing. I 6 / 23 Canonical Form R Q Trans Trans will have the following canonical form • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient Absorb . ( ) P = Absorb .
Absorbing Markov Chain Trans Week 9 Math 20, Fall 2017 Edgar Costa • The first t states are transient and the last r states are absorbing. I 0 R Canonical Form Q will have the following canonical form Trans • If there are r absorbing states and t transient states, the transition matrix states come first. • For an absorbing Markov chain, renumber the states so that the transient 6 / 23 Absorb . ( ) P = Absorb . • I is an r × r identity matrix, 0 is an r × t zero matrix, R is a nonzero t × r matrix, and Q is an t × t matrix.
Canonical Form Q n Week 9 Math 20, Fall 2017 Edgar Costa • What is the probability that the process will be absorbed? I 0 Trans • where Trans ij 7 / 23 • Recall that the entry p ( n ) of the matrix P n is the probability of being in the state s j after n steps, when the chain is started in state s i . Absorb . ( ? ) P n = Absorb .
Canonical Form Q n Week 9 Math 20, Fall 2017 Edgar Costa • What is the probability that the process will be absorbed? I 0 Trans • where Trans ij 7 / 23 • Recall that the entry p ( n ) of the matrix P n is the probability of being in the state s j after n steps, when the chain is started in state s i . Absorb . ( ? ) P n = Absorb .
Probability of Absorption Absorb Week 9 Math 20, Fall 2017 Edgar Costa I 0 Absorb Q n Trans Trans Theorem P n 0 Q n n lim In other words, is 1. In an absorbing Markov chain, the probability that the process will be absorbed 8 / 23
Probability of Absorption Q n Week 9 Math 20, Fall 2017 Edgar Costa I 0 Absorb Trans Theorem Absorb Trans P n lim is 1. In other words, In an absorbing Markov chain, the probability that the process will be absorbed 8 / 23 n → + ∞ Q n = 0 .
Probability of Absorption Theorem Week 9 Math 20, Fall 2017 Edgar Costa I 0 Q n Trans 8 / 23 Trans lim is 1. In other words, In an absorbing Markov chain, the probability that the process will be absorbed n → + ∞ Q n = 0 . Absorb . P n = ( ? ) Absorb .
How many steps until the process gets absorbed? then Week 9 Math 20, Fall 2017 Edgar Costa 1 Q n Q 2 Q I Equivalently, B 9 / 23 Write I 0 B n Q n Trans Trans Absorb . ( ) P n = , Absorb . ( I + Q + Q 2 + · · · + Q n − 1 ) R
How many steps until the process gets absorbed? 0 Week 9 Math 20, Fall 2017 Edgar Costa Equivalently, then Write I B n Q n Trans Trans 9 / 23 Absorb . ( ) P n = , Absorb . ( I + Q + Q 2 + · · · + Q n − 1 ) R − → B =? I + Q + Q 2 + · · · + Q n − 1 − → ?
The Fundamental Matrix I Week 9 Math 20, Fall 2017 Edgar Costa that it started in s i . I 0 0 Theorem 10 / 23 0 Q n matrix for P . For an absorbing Markov chain the matrix I − Q is invertible and ( I − Q ) − 1 = N = I + Q + Q 2 + · · · For an absorbing Markov chain the matrix N = ( I − Q ) − 1 is called the fundamental ( ( I + Q + Q 2 + · · · + Q n − 1 ) R ) ( ) N · R P n = − → ( as n → + ∞ ) Can you interpret the entries of B = N · R and N ? B i , j = probability of being absorbed by the state s j , given that it started in s i . N i , j = number of expected times that the process is in the transient state s j , given
Absorption Probabilities Time to Absorption Theorem Write and given that it started in s i . Proof idea: How did you compute the expected value of the geometric distribution? Edgar Costa Math 20, Fall 2017 Week 9 11 / 23 N = ( I − Q ) − 1 = I + Q + Q 2 + · · · + Q n + · · · B = N · R . • N i , j = number of expected times that the process is in the transient state s j , • B i , j = probability of being absorbed by the state s j , given that it started in s i .
Absorption Probabilities Time to Absorption Theorem Write and given that it started in s i . Proof idea: How did you compute the expected value of the geometric distribution? Edgar Costa Math 20, Fall 2017 Week 9 11 / 23 N = ( I − Q ) − 1 = I + Q + Q 2 + · · · + Q n + · · · B = N · R . • N i , j = number of expected times that the process is in the transient state s j , • B i , j = probability of being absorbed by the state s j , given that it started in s i .
Recommend
More recommend
