Markov Chains Transition Matrix P = ( p ij ) where p ij = P ( j | i ) - PowerPoint PPT Presentation

Markov Chains Transition Matrix P = ( p ij ) where p ij = P ( j | i ) . Defining Properties: 0 ≤ p ij ≤ 1 , � j p ij = 1 . Regular P n for some n > 0 has all positive entries only. Main Property: An equilibrium/fixed vector exists and is unique. v · P = v where v is a probability row vector. Absorbing There is an i such that p ii = 1 . If so, all OTHER entries in row i are 0; p ij = 0 for j � = i . Also there is a positive probability to go from any state to an absorbing state. MAY NOT BE AT THE FIRST STEP. Main Property: Long Term Trend. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 1 / 19

Examples � 1 � 0 1 0 . 5 0 . 5   0 . 4 0 . 3 0 . 3 0 0 0 . 3 0 0 . 7   2   0 . 2 0 . 2 0 . 3 0 . 3   0 0 . 6 0 0 . 4 � 0 . 2 � 0 . 8 3 0 . 5 0 . 5 � 0 � 1 4 1 0   0 . 2 0 . 8 0 0 0 . 8 0 . 2 5   0 . 8 0 . 2 0   0 1 0 0 0 1 6   1 0 0 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 2 / 19

Example Continued   0 . 68 0 . 16 0 . 16 For (5), P 2 =  . So the matrix is regular. 0 . 16 0 . 68 0 . 16  0 . 16 . 16 0 . 68 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 3 / 19

Application Example (Problem 25, Section 10.2) The probability that a complex assembly line works correctly depends only on whether the line worked the last time it was used. There is a 0 . 9 chance that the line will work correctly if it worked correctly the last time, and a 0 . 8 chance that it will work correctly if it failed last time. Set up a transition matrix, and find the long range probability that the line will work correctly. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 4 / 19

Answer. � 0 . 9 � 0 . 1 The matrix is . This is regular. We need to solve the system 0 . 8 0 . 2 � 0 . 9 , � 0 . 1 � � � � x , y · = x , y . 0 . 8 , 0 . 2 � � � � This is 0 . 9 x + 0 . 8 y , 0 . 1 x + 0 . 2 y = x , y We can rewrite it as � − 0 . 1 x + 0 . 8 y = 0 = 0. The two equations are x = 8 y . So the equilibrium 0 . 1 x − 0 . 8 y � � vector is 8 / 9 1 / 9 . In general, the coefficient matrix of the system is P T − I , and you set every equation equal to 0: [ P T − I | 0]. Transpose means that you interchange rows and columns. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 5 / 19

Regular Markov Chains, Summary 1 The matrix P has positive entries only. 2 There is a unique equilibrium (probability) vector V such that V · P = V .   V 1 . . . V n . . . 3 lim m →∞ P m = lim m →∞ v · P m = V . . . .  ,   . . .  . . . V 1 V n 4 To solve for V , you need to solve the system [ P t − I n × n | 0] . There is a unique solution satisfying 0 ≤ V j ≤ 1 and V 1 + · · · + V n = 1 . Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 6 / 19

Previous Example  − 0 . 1 0 . 8 0   0 0 0  � 0 � � 0 � − 9 − 1 1 1 / 9  �→  �→ 0 . 1 − 0 . 8 0 1 − 8 0 �→ �→   1 1 1 1 1 1 1 1 1 1 1 1 � 0 � � 1 � 1 1 / 9 0 8 / 9 �→ �→ 1 0 8 / 9 0 1 1 / 9 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 7 / 19

Absorbing Markov Chains 1 The matrix P has some p ii = 1 . You rearrange the states so that the � I � 0 matrix is P = . For each state there must a positive R Q probability for it to go to an equilibrium state EVENTUALLY. This means each row of FR must have at least one nonzero entry. � � 0 I 2 lim m →∞ P m = . The rows of ( I − Q ) − 1 R tell you ( I − Q ) − 1 R 0 the probabilities of ending up in the state corresponding to the column. The matrix F = ( I − Q ) − 1 is called the fundamental matrix. The number f ij give the number of visits to state j before being absorbed, given that the current state is i . The explanation comes from the fact that ( I − Q ) − 1 = I + Q + Q 2 + · · · + Q m + . . . appears when you compute P m . Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 8 / 19

Gambler’s Ruin Two players A and B , play a game. They toss a coin. If heads, then A pays B $1. If tails, B pays A $1. They start out with a total of $3. The game ends whenever on of the players is broke. Analyze the game. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 9 / 19

Examples   1 0 0 0 0 1 0 0   1   0 0 . 4 0 . 6 0   0 . 2 0 0 . 4 0 . 4   1 0 0 0 . 4 0 . 6 0 2   0 0 . 6 0 . 4 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 10 / 19

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 11 / 19

Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 12 / 19

Classification of States, Optional I Question: How to tell if a Markov Chain is regular. 1 We say j is accessible from i if p m ( i , j ) > 0 for some m > 0 . i , j communicate, i ≃ j , if p n ( i , j ) > 0 and p m ( j , i ) > 0 for some m , n > 0 . 2 i ≃ i 3 i ≃ j implies j ≃ i 4 i ≃ j and j ≃ k implies i ≃ j 5 The sets of equivalent states are called classes 6 A Markov Chain is irreducible if all states are in a single class. 7 Regular implies irreducible. 8 Absorbing states are single classes. 9 i is called transient if f i = P ( i for some m | i ) = 1 . 10 i is called recurrent if f i = P ( i for some m | i ) < 1 . Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 13 / 19

Classification of States, Optional II To find the equivalence classes: 1 Draw the states 2 Forget the probabilities on the diagonal 3 Join two states by an edge if p ( i , j ) > 0 and p ( j , i ) > 0 . 4 If only one is > 0 , draw an arrow. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 14 / 19

Genetics I Genetics somethimes studied the case when offspring from the same parents are mated; two of these offspring are mated,and so on. Let A be the dominant gene and a the recessive one. The original offspring can carry genes AA , Aa = aA , aa . There are six ways in which two offspring can mate: State Mating 1 AA & AA 2 AA & Aa 3 AA & aa 4 Aa & Aa 5 Aa & aa 6 aa & aa Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 15 / 19

Genetics II Model the inheritance of the genes by a Markov chain. Suppose that the offspring are randomly mated with each other. Find the transition matrix. 1 2 3 4 5 6 1 | 1 0 0 0 0 0 | 2 | 1 / 4 1 / 2 0 1 / 4 0 0 | P = 3 | 0 0 0 1 0 0 | 4 | 1 / 16 1 / 4 1 / 8 1 / 4 1 / 4 1 / 16 | 5 | 0 0 0 1 / 4 1 / 2 1 / 4 | 6 | 0 0 0 0 0 1 | The principle is that an offspring inherits one gene from the two of one parent with equal probability, and independently another gene from the other parent in the same way. The inheritances from the two parents are independent. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 16 / 19

Genetics III  AA 1 / 4   Aa & Aa = Aa 1 / 4 + 1 / 4 = 1 / 2  aa 1 / 4  AA & AA 1 / 16 = 1 / 4 · 1 / 4 AA & Aa 1 / 4 = 1 / 4 · 1 / 2 + 1 / 2 · 1 / 4 AA & aa 1 / 8 = 1 / 4 · 1 / 4 + 1 / 4 · 1 / 4 Aa & Aa 1 / 4 = 1 / 2 · 1 / 2 Aa & aa 1 / 8 = 1 / 2 · 1 / 4 + 1 / 4 · 1 / 2 aa & aa 1 / 16 = 1 / 4 · 1 / 4 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 17 / 19

Questions 1 Identify the absorbing states. 2 Find the matrix Q . 3 Find the fundamental matrix F . 4 If two parents with genes Aa are mated, find the number of pairs of offspring that can be expected before the either the dominant or the recessive gene no longer appears. 5 If two parents with the genes Aa are mated, find the probability that the recessive gene will disappear. Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 18 / 19

Answers For item (4), you start with state (4), and ask how many times you pass through states (2), (3), (4), (5) before being absorbed; the sum of entries in the row for (4) in the matrix F . For item (5), you add the entries in the matrix FR in the row for (4).  1 0 0 0 0 0  0 1 0 0 0 0     1 / 4 0 1 / 2 0 1 / 4 0   The matrix P is relabelled   0 0 0 0 1 0     1 / 16 1 / 16 1 / 4 1 / 8 1 / 4 1 / 4   0 1 / 4 0 0 1 / 4 1 / 2     1 / 4 0 1 / 2 0 1 / 4 0 0 0 0 0 1 0     So R =  and Q =     1 / 16 1 / 16 1 / 4 1 / 8 1 / 4 1 / 4    0 1 / 4 0 0 1 / 4 1 / 2 Dan Barbasch Math 1105 Chapter 10, November 6 Week of November 6 19 / 19