The Symmetric Two-State Chain Different Initial Distributions? Let - PowerPoint PPT Presentation

The Symmetric Two-State Chain Different Initial Distributions? Let µ ( 0 ) = [ p ( 1 − p )] be some initial distribution More Markov Chains: on the symmetric two state chain. Classification of States, What is µ ( n ) = µ ( 0 ) P n as n → ∞ ? Stationary Distribution Transition matrix P = Last time: P n = CS 70, Summer 2019 Observe : µ ( 0 ) = [ 1 1 2 ] is the only initial As n → ∞ , P n → 2 distribution such that Lecture 27, 8/8/19 1 / 23 2 / 23 3 / 23 Stationary Distribution Stationary Distribution: A Visual Initial Distributions: A Visual Let S , P be the states and transition matrix of a Markov chain. A distribution µ over states is π m (1) stationary or invariant if π 0 = [0 , 1 , 0] π 0 = [1 , 0 , 0]   m 0 . 8 0 . 2 0 π m (1) π m (1) π m = π 0 P m = π 0 0 0 . 3 0 . 7 .   0 . 6 0 . 4 0 π m (2) π m (2) π m (2) π m (3) Intuition: π m (3) π m (3) m m m 4 / 23 5 / 23 6 / 23

Asymmetric Two State Chain Loopy Two State Chain Irreducibility Similar example to the one before: A funny looking chain: A Markov chain is irreducible we can go from every state i ∈ S to every other state j ∈ S , possibly in multiple steps . Are these chains irreducible: Is there a stationary distribution? If so, what is it? Is there a stationary distribution? If so, what is it? Two state asymmetric chain? Gambling chain (from yesterday)? Q: When do we have a stationary distribution? When do we have exactly 1? 7 / 23 8 / 23 9 / 23 Long Run Behavior Irreducibility Implies... Break Theorem: Let I { X m = i } be an indicator for whether X m = i . Let S , P be an irreducible Markov chain. S is a finite set. How do we interpret the quantity below? The stationary π exists and is unique . If you were a random variable, which one would n − 1 For any initial µ ( 0 ) and all states i ∈ S : 1 � I { X m = i } you be and why? n m = 0 What happens as n → ∞ ? 10 / 23 11 / 23 12 / 23

Non-Loopy Two State Chain Two Scenarios... Periodicity A simple looking chain: For a state i , its periodicity is the gcd of the length of all tours (i.e. walks from i to i ). Examples: Asymmetric two state chain? Is there a stationary distribution? If so, what is it? Gambling chain from yesterday? If X 0 = a , what is X 1000000 ? What is X 1000001 ? 13 / 23 14 / 23 15 / 23 Periodicity + Irreducibility Cars and Trucks Cars and Trucks Let S , P define an irreducible Markov chain. Three out of every four trucks on the road are Step 2: Compute the stationary distribution. Then, every state has the same period . followed by a car, while only one out of every five cars is followed by a truck. What fraction of vehicles on the road are trucks? If the chain is also aperiodic , then as n → ∞ : Step 1: Draw the Markov Chain. P [ X n = 1 ] → π i 16 / 23 17 / 23 18 / 23

Markov Chain on a Graph Markov Chain on a Graph Sanity Check Let G be any loopless, connected graph. The unique stationary distribution π is given by: Let G be a complete graph. Each vertex represents a state, and at each What do we know about its long run behavior? π = vertex, we transition to a neighbor each with the same probability. Can we verify this? Q: Is this Markov chain irreducible? Let G be an odd cycle. What do we know about its long run behavior? 19 / 23 20 / 23 21 / 23 Sanity Check Summary Let G be a hypercube. ◮ Stationary distributions do not change when What do we know about its long run behavior? we multiply them by the transition matrix. ◮ Irreducible chains always have a unique stationary distribution. ◮ We can say something about fraction of What fraction of time does it spend on strings time spent in state i if a chain is irreducible with exactly k zeros? ◮ If an irreducible chain is also aperiodic, the probability of being in a state at any time far enough out approaches π i . Next week: Conceptual review! 22 / 23 23 / 23