Markov Chains: Classification of States IE 502: Probabilistic - PowerPoint PPT Presentation

Markov Chains: Classification of States IE 502: Probabilistic Models Jayendran Venkateswaran IE & OR IEOR @ IITBombay IE502: Probabilistic Models

Classification of States • State j is accessible from state i iff, starting in i , it is possible to enter j at some time (or step). • Two states i and j that are accessible to each other are said to communicate ( i ↔ j ) • Communication is an equivalence relation – i ↔ i – If i ↔ j then j ↔ i – If i ↔ j and j ↔ k then i ↔ k IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Class • Two states that communicate are in same class . – MC can have 1 or more classes. – Classes are either disjoint or identical. • If the Markov chain has only one class, then the chain is irreducible – That is, all states communicate with one another • Identify the classes of the following MCs: 0.4 0.6 0 0.4 0.6 0 0.2 0.7 0.1 0.7 0.3 0 0 0.5 0.5 0.2 0.4 0.4 IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Absorbing State • State i is called absorbing if it is impossible to leave it (i.e. p ii = 1). • A Markov chain is absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step) – When a process reaches an absorbing state, then it said to be absorbed. IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Transient and Recurrent states • State i is said to be transient if the probability to return to i is less than 1. – That is, State i is transient if, starting in state i , there is a non-zero probability that we never return to i • State i is said to be recurrent if the probability that, starting in state i , the process will ever return to state i is 1. IEOR @ IITBombay IE502: Probabilistic Models

Transient and Recurrent states (2) • If state i is transient then, starting in i, the process visits state i finite number of times • If state i is recurrent then, starting in i, the process will reenter state i again and again, infinitely often. • Recurrence is a class property  Transience is also a class property! • In a finite state MC, all states cannot be transient! IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Periodicity • State i has period d if, starting in state i , any return to i must occur in multiples of d time steps • State with period 1 is said to be aperiodic . • Determine period of state 0 in the following chains 1 p 1/2 1/2 0 1 2 1/2 1/2 1 0 1 2 3 1/2 1/2 1-p 1 • Periodicity is a class property – If state i has period d and i ↔ j then state j has period d • Markov chain is aperiodic if all states are aperiodic IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Ergodicity • If state i is recurrent, and (starting in i ) the mean time to return to i is finite, then it is said to be positive recurrent . • If mean time to return is infinite, then it is null recurrent . ( Example null recurrent chain ) 1/4 1/3 1/2 2/3 1 . . . 0 1 2 3 1/2 • Positive recurrence is a class property – In a finite state Markov chain all recurrent states are positive recurrent IEOR @ IITBombay IE502: Probabilistic Models

Classification of States: Ergodicity (2) • Positive recurrent, aperiodic states are called as ergodic • Ergodicity is a class property • If all states are ergodic then the chain is called as Ergodic Markov Chain IEOR @ IITBombay IE502: Probabilistic Models

Example • Consider the chain: 1/2 1/2 0 0 0 3/4 1/4 0 0 0 0 0 0 1 0 0 0 1 0 0 1/4 0 1/4 0 1/2 • Identify classes: are they transient or recurrent? • Comment on the periodicity of the classes – Is the Markov Chain periodic or aperiodic? • Comment on the ergodicity of the classes – Is the Markov Chain ergodic? IEOR @ IITBombay IE502: Probabilistic Models

Terminology • " Transition into state i " – enter state i from some other state – Change or move from other state to state i • " Transition out of state i " – Leave state i and go to some other state – Change or move to another state from state i • Stages, Steps, Round of a game, time step and time could be used interchangeably. Learn to interpret correctly given the problem context. IEOR @ IITBombay IE502: Probabilistic Models