Computing limit expectations of imprecise continuous-time Markov - PowerPoint PPT Presentation

Computing limit expectations of imprecise continuous-time Markov chains Alexander Erreygers Jasper De Bock WPMSIIP 2018 Ghent University, ELIS, SYSTeMS

? without explicitly evaluating 1 Can we determine E ∞ ( f ) ( ) ) n f I + t ( t → + ∞ E M 0 lim lim n Q n → + ∞

Continuous-time Markov chains

Basic set-up Objective [homogeneity] [Markov property] 2 2. time parameter is continuous 1. state space is finite Assumptions making inferences about the state X t of some system 3. dynamics are non-deterministic, Markovian & homogeneous X t 1 X t n X t X t + ∆ t 1 t n t t + ∆ P ( X t + ∆ = y | X t 1 = x 1 , . . . , X t n = x n , X t = x )

Basic set-up Objective [homogeneity] [Markov property] 2 2. time parameter is continuous 1. state space is finite Assumptions making inferences about the state X t of some system 3. dynamics are non-deterministic, Markovian & homogeneous X t 1 X t n X t X t + ∆ t 1 t n t t + ∆ P ( X t + ∆ = y | X t 1 = x 1 , . . . , X t n = x n , X t = x ) = P ( X t + ∆ = y | X t = x )

Basic set-up Objective [homogeneity] [Markov property] 2 2. time parameter is continuous 1. state space is finite Assumptions making inferences about the state X t of some system 3. dynamics are non-deterministic, Markovian & homogeneous X t 1 X t n X 0 X ∆ t 1 t n 0 ∆ P ( X t + ∆ = y | X t 1 = x 1 , . . . , X t n = x n , X t = x ) = P ( X t + ∆ = y | X t = x ) = P ( X ∆ = y | X 0 = x )

Characterisation A homogeneous CTMC is fully characterised by [nonnegative ofg-diagonal elements and zero row sums] 3 1. a (finite) state space X ; 2. an initial distribution π 0 ; [ P ( X 0 = x ) = π 0 ( x ) ] 3. a transition rate matrix Q .

Marginal expectations 1. solve the difgerential equation i.e., evaluate 2. compute 4 How do we compute E ( f ( X t )) ? d d τ T τ f = QT τ f with initial condition T 0 f = f [Note: T τ f : X → R ] E ( f ( X t )) = E π 0 ( T t f )

Marginal expectations i.e., evaluate 2. compute 4 1. solve the difgerential equation How do we compute E ( f ( X t )) ? d d τ T τ f = QT τ f with initial condition T 0 f = f [Note: T τ f : X → R ] ) n ( I + t T t f = e tQ f : = lim nQ f n → + ∞ E ( f ( X t )) = E π 0 ( T t f )

5 Typical temporal behaviour of T t f T t f t

Limit expectations In many applications, one is interested in the limit expectation Definition (Ergodicity) or equivalently, 6 E ∞ ( f ) : = lim t → + ∞ E ( f ( X t )) = lim t → + ∞ E π 0 ( T t f ) . The transition rate matrix Q is ergodic if, for all f : X → R , E ∞ ( f ) does not depend on π 0 t → + ∞ T t f is a constant function. lim

Imprecise continuous-time Markov chains

Thomas Krak, Jasper De Bock, and Arno Siebes. “Imprecise Approximate Reasoning 88 (2017), pp. 452–528 Jasper De Bock. “The Limit Behaviour of Imprecise Science 27.1 (2017), pp. 159–196 7 continuous-time Markov chains”. In: International Journal of Continuous-Time Markov Chains”. In: Journal of Nonlinear

Characterisation An imprecise CTMC is characterised by Problem These sets do not characterise a single CTMC! Solution Consider the set of stochastic processes that is consistent with and : the set of consistent homogeneous CTMCs, the set of consistent CTMCs, the set of consistent stochastic processes. 8 1. a (finite) state space X ; 2. an initial distribution π 0 ; 3. a transition rate matrix Q .

Characterisation An imprecise CTMC is characterised by Problem These sets do not characterise a single CTMC! Solution Consider the set of stochastic processes that is consistent with and : the set of consistent homogeneous CTMCs, the set of consistent CTMCs, the set of consistent stochastic processes. 8 1. a (finite) state space X ; 2. a set of initial distributions M 0 ; 3. a set of transition rate matrices Q .

Characterisation An imprecise CTMC is characterised by Problem These sets do not characterise a single CTMC! Solution Consider the sets of stochastic processes that are consistent with the set of consistent CTMCs, the set of consistent stochastic processes. 8 1. a (finite) state space X ; 2. a set of initial distributions M 0 ; 3. a set of transition rate matrices Q . M 0 and Q : P HM M 0 , Q the set of consistent homogeneous CTMCs,

Characterisation An imprecise CTMC is characterised by Problem These sets do not characterise a single CTMC! Solution Consider the sets of stochastic processes that are consistent with 8 1. a (finite) state space X ; 2. a set of initial distributions M 0 ; 3. a set of transition rate matrices Q . M 0 and Q : P HM M 0 , Q the set of consistent homogeneous CTMCs, P M M 0 , Q the set of consistent CTMCs, P M 0 , Q the set of consistent stochastic processes.

Lower envelopes lower envelope [superadditive, nonneg. hom., ~ zero row sums, ~ nonneg. ofg-diagonal elements] is defined by operator . The lower transition rate They also define a lower envelope of lower envelope Krak et al. (2017) define the coherent lower expectations 9 lower envelope P HM → E HM − − − − − − − − M 0 , Q M 0 , Q P M → E M − − − − − − − − M 0 , Q M 0 , Q − − − − − − − − → E M 0 , Q P M 0 , Q

Lower envelopes lower envelope [superadditive, nonneg. hom., ~ zero row sums, ~ nonneg. ofg-diagonal elements] lower envelope Krak et al. (2017) define the coherent lower expectations 9 lower envelope P HM → E HM − − − − − − − − M 0 , Q M 0 , Q P M → E M − − − − − − − − M 0 , Q M 0 , Q − − − − − − − − → E M 0 , Q P M 0 , Q They also define a lower envelope of Q . The lower transition rate operator Q : L ( X ) → L ( X ) is defined by [ Q f ]( x ) : = inf { [ Q f ]( x ) : Q ∈ Q } .

Marginal lower expectations Observe that This implies that 10 P M 0 , Q ⊇ P M M 0 , Q ⊇ P HM M 0 , Q . E M 0 , Q ( f ( X t )) ≤ E M M 0 , Q ( f ( X t )) ≤ E HM M 0 , Q ( f ( X t )) . Furthermore, Krak et al. (2017) show that [under some conditions on Q ] E M 0 , Q ( f ( X t )) = E M M 0 , Q ( f ( X t )) ≤ E HM M 0 , Q ( f ( X t )) .

11 i.e., evaluate 1. solve the difgerential equation 2. compute Determining E M 0 , Q ( f ( X t )) How do we compute E M 0 , Q ( f ( X t )) ? d d τ T τ f = QT τ f with initial condition T 0 f = f [Note: T τ f : X → R ] ) n ( I + t T t f = lim nQ f n → + ∞ E ( f ( X t )) = E π 0 ( T t f )

11 i.e., evaluate 1. solve the difgerential equation 2. compute Determining E M 0 , Q ( f ( X t )) How do we compute E M 0 , Q ( f ( X t )) ? d d τ T τ f = QT τ f with initial condition T 0 f = f [Note: T τ f : X → R ] Underline all the operators! ) n ( I + t T t f = lim nQ f n → + ∞ E ( f ( X t )) = E π 0 ( T t f )

11 i.e., evaluate 1. solve the difgerential equation 2. compute Determining E M 0 , Q ( f ( X t )) How do we compute E M 0 , Q ( f ( X t )) ? d d τ T τ f = QT τ f with initial condition T 0 f = f [Note: T τ f : X → R ] ) n ( I + t T t f = lim nQ f . n → + ∞ E M 0 , Q ( f ( X t )) = E M 0 ( T t f )

12 Typical temporal behaviour of T t f T t f t

Limit expectations We now turn to the limit lower expectations is a constant function. or equivalently, does not depend on , is ergodic if, for all The lower transition rate operator Definition (De Bock, 2017) 13 and E ∞ ( f ) : = lim t → + ∞ E M 0 ( T t f ) t → + ∞ E M = lim t → + ∞ E M 0 , Q ( f ( X t )) = lim M 0 , Q ( f ( X t )) E HM t → + ∞ E HM ∞ ( f ) : = lim M 0 , Q ( f ( X t )) .

Limit expectations We now turn to the limit lower expectation Definition (De Bock, 2017) or equivalently, 13 E ∞ ( f ) : = lim t → + ∞ E M 0 ( T t f ) . The lower transition rate operator Q is ergodic if, for all f : X → R , E ∞ ( f ) does not depend on M 0 t → + ∞ T t f is a constant function. lim

? without explicitly evaluating 13 Can we determine E ∞ ( f ) ( ) ) n f I + t ( t → + ∞ E M 0 lim lim n Q n → + ∞

A well-known result Theorem operator: 14 If Q is an ergodic transition rate matrix, then for all δ > 0 with δ ∥ Q ∥ < 2 , E ∞ is the unique ( I + δ Q ) -invariant expectation E ∞ (( I + δ Q ) f ) = E ∞ ( f ) for all f ∈ L ( X ) ⇔ E ∞ ( Q f ) = 0 for all f ∈ L ( X ) . + simply solve the linear system of equations π ∞ Q = 0

A well-known result Theorem operator: 14 If Q is an ergodic transition rate matrix, then for all δ > 0 with δ ∥ Q ∥ < 2 , E ∞ is the unique ( I + δ Q ) -invariant expectation E ∞ (( I + δ Q ) f ) = E ∞ ( f ) for all f ∈ L ( X ) Underline all the operators? ⇔ E ∞ ( Q f ) = 0 for all f ∈ L ( X ) . + simply solve the linear system of equations π ∞ Q = 0

¿ some alternative general upper bound ? Conjecture ¿ effjcient way to solve this “set of equations” ? 15 Explicitly determining E ∞ If Q is an ergodic lower transition rate operator, then 1. for all δ > 0 with δ ∥ Q ∥ < 2 , E ∞ is the unique ( I + δ Q ) -invariant lower expectation operator: E ∞ (( I + δ Q ) f ) = E ∞ ( f ) for all f ∈ L ( X ) ; 2. E ∞ ( Q f ) = 0 for all f ∈ L ( X ) .