Markov Processes in Isabelle/HOL Applications Probabilistic - PowerPoint PPT Presentation

CPP 2017 Johannes Hölzl (Technical University of Munich) Markov Processes in Isabelle/HOL

Applications Probabilistic programming, Continuous-time Markov Example proc x — randomised walk on proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes Next step is Normal distributed with variance 0 1. physical processes with errors, … chains (queuing theory, biological processes, …), 1

Example proc x — randomised walk on proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes Next step is Normal distributed with variance 0 1. physical processes with errors, … chains (queuing theory, biological processes, …), Continuous-time Markov 1 Applications Probabilistic programming,

Example proc x — randomised walk on proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes physical processes with errors, … Next step is Normal distributed with variance 0 1. chains (queuing theory, biological processes, …), 1 Applications Probabilistic programming, Continuous-time Markov

proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes Next step is Normal distributed with variance 0 1. — randomised walk on physical processes with errors, … chains (queuing theory, biological processes, …), 1 Applications Probabilistic programming, Continuous-time Markov Example proc x

proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes Next step is Normal distributed with variance 0 1. physical processes with errors, … chains (queuing theory, biological processes, …), 1 Applications Probabilistic programming, Continuous-time Markov Example proc x — randomised walk on R

proc stream m proc x do x Normal x 0 1 proc x return x Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes physical processes with errors, … chains (queuing theory, biological processes, …), 1 Applications Probabilistic programming, Continuous-time Markov Example proc x — randomised walk on R Next step is Normal distributed with variance 0 . 1.

Wanted A general method to construct processes: proc x do y K x proc y return y Formalization for Modelling Stochastic Processes 1 physical processes with errors, … chains (queuing theory, biological processes, …), Applications Probabilistic programming, Continuous-time Markov Example proc x — randomised walk on R Next step is Normal distributed with variance 0 . 1. proc ∈ R → m Pr ( stream ( R )) proc x = do { x ′ ← Normal ( x , 0 . 1 ) ω ← proc x ′ return x ′ · ω }

Formalization for Modelling Stochastic Processes chains (queuing theory, biological processes, …), physical processes with errors, … 1 Applications Probabilistic programming, Continuous-time Markov Example proc x — randomised walk on R Next step is Normal distributed with variance 0 . 1. proc ∈ R → m Pr ( stream ( R )) proc x = do { x ′ ← Normal ( x , 0 . 1 ) ω ← proc x ′ return x ′ · ω } Wanted A general method to construct processes: proc x = do { y ← K x ; ω ← proc y ; return ( y · ω ) }

Formalization for Modelling Stochastic Processes physical processes with errors, … 1 Applications Probabilistic programming, Continuous-time Markov chains (queuing theory, biological processes, …), Example proc x — randomised walk on R Next step is Normal distributed with variance 0 . 1. proc ∈ R → m Pr ( stream ( R )) proc x = do { x ′ ← Normal ( x , 0 . 1 ) ω ← proc x ′ return x ′ · ω } Wanted A general method to construct processes: proc x = do { y ← K x ; ω ← proc y ; return ( y · ω ) }

• Formalize Markov Processes • Transition Function • Construction Method • Basic Properties • Transition Rates • Construction Method • Properties • Discussion 2 Overview • Application: Continuous-Time Markov Chains

2 Markov Processes

Monad Combinators Map map m m Bind bind m Return return m Note: Functions are regular HOL functions, We use do theorems have measurability assumptions. notation 3 Giry Monad Monad on probability spaces Pr ( S ) for a measurable space S

Bind bind m Return return m Note: Functions are regular HOL functions, We use do theorems have measurability assumptions. notation 3 Giry Monad Monad on probability spaces Pr ( S ) for a measurable space S Monad Combinators Map map ∈ ( S → m T ) → ( Pr ( S ) → m Pr ( T ))

Return return m Note: Functions are regular HOL functions, We use do theorems have measurability assumptions. notation 3 Giry Monad Monad on probability spaces Pr ( S ) for a measurable space S Monad Combinators Map map ∈ ( S → m T ) → ( Pr ( S ) → m Pr ( T )) Bind bind ∈ Pr ( S ) → ( S → m Pr ( T )) → Pr ( T )

Note: Functions are regular HOL functions, We use do theorems have measurability assumptions. notation 3 Giry Monad Monad on probability spaces Pr ( S ) for a measurable space S Monad Combinators Map map ∈ ( S → m T ) → ( Pr ( S ) → m Pr ( T )) Bind bind ∈ Pr ( S ) → ( S → m Pr ( T )) → Pr ( T ) Return return ∈ S → m Pr ( S )

theorems have measurability assumptions. 3 Giry Monad Monad on probability spaces Pr ( S ) for a measurable space S Monad Combinators Map map ∈ ( S → m T ) → ( Pr ( S ) → m Pr ( T )) Bind bind ∈ Pr ( S ) → ( S → m Pr ( T )) → Pr ( T ) Return return ∈ S → m Pr ( S ) Note: Functions are regular HOL functions, We use do { . . . } notation

coalgebraic T S S S are the discrete distributions on S generalized K m is a measurable space on S , • nonnegative & rows sum up to 1 • • • are probability distributions on 4 Markov Kernels (a.k.a. stochastic relations) Transition functions for Markov chains on state spaces S traditional T : S → S → R

generalized K m is a measurable space on S , • nonnegative & rows sum up to 1 • • are probability distributions on 4 Markov Kernels (a.k.a. stochastic relations) Transition functions for Markov chains on state spaces S traditional T : S → S → R coalgebraic T : S → D ( S ) • D ( S ) are the discrete distributions on S

• nonnegative & rows sum up to 1 4 Markov Kernels (a.k.a. stochastic relations) Transition functions for Markov chains on state spaces S traditional T : S → S → R coalgebraic T : S → D ( S ) • D ( S ) are the discrete distributions on S generalized K ∈ S → m Pr ( S ) • S is a measurable space on S , • Pr ( S ) are probability distributions on S

5 . . . Extension Theorem by Ionescu-Tulcea There exists proc x ∈ S → m Pr ( stream ( S )) where do { y 1 ← K x y 2 ← K y 1 do { y 3 ← K y 2 y ← K x proc x y 4 ← K y 3 ω ← proc y = = y 5 ← K y 4 return ( y · ω ) y 6 ← K y 5 } return ( y 1 · y 2 · y 3 · y 4 · y 5 · · · ) }

Yes: Bisimulation implies equality Bisimulation relation R stream stream R M N exists K , and M N stream m • M do y K M y return y • N do y K N y return y K y R M y N y Bisimulation implies equality (a.k.a coinduction rule for equality) R bisimulation relation: R M N M N . 6 1. • , and , s.t. Uniqueness Ionescu-Tulcea proves existence. Is it unique?

Bisimulation relation R stream stream R M N exists K , and M N stream m • M do y K M y return y • N do y K N y return y K y R M y N y Bisimulation implies equality (a.k.a coinduction rule for equality) R bisimulation relation: R M N M N . 1. 6 • , and , s.t. Uniqueness Ionescu-Tulcea proves existence. Is it unique? Yes: Bisimulation implies equality

Bisimulation implies equality (a.k.a coinduction rule for equality) R bisimulation relation: R M N M N . 6 Uniqueness Ionescu-Tulcea proves existence. Is it unique? Yes: Bisimulation implies equality Bisimulation relation R : Pr ( stream ( S )) → Pr ( stream ( S )) → B R M N exists K ∈ Pr ( S ) , and M ′ , N ′ ∈ S → m Pr ( stream ( S )) s.t. • M = do { y ← K ; ω ← M ′ y ; return ( y · ω ) } , • N = do { y ← K ; ω ← N ′ y ; return ( y · ω ) } , and • Pr K { y | R ( M ′ y ) ( N ′ y ) } = 1.

6 Uniqueness Ionescu-Tulcea proves existence. Is it unique? Yes: Bisimulation implies equality Bisimulation relation R : Pr ( stream ( S )) → Pr ( stream ( S )) → B R M N exists K ∈ Pr ( S ) , and M ′ , N ′ ∈ S → m Pr ( stream ( S )) s.t. • M = do { y ← K ; ω ← M ′ y ; return ( y · ω ) } , • N = do { y ← K ; ω ← N ′ y ; return ( y · ω ) } , and • Pr K { y | R ( M ′ y ) ( N ′ y ) } = 1. Bisimulation implies equality (a.k.a coinduction rule for equality) R bisimulation relation: R M N = ⇒ M = N .

a a b c a 7 time Markov Property 0 . 5 c b 0 . 33 0 . 5 0 . 67 a 0 . 5 0 . 5

a b c a 7 time Markov Property 0 . 5 c b 0 . 33 0 . 5 0 . 67 a 0 . 5 0 . 5 a