progress on parameter synthesis for markov models
play

Progress on Parameter Synthesis for Markov Models Joost-Pieter - PowerPoint PPT Presentation

Dec 27, 2023 •173 likes •1.21k views

Progress on Parameter Synthesis for Markov Models Joost-Pieter Katoen Joint with: Christian Dehnert, Nils Jansen, Sebastian Junges, Tim Quatman, Erika brahm, Harold Bruintjes, Florian Corzilius, Ufuk Topcu, Murat Cubutceke, Ivan Papusha,

  1. Hierarchical SCC Decomposition [Jansen et al. , 2014] S S 1 0 . 8 0 . 2 2 3 0 . 4 q 1 1 1 − q 4 5 1 0 . 2 0 . 2 0 . 2 0 . 4 0 . 5 6 7 9 1 p 0 . 3 1 − p 0 . 8 8 S 2 . 1 S 2 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  2. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 0 . 2 2 3 0 . 4 q 1 1 1 − q 4 5 1 0 . 2 0 . 2 0 . 2 0 . 4 0 . 5 6 7 9 1 p 0 . 3 1 − p 0 . 8 8 S 2 . 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  3. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 0 . 2 2 3 0 . 4 q 1 1 1 − q 4 5 1 0 . 2 0 . 2 0 . 2 p 0 . 4 0 . 3 p 0 . 5 p 1 − p S 2 . 1 6 9 1 0 . 8 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  4. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 0 . 2 2 3 0 . 4 q 1 1 1 − q 4 5 1 0 . 2 0 . 2 0 . 2 p 1 − 0 . 3 p 0 . 5 p 0 . 4 1 − 0 . 3 p 1 − p 1 − 0 . 3 p S 2 . 1 6 9 1 0 . 8 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  5. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 0 . 2 2 3 0 . 4 q 1 1 1 − q 4 5 1 0 . 2 0 . 2 0 . 2 p 1 − 0 . 3 p 0 . 5 p 0 . 4 1 − 0 . 3 p 1 − p 1 − 0 . 3 p S 2 . 1 6 9 1 S 2 0 . 8 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  6. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 0 . 2 2 3 0 . 4 q 1 0 . 2 − 0 . 06 p 1 1 − q 1 − 0 . 7 p 4 5 1 0 . 2 0 . 16 p 1 − 0 . 7 p 0 . 4 0 . 8 − 0 . 8 p 1 − 0 . 7 p S 2 9 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  7. Hierarchical SCC Decomposition [Jansen et al. , 2014] S 1 0 . 8 0 . 2 2 3 0 . 4 q 1 0 . 2 − 0 . 06 p 1 1 − q 1 − 0 . 7 p 4 5 1 0 . 2 0 . 16 p 1 − 0 . 7 p 0 . 4 0 . 8 − 0 . 8 p 1 − 0 . 7 p S 2 9 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  8. Hierarchical SCC Decomposition [Jansen et al. , 2014] 0 . 8 − 0 . 8 q 1 − 0 . 8 q 0 . 2 S 12 S 13 1 − q 1 − 0 . 8 q 0 . 4 0 . 2 1 − 0 . 8 q 0 . 2 − 0 . 06 p 1 1 − 0 . 7 p 5 1 0 . 2 q 1 − 0 . 8 q 0 . 16 p 1 − 0 . 7 p 0 . 4 0 . 8 − 0 . 8 p 1 − 0 . 7 p S 2 9 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  9. Hierarchical SCC Decomposition [Jansen et al. , 2014] S 0 . 8 − 0 . 8 q 1 − 0 . 8 q 0 . 2 S 12 S 13 1 − q 1 − 0 . 8 q 0 . 4 0 . 2 1 − 0 . 8 q 0 . 2 − 0 . 06 p 1 1 − 0 . 7 p 5 1 0 . 2 q 1 − 0 . 8 q 0 . 16 p 1 − 0 . 7 p 0 . 4 0 . 8 − 0 . 8 p 1 − 0 . 7 p S 2 9 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  10. Hierarchical SCC Decomposition [Jansen et al. , 2014] − 0 . 2872 p − 0 . 52 q + 0 . 3192 pq + 0 . 52 − 0 . 6712 p − 0 . 744 q + 0 . 5432 pq + 0 . 904 S 5 1 − 0 . 384 p − 0 . 224 q + 0 . 224 pq + 0 . 384 − 0 . 6712 p − 0 . 744 q + 0 . 5432 pq + 0 . 904 9 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  11. Hierarchical SCC Decomposition [Jansen et al. , 2014] − 0 . 2872 p − 0 . 52 q + 0 . 3192 pq + 0 . 52 − 0 . 6712 p − 0 . 744 q + 0 . 5432 pq + 0 . 904 S 5 1 − 0 . 384 p − 0 . 224 q + 0 . 224 pq + 0 . 384 − 0 . 6712 p − 0 . 744 q + 0 . 5432 pq + 0 . 904 9 1 For which (combinations of) values for p and q is the probability of reaching5smaller than c ∈ [ 0 , 1 ] ? ⇒ Evaluate rational function. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 23/51

  12. Exploiting SMT Goal: partition parameter space in regions R that are either safe or unsafe Idea: generate region candidates R and ask SMT solver 2 for counterexample 2 Over non-linear real arithmetic using Z3 or SMT-RAT . Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 24/51

  13. Exploiting SMT Goal: partition parameter space in regions R that are either safe or unsafe Idea: generate region candidates R and ask SMT solver 2 for counterexample 2 Over non-linear real arithmetic using Z3 or SMT-RAT . Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 24/51

  14. CEGAR-Like Parameter Synthesis Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 25/51

  15. CEGAR-Like Parameter Synthesis For which 1 / 10 ⩽ p ⩽ 9 / 10 and 2 / 5 ⩽ q ⩽ 3 / 5 does Pr (◇ 2 ) ⩾ 3 / 20 hold? Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

  16. CEGAR-Like Parameter Synthesis For which 1 / 10 ⩽ p ⩽ 9 / 10 and 2 / 5 ⩽ q ⩽ 3 / 5 does Pr (◇ 2 ) ⩾ 3 / 20 hold? Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

  17. CEGAR-Like Parameter Synthesis For which 1 / 10 ⩽ p ⩽ 9 / 10 and 2 / 5 ⩽ q ⩽ 3 / 5 does Pr (◇ 2 ) ⩾ 3 / 20 hold? Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

  18. CEGAR-Like Parameter Synthesis For which 1 / 10 ⩽ p ⩽ 9 / 10 and 2 / 5 ⩽ q ⩽ 3 / 5 does Pr (◇ 2 ) ⩾ 3 / 20 hold? Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 26/51

  19. Experimental Results [Dehnert et al. , 2015] competitors ▸ PARAM [Hahn et al., 2010] ▸ PRISM [Parker et al., 2011] models ▸ Bounded retransmission protocol ▸ NAND multiplexing ▸ Zeroconf, Crowds protocol ▸ 10 4 to 7 . 5 ⋅ 10 6 states experiments: ▸ best set-up for each tool ▸ log-scale x - and y -axis runner-up in the CAV 2015 artefact evaluation Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 27/51

  20. Experimental Results [Dehnert et al. , 2015] competitors ▸ PARAM [Hahn et al., 2010] ▸ PRISM [Parker et al., 2011] ▸ prototype [Baier et al., 2014] models ▸ Bounded retransmission protocol ▸ NAND multiplexing ▸ Zeroconf, Crowds protocol ▸ 10 4 to 7 . 5 ⋅ 10 6 states experiments: ▸ best set-up for each tool ▸ log-scale x - and y -axis runner-up in the CAV 2015 artefact evaluation Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 27/51

  21. Parameter Synthesis using SMT Pros: ▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools PARAM and PRISM ▸ User-friendly representation Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

  22. Parameter Synthesis using SMT Pros: ▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools PARAM and PRISM ▸ User-friendly representation Cons: > 4 parameters? ▸ Rational function requires many gcd-computations ▸ SMT performance unpredictable heuristics hard Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

  23. Parameter Synthesis using SMT Pros: ▸ Exact results: rational function is an exact symbolic object ▸ Drastic improvements over existing tools PARAM and PRISM ▸ User-friendly representation Cons: > 4 parameters? ▸ Rational function requires many gcd-computations ▸ SMT performance unpredictable heuristics hard Can we do better by sacrificing exactness? Yes. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 28/51

  24. Approximate Parameter Synthesis [Quatmann et al, , 2016] Let transition probabilities be linear in each variable. That is, transition functions f are multi-affine multivariate polynomials of form: f = ∑ a i ⋅ (∏ x ) with a i ∈ Q x ∈ V Examples: 3 x ⋅ y + 4 y ⋅ z , 1 − x , x ⋅ y ⋅ z etc. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

  25. Approximate Parameter Synthesis [Quatmann et al, , 2016] Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

  26. Approximate Parameter Synthesis [Quatmann et al, , 2016] Two-phase approach: first remove dependencies, then substitute extremal values Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

  27. Approximate Parameter Synthesis [Quatmann et al, , 2016] Two-phase approach: first remove dependencies, then substitute extremal values Also applicable to parametric MDPs. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 29/51

  28. Phase 1: Relaxation Parameter dependencies are removed; Pr (◇ 2 ) = ( 1 − z ) ⋅ 1 − q 1 − p ⋅ q ⇒ each state is equipped with its own parameter Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 30/51

  29. Phase 1: Relaxation Correctness: ▸ Relaxed regions contain more valuations than original regions ⇒ Relaxation yields over-approximations ⇒ Relaxation preserves upper-bounds on reachability probs Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 31/51

  30. Phase 1: Relaxation Correctness: ▸ Relaxed regions contain more valuations than original regions ⇒ Relaxation yields over-approximations ⇒ Relaxation preserves upper-bounds on reachability probs Complexity of parameter synthesis : ▸ Relaxation increases the number of parameters ▸ Extremal values of the state parameters attain maximal probabilities ⇒ Valuations for maximal probabilities are easier to find Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 31/51

  31. Phase 2: Substitution Local parameters per state ⇒ extremal values at states suffice Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 32/51

  32. Phase 2: Substitution Local parameters per state ⇒ extremal values at states suffice Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 33/51

  33. Phase 2: Substitution This results in a Markov decision process. Its extremal reachability probabilities provide bounds for parametric MC. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 34/51

  34. Parameter Synthesis Until ≈ 95% of the parameter space is covered Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 35/51

  35. Parameter Synthesis Until 95% of the parameter space is covered Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 36/51

  36. Coverage # states # trans % p safe unsafe neither unkn ϕ n t brp E 2 20 744 27 651 48% 51 14.9% 79.2% 5.8% 0.2% 4 20 744 27 651 48% 71 7.5% 51.0% 40.6% 0.8% pMC E crowds 2 104 512 246 082 19% 44 54.4% 41.1% 4.2% 0.3% P nand P 2 35 112 52 647 47% 21 21.4% 68.5% 6.9% 3.2% brp 2 40 721 55 143 50% 153 6.6% 90.4% 3.0% 0.0% P pMDP cons P 4 22 656 75 232 41% 357 2.6% 87.0% 10.4% 0.0% sav P 4 379 1 127 50% 2 44.0% 15.4% 35.4% 5.3% zconf P 2 88 858 203 550 40% 186 16.6% 77.3% 5.6% 0.5% Parameter space R = [ 10 − 5 , 1 − 10 − 5 ] n until 95% coverage for n parameters for 625 equally-sized regions without region refinement single core, 2.0 GHz, 30GB RAM, TO = one hour Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 37/51

  37. Parametric Markov Chain Benchmarks PLA PRISM benchmark instance ϕ #pars #states #trans #regions direct bisim best (256,5) P 2 19 720 26 627 37 6 14 TO (4096,5) P 2 315 400 425 987 13 233 TO TO (256,5) E 2 20 744 27 651 195 8 15 TO (4096,5) E 2 331 784 442 371 195 502 417 TO brp (16,5) E 4 1 304 1 731 1 251 220 2 764 1 597 TO (32,5) E 4 2 600 3 459 1 031 893 TO 2 722 TO (256,5) E 4 20 744 27 651 – TO TO TO (10,5) P 2 104 512 246 082 123 17 6 2038 crowds (15,7) 2 8 364 409 25 108 729 116 1 880 518 TO P (20,7) P 2 45 421 597 164 432 797 119 TO 2 935 TO (10,5) 2 35 112 52 647 469 22 30 TO P nand (25,5) P 2 865 592 1 347 047 360 735 2 061 TO coverage of 95%; refinement into four equally-sized regions SMT approach needs > one hour on all instances. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 38/51

  38. Parametric MDP Benchmarks PLA PRISM benchmark instance ϕ #pars #states #trans #regions direct bisim best (256,5) 2 40 721 55 143 37 35 3 359 TO P brp (4096,5) P 2 647 441 876 903 13 3 424 TO TO < 1 < 1 (2,2) 2 272 492 119 31 P (2,32) P 2 4 112 7 692 108 113 141 TO consensus (4,2) 4 22 656 75 232 6 125 1 866 2 022 TO P (4,4) P 4 43 136 144 352 – TO TO TO < 1 < 1 (6,2,2) 2 379 1 127 162 TO P (100,10,10) P 2 1 307 395 6 474 535 37 1 612 TO TO sav (6,2,2) 4 379 1 127 621 175 944 917 TO P (10,3,3) P 4 1 850 6 561 TO TO TO (2) P 2 88 858 203 550 186 86 1 295 TO zeroconf (5) P 2 494 930 1 133 781 403 2 400 TO TO coverage of 95% Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 39/51

  39. Summary So Far SMT-based approach: ▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 10 6 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 40/51

  40. Summary So Far SMT-based approach: Parameter lifting approach: ▸ Exact ▸ Approximative ▸ Requires rational functions ▸ Off-the-shelf model checking ▸ Fickle SMT performance ▸ No SMT, no rational functions ▸ ≈ 10 6 states, 2 parameters ▸ ≈ 10 7 states, 4–5 parameters ▸ Restricted to Markov chains ▸ Applicable to MDPs and games ▸ CEGAR-like refinement ▸ CEGAR-like refinement Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 40/51

  41. Multiple Objectives Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

  42. Multiple Objectives Inputs: 1. a (finite) parametric MDP M over V = { x 1 ,... , x n } with signomial parameter functions c ⋅ x a 1 1 ⋅ ... ⋅ x a n n for c ∈ R 2. multiple objectives ϕ 1 ,... ,ϕ m (reachability, expected reward) N c k ⋅ x a 1 k ⋅ ... ⋅ x a nk for c k ∈ R ∑ 3. objective function f over V : n 1 k = 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

  43. Multiple Objectives Inputs: 1. a (finite) parametric MDP M over V = { x 1 ,... , x n } with signomial parameter functions c ⋅ x a 1 1 ⋅ ... ⋅ x a n n for c ∈ R 2. multiple objectives ϕ 1 ,... ,ϕ m (reachability, expected reward) N c k ⋅ x a 1 k ⋅ ... ⋅ x a nk for c k ∈ R ∑ 3. objective function f over V : n 1 k = 1 Output: A (randomised) policy σ and valuation u such that: M σ [ u ] ⊧ ϕ 1 ∧ ... ∧ ϕ m and the objective f is minimised �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� “optimality” “feasibility” Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

  44. Multiple Objectives Inputs: 1. a (finite) parametric MDP M over V = { x 1 ,... , x n } with signomial parameter functions c ⋅ x a 1 1 ⋅ ... ⋅ x a n n for c ∈ R 2. multiple objectives ϕ 1 ,... ,ϕ m (reachability, expected reward) N c k ⋅ x a 1 k ⋅ ... ⋅ x a nk for c k ∈ R ∑ 3. objective function f over V : n 1 k = 1 Output: A (randomised) policy σ and valuation u such that: M σ [ u ] ⊧ ϕ 1 ∧ ... ∧ ϕ m and the objective f is minimised �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� “optimality” “feasibility” multi-objective MDP: use LP [Etessami et al. , 2008] multi-objective parametric MDP: use special type NLP [Cubuktepe et al. , 2017] Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 41/51

  45. NLP for Two Objectives Objectives: minimise f , reach T with probability ⩽ p , expected cost to reach G ⩽ c Subject to: p s I ⩽ p reachability objective c s I ⩽ c expected reward objective Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

  46. NLP for Two Objectives Objectives: minimise f , reach T with probability ⩽ p , expected cost to reach G ⩽ c Subject to: p s I ⩽ p reachability objective c s I ⩽ c expected reward objective σ s ,α = 1 ∀ s ∶ ∑ randomised scheduler 0 ⩽ σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

  47. NLP for Two Objectives Objectives: minimise f , reach T with probability ⩽ p , expected cost to reach G ⩽ c Subject to: p s I ⩽ p reachability objective c s I ⩽ c expected reward objective σ s ,α = 1 ∀ s ∶ ∑ randomised scheduler 0 ⩽ σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) = 1 ∑ probabilistic choice ∀ s , t ,α ∶ 0 ⩽ P( s ,α, t ) ⩽ 1 t ∈ S Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

  48. NLP for Two Objectives Objectives: minimise f , reach T with probability ⩽ p , expected cost to reach G ⩽ c Subject to: p s I ⩽ p reachability objective c s I ⩽ c expected reward objective σ s ,α = 1 ∀ s ∶ ∑ randomised scheduler 0 ⩽ σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) = 1 ∑ probabilistic choice ∀ s , t ,α ∶ 0 ⩽ P( s ,α, t ) ⩽ 1 t ∈ S ∀ s ∈ T ∶ p s = 1 reach prob of T σ s ,α ⋅ ∑ ∀ s / ∈ T ∶ p s = P( s ,α, t )⋅ p t ∑ transition probabilities α ∈ Act ( s ) t ∈ S Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

  49. NLP for Two Objectives Objectives: minimise f , reach T with probability ⩽ p , expected cost to reach G ⩽ c Subject to: p s I ⩽ p reachability objective c s I ⩽ c expected reward objective σ s ,α = 1 ∀ s ∶ ∑ randomised scheduler 0 ⩽ σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) = 1 ∑ probabilistic choice ∀ s , t ,α ∶ 0 ⩽ P( s ,α, t ) ⩽ 1 t ∈ S ∀ s ∈ T ∶ p s = 1 reach prob of T σ s ,α ⋅ ∑ ∀ s / ∈ T ∶ p s = P( s ,α, t )⋅ p t ∑ transition probabilities α ∈ Act ( s ) t ∈ S ∀ s ∈ G ∶ c s = 0 expected cost of G σ s ,α ⋅ ( c ( s ,α ) + ∑ ∀ s / ∈ G ∶ c s = P( s ,α, t )⋅ c t ) ∑ expected costs α ∈ Act ( s ) t ∈ S Theorem: This NLP is sound and complete. But solving NLPs is exponential. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 42/51

  50. Can We Do Better? Yes. 1. Get a feasible solution in polynomial time 3 . How? Geometric programming. 2. Get local optimum. How? Sequential convex programming. Solutions are approximations that can be arbitrarily close. 3 Approximation of arbitrarily precise results by interior point methods with barriers Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 43/51

  51. Geometric Programming N Objective: minimise f ∶∶ c k ⋅ x a 1 k ⋅ ... ⋅ x a nk for c k ∈ R ⩾ 0 ∑ n 1 k = 1 Subject to: ∀ i ∈ [ 1 .. m ] ∶ g i ⩽ 1 posynomial g i ∀ j ∈ [ 1 ..ℓ ] ∶ h j = 1 monomial h j Division transformation: f ⩽ h if and only if f h ⩽ 1 Relaxation: f = h implies f ⩽ h if and only if f h ⩽ 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 44/51

  52. Convexification Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 45/51

  53. Lifting Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 46/51

  54. GP for Two Objectives Objectives: reach T with probability ⩽ p , expected cost to reach G ⩽ c ⩽ 1 Subject to: p sI reachability p ⩽ 1 c sI expected reward c Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

  55. GP for Two Objectives Objectives: reach T with probability ⩽ p , expected cost to reach G ⩽ c ⩽ 1 Subject to: p sI reachability p ⩽ 1 c sI expected reward σ s ,α ⩽ 1 c ∀ s ∶ ∑ randomised scheduler σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

  56. GP for Two Objectives Objectives: reach T with probability ⩽ p , expected cost to reach G ⩽ c ⩽ 1 Subject to: p sI reachability p ⩽ 1 c sI expected reward σ s ,α ⩽ 1 c ∀ s ∶ ∑ randomised scheduler σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) ⩽ 1 ∑ probabilistic choice t ∈ S ∀ s , t ,α ∶ P( s ,α, t ) ⩽ 1 Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

  57. GP for Two Objectives Objectives: reach T with probability ⩽ p , expected cost to reach G ⩽ c ⩽ 1 Subject to: p sI reachability p ⩽ 1 c sI expected reward σ s ,α ⩽ 1 c ∀ s ∶ ∑ randomised scheduler σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) ⩽ 1 ∑ probabilistic choice t ∈ S ∀ s , t ,α ∶ P( s ,α, t ) ⩽ 1 ∀ s ∈ T ∶ p s = 1 reach prob of T ∑ α σ s ,α ⋅ ∑ t ∈ S P( s ,α, t )⋅ p t ∀ s / ∈ T ∶ ⩽ 1 transition probabilities p s Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

  58. GP for Two Objectives Objectives: reach T with probability ⩽ p , expected cost to reach G ⩽ c ⩽ 1 Subject to: p sI reachability p ⩽ 1 c sI expected reward σ s ,α ⩽ 1 c ∀ s ∶ ∑ randomised scheduler σ s ,α ⩽ 1 α ∈ Act ( s ) ∀ s ,α ∶ ∀ s ,α ∶ P( s ,α, t ) ⩽ 1 ∑ probabilistic choice t ∈ S ∀ s , t ,α ∶ P( s ,α, t ) ⩽ 1 ∀ s ∈ T ∶ p s = 1 reach prob of T ∑ α σ s ,α ⋅ ∑ t ∈ S P( s ,α, t )⋅ p t ∀ s / ∈ T ∶ ⩽ 1 transition probabilities p s ∑ α σ s ,α ⋅ ( c ( s ,α ) + ∑ t ∈ S P( s ,α, t )⋅ c t ) ∀ s / ∈ G ∶ ⩽ 1 expected costs c s Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 47/51

  59. Correctness Use the objective function F now 4 p + ∑ p + ∑ ∑ 1 1 1 Minimise σ s ,α s ,α p ∈ V p ∈ L yields that all variables p , p and σ s ,α are maximised. Theorem: The GP with objective function F yields a feasible solution. Solving this GP can be done in polynomial time. 4 Note: the original objective function f is dropped. Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 48/51

  60. Experimental Results Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 49/51

  61. Experimental Results Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 50/51

  62. Epilogue SMT-based approach: ▸ Exact ▸ Requires rational functions ▸ Fickle SMT performance ▸ ≈ 10 6 states, 2 parameters ▸ Restricted to Markov chains ▸ CEGAR-like refinement Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

  63. Epilogue SMT-based approach: Parameter lifting approach: ▸ Exact ▸ Approximative ▸ Requires rational functions ▸ Off-the-shelf model checking ▸ Fickle SMT performance ▸ No SMT, no rational functions ▸ ≈ 10 6 states, 2 parameters ▸ ≈ 10 7 states, 4–5 parameters ▸ Restricted to Markov chains ▸ Applicable to MDPs and games ▸ CEGAR-like refinement ▸ CEGAR-like refinement Joost-Pieter Katoen Progress on Parameter Synthesis for Markov Models 51/51

Recommend

CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation Reminder: Proteins Read DNA and X-Inactivation E.g.: 2 copies (mom/dad) of each

1.28k views • 21 slides
A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu #

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu #

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu # Sham M. Kakade # @ University of California, Irvine # Microsoft Research, New England Outline 1. Latent class models and parameter estimation 2.

1.44k views • 82 slides
CSE 527 &quot; Markov Models and Hidden Markov Models !

CSE 527 &quot; Markov Models and Hidden Markov Models !

CSE 527 &quot; Markov Models and Hidden Markov Models ! http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat ! Dosage Compensation Reminder: Proteins Read DNA ! and X-Inactivation ! E.g.: ! 2 copies (mom/dad) of each chromosome

770 views • 17 slides
CSE 427 Markov Models and Hidden Markov Models 2

CSE 427 Markov Models and Hidden Markov Models 2

CSE 427 Markov Models and Hidden Markov Models 2 http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation and X-Inactivation 2 copies (mom/dad) of each chromosome 1-23 Mostly, both copies of each gene are

1.07k views • 60 slides
CSE P 527 Markov Models and Hidden Markov Models 1 2

CSE P 527 Markov Models and Hidden Markov Models 1 2

CSE P 527 Markov Models and Hidden Markov Models 1 2 http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation and X-Inactivation 2 copies (mom/dad) of each chromosome 1-23 Mostly, both copies of each gene are

970 views • 66 slides
Parameter synthesis for probabilistic real-time systems Marta Kwiatkowska Department of Computer

Parameter synthesis for probabilistic real-time systems Marta Kwiatkowska Department of Computer

Parameter synthesis for probabilistic real-time systems Marta Kwiatkowska Department of Computer Science, University of Oxford SynCoP 2015, London Quantitative (probabilistic) verification Result Probabilistic model System e.g. Markov chain

890 views • 59 slides
Hidden Markov Models (HMM) Many slides from Michael Collins and HMMs Overview I The Tagging

Hidden Markov Models (HMM) Many slides from Michael Collins and HMMs Overview I The Tagging

Hidden Markov Models (HMM) Many slides from Michael Collins and HMMs Overview I The Tagging Problem I Generative models, and the noisy-channel model, for supervised learning I Hidden Markov Model (HMM) taggers I Basic definitions I Parameter

968 views • 37 slides
Precise Parameter Synthesis for Stochastic Biochemical Systems Milan ska 1 , 2 , Frits

Precise Parameter Synthesis for Stochastic Biochemical Systems Milan ska 1 , 2 , Frits

Introduction Problem Formulation Parameter Synthesis Case Studies Conclusion Precise Parameter Synthesis for Stochastic Biochemical Systems Milan ska 1 , 2 , Frits Dannenberg 2 , Marta Kwiatkowska 2 , Nicola Paoletti 2 Ce Faculty of

579 views • 28 slides
1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

Hidden Markov Models 1 Hidden Markov Models Markov chains not so useful for most agents Need observations to update your beliefs Hidden Markov models (HMMs) Underlying Markov chain over states X You observe outputs (effects) at

617 views • 7 slides
Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov

Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov

Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov chains Hidden Markov Models (HMMs) Algorithms: Viterbi, forward, backward, posterior decoding Profile HMMs Baum-Welch algorithm 9001

1.16k views • 87 slides
CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2 adjacent nts, same strand (not Watson-Crick pair; p mnemonic for the phosphodiester bond of the DNA backbone) C of CpG is often (70-80%)

690 views • 55 slides
CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2 adjacent nts, same strand (not Watson-Crick pair; p mnemonic for the phosphodiester bond of the DNA backbone) C of CpG is often (70-80%)

780 views • 55 slides
Max Likelihood for Log-Linear Models Daphne Koller Log-Likelihood for Markov Nets A B C

Max Likelihood for Log-Linear Models Daphne Koller Log-Likelihood for Markov Nets A B C

Learning Probabilistic Graphical Parameter Estimation Models Max Likelihood for Log-Linear Models Daphne Koller Log-Likelihood for Markov Nets A B C Partition function couples the parameters No decomposition of likelihood No

799 views • 21 slides
HMM Review Lecture Outline 1. Markov models 2. Hidden Markov

HMM Review Lecture Outline 1. Markov models 2. Hidden Markov

HMM Review Lecture Outline 1. Markov models 2. Hidden Markov models 3. Viterbi algorithm MARKOV MODELS One View of Text Sequence of symbols (bytes,

489 views • 46 slides
Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models Luca

Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models Luca

Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models Luca Bortolussi 1 Simone Silvetti 2,3 1DMG, University of Trieste, Trieste, Italy lbortolussi@units.it 2DIMA, University of Udine, Udine, Italy 3Esteco

1.01k views • 48 slides
Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University 2 Markov Chains

Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University 2 Markov Chains

1 Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University 2 Markov Chains and Hidden Markov Models Modeling the statistical properties of biological sequences and distinguishing regions based on these models For the

437 views • 32 slides
Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University Markov Chains and

Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University Markov Chains and

Markov Chains and Hidden Markov Models COMP 571 Luay Nakhleh, Rice University Markov Chains and Hidden Markov Models Modeling the statistical properties of biological sequences and distinguishing regions based on these models For the

1.33k views • 96 slides
Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and

Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and

Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and slides are used from: 1. CS188 UC Berkeley 2. RN, AIMA Markov Models We have already seen that an MDP provides a useful framework for modeling

846 views • 62 slides
Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and

Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and

Markov Models and Hidden Markov Models Robert Platt Northeastern University Some images and slides are used from: 1. CS188 UC Berkeley 2. RN, AIMA Markov Models We have already seen that an MDP provides a useful framework for modeling

1k views • 69 slides
CSE 427 Markov Models and Hidden Markov Models How Proteins Read DNA E.g.:

CSE 427 Markov Models and Hidden Markov Models How Proteins Read DNA E.g.:

CSE 427 Markov Models and Hidden Markov Models How Proteins Read DNA E.g.: Helix-Turn-Helix Motif Leucine Zipper Motif Down in the Groove Different patterns of hydrophobic methyls, potential H bonds, etc. at edges of different

877 views • 58 slides
A C Standard model has 1-1 correspondence between symbols and states, P(A | T) thus P ( x i

A C Standard model has 1-1 correspondence between symbols and states, P(A | T) thus P ( x i

Outline Markov chains CSCE 471/871 Lecture 3: Markov Chains and Hidden Markov Models Hidden Markov models (HMMs) Formal definition Finding most probable state path (Viterbi algorithm) Stephen D. Scott Forward and backward

640 views • 5 slides
1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

Hidden Markov Models CSE 473: Artificial Intelligence Markov chains not so useful for most agents Hidden Markov Models Eventually you dont know anything anymore Need observations to update your beliefs Hidden Markov models

567 views • 9 slides
STA 331 2.0 Stochastic Processes 5. Continuous Parameter Markov Chains Dr Thiyanga S. Talagala

STA 331 2.0 Stochastic Processes 5. Continuous Parameter Markov Chains Dr Thiyanga S. Talagala

STA 331 2.0 Stochastic Processes 5. Continuous Parameter Markov Chains Dr Thiyanga S. Talagala September 08, 2020 Department of Statistics, University of Sri Jayewardenepura Goals 1. Explain the Markov property in the continuous-time

358 views • 12 slides
1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

Hidden Markov Models CSE 473: Artificial Intelligence Hidden Markov Models Markov chains not so useful for most agents Eventually you dont know anything anymore Need observations to update your beliefs Hidden Markov models (HMMs)

253 views • 7 slides

More recommend