SFM-11:CONNECT Summer School, Bertinoro, June 2011 EU-FP7: CONNECT - PowerPoint PPT Presentation

Probability space - Example • Paths where sending fails the first time 1 − ω = s 0 s 1 s 2 {fail} − C( ω ) = all paths starting s 0 s 1 s 2 … s 2 0.01 {try} − P s0 ( ω ) = P(s 0 ,s 1 ) · P(s 1 ,s 2 ) s 0 s 1 1 0.98 = 1 · 0.01 = 0.01 1 s 3 − Pr s0 (C( ω )) = P s0 ( ω ) = 0.01 {succ} 0.01 • Paths which are eventually successful and with no failures − C(s 0 s 1 s 3 ) ∪ C(s 0 s 1 s 1 s 3 ) ∪ C(s 0 s 1 s 1 s 1 s 3 ) ∪ … − Pr s0 ( C(s 0 s 1 s 3 ) ∪ C(s 0 s 1 s 1 s 3 ) ∪ C(s 0 s 1 s 1 s 1 s 3 ) ∪ … ) = P s0 (s 0 s 1 s 3 ) + P s0 (s 0 s 1 s 1 s 3 ) + P s0 (s 0 s 1 s 1 s 1 s 3 ) + … = 1·0.98 + 1·0.01·0.98 + 1·0.01·0.01·0.98 + … = 0.9898989898… = 98/99 23

Overview (Part 2) • Discrete-time Markov chains (DTMCs) • PCTL: A temporal logic for DTMCs • PCTL model checking • Other properties: LTL, costs and rewards • Case study: Bluetooth device discovery 24

PCTL • Temporal logic for describing properties of DTMCs − PCTL = Probabilistic Computation Tree Logic [HJ94] − essentially the same as the logic pCTL of [ASB+95] • Extension of (non-probabilistic) temporal logic CTL − key addition is probabilistic operator P − quantitative extension of CTL’s A and E operators • Example − send → P ≥ 0.95 [ true U ≤ 10 deliver ] − “if a message is sent, then the probability of it being delivered within 10 steps is at least 0.95” 25

PCTL syntax ψ is true with • PCTL syntax: probability ~p − φ ::= true | a | φ ∧ φ | ¬ φ | P ~p [ ψ ] (state formulas) − ψ ::= X φ | φ U ≤ k φ | φ U φ (path formulas) “bounded “next” “until” until” − where a is an atomic proposition, used to identify states of interest, p ∈ [0,1] is a probability, ~ ∈ {<,>, ≤ , ≥ }, k ∈ ℕ • A PCTL formula is always a state formula − path formulas only occur inside the P operator 26

PCTL semantics for DTMCs • PCTL formulas interpreted over states of a DTMC − s ⊨ φ denotes φ is “true in state s” or “satisfied in state s” • Semantics of (non-probabilistic) state formulas: − for a state s of the DTMC (S,s init ,P,L): − s ⊨ a ⇔ a ∈ L(s) − s ⊨ φ 1 ∧ φ 2 ⇔ s ⊨ φ 1 and s ⊨ φ 2 − s ⊨ ¬ φ ⇔ s ⊨ φ is false 1 {fail} • Examples s 2 0.01 {try} − s 3 ⊨ succ s 0 s 1 1 0.98 − s 1 ⊨ try ∧ ¬fail 1 s 3 {succ} 0.01 27

PCTL semantics for DTMCs • Semantics of path formulas: − for a path ω = s 0 s 1 s 2 … in the DTMC: − ω ⊨ X φ ⇔ s 1 ⊨ φ − ω ⊨ φ 1 U ≤ k φ 2 ⇔ ∃ i ≤ k such that s i ⊨ φ 2 and ∀ j<i, s j ⊨ φ 1 − ω ⊨ φ 1 U φ 2 ⇔ ∃ k ≥ 0 such that ω ⊨ φ 1 U ≤ k φ 2 • Some examples of satisfying paths: − X succ {try} {succ} {succ} {succ} 1 {fail} s 1 s 3 s 3 s 3 s 2 0.01 {try} − ¬fail U succ s 0 s 1 1 0.98 1 s 3 {try} {try} {succ} {succ} s 0 s 1 s 1 s 3 s 3 {succ} 0.01 28

PCTL semantics for DTMCs • Semantics of the probabilistic operator P − informal definition: s ⊨ P ~p [ ψ ] means that “the probability, from state s, that ψ is true for an outgoing path satisfies ~p” − example: s ⊨ P <0.25 [ X fail ] ⇔ “the probability of atomic proposition fail being true in the next state of outgoing paths from s is less than 0.25” − formally: s ⊨ P ~p [ ψ ] ⇔ Prob(s, ψ ) ~ p − where: Prob(s, ψ ) = Pr s { ω ∈ Path(s) | ω ⊨ ψ } − (sets of paths satisfying ψ are always measurable [Var85]) ¬ ψ s ψ Prob(s, ψ ) ~ p ? 29

More PCTL… • Usual temporal logic equivalences: − false ≡ ¬true (false) − φ 1 ∨ φ 2 ≡ ¬(¬ φ 1 ∧ ¬ φ 2 ) (disjunction) − φ 1 → φ 2 ≡ ¬ φ 1 ∨ φ 2 (implication) − F φ ≡ ◊ φ ≡ true U φ (eventually, “future”) − G φ ≡ □ φ ≡ ¬(F ¬ φ ) (always, “globally”) − bounded variants: F ≤ k φ , G ≤ k φ • Negation and probabilities − e.g. ¬P >p [ φ 1 U φ 2 ] ≡ P ≤ p [ φ 1 U φ 2 ] − e.g. P >p [ G φ ] ≡ P <1-p [ F ¬ φ ] 30

Qualitative vs. quantitative properties • P operator of PCTL can be seen as a quantitative analogue of the CTL operators A (for all) and E (there exists) • A PCTL property P ~p [ ψ ] is… − qualitative when p is either 0 or 1 {heads} − quantitative when p is in the range (0,1) 1 s 1 0.5 • P >0 [ F φ ] is identical to EF φ 1 s 0 − there exists a finite path to a φ -state s 2 0.5 {tails} • P ≥ 1 [ F φ ] is (similar to but) weaker than AF φ − e.g. AF “tails” (CTL) ≠ P ≥ 1 [ F “tails” ] (PCTL) 31

Quantitative properties • Consider a PCTL formula P ~p [ ψ ] − if the probability is unknown, how to choose the bound p? • When the outermost operator of a PTCL formula is P − we allow the form P =? [ ψ ] − “what is the probability that path formula ψ is true?” • Model checking is no harder: compute the values anyway • Useful to spot patterns, trends • Example − P =? [ F err/total>0.1 ] − “what is the probability 
 that 10% of the NAND 
 gate outputs are erroneous?” 32

Some real PCTL examples reliability • NAND multiplexing system − P =? [ F err/total>0.1 ] − “what is the probability that 10% of the NAND gate outputs are erroneous?” performance • Bluetooth wireless communication protocol − P =? [ F ≤ t reply_count=k ] − “what is the probability that the sender has received k acknowledgements within t clock-ticks?” fairness • Security: EGL contract signing protocol − P =? [ F (pairs_a=0 & pairs_b>0) ] − “what is the probability that the party B gains an unfair advantage during the execution of the protocol?” 33

Overview (Part 2) • Discrete-time Markov chains (DTMCs) • PCTL: A temporal logic for DTMCs • PCTL model checking • Other properties: LTL, costs and rewards • Case study: Bluetooth device discovery 34

PCTL model checking for DTMCs • Algorithm for PCTL model checking [CY88,HJ94,CY95] − inputs: DTMC D=(S,s init ,P,L), PCTL formula φ − output: Sat( φ ) = { s ∈ S | s ⊨ φ } = set of states satisfying φ • What does it mean for a DTMC D to satisfy a formula φ ? − sometimes, want to check that s ⊨ φ ∀ s ∈ S, i.e. Sat( φ ) = S − sometimes, just want to know if s init ⊨ φ , i.e. if s init ∈ Sat( φ ) • Sometimes, focus on quantitative results − e.g. compute result of P=? [ F error ] − e.g. compute result of P=? [ F ≤ k error ] for 0 ≤ k ≤ 100 35

PCTL model checking for DTMCs • Basic algorithm proceeds by induction on parse tree of φ − example: φ = (¬fail ∧ try) → P >0.95 [ ¬fail U succ ] • For the non-probabilistic operators: − Sat(true) = S → − Sat(a) = { s ∈ S | a ∈ L(s) } − Sat(¬ φ ) = S \ Sat( φ ) ∧ P >0.95 [ · U · ] − Sat( φ 1 ∧ φ 2 ) = Sat( φ 1 ) ∩ Sat( φ 2 ) • For the P ~p [ ψ ] operator ¬ try ¬ succ − need to compute the 
 probabilities Prob(s, ψ ) 
 fail fail for all states s ∈ S − focus here on “until” 
 case: ψ = φ 1 U φ 2 36

PCTL until for DTMCs • Computation of probabilities Prob(s, φ 1 U φ 2 ) for all s ∈ S • First, identify all states where the probability is 1 or 0 − S yes = Sat(P ≥ 1 [ φ 1 U φ 2 ]) − S no = Sat(P ≤ 0 [ φ 1 U φ 2 ]) • Then solve linear equation system for remaining states • We refer to the first phase as “precomputation” − two algorithms: Prob0 (for S no ) and Prob1 (for S yes ) − algorithms work on underlying graph (probabilities irrelevant) • Important for several reasons − reduces the set of states for which probabilities must be computed numerically (which is more expensive) − gives exact results for the states in S yes and S no (no round-off) − for P ~p [·] where p is 0 or 1, no further computation required 37

PCTL until - Linear equations • Probabilities Prob(s, φ 1 U φ 2 ) can now be obtained as the unique solution of the following set of linear equations: − can be reduced to a system in |S ? | unknowns instead of |S| where S ? = S \ (S yes ∪ S no ) • This can be solved with (a variety of) standard techniques − direct methods, e.g. Gaussian elimination − iterative methods, e.g. Jacobi, Gauss-Seidel, … 
 (preferred in practice due to scalability) 38

PCTL until - Example • Example: P >0.8 [¬a U b ] 1 0.3 a 0.6 1 3 5 0.3 0.7 0.4 0.1 0.1 b 0 2 4 0.9 0.5 1 0.1 39

PCTL until - Example • Example: P >0.8 [¬a U b ] S no = Sat(P ≤ 0 [¬a U b ]) 1 0.3 a 0.6 1 3 5 S yes = 0.3 0.7 0.4 0.1 0.1 Sat(P ≥ 1 [¬a U b ]) b 0 2 4 0.9 0.5 1 0.1 40

PCTL until - Example • Example: P >0.8 [¬a U b ] S no = Sat(P ≤ 0 [¬a U b ]) • Let x s = Prob(s, ¬a U b) 1 0.3 a 0.6 • Solve: 1 3 5 S yes = 0.3 0.7 0.4 0.1 0.1 Sat(P ≥ 1 [¬a U b ]) x 4 = x 5 = 1 b 0 2 4 x 1 = x 3 = 0 0.9 0.5 1 0.1 x 0 = 0.1x 1 +0.9x 2 = 0.8 x 2 = 0.1x 2 +0.1x 3 +0.3x 5 +0.5x 4 = 8/9 Prob(¬a U b) = x = [0.8, 0, 8/9, 0, 1, 1] Sat(P >0.8 [ ¬a U b ]) = { s 2 ,s 4 ,s 5 } 41

PCTL model checking - Summary • Computation of set Sat( Φ ) for DTMC D and PCTL formula Φ − recursive descent of parse tree − combination of graph algorithms, numerical computation • Probabilistic operator P: − X Φ : one matrix-vector multiplication, O(|S| 2 ) − Φ 1 U ≤ k Φ 2 : k matrix-vector multiplications, O(k|S| 2 ) − Φ 1 U Φ 2 : linear equation system, at most |S| variables, O(|S| 3 ) • Complexity: − linear in | Φ | and polynomial in |S| 42

Overview (Part 2) • Discrete-time Markov chains (DTMCs) • PCTL: A temporal logic for DTMCs • PCTL model checking • Other properties: LTL, costs and rewards • Case study: Bluetooth device discovery 43

Limitations of PCTL • PCTL, although useful in practice, has limited expressivity − essentially: probability of reaching states in X, passing only through states in Y (and within k time-steps) • More expressive logics can be used, for example: − LTL [Pnu77] – (non-probabilistic) linear-time temporal logic − PCTL* [ASB+95,BdA95] - which subsumes both PCTL and LTL − both allow path operators to be combined − (in PCTL, P ~p […] always contains a single temporal operator) • Another direction: extend DTMCs with costs and rewards… 44

LTL - Linear temporal logic • LTL syntax (path formulae only) − ψ ::= true | a | ψ ∧ ψ | ¬ ψ | X ψ | ψ U ψ − where a ∈ AP is an atomic proposition − usual equivalences hold: F φ ≡ true U φ , G φ ≡ ¬(F ¬ φ ) − evaluated over paths of a model • Examples − (F tmp_fail 1 ) ∧ (F tmp_fail 2 ) − “both servers suffer temporary failures at some point” − GF ready − “the server always eventually returns to a ready-state” − FG error − “an irrecoverable error occurs” − G (req → X ack) − “requests are always immediately acknowledged” 45

LTL for DTMCs • Same idea as PCTL: probabilities of sets of path formulae − for a state s of a DTMC and an LTL formula ψ : − Prob(s, ψ ) = Pr s { ω ∈ Path(s) | ω ⊨ ψ } − all such path sets are measurable [Var85] • A (probabilistic) LTL specification often comprises 
 an LTL (path) formula and a probability bound − e.g. P ≥ 1 [ GF ready ] – “with probability 1, the server always eventually returns to a ready-state” − e.g. P <0.01 [ FG error ] – “with probability at most 0.01, an irrecoverable error occurs” • PCTL* subsumes both LTL and PCTL − e.g. P >0.5 [ GF crit 1 ] ∧ P >0.5 [ GF crit 2 ] 46

Fundamental property of DTMCs • Strongly connected component (SCC) − maximally strongly connected set of states • Bottom strongly connected component (BSCC) − SCC T from which no state outside T is reachable from T 0.5 • Fundamental property of DTMCs: 0.25 s 1 − “with probability 1, 
 s 0 s 2 0.5 a BSCC will be reached 
 0.5 and all of its states 
 0.25 1 1 visited infinitely often” s 4 s 5 s 3 1 1 • Formally: − Pr s { ω ∈ Path(s) | ∃ i ≥ 0, ∃ BSCC T such that 
 ∀ j ≥ i ω (i) ∈ T and 
 ∀ s’ ∈ T ω (k) = s' for infinitely many k } = 1 47

LTL model checking for DTMCs • Steps for model checking LTL property ψ on DTMC D − i.e. computing Prob D (s, ψ ) • 1. Build a deterministic Rabin automaton (DRA) A for ψ − i.e. a DRA A over alphabet 2 AP accepting ψ -satisfying traces • 2. Build the “product” DTMC D ⊗ A − records state of A for path through D so far • 3. Identify states T acc in “accepting” BSCCs of D ⊗ A − i.e. those that meet the acceptance condition of A • 4. Compute probability of reaching T acc in D ⊗ A − which gives Prob D (s, ψ ), as required 48

Example: LTL for DTMCs DTMC D DRA A ψ for ψ = G¬b ∧ GF a {b} {a} a ∧ ¬b 0.1 0.5 s 0 s 1 s 2 q 0 q 1 a ∧ ¬b 0.6 0.2 0.3 0.1 ¬a ∧ ¬b 1 b 0.3 b ¬a ∧ ¬b 0.9 1 q 2 Acc ={ ({},{q 1 }) } s 3 s 4 s 5 true 1 {a} {a} Product DTMC D ⊗ A ψ Prob D (s, ψ ) 0.1 0.5 s 0 q 0 s 1 q 2 s 2 q 2 = Prob D ⊗ A ψ (F T 1 ) 0.6 0.2 T 1 T 2 0.3 0.1 = 3/4. 0.3 1 0.9 1 1 s 4 q 0 s 4 q 2 s 3 q 1 s 3 q 2 s 5 q 2 T 3 {k 1 } 1 1 49

Costs and rewards • We augment DTMCs with rewards (or, conversely, costs) − real-valued quantities assigned to states and/or transitions − these can have a wide range of possible interpretations • Some examples: − elapsed time, power consumption, size of message queue, number of messages successfully delivered, net profit, … • Costs? or rewards? − mathematically, no distinction between rewards and costs − when interpreted, we assume that it is desirable to minimise costs and to maximise rewards − we will consistently use the terminology “rewards” regardless 50

Reward-based properties • Properties of DTMCs augmented with rewards − allow a wide range of quantitative measures of the system − basic notion: expected value of rewards − formal property specifications will be in an extension of PCTL • More precisely, we use two distinct classes of property… • Instantaneous properties − the expected value of the reward at some time point • Cumulative properties − the expected cumulated reward over some period 51

DTMC reward structures • For a DTMC (S,s init ,P,L), a reward structure is a pair ( ρ , ι ) − ρ : S → ℝ ≥ 0 is the state reward function (vector) − ι : S × S → ℝ ≥ 0 is the transition reward function (matrix) • Example (for use with instantaneous properties) − “size of message queue”: ρ maps each state to the number of jobs in the queue in that state, ι is not used • Examples (for use with cumulative properties) − “time-steps”: ρ returns 1 for all states and ι is zero (equivalently, ρ is zero and ι returns 1 for all transitions) − “number of messages lost”: ρ is zero and ι maps transitions corresponding to a message loss to 1 − “power consumption”: ρ is defined as the per-time-step energy consumption in each state and ι as the energy cost of each transition 52

PCTL and rewards • Extend PCTL to incorporate reward-based properties − add an R operator, which is similar to the existing P operator expected reward is ~r − φ ::= … | P ~p [ ψ ] | R ~r [ I =k ] | R ~r [ C ≤ k ] | R ~r [ F φ ] “instantaneous” “cumulative” “reachability” − where r ∈ ℝ ≥ 0 , ~ ∈ {<,>, ≤ , ≥ }, k ∈ ℕ • R ~r [ · ] means “the expected value of · satisfies ~r” 53

Types of reward formulas • Instantaneous: R ~r [ I =k ] − “the expected value of the state reward at time-step k is ~r” − e.g. “the expected queue size after exactly 90 seconds” • Cumulative: R ~r [ C ≤ k ] − “the expected reward cumulated up to time-step k is ~r” − e.g. “the expected power consumption over one hour” • Reachability: R ~r [ F φ ] − “the expected reward cumulated before reaching a state satisfying φ is ~r” − e.g. “the expected time for the algorithm to terminate” 54

Reward formula semantics • Formal semantics of the three reward operators − based on random variables over (infinite) paths • Recall: − s ⊨ P ~p [ ψ ] ⇔ Pr s { ω ∈ Path(s) | ω ⊨ ψ } ~ p • For a state s in the DTMC: − s ⊨ R ~r [ I =k ] ⇔ Exp(s, X I=k ) ~ r − s ⊨ R ~r [ C ≤ k ] ⇔ Exp(s, X C ≤ k ) ~ r − s ⊨ R ~r [ F Φ ] ⇔ Exp(s, X F Φ ) ~ r where: Exp(s, X) denotes the expectation of the random variable X : Path(s) → ℝ ≥ 0 with respect to the probability measure Pr s 55

Reward formula semantics • Definition of random variables: − for an infinite path ω = s 0 s 1 s 2 … − where k φ =min{ j | s j ⊨ φ } 56

Model checking reward properties • Instantaneous: R ~r [ I =k ] • Cumulative: R ~r [ C ≤ t ] − variant of the method for computing bounded until probabilities − solution of recursive equations • Reachability: R ~r [ F φ ] − similar to computing until probabilities − precomputation phase (identify infinite reward states) − then reduces to solving a system of linear equation • For more details, see e.g. [KNP07a] 57

Overview (Part 2) • Discrete-time Markov chains (DTMCs) • PCTL: A temporal logic for DTMCs • PCTL model checking • Other properties: LTL, costs and rewards • Case study: Bluetooth device discovery 58

The PRISM tool • PRISM: Probabilistic symbolic model checker − developed at Birmingham/Oxford University, since 1999 − free, open source (GPL), runs on all major OSs • Support for: − discrete-/continuous-time Markov chains (D/CTMCs) − Markov decision processes (MDPs) − probabilistic timed automata (PTAs) − PCTL, CSL, LTL, PCTL*, costs/rewards, … • Multiple efficient model checking engines − mostly symbolic (BDDs) (up to 10 10 states, 10 7 -10 8 on avg.) • Successfully applied to a wide range of case studies − communication protocols, security protocols, dynamic power management, cell signalling pathways, … • See: http://www.prismmodelchecker.org/ 59

Bluetooth device discovery • Bluetooth: short-range low-power wireless protocol − widely available in phones, PDAs, laptops, ... − open standard, specification freely available • Uses frequency hopping scheme − to avoid interference (uses unregulated 2.4GHz band) − pseudo-random selection over 32 of 79 frequencies • Formation of personal area networks (PANs) − piconets (1 master, up to 7 slaves) − self-configuring: devices discover themselves • Device discovery − mandatory first step before any communication possible − relatively high power consumption so performance is crucial − master looks for devices, slaves listens for master 60

Master (sender) behaviour • 28 bit free-running clock CLK, ticks every 312.5µs • Frequency hopping sequence determined by clock: − freq = [CLK 16-12 +k+ (CLK 4-2,0 - 
 CLK 16-12 ) mod 16] mod 32 − 2 trains of 16 frequencies 
 (determined by offset k), 
 128 times each, swap between 
 every 2.56s • Broadcasts “inquiry packets” on 
 two consecutive frequencies, 
 then listens on the same two 61

Slave (receiver) behaviour • Listens (scans) on frequencies for inquiry packets − must listen on right frequency at right time − cycles through frequency sequence at much slower speed (every 1.28s) 
 • On hearing packet, pause, send reply and then wait for a random delay before listening for subsequent packets − avoid repeated collisions with other slaves 62

Bluetooth – PRISM model • Modelled/analysed using PRISM model checker [DKNP06] − model scenario with one sender and one receiver − synchronous (clock speed defined by Bluetooth spec) − model at lowest-level (one clock-tick = one transition) − randomised behaviour so model as a DTMC − use real values for delays, etc. from Bluetooth spec • Modelling challenges − complex interaction between sender/receiver − combination of short/long time-scales – cannot scale down − sender/receiver not initially synchronised, so huge number of possible initial configurations (17,179,869,184) 63

Bluetooth - Results • Huge DTMC – initially, model checking infeasible − partition into 32 scenarios, i.e. 32 separate DTMCs − on average, approx. 3.4 x 10 9 states (536,870,912 initial) − can be built/analysed with PRISM's MTBDD engine • We compute: − R=? [ F replies=K {“init”}{max} ] − “worst-case expected time to hear K replies over all possible initial configurations” • Also look at: − how many initial states for each possible expected time − cumulative distribution function (CDF) for time, assuming equal probability for each initial state 64

Bluetooth - Time to hear 1 reply • Worst-case expected time = 2.5716 sec − in 921,600 possible initial states − best-case = 635 µs 65

Bluetooth - Time to hear 2 replies • Worst-case expected time = 5.177 sec − in 444 possible initial states − compare actual CDF with derived version which assumes times to reply to first/second messages are independent 66

Bluetooth - Results • Other results: (see [DKNP06]) − compare versions 1.2 and 1.1 of Bluetooth, confirm 1.1 slower − power consumption analysis (using costs + rewards) • Conclusions: − successful analysis of complex real-life model − detailed model, actual parameters used − exhaustive analysis: best/worst-case values • can pinpoint scenarios which give rise to them • not possible with simulation approaches − model still relatively simple • consider multiple receivers? • combine with simulation? 67

Summary (Parts 1 & 2) • Probabilistic model checking − automated quantitative verification of stochastic systems − to model randomisation, failures, … • Discrete-time Markov chains (DTMCs) − state transition systems + discrete probabilistic choice − probability space over paths through a DTMC • Property specifications − probabilistic extensions of temporal logic, e.g. PCTL, LTL − also: expected value of costs/rewards • Model checking algorithms − combination of graph-based algorithms, numerical computation, automata constructions • Next: Markov decision processes (MDPs) 68

Part 3 Markov decision processes

Overview • Lectures 1 and 2: − 1 – Introduction − 2 – Discrete-time Markov chains − 3 – Markov decision processes − 4 – Compositional probabilistic verification • Course materials available here: − http://www.prismmodelchecker.org/courses/sfm11connect/ − lecture slides, reference list, tutorial chapter, lab session 70

Probabilistic models Fully probabilisti tic Nondete terministi tic Discrete-time Markov decision Discrete Di te Markov chains processes (MDPs) time ti (DTMCs) (probabilistic automata) Probabilistic timed automata (PTAs) Continuous-time Conti tinuous Markov chains time ti (CTMCs) CTMDPs/IMCs 71

Overview (Part 3) • Markov decision processes (MDPs) • Adversaries & probability spaces • Properties of MDPs: The temporal logic PCTL • PCTL model checking for MDPs • Case study: Firewire root contention 72

Recap: Discrete-time Markov chains • Discrete-time Markov chains (DTMCs) − state-transition systems augmented with probabilities • Formally: DTMC D = (S, s init , P, L) where: − S is a set of states and s init ∈ S is the initial state − P : S × S → [0,1] is the transition probability matrix − L : S → 2 AP labels states with atomic propositions − define a probability space Pr s over paths Path s 1 {fail} • Properties of DTMCs s 2 0.01 {try} − can be captured by the logic PCTL s 0 s 1 0.98 1 1 − e.g. send → P ≥ 0.95 [ F deliver ] s 3 − key question: what is the probability 
 {succ} 0.01 of reaching states T ⊆ S from state s? − reduces to graph analysis + linear equation system 73

Nondeterminism • Some aspects of a system may not be probabilistic and should not be modelled probabilistically; for example: • Concurrency - scheduling of parallel components − e.g. randomised distributed algorithms - multiple probabilistic processes operating asynchronously • Underspecification - unknown model parameters − e.g. a probabilistic communication protocol designed for message propagation delays of between d min and d max • Unknown environments − e.g. probabilistic security protocols - unknown adversary 74

Markov decision processes • Markov decision processes (MDPs) − extension of DTMCs which allow nondeterministic choice • Like DTMCs: − discrete set of states representing possible configurations of the system being modelled − transitions between states occur in discrete time-steps • Probabilities and nondeterminism {heads} s 2 − in each state, a nondeterministic 
 {init} a 0.5 a 1 choice between several discrete 
 1 s 0 s 1 c 1 probability distributions over 
 a s 3 0.7 successor states b 0.5 0.3 {tails} 75

Markov decision processes • Formally, an MDP M is a tuple (S,s init , α , δ ,L) where: {heads} − S is a set of states (“state space”) s 2 {init} a 0.5 − s init ∈ S is the initial state a 1 1 s 0 s 1 c − α is an alphabet of action labels 1 a s 3 0.7 b − δ ⊆ S × α × Dist(S) is the transition 
 0.5 0.3 probability relation, where Dist(S) is the set 
 {tails} of all discrete probability distributions over S − L : S → 2 AP is a labelling with atomic propositions • Notes: − we also abuse notation and use δ as a function − i.e. δ : S → 2 α× Dist(S) where δ (s) = { (a,µ) | (s,a,µ) ∈ δ } − we assume δ (s) is always non-empty, i.e. no deadlocks − MDPs, here, are identical to probabilistic automata [Segala] 76

Simple MDP example • A simple communication protocol − after one step, process starts trying to send a message − then, a nondeterministic choice between: (a) waiting a step because the channel is unready; (b) sending the message − if the latter, with probability 0.99 send successfully and stop − and with probability 0.01, message sending fails, restart restart {fail} 1 s 2 {try} 0.01 start send s 0 s 1 1 stop 0.99 s 3 1 wait 1 {succ} 77

Example - Parallel composition 1 Asynchronous parallel 
 0.5 t 0 t 1 t 2 1 composition of two 
 0.5 3-state DTMCs 1 0.5 Action labels 
 s 0 s 0 t 0 s 0 t 1 1 s 0 t 2 0.5 omitted here 0.5 0.5 0.5 0.5 1 1 1 1 1 0.5 s 1 s 1 t 0 s 1 t 1 s 1 t 2 0.5 1 0.5 0.5 0.5 0.5 1 0.5 s 2 s 2 t 0 s 2 t 1 s 2 t 2 0.5 1 1 1 1 1 78

Paths and probabilities • A (finite or infinite) path through an MDP M − is a sequence of states and action/distribution pairs − e.g. s 0 (a 0 ,µ 0 )s 1 (a 1 ,µ 1 )s 2 … − such that (a i ,µ i ) ∈ δ (s i ) and µ i (s i+1 ) > 0 for all i ≥ 0 − represents an execution (i.e. one possible behaviour) of the system which the MDP is modelling − note that a path resolves both types of choices: nondeterministic and probabilistic − Path M,s (or just Path s ) is the set of all infinite paths starting from state s in MDP M; the set of finite paths is PathFin s • To consider the probability of some behaviour of the MDP − first need to resolve the nondeterministic choices − …which results in a DTMC − …for which we can define a probability measure over paths 79

Overview (Part 3) • Markov decision processes (MDPs) • Adversaries & probability spaces • Properties of MDPs: The temporal logic PCTL • PCTL model checking for MDPs • Case study: Firewire root contention 80

Adversaries • An adversary resolves nondeterministic choice in an MDP − also known as “schedulers”, “strategies” or “policies” • Formally: − an adversary σ of an MDP is a function mapping every finite path ω = s 0 (a 0 ,µ 0 )s 1 ...s n to an element of δ (s n ) • Adversary σ restricts the MDP to certain paths − Path s σ ⊆ Path s σ and PathFin s σ ⊆ PathFin s σ • Adversary σ induces a probability measure Pr s σ over paths − constructed through an infinite state DTMC (PathFin s σ , s, P s σ ) − states of the DTMC are the finite paths of σ starting in state s − initial state is s (the path starting in s of length 0) − P s σ ( ω , ω ’)=µ(s) if ω ’= ω (a,µ)s and σ ( ω )=(a,µ) − P s σ ( ω , ω ’)=0 otherwise 81

Adversaries - Examples • Consider the simple MDP below − note that s 1 is the only state for which | δ (s)| > 1 − i.e. s 1 is the only state for which an adversary makes a choice − let µ b and µ c denote the probability distributions associated with actions b and c in state s 1 {heads} • Adversary σ 1 s 2 {init} a 0.5 a 1 − picks action c the first time 1 s 0 s 1 c 1 − σ 1 (s 0 s 1 )=(c,µ c ) a s 3 0.7 b 0.5 0.3 {tails} • Adversary σ 2 − picks action b the first time, then c − σ 2 (s 0 s 1 )=(b,µ b ), σ 2 (s 0 s 1 s 1 )=(c,µ c ), σ 2 (s 0 s 1 s 0 s 1 )=(c,µ c ) 82

Adversaries - Examples • Fragment of DTMC for adversary σ 1 − σ 1 picks action c the first time {heads} s 2 {init} 0.5 a a 1 1 s 0 s 1 c 1 a s 3 0.7 b 0.5 0.3 {tails} 1 0.5 s 0 s 1 s 2 s 0 s 1 s 2 s 2 1 s 0 s 0 s 1 s 0 s 1 s 3 s 0 s 1 s 3 s 3 0.5 1 83

Adversaries - Examples {heads} • Fragment of DTMC for adversary σ 2 s 2 − σ 2 picks action b, then c {init} 0.5 a a 1 1 s 0 s 1 c 1 a s 3 0.7 b 0.5 0.3 {tails} 0.5 s 0 s 1 s 0 s 1 s 2 1 s 0 s 1 s 0 s 0 s 1 s 0 s 1 0.7 s 0 s 1 s 0 s 1 s 3 0.5 1 s 0 s 1 s 0 1 0.5 s 0 s 1 s 1 s 2 s 0 s 1 s 1 s 2 s 2 0.3 s 0 s 1 s 1 s 0 s 1 s 1 s 3 s 0 s 1 s 1 s 3 s 3 0.5 1 84

Memoryless adversaries • Memoryless adversaries always pick same choice in a state − also known as: positional, simple, Markov − formally, for adversary σ : − σ (s 0 (a 0 ,µ 0 )s 1 ...s n ) depends only on s n − resulting DTMC can be mapped to a |S|-state DTMC • From previous example: − adversary σ 1 (picks c in s 1 ) is memoryless, σ 2 is not {heads} {heads} s 2 s 2 σ 1 {init} {init} 0.5 a 0.5 a a 1 a 1 1 1 s 0 s 1 c s 0 s 1 c 1 1 a a s 3 s 3 0.7 b 0.5 0.5 0.3 {tails} {tails} 85

Overview (Part 3) • Markov decision processes (MDPs) • Adversaries & probability spaces • Properties of MDPs: The temporal logic PCTL • PCTL model checking for MDPs • Case study: Firewire root contention 86

PCTL • Temporal logic for properties of MDPs (and DTMCs) − extension of (non-probabilistic) temporal logic CTL − key addition is probabilistic operator P − quantitative extension of CTL’s A and E operators • PCTL syntax: − φ ::= true | a | φ ∧ φ | ¬ φ | P ~p [ ψ ] (state formulas) − ψ ::= X φ | φ U ≤ k φ | φ U φ (path formulas) − where a is an atomic proposition, used to identify states of interest, p ∈ [0,1] is a probability, ~ ∈ {<,>, ≤ , ≥ }, k ∈ ℕ • Example: send → P ≥ 0.95 [ true U ≤ 10 deliver ] 87

PCTL semantics for MDPs • PCTL formulas interpreted over states of an MDP − s ⊨ φ denotes φ is “true in state s” or “satisfied in state s” • Semantics of (non-probabilistic) state formulas: − for a state s of the MDP (S,s init , α , δ ,L): − s ⊨ a ⇔ a ∈ L(s) − s ⊨ φ 1 ∧ φ 2 ⇔ s ⊨ φ 1 and s ⊨ φ 2 − s ⊨ ¬ φ ⇔ s ⊨ φ is false • Semantics of path formulas: − for a path ω = s 0 (a 0 ,µ 0 )s 1 (a 1 ,µ 1 )s 2 … in the MDP: − ω ⊨ X φ ⇔ s 1 ⊨ φ − ω ⊨ φ 1 U ≤ k φ 2 ⇔ ∃ i ≤ k such that s i ⊨ φ 2 and ∀ j<i, s j ⊨ φ 1 − ω ⊨ φ 1 U φ 2 ⇔ ∃ k ≥ 0 such that ω ⊨ φ 1 U ≤ k φ 2 88

PCTL semantics for MDPs • Semantics of the probabilistic operator P − can only define probabilities for a specific adversary σ − s ⊨ P ~p [ ψ ] means “the probability, from state s, that ψ is true for an outgoing path satisfies ~p for all adversaries σ ” − formally s ⊨ P ~p [ ψ ] ⇔ Pr s σ ( ψ ) ~ p for all adversaries σ − where we use Pr s σ ( ψ ) to denote Pr s σ { ω ∈ Path s σ | ω ⊨ ψ } ¬ ψ s Pr s σ ( ψ ) ~ p ψ • Some equivalences: − F φ ≡ ◊ φ ≡ true U φ (eventually, “future”) − G φ ≡ □ φ ≡ ¬(F ¬ φ ) (always, “globally”) 89

Minimum and maximum probabilities • Letting: − Pr s max ( ψ ) = sup σ Pr s σ ( ψ ) − Pr s min ( ψ ) = inf σ Pr s σ ( ψ ) • We have: − if ~ ∈ { ≥ ,>}, then s ⊨ P ~p [ ψ ] ⇔ Pr s min ( ψ ) ~ p − if ~ ∈ {<, ≤ }, then s ⊨ P ~p [ ψ ] ⇔ Pr s max ( ψ ) ~ p • Model checking P ~p [ ψ ] reduces to the computation over all adversaries of either: − the minimum probability of ψ holding − the maximum probability of ψ holding • Crucial result for model checking PCTL on MDPs − memoryless adversaries suffice, i.e. there are always memoryless adversaries σ min and σ max for which: − Pr s σ min ( ψ ) = Pr s min ( ψ ) and Pr s σ max ( ψ ) = Pr s min ( ψ ) 90

Quantitative properties • For PCTL properties with P as the outermost operator − quantitative form (two types): P min=? [ ψ ] and P max=? [ ψ ] − i.e. “what is the minimum/maximum probability (over all adversaries) that path formula ψ is true?” − corresponds to an analysis of best-case or worst-case behaviour of the system − model checking is no harder since compute the values of 
 Pr s min ( ψ ) or Pr s max ( ψ ) anyway − useful to spot patterns/trends • Example: CSMA/CD protocol − “min/max probability that a message is sent within the deadline” 91

Other classes of adversary • A more general semantics for PCTL over MDPs − parameterise by a class of adversaries Adv • Only change is: − s ⊨ Adv P ~p [ ψ ] ⇔ Pr s σ ( ψ ) ~ p for all adversaries σ ∈ Adv • Original semantics obtained by taking Adv to be the set of all adversaries for the MDP • Alternatively, take Adv to be the set of all fair adversaries − path fairness: if a state is occurs on a path infinitely often, then each non-deterministic choice occurs infinite often − see e.g. [BK98] 92

Some real PCTL examples • Byzantine agreement protocol − P min=? [ F (agreement ∧ rounds ≤ 2) ] − “what is the minimum probability that agreement is reached within two rounds?” • CSMA/CD communication protocol − P max=? [ F collisions=k ] − “what is the maximum probability of k collisions?” • Self-stabilisation protocols − P min=? [ F ≤ t stable ] − “what is the minimum probability of reaching a stable state within k steps?” 93

Overview (Part 3) • Markov decision processes (MDPs) • Adversaries & probability spaces • Properties of MDPs: The temporal logic PCTL • PCTL model checking for MDPs • Case study: Firewire root contention 94

PCTL model checking for MDPs • Algorithm for PCTL model checking [BdA95] − inputs: MDP M=(S,s init , α , δ ,L), PCTL formula φ − output: Sat( φ ) = { s ∈ S | s ⊨ φ } = set of states satisfying φ • Basic algorithm same as PCTL model checking for DTMCs − proceeds by induction on parse tree of φ − non-probabilistic operators (true, a, ¬, ∧ ) straightforward • Only need to consider P ~p [ ψ ] formulas − reduces to computation of Pr s min ( ψ ) or Pr s max ( ψ ) for all s ∈ S − dependent on whether ~ ∈ { ≥ ,>} or ~ ∈ {<, ≤ } − these slides cover the case Pr s min ( φ 1 U φ 2 ), i.e. ~ ∈ { ≥ ,>} − case for maximum probabilities is very similar − next (X φ ) and bounded until ( φ 1 U ≤ k φ 2 ) are straightforward extensions of the DTMC case 95

PCTL until for MDPs • Computation of probabilities Pr s min ( φ 1 U φ 2 ) for all s ∈ S • First identify all states where the probability is 1 or 0 − “precomputation” algorithms, yielding sets S yes , S no • Then compute (min) probabilities for remaining states (S ? ) − either: solve linear programming problem − or: approximate with an iterative solution method − or: use policy iteration 0.5 {a} 0.4 Example: s 2 s 1 P ≥ p [ F a ] 1 0.1 ≡ 1 1 1 0.5 P ≥ p [ true U a ] s 0 s 3 0.25 96 0.25

PCTL until - Precomputation • Identify all states where Pr s min ( φ 1 U φ 2 ) is 1 or 0 − S yes = Sat(P ≥ 1 [ φ 1 U φ 2 ]), S no = Sat(¬ P >0 [ φ 1 U φ 2 ]) • Two graph-based precomputation algorithms: − algorithm Prob1A computes S yes • for all adversaries the probability of satisfying φ 1 U φ 2 is 1 − algorithm Prob0E computes S no • there exists an adversary for which the probability is 0 0.5 S yes = Sat(P ≥ 1 [ F a ]) {a} 0.4 s 1 s 2 Example: 1 0.1 P ≥ p [ F a ] 1 1 1 0.5 s 0 s 3 0.25 S no = Sat( ¬P >0 [ F a ]) 0.25 97

Method 1 - Linear programming • Probabilities Pr s min ( φ 1 U φ 2 ) for remaining states in the set S ? = S \ (S yes ∪ S no ) can be obtained as the unique solution of the following linear programming (LP) problem: maximize x s subject to the constraints : ∑ s ∈ S ? x s ≤ µ (s') ⋅ x s' + µ (s') ∑ ∑ s' ∈ S ? s' ∈ S yes for all s ∈ S ? and for all (a, µ ) ∈ δ (s) • Simple case of a more general problem known as the stochastic shortest path problem [BT91] • This can be solved with standard techniques − e.g. Simplex, ellipsoid method, branch-and-cut 98

Example - PCTL until (LP) 0.5 Let x i = Pr si min (F a) S yes {a} S yes : x 2 =1, S no : x 3 =0 0.4 s 1 s 2 For S ? = {x 0 , x 1 } : 1 0.1 Maximise x 0 +x 1 subject to constraints: 1 1 ● x 0 ≤ x 1 1 0.5 s 3 s 0 ● x 0 ≤ 0.25·x 0 + 0.5 0.25 ● x 1 ≤ 0.1·x 0 + 0.5·x 1 + 0.4 S no 0.25 99

Example - PCTL until (LP) 0.5 Let x i = Pr si min (F a) S yes {a} S yes : x 2 =1, S no : x 3 =0 0.4 s 1 s 2 For S ? = {x 0 , x 1 } : 1 0.1 Maximise x 0 +x 1 subject to constraints: 1 1 ● x 0 ≤ x 1 1 0.5 s 3 s 0 ● x 0 ≤ 2/3 0.25 ● x 1 ≤ 0.2·x 0 + 0.8 S no 0.25 x 1 x 1 x 1 1 1 1 x 0 ≤ x 1 0.8 x 0 ≤ 2/3 x 1 ≤ 0.2·x 0 + 0.8 x 0 x 0 x 0 0 0 0 2/3 0 1 0 1 0 1 100