PAC Statistical Model Checking for Markov Decision Processes and - PowerPoint PPT Presentation

PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games 1 Pranav Ashok, Jan Kˇ ret´ ınsk´ y, Maximilian Weininger Technical University of Munich Highlights of Logic, Automata and Games Warsaw, Poland September 19, 2019 1 based on paper presented at CAV 2019

Stochastic Game Reachability 0 . 2 a b c 0 . 8 Objective player: maximize P(F ) player: minimize P(F ) Reachability in limited information stochastic games 2/6

This work: Black-box (limited information setting) Unknown successor distribution Problem statement Compute V ( s ) = max σ min τ P σ,τ ( F ) = min τ max σ P σ,τ ( F ) s s with guarantees Reachability in limited information stochastic games 3/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration Reachability in limited information stochastic games 4/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration ◮ Algos not directly applicable on general SG ◮ First practical algorithm for general SG giving guarantees [ Kelmendi et. al. 2018 ] Reachability in limited information stochastic games 4/6

Background ◮ Seminal paper on Stochastic Games [ Condon 90 ] quadratic programming, strategy iteration, value iteration ◮ Algos not directly applicable on general SG ◮ First practical algorithm for general SG giving guarantees [ Kelmendi et. al. 2018 ] ◮ This work: first algorithm for limited information SG Reachability in limited information stochastic games 4/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence ◮ Information conservatively back-propagated Reachability in limited information stochastic games 5/6

The Algorithm Similar to Kelmendi et. al. 2018 while U − L is large 1. Simulate and estimate 2. Back-propagate The how ◮ Simulation finds important parts of state space ◮ Simulation computes Hoeffding confidence intervals ball around estimate such that real prob. falls in the ball with high confidence ◮ Information conservatively back-propagated ◮ Other tricks to ensure fixpoint convergence Reachability in limited information stochastic games 5/6

Conclusion ◮ Algorithm for reachability in limited information MDP/SG result ∈ [0 . 6 − ǫ, 0 . 6 + ǫ ] with prob of going wrong 10 − 8 ◮ Implemented and benchmarked in PRISM Model Checker ◮ First algorithm to do so for SG ◮ First practical algorithm for MDPs Reachability in limited information stochastic games 6/6

PAC Statistical Model Checking for Markov Decision Processes and - PowerPoint PPT Presentation

PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games 1 Pranav Ashok, Jan K ret nsk y, Maximilian Weininger Technical University of Munich Highlights of Logic, Automata and Games Warsaw, Poland September

Statistical Statistical Statistical Model Statistical Model Model Checking Model Checking

Statistical Model Checking for Markov Decision Processes David Henriques Joint work with Jo

Model checking quantum Markov chains Yuan Feng, Nengkun Yu, and Mingsheng Ying University of

Model checking perhaps the most important part of applied statistical modelling Simon Wood

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

Hidden Markov Models Pratik Lahiri Introduction A hidden Markov model (HMM) is a

Statistical Model Checking and Rare Events Paolo Zuliani Joint work with Edmund M. Clarke

time Markov Models Mohmmadsadegh Mohagheghi And Behrang Chaboki Vali-e-Asr University of

Statistical Model Checking of Simulink Models Simulink Models Ed Edmund M. Clarke d M Cl k

Probabilistic Model Checking Probabilistic Model Checking Marta Kwiatkowska Kwiatkowska Marta

Hoare Logic and Model Checking Model Checking Lecture 11: Model checking for Computation Tree

Limit-Deterministic Bchi Automata for Probabilistic Model Checking Jan Ketnsk Javier

Limit-Deterministic Bchi Automata for Probabilistic Model Checking Javier Esparza Jan K

Games for discrete-time Markov chain and their application to verification Shota Nakagawa The

Model Checking Continuous-Time Markov Chains Joost-Pieter Katoen Software Modeling and

Bayesian Statistical Model Checking with Application to Stateflow/Simulink Verification Paolo

Statistical Model Checking for Distributed Probabilistic-Control Hybrid Automata with Smart Grid

Statistical Model Checking for Biological Systems Paolo Zuliani Department of Computer Science

Bounded Model Checking bmc Revision: 1.11 1 [BiereCimattiClarkeZhu99] uses SAT for model

CTL Chapter 6 Part 2 Overview Review CTL Model Checking CTL model Checking algorithms

Statistical model checking of hazards in an autonomous tramway positioning system Davide Basile 1

Parametric Statistical model checking of UAV flight plan 1 Ran Bao 12 e 2 t Delahaye 2 Christian

Parallel Statistical Model Checking for Safety Verification in Smart Grids Enrico Tronci joint