hybrid system falsification and reinforcement learning
play

Hybrid System Falsification and Reinforcement Learning Formal - PowerPoint PPT Presentation

Mar 22, 2024 •163 likes •1.01k views

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems Clovis Eberhart David Sprunger National Institute of Technology, Japan SOKENDAI lesson, July 1, 8, and 22 1 / 50 Lecture structure Lectures

  1. Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems Clovis Eberhart David Sprunger National Institute of Technology, Japan SOKENDAI lesson, July 1, 8, and 22 1 / 50

  2. Lecture structure Lectures 1st: falsification (problem, framework, logics. . . ), by me 2nd: deep learning for falsification (learning, reinforcement learning, application to falsification. . . ), by David 3rd: advanced techniques in falsification and reinforcement learning, by David and me Evaluation Easy practical assignment (in Python). Questions? Ask them during the lesson. Find me at my desk (Palaceside building). clovis.eberhart@gmail.com 2 / 50

  3. First lecture overview Formal methods landscape 1 Framework 2 Hybrid systems 3 Formulas 4 Optimisation 5 3 / 50

  4. Table of Contents Formal methods landscape 1 Framework 2 Hybrid systems 3 Formulas 4 Optimisation 5 4 / 50

  5. Formal method landscape Kapinski, Deshmukh, Jin, Ito, Butts, Simulation-Based Approaches for Verification of Embedded Control Systems , IEEE Control Magazine, 2010 5 / 50

  6. Testing Testing Given: a system S , a property ϕ . Goal: generate a test suite { t i } i ∈ I . 6 / 50

  7. Testing Testing Given: a system S , a property ϕ . Goal: generate a test suite { t i } i ∈ I . Characteristics simple (run tests) black-box (unknown systems) versatile (guarantees, explainable failures. . . ) no formal guarantee too general 6 / 50

  8. Verification Verification Given: a model M , a property ϕ . Goal: automatically prove that M � ϕ . 7 / 50

  9. Verification Verification Given: a model M , a property ϕ . Goal: automatically prove that M � ϕ . Characteristics complex (design model, use specific techniques, so typically not used by engineers) white-box (known systems only) formal proof (strong guarantee) ill-suited to CPS (continuous systems) 7 / 50

  10. Verification Verification Given: a model M , a property ϕ . Goal: automatically prove that M � ϕ . Characteristics complex (design model, use specific techniques, so typically not used by engineers) white-box (known systems only) formal proof (strong guarantee) ill-suited to CPS (continuous systems) Theorem proving Given: a model M , a property ϕ . Goal: prove that M � ϕ . 7 / 50

  11. Falsification Falsification Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . 8 / 50

  12. Falsification Falsification Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Characteristics particular case of testing black-box (unknown systems) relatively simple no proof (no formal guarantee) 8 / 50

  13. Verification versus falsification Verification: finds a proof: system verifies property, finds nothing: Falsification: finds a counterexample: system violates property, finds nothing: 9 / 50

  14. Verification versus falsification Verification: finds a proof: system verifies property, finds nothing: nothing can be said. Falsification: finds a counterexample: system violates property, finds nothing: nothing can be said. 9 / 50

  15. Verification versus falsification Verification: finds a proof: system verifies property, finds nothing: nothing can be said. Falsification: finds a counterexample: system violates property, finds nothing: nothing can be said. Interaction: verification for falsification: constraining state space by reachability analysis, falsification for verification: coverage-based falsification techniques. 9 / 50

  16. Table of Contents Formal methods landscape 1 Framework 2 Hybrid systems 3 Formulas 4 Optimisation 5 10 / 50

  17. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . 11 / 50

  18. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? 11 / 50

  19. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system 11 / 50

  20. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system What is a property? 11 / 50

  21. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system What is a property? � logical formula 11 / 50

  22. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system What is a property? � logical formula What is a counterexample? 11 / 50

  23. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system What is a property? � logical formula What is a counterexample? � an input (and output) signal to the system that violates the property 11 / 50

  24. General framework Reminder Given: a system S , a property ϕ . Goal: generate a counterexample to S � ϕ . Questions What is a system? � hybrid system What is a property? � logical formula What is a counterexample? � an input (and output) signal to the system that violates the property Challenges infinite (and high-dimensional) search space non-linear dynamics 11 / 50

  25. The falsification loop σ in S σ out falsification σ out � ϕ generator checker ϕ σ out � ϕ ϕ ϕ σ in / ( σ in ,σ out ) 12 / 50

  26. The falsification loop falsification σ in S σ out σ out � ϕ generator checker ϕ σ out � ϕ ϕ ϕ σ in / ( σ in ,σ out ) 12 / 50

  27. The falsification algorithm Input: A system S , a formula ϕ , a satisfaction predicate � , and a timeout t max Output: A signal σ in such that S ( σ in ) � ϕ found = false; while not( found ) and t < t max do σ in = generate(); σ out = S ( σ in ); found = σ out � ϕ ; end if found then return σ in else return “timeout” end 13 / 50

  28. Optimisation-based falsification σ in S σ out falsification ρ ( σ out ,ϕ ) ≥ 0 generator checker ϕ ρ ( σ out ,ϕ ) < 0 ϕ ϕ σ in / ( σ in ,σ out ) 14 / 50

  29. Optimisation-based falsification falsification σ in S σ out ρ ( σ out ,ϕ ) ≥ 0 generator checker ϕ ρ ( σ out ,ϕ ) < 0 ϕ ϕ σ in / ( σ in ,σ out ) 14 / 50

  30. The optimisation-based falsification algorithm Input: A system S , a formula ϕ , a robustness function ρ , and a timeout t max Output: A signal σ in such that S ( σ in ) � ϕ found = false; while not( found ) and t < t max do σ in = search minimum( ρ ); σ out = S ( σ in ); found = ρ ( σ out , ϕ ) < 0; end if found then return σ in else return “timeout” end Required: ρ ( σ, ϕ ) ≥ 0 ⇐ ⇒ σ � ϕ 15 / 50

  31. Table of Contents Formal methods landscape 1 Framework 2 Hybrid systems 3 Formulas 4 Optimisation 5 16 / 50

  32. Hybrid systems Definition A hybrid system is a dynamical system that exhibits both continuous and discrete dynamic behavior – a system that can both flow (described by a differential equation) and jump (described by a state machine or automaton). Wikipedia Fehnker, Ivanˇ ci´ c, Benchmarks for Hybrid Systems Verification , Hybrid Systems: Computation and Control, pp 326–341 17 / 50

  33. Hybrid system: definition Hybrid system A Hybrid system is a tuple H = ( Q , X , GUARD , JUMP , U , FLOW) of: a finite set of modes Q , a family of continuous state spaces X = { X q ⊆ R n q | q ∈ Q } , GUARD q , q ′ ⊆ X q is the set of states in X q that can transition to mode q ′ , JUMP q , q ′ : X q → X q ′ describes the transition from q to q ′ , U is the input space, FLOW q , is a set of differential equations in X q and U , seen as a function X q × U × R ≥ 0 → X q , 18 / 50

  34. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · 19 / 50

  35. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · Q = { q 0 , q 1 } , 19 / 50

  36. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · Q = { q 0 , q 1 } , X q 0 = { T ∈ R | T ≥ T 0 } , 19 / 50

  37. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · Q = { q 0 , q 1 } , X q 0 = { T ∈ R | T ≥ T 0 } , X q 1 = { T ∈ R | T ≤ T 1 } , 19 / 50

  38. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · Q = { q 0 , q 1 } , X q 0 = { T ∈ R | T ≥ T 0 } , X q 1 = { T ∈ R | T ≤ T 1 } , GUARD q 0 , q 1 = { m } 19 / 50

  39. Example of hybrid system: thermostat T = T 0 | · q 0 q 1 dT dT dt = − KT dt = K ( h − T ) T ≥ T 0 T ≤ T 1 T = T 1 | · Q = { q 0 , q 1 } , X q 0 = { T ∈ R | T ≥ T 0 } , X q 1 = { T ∈ R | T ≤ T 1 } , GUARD q 0 , q 1 = { m } GUARD q 1 , q 0 = { M } 19 / 50

Recommend

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems Clovis Eberhart David Sprunger National Institute of Technology, Japan SOKENDAI lesson, July 1, 8, and 22 1 / 31 Quick reminder Falsification:

405 views • 36 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Information, Learning and Falsification David Balduzzi December 17, 2011 Max Planck Institute

Information, Learning and Falsification David Balduzzi December 17, 2011 Max Planck Institute

Effective information Algorithmic information Learning theory Falsification Conclusion Information, Learning and Falsification David Balduzzi December 17, 2011 Max Planck Institute for Intelligent Systems T ubingen, Germany Effective

695 views • 43 slides
Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides
Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are

Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are

Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are trained? They learn through a simple system of reinforcement learning, where the desired behaviors are met with rewards such as raw fish and back

248 views • 5 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald

A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald

A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald Tesauro, Nicholas K. Jongt, Rajarshi Das and Mohamed N. Bennanit Presentation: Nat McAleese (nm583) What is autonomic computing? computing systems

288 views • 11 slides
Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1 Research interests Clustering for building locally linear models Reinforcement learning for continuous dynamic systems Neural

604 views • 31 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views • 10 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

CSE 573: Artificial Intelligence Todo Reinforcement Learning Add simulations from 473 Dan Weld Add UCB bound (cut bolzman &amp; constant epsilon Add snazzy videos (pendulum, zico kolter See http://www See http://www-

462 views • 11 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

1 Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5 Environment Dynamics Gillian Hayes Components of a Problem 18th January 2007 Values and Action Values, V and Q Optimal Policies

109 views • 7 slides
Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007 sequential decision making under uncertainty ? How Can I ... ? Move around in the physical world (e.g. driving, navigation) Play and win a game

519 views • 21 slides

More recommend