Verified Runtime Monitoring: From Foundations to Practice Presenter: - PowerPoint PPT Presentation

Verified Runtime Monitoring: From Foundations to Practice Presenter: Brandon Bohrer 1 , based on works with: Yong Kiam Tan 1 , Stefan Mitsch 1 , Andrew Sogokon 1 , Edward Ahn 1 , David Held 1 , John Dolan 1 , Aman Khurana 1 , Magnus O. Myreen 2 , and Andr´ e Platzer 1 Carnegie Mellon University 1 Chalmers University of Technology 2 CMU V&V Workshop, Dec 12 2018

A Real Cyber-Physical System 2

A Scary Cyber-Physical System 2

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } {

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } { Far Enough?

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } { Far Velocity Enough? Envelope

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } { Fallback Far Velocity Enough? Envelope

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } { Fallback Far Velocity Enough? Physics Envelope

HPs Model Control and Environment 4 drive stop � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } { Fallback Far Velocity Enough? Physics Envelope Constraint

KeYmaera X Enables Model Verification 5

ModelPlex: Provably Correct Monitors 6 Monitor whether transitions from previous state � x to next state � x + are consistent with control, environment models. stop drive � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } {

ModelPlex: Provably Correct Monitors 6 Monitor whether transitions from previous state � x to next state � x + are consistent with control, environment models. stop drive � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } {

ModelPlex: Provably Correct Monitors 6 Monitor whether transitions from previous state � x to next state � x + are consistent with control, environment models. stop drive � � �� � � �� � α ≡ ( ? d ≥ ε V ; v := ∗ ; ? 0 ≤ v ≤ V ∪ v := 0 ); t := 0 ; env . � �� � � ∗ d ′ = − v , t ′ = 1 & t ≤ ε } {

Provable Monitor � Provable Sandbox 7 Sandboxed controller uses external controller when decision is safe, else uses verified fallback. Detects non-compliant plants. V := ∗ ; ε := ∗ ; d := ∗ ; t := ∗ ; // � x := ∗ ? d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0 ; // ? φ � t + := ∗ ; v + := ∗ ; d + := d ; x + := extCtrl // � ? ctrlMon ( d , t , v , d + , t + , v + ) ( ∪ t + := 0 ; v + := 0 ); x + := fallback // � t := t + ; v := v + ; x + // � x := � d + := ∗ ; t + := ∗ ; x + := ∗ // � x + ); ? plantMon ( � x ,� x + � ∗ d := d + ; t := t + // � x := �

Intervals Make ctrlMon and plantMon Computable 8 Example: Check whether π < e , efficiently. Solution: Conservative interval approximation Example Let ν I = { pi �→ [ 3 , 4 ] , e �→ [ 2 , 3 ] } , then • pi < w e is false ( ⊥ )

Intervals Make ctrlMon and plantMon Computable 8 Example: Check whether π < e , efficiently. Solution: Conservative interval approximation Example Let ν I = { pi �→ [ 3 , 4 ] , e �→ [ 2 , 3 ] } , then • pi < w e is false ( ⊥ ) • pi < w e + 3 is true ( ⊤ )

Intervals Make ctrlMon and plantMon Computable 8 Example: Check whether π < e , efficiently. Solution: Conservative interval approximation Example Let ν I = { pi �→ [ 3 , 4 ] , e �→ [ 2 , 3 ] } , then • pi < w e is false ( ⊥ ) • pi < w e + 3 is true ( ⊤ ) • pi < w e + 1 is ???

Intervals Make ctrlMon and plantMon Computable 8 Example: Check whether π < e , efficiently. Solution: Conservative interval approximation Example Let ν I = { pi �→ [ 3 , 4 ] , e �→ [ 2 , 3 ] } , then • pi < w e is false ( ⊥ ) • pi < w e + 3 is true ( ⊤ ) • pi < w e + 1 is a known unknown ( U ) When truth values can be unknown, resulting logic is 3-valued

Interval d L is 3-Valued 9 ∧ ⊤ U ⊥ ∨ ⊤ U ⊥ ⊤ ⊤ U ⊥ ⊤ ⊤ ⊤ ⊤ U U U ⊥ U ⊤ U U ⊥ ⊥ ⊥ ⊥ ⊥ ⊤ U ⊥ ] = [ l 1 ˇ + w l 2 , u 1 ˆ ω I [ ( θ 1 + θ 2 ) + w u 2 ] where ω I [ ( θ i ) ] = [ l i , u i ]  ⊤ if ω I [ ( θ i ) ] = ( l i , u i ) and u 1 < l 2    ω I [ ( θ 1 <θ 2 ) ] = ⊥ if ω I [ ( θ i ) ] = ( l i , u i ) and l 1 ≥ u 2    otherwise U ( ω I , ν I ) ∈ [ ( α ∪ β ) ] iff ( ω I , ν I ) ∈ [ ( α ) ] or ( ω I , ν I ) ∈ [ ( β ) ]

Interval d L is a Sound Approximation 10 Theorem (Interval Soundness for Formulas) • If ω ∈ ω I and ω I [ ( φ ) ]= ⊤ then ω ∈ [ [ φ ] ] • If ω ∈ ω I and ω I [ ( φ ) ]= ⊥ then ω / ∈ [ [ φ ] ] • No claims when ω I [ ( φ ) ]= U Generalizes naturally to programs, but CakeML sandbox only runs simpler formula case

Sandbox HP Already Verified 11 // � V := ∗ ; ε := ∗ ; d := ∗ ; t := ∗ ; x := ∗ ? d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0 ; // ? φ t + := ∗ ; v + := ∗ ; d + := d ; x + := extCtrl � // � ? ctrlMon ( d , t , v , d + , t + , v + ) ( ∪ t + := 0 ; v + := 0 ); x + := fallback // � t := t + ; v := v + ; // � x := � x + d + := ∗ ; t + := ∗ ; x + := ∗ // � � � ? 0 ≤ t + ≤ ε ∧ d + ≥ v ( ε − t + ) ; // ? plantMon ( � x ,� x + ) x + � ∗ d := d + ; t := t + // � x := �

Verified CakeML Source is Generated 11 CakeML source incorporates external control, actuation, sensing fun cmlSandboxBody state = if not (stop ()) then state.ctrl + := extCtrl state; state.ctrl := if intervalSem ctrlMon state = ⊤ then state.ctrl + else fallback state; actuate state.ctrl; state.sensors + := sense (); if intervalSem plantMon state = ⊤ then Runtime.fullGC (); state.sensors := state.sensors + ; cmlSandboxBody state else violation "Plant Violation"

CakeML Sandbox is Sound 12 Theorem (Soundness for CakeML Sandbox, Main Case) � ∈ [ � [ If { ω } ] , [ { ν } ] { cmlSandbox } ] then ([ ( ω ) ] , [ ( ν ) ]) ∈ [ ( sandbox ) ]

CakeML Compiler Preserves Guarantees 13

Code Executed on Sim, Soon Bot 14 Speed Ctrl Fail. Phys Fail. Collide World Sim Human Sim Human Sim Human Sim Human 1 6.69 17.4 .431 .913 .045 .377 0 0 2 5.78 10.7 .632 .890 .011 .417 0 0 3 7.89 29.9 1 .996 .01 .151 0 0 Table : Average speed, Monitor failure rates, safety violation rates, for AirSim, F1/10, and human driver in Rectangular World, NeighborHood, and Free-Range for Patrol and Goto missions

Proof Chain Justifies Transformations 15 ν | = ψ ⇑ Real arithmetic, ( ω, ν ) ∈ [ [ sandbox ] ] nondeterministic d L (KeYmaera X) ⇑ � ∈ Interval word arithmetic, � ω I , ν I [ ( sandbox ) ] nondeterministic d L (Isabelle/HOL) ⇑ � ∈ [ � [ Interval word arithmetic, { ω } ] , [ { ν } ] { cmlSandbox } ] deterministic CakeML (HOL4) ⇑ � ∈ { Interval word arithmetic, � { | ω | } , { | ν | } | CML ( cmlSandbox ) | } machine-executable ARM/x64

Takeaway Metaphor 16

Takeaway Metaphor 16

References I 17 Brandon Bohrer, Vincent Rahli, Ivana Vukotic, Marcus V¨ olp, and Andr´ e Platzer, Formally verified differential dynamic logic , Certified Programs and Proofs - 6th ACM SIGPLAN Conference, CPP 2017, Paris, France, January 16-17, 2017 (Yves Bertot and Viktor Vafeiadis, eds.), ACM, 2017, pp. 208–221. Joe Hurd, The OpenTheory standard theory library , NFM (Mihaela Gheorghiu Bobaru, Klaus Havelund, Gerard J. Holzmann, and Rajeev Joshi, eds.), LNCS, vol. 6617, Springer, 2011, pp. 177–191. Magnus O. Myreen and Scott Owens, Proof-producing synthesis of ML from higher-order logic , ICFP (Peter Thiemann and Robby Bruce Findler, eds.), ACM, 2012, pp. 115–126.