Planning for Change in a Formal Verification of the Raft Consensus - PowerPoint PPT Presentation

Planning for Change 
 in a Formal Verification of the 
 Raft Consensus Protocol Doug James Steve Zach Mike Tom Woos Wilcox Anton Tatlock Ernst Anderson

Contributions First formal proof of Raft’s safety first verified implementation! Large-scale Verdi case study stress test; reverification inevitable Proof engineering lessons affinity lemmas, etc.

Distributed Systems

Reliably deliver procrastination

Also serious infrastructure

One day last summer...

One day last summer...

One day last summer...

How distributed systems fail

Related Work EventML [LADA12, AVoCS15] language for verified distributed systems IronFleet [SOSP15] liveness, log compaction, serialization Verdi [PLDI15] network semantics, transformers, higher-order

Verdi background Network semantics operational semantics define network behavior Verified system transformers prove property transfer to adversarial network App App VST App App App App

Big Picture Past: Verdi Framework compositional fault tolerance Present: Verified Raft critical piece of infrastructure Future: dynamically upgrading systems program logic

Outline Verification Challenge ⇒ state machine replication Raft Algorithm implemented in Verdi Proof Overview and lessons learned

Replication for fault tolerance critical components must not fail

Replication for fault tolerance replicas must be consistent with each other ⇒ available if n/2 nodes are up

Replication for fault tolerance ⇒

Replication correctness ⇒

Replication correctness ≈ linearizability cluster presents consistent order of operations to clients

Internal Correctness ≈ linearizability follows from internal correctness: state machine safety

Goal: Verify Raft Prove State Machine Safety Reduce linearizability to State Machine Safety ⇒ [PLDI15]

Goal: Verify Raft 45k ⇒ LOC 5k Lin. SMS

Outline Verification Challenge ⇒ state machine replication Raft Algorithm implemented in Verdi Proof Overview and lessons learned

Formalizing the network state of the world packets in flight history of I/O data @ nodes

Formalizing the network

Formalizing the network

Defining network semantics Σ 0 = Σ [ dst 7! σ 0 ] H net ( dst, Σ [ dst ] , src, m )=( σ 0 , o, P 0 ) Deliver ( { ( src, dst, m ) } ] P, Σ , T ) ( P ] P 0 , Σ 0 , T ++ h o i )

Defining network semantics Σ 0 = Σ [ dst 7! σ 0 ] H net ( dst, Σ [ dst ] , src, m )=( σ 0 , o, P 0 ) Deliver ( { ( src, dst, m ) } ] P, Σ , T ) ( P ] P 0 , Σ 0 , T ++ h o i ) p 2 P Duplicate ( P, Σ , T ) ( P ] { p } , Σ , T ) Drop ( { p } ] P, Σ , T ) ( P, Σ , T ) Σ 0 = Σ [ n 7! σ 0 ] H tmt ( n, Σ [ n ]) = ( σ 0 , o, P 0 ) Timeout ( P, Σ , T ) ( P ] P 0 , Σ 0 , T ++ h tmt , o i )

Defining network semantics Σ 0 = Σ [ dst 7! σ 0 ] H net ( dst, Σ [ dst ] , src, m )=( σ 0 , o, P 0 ) Deliver ( { ( src, dst, m ) } ] P, Σ , T ) ( P ] P 0 , Σ 0 , T ++ h o i ) p 2 P Duplicate systems defined by handlers ( P, Σ , T ) ( P ] { p } , Σ , T ) Drop ( { p } ] P, Σ , T ) ( P, Σ , T ) Σ 0 = Σ [ n 7! σ 0 ] H tmt ( n, Σ [ n ]) = ( σ 0 , o, P 0 ) Timeout ( P, Σ , T ) ( P ] P 0 , Σ 0 , T ++ h tmt , o i )

Implementing Raft election replication ... Term 1 Term 2 Term 3

Implementing Raft: Leader Election Followers Vote ReqVote Candidate ... Term 1 Term 2 Term 3

Implementing Raft ... Term 1 Term 2 Term 3

Implementing Raft: Log Replication Followers AppendAck Append Leader commits entry when receiving n/2 acks Leader ... Term 1 Term 2 Term 3

Outline Verification Challenge ⇒ state machine replication Raft Algorithm implemented in Verdi Proof Overview and lessons learned

Verifying Raft: Show linearizability ≈

Verifying Raft: Approach ⇒

State Machine Safety Nodes agree about committed entries ⇒ since only committed entries executed proof by induction on an execution

State Machine Safety: Proof I ⇒ I not inductive!

State Machine Safety: Proof 90 invariants Lemma … in total Lemma Lemma I I true initially preserved I I ⇒

The burden of proof Lemma … Lemma Lemma P true initially P preserved Re-verification is the primary challenge: P with ghost state - invariants are not inductive - not-yet-verified code is wrong P ⇒ P - need additional invariants

The burden of proof Lemma … Re-verification is the primary challenge Lemma Lemma P true initially P preserved P with ghost state Proof engineering techniques help: - affinity lemmas P ⇒ P - intermediate reachability - structural tactics - information hiding

Ghost State: Example Capture all entries received by a node Log (real) allEntries (ghost) Follower A ,B,C A ,D {A,B,C,D} {A,D} Append [A],B,C Leader A ,B,C {A,B,C}

Affinity Lemmas: Example every invariant of entries in logs is Affinity Lemma e log ⇒ invariant of entries ∈ in allEntries e.term > 0 ⇒ e.term > 0 e allEntries ∈

Affinity Lemmas: Example every invariant of entries in logs is Affinity Lemma e log ⇒ invariant of entries ∈ in allEntries P e ⇒ P e e allEntries ∈

Affinity Lemmas Ex 1: Relate ghost state to real state transfer properties once and for all Ex 2: Relate current messages to past response => past request

Structured Handlers: Example handler = update_state ; respond net net update_state handler net i respond net’ net’

Structured Handlers: Example handler = update_state ; respond I net net update_state handler net i respond I net’ net’

Structured Handlers: Example handler = update_state ; respond I I net net update_state I handler net i respond I I net’ net’

The burden of proof Lemma … Re-verification is the primary challenge Lemma Lemma P true initially P preserved P with ghost state Proof engineering techniques help: - affinity lemmas P ⇒ P - intermediate reachability - structural tactics - information hiding

Contributions First formal proof of Raft’s safety first verified implementation! Large-scale Verdi case study stress test; reverification inevitable Proof engineering lessons affinity lemmas, etc.

Planning for Change 
 in a Formal Verification of the 
 Raft Consensus Protocol Doug James Steve Zach Michael Tom Woos Wilcox Anton Tatlock Ernst Anderson