[PPT] - Verifying concurrent software using movers in CSPEC Tej Chajed , PowerPoint Presentation

SLIDE 1

Verifying concurrent software using movers in CSPEC

Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich MIT CSAIL and *Microsoft

SLIDE 2

Concurrent software is difficult to get right

2

Programmer cannot reason about code in sequence…

SLIDE 3

Concurrent software is difficult to get right

3

instead, must consider many executions: Programmer cannot reason about code in sequence…

SLIDE 4

Concurrent software is difficult to get right

3

instead, must consider many executions: … Programmer cannot reason about code in sequence…

SLIDE 5

Goal: verify concurrent software

4

SLIDE 6

Challenge for formal verification

Proofs must also cover every execution
Many approaches to managing this complexity
movers [Lipton, 1975]
rely-guarantee [1983]
RGSep [CONCUR 2007]
FCSL [PLDI 2015]
Iris [POPL 2017, LICS 2018, others]
many others

5

SLIDE 7

Challenge for formal verification

Proofs must also cover every execution
Many approaches to managing this complexity
movers [Lipton, 1975]
rely-guarantee [1983]
RGSep [CONCUR 2007]
FCSL [PLDI 2015]
Iris [POPL 2017, LICS 2018, others]
many others
This work: our experience using movers

5

SLIDE 8

Movers: reduce concurrent executions to sequential ones

6

time B 3 2 A 1

blue thread green thread

1 2 3 A B

SLIDE 9

Movers: reduce concurrent executions to sequential ones

6

has the same effect as movers B 3 2 A 1

blue thread green thread

1 2 3 A B

B 3 2 A 1

SLIDE 10

Movers: reduce concurrent executions to sequential ones

6

has the same effect as movers sequential reasoning B 3 2 A 1

blue thread green thread

1 2 3 A B

B 3 2 A 1 B 3 2 A 1

SLIDE 11

Prior systems with mover reasoning

7

CIVL [CAV ’15, CAV ’18] framework relies pen & paper proofs IronFleet [SOSP ’15]

nly move network send/receive

SLIDE 12

Contribution: CSPEC

Framework for verifying concurrency in systems software
general-purpose movers
patterns to support mover reasoning
machine checked in Coq to support extensibility

8

SLIDE 13

Contribution: CSPEC

Framework for verifying concurrency in systems software
general-purpose movers
patterns to support mover reasoning
machine checked in Coq to support extensibility
Case studies using CSPEC
Lock-free file-system concurrency
Spinlock on top of x86-TSO (see paper)

8

SLIDE 14

Case study: mail server using file-system concurrency

9

spool file system mbox

SLIDE 15

Mail servers exploit file-system concurrency

10

spool file system # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”) mbox

1 2 3

SLIDE 16

Mail servers exploit file-system concurrency

11

spool file system msg mbox

1 2 3

# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 17

Spooling avoids reading partially-written messages

12

spool file system mbox

1 2 3

$TID =10 # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 18

Spooling avoids reading partially-written messages

12

spool file system

10

mbox

1 2 3

$TID =10 # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

10

SLIDE 19

Threads use unique IDs to avoid conflicts

13

spool file system

10

msg mbox

1 2 3

$TID =10 $TID =11 # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 20

Threads use unique IDs to avoid conflicts

14

spool file system

10

mbox

1 2 3

$TID =10 $TID =11 # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 21

Threads use unique IDs to avoid conflicts

14

spool file system

10 11

mbox

1 2 3

$TID =10 $TID =11 # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 22

Timestamps help generate unique message names

15

spool file system mbox

1 2 3 4 10 11

link(/spool/11, /mbox/4) # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 23

Timestamps help generate unique message names

16

spool mbox

1 2 3

file system

4 10 11

link(/spool/10, /mbox/4)

EEXISTS ✗

# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 24

Timestamps help generate unique message names

17

spool file system mbox

1 2 3 4 5 10 11

link(/spool/10, /mbox/5) # accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 25

Delivery concurrency does not use locks

18

spool file system mbox

1 2 3 5 4 10

# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 26

Delivery concurrency does not use locks

19

spool file system mbox

1 2 3 4 5

# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup unlink(“/spool/$TID”)

SLIDE 27

Proving delivery correct in CSPEC

20

file-system spec delivery specification

implementation and proof

CSPEC CSPEC provides supporting definitions and theorems

SLIDE 28

Proof engineer reasons about file-system

perations

21

def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break unlink(“/spool/$TID”) create( /sp/$TID, msg) ✓ link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t) ✓ unlink( /sp/$TID) ✓

SLIDE 29

Proof engineer reasons about file-system

perations

21

def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break unlink(“/spool/$TID”) create( /sp/$TID, msg) ✓ link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t) ✓ unlink( /sp/$TID) ✓ create(“/spool/$TID”) write(“/spool/$TID”, msg)

collapsed to

ne operation

SLIDE 30

Proof engineer reasons about interleaving of file- system operations

22

def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break unlink(“/spool/$TID”)

We assume file-system operations are atomic

create( /sp/$TID, msg) ✓ link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t) ✓ unlink( /sp/$TID) ✓

create

✓

link unlink

SLIDE 31

Proving atomicity of delivery

23

atomicity: concurrent deliveries appear to execute all at once (in some order)

create

✓

link

✗

link unlink create

✓

link unlink create

✓

link unlink

deliver

create

✓

link

✗

link unlink

deliver

SLIDE 32

Proving atomicity of delivery

23

atomicity: concurrent deliveries appear to execute all at once (in some order)

create

✓

link

✗

link unlink create

✓

link unlink create

✓

link unlink

deliver

create

✓

link

✗

link unlink

deliver

Step 1: developer identifies commit point

SLIDE 33

Proving atomicity of delivery

23

atomicity: concurrent deliveries appear to execute all at once (in some order)

create

✓

link

✗

link unlink create

✓

link unlink create

✓

link unlink

deliver

create

✓

link

✗

link unlink

deliver

Step 1: developer identifies commit point Step 2: prove operation occurs logically at commit point

SLIDE 34

Example of movers for this execution

24

create

✓

link

✗

link unlink create

✓

link unlink

SLIDE 35

Example of movers for this execution

24

create

✓

link

✗

link unlink create

✓

link unlink create

✓

link

✗

link unlink create

✓

link unlink

SLIDE 36

Example of movers for this execution

24

create

✓

link

✗

link unlink create

✓

link unlink create

✓

link

✗

link unlink create

✓

link unlink create

✓

link

✗

link unlink create

✓

link unlink

SLIDE 37

Right mover can be reordered after any green thread operation

25

A r A r

SLIDE 38

Right mover can be reordered after any green thread operation

25

A r A r

left movers are the converse

SLIDE 39

Movers need to consider only possible operations from other threads

26

create( /sp/$TID, msg) link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t) unlink( /sp/$TID) is one of

A r A r

for all green operations , is a right mover if A left movers are the converse

SLIDE 40

Example mover proof: failing link is a right mover

27

Proof sketch (only case):

link

link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

SLIDE 41

Example mover proof: failing link is a right mover

27

Proof sketch (only case):

link

link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t) ✓

$t $t ≠

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

✗

link

✓

link

(otherwise then is impossible)

SLIDE 42

Example mover proof: failing link is a right mover

27

Proof sketch (only case):

link

link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t) ✓

$t $t ≠

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

✗

link

✓

link

(otherwise then is impossible) link operations are independent

⟹

SLIDE 43

Failing link does not move left

28

SLIDE 44

Failing link does not move left

28

link( /sp/$TID, /mbox/$t)

EEXISTS ✗

link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t) ✓ link( /sp/$TID, /mbox/$t)

EEXISTS ✗

if =

$t $t

SLIDE 45

Challenge: how to limit what other operations to consider in mover proofs?

29

Delivery File system

deliver
create(f, d)
link(f1, f2)
unlink(f)
rename(f1, f2)

mover proof?

SLIDE 46

Challenge: how to limit what other operations to consider in mover proofs?

29

Delivery File system

deliver
create(f, d)
link(f1, f2)
unlink(f)
rename(f1, f2)

mover proof?

create( f1, d) create( f2, d) create( f1, d) create( f2, d)

if filenames are identical

SLIDE 47

Layers enable mover reasoning

30

Delivery File system

deliver
create(f, d)
link(f1, f2)
unlink(f)
rename(f1, f2)

Layers limit what operations are available use multiple layers to make operations movers

⟹

SLIDE 48

Layers enable mover reasoning

31

Delivery File system Restricted file system restrict arguments to include $TID

create(/spool/$TID, d)
link(/spool/$TID, /mbox/$t)
unlink(/spool/$TID)

mover proof ✓

Layers limit what operations are available use multiple layers to make operations movers

⟹

SLIDE 49

Layers enable mover reasoning

31

Delivery File system Restricted file system upper layers can only use restricted operations

create(/spool/$TID, d)
link(/spool/$TID, /mbox/$t)
unlink(/spool/$TID)

mover proof ✓

Layers limit what operations are available use multiple layers to make operations movers

⟹

SLIDE 50

Movers are a layer proof pattern

32

mover pattern Obligation for developer: movers for each implementation

layer 1 layer 2

A B C D

foo bar

SLIDE 51

Movers are a layer proof pattern

32

mover pattern

A B D C B A C

def foo: def bar: Obligation for developer: movers for each implementation

layer 1 layer 2

A B C D

foo bar

SLIDE 52

Movers are a layer proof pattern

32

mover pattern

A B D C B A C

def foo: def bar: Obligation for developer: movers for each implementation CSPEC theorem: entire layer implementation is atomic

layer 1 layer 2

A B C D

foo bar

SLIDE 53

CSPEC provides other patterns to support mover reasoning

Abstraction / forward simulation
Invariants
Error state
Protocols
Retry loops
Partitioning

33

(see paper for details)

pattern

bligations

proof connecting layers

SLIDE 54

Using CSPEC to verify CMAIL

34

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

CSPEC

auto generated framework

SLIDE 55

Using CSPEC to verify CMAIL

34

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

  Coq extraction CSPEC

auto generated framework

SLIDE 56

Using CSPEC to verify CMAIL

34

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

  Coq extraction

GHC

CSPEC executable

auto generated framework

Linux

SLIDE 57

What is proven vs. assumed correct?

35

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

Coq extraction GHC CSPEC executable

✓

k

Coq proof checker Linux proven

auto generated

assumed correct

SLIDE 58

Concurrency inside CMAIL is proven

36

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

Coq extraction GHC CSPEC executable

✓

k

Coq proof checker Linux proven

auto generated

assumed correct

SLIDE 59

Trust that the tools and OS are correct

37

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

Coq extraction GHC CSPEC executable

✓

k

Coq proof checker Linux proven

auto generated

assumed correct

SLIDE 60

Mail server-specific assumptions

38

file-system spec mail library spec implementation layers patterns

CMAIL (Coq) Coq

calls to file-system SMTP + POP3 extracted implementation

CMAIL (Haskell)

Coq extraction GHC CSPEC executable

✓

k

Coq proof checker Linux proven

auto generated

assumed correct

SLIDE 61

Evaluation

Can CMAIL exploit file-system concurrency for speedup?
How much effort was verifying CMAIL?
What is the benefit of CSPEC’s machine-checked proofs?

39

SLIDE 62

CMAIL achieves speedup with multiple cores

40

kreq/s 35 70 105 140 # cores 1 2 3 4 5 6 7 8 9 10 11 12

CMAIL GoMail

SLIDE 63

CMAIL was work but doable

41

proof:code ratio CMAIL 11.5x CertiKOS 13.8x IronFleet 7.7x IronClad 4.8x CompCert 4.6x

Took two authors 6 months

{

concurrent

{

sequential

SLIDE 64

Machine-checked proofs give confidence in framework changes

42

Three anecdotes of changes to CSPEC: Machine-checked proofs ensure soundness of entire system

SLIDE 65

Machine-checked proofs give confidence in framework changes

42

Implemented partitioning pattern to support multiple users

Three anecdotes of changes to CSPEC: Machine-checked proofs ensure soundness of entire system

SLIDE 66

Machine-checked proofs give confidence in framework changes

42

Implemented partitioning pattern to support multiple users
Improved mover pattern for a CMAIL left mover proof

Three anecdotes of changes to CSPEC: Machine-checked proofs ensure soundness of entire system

SLIDE 67

Machine-checked proofs give confidence in framework changes

42

Implemented partitioning pattern to support multiple users
Improved mover pattern for a CMAIL left mover proof
Implemented error-state pattern for the x86-TSO lock proof

Three anecdotes of changes to CSPEC: Machine-checked proofs ensure soundness of entire system

SLIDE 68

CSPEC is a framework for verifying concurrency in systems software

Layers and patterns (esp. movers) make proofs manageable
Machine-checked framework supports adding new patterns
Evaluated by verifying mail server and x86-TSO lock

github.com/mit-pdos/cspec

43

SLIDE 69

CSPEC is a framework for verifying concurrency in systems software

Layers and patterns (esp. movers) make proofs manageable
Machine-checked framework supports adding new patterns
Evaluated by verifying mail server and x86-TSO lock

github.com/mit-pdos/cspec

43

poster #1

SLIDE 70

44

Backup slides

CMAIL perf experimental setup

SLIDE 71

Performance experiment setup for CMAIL

45

in-memory file system

process

client

deliver + pickup

CMAIL

core 1

SLIDE 72

Performance experiment setup for CMAIL

45

…

in-memory file system

process

client

deliver + pickup

CMAIL

core 1

process

client

deliver + pickup

CMAIL

core 2

process

client

deliver + pickup

CMAIL

core 12