Understanding and Finding Crash-Consistency Bugs in Parallel File - PowerPoint PPT Presentation

Understanding and Finding Crash-Consistency Bugs in Parallel File Systems Jinghan Sun , Chen Wang, Jian Huang, and Marc Snir University of Illinois at Urbana-Champaign Contact: Jinghan Sun (js39@illinois.edu)

PFS failures are frequent and expensive PFS Recovery Time Single Day Failure Cost PFS Failure Frequency 40% 75% 50% 34% 34% 41% 59% 60% 40% 32% 35% 45% 30% 24% 30% 24% 20% 14% 16% 12% 14% 8% 6% 15% 10% 4% 8% 3% 0% 0% 0% <$100K $100K-$500K $500K-$1M Weekly Monthly Never Not Reported <1 day 2-3 days 1 week >1 week >$1M Not Reported 41% of PFSes suffer from monthly or weekly failures, their recovery process is expensive & time consuming Source: Hyperion Research 2019 1

Introduction to parallel file systems Parallel I/O library • Higher level abstractions: Datasets, groups, collective I/O APIs Parallel file system • Data striping • Separate metadata management • POSIX-compliant HPC I/O stack is much more complex than the traditional I/O stack 2

A PFS failure example PFS may experience severe data loss after system-wide power outage 3

A study of crash vulnerabilities on PFSes Atomic Replace via Rename Data loss Write-ahead Logging HDF5 DoS create delete rename Inaccessible resize dataset update + = Two Filesystems Seven Workloads 34 Vulnerabilities 4

PFS crash vulnerabilities Number of Vulnerabilities on Different Filesystems 10 9 Vulnerability : Parallel I/O stack may BeeGFS OrangeFS ext4 8 corrupt user files if crash happens in 7 the middle of the computation 6 (depending on the precise timing of 5 disk accesses) 4 3 2 1 0 ARVR WAL H5-create H5-delete H5-resize H5-rename H5-write The complexity of PFS stack makes it more vulnerable to system crashes 5

Crash vulnerability example // atomic replace via rename (ARVR) bool atomic_update(){ int fd = create("file.tmp"); write(fd, new, size); close(fd); storage #1 metadata storage #2 rename("file.tmp","file.txt"); } The function tries to update a file content atomically BeeGFS with 2 storage and 1 metadata server 6

Crash vulnerability example creat idfile beegfs-client idfile link dentries/tmp // atomic replace via rename (ARVR) creat chunk bool atomic_update(){ int fd = creat("file.tmp"); append chunk write(fd, new, size); close(fd); rename("file.tmp","file.txt"); dentries/tmp rename } dentries/file unlink idfile_2 Persistence order ≠ Program order! unlink old_chunk unlink old_chunk Two vulnerabilities discovered at system crash! storage #2 storage #1 metadata 7

Crash vulnerability example creat idfile Inconsistency No.1 idfile link dentries/tmp Cause rename() persisted before append() creat chunk Ordering Cross-node dependency append chunk Consequence Data loss dentries/tmp Fixed by fsck? No rename dentries/file unlink idfile_2 unlink unlink old_chunk old_chunk Persisted operations op param Non-persisted operations op param storage #2 storage #1 metadata 8

Crash vulnerability example creat idfile Inconsistency No.2 idfile link dentries/tmp Cause unlink() persisted before rename() creat chunk Ordering Cross-node dependency append chunk Consequence Data loss dentries/tmp Fixed by fsck? No rename dentries/file unlink idfile_2 unlink old_chunk unlink old_chunk Persisted operations op param Non-persisted operations op param storage #2 storage #1 metadata 9

Crash vulnerability example creat idfile Inconsistency No.3 idfile link dentries/tmp Cause unlink() persisted before rename() creat chunk Ordering Intra-node dependency append chunk Consequence Data loss dentries/tmp Fixed by fsck? Yes rename dentries/file unlink idfile_2 unlink old_chunk unlink old_chunk Persisted operations op param Non-persisted operations op param storage #2 storage #1 metadata 10

PFSCheck design filesystem workload checker & app-level recovery 1 Record server-side crash state traces failed crash state 3 5 2 Test Classification Crash Report … … passed crash state client-side traces 4 … Legal replay legal state legal state legal state File system images that satisfy the given consistency model consistency model Discovering PFS crash vulnerabilities systematically & efficiently 11

The PFSCheck design Automated workload generation filesystem workload checker & app-level recovery Unified API for I/O libraries 1 • Record server-side crash state 1. Multi-level tracing traces failed crash state Test Classification Crash Report … … Joint server-side & client-side I/O calls • tracing passed crash state client-side traces Network packet tracing • … Legal replay legal state legal state legal state File system images that satisfy the given consistency model consistency Correlation between server & client • model operations 12

The PFSCheck design 2. Efficient crash state emulation filesystem workload checker & app-level recovery 1 Record Automated crash state generation via • trace permutation server-side crash state traces failed crash state 3 2 Perform necessary post-crash recovery • Test Classification Crash Report … … passed crash state 3. Consistency testing client-side traces Workload-specific consistency checker • … Legal replay legal state legal state legal state File system images that satisfy the given consistency model consistency model 13

The PFSCheck design 4. Legal replay based on given consistency filesystem workload checker & app-level model recovery 1 Record Crash consistency model specifies the • server-side crash state traces legitimate crash states of the parallel file failed crash state 3 5 2 Test Classification system Crash Report … … passed crash state client-side 5. Crash vulnerability classification traces 4 … If a vulnerable crash state is not a legal Legal replay • legal state legal state legal state state, we attribute it to PFS File system images that satisfy the given consistency model consistency model Otherwise, I/O libraries are blamed • 14

Conclusion • Motivation: crash vulnerabilities could be exacerbated on PFSes, due to the complexity of the parallel I/O stack • Study: – the number of crash consistency bugs on BeeGFS and OrangeFS is higher than local filesystem – the workload can fail in more ways on PFSes – the consistency relies on persistency reordering across nodes • Proposed framework: PFS-specific crash consistency checker with a focus on automation and efficiency 15

Thank you! Contact: Jinghan Sun (js39@illinois.edu) 16