Krace: Data Race Fuzzing for Kernel File Systems Meng Xu , Sanidhya - PowerPoint PPT Presentation

Introduction Title Krace: Data Race Fuzzing for Kernel File Systems Meng Xu , Sanidhya Kashyap, Hanqing Zhao, Taesoo Kim May 1, 2020 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems 1 May 1, 2020

Introduction Data race concept Let’s talk about data race De fi nition: Two memory accesses from di ff erent threads such that 1. They access the same memory location 2. At least one of them is a write operation 3. They may interleave without restrictions (i.e., locks, orderings, etc) Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 2

Introduction A classic data race example The classic race condition example counter = 0 for(i=0; i<50000; i++) { for(i=0; i<50000; i++) { counter++; counter++; } } Thread 1 Thread 2 What is the value of counter when both threads terminate? Any value between 50,000 to 100,000 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 3

Introduction A classic data race example The classic race condition example counter = 0 for(i=0; i<50000; i++) { for(i=0; i<50000; i++) { lock(mutex); lock(mutex); counter++; counter++; unlock(mutex); unlock(mutex); } } Thread 1 Thread 2 What is the value of counter when both threads terminate? 100,000 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 4

Introduction Kernel concurrency High level of concurrency in the Linux kernel 22 threads run in the background! Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 5

Introduction A data race in the kernel A data race in the kernel p is a global pointer initialized to null if (! p ) if (! p ) p = kmalloc(...); p = kmalloc(...); Information lost! Thread 1 Thread 2 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 6

Introduction A data race in the kernel A data race in the kernel p is a global pointer initialized to null This data race can be easily detected… if we drive the execution into these code paths at runtime if (! p ) if (! p ) p = kmalloc(...); p = kmalloc(...); Information lost! Thread 1 Thread 2 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 7

Background Fuzzing in general Fuzzing as a way to explore the program Start 1 2 3 4 5 6 7 8 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 8

Background Edge coverage Code coverage as an approximation Start 1 open(“some-file”, O_READ, ...) 2 3 4 5 6 7 8 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 9

Background Edge coverage Code coverage as an approximation Start 1 open(“some-file”, O_READ, ...) 2 3 open(“some-file”, O_WRITE, ...) 4 5 6 7 8 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 10

Background Edge coverage Code coverage as an approximation Start 1 open(“some-file”, O_READ, ...) 2 3 open(“some-file”, O_WRITE, ...) 4 open(“new-file”, O_READ, ...) 5 6 7 8 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 11

Background Edge coverage Code coverage as an approximation Start 1 open(“some-file”, O_READ, ...) 2 3 open(“some-file”, O_WRITE, ...) Coverage growth stalled! 4 open(“new-file”, O_READ, ...) 5 ...... 20 trials 6 open(“some-file”, O_RDWR, ...) 7 8 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 12

Background Edge coverage Code coverage as an approximation Start 1 open(“some-file”, O_READ, ...) 2 3 open(“some-file”, O_WRITE, ...) 10 4 open(“new-file”, O_READ, ...) 11 12 5 ...... 20 trials 13 6 open(“some-file”, O_RDWR, ...) 7 8 14 rename(“new-file”, “old-file”) 9 End Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 13

Background Existing kernel fuzzers The conventional fuzzing process Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 14

Background Existing kernel fuzzers The conventional fuzzing process Syscall Program Crashed? Test case generator executor Memory error The code coverage metric backs all modern kernel fuzzers including Syzkaller, kAFL, and their follow-ups, and is one of the key Feedback reason why over 200 memory errors were found and reported during the past few years! code coverage Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 15

Background Motivation Back to our data race example p is a global pointer initialized to null if (! p ) if (! p ) p = kmalloc(...); p = kmalloc(...); Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 16

Background Motivation Back to our data race example p is a global pointer initialized to null No CRASH when the data race is triggered! if (! p ) if (! p ) p = kmalloc(...); p = kmalloc(...); Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 17

Design Data race checker Bring out data races explicitly with a checker Signaled? Data race checker Data race Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 18

Design Data race checker Checking data races - locking Syscall Workqueue Fork-style - Work queues - Kernel threads - RCU callbacks - Timer functions lock - Software-based interrupts - Inter-processor interrupts W lock Join-style unlock R - Wait_* (e.g., wait_event) - Semaphores unlock Publisher-subscriber - RCU pointer operations Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 19

Design Data race checker Checking data races - ordering (causality) Syscall Timer Workqueue Fork-style - Work queues - Kernel threads W - RCU callbacks - Timer functions delayed_work - Software-based interrupts <timer start> - Inter-processor interrupts Join-style - Wait_* (e.g., wait_event) <timer end> - Semaphores queue_work <work start> Publisher-subscriber R - RCU pointer operations Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 20

Design Data race checker Bring out data races explicitly with a checker Signaled? Data race checker Data race Syscall Program Crashed? Test case generator executor Memory error Feedback code coverage Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 21

Design Interactions between threads A slightly complicated data race G[ … ] is all null at initialization sys_readlink (path, ...): sys_truncate (size, ...): global A = 1; global A = 0; local y; local x; if (size > 4096) { if (IS_DIR(path)) { x = A + 1; y = A * 2; if (! G[ x ] ) if (! G[ y ] ) G[ x ] = kmalloc(...); G[ y ] = kmalloc(...); } } Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 22

Design Interactions between threads A slightly complicated data race G[ … ] is all null at initialization sys_readlink (path, ...): sys_truncate (size, ...): global A = 1; global A = 0; local y; local x; if (size > 4096) { if (IS_DIR(path)) { x = A + 1; y = A * 2; if (! G[ x ] ) if (! G[ y ] ) G[ x ] = kmalloc(...); G[ y ] = kmalloc(...); } } Thread 1 Thread 2 *Assume sequential consistency. Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 23

Design Interactions between threads Case simpli fi ed A = 1; A = 0; x = A + 1; y = A * 2; Thread 1 Thread 2 Can we reach x == y? Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 24

Design Interactions between threads Case simpli fi ed A = 1; A = 1; A = 1; x = A + 1; A = 0; A = 0; A = 0; x = A + 1; y = A * 2; y = A * 2; y = A * 2; x = A + 1; A = 1; A = 0; x = A + 1; x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 y = A * 2; Thread 1 Thread 2 A = 0; A = 0; A = 0; Can we reach x == y? y = A * 2; A = 1; A = 1; x = A + 1; A = 1; y = A * 2; x = A + 1; x = A + 1; y = A * 2; x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 25

Design Interactions between threads All interleavings yield to the same code coverage! A = 1; A = 1; A = 1; global A = 1; local x; x = A + 1; A = 0; A = 0; if (IS_DIR(path)) A = 0; x = A + 1; y = A * 2; x = A + 1; y = A * 2; y = A * 2; x = A + 1; if (! G[ x ] ) x = 2, y = 0 x = 1, y = 0 x = 1, y = 0 G[ x ] = kmalloc(...); ... A = 0; A = 0; A = 0; global A = 0; local y; y = A * 2; A = 1; A = 1; if (size > 4096) x = A + 1; A = 1; y = A * 2; y = A * 2; x = A + 1; x = A + 1; y = A * 2; if (! G[ y ] ) G[ y ] = kmalloc(...); x = 2, y = 0 x = 2, y = 2 x = 2, y = 2 ... Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 26

Design Missing information in edge coverage Incompleteness of CFG edge coverage Meng Xu (Georgia Tech) Krace: Data Race Fuzzing for Kernel File Systems May 1, 2020 27