Badger : Complexity Analysis with Fuzzing and Symbolic Execution - PowerPoint PPT Presentation

Badger : Complexity Analysis with Fuzzing and Symbolic Execution Yannic Noller Rody Kersten Corina S. Pasareanu yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 1

Problem Solution Example Evaluation Related Summary Complexity Analysis discover vulnerabilities related to worst-case time/ space complexity, e.g., Denial-of-Service 0 public void sort ( int [] a) { find worst-case input: 
 1 int N = a.length; automated + fast + concrete 2 for ( int i = 1; i < N; i++) { 3 int j = i - 1; 4 int x = a[i]; • worst-case complexity: 5 while ((j >= 0) && (a[j] > x)) { O(n 2 ) 6 a[j + 1] = a[j]; 7 j--; • e.g. a=[8, 7, 6] (n=3) 8 } 9 a[j + 1] = x; 10 } 11 } Insertion Sort yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 2

Problem Solution Example Evaluation Related Summary Our Contributions • combine fuzzing and symbolic execution to find algorithmic complexity vulnerabilities • Badger, a framework for analysis of Java applications • analysis parameterized by a cost metric • handling of user-defined cost yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 3

Problem Solution Example Evaluation Related Summary Badger KelinciWCA (based on AFL) increased fuzzer coverage or increased cost exchange interesting 
 inputs symbolic execution based on Symbolic PathFinder (SPF) fuzzer and symbolic execution run in parallel yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 4

Problem Solution Example Evaluation Related Summary KelinciWCA • based on AFL, extends Kelinci [Kersten2017] • mutation-based greybox fuzzing • cost-guided fuzzer: coverage + cost • cost metrics: timing / memory / user-defined • maintain current highscore yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 5

Problem Solution Example Evaluation Related Summary SymExe with SPF fuzzer export inputs import inputs 1 5 SymExe interesting input Trie Extension / 
 2 concolic execution Input Assessment includes most promising node worst-case analysis new input Input 
 4 3 Exploration Generation trie-guided symbolic model generation execution path condition bounded symbolic input generation execution yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 6

Problem Solution Example Evaluation Related Summary Example Trie extension with initial input. The most promising node get selected. initial input 
 a=[37, 42, 48] 0 public void sort ( int [] a) { 1 int N = a.length; 2 for ( int i = 1; i < N; i++) { id=0 
 3 int j = i - 1; ROOT 4 int x = a[i]; score=7.0 5 while ((j >= 0) && (a[j] > x)) { 6 a[j + 1] = a[j]; id=1 
 7 j--; line=5 8 } choice=0 
 9 a[j + 1] = x; score=7.0 10 } 11 } id=2 line=5 Insertion Sort choice=0 
 score=7.0 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 7

Problem Solution Example Evaluation Related Summary Example Exploration and input generation . id=0 
 ROOT 0 public void sort ( int [] a) { score=7.0 1 int N = a.length; 2 for ( int i = 1; i < N; i++) { id=1 
 3 int j = i - 1; line=5 4 int x = a[i]; choice=0 
 5 while ((j >= 0) && (a[j] > x)) { score=7.0 6 a[j + 1] = a[j]; 7 j--; id=2 id=3 8 } line=5 line=5 9 a[j + 1] = x; choice=0 
 choice=1 
 score=7.0 10 } score=? 11 } pc = sym _0 ≤ sym _1 ∧ sym _1 > sym _2 Insertion Sort new input 
 a=[0, 1, 0] yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 8

Problem Solution Example Evaluation Related Summary Example Assessment of new input and extension of the trie. New most promising node gets selected. new input 
 a=[0, 1, 0] id=0 
 0 public void sort ( int [] a) { ROOT 1 int N = a.length; score=8.5 2 for ( int i = 1; i < N; i++) { 3 int j = i - 1; id=1 
 4 int x = a[i]; line=5 5 while ((j >= 0) && (a[j] > x)) { choice=0 
 6 a[j + 1] = a[j]; score=8.5 7 j--; 8 } id=2 id=3 9 a[j + 1] = x; line=5 line=5 10 } choice=0 
 choice=1 
 score=7.0 11 } score=10 Insertion Sort id=4 line=5 choice=0 
 score=10 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 9

Problem Solution Example Evaluation Related Summary Research Questions RQ1 : Since Badger combines fuzzing and symbolic execution, is it better than each part on their own in terms of: (a) Quality of worst-case, and (b) Speed? RQ2 : Is KelinciWCA better than Kelinci in terms of: (a) Quality of worst-case, and (b) Speed? RQ3 : Can Badger reveal worst-case vulnerabilities? yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 10

Problem Solution Example Evaluation Related Summary Experiments ID Subject 1 Insertion Sort each experiment for 5 2 Quicksort hours and 5 times 3a Regular Expression (fixed input) 3b Regular Expression (fixed regex) we report the average values 
 4 Hash Table (our full data set is available 5 Compression online) 6 Image Processor 7 Smart Contract yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 11

Problem Solution Example Evaluation Related Summary KelinciWCA 9305 Badger after 20min: 9305 after 2.85 hours Insertion Sort (N=64) 9850 19.35x 10000 9533 18.73x 7500 6701 costs (# jumps) 5000 3025 2500 Kelinci KelinciWCA SymExe Badger 0 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300 time (minutes) initial input score: 509 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 12

Problem Solution Example Evaluation Related Summary no significant di ff erence between Badger and KelinciWCA Quicksort (N=64) 3719 1.31x 3800 3683 1.30x 3161 2970 2850 costs (# jumps) 1900 950 Kelinci KelinciWCA SymExe Badger 0 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300 time (minutes) initial input score: 2829 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 13

Problem Solution Example Evaluation Related Summary Image Processor (2x2 JPEG) 400000 349,438 40.11x 300000 291,384 costs (# jumps) 193,730 22.24x 200000 188,719 100000 Kelinci KelinciWCA SymExe Badger 0 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 300 time (minutes) initial input score: 8712 yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 14

Problem Solution Example Evaluation Related Summary Existing Solutions • Fuzzing 
 e.g. SlowFuzz [Petsios2017] • Symbolic Execution 
 e.g. WISE , SPF-WCA [Burnim2009] [Luckow2017] • Fuzzing + Symbolic Execution 
 e.g. Driller [Stephens2016] yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 15

Problem Solution Example Evaluation Related Summary Badger : Complexity Analysis with Fuzzing and Symbolic Execution git clone https://github.com/isstac/badger.git yannic.noller@hu-berlin.de International Symposium on Software Testing and Analysis (ISSTA) 2018 ! 16