Semi-valid Input Coverage for Fuzz Testjng Petar Tsankov , Mohammad - PowerPoint PPT Presentation

Semi-valid Input Coverage for Fuzz Testjng Petar Tsankov , Mohammad Torabi Dashtj, David Basin Instjtute of Informatjon Security ETH Zurich

Fuzz Testjng Testjng a PDF Viewer Pass / Fail PDF Viewer Valid inputs Test Oracle Are the PDF fjles displayed correctly? 2

Fuzz Testjng Fuzz-testjng a PDF viewer testjng Pass / Fail PDF Viewer Invalid inputs Test Oracle Are there any security faults? (e.g. memory errors) 3

Semi-valid Inputs PDF Viewer Open Inputs 4

Semi-valid Inputs PDF Viewer Open View Valid Inputs 5

Semi-valid Inputs PDF Viewer Open View Valid Inputs 6

Semi-valid Inputs PDF Viewer Open View Valid Block Inputs 7

Semi-valid Inputs PDF Viewer Open View Valid Block Inputs 8

Semi-valid Inputs PDF Viewer Open View Valid Block Inputs 9

Semi-valid Inputs PDF Viewer Entjrely-invalid Open View Semi-valid Valid Block Inputs ● Entjrely-invalid inputs get blocked. ● Semi-valid inputs are essentjal for fuzz testjng. 10

Coverage Criteria Measure Generate Test set Coverage Improve ● Low coverage hints at missing test cases. ● No existjng coverage metric tailored to fuzz testjng. - existjng metrics do not tell us how thoroughly we have tested with semi-valid inputs. 11

Coverage for Fuzz Testjng 12

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) 13

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Input Domain 14

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 Input Domain 15

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 C3 C2 Input Domain 16

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 Valid inputs C3 C2 Input Domain 17

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 Valid inputs C3 C2 Semi-valid input Input Domain 18

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 Valid inputs C3 C2 Semi-valid input Entjrely-invalid inputs Input Domain 19

Semi-valid Input Coverage ( SVCov ) ● Constraints defjne whether an input is valid or not. “The third byte is the XOR of the fjrst two bytes.” ( C1 ) Inputs that satjsfy C1 C1 Valid inputs C3 C2 Semi-valid input Entjrely-invalid inputs Input Domain SVCov = # covered semi-valid partjtjons # total semi-valid partjtjons 20

SVCov Propertjes C1 # covered semi-valid partjtjons SVCov = # total semi-valid partjtjons C2 C3 Independent to test generatjon method. Valid inputs do not contribute to SVCov. The usefulness of SVCov depends on the constraints. 100% SVCov does not guarantee that the tests reveal all faults. 21

Using SVCov C1 C2 C3 Fuzzing tool Test set SVCov 22

Using SVCov Problems with the fuzzing tool C1 C2 C3 Fuzzing tool Test set SVCov 23

Using SVCov Valid inputs Problems with the fuzzing tool C1 C2 C3 Fuzzing tool Test set SVCov 24

Using SVCov Missing valid inputs Valid inputs Problems with the fuzzing tool C1 C2 C3 Fuzzing tool Test set SVCov 25

Using SVCov Missing valid inputs Valid inputs Redundant constraints Problems with the fuzzing tool C1 C2 C3 Fuzzing tool Test set SVCov 26

Case Study 27

Case Study Research questjons: ● RQ1 : Feasibility Can we precisely defjne the semi-valid inputs of the SUT and effjciently measure SVCov? ● RQ2 : Relevance to coverage Does measuring SVCov provide meaningful informatjon on how to improve a test set's coverage? ● RQ3: Relevance to discovering faults Does increasing SVCov result in discovering additjonal faults? 28

Case Study: Artjfacts ● Test subject: OpenSwan - IKE implementatjon for Linux, 600K LOC. - Input specifjcatjon: RFC2407, RFC2408, RFC2409. ● Fuzzing tool: SecFuzz - Mutatjon-based fuzzer for security protocols. ● Test oracle: MemCheck - Detects memory errors. ● SVCov checker - Currently supports only IKE. 29

RQ1 : Feasibility ● We focused on “must (not) sentences” in the RFCs: “If a message contains a proposal payload, then the proposal’s next-payload fjeld must be set to 2 or 0.” ● The specifjcatjon of constraints for IKE is straightgorward: – Number of constraints: 217 . – Time to extract the constraints: 8 person hours . ● Negligible overhead for measuring SVCov: – Time to check all constraints for each test case: 41 ms . – Time to execute a test case: 1000 ms . 30

RQ2 : Relevance to Coverage SVCov (initjal) Violated SVCov 1 Missing valid inputs or fuzz-operators 0.8 Imprecise Coverage 0.6 fuzz-operators 0.4 0.2 0 0K 10K 20K 30K Number of test cases ● Many constraints are violated, but not uniquely. ● Some constraints are never violated. 31

RQ2 : Relevance to Coverage SVCov analysis ● Problems in the fuzzing tool - Imprecision in the “insert payload” fuzz operator. - Insert random numbers limited to [0, 100]. - ... ● Missing valid inputs - No valid inputs for IPv6 and ASN.1 X500 DN. ● Redundant constraints C1 C3 C2 32

RQ2 : Relevance to Coverage SVCov (afuer improvements) Violated SVCov 1 0.8 Coverage 0.6 0.4 0.2 0 0K 10K 20K 30K Number of test cases ● SVCov improved from 41% to 89%. ● All constraints are violated. ● 9% of the constraints are not uniquely violated. 33

RQ3 : Relevance to Discovering Faults ● A previously unknown security fault revealed afuer improving SVCov. SecFuzz OpenSwan Valid input Test case MemCheck Unallocated memory access ● The valid input was missing in the fjrst experiment. ● The test case belongs to a semi-valid partjtjon. 34

SVCov Contributjons C1 C2 C3 Easy-to-use coverage Independent of the for fuzz testjng fuzz-testjng technique Pinpoint subtle problems Promising initjal in fuzz testjng empirical results 35

Backup Slides 36

Redundant Constraints Input Domain C1 C3 C2 ● Constraint C1 is redundant. - removing C1 does not change the set of valid inputs. ● Constraint C1 cannot be uniquely violated. - Any input that violates C1 also violates C2. 37

Missing Valid Inputs Violated C guard Non-vacuously satjsfjed C target Vacuously satjsfjed ● To violate a constraint we need an input that satjsfjes the constraint non-vacuously. 38

Case Study: Setup Valid inputs Fuzzed inputs SUT OpenSwan OpenSwan (initjator) (responder) MemCheck SecFuzz ● We measure and report SVCov of the fuzzed inputs. ● Measure SVCov of the valid inputs to check for missing inputs. 39