Shadow of a Doubt: Testing for Divergences Between Software Versions Hristina Palikareva Tomasz Kuchta Cristian Cadar ICSE’16, 20 th May 2016 This work is supported by EPSRC and Microsoft Research
Motivation § Software patches § Frequent, at the core of software evolution § New features, bug fixes, better performance, usability § Poorly tested in practice § May introduce bugs 2
Motivation: an example Old 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 3
Motivation: an example Old New 01 int gt_100(unsigned x) { 01 int gt_100(unsigned x) { 02 unsigned y = x; 02 unsigned y = x + 1 x + 1; 03 if (y > 100) 03 if (y > 100) 04 return 1; 04 return 1; 05 else 05 else 06 return 0; 06 return 0; 07 } 07 } § Test cases: x = 0, x = 100, x = 101 4
Motivation: an example Old New 01 int gt_100(unsigned x) { 01 int gt_100(unsigned x) { 02 unsigned y = x; 02 unsigned y = x + 1 x + 1; 03 if (y > 100) 03 if (y > 100) 04 return 1; 04 return 1; 05 else 05 else 06 return 0; 06 return 0; 07 } 07 } § Test cases: x = 0, x = 100, x = 101, 100% 100% code coverage 5
Motivation: an example Old New 01 int gt_100(unsigned x) { 01 int gt_100(unsigned x) { 02 unsigned y = x; 02 unsigned y = x + 1 x + 1; 03 if (y > 100) 03 if (y > 100) 04 return 1; 04 return 1; 05 else 05 else 06 return 0; 06 return 0; 07 } 07 } § Test cases: x = 0, x = 100, x = 101, 100% 100% code coverage § Only 50% 50% new behaviour coverage § Code coverage not sufficient! 6
Contributions § Shadow symbolic execution technique § Focuses on the new behaviours of the patch § Technique for unifying two versions of a program § Execute in a single symbolic execution instance § A patch testing approach § Shadow symbolic execution § Enhanced cross-version checks 7
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 8
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else 06 return 0; 07 } 9
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else 06 return 0; 07 } 10
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else 06 return 0; 07 } 11
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else 06 return 0; 07 } 12
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x + 1 > 100 06 return 0; 07 } x + 1 ≤ 100 13
Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1 x + 1; 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x + 1 > 100 06 return 0; 07 } x + 1 ≤ 100 e.g. x = 1000 e.g. x = 0 14
Shadow symbolic execution
Shadow symbolic execution § Old and new version in the same instance § The two versions are combined § Executed in lock-step fashion § The old version shadows the new one § Focus on the new behaviour § Versions take different sides of a branch 16
Shadow symbolic execution Old New 01 int gt_100(unsigned x) { 01 int gt_100(unsigned x) { 02 unsigned y = x; 02 unsigned y = x + 1 x + 1; 03 if (y > 100) 03 if (y > 100) 04 return 1; 04 return 1; 05 else 05 else 06 return 0; 06 return 0; 07 } 07 } 17
Shadow symbolic execution Combined 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1) change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 18
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1) change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 19
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1) change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 20
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1) change(x, x + 1); 03 if (y > 100) 04 return 1; 4-way fork 05 else 06 return 0; 07 } 21
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, change(x, x + 1 x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else 06 return 0; 07 } 22
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change( change(x, x + 1) , x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x ≤ 100 x > 100 x ≤ 100 x > 100 06 return 0; 07 } 23
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change( change(x, x + 1 x, x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x ≤ 100 x > 100 x ≤ 100 x > 100 06 return 0; new:else new:else new:then new:then old:else old:then old:else old:then 07 } 24
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change( change(x, x + 1 x, x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x ≤ 100 x > 100 x ≤ 100 x > 100 06 return 0; new:else new:else new:then new:then old:else old:then old:else old:then 07 } 25
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change( change(x, x + 1 x, x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x ≤ 100 x > 100 x ≤ 100 x > 100 06 return 0; new:else new:else new:then new:then old:else old:then old:else old:then 07 } 100 ✓ 26
Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change( change(x, x + 1 x, x + 1); 03 if (y > 100) x+1 ≤ 100 x+1 > 100 04 return 1; 05 else x ≤ 100 x > 100 x ≤ 100 x > 100 06 return 0; new:else new:else new:then new:then old:else old:then old:else old:then 07 } 100 ✓ Max Int 27
Shadow symbolic execution Divergence not always possible 01 int gt_100(unsigned x) { 02 unsigned y = x; y > 200 03 if (change(y > 100, y change(y > 100, y ≥ 100) 100)) 04 return 1; 05 else 06 return 0; y < 100 y < 100 y ≥ 100 y ≥ 100 y ≤ 100 y > 100 y ≤ 100 y > 100 07 } 28
Shadow symbolic execution § Advantages of shadow symbolic execution § Pruning execution paths – smaller search space § Space efficiency § Two versions combined into one § Expression sharing via shadow expressions § Does not execute unchanged path prefix twice 29
Patch annotations
Expressing patches § Annotations § change(old, new) macro § Currently manual, automation possible § A set of 15 rules § See project web-site for annotated patches http://srg.doc.ic.ac.uk/projects/shadow/ 31
Annotation rules § Modifying an rvalue expression Old New 01 if (argc – optind argc – optind < 1 < 1) 01 if (n_args n_args < 1 < 1) 02 { 02 { 03 error (...); 03 error (...); 04 usage (EXIT_FAILURE); 04 usage (EXIT_FAILURE); 05 } 05 } Combined 01 if (change(argc - optind, n_args) change(argc - optind, n_args) < 1 < 1) 02 { 03 error (...); 04 usage (EXIT_FAILURE); 05 } 32
Annotation rules § Adding an assignment Old New 01 byte_idx = 0; 01 byte_idx = 0; 02 print_delimiter = false; 02 print_delimiter = false; 03 03 current_rp = rp; current_rp = rp; Combined 01 byte_idx = 0; 02 print_delimiter = false; 03 current_rp = change(current_rp, rp) change(current_rp, rp); 33
Patch testing approach
Shadow approach overview Old New Test suite version version 35
Shadow approach overview Old New Test suite version version Unify Select Enhanced Shadow checks versions test cases 36
Shadow approach overview Old New Test suite version version Unify Select Enhanced Shadow checks versions test cases Regression Expected bugs divergences 37
Shadow approach overview Old New Test suite version version Unify Select Enhanced Shadow checks versions test cases Regression Expected bugs divergences 38
Shadow approach overview Unify Select Enhanced Shadow checks versions test cases § Combine old and new version § change() macro § Set of rules 39
Shadow approach overview Unify Select Enhanced Shadow checks versions test cases § Select test cases that touch the patch § Run test suite on the new version § Use coverage data § Cover at least one line of the patch 40
Recommend
More recommend
