Lecture 20 Delta Debugging Regression Testing EE 382V Spring 2009 - PowerPoint PPT Presentation

Lecture 20 Delta Debugging Regression Testing EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Today’s Agenda • Regression Testing • Presentation by Divya (advocate) • Presentation by David (skeptic) • Delta Debugging: • Understanding its problem characterization one more time!!! • Quiz answers • Isolating Cause and Effect Chain EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Delta Debugging Problem (1) Circumstance δ All circumstances C = { δ 1 , δ 2 , . . . } Configuration c ⊆ C c = { δ 1 , δ 2 , . . . δ n } EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Delta Debugging Problem (2) Testing function test (c) ∈ { ✔ , ✘ , ? } Failure-inducing configuration test (c ✘ ) = ✘ Relevant configuration c � ✘ ⊆ c ✘ ∀ δ i ∈ c � c � � � ≠ ✘ ✘ · test ✘ \ { δ i } EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Mappings to DD Framework Testing Function test(c) esting Function test(c) esting Function test(c) Circumstances ( δ ) Circumstances ( δ ) Configuration (c) Configuration (c) ? ✔ ✖ Simplifying Inputs A subset of the A set of inputs Running a test function on the input c Running a test function on the input c Running a test function on the input c (Zeller, FSE 99) inputs Quiz Identifying Failure A subset of the Running a test function on a base Running a test function on a base Running a test function on a base Inducing Changes Changes changes program + changes (c) ogram + changes (c) ogram + changes (c) (Zeller, FSE 99) Invoking DDD with the name of a oking DDD with the name of a oking DDD with the name of a 344 textual deltas DDD 3.1.1 and non-existing file non-existing file non-existing file DDD 3.1.2 case between 3.1.1 and between 3.1.1 and deltas up to a deltas up to a study study 3.1.2 particular date no core can’t compile core dump DDD dump GDB 4.17 GDB 4.17 EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Testing Function test(c) esting Function test(c) esting Function test(c) Circumstances ( δ ) Circumstances ( δ ) Configuration (c) Configuration (c) ? ✔ ✖ Simplifying Inputs A subset of the A set of inputs Running the test code on the input c Running the test code on the input c Running the test code on the input c (Zeller, FSE 99) inputs A sequence of A subsequence of Quiz testSort that tak testSort that takes a sequence es a sequence values in an array values in an array Identifying Failure A subset of the Running the test code on a base Running the test code on a base Running the test code on a base Changes Inducing Changes changes program + changes (c) ogram + changes (c) ogram + changes (c) (Zeller, FSE 99) Invoking DDD with the name of a oking DDD with the name of a oking DDD with the name of a 344 textual deltas DDD 3.1.1 and non-existing file non-existing file non-existing file DDD 3.1.2 case between 3.1.1 and deltas up to a study study 3.1.2 particular date no core can’t compile core dump dump DDD Passing arguments in DDD fr Passing arguments in DDD fr Passing arguments in DDD front-end to GDB GDB 4.16 and a GDB 4.16 and a GDB 4.17 GDB 4.17 8721 textual deltas 8721 textual deltas subset of deltas Arguments Arguments Can’t compile GDB passed not passed EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

Testing Function test(c) esting Function test(c) esting Function test(c) Circumstances ( δ ) Circumstances ( δ ) Configuration (c) Configuration (c) ? ✔ ✖ Identifying Failure Inducing Thread A set of context A subset of the Run a program with the schedule Run a program with the schedule ogram with the schedule Schedule (ISSTA switch events events 2002) Identifying Cause A subset of A set of (variable, Resume the debugger with the Resume the debugger with the Resume the debugger with the Effect Chain (FSE (variable, value) value) pairs modified (variable modified (variable, value) pairs value) pairs 2002) pairs a set of (variable Running GCC on the fail.c as input Running GCC on the fail.c as input Running GCC on the fail.c as input a subset of and value) pairs at a GCC GCC (variable, value) (variable, value) particular debugger pairs no crash crash breakpoint A set of debug A subset of debug Locating Failure breakpoints that breakpoints that Resume the debugger with the Resume the debugger with the Resume the debugger with the Causes (ICSE include failure- include failure- modified (variable, value) pairs modified (variable value) pairs 2005) inducing program inducing program states states EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim

ddmin algorithm def ddmin(circumstances, n): while len(circumstances) >= 2: subsets = split(circumstances, n) some_complement_return_false = 0 for subset in subsets: complement = listminus(circumstances, subset) if testSort(complement) == False: circumstances = complement n = max(n - 1, 2) some_complement_return_false = 1 break if not some_complement_return_false: if n == len(circumstances): break n = min(n * 2, len(circumstances)) return circumstances

ddmin algorithm Input: [0, 1, 2, 3, 5, 4, 5, 6] Step n circumstances complement testSort (complement) 1 2 [0, 1, 2, 3, 5, 4, 5, 6] [5,4,5,6] false

ddmin algorithm Input: [0, 1, 2, 3, 5, 4, 5, 6] Step n circumstances complement testSort (complement) 1 2 [0, 1, 2, 3, 5, 4, 5, 6] [5,4,5,6] false 2 2 [5,4,5,6] [5,6] true

ddmin algorithm Input: [0, 1, 2, 3, 5, 4, 5, 6] Step n circumstances complement testSort (complement) 1 2 [0, 1, 2, 3, 5, 4, 5, 6] [5,4,5,6] false 2 2 [5,4,5,6] [5,6] true 3 2 [5,4,5,6] [5,4] false

ddmin algorithm Input: [0, 1, 2, 3, 5, 4, 5, 6] Step n circumstances complement testSort (complement) 1 2 [0, 1, 2, 3, 5, 4, 5, 6] [5,4,5,6] false 2 2 [5,4,5,6] [5,6] true 3 2 [5,4,5,6] [5,4] false 4 2 [5,4] [4] true

ddmin algorithm Input: [0, 1, 2, 3, 5, 4, 5, 6] Step n circumstances complement testSort (complement) 1 2 [0, 1, 2, 3, 5, 4, 5, 6] [5,4,5,6] false 2 2 [5,4,5,6] [5,6] true 3 2 [5,4,5,6] [5,4] false 4 2 [5,4] [4] true 5 2 [5,4] [5] true

ddmin algorithm Input: [3, 5, 7, 6, 8, 9, 13, 11] Step n circumstances complement testSort (complement) 1 2 [3,5,7,6,8,9,13,11] [8,9,13,11] false

ddmin algorithm Input: [3, 5, 7, 6, 8, 9, 13, 11] Step n circumstances complement testSort (complement) 1 2 [3,5,7,6,8,9,13,11] [8,9,13,11] false 2 2 [8,9,13,11] [13,11] false

ddmin algorithm Input: [3, 5, 7, 6, 8, 9, 13, 11] Step n circumstances complement testSort (complement) 1 2 [3,5,7,6,8,9,13,11] [8,9,13,11] false 2 2 [8,9,13,11] [13,11] false 3 2 [13,11] [11] true

ddmin algorithm Input: [3, 5, 7, 6, 8, 9, 13, 11] Step n circumstances complement testSort (complement) 1 2 [3,5,7,6,8,9,13,11] [8,9,13,11] false 2 2 [8,9,13,11] [13,11] false 3 2 [13,11] [11] true 4 2 [13,11] [13] true

Isolating Cause-E fg ect Chains Andreas Zeller

bug.c double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } 19

What is the cause of this failure? 20

From Defect to Failure 1. The programmer creates a defect – an error in the code. Variables ✘ ✘ 2. When executed, the defect creates an infection – an ✘ error in the state. 3. The infection propagates. ✘ ✘ 4. The infection causes a failure. This infection chain must be ✘ t traced back – and broken. 21

Tracing Infections • For every infection, we must find the earlier infection that causes it. • Program analysis tells us possible causes ✘ 22

Tracing Infections ✘ 23

Isolating Input Input Input Difference causes failure ✘ ✔ 24

Isolating States Variables Variables ✘ ✘ Difference causes failure ✘ t ✔ 25

Comparing States • What is a program state, anyway? • How can we compare states? • How can we narrow down differences? 26

A Sample Program $ sample 9 8 7 Output: 7 8 9 $ sample 11 14 Output: 0 11 Where is the defect which causes this failure? 27

int main(int argc, char *argv[]) { int *a; // Input array a = (int *)malloc((argc - 1) * sizeof(int)); for (int i = 0; i < argc - 1; i++) a[i] = atoi(argv[i + 1]); // Sort array shell_sort(a, argc); // Output array printf("Output: "); for (int i = 0; i < argc - 1; i++) printf("%d ", a[i]); printf("\n"); free(a); return 0; } 28

A sample state • We can access the entire state via the debugger: 1. List all base variables 2. Expand all references… 3. …until a fixpoint is found 29

Sample States Variable Value Variable Value in r � in r � in r � in r � argc i 4 5 3 2 argv [ 0 ] "./sample" "./sample" a[ 0 ] 9 11 argv [ 1 ] "9" "11" a[ 1 ] 8 14 argv [ 2 ] "8" "14" a[ 2 ] 7 0 argv [ 3 ] "7" 0x0 (NIL) a[ 3 ] 1961 1961 i ′ 1073834752 1073834752 a ′ [ 0 ] 9 11 a ′ [ 1 ] j 1074077312 1074077312 8 14 h 1961 1961 a ′ [ 2 ] 7 0 a ′ [ 3 ] 1961 1961 size 4 3 at shell_sort() 30