Improved Debugging Using Automatic Fault-localization Techniques - PDF document

Improved Debugging Using Automatic Fault-localization Techniques Mary Jean Harrold ADVANCE Professor of Computing Georgia Institute of Technology Supported by NSF, NASA Boeing Commercial Airplanes, Tata Consultancy Services MASPLAS 2007 My Research Group Postdoc • Carsten Görg Graduate students • Ahmed Ghiduk • Saswat Anand Summer 2006 • Term Apiwattanapong • George Baah • Jim Jones Aristotle Research Group • Irina Kishiniovski • Paul Li Research Scientists • Raul Santelices • Rashmitha Bupathi • Hina Shah • Pavan Kumar Chittimalli • Yanbing Yu MASPLAS 2007 1

High Cost of Software Failure Denver Airport Baggage System (1995): $280M Ariane 5 Explosion (1996): $7B Mars Rover (2004): Unknown cost MASPLAS 2007 High Cost of Software Failure Denver Airport Baggage System (1995): $280M Ariane 5 Explosion (1996): $7B Mars Rover (2004): Unknown cost Software bugs are costing the U.S. economy an estimated $59.5 billion each year. Improvements in testing and debugging could reduce this cost by about a third, or $22.5 billion. (from NIST Estimated Planning Report 02-3) MASPLAS 2007 2

Testing and Debugging Fault Localization Usage scenarios Nightly-build process • Run set tests (regression, breadth) each night • Report pass/fail status Test-driven development Use fault-localization to identify most likely • Create and run tests faulty parts of the (regression, breadth) after software changes • Report pass/fail status Regression testing • Run regression tests after changes • Report pass/fail status MASPLAS 2007 Research Projects Outline of Talk • Introduction • Fault localization—Tarantula • General technique • Visualization and demo • Test-case clustering to improve efficiency • Related work • Conclusion MASPLAS 2007 3

General Technique—Tarantula mid() { int x,y,z,m; 1:read(“Enter 3 integers:”,x,y,z); 2:m = z; 3:if (y<z) 4: if (x<y) 5: m = y; 6: else if (x<z) 7: m = y; 8:else 9: if (x>y) 10: m = y; 11: else if (x>z) 12: m = x; 13:print(“Middle number is:”, m); } MASPLAS 2007 General Technique—Tarantula Technique uses Dynamic information mid() { • statements executed int x,y,z,m; 1:read(“Enter 3 integers:”,x,y,z); • outcome (pass/fail) 2:m = z; Statistical analysis 3:if (y<z) • computes suspiciousness 4: if (x<y) 5: m = y; of each statement 6: else if (x<z) 7: m = y; 8:else Intuition: Statements primarily 9: if (x>y) executed by failed test cases 10: m = y; 11: else if (x>z) are more suspicious than 12: m = x; statements primarily executed 13:print(“Middle number is:”, m); by passed test cases } MASPLAS 2007 4

General Technique—Tarantula t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { int x,y,z,m; h h h h h h h h h h 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 2:m = z; h h h h h h h h h h 3:if (y<z) h h h h h h 4: if (x<y) h 5: m = y; h h h h h 6: else if (x<z) h h h h 7: m = y; h h h h 8:else h h h h 9: if (x>y) h h h 10: m = z; h 11: else if (x>z) 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); } P P P P P P F F F F Pass/fail Status MASPLAS 2007 General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { int x,y,z,m; h h h h h h h h h h 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 2:m = z; h h h h h h h h h h 3:if (y<z) h h h h h h 4: if (x<y) h 5: m = y; h h h h h 6: else if (x<z) h h h h 7: m = y; h h h h 8:else h h h h 9: if (x>y) h h h 10: m = z; h 11: else if (x>z) 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); } P P P P P P F F F F Pass/fail Status MASPLAS 2007 5

General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { int x,y,z,m; h h h h h h h h h h 1:read(“Enter 3 integers:”,x,y,z); 0.50 h h h h h h h h h h 0.50 2:m = z; h h h h h h h h h h 0.50 3:if (y<z) 4 h h h h h h 4: if (x<y) 0.43 4 h = = 0.00 5: m = y; ( 1 ) 0 . 50 susp h h 6 h 4 h h 6: else if (x<z) 0.50 + h h h h 7: m = y; 0.60 6 4 h h h h 0.60 8:else h h h h 0.60 9: if (x>y) h h h 0.75 10: m = z; h 11: else if (x>z) 0.00 12: m = x; 0.00 h h h h h h h h h h 0.50 13:print(“Middle number is:”, m); } P P P P P P F F F F Pass/fail Status MASPLAS 2007 General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { int x,y,z,m; h h h h h h h h h h 2 0.50 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 0.50 2:m = z; 4 = = ( 7 ) 0 . 60 h susp h h h h h h h h h 3:if (y<z) 0.50 2 2 + h h h h h h 0.43 4: if (x<y) 6 4 h 5: m = y; 0.00 h h h h h 0.50 6: else if (x<z) h h h h 0.60 7: m = y; h h h h 8:else 0.60 h h h h 0.60 9: if (x>y) h h h 10: m = z; 0.75 h 0.00 11: else if (x>z) 0.00 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); 0.50 } P P P P P P F F F F Pass/fail Status MASPLAS 2007 6

General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { rank int x,y,z,m; h h h h h h h h h h 1:read(“Enter 3 integers:”,x,y,z); 0.50 5 h h h h h h h h h h 0.50 5 2:m = z; h h h h h h h h h h 0.50 5 3:if (y<z) h h h h h h 4: if (x<y) 0.43 10 h 0.00 11 5: m = y; h h h h h 6: else if (x<z) 0.50 5 h h h h 2 7: m = y; 0.60 h h h h 0.60 2 8:else h h h h 0.60 2 9: if (x>y) h h h 0.75 1 10: m = z; h 11: else if (x>z) 0.00 11 11 12: m = x; 0.00 h h h h h h h h h h 0.50 5 13:print(“Middle number is:”, m); } P P P P P P F F F F Pass/fail Status MASPLAS 2007 General Technique—Tarautula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { rank int x,y,z,m; h h h h h h h h h h 0.50 5 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 0.50 5 2:m = z; h h h h h h h h h h 3:if (y<z) 0.50 5 h h h h h h 0.43 10 4: if (x<y) h 5: m = y; 0.00 11 h h h h h 0.50 5 6: else if (x<z) h h h h 0.60 2 7: m = y; h h h h 8:else 0.60 2 h h h h 0.60 2 9: if (x>y) h h h 10: m = z; //bug;correct m=y 0.75 1 h 0.00 11 11: else if (x>z) 0.00 11 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); 0.50 5 } P P P P P P F F F F Pass/fail Status MASPLAS 2007 7

General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { rank int x,y,z,m; h h h h h h h h h h 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 2:m = z; h h h h h h h h h h 3:if (y<z) h h h h h h 4: if (x<y) h 5: m = y; h h h h h 6: else if (x<z) h h h h 7: m = y; h h h h 8:else h h h h 9: if (x>y) h h h //fixed 10: m = y; h 11: else if (x>z) 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); } P P P P P P P F P F Pass/fail Status MASPLAS 2007 General Technique—Tarantula suspiciousness t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 3,2,1 3,3,5 1,2,3 3,2,2 5,5,5 1,1,4 5,3,4 2,1,3 5,4,2 5,2,6 mid() { rank int x,y,z,m; h h h h h h h h h h 0.50 4 1:read(“Enter 3 integers:”,x,y,z); h h h h h h h h h h 0.50 4 2:m = z; h h h h h h h h h h 3:if (y<z) 0.50 4 h h h h h h 0.67 3 4: if (x<y) h 5: m = y; 0.00 8 h h h h h 0.73 2 6: else if (x<z) h h h h //bug;correct:m=x; 0.80 1 7: m = y; h h h h 8:else 0.00 8 h h h h 0.00 8 9: if (x>y) h h h //fixed 10: m = y; 0.00 8 h 0.00 8 11: else if (x>z) 0.00 8 12: m = x; h h h h h h h h h h 13:print(“Middle number is:”, m); 0.50 4 } P P P P P P P F P F Pass/fail Status MASPLAS 2007 8