Automated Program Repair Opportunities, Challenges, Advances Chris - PowerPoint PPT Presentation

Automated Program Repair Opportunities, Challenges, Advances Chris Timperley 1

About me... Research Interests: Postdoc Carnegie Mellon University, USA ● Automated Program Repair w/ Claire Le Goues ● Fault Localisation ● GI for Robotics ● Automated Test Generation Recent Projects: MEng (2013), PhD (2017) ● BugZoo: reproducible studies of University of York, UK historical bugs w/ Susan Stepney ctimperley@cmu.edu ● Rooibos: language-independent, syntax-aware search and transformation. ● Houston : automated testing for robotics 2

Purpose of Talk: Challenge existing views, and identify opportunities. 3

4

2008 Genetic Programming to modify existing programs, rather than building them from scratch. Demonstrates concept of automated program repair. Evolutionary repair of faulty software. Andrea Arcuri. 2011. 5 Applied Soft Computing 11, 4 (June 2011), 3494-3514.

2009 GenProg demonstrates program repair on real-world C programs Automatically Finding Patches Using Genetic Programming. Westley Weimer, ThanVu Nguyen, Claire Le Goues, Stephanie Forrest. 6 Proceedings of International Conference on Software Engineering. ICSE ‘09. 2009.

2009– Program Analysis Fault Patch Localisation Generation BUGGY LOCALISED FIXED CODE CODE CODE 7

Where are the program repair bots? Or, why aren’t we all filthy rich yet? 8

ICSE SEIP ‘18 they exist! (note: we found out about them a few weeks ago) How to Design a Program Repair Bot? Insights from the Repairnator Project. Simon Urli, Zhongxing Yu, Lionel Seinturier, and Martin Monperrus. Proceedings of the International Conference on Software Engineering . ICSE ‘18. 9

How do we deploy? 10

What do developers need? Easy Integration Timeliness Bug Information Patches 11

The big assumption: The existence of tests requires test cases 12

Software engineering has changed since APR was introduced. 13

14

[ triggers CI build ] [ pushes results ] 15

requires test cases 16

17

What are the challenges? 18

Challenges Patch Quality Scalability Expressiveness 19

Challenges: Patch Quality BUGGY CORRECT PLAUSIBLE max = 0; max = a; max = 7; if(a > b) if(a > b) if(a > b) max = a; max = a; largest = a; if(b > a) if(b > a) if(b > a) max = b; max = b; largest = b; a b max a b max a b max 3 2 3 3 2 3 3 2 3 4 5 5 4 5 5 4 5 5 7 7 0 7 7 7 7 7 7 20

L A I S R E V O R T N O C Not all good patches are correct. 21

ISSTA ‘15 GPCE ‘06 “In many cases the Kali patch cleanly identifies the exact functionality and location that the developer patch “Bug reports also accompanied by modifies” [machine-generated] patches were three times as likely to be addressed as “The Kali and developer patches standard bug reports.” typically modify common functionality and variables.” 22

Correctness is a major challenge, but overfitted patches can still be useful. 23

Challenges Patch Quality Scalability Expressiveness 24

Challenges: Expressiveness vs. Scalability Allows the program to be changed in a greater number of ways, increasing the odds of finding a modification that produces a repair. ● larger corpus of fix ingredients ● wider set of program transformations ● granular modifications to the program Expressiveness 25

Challenges: Expressiveness vs. Scalability Time taken to discover a patch is a function of: ● patch size ● program size ● expressiveness ● ... Scalability 26

Scalability Expressiveness 27

How can we make APR both scalable and expressive? 28

Observation: APR inherited most of its technologies. 29

compiler optimisation test suite prioritisation abstract syntax trees concolic execution specification mining ... Program Program Analysis Analysis Fault Patch Patch Localisation Generation Generation BUGGY LOCALISED FIXED CODE CODE CODE genetic algorithms delta-debugging minimisation random search program synthesis metaprogramming mutation testing ... 30

APPEND 12 3 SWAP 5 6 DELETE 4 Patch Representation Abstract Syntax Tree 31

compiler optimisation test suite prioritisation abstract syntax trees concolic execution specification mining ... Program Program Analysis Analysis Fault Fault Patch Patch Localisation Localisation Generation Generation BUGGY LOCALISED FIXED spectrum-based fault localisation CODE CODE CODE genetic algorithms delta-debugging minimisation random search program synthesis metaprogramming mutation testing ... 32

Everyone is still using spectrum-based fault localisation! ● 2017: ssFix, Repairnator, NOPOL, ... ● 2016: History-Driven Program Repair, ... ● 2015: Angelix, SearchRepair, ... ● 2014: Astor, RSRepair, ... ● 2013: SemFix, ... ● 2011: AE, ... ● ... ● 2009: GenProg 33

Recap: Spectra-Based Fault Localisation int year, days; int year, days; int year, days; int year, days; year = 1980; year = 1980; year = 1980; year = 1980; days = atoi(argv[1]); days = atoi(argv[1]); days = atoi(argv[1]); days = atoi(argv[1]); while (days > 365) { while (days > 365) { while (days > 365) { while (days > 365) { if (isLeapYear(year)){ if (isLeapYear(year)){ if (isLeapYear(year)){ if (isLeapYear(year)){ if (days > 366) { if (days > 366) { if (days > 366) { if (days > 366) { days -= 366; days -= 366; days -= 366; days -= 366; year += 1; year += 1; year += 1; year += 1; } else { } } else { } } else { } } else { } } else { } else { } else { } else { days -= 365; days -= 365; days -= 365; days -= 365; year += 1; year += 1; year += 1; year += 1; } } } } } } } } return year; return year; return year; return year; Passing Test #1 Passing Test #2 Passing Test #3 Failing Test #1 34

Jaccard Ochiai AMPLE Tarantula GenProg 35 Shin Yoo. 2012. Evolving human competitive spectra-based fault localisation techniques. In Proceedings of the 4th international conference on Search Based Software Engineering (SSBSE'12), Gordon Fraser and Jerffeson Teixeira de Souza (Eds.). Springer-Verlag, Berlin, Heidelberg, 244-258.

Assumption: several failing test cases (6 is optimal). Reality: usually one failing test. Takes > 12 hours to run GenProg and SearchRepair. 36

Can we tailor fault localisation to CI-based program repair? 37

time 38

BugZoo 39