Leveraging Program Invariants to Promote Population Diversity in - PowerPoint PPT Presentation

Leveraging Program Invariants to Promote Population Diversity in Search-Based Automatic Program Repair Zhen Yu Ding † , Yiwei Lyu ‡ , Christopher S. Timperley ‡ , Claire Le Goues ‡ † University of Pittsburgh, ‡ Carnegie Mellon University 1

Bugs aren’t great... In 2017 ● 3.7 billion people affected ● Over $1.7 trillion of assets affected Reduces developer productivity ● Loss of time ● Frustration Z 2

Automatic Bug Repair Z 3

Automatic Bug Repair Z 4

Buggy program Pos. Tests Neg. Tests Z 5

Buggy Candidate program patches Pos. Tests Neg. Tests Z 6

Fitness function: weighted sum How many test cases does Buggy Passing Candidate program patches each candidate positive tests patch pass? Passing negative tests A patch passes all Pos. Tests test cases. Neg. Repair Tests found! Z 7

Fitness function: weighted sum How many test cases does Buggy Passing Candidate Selected program patches each candidate positive tests patches patch pass? Passing negative tests A patch passes all Pos. Tests test cases. Neg. Repair Tests found! Z 8

Y 9

Warning: this is not the normal GCD bug often seen in APR! Y 10

Y 11

Y 12

Y 13

Y 14

Works correctly when a != 0 Should return b when a = 0 This program returns 0 instead Y 15

Works correctly when a != 0 Should return b when a = 0 This program returns 0 instead Test Cases: a b Expected Actual Passed? result Result 5 7 1 0 2 2 12 16 4 3 0 3 0 10 10 ... ... ... Y 16

Works correctly when a != 0 Should return b when a = 0 This program returns 0 instead Test Cases: a b Expected Actual Passed? result Result 5 7 1 1 Yes 0 2 2 0 No 12 16 4 4 Yes 3 0 3 3 Yes 0 10 10 0 No ... ... ... ... ... Y 17

Problem: Should return b when a is 0 This program returns 0 instead Y 18

Problem: result=b; Should return b when a is 0 This program returns 0 instead Simplest fix is 2 steps: (1) Delete line 16 (2) Replace line 4 with line 8 Y 19

Simplest fix is 2 steps: (1) Delete line 16 (2) Replace line 4 with line 8 If we only perform step 1 (partial repair): Y 20

Simplest fix is 2 steps: (1) Delete line 16 (2) Replace line 4 with line 8 If we only perform step 1 (partial repair): ● Still fails when a=0 , passes otherwise ● Cannot be differentiated just from test results. Y 21

Patch indistinguishability Test cases often fail to distinguish between different candidate patches. Plateau-like fitness landscape. S. Forrest, W. Weimer, T. Nguyen, and C. Le Goues, “A genetic programming approach to automated software repair,” in Genetic and Evolutionary Computation Conference (GECCO), 2009, pp. 947–954. E. Fast, C. Le Goues, S. Forrest, and W. Weimer, “Designing better fitness functions for automated program repair,” in Genetic and Evolutionary Computation Conference, ser. GECCO ’10, 2010, pp. 965–972. E. F. de Souza, C. Le Goues, and C. G. Camilo-Junior, “A novel fitness function for automated program repair based on source code checkpoints,” in Z 22 Genetic and Evolutionary Computation Conference, ser. GECCO ’18, 2018.

Goal: distinguish patches better Z 23

Goal: distinguish patches better Infer invariants to semantically describe candidate patches. Find semantically unique/diverse candidate patches. Z 24

An intuition on why. Y 25

An intuition on why. Invariants when running positive tests (gcd(5,7), gcd(12,16), gcd(3,0), etc): ● a>=0 ● b>=0 ● result>=0 ● a%result==0 ● b%result==0 ● ... Y 26

An intuition on why. One simple fix: result=b; (1) Delete line 16 (2) Replace line 4 with line 8 Y 27

An intuition on why. One simple fix: (1) Delete line 16 (2) Replace line 4 with line 8 If we only perform step 1 (partial repair): ● Still fails when a=0 , passes otherwise ● Cannot be differentiated just from test results. Y 28

An intuition on why. Invariant a%result==0 : ● True when a != 0 ● False when a=0 (result is 0 ) Y 29

An intuition on why. Invariant a%result==0 : ● True when a != 0 ● False when a=0 (result is 0 ) ● True when a=0 in partial repair Y 30

An intuition on why. Invariant a%result==0 : ● True when a != 0 ● False when a=0 (result is 0 ) ● True when a=0 in partial repair Partial repair results in invariant behavior change! Y 31

Daikon – an invariant detection tool A mature dynamic invariant detection technique ● Runs the program and record traces of intermediate variable values ● Analyze the traces to learn invariants Y 32

Invariants when running positive tests (gcd(5,7), gcd(12,16), gcd(3,0), etc): ● a>=0 ● b>=0 ● result>=0 ● a%result==0 ● b%result==0 ● ... All were detected by Daikon Y 33

Fitness function: weighted sum How many test cases does Buggy Passing Candidate Selected program patches each candidate positive tests patches patch pass? Passing negative tests A patch passes all Pos. Tests test cases. Neg. Repair Tests found! Z 34

Fitness function: weighted sum How many test cases does Buggy Passing Candidate Selected program patches each candidate positive tests patches patch pass? Passing negative tests A patch passes all Pos. Tests test cases. Neg. Repair Tests found! Starting set of Daikon invariants. Z 35

Fitness function: weighted sum How many test cases does Buggy Passing Candidate Selected program patches each candidate positive tests patches patch pass? Passing negative tests A patch passes all Pos. Tests test cases. Neg. Repair Tests found! Do these Starting invariants still set of Daikon hold in invariants. candidate patches? Z 36

Starting set of invariants a%result==0 b%result==0 result>=0 Z 37

Starting set Candidate of invariants patch 0 a%result==0 b%result==0 result>=0 Z 38

Starting set Candidate Tested = Invariant never violated of invariants patch 0 against during program execution. a%result==0 Pos. tests b%result==0 result>=0 Z 39

Starting set Candidate Tested = Invariant never violated of invariants patch 0 against during program execution. a%result==0 Pos. tests ✘ = Invariant violated at least once. ✘ Neg. tests b%result==0 result>=0 Z 40

Starting set Candidate Tested = Invariant never violated of invariants patch 0 against during program execution. a%result==0 Pos. tests ✘ = Invariant violated at least once. ✘ Neg. tests b%result==0 Pos. tests result>=0 Z 41

Starting set Candidate Tested = Invariant never violated of invariants patch 0 against during program execution. a%result==0 Pos. tests ✘ = Invariant violated at least once. ✘ Neg. tests ? = Invariant not testable. b%result==0 Pos. tests Neg. tests ? result>=0 Z 42

Starting set Candidate Tested = Invariant never violated of invariants patch 0 against during program execution. a%result==0 Pos. tests ✘ = Invariant violated at least once. ✘ Neg. tests ? = Invariant not testable. b%result==0 Pos. tests Neg. tests ? result>=0 Pos. tests ✘ Neg. tests Z 43

Starting set Candidate Candidate = Invariant never violated of invariants patch 0 patch 1 during program execution. a%result==0 ✘ = Invariant violated at least once. ✘ ? = Invariant not testable. b%result==0 ✘ ? result>=0 ✘ ✘ Z 44

Starting set Candidate Candidate Invariant profile: of invariants patch 0 patch 1 Describes the semantics of a a%result==0 program based on a set of ✘ predicates. b%result==0 ✘ ? result>=0 ✘ ✘ Z 45

Starting set Candidate Candidate Invariant profile: of invariants patch 0 patch 1 Describes the semantics of a a%result==0 program based on a set of ✘ predicates. b%result==0 We use string comparisons ✘ ? to compare program semantics. result>=0 ● We use Hamming ✘ ✘ distances. Z 46

Starting set Candidate Candidate Invariant profile: of invariants patch 0 patch 1 Describes the semantics of a a%result==0 program based on a set of ✘ predicates. b%result==0 We use string comparisons ✘ ? to compare program semantics. result>=0 ● We use Hamming ✘ ✘ distances. Δ(p0, p1) = 2 Z 47

Starting set Candidate Candidate Candidate Invariant profile: of invariants patch 0 patch 1 patch 2 Describes the semantics of a ✘ a%result==0 program based on a set of ✘ ✘ predicates. b%result==0 We can use string ✘ ? ? comparisons to compare program semantics. result>=0 ● We use Hamming ✘ ✘ ✘ distances. We can calculate semantic Δ(p0, p1) = 2 Δ(p1, p2) = 3 diversity. ● Sum the Hamming Δ(p0, p2) = 1 distances. diversity (p0) = Δ(p0, p1) + Δ(p0, p2) = 2 + 1 = 3 diversity (p1) = Δ(p1, p0) + Δ(p1, p2) = 2 + 3 = 5 Z 48 diversity (p2) = Δ(p2, p0) + Δ(p2, p1) = 1 + 3 = 4