A Revisit of the Integration of Metamorphic Testing and Test Suite - PowerPoint PPT Presentation

A Revisit of the Integration of Metamorphic Testing and 
 Test Suite Based 
 Automated Program Repair Mingyue Jiang 1,2, Tsong Yueh Chen 2, 
 Fei-Ching Kuo 2, Zuohua Ding 1, 
 Eun-Hye Choi 3, Osamu Mizuno 4 1 Zhejiang Sci-Tech University, China 2 Swinburne University of Technology, Australia 3 National Institute of Advanced Industrial Science and Technology (AIST), Japan 4 Kyoto Institute of Technology, Kyoto, Japan 1

Automatic Program Repair (APR) APR Test Suite Repair Program Under Failing test cases passing 
 Repair (PUR) Passing test cases all test cases Test oracle is necessary. Ex. int Min(int x, int y){ int Min(int x, int y){ Inputs Expected Pass/ 
 if(x<=y-2) //fault if(x<=y) (x,y) Output Fail return x; return x; t 1 (8,1) 1 Pass else else t 2 (0,5) 0 Pass return y; return y; } } t 3 (1,2) 1 Fail 4

Metamorphic Testing (MT) To identify the correctness of test outputs, use metamorphic relations (MRs) instead of test oracle . Min MR Min(a,b) = Min(b,a) int Min(int x, int y){ if(x<=y-2) //fault Metamorphic test Conditions to be return x; group (MTG) satisfied else return y; mtg 1 ((8,1),(1,8)) Min(8,1) = Min(1,8) } mtg 2 ((0,5),(5,0)) Min(0,5) = Min(5,0) mtg 3 ((1,2),(2,1)) Min(1,2) = Min(2,1) 5

Metamorphic Testing (MT) To identify the correctness of test outputs, use metamorphic relations (MRs) instead of test oracle . Test oracle MR Ex. int Min(int x, int y) Min(a,b) = Min(b,a) Expected Pass/ 
 { Inputs Output Fail if(x<=y-2) 
 MTG MR is return x; (8,1) 1 Pass ((8,1),(1,8)) non-violated else (0,5) 0 Pass ((0,5),(5,0)) non-violated return y; } (1,2) 1 Fail ((1,2),(2,1)) violated Min(1,2)=2 
 Min(2,1)=1 6

APR + MT Use MR. APR-MT Metamorphic 
 Repair Program Under test groups satisfying MR 
 Repair (PUR) violating ones for all MTGs non-violating ones Use test oracle. APR Test Suite Repair Program Under Failing test cases passing 
 Repair (PUR) Passing test cases all test cases 7

APR + MT Use MR. APR-MT Metamorphic 
 Repair Program Under test groups satisfying MR 
 Repair (PUR) violating ones for all MTGs non-violating ones Ex. Min(a,b) = Min(b,a) int Min(int x, int y){ int Min(int x, int y){ if(x<=y-2) //fault if(x<=y) MTG MR is return x; return x; ((8,1),(1,8)) non-violated else else ((0,5),(5,0)) non-violated return y; return y; } } ((1,2),(2,1)) violated 8

APR + MT • We apply MT to a semantics based APR technique. • CETI-MT = CETI [Nguyen+17] + MT • MT is applied to a generate-and-validate APR technique [Jiang+17]. • GenProg-MT = GenProg [Forrest+09] + MT PUR variants Repair Test APR P 1 passing 
 suite all test cases ↓ P 2 PUR ↓ : satisfying 
 APR-MT MTGs P n all MTGs 9

CETI: Semantics based APR int Min(int x, int y, int uk_0){ if(x<=2) //fault 1. Fault localization*. return x; else return y; } *tarantula [Jones+05] . 10

CETI: Semantics based APR 1. Fault localization. int Min(int x, int y, int uk_0){ 2. Make a 
 if( x<=y-uk_0 ) //fault parameterized 
 return x; else statement with 
 return y; a suspicious statement 
 } using a repair template*. if(x<=y-2) *constant, operator, etc. 11

CETI: Semantics based APR int Min(int x, int y, int uk_0){ 1. Fault localization. if(x<=y-uk_0) //fault 2. Generate 
 return x; a parameterized 
 else statement. 
 return y; } 3. Generate 
 void main(){ a reachability 
 int uk_0; instance program. if( Min(8, 1, uk_0) == 1 && Min(0, 5, uk_0) == 0 && Min(1, 2, uk_0) == 1) { Precondition 
 Expected Location L; Input = passing all Output } (8,1) 1 test cases. } (0,5) 0 (1,2) 1 12

CETI: Semantics based APR int Min(int x, int y, int uk_0){ 1. Fault localization. if(x<=y-uk_0) //fault 2. Generate 
 return x; a parameterized 
 else statement. return y; 3. Generate 
 } a reachability 
 void main(){ instance program. 
 int uk_0; if( Min(8, 1, uk_0) == 1 && 4. Explore values that 
 Min(0, 5, uk_0) == 0 && make L reachable, 
 Min(1, 2, uk_0) == 1) { i.e. construct a repair. Location L; } } 13

CETI-MT int Min(int x, int y, int uk_0){ if(x<=y-uk_0) //fault Conditions to be return x; MTG satisfied else ((8,1),(1,8)) Min(8,1) = Min(1,8) return y; ((0,5),(5,0)) Min(0,5) = Min(5,0) } ((1,2),(2,1)) Min(1,2) = Min(2,1) int MR1Checker(int a, int b) { Precondition 
 if(a == b) return 1; else return 0; = satisfying MR } for all MTGs void main(){ int uk_0; /* Apply the MTG set. */ if ( MR1Checker(Min(8, 1, uk_0), Min(1, 8, uk_0)) == 1 && MR1Checker(Min(0, 5, uk_0), Min(5, 0, uk_0)) == 1 && 
 MR1Checker(Min(1, 2, uk_0), Min(2, 1, uk_0)) == 1 ) { Location L ; }} 14

CETI-MT CETI CETI-MT • A PUR • A PUR • A set of MTGs (violating & • A test suite (passing & Inputs non-violating test inputs failing test cases) for MRs) Repair Use information of the test Use information of the Process suite. MTGs. A repair passing all test A repair satisfying all MTGs Output cases or null. or null. 15

Experiments Using 6 IntroClass benchmark programs in C [Goues+15] . Totally, 1143 versions. 16

Experiments CETI-MT CETI MR1 MR2 black-box CE-BTS MCE-BTG1 MCE-BTG2 groupA test suite white-box CE-WTS MCE-WTG1 MCE-WTG2 groupB test suite We compare the success rate and the repair quality. 17

Result: Success Rate # of program versions successfully repaired / # of all program versions CETI-MT CETI-MT CETI CETI CE-BTS MCE-BTG1 MCE-BTG2 CE-WTS MCE-WTG1 MCE-WTG2 checksum 0.313 0.000 0.100 0.132 0.000 0.031 digits 0.122 0.457 0.133 0.217 0.268 0.278 grade 0.521 0.521 0.582 0.534 0.528 0.596 median 0.958 1.000 1.000 0.993 1.000 1.000 smallest 0.832 0.733 0.750 1.000 0.736 0.830 syllables 0.318 0.500 0.727 0.067 0.500 0.773 Total 0.636 0.657 0.659 0.626 0.628 0.636 → CETI-MT is comparable to CETI. 18

Result: Repair Quality • Mann-Whitney U Test [Wilcoxon1945] : to verify whether CETI and CETI-MT are di ff erent with p=0.05. • statistic [Arcuri+11, Vargha&Delaney2000] : ˆ A 12 to measure the probability that program repairs by CETI are of high quality than those by CETI-MT. • > 0.56: CETI is better. ˆ A 12 • < 0.44: CETI-MT is better. ˆ A 12 • Others: similar. 19

Result: Repair Quality • For 5 cases, CETI-MT is better. → CETI-MT is comparable to 
 • For 5 cases, both are similar. CETI. • For 2 cases, CETI is better. CE-BTS vs CE-BTS vs CE-WTS vs CE-WTS vs Eval. Eval. MCE-BTG1 MCE-BTG2 MCE-WTG1 MCE-WTG2 Data Data p < 0.05 p < 0.05 p=0.4345 p < 0.05 ˆ = 0.436 ˆ = 0.544 ˆ = 0.483 ˆ = 0.565 A 12 A 12 A 12 A 12 T w T b CETI-MT is better similar similar CETI is better p < 0.05 p < 0.05 p < 0.05 p=0.090 ˆ = 0.380 ˆ = 0.566 ˆ = 0.370 ˆ = 0.473 A 12 A 12 A 12 A 12 M 1 M 1 b w CETI-MT is better CETI-MT is better CETI is better similar p < 0.05 p < 0.05 p=0.648 p < 0.05 ˆ ˆ ˆ ˆ = 0.377 = 0.344 = 0.492 = 0.462 A 12 A 12 A 12 A 12 M 2 M 2 b w CETI-MT is better CETI-MT is better similar similar 20

Conclusion • We applied MT to a semantics based APR, CETI, and investigated its e ff ectiveness. • A test oracle is stronger than an MR. Nevertheless, CETI-MT shows comparable repair e ff ectiveness to CETI. • Two major factors a ff ecting CETI-MT: MRs and test cases. • Challenges: Automatic identification of MRs. Investigating a way of CETI-MT. 21