Evolving Fault Localisation Shin Yoo, University College London, UK - PowerPoint PPT Presentation

Evolving Fault Localisation Shin Yoo, University College London, UK Human Competitive Award, GECCO 2012, 09 July 2012

Spectra Based Fault Localisation Program

Spectra Based Fault Localisation Program Tests

Spectra Based Fault Localisation Program Spectrum Tests

Spectra Based Fault Localisation e p e f − e p + n p + 1 Program Spectrum Risk Evaluation Formula Tests

Spectra Based Fault Localisation e p e f − e p + n p + 1 Program Spectrum Risk Evaluation Formula Tests Ranking

Evolving SBFL e p e f − e p + n p + 1 Program Spectrum Risk Evaluation Formula Tests Ranking

Evolving SBFL Program Spectrum P S e p e f − Tests e p + n p + 1 T P S T Risk Evaluation Formula Ranking

Evolving SBFL Program Spectrum P S GP Tests T P S T Ranking

Evolving SBFL Program Spectrum P S GP Tests T P S T Fitness

Evolving SBFL Program Spectrum P S GP Tests T P S T Fitness

Evolving SBFL e 2 f (2 e p + 2 e f + 3 n p ) Program Spectrum P S GP Tests e 2 f ( e 2 f + √ n p ) T P S . . . T Fitness

State of the Art Over 30 formulæ in the literature, with various empirical studies with slightly different results

State of the Art e f 2 e f e f + n p + 2( e p + n f ) e f + n f + e p 2( e f + n p ) e f 2( e f + n p ) + e p + n f e f e f + n f + e p e f + 2( n f + e p ) e f Over 30 formulæ in the literature, with various e f + n p n f + e p empirical studies with slightly different results n f + e p e f 2 e f e f + n p 2 e f + n f + e p e f + n f + e p + n p e f + n f + e p + n p e f + n p − n f − e p 1 e f e f 2( + ) e f e f + n f + e p + n p e f + n f e f + e p e f + n f e p e f e p + n p + e f + n f

State of the Art Optimality Proof (Naish et al. 2011) ( − 1 if n f > 0 e p Op 2 = e f − Op 1 = e p + n p + 1 n p otherwise

State of the Art Optimality Proof (Naish et al. 2011) ( − 1 if n f > 0 e p Op 2 = e f − Op 1 = e p + n p + 1 n p otherwise But the proof is against a specific model · if (t1()) s1(); /* S1 */ else s2(); /* S2 */ if (t2()) x = True; /* S3 */ else x = t3(); /* S4 - BUG */

State of the Art Optimality Proof (Naish et al. 2011) ( − 1 if n f > 0 e p Op 2 = e f − Op 1 = e p + n p + 1 n p otherwise But the proof is against a specific model · if (t1()) s1(); /* S1 */ else s2(); /* S2 */ if (t2()) x = True; /* S3 */ else x = t3(); /* S4 - BUG */ ... not to mention hard.

Human Competitiveness How many of 9 Existing Techniques can 30 GP runs match and/or outperform? ✤ 6 runs outperform 8 existing 10% 20% techniques and match/outperform 13% one of the state of the art with proof (Op1 and Op2). 13% ✤ 16 runs outperform all 7 existing 23% techniques without proof. 20% < 5 5 6 7 8 9 Four Unix tools with 92 faults: 20 random faults for training, 72 for evaluation.

Human Competitiveness 100 100 ● ● 80 80 ● ● 61 61 10 10 ● ● 60 60 tarantula tarantula ● ● ● ● ✤ Per-fault view shows that ● ● ● ● evolved techniques can ● ● 40 40 ● ● ● ● ● ● ● ● outperform ones with 1 1 ● ● ● ● ● ● ● ● optimality proofs. ● ● ● ● ● ● 20 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 0 0 0 20 20 40 40 60 60 80 80 100 100 GP15 GP15

Human Competitiveness 100 100 80 80 3 3 59 59 60 60 ✤ Per-fault view shows that op2 op2 evolved techniques can 40 40 outperform ones with 10 10 ● ● ● ● ● ● optimality proofs. ● ● 20 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 0 0 0 20 20 40 40 60 60 80 80 100 100 GP08 GP08

Human Competitiveness 100 100 80 80 0 0 72 72 60 60 ✤ Per-fault view shows that op2 op2 evolved techniques can 40 40 outperform ones with 0 0 ● ● ● ● ● ● optimality proofs. 20 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 0 0 0 20 20 40 40 60 60 80 80 100 100 GP11 GP11

Future of Search-Based Software Engineering From Solutions to To Techniques and Strategies Generic Problems... for Your Problems.

The most effective way to do it, is to do it. ✤ GP provides a structured, automated way of doing iterative design. ✤ It can cope with a much diverse spectra and other meta-data. ✤ GP can evolve to suit your project .

The most effective way to do it, is to do it. ✤ GP provides a structured, automated way of doing iterative design. Think Hard ✤ It can cope with a much diverse Write Formula spectra and other meta-data. Experiment ✤ GP can evolve to suit your project . Human

The most effective way to do it, is to do it. ✤ GP provides a structured, automated way of doing iterative design. Think Hard Genetic Op. ✤ It can cope with a much diverse Write Formula Evaluate spectra and other meta-data. Experiment Select ✤ GP can evolve to suit your project . Human GP

The most effective way to do it, is to do it. ✤ GP provides a structured, Spectrum automated way of doing iterative design. Think Hard Genetic Op. ✤ It can cope with a much diverse Write Formula Evaluate spectra and other meta-data. Experiment Select ✤ GP can evolve to suit your project . Human GP

The most effective way to do it, is to do it. ✤ GP provides a structured, Spectrum Spectrum automated way of doing iterative design. Think Hard Genetic Op. ✤ It can cope with a much diverse Write Formula Evaluate spectra and other meta-data. Experiment Select ✤ GP can evolve to suit your project . Human GP

The most effective way to do it, is to do it. Dependency Change Hist. ✤ GP provides a structured, Spectrum Spectrum automated way of doing iterative design. Think Hard Genetic Op. ✤ It can cope with a much diverse Write Formula Evaluate spectra and other meta-data. Experiment Select ✤ GP can evolve to suit your project . Human GP

The most effective way to do it, is to do it. Dependency Dependency Change Hist. Change Hist. ✤ GP provides a structured, Spectrum Spectrum Spectrum automated way of doing iterative design. Think Hard Genetic Op. ✤ It can cope with a much diverse Write Formula Evaluate spectra and other meta-data. Experiment Select ✤ GP can evolve to suit your project . Human GP

Evolving SBFL e p e f − e p + n p + 1 Program Spectrum Risk Evaluation Formula Tests Ranking

Human Competitiveness Evolving SBFL How many of 9 Existing Techniques can 30 GP runs match and/or outperform? e p e f − e p + n p + 1 ✤ 6 runs outperform 8 existing 20% 10% techniques and match/outperform 13% Program Spectrum Risk Evaluation Formula one of the state of the art with proof (Op1 and Op2). 13% ✤ 16 runs outperform all 7 existing 23% techniques without proof. 20% < 5 5 6 7 Tests Ranking 8 9 Four Unix tools with 92 faults: 20 random faults for training, 72 for evaluation.