Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 - PowerPoint PPT Presentation

Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 baudry@kth.se

Test Your Tests • What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 2 Benoit Baudry, KTH, EclipseCon'17

Test Your Tests • What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 3 Benoit Baudry, KTH, EclipseCon'17

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } 4 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { @Test if (n == 0) return 1; factorialWith5Test() { long result = 1; long obs = fact(5); for (int i=2; i<=n; i++) assertTrue(5 < obs); result = result * i; } return result;} 5 Benoit Baudry, KTH, EclipseCon'17

Coverage long fact(int n) { @Test if (n == 0) return 1; factorialWith5Test() { long result = 1; long obs = fact(5); for (int i=2; i<=n; i++) assertTrue(5 < obs); result = result * i; } return result;} 6 Benoit Baudry, KTH, EclipseCon'17

Coverage long fact(int n) { @Test if (n == 0) return 1; factorialWith5Test() { long result = 1; long obs = fact(5); for (int i=2; i<=n; i++) assertTrue(5 < obs); result = result * i; } return result;} @Test factorialWith5Test() { assertEqual(1, fact(0));} 7 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { @Test if (n == 0) return 1; Are these test cases good at factorialWith5Test() { long result = 1; long obs = fact(5); detecting bugs? for (int i=2; i<=n; i++) assertTrue(5 < obs); result = result * i; } return result;} @Test factorialWith5Test() { assertEqual(1, fact(0));} 8 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { @Test if (n == 0) return 1; Are these test cases good at factorialWith5Test() { long result = 1; long obs = fact(5); detecting bugs? for (int i=2; i<=n; i++) assertTrue(5 < obs); result = result * i; } return result;} Let’s mutate our code to see. @Test factorialWith5Test() { assertEqual(1, fact(0));} 9 Benoit Baudry, KTH, EclipseCon'17

Mutation analysis • Tests are good if they can detect bugs • Principle: generate bugs and test the tests 10 Benoit Baudry, KTH, EclipseCon'17

Mutation analysis • Tests are good if they can detect bugs • Principle: generate bugs and test the tests • Mutation operators • Mutant = Program with one mutation 11 Benoit Baudry, KTH, EclipseCon'17

Mutation analysis Inputs : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M) 12 Benoit Baudry, KTH, EclipseCon'17

PIT • Open source • Actively developed and maintained • State of the art • For Java • Integrates with major build systems 13 Benoit Baudry, KTH, EclipseCon'17

PIT mutation operators • Conditions • Constants • Return • Delete method calls • Constructor call 14 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { if (n == 0) return 1; long result = 1; for (int i=2; i<=n; i++) result = result * i; return result;} 15 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { if (n == 0) return 1; n != 0 return 1+1 long result = 1; for (int i=2; i<=n; i++) < !(i<=n) -- result = result * i; result/i return result;} result+1 16 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { if (n == 0) return 1; long result = 1; for (int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < -- !(i<=n) result/i result+1 17 Benoit Baudry, KTH, EclipseCon'17

@Test long fact(int n) { factorialWith5Test() { if (n == 0) return 1; long obs = fact(5); long result = 1; assertTrue(5 < obs); } for (int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < -- !(i<=n) result/i result+1 18 Benoit Baudry, KTH, EclipseCon'17

long fact(int n) { if (n == 0) return 1; long result = 1; for (int i=2; i<=n; i++) @Test result = result * i; factorialWith5Test() { assertEqual(1, fact(0));} return result;} n != 0 return 1+1 < -- !(i<=n) result/i result+1 19 Benoit Baudry, KTH, EclipseCon'17

@Test long fact(int n) { factorialWith5Test() { if (n == 0) return 1; long obs = fact(5); long result = 1; assertTrue(5 < obs); } for (int i=2; i<=n; i++) @Test result = result * i; factorialWith5Test() { assertEqual(1, fact(0));} return result;} n != 0 return 1+1 < -- !(i<=n) result/i result+1 20 Benoit Baudry, KTH, EclipseCon'17

Mutation reveals bugs in the test suite @Test long fact(int n) { factorialWith5Test() { if (n == 0) return 1; long obs = fact(5); long result = 1; assertTrue(5 < obs); } for (int i=2; i<=n; i++) @Test result = result * i; factorialWith5Test() { assertEqual(1, fact(0));} return result;} Bugs in the test suite: n != 0 return 1+1 < -- !(i<=n) - Weak oracle result/i - Missing input result+1 21 Benoit Baudry, KTH, EclipseCon'17

@Test long fact(int n) { factorialWith5Test() { if (n == 0) return 1; long obs = fact(5); long result = 1; assertEqual(120, obs) ; } for (int i=2; i<=n; i++) @Test result = result * i; factorialWith5Test() { return result;} assertEqual(1, fact(0));} @Test n != 0 return 1+1 factorialWith1Test() { assertEqual(1, fact(1));} < -- !(i<=n) result/i result+1 22 Benoit Baudry, KTH, EclipseCon'17

PIT – reporting 23 Benoit Baudry, KTH, EclipseCon'17

PIT – reporting 24 Benoit Baudry, KTH, EclipseCon'17

Project #LOC #Mutants Time (h) Score (%) Amazon Web Services SDK 1703892 2141690 04:25:35 76.28 XWiki Rendering Engine 104727 112609 01:59:55 50.89 Apache Commons Math 180951 104786 03:22:18 83.81 JFreeChart 134353 89592 00:41:28 58.04 Apache PdfBox 137099 79763 06:20:25 58.89 Java Git 128874 78316 16:02:00 73.86 SCIFIO 55347 62768 03:12:11 45.92 Joda-Time 85911 31233 00:16:32 81.65 Apache Commons Lang 60733 30361 00:21:02 86.17 Apache Commons Collections 51632 20394 00:05:41 85.94 Urban Airship Client Library 40885 17345 00:11:31 82.26 SAT4J 26415 17067 11:11:04 68.58 ImageJ Common 23868 15592 00:29:09 54.77 jsoup 17839 14054 00:12:49 78.34 Jaxen XPath Engine 21204 12210 00:24:40 67.13 Apache Commons Codec 17267 9233 00:07:57 87.82 Apache Commons IO 24334 8809 00:12:48 84.73 Google Gson 20068 7353 00:05:34 81.76 AuthZForce PDP Core 16602 7296 01:23:50 88.18 Apache Commons CLI 7005 2560 00:01:26 88.71 25 Benoit Baudry, KTH, EclipseCon'17 JOpt Simple 9214 2271 00:01:36 93.52

Descartes • Mutation operators: extreme mutation • Active development • Open source • Compared to PIT • Less mutants • Different type of feedback • Same framework 26 Benoit Baudry, KTH, EclipseCon'17

Descartes - example long fact(int n) { if (n == 0) return 1; long result = 1; for (int i=2; i<=n; i++) result = result * i; return result;} 27 Benoit Baudry, KTH, EclipseCon'17

Descartes - example long fact(int n) { if (n == 0) return 1; long result = 1; for (int i=2; i<=n; i++) PIT : 7 mutants result = result * i; return result;} Descartes : 2 mutants long fact(int n){ long fact(int n){ return 1;} return 0;} 28 Benoit Baudry, KTH, EclipseCon'17

Scalability of extreme mutation public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants 29 Benoit Baudry, KTH, EclipseCon'17

Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45 Jaxen XPath Engine 12210 1252 00:24:40 00:01:34 Apache Commons Codec 9233 979 00:07:57 00:02:02 Apache Commons IO 8809 1164 00:12:48 00:02:18 Google Gson 7353 848 00:05:34 00:01:11 AuthZForce PDP Core 7296 626 01:23:50 00:08:45 Apache Commons CLI 2560 271 00:01:26 00:00:09 30 Benoit Baudry, KTH, EclipseCon'17 JOpt Simple 2271 412 00:01:36 00:00:25

Coarser grain than PIT @Test long fact(int n) { factorialWith5Test() { if (n == 0) return 1; long obs = fact(5); long result = 1; assertTrue(5 < obs); } for (int i=2; i<=n; i++) @Test result = result * i; factorialWith5Test() { assertEqual(1, fact(0));} return result;} long fact(int n){ long fact(int n){ return 1;} return 0;} 31 Benoit Baudry, KTH, EclipseCon'17

bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} 32 Benoit Baudry, KTH, EclipseCon'17