coverage driven test code generation
play

Coverage-Driven Test Code Generation for Concurrent Classes Valerio - PowerPoint PPT Presentation

Coverage-Driven Test Code Generation for Concurrent Classes Valerio Terragni Shing-Chi Cheung Department of Computer Science and Engineering The Hong Kong University of Science and Technology {vterragni, scc}@cse.ust.hk 20 May 2016 1


  1. ConSuite Context and synchronization insensitivity misses the [Steenbuck et al., generation of this failure inducing test code ICST 2013] public class CUT{ final CUT sout = new CUT(); int x= 0, z =0; T1 T2 sout .m1(0); sout .m2() z = 1 sout .m3(10); public void m3(int v1){ if(z ==0){ synchronized (this){ m4(v1); private void m4(int v1){ } $temp = x; } else x = $temp + v1; m4(v1); } } 9

  2. ConSuite Context and synchronization insensitivity misses the [Steenbuck et al., generation of this failure inducing test code ICST 2013] public class CUT{ final CUT sout = new CUT(); int x= 0, z =0; T1 T2 sout .m1(0); sout .m2() z = 1 sout .m3(10); public void m3(int v1){ if(z ==0){ synchronized (this){ m4(v1); private void m4(int v1){ } $temp = x; } else x = $temp + v1; m4(v1); } } 9

  3. ConSuite Context and synchronization insensitivity misses the [Steenbuck et al., generation of this failure inducing test code ICST 2013] public class CUT{ final CUT sout = new CUT(); int x= 0, z =0; T1 T2 sout .m1(0); sout .m2() z = 1 sout .m3(10); public void m3(int v1){ if(z ==0){ synchronized (this){ m4(v1); private void m4(int v1){ } $temp = x; } else x = $temp + v1; m4(v1); } } 9

  4. ConSuite Context and synchronization insensitivity misses the [Steenbuck et al., generation of this failure inducing test code ICST 2013] public class CUT{ final CUT sout = new CUT(); int x= 0, z =0; T1 T2 sout .m1(0); sout .m2() z = 1 sout .m3(10); FEASIBLE interleaving public void m3(int v1){ $temp = x; if(z ==0){ x= n synchronized (this){ m4(v1); x; = $temp + v1; private void m4(int v1){ } $temp = x; } else x = $temp + v1; m4(v1); } } 9

  5. ConSuite Context and synchronization insensitivity misses the [Steenbuck et al., generation of this failure inducing test code ICST 2013] public class CUT{ final CUT sout = new CUT(); int x= 0, z =0; T1 T2 sout .m1(0); sout .m2() z = 1 sout .m3(10); FEASIBLE interleaving public void m3(int v1){ $temp = x; if(z ==0){ x= n synchronized (this){ m4(v1); x; = $temp + v1; private void m4(int v1){ } $temp = x; } else x = $temp + v1; Without context and synchronization m4(v1); sensitivity this test is unlikely generated } } 9

  6. Our Intuition Step 1 Collect context-insensitive coverage requirements (statically) Step 2 Coverage-driven test code generation 10

  7. Our Intuition Step 1 Collect context-insensitive coverage requirements (statically) Step 2 Coverage-driven test code generation  Context-sensitive and synchronization-sensitive information can be collected precisely and efficiently during sequential test code generation 10

  8. Our Intuition Step 1 Collect context-insensitive coverage requirements (statically) Step 2 Coverage-driven test code generation  Context-sensitive and synchronization-sensitive information can be collected precisely and efficiently during sequential test code generation  Coverage metric (Sequential coverage)  Granularity at outer-most method call  Ordered sequence of object’s fields accesses  Context and synchronization sensitive  lock acquires and releases  calling context 10

  9. AutoConTest 11

  10. AutoConTest class Call Sequence CUT Generator increase sequential coverage 11

  11. AutoConTest class Call Sequence Concurrent Test final CUT sout = new CUT(); CUT Generator Assembler T1 T2 increase sequential coverage pairwise combinations 11

  12. AutoConTest class Call Sequence Concurrent Test final CUT sout = new CUT(); CUT Generator Assembler T1 T2 increase sequential coverage pairwise combinations Interleaving Explorer compute the interleaving based coverage requirements $temp = x; x= n x; = $temp + v1; 11

  13. Call Sequence Generator class Call Sequence CUT Generator 12

  14. Call Sequence Generator class Call Sequence CUT Generator CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  15. Call Sequence Generator public void m3(int v1){ class Call Sequence if(z ==0){ CUT synchronized (this){ Generator m4(v1); } } else m4(v1); } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  16. Call Sequence Generator public void m3(int v1){ class Call Sequence if(z ==0){ CUT synchronized (this){ Generator m4(v1); } } else m4(v1); } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  17. Call Sequence Generator public void m3(int v1){ class Call Sequence if(z ==0){ CUT synchronized (this){ Generator m4(v1); } } else m4(v1); } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  18. Call Sequence Generator public void m3(int v1){ class Call Sequence if(z ==0){ CUT synchronized (this){ Generator m4(v1); } } else m4(v1); } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  19. Call Sequence Generator private void m4(int v1){ class Call Sequence $temp = x; CUT x = $temp + 1; Generator } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  20. Call Sequence Generator private void m4(int v1){ class Call Sequence $temp = x; CUT x = $temp + 1; Generator } CUT sout = new CUT(); sout .m3(10); sout .m2(); 12

  21. Call Sequence Generator private void m4(int v1){ class Call Sequence $temp = x; CUT x = $temp + 1; Generator } CUT sout = new CUT(); new m3’s behaviour sout .m3(10); sout .m2(); 12

  22. Call Sequence Generator class Call Sequence CUT Generator public void m2(){ z = 1; } CUT sout = new CUT(); new m3’s behaviour sout .m3(10); sout .m2(); 12

  23. Call Sequence Generator class Call Sequence CUT Generator public void m2(){ z = 1; } CUT sout = new CUT(); new m3’s behaviour sout .m3(10); new m2’s behaviour sout .m2(); 12

  24. Call Sequence Generator class Call Sequence CUT Generator increase sequential public void m2(){ coverage z = 1; } CUT sout = new CUT(); new m3’s behaviour sout .m3(10); new m2’s behaviour sout .m2(); 12

  25. Call Sequence Generator class Call Sequence CUT Generator public void m2(){ z = 1; } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  26. Call Sequence Generator class Call Sequence CUT Generator public void m2(){ z = 1; } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  27. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  28. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  29. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  30. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  31. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  32. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator m4(v1); } } public void m2(){ } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); 12

  33. Call Sequence Generator public void m3(int v1){ private void m4(int v1){ class Call Sequence if(z ==0){ $temp = x; CUT synchronized (this){ x = $temp + 1; Generator increase m4(v1); } sequential } public void m2(){ coverage } else z = 1; m4(v1); } } CUT sout = new CUT(); sout .m2(); sout .m3(10); new m3’s behaviour 12

  34. Call Sequence Generator CUT sout = new CUT(); Search Space   CUT methods  fixed pool of parameters values (at each iteration) 13

  35. Call Sequence Generator CUT sout = new CUT(); Search Space   CUT methods sout .m1(1);  fixed pool of parameters values (at each iteration) 13

  36. Call Sequence Generator CUT sout = new CUT(); Search Space   CUT methods sout .m1(1); sout .m3(10);  fixed pool of parameters sout .m2(); sout .m1(0); values (at each iteration) ……. ……. 13

  37. Call Sequence Generator CUT sout = new CUT(); Search Space   CUT methods sout .m1(1); sout .m3(10);  fixed pool of parameters sout .m2(); sout .m1(0); values (at each iteration) ……. ……. sout .m1(1); sout .m3(10); sout .m1(0); sout .m2(); sout .m2(); ……. ……. 13

  38. Call Sequence Generator CUT sout = new CUT(); Search Space   CUT methods sout .m1(1); sout .m3(10);  fixed pool of parameters sout .m2(); sout .m1(0); values (at each iteration) ……. ……. sout .m1(1); sout .m3(10); sout .m1(0); sout .m2(); sout .m2(); ……. ……. CUT sout = new CUT(); sout .m3(10); sout .m2(); 13

  39. Call Sequence Generator CUT sout = new CUT();   sout .m3(10);  ……. ……. sout .m2(); ……. ……. CUT sout = new CUT(); sout .m3(10); sout .m2(); 13

  40. Call Sequence Generator Systematic exploration  CUT sout = new CUT();  DFS (search order) sout .m1(1); sout .m3(10); sout .m2(); sout .m1(0); ……. ……. sout .m1(1); sout .m3(10); sout .m1(0); sout .m2(); sout .m2(); ……. ……. Saturation-based stopping criterion  Stop extending a node if the latest K extensions have not increased the  sequential coverage 14

  41. Call Sequence Generator Systematic exploration  CUT sout = new CUT();  DFS (search order) sout .m1(1); sout .m3(10); sout .m2(); sout .m1(0); ……. ……. sout .m1(1); sout .m3(10); sout .m1(0); sout .m2(); sout .m2(); ……. ……. 15

  42. Call Sequence Generator Systematic exploration   DFS (search order) 15

  43. Call Sequence Generator   To minimize the number of tests and maximize the coverage  At each iteration identify an optimal sequence  with the highest coverage improvement # method calls that increase sequential coverage 15

  44. Call Sequence Generator   To minimize the number of tests and maximize the coverage  At each iteration identify an optimal sequence  with the highest coverage improvement # method calls that increase sequential coverage search space is too huge! 15

  45. Search Space Pruning Strategies Depth pruning 16

  46. Search Space Pruning Strategies Depth pruning 16

  47. Search Space Pruning Strategies Breadth pruning Depth pruning 16

  48. Search Space Pruning Strategies Breadth pruning Depth pruning 16

  49. Search Space Pruning Strategies Breadth pruning Depth pruning 16

  50. Search Space Pruning Strategies Assumption: The sequential execution of call sequences is deterministic Breadth pruning Depth pruning 16

  51. Search Space Pruning Strategies Assumption: The sequential execution of call sequences is deterministic Redundancy  Based on program states and sequential  coverage Breadth pruning Depth pruning 16

  52. Search Space Pruning Strategies Assumption: The sequential execution of call sequences is deterministic Redundancy  Based on program states and sequential  coverage Breadth pruning Depth pruning Theorem 1 The unexplored descendants of a node representing a redundant call sequence do not need to be explored in order to reach the optimal solution 16

  53. Search Space Pruning Strategies   Breadth pruning Depth pruning Theorem 1 The unexplored descendants of a node representing a redundant call sequence do not need to be explored in order to reach the optimal solution 16

  54. Concurrent Test Assembler class Call Sequence CUT Generator 17

  55. Concurrent Test Assembler class Call Sequence CUT Generator 17

  56. Concurrent Test Assembler class Call Sequence Concurrent Test CUT Generator Assembler 17

  57. Concurrent Test Assembler final CUT sout = new CUT(); class Call Sequence Concurrent Test T1 T2 CUT Generator Assembler final CUT sout = new CUT(); T1 T2 final CUT sout = new CUT(); The returned call sequences returned call  T1 T2 sequences are concurrently pair-wise tested Necessary condition for increasing  interleaving coverage (Theorem 2) 17

  58. Interleaving Explorer final CUT sout = new CUT(); class Call Sequence Concurrent Test T1 T2 CUT Generator Assembler sout .m1(0); sout .m2() sout .m3(10); Interleaving Explorer 18

  59. Interleaving Explorer final CUT sout = new CUT(); class Call Sequence Concurrent Test T1 T2 CUT Generator Assembler sout .m1(0); sout .m2() sout .m3(10); Interleaving Explorer Predictive Trace Analysis To compute the interleaving coverage (PTA) [Lai et al. ICSE 2016] requirements of the test code 18

  60. Interleaving Explorer final CUT sout = new CUT(); class Call Sequence Concurrent Test T1 T2 CUT Generator Assembler sout .m1(0); sout .m2() sout .m3(10); Interleaving Explorer To check if the requirements are Thread Scheduler feasible or infeasible 18

  61. Interleaving Explorer final CUT sout = new CUT(); class Call Sequence Concurrent Test T1 T2 CUT Generator Assembler sout .m1(0); sout .m2() sout .m3(10); Interleaving Explorer Challenge: non-deterministic interferences 18

  62. Subjects BUG Code Base Class Under Test CUT SLOC ID 1 Apache Commons 2.4 [..].lang.math.IntRange 278 2 Google Commons 1.0 [...]AbstractMultiMap$AsMap 1125 3 Java JDK 1.1.7 java.util.Vector 216 4 JFreeChart 0.9 [...]chart.axis.NumberAxis 1298 5 Java JDK 1.1.7 java.util.logging.Logger 992 6 Java JDK 1.4.2 java.util.Vector 326 6 real, known concurrency bugs atomic-set serializability violations 19

  63. RQ1:Cost-Effectiveness BUG ID Code Base CUT Time first SLOC fault 1 Apache Commons 2.4 278 22 sec 2 Google Commons 1.0 1125 29 sec 3 Java JDK 1.1.7 216 65 sec 4 JFreeChart 0.9 1298 35 sec 5 Java JDK 1.1.7 992 45 sec 6 Java JDK 1.4.2 326 30 sec 20

  64. RQ1:Cost-Effectiveness BUG ID Code Base CUT Time first SLOC fault 1 Apache Commons 2.4 278 22 sec 2 Google Commons 1.0 1125 29 sec 3 Java JDK 1.1.7 216 65 sec 4 JFreeChart 0.9 1298 35 sec 5 Java JDK 1.1.7 992 45 sec 6 Java JDK 1.4.2 326 30 sec 20

  65. RQ1:Cost-Effectiveness generate the first failing test code BUG ID Code Base CUT Time first and SLOC fault trigger the first faulty interleaving 1 Apache Commons 2.4 278 22 sec 2 Google Commons 1.0 1125 29 sec 3 Java JDK 1.1.7 216 65 sec 4 JFreeChart 0.9 1298 35 sec 5 Java JDK 1.1.7 992 45 sec 6 Java JDK 1.4.2 326 30 sec 20

  66. RQ1:Cost-Effectiveness generate the first failing test code BUG ID Code Base CUT Time first and SLOC fault trigger the first faulty interleaving 1 Apache Commons 2.4 278 22 sec 2 Google Commons 1.0 1125 29 sec 3 Java JDK 1.1.7 216 65 sec For all subjects the first generated 4 JFreeChart 0.9 1298 35 sec test manifested the bug 5 Java JDK 1.1.7 992 45 sec 6 Java JDK 1.4.2 326 30 sec 20

Recommend


More recommend