Testing in Resource Constrained Execution Environments Gregory M. Kapfhammer Mary Lou Soffa Department of Computer Science Department of Computer Science Allegheny College University of Virginia Daniel Mossé Department of Computer Science University of Pittsburgh Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 1/19
Contributions Use of native code unloading during test suite execution in a resource constrained environment Identification of the testing techniques that yield the greatest reduction in execution time and native code size Characterization of how software applications and test suites restrict and/or support resource constrained testing Cost-benefit analysis for the use of sample-based and exhaustive profiles of program testing behavior Executes test suites faster when memory resources are limited! Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 2/19
Test Suite Execution with a JVM T Input Output P Byte Code Fast? Interpreter? Virtual Machine Adaptive? JIT? methodA testOne Heap Program Native Code Cache Stack During testing the JVM must manage limited resources Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 3/19
Resource Constrained Testing Memory Resident Native Code Bodies P Test Suite T Program ... ... m s m t T 1 T n Test Executor inv ct: 8 inv ct: 1200 inv ct: 2 inv ct: 1 exec time: 1% exec time: 15% exec time: 2% exec time: 1% ... size: 100 KB size: 64 KB TE u TE v All Tests size: 128 KB inv ct: 50 inv ct: 15 exec time: 22% exec time: 2% size: 75 KB size: 50 KB JIT compiler produces native code that exhausts limited memory resources Frequent invocation of GC increases testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 4/19
Test Suite Execution Strategies Omit tests? - Could reduce overall confi dence in the correctness of P Use non-constrained environment? - Defects related to P ’s interaction with environment might not be isolated Execute tests individually? - Might increase overall testing time and violate test order dependencies Unload with offline profile? - Not useful if P and T change frequently during regression testing Our Approach : Use online behavior profi les to guide the unloading of native code Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 5/19
Experiment Goals and Design Research Question: Can an adaptive code unloading JVM reduce time and space overheads associated with memory constrained testing? Experiment Metrics : percent reduction in time, T % R ( P, T ) and space, S % R ( P, T ) Jikes RVM 2.2.1, JUnit 3.8.1, GNU/Linux 2.4.18 Sample-based ( S ) and exhaustive ( X ) program profi les Timer ( TM ), garbage collection ( GC ), and code cache size ( CS ) triggers the unloading technique Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 6/19
Case Study Applications Name Min Size (MB) # Tests NCSS UniqueBoundedStack ( UBS ) 8 24 362 Library ( L ) 8 53 551 ShoppingCart ( SC ) 8 20 229 Stack ( S ) 8 58 624 JDepend ( JD ) 10 53 2124 IDTable ( ID ) 11 24 315 Empirically determined the MIN Jikes RVM heap size Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 7/19
Testing Time Overhead: Full RVM UBS L SC S JD I 5 4 3.67 Execution Time � sec � 3 2 1 0.688 0.52 0.48 0.47 0.468 UBS L SC S JD I Application, Full RVM When memory is not constrained, testing time is acceptable Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 8/19
Testing Time Overhead: Min RVM UBS L SC S JD I 25 21.108 20 Execution Time � sec � 15 10.644 10 5.196 4.276 5 3.686 3.376 UBS L SC S JD I Application, Min JVM Testing time increases signifi cantly when memory is Min Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 9/19
Summary of Reductions for Library T % S % R ( P, T ) R ( P, T ) Name S-GC 32.7 78.8 � X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 � 61.4 X-CS 33.4 59.8 Signifi cant reductions in time and space required for testing Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 10/19
Testing Time Overhead: Library S � GC X � GC S � CS X � CS S � TM X � TM 12 10 Execution Time � sec � 8 7.292 7.222 7.238 7.168 7.09 6.998 6 4 2 S � GC X � GC S � CS X � CS S � TM X � TM Library S vs. X : Similar reductions for all code unloading techniques Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 11/19
Testing Space Overhead: Library S � GC X � GC S � CS X � CS S � TM X � TM 80000 60000 Code Size � KB � 40000 30444.1 29212.1 28545.1 26463.7 20612.2 16052.7 20000 S � GC X � GC S � CS X � CS S � TM X � TM Library Code size reduction does not always yield best testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 12/19
Code Size Fluctuation: Library S � CS S � GC 40000 Code Size � KB � 30000 20000 10000 0 0 1 2 3 4 5 6 7 Time � sec � S-GC causes code size fluctuation that increases testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 13/19
Summary of Reductions for ID T % S % R ( P, T ) R ( P, T ) Name S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 � 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 � Unloading can decrease code size while still creating an overall increase in testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 14/19
Number of Code Unloads S � CS S � TM S � GC 2.0 UBS 8.0 UBS 11.4 2.0 L 9.0 L 11.0 1.4 Application SC 4.0 SC 4.0 2.0 S 6.0 S 12.0 14.0 JD 16.6 JD 16.0 4.0 ID 4.0 ID 2.0 2.5 5 7.5 10 12.5 15 17.5 20 # Code Unloading Times All techniques cause ID to perform few unloading sessions Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 15/19
Number of Unloaded Code Bodies S � CS S � TM S � GC 238.6 UBS 398.4 UBS 469.4 272.4 L 533.0 L 542.8 194.0 Application SC 303.4 SC 163.0 326.6 S 437.4 S 589.8 1953.8 JD 2228.2 JD 1970.2 664.4 ID 635.8 ID 164.0 500 1000 1500 2000 2500 # Unloaded Native Code Bodies ID ’s large working set forces unloading of many code bodies Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 16/19
Summary of Reductions T % S % R ( P, T ) R ( P, T ) Name S-GC 16.1 68.4 � X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 � 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 17/19
Conclusions and Future Work Dynamic compilation in JVMs can increase testing time if memory is constrained Adaptive unloading of native code often reduces memory overhead, avoids GC invocation, and reduces testing time Impact of unloading varies with respect to size of application’s working set and program testing behavior Regression test suite prioritization and reduction techniques that consider structural coverage and time and space overheads Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 18/19
Additional Resources Kapfhammer et al. Testing in Resource Constrained Execution Environments. In IEEE/ACM Automated Software Engineering . November 7 - 11, 2005. http : // cs . allegheny . edu / ˜ gkapfham / research / juggernaut / Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 19/19
Recommend
More recommend
