important contributions
play

Important Contributions JD 100 Program Reduction 80 Execution - PowerPoint PPT Presentation

Dec 07, 2022 •177 likes •1.2k views

Introduction Regression Testing Empirical Evaluation Conclusion Practical Suggestions for Improving and Empirically Studying Greedy Test Suite Reduction and Prioritization Methods Gregory M. Kapfhammer Department of Computer Science

  1. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of a Test Suite Program Calculating Coverage Test Suite Use instrumentation probes Adequacy Criterion to capture and analyze a Instrumented Program test suite’s coverage of the Instrumentation program state and structure Instrumented Test Suite Regression Testing Test Coverage Coverage Monitoring Results The coverage results and Test Adequacy adequacy measurements can Requirements Calculation support both test suite Adequacy reduction and prioritization Measurements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 6 / 30

  2. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of a Test Suite Program Calculating Coverage Test Suite Use instrumentation probes Adequacy Criterion to capture and analyze a Instrumented Program test suite’s coverage of the Instrumentation program state and structure Instrumented Test Suite Regression Testing Test Coverage Coverage Monitoring Results The coverage results and Test Adequacy adequacy measurements can Requirements Calculation support both test suite Adequacy reduction and prioritization Measurements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 6 / 30

  3. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  4. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  5. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 2 R 1 R 3 R 4 R 7 R 5 R 6 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  6. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  7. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  8. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  9. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  10. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 9 / 30

  11. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 9 / 30

  12. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Reduction Factor for Size (RFFS): How small is the reduced test suite? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  13. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Reduction Factor for Time (RFFT): How fast is the reduced test suite? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  14. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Common Rate (CR): How similar are differently reduced test suites? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  15. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  16. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  17. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  18. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  19. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  20. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  21. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Tree Models: Use recursive partitioning to create hierarchical view of data Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  22. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Explanatory Variable: Configuration of the testing methods (e.g., GCM) Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  23. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Response Variable: One of the evaluation metrics (e.g., CE or RFFT) Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  24. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  25. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  26. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  27. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  28. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Many Trees: Randomly construct a large collection of trees in order to avoid bias and identify the most important explanatory variables Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  29. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  30. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  31. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  32. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Do the greedy reducers and prioritizers efficiently identify test suites that improve effectiveness? What are the fundamental trade-offs ? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  33. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  34. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  35. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  36. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  37. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  38. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  39. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Objective: Simulate test suite execution on a centralized server Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  40. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Objective: Simulate test suite execution on a centralized server Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  41. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Construction: Combine all of the test suites and coverage reports Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  42. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  43. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  44. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  45. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  46. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Importance: Random prioritization serves as a valuable experimental control and often produces orderings better than the initial suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  47. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Strategy: Use the modern and efficient implementation of the Fisher-Yates shuffle to produce the reordered test suite T p = � t 4 , t 1 , t 3 , t 2 � Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  48. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends Reduction Factor for Time (RFFT) metric: cost | alg: 2OPT,GRD 0.4889 0.2101 0.4946 The myopic focus on cost leads to low RFFT values for 2OPT and GRD Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 19 / 30

  49. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends Reduction Factor for Time (RFFT) metric: cost | alg: 2OPT,GRD 0.4889 0.2101 0.4946 The myopic focus on cost leads to low RFFT values for 2OPT and GRD Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 19 / 30

  50. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends Reduction Factor for Size (RFFS) alg: 2OPT,GRD | metric: cost 0.6136 metric: coverage 0.1130 0.5967 0.4959 DGR and HGS are the best at creating test suites that improve RFFS Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 20 / 30

  51. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends Reduction Factor for Size (RFFS) alg: 2OPT,GRD | metric: cost 0.6136 metric: coverage 0.1130 0.5967 0.4959 DGR and HGS are the best at creating test suites that improve RFFS Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 20 / 30

  52. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends Coverage Effectiveness (CE) alg: HGS | metric: coverage 0.7520 alg: DGR 0.8231 0.8344 0.9388 Using ratio and cost improves the CE of the prioritized test suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 21 / 30

  53. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends Coverage Effectiveness (CE) alg: HGS | metric: coverage 0.7520 alg: DGR 0.8231 0.8344 0.9388 Using ratio and cost improves the CE of the prioritized test suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 21 / 30

  54. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time - SK cost coverage ratio SK 0.4 Reduction Factor for Time (RFFT) 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique For 2OPT and GRD, ratio and coverage create the best test suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 22 / 30

  55. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time - SK cost coverage ratio SK 0.4 Reduction Factor for Time (RFFT) 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique For 2OPT and GRD, ratio and coverage create the best test suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 22 / 30

  56. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size - SK cost coverage ratio SK 0.6 Reduction Factor for Size (RFFS) 0.5 0.4 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique It is often easy to construct test suites with high RFFS values Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 23 / 30

  57. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size - SK cost coverage ratio SK 0.6 Reduction Factor for Size (RFFS) 0.5 0.4 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique It is often easy to construct test suites with high RFFS values Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 23 / 30

  58. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  59. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique DGR and HGS exhibit lackluster performance when reordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  60. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Greedily reordered test suites are better than randomly prioritized ones Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  61. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Greedy fooling test suites cause GRD and DGR to make low CE suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  62. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique 2OPT uses lookahead and can construct high CE test prioritizations Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  63. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  64. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  65. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  66. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  67. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Value of the common rate is relatively stable across methods Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  68. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  69. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Algorithm choice is most important for improving the CE of ordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  70. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Greedy choice metric has the greatest impact on the test suite reducers Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  71. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements JD 100 80 Execution Time (ms) 60 40 20 0 2OPT DGR GRD HGS Prioritization Technique For the chosen case study applications, the techniques are fast Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 27 / 30

  72. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements JD 100 80 Execution Time (ms) 60 40 20 0 2OPT DGR GRD HGS Prioritization Technique For the chosen case study applications, the techniques are fast Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 27 / 30

  73. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  74. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  75. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  76. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  77. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - R educe A nd pr I ortize S uit E s http://raise.googlecode.com/ provides tools, data sets, and resources Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 29 / 30

  78. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - R educe A nd pr I ortize S uit E s Interactive visualization methods enable testers to find best ordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 29 / 30

  79. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  80. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  81. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  82. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

Recommend

Why This Matters Bayesian networks have been one of the most important contributions to the

Why This Matters Bayesian networks have been one of the most important contributions to the

Why This Matters Bayesian networks have been one of the most important contributions to the field of CS 331: Artificial Intelligence AI in the last 10-20 years Bayesian Networks Provide a way to represent knowledge in an uncertain

114 views • 7 slides
Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray

Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray

Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray Areas Surrounding FAS 116 and FAS 117 TUESDAY, MAY 13, 2014, 1:00-2:50 pm Eastern IMPORTANT INFORMATION This program is approved for 2 CPE credit

1.38k views • 88 slides
ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and

ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and

ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and Contributions Made Kimberly Griffith, CPA Audit Shareholder Danielle Miceli, CPA Audit Manager Objectives Distinguish between contributions

442 views • 40 slides
Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016

Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016

Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016 November 8, 2016 Payment Policy &amp; Financial Management Group, Division of Reinsurance Operations Training Series HTTPS://WWW.REGTAP.INFO/ Session

991 views • 64 slides
Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa

Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa

Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa Gymrek University of California San Diego COASP 2016 Outline Why is authorship order so important? The sandwich model of authorship It

325 views • 15 slides
Contributions to the impedance budget impedance of pumping holes and interconnects Bernard

Contributions to the impedance budget impedance of pumping holes and interconnects Bernard

Contributions to the impedance budget impedance of pumping holes and interconnects Bernard Riemann FCC-hh impedance and beamscreen workshop 2017-03-30 1 / 17 Contributions to impedance budget A list of (passive) contributions beam pipe

335 views • 17 slides
ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr.

ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr.

ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr. Ch.Srinivasa Rao Director ICAR-Central Research Institute for Dryland Agriculture Hyderabad Climate Change Impacts on Agriculture Sector Thus, currently,

231 views • 22 slides
CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions

CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions

CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions Isabel Braun Insttut for Partcle Physics, ETH Zrich CHIPP Plenary Meeting 2009, Appenberg Participating Institutes Participating Institutes

442 views • 16 slides
CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein

CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein

CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein Tuesday/Thursday 10:30 to 12:00 CSE 403 Today Contributions in HCI Project Discussions Abstract (from original version) Research in human-computer

707 views • 41 slides
Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy

Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy

Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy Model in China Beijing 15- 23.5.2107 Katja Granlund Planoras main sectors of activities Consulting Technical design Contributions of Planora to

452 views • 22 slides
DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and

DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and

DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and Zhe Zhang DIANA Contributions (Since July) Since the F2F meeting, DIANA contributions have focused on the following areas: Parallelism :

342 views • 16 slides
Why is this important? Why is this important? Anyone know what song this is? Yes thats right,

Why is this important? Why is this important? Anyone know what song this is? Yes thats right,

Why is this important? Why is this important? Anyone know what song this is? Yes thats right, Crowded House Weather With You UK release in 1992. I was 5 years old. Why is this important? The estimated average annual economic

332 views • 16 slides
Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In

Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In

Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In collaboration with Ayres Freitas, Hiren Patel, Michael J. Ramsey-Musolf LoopFest XVIII, August 13, 2019 FermiLab, IL Two-loop fermionic contributions to

414 views • 30 slides
o All functionalities appear at least once o 5 functionalities stick out with 4 mentions Mobility

o All functionalities appear at least once o 5 functionalities stick out with 4 mentions Mobility

Workshop Cross-border Cooperation for CCAM Testing o 8 contributions received, still limited data but some trends appear o (Understandably) not all contributions cover all aspects (i.e. functionalities and all enablers) o Focus on elements that

416 views • 5 slides
Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector

Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector

Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector 7-26-2018 1 Conditional Contributions Completed Independent review of the Amaila Falls Project with the assistance of the Kingdom of

781 views • 25 slides
Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions

Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions

Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions by: Giacinto Donvito, INFN Marcus Hardt, KIT Paul Millar, DESY Alvaro Garcia, CSIC Alvaro Garcia, CSIC With ki d With kind contributions by t

760 views • 38 slides
Adam Means treasurerCCCGNV@gmail Current contributions $464,615 Current expenses $480,737 Faith 2

Adam Means treasurerCCCGNV@gmail Current contributions $464,615 Current expenses $480,737 Faith 2

Adam Means treasurerCCCGNV@gmail Current contributions $464,615 Current expenses $480,737 Faith 2 Sight Contributions $60,737 Money spent from Savings $16,122 Year to Date Budget: $519,930 $39,193 under projected budget 300 Average attendance

539 views • 18 slides
Ultrastructural Contributions to Contributions to Ultrastructural Desensitization at Cerebellar

Ultrastructural Contributions to Contributions to Ultrastructural Desensitization at Cerebellar

Ultrastructural Contributions to Contributions to Ultrastructural Desensitization at Cerebellar Cerebellar Desensitization at Mossy Fiber to Granule Cell Mossy Fiber to Granule Cell Synapses Synapses Matthew A. Xu Xu- -Friedman and Wade

288 views • 28 slides
Contributions to Analysis and Functional Analysis in memoriam Pawe l Doma nski Dietmar

Contributions to Analysis and Functional Analysis in memoriam Pawe l Doma nski Dietmar

Contributions to Analysis and Functional Analysis in memoriam Pawe l Doma nski Dietmar Vogt Bergische Universit at Wuppertal Pawe l Doma nski Memorial Conference Bedlewo 2018 Contributions to: Real analytic functions

758 views • 72 slides
Contributions for 5G Development at Brazil May 22 nd 2018 Dr. Henry Douglas Rodrigues Agenda

Contributions for 5G Development at Brazil May 22 nd 2018 Dr. Henry Douglas Rodrigues Agenda

Contributions for 5G Development at Brazil May 22 nd 2018 Dr. Henry Douglas Rodrigues Agenda Motivations for 5G Inatel Contributions for 5G Demos and Performance Future Work Conclusions Motivations Motivations 5G

380 views • 26 slides
Outline ITAINNOVA at a glance: 01. OpenFOAM group Contributions to Industrial Manufacturing:

Outline ITAINNOVA at a glance: 01. OpenFOAM group Contributions to Industrial Manufacturing:

Outline OPENFOAM CONTRIBUTIONS TO INDUSTRIAL MANUFACTURING Outline ITAINNOVA at a glance: 01. OpenFOAM group Contributions to Industrial Manufacturing: 02. Rubber Injection Molding 03. Microfluidic Chip Control 04. Ongoing/Future projects

1.01k views • 24 slides
CSIRO Decadal Climate Forecasting Project Richard Matear, Project Leader with important

CSIRO Decadal Climate Forecasting Project Richard Matear, Project Leader with important

CSIRO Decadal Climate Forecasting Project Richard Matear, Project Leader with important contributions from the Decadal Climate Forecasting Team Climate Science Centre O&amp;A Business Unit OCEANS AND ATMOSPHERE https://research.csiro.au/dfp/

670 views • 26 slides
innovation Zaragoza, 25 th of October 2013 ECITL Parallel B (Friday 12:30): ICT contributions and

innovation Zaragoza, 25 th of October 2013 ECITL Parallel B (Friday 12:30): ICT contributions and

ICT contributions and challenges for logistics innovation Zaragoza, 25 th of October 2013 ECITL Parallel B (Friday 12:30): ICT contributions and challenges for logistics innovation Moderators: Georgia Aifadopoulou, Fernando Liesa Participants:

506 views • 14 slides
Contributions and Missed Opportunities S T . C O L U M B A S ( O A K L A N D , C A ) F R .

Contributions and Missed Opportunities S T . C O L U M B A S ( O A K L A N D , C A ) F R .

The 2018 Racism Pastoral Letter: Contributions and Missed Opportunities S T . C O L U M B A S ( O A K L A N D , C A ) F R . B R Y A N M A S S I N G A L E F O R D H A M U N I V E R S I T Y November 2018: The first since 1979 Overview

750 views • 53 slides

More recommend