An Experiment in Estimating Reliability Growth Under Both Representative and Directed Testing An Experiment in Estimating Reliability Brian Mitchell and Steven J Zeil Growth Under Both Representative and Order Statistic Directed Testing Model Assumptions Testing as Biased Selection Order Statistics and Testing Brian Mitchell and Steven J Zeil Combining Representative and Directed Tests Old Dominion Univ. Scenario Experimental Design Results March 1998 Open Questions & Future Directions Proceedings of the ACM Sigsoft International Symposium on Software Testing and Analysis 1
An Experiment in Estimating Reliability Growth An Experiment in Estimating Reliability Growth Under Both Representative and Directed Testing Under Both Representative and Directed Testing Brian Mitchell and Steven J Zeil Order Statistic Model Order Statistic Assumptions Model Assumptions Testing as Biased Selection Testing as Biased Selection Order Statistics and Testing Order Statistics and Testing Combining Representative and Combining Representative and Directed Tests Directed Tests Scenario Scenario Experimental Design Results Experimental Design Open Questions & Results Future Directions Open Questions & Future Directions 2
Context: Fault-Based Reliability Modeling An Experiment in Estimating Reliability Growth Under Both Representative and Goals: Directed Testing ◮ Model Flexibility Brian Mitchell and Steven J Zeil ◮ reliability estimates using either representative or directed test data Order Statistic Model ◮ tolerance of “normal” variations in data Assumptions Testing as Biased ◮ Improved Data Collection Selection Order Statistics and Testing ◮ reduced noise Combining ◮ integrating multiple sources of information Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 3
Presentation Outline An Experiment in Estimating Reliability Growth Under Both Representative and 1. Overview of Order Statistic Model Directed Testing 2. Experiment: combining representative and directed tests Brian Mitchell and Steven J Zeil Order Statistic Model Assumptions Testing as Biased Selection Order Statistics and Testing Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 4
Order Statistic Model An Experiment in Estimating Reliability Growth Under Both Representative and Directed Testing Order Statistic Model Brian Mitchell and Steven J Zeil Assumptions Order Statistic Testing as Biased Selection Model Assumptions Order Statistics and Testing Testing as Biased Selection Order Statistics and Testing Combining Combining Representative and Directed Tests Representative and Directed Tests Scenario Scenario Experimental Design Experimental Design Results Results Open Questions & Future Directions Open Questions & Future Directions 5
Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Basic idea: Directed Testing Brian Mitchell and ◮ The faults present in a program have operational failure Steven J Zeil rates randomly selected from a distribution F . Order Statistic ◮ F depends upon Model ◮ program structure and semantics Assumptions Testing as Biased ◮ operational input distribution Selection Order Statistics and Testing ◮ Test methods tend to find the largest faults first. Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 6
Detailed assumptions An Experiment in Estimating Reliability Growth Under Both Representative and ◮ Operational program failure rate between repairs is Directed Testing constant. Brian Mitchell and Steven J Zeil ◮ Faults manifest independently. Order Statistic ◮ Detected faults are repaired perfectly. Model Assumptions ◮ The testing process is biased towards early detection of Testing as Biased Selection Order Statistics and faults with the largest failure rates. Testing Combining ◮ Faults in a program have failure rates φ with Representative and Directed Tests distribution F ( φ ). Scenario Experimental Design ◮ The program contains a finite number of faults. Results Open Questions & Future Directions 7
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ Operational program failure rate between repairs is Order Statistic constant. Model Assumptions ◮ Faults manifest independently. Testing as Biased Selection Order Statistics and ◮ Detected faults are repaired perfectly. Testing Combining ◮ The testing process is biased towards early detection of Representative and Directed Tests faults with the largest failure rates. Scenario Experimental Design ◮ Faults in a program have failure rates φ with Results distribution F ( φ ). Open Questions & Future Directions ◮ The program contains a finite number of faults. 8
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ Operational program failure rate between repairs is Order Statistic constant. Model Assumptions ◮ Operational program failure rate between repairs is Testing as Biased Selection Order Statistics and constant. Testing Combining ◮ Faults manifest independently. Representative and Directed Tests ◮ Faults manifest independently. Scenario Experimental Design Results Open Questions & Future Directions 9
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ Detected faults are repaired perfectly. Order Statistic Model ◮ Detected faults are repaired perfectly and Assumptions instantaneously. Testing as Biased Selection Order Statistics and Testing Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 10
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ The testing process is biased towards early detection of Order Statistic faults with the largest failure rates. Model Assumptions ◮ The test process finds the faults in decreasing order of Testing as Biased Selection Order Statistics and failure rate. Testing Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 11
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ Faults in a program have failure rates φ with Order Statistic distribution F ( φ ). Model Assumptions ◮ Faults f i in a program have failure rates φ i whose Testing as Biased Selection Order Statistics and expected value is a monotonic non-increasing function Testing g α,β ( i ) . Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 12
Comparing Assumptions An Experiment in Estimating Reliability Growth Under Both Representative and Our assumptions are actually less restrictive than most Directed Testing existing R GM s Brian Mitchell and Steven J Zeil ◮ The program contains a finite number of faults. Order Statistic Model ◮ The program contains a finite number of faults. , or Assumptions Testing as Biased ◮ The program contains an infinite number of faults. Selection Order Statistics and Testing Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 13
Testing as Biased Selection An Experiment in Estimating Reliability Growth Under Both Representative and ◮ Representative testing tends to find largest (failure rate) Directed Testing faults first Brian Mitchell and ◮ But is not, as usually assumed, guaranteed to do so Steven J Zeil ◮ Existing RGMs may be sensitive to these permutations Order Statistic Model ◮ Directed testing is often assumed to find faults in Assumptions arbitrary order Testing as Biased Selection Order Statistics and ◮ No evidence for this assumption Testing ◮ Counter-intuitive Combining Representative and Directed Tests Scenario Experimental Design Results Open Questions & Future Directions 14
Recommend
More recommend
