Combinatorial Interaction Testing Justyna Petke C entre for R - PowerPoint PPT Presentation

Combinatorial Interaction Testing Problem: Testing all combinations is too expensive. Solution: Test all interactions between any set of t parameters. Combinatorial Interaction Testing Justyna Petke

Combinatorial Interaction Testing Problem: Testing all combinations is too expensive. Solution: t -way testing Combinatorial Interaction Testing Justyna Petke

Combinatorial Interaction Testing A CIT test suite covers all t -way interactions between any t parameters. Such a test suite is also known as a covering array. Pairwise (2-way) testing is the most widely studied CIT technique. Combinatorial Interaction Testing Justyna Petke

Covering Array Definition : A covering array CA ( t, k, v ) of size N is a table with N rows and k columns. Each field of CA contains a value in the range 0 , · · · , v − 1 . CA has the following property: every combination of t values between any t parameters occurs in at least one row. t is called the strength of a covering array. Combinatorial Interaction Testing Justyna Petke

Combinatorial Interaction Testing CIT problem: Find a minimal test suite that covers all t -way interactions. CAN ( t, k, v ) denotes the size of the smallest covering array. Combinatorial Interaction Testing Justyna Petke

Covering Array Tables �� ∼ �� Combinatorial Interaction Testing Justyna Petke

Pairwise testing example Pairwise (2-way) interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Spain Barcelona 21 Aug 26 Aug Spain Barcelona 20 Aug 26 Aug Spain Barcelona 21 Aug 27 Aug Spain Madrid 21 Aug 26 Aug Spain Madrid 20 Aug 27 Aug Are all 3-way interactions covered? Combinatorial Interaction Testing Justyna Petke

3-way testing example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Spain Barcelona 21 Aug 26 Aug Spain Barcelona 20 Aug 26 Aug Spain Barcelona 21 Aug 27 Aug Spain Madrid 21 Aug 26 Aug Spain Madrid 20 Aug 27 Aug Spain Madrid 20 Aug 26 Aug Spain Madrid 21 Aug 27 Aug Combinatorial Interaction Testing Justyna Petke

Applications Combinatorial Interaction Testing Justyna Petke

Web Form Example Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Madrid 21 Aug 26 Aug Combinatorial Interaction Testing Justyna Petke

Web Browser Example Load content Notify pop-up Cookies Warn before Remember blocked add-ons install downloads Allow Yes Allow Yes Yes Restrict No Restrict No No Block Block Combinatorial Interaction Testing Justyna Petke

Car Model Example Automated Collision Parallel Lateral Forward Driving Avoidance Parking Range Range Controller Braking Finder Finder Included StandardAvoidance Included Included Included None EnhancedAvoidance None None None None Combinatorial Interaction Testing Justyna Petke

Cellphone Example MMS WLAN Bluetooth MP3 Camera Included Included Included Included Black&White None None None None Colour None Combinatorial Interaction Testing Justyna Petke

Applications Web Forms Web Browsers Automotive Industry Cellphone Industry .. and many other configurable systems Combinatorial Interaction Testing Justyna Petke

Fault detection Will a pairwise test suite discover a fault triggered by interactions of 3 or more parameters? Combinatorial Interaction Testing Justyna Petke

Fault detection Pairwise testing discovers at least 53% of the known faults. 6-way testing discovers 100% of the known faults. Combinatorial Interaction Testing Justyna Petke

Fault detection * results avilable at �� Combinatorial Interaction Testing Justyna Petke

Example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Spain Barcelona 21 Aug 26 Aug Spain Barcelona 20 Aug 26 Aug Spain Barcelona 21 Aug 27 Aug Spain Madrid 21 Aug 26 Aug Spain Madrid 20 Aug 27 Aug Spain Madrid 20 Aug 26 Aug Spain Madrid 21 Aug 27 Aug What can we do to discover a pairwise interaction fault more quickly? Combinatorial Interaction Testing Justyna Petke

Example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Spain Barcelona 21 Aug 26 Aug Spain Barcelona 20 Aug 26 Aug Spain Barcelona 21 Aug 27 Aug Spain Madrid 21 Aug 26 Aug Spain Madrid 20 Aug 27 Aug Spain Madrid 20 Aug 26 Aug Spain Madrid 21 Aug 27 Aug Test case 5 is the first to discover the fault.how quickly will test case discover fault Combinatorial Interaction Testing Justyna Petke

Example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug 6 new pairs Spain Barcelona 21 Aug 26 Aug 5 new pairs Spain Barcelona 20 Aug 26 Aug 1 new pair Spain Barcelona 21 Aug 27 Aug 1 new pair Spain Madrid 21 Aug 26 Aug 3 new pairs Spain Madrid 20 Aug 27 Aug 2 new pairs Spain Madrid 20 Aug 26 Aug 0 new pairs Spain Madrid 21 Aug 27 Aug 0 new pairs How many new pairs does each test case cover? Combinatorial Interaction Testing Justyna Petke

Example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug 1 new pair Spain Barcelona 21 Aug 26 Aug 5 new pairs Spain Barcelona 20 Aug 26 Aug 1 new pair Spain Barcelona 21 Aug 27 Aug 1 new pair Spain Madrid 21 Aug 26 Aug 3 new pairs Spain Madrid 20 Aug 27 Aug 2 new pairs Spain Madrid 20 Aug 26 Aug 0 new pairs Spain Madrid 21 Aug 27 Aug 0 new pairs How many new pairs does each test case cover? Combinatorial Interaction Testing Justyna Petke

Example 3-way interaction test suite: Country City Date Return Date Spain Barcelona 20 Aug 27 Aug 6 new pairs Spain Barcelona 21 Aug 26 Aug 1 new pair Spain Barcelona 20 Aug 26 Aug 1 new pair Spain Barcelona 21 Aug 27 Aug 1 new pair Spain Madrid 21 Aug 26 Aug 3 new pairs Spain Madrid 20 Aug 27 Aug 2 new pairs Spain Madrid 20 Aug 26 Aug 0 new pairs Spain Madrid 21 Aug 27 Aug 0 new pairs How many new pairs does each test case cover? Combinatorial Interaction Testing Justyna Petke

Prioritisation 2-way prioritised 3-way interaction test suite Country City Date Return Date Spain Barcelona 20 Aug 27 Aug Spain Madrid 21 Aug 26 Aug Spain Barcelona 21 Aug 26 Aug Spain Madrid 20 Aug 27 Aug Spain Barcelona 20 Aug 26 Aug Spain Barcelona 21 Aug 27 Aug Spain Madrid 20 Aug 26 Aug Spain Madrid 21 Aug 27 Aug How quickly will a fault caused by Spain-Madrid interaction be discovered? Combinatorial Interaction Testing Justyna Petke

Prioritisation 2-way prioritised 3-way interaction test suite Country City Date Return Date Spain Barcelona 20 Aug 27 Aug 6 new pairs Spain Madrid 21 Aug 26 Aug 6 new pairs Spain Barcelona 21 Aug 26 Aug 2 new pairs Spain Madrid 20 Aug 27 Aug 2 new pairs Spain Barcelona 20 Aug 26 Aug 1 new pair Spain Barcelona 21 Aug 27 Aug 1 new pair Spain Madrid 20 Aug 26 Aug 0 new pairs Spain Madrid 21 Aug 27 Aug 0 new pairs How quickly will a fault caused by Spain-Madrid interaction be discovered? Combinatorial Interaction Testing Justyna Petke

Prioritisation 2-way prioritised 3-way interaction test suite Country City Date Return Date Spain Barcelona 20 Aug 27 Aug 6 new pairs Spain Madrid 21 Aug 26 Aug 6 new pairs Spain Barcelona 21 Aug 26 Aug 2 new pairs Spain Madrid 20 Aug 27 Aug 2 new pairs Spain Barcelona 20 Aug 26 Aug 1 new pair Spain Barcelona 21 Aug 27 Aug 1 new pair Spain Madrid 20 Aug 26 Aug 0 new pairs Spain Madrid 21 Aug 27 Aug 0 new pairs Test case 2 is the first to discover the fault.how quickly will test case discover fault Combinatorial Interaction Testing Justyna Petke

Higher-strength prioritisation Justyna Petke, Myra B. Cohen, Mark Harman, and Shin Yoo. Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection . IEEE Transactions on Software Engineering (TSE) volume 99: 1-26 (2015) Justyna Petke, Shin Yoo, Myra B. Cohen and Mark Harman. Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing. The 9th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2013) Combinatorial Interaction Testing Justyna Petke

sed 5-way Example sed generated:5 (unordered) 100 80 Interaction Coverage 60 40 20 cov. for str. 2 (APCC=84.93) cov. for str. 3 (APCC=80.27) cov. for str. 4 (APCC=76.26) cov. for str. 5 (APCC=72.69) 0 0 20 40 60 80 100 Percentage of Test Suite (324 Test Cases) * Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing Combinatorial Interaction Testing Justyna Petke

sed 5-way Example sed generated:5 ordered:2 100 80 Interaction Coverage 60 40 20 cov. for str. 2 (APCC=97.48) cov. for str. 3 (APCC=93.28) cov. for str. 4 (APCC=88.36) ● cov. for str. 5 (APCC=83.44) ● 0 0 20 40 60 80 100 Percentage of Test Suite (324 Test Cases) * Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing Combinatorial Interaction Testing Justyna Petke

sed 5-way Example sed generated:5 ordered:5 100 80 Interaction Coverage 60 40 20 cov. for str. 2 (APCC=97.40) cov. for str. 3 (APCC=94.52) cov. for str. 4 (APCC=90.95) ● cov. for str. 5 (APCC=87.02) ● 0 0 20 40 60 80 100 Percentage of Test Suite (324 Test Cases) * Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing Combinatorial Interaction Testing Justyna Petke

Prioritisation Pairwise prioritisation is no worse than higher-strength prioritisation in terms of interaction coverage and early fault detection. Combinatorial Interaction Testing Justyna Petke

Combinatorial Interaction Testing CIT is a black-box testing technique. A CIT test suite covers all t -way interactions between any t parameters. Pairwise (2-way) testing is the most widely studied CIT technique. Pairwise prioritisation is no worse than higher-strength prioritisation in terms of interaction coverage and early fault detection. The two most popular CIT test suite generation techniques are: greedy and meta-heuristic search. Combinatorial Interaction Testing Justyna Petke

Algorithms Approaches for generating t -way interaction test suites: Greedy Meta-heuristic search Combinatorial Interaction Testing Justyna Petke

Greedy approach - example Three parameters: P 1 , P 2 , P 3 Values for parameter P 1 : 0,1 Values for parameter P 2 : 0,1 Values for parameter P 3 : 0,1,2 Objective: find a pairwise interaction test suite Combinatorial Interaction Testing Justyna Petke

Greedy algorithm for CIT IPOG (In-Parameter-Order-General) IPOG: A General Strategy for T-Way Software Testing Yu Lei et al., 2007 Combinatorial Interaction Testing Justyna Petke

Greedy algorithm IPOG-Test (int t , ParameterSet ps ) 1. initialize test set ts to be an empty set 2. denote the parameters in ps , in an arbitrary order, as P 1 , P 2 , · · · , and P n 3. add into ts a test for each combination of values of the first t parameters 4. for (int i = t + 1; i ≤ n ; i + + ) { 5. let π be the set of t -way combinations of values involving parameter P i and t -1 parameters among the first i -1 parameters 6. for (each test γ = ( v 1 , v 2 , · · · , v i − 1 ) in test set ts ) { 7. choose a value v i of P i and replace γ with γ ’ = ( v 1 , v 2 , · · · , v i − 1 , v i ) so that γ ’ covers the most number of combinations of values in π 8. remove from π the combinations of values covered by γ ’ } 9. for (each combination α in set π ) { 10. if (there exists a test that already covers α ) { 11. remove α from π 12. } else { 13. change an existing test, if possible, or otherwise add a new test to cover α and remove it from π } } } 14. return ts ; Combinatorial Interaction Testing Justyna Petke

Greedy approach - example Adding all combinations of values between the first 2 parameters: P 1 P 2 0 0 0 1 1 0 1 1 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example Set π = pairs to cover involving P 3 : P 1 P 2 P 3 0 0 0 0 0 1 0 1 0 2 0 2 1 0 1 0 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Adding values for P 3 in ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 0 1 0 1 0 1 1 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Adding values for P 3 in ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 1 0 1 0 1 0 1 1 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Adding values for P 3 in ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Adding values for P 3 in ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Extending ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 0 0 Extending ts : 0 1 0 1 P 1 P 2 P 3 0 2 0 0 0 0 2 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 0 0 2 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy approach - example P 1 P 2 P 3 0 0 Extending ts : 0 0 0 1 0 1 P 1 P 2 P 3 0 0 0 0 2 0 1 1 0 2 1 0 1 1 0 1 1 0 1 0 0 0 2 1 1 1 1 2 1 1 1 2 1 2 Combinatorial Interaction Testing Justyna Petke

Greedy algorithm & prioritisation In a greedy algorithm values are added to cover the largest amount of t -way interactions. Thus it already prioritises the test suite! Combinatorial Interaction Testing Justyna Petke

Meta-heuristics Meta-heuristics are strategies that guide the search process. They efficiently explore the search space in order to find near-optimal solutions. They are not problem-specific. Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT Outer Search (guess test suite size) Inner Search (populates test suite) Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT Outer Search (e.g. binary) Inner Search (e.g. simulated-annealing) Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT Evaluating improvements to a meta-heuristic search for constrained interaction testing. Brady J. Garvin et. al, 2011 Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT Bluetooth MP3 Camera Included Included Black&White None None None Colour Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT pick test suite size : 6 Bluetooth MP3 Camera Combinatorial Interaction Testing Justyna Petke

Meta-heuristic search for CIT randomly generate test suite of size 6 Bluetooth MP3 Camera Included Included Black&White None None None None None Colour None None Black&White Included Included None None None Colour Combinatorial Interaction Testing Justyna Petke

Combinatorial Interaction Testing Justyna Petke C entre for R - PowerPoint PPT Presentation

Combinatorial Interaction Testing Justyna Petke C entre for R esearch in E volution, S earch and T esting University College London CREST Using materials from: http://cse.unl.edu/ citportal/ http://csrc.nist.gov/groups/SNS/acts/ftfi.html

Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing Elke Salecker,

Combinatorial Security Testing: Combinatorial Testing Meets Information Security Dimitris E.

NIST Combinatorial Testing project Goals reduce testing cost, improve cost-benefit ratio

Combinatorial Testing Automated testing and J.P . Galeotti - Alessandra Gorla verification

Combinatorial Testing Rick Kuhn NIST Computer Security Division NIST Combinatorial Testing

Combinatorial Testing and Covering Arrays Lucia Moura School of Electrical Engineering and

Learning objectives Understand rationale and basic approach for systematic combinatorial

Combinatorial Auctions Do Need Modest Interaction Sepehr Assadi University of Pennsylvania

Combinatorial Testing Rick Kuhn National Institute of Standards and Technology Gaithersburg, MD

Interaction Testing Chapter 15 Interaction faults and failures Subtle Difficult to detect

Interaction Testing Chapter 15 Interaction faults and failures Subtle Difficult to detect

Validation of Constraints Among Configuration Parameters Using (Search-Based) Combinatorial

Combinatorial Testing Rick Kuhn National Institute of Standards and Technology Gaithersburg, MD

Combinatorial Group Testing in Telecommunications II. Hosszu va Budapesti Mszaki s

Combinatorial Testing on Im Implementations of f HTML5 Support Xi Deng, Tianyong Wu, Jun Yan

Detection of Data Corruption via Combinatorial Group Testing and beyond Kazuhiko Minematsu

Combinatorial Group Testing Problems on Various Models Speaker: Huilan Chang Advisor: Hung-Lin

Introduction to Combinatorial Testing Rick Kuhn National Institute of Standards and Technology

Combinatorial Testing Of ACTS: A Case Study Mehra N.Borazjany, Linbin Yu, Yu Lei - UTA Raghu

Combinatorial Testing Rick Kuhn Raghu Kacker National Institute of Standards and

Rigorous Evaluation Usability Testing What is Usability Testing? Formal and rigorous testing

Local Consistency Combinatorial Problem Solving (CPS) Enric Rodr guez-Carbonell (based on

A .NET-based Test-Data Generator for Combinatorial Grammar- and Schema-based Testing Vadim

Testing Techniques Applied to Virt Devel Cleber Rosa Red Hat, Inc. Agenda Software Testing