Automated Test Data Generation on the Analyses of Feature Models A Metamorphic Testing Approach Sergio Segura 1 , Robert M. Hierons 2 , David Benavides 1 and Antonio Ruiz-Cortés 1 1 Department of Computer Languages and Systems, Univesity of Seville, Spain 2 School of Information Systems, Computing and Mathematics, Brunel University, Uxbridge, Middlesex, UK ICST 2010, Paris
Introduction Problem Contribution Conclusions Future work
Introduction Problem Contribution Conclusions Future work
Introduction A Feature Model represents a set of products in terms of features and relations among them. Feature Feature Mandatory Optional Alternative Or (choose 1) Excludes Requires
Introduction A Feature Model represents a set of products in terms of features and relations among them. Non valid product! = {MobilePhone, Calls, Screen, Basic, GPS , Media, MP3}
Introduction Automated analysis of feature models: Computer-aided extraction of information from feature models How many 14 products?
Introduction Automated analysis of feature models: Computer-aided extraction of information from feature models Yes, feature Any error? “Basic” is dead
Introduction Automated Analyses on FMs Testing Manual Test Suite Automated Test Data Generation Mechanisms Tools Description Constraint BDD OWL SAT Techniques Logic Programming Valid Number of Void Products Filter Products Product Operations Corrective Dead Variability Explanations Optimization Features Explanations over 30 operations
Introduction Input Analysis tool Output The model represents 2016 products How many products? Combinatorial complexity! Combinatorial complexity!
Introduction Metamorphic Testing Follow-up test case x = 42 x = - 42 cos(x) = cos(-x) cos(x) cos(x) Metamorphic Relation - 0.399 - 0.399 - 0.426
Introduction Problem Contribution Conclusions Future work
Problem How can we avoid the oracle problem on the analyses of feature models? How can we automate the generation of test data?
Introduction Problem Contribution Conclusions Future work
Contribution How can we avoid the oracle problem on the analyses of feature models? We propose a set of metamorphic relations on feature models How can we automate the generation of test data? We present an automated test data generator
Contribution How can we avoid the oracle problem on the analyses of feature models? We propose a set of metamorphic relations on feature models How can we automate the generation of test data? We present an automated test data generator
Contribution Metamorphic relations on Feature Models FM’ FM FM’ = FM + mandatory Neighbours products(FM) products(FM) Metamorphic relation P1 = {A,C,D} P1 = {A,C} P2 = {A,B,C,D} P2 = {A,B,C}
Contribution Metamorphic relations on Feature Models FM’ FM FM’ = FM + mandatory (pf) Neighbours (f) # products ( FM ' ) # products ( FM ) products(FM) products(FM) P ' ( P ' products ( FM ' ) P products ( FM ) ( pf features ( P ) P ' P { f }) ( pf features ( P ) P ' P )) Metamorphic relation P1 = {A,C,D} P1 = {A,C} P2 = {A,B,C,D} P2 = {A,B,C}
Contribution Metamorphic relations on Feature Models FM FM’ Metamorphic relation MANDATORY # products ( FM ' ) # products ( FM ) P ' ( P ' products ( FM ' ) P products ( FM ) ( pf features ( P ) P ' P { f }) ( pf features ( P ) P ' P )) OPTIONAL # products ( FM ' ) # products ( FM ) filter ( FM , { pf }, ) P ' ( P ' products ( FM ' ) P products ( FM ) P ' P ( pf features ( P ) P ' P { f })) # products ( FM ' ) # products ( FM ) (# C 1 )# filter ( FM , { pf }, ) ALTERNATIVE P ' ( P ' products ( FM ' ) P products ( FM ) ( pf features ( P ) c C P ' P { c }) ( pf features ( P ) P ' P )) # products ( FM ' ) # products ( FM ) ( 2 # C 1 )# filter ( FM , { pf }, ) P ' ( P ' products ( FM ' ) P products ( FM ) OR ( pf features ( P ) S ( C ) P ' P S ) ( pf features ( P ) P ' P ))) REQUIRES products ( FM ' ) products ( FM ) \ filter ( FM , { f }, { g }) EXCLUDES products ( FM ' ) products ( FM ) \ filter ( FM , { f , g }, )
Contribution How can we avoid the oracle problem on the analyses of feature models? We propose a set of metamorphic relations on feature models How can we automate the generation of test data? We present an automated test data generator
Contribution Automated Test Data Generation 2. Create a neighbour by adding a (random) 2. Create a neighbour by adding a (random) 1. Start with a trivial test case relationship to the model relationship to the model Or P1 = {A,C} P1 = {A,C} P2 = {A,B,C,D} P2 = {A,B,C} P3 = {A,B,C,E} P4 = {A,B,C,D,E} 3. Construct the set of products represented by the new 3. Construct the set of products represented by the new model according to our metamorphic relations model according to our metamorphic relations
Contribution Automated Test Data Generation Or Alternative Excludes Optional P1 = {A,C} P1 = {A,C} P1 = {A,C,F} P1 = {A,C,F} P2 = {A,B,C} P2 = {A,B,C,D} P2 = {A,B,G} P2 = {A,B,C,D,F} P3 = {A,B,C,E} P1 = {A,C,F} P3 = {A,B,C,D,F} P3 = {A,B,C,E,F} P4 = {A,B,C,D,E} P2 = {A,B,C,D,F} P4 = {A,B,C,E,F} P4 = {A,B,C,D,E,F} P3 = {A,B,C,E,F} P5 = {A,B,C,D,G} P4 = {A,B,C,D,E,F} P6 = {A,B,C,E,G} P5 = {A,B,C,E,F,H} P7 = {A,B,C,D,E,G} P6 = {A,B,C,D,E,F,H} P8 = {A,B,C,D,E,F} Yes, feature „G‟ Does the model How many Yes, it represents at Is the model is not included 6 products Commonality(B) = 83.3% Metrics? contain any dead products it least one product consistent? in any product. Variability = 0.023 represents? feature?
Contribution Automated Test Data Generation 50 features 5% cross-tree constraints 13,741,632 products
Contribution Evaluation Mutation Testing Reasoner LoC #Mutants Av Time (s) Max Time (s) Max TCs Score Sat4jReasoner 743 262 7.4 566.5 414 99.3% JavaBDDReasoner 625 302 1.6 111.7 350 98.7% JaCoPReasoner 686 185 0.7 8.3 83 100% Total 2054 749 3.2 566.5 414 99.3% 6 operations. Timeout: 600s Real faults and tools Score: 91.6% Average time: 20.2 seconds Batory’06 - SPLC 2 defects found: • Sat4jStagedConfigurations Av Time: 0.5 seconds • JaCoPProducts Av Time: 142.9 seconds FaMa Framework v1.0 alpha
Introduction Problem Contribution Conclusions Future work
Conclusions • Contributions: • Set of metamorphic relations on feature models overcoming the oracle problem. • Automated test data generator relying on them. • Main points of our approach: • Efficient. Hundred of complex feature models and their associated set of products can be generated within seconds. • Flexible . It can be used to test any operation extracting information from the set of products of a feature model (most of them) • Effective . Mutation score of 99.3%. Two defects identified in a recent release of FaMa .
Introduction Problem Contribution Conclusions Future work
Ongoing and Future Work • Comparison of our automated generator and FaMa manual suite Results have revealed a great superiority of our automated approach. • Combination of our manual suite and our automated test data generator. We have got significant improvements in terms of efficiency • Evaluation with other tools . Two defects found in SPLOT . http://www.splot-research.org • Study the applicability of our approach in other domains.
Thanks! www.isa.us.es sergiosegura@us.es www.isa.us.es/fama
Recommend
More recommend
