Combinatorial Interaction Testing for Test Selection in - PowerPoint PPT Presentation

Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing Elke Salecker, Sabine Glesner Technical University of Berlin, Germany Workshop on Combinatorial Testing (CT) 2012 Montreal, Canada

Introduction Background Approach Evaluation Conclusions Motivation Grammar-based Testing • black-box testing approach • test data derived from context-free grammar ➠ automatic generation of test data Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions Motivation Grammar-based Testing • black-box testing approach • test data derived from context-free grammar ➠ automatic generation of test data Problem • exhaustive testing of test sets infeasible • controlled/random-based enumaration process • rule combination coverage not in focus Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions Motivation Grammar-based Testing • black-box testing approach • test data derived from context-free grammar ➠ automatic generation of test data Problem • exhaustive testing of test sets infeasible • controlled/random-based enumaration process • rule combination coverage not in focus New Approach ➠ test selection based on combinatorial interaction testing ➠ automatic generation of CIT test specification Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions Outline 1 Introduction 2 CIT and Grammars 3 Test Set Generation 4 Evaluation 5 Conclusions Elke Salecker, Sabine Glesner 2

Introduction Background Approach Evaluation Conclusions Combinatorial Interaction Testing CIT specification: P1 P2 P3 P4 parameters, values a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2 a 3 Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions Combinatorial Interaction Testing CIT specification: P1 P2 P3 P4 parameters, values a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2 a 3 test case (a 1 , b 1 , c 1 , d 2 ) Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions Combinatorial Interaction Testing CIT specification: P1 P2 P3 P4 parameters, values a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2 a 3 test case (a 1 , b 1 , c 1 , d 2 ) test set P1 × P2 × P3 × P4, 24 test cases Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions Combinatorial Interaction Testing CIT specification: P1 P2 P3 P4 parameters, values a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2 a 3 test case (a 1 , b 1 , c 1 , d 2 ) test set P1 × P2 × P3 × P4, 24 test cases coverage criterion (t=2) 1 a 3 b 1 c 1 d 2 2 a 2 b 2 c 2 d 2 3 a 1 b 2 c 1 d 2 4 a 1 b 1 c 2 d 1 5 a 2 b 1 c 1 d 1 6 a 3 b 2 c 2 d 1 Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions Combinatorial Interaction Testing CIT specification: P1 P2 P3 P4 parameters, values a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2 a 3 test case (a 1 , b 1 , c 1 , d 2 ) test set P1 × P2 × P3 × P4, 24 test cases coverage criterion (t=2) 1 a 3 b 1 c 1 d 2 2 a 2 b 2 c 2 d 2 3 a 1 b 2 c 1 d 2 4 a 1 b 1 c 2 d 1 5 a 2 b 1 c 1 d 1 6 a 3 b 2 c 2 d 1 constraints ¬ ( b 1 ∧ c 2 ) Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals ǫ → Assign((ObjAddr a), r) rules def: (lhs → rhs) r → Plus(r, r) add: r → Plus(r, imm) addimm: use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c start symbol ǫ Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals ǫ → Assign((ObjAddr a), r) rules def: (lhs → rhs) r → Plus(r, r) add: r → Plus(r, imm) addimm: use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c start symbol ǫ sequence of rule applications derivation < add, addimm, r2c > r Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals ǫ → Assign((ObjAddr a), r) rules def: (lhs → rhs) r → Plus(r, r) add: r → Plus(r, imm) addimm: use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c start symbol ǫ sequence of rule applications derivation < add, addimm, r2c > r Plus r r Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals ǫ → Assign((ObjAddr a), r) rules def: (lhs → rhs) r → Plus(r, r) add: r → Plus(r, imm) addimm: use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c start symbol ǫ sequence of rule applications derivation < add, addimm, r2c > r Plus Plus r r r Plus r imm Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Grammars nonterminals N = { ǫ , r, imm } Σ = { Const, ObjAddr, Content, Plus, Assign } terminals ǫ → Assign((ObjAddr a), r) rules def: (lhs → rhs) r → Plus(r, r) add: r → Plus(r, imm) addimm: use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c start symbol ǫ sequence of rule applications derivation < add, addimm, r2c > r Plus Plus Plus r r r r Plus Plus r imm Const 2 imm Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions Outline 1 Introduction 2 CIT and Grammars 3 Test Set Generation 4 Evaluation 5 Conclusions Elke Salecker, Sabine Glesner 5

Introduction Background Approach Evaluation Conclusions Symmetry in Derivations r2c r → Const c i2c imm → Const c add r → Plus(r, r) r → Plus(r, imm) addimm derivation: add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c Plus Plus Const 4 Const 2 Const 2 Elke Salecker, Sabine Glesner 6

Introduction Background Approach Evaluation Conclusions Symmetry in Derivations r2c r → Const c i2c imm → Const c add r → Plus(r, r) r → Plus(r, imm) addimm derivation: derivation: add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c add ⇒ r2c ⇒ addimm ⇒ r2c ⇒ i2c Plus Plus Plus Const 4 Const 4 Plus Const 2 Const 2 Const 2 Const 2 Elke Salecker, Sabine Glesner 6

Introduction Background Approach Evaluation Conclusions Principle of Specification Generation Derivations with Fixed Length 1 2 3 4 5 6 1 def add add r2c r2c r2c 2 def add use add r2c r2c 3 def add r2c add r2c r2c . . . 6 def add addimm r2c i2c use 7 def add addimm use i2c use . . . 30 def add use add use use Elke Salecker, Sabine Glesner 7

Introduction Background Approach Evaluation Conclusions Principle of Specification Generation Derivations with Fixed Length 1 2 3 4 5 6 1 def add add r2c r2c r2c 2 def add use add r2c r2c 3 def add r2c add r2c r2c . . . 6 def add addimm r2c i2c use 7 def add addimm use i2c use . . . 30 def add use add use use P1 P2 P3 P4 P5 P6 derivation length = number of parameters value of parameter i = rule can be applied in step i constraints for invalid combinations Elke Salecker, Sabine Glesner 7

Introduction Background Approach Evaluation Conclusions CIT Specification Generation 1 Generate compact representation of derivations 2 Calculate value sets 3 Calculate constraints Elke Salecker, Sabine Glesner 8

Introduction Background Approach Evaluation Conclusions Generate Compact Representation of Derivations Recursive Construction: Base Case derivation length = 1 store for terminal rules lhs nonterminal, rule identifier derivation length > 1 Recursion Elke Salecker, Sabine Glesner 9

Introduction Background Approach Evaluation Conclusions Generate Compact Representation of Derivations Recursive Construction: Base Case derivation length = 1 store for terminal rules lhs nonterminal, rule identifier derivation length > 1 Recursion check remaining rules split overall length for number of nonterminals in rule find smaller derivations store lhs nonterminal, rule identifier, extended pattern Elke Salecker, Sabine Glesner 9

Introduction Background Approach Evaluation Conclusions Generate Compact Representation of Derivations def: L NT Rule RHS Pattern ǫ → Assign((ObjAddr a),r) ∅ 1 r use add: ∅ r2c r → Plus(r, r) imm i2c ∅ addimm: r → Plus(r, imm) use: r → Content(ObjAddr a) r2c: r → Const c r2i: r → imm i2c: imm → Const c Elke Salecker, Sabine Glesner 10