A .NET-based Test-Data Generator for Combinatorial Grammar- and - PowerPoint PPT Presentation

A .NET-based Test-Data Generator for Combinatorial Grammar- and Schema-based Testing Vadim Zaytsev with: Ralf L¨ ammel (VU), Wolfram Schulte (MSR) 14 April 2004

Grammar ware • grammars • grammar-dependent software • In this project: – XML Validators – W3C XML Schemata as grammars http://www.cs.vu.nl/grammarware/ 1

Scenarios for grammar-based testing • testing virtual processors – virtual machines – just-in-time compilers • testing front-ends – automated software modification & analysis • testing implementations – optimisation of XPath 2

Combinatorial exploration Grammar Explosion Term Term Term Term Term Term Term Term Term Term Term Term Term ... . . . Adversary of stochastic testing 3

Explosion examples Cardinalities per depth 4

Controlled explosion Grammar Term Depth control Term Term Recursion control Term Term Term Term Term Term Term Term Term Term ... . . . + other mechanisms 5

Control mechanisms • depth control – maximum “length” of terms • recursion control – nested constructor application • equivalence control – build equivalence classes 6

Control mechanisms (contd.) • balance control – limit the preceding levels • combination control – limit Cartesian product – pair-wise testing • context control – enforce context conditions 7

Emphases in this project • The case study of XSD usage in testing XML Validators • Implementing and using control mechanisms for test data generation • Developing a tool to support combinatorial testing 8

Problem System Under Test XML XSD Validator YES NO 9

Solution Stage A: TDGenerator XML XSD Validator Oracle YES YES NO GOOD/BAD 10

Solution Stage B: TDGenerator XSD XML XSD Validator Oracle YES YES NO GOOD/BAD 11

What kind of Oracle ? • Differential testing – run two or more against one another – if the outputs are different, something has to be wrong – in our case: different XML Validators ∗ using Microsoft .NET API ∗ Sun Multi-Schema XML Validator (JAXB) ∗ Ant Validation Task (JBind) 12

The Tool we have grammar terms Tree Nil = Nil Node-1(Nil,Nil) | Node(Tree,Tree) ... ; Terms as objects Constructors of all sorts +Serialisation 13

Solution proposition grammar terms The Tool in BNF in text Constructors Serialisation Sorts and Parsing Terms grammar test data Mapping Mapping XSD XML XML XSD 14

Problems underway • XSD is not meant to be implemented – (as a whole) • YACC ification • how to deal with XML attributes • implementing control mechanisms • . . . 15

Conclusion The buzzwords are: • Test data generation • Combinatorial testing • Controlled explosion • Differential testing • The .NET Framework 16