Syntax-based test coverage (part 2) - grammar-based testing - - PowerPoint PPT Presentation

Syntax-based test coverage (part 2) - grammar-based testing - Basic grammar concepts Syntax-based Coverage Criteria Specification-based Grammars Input Space Grammars Verificação e Validação de Software Departamento de Informática Faculdade de Ciências da Universidade de Lisboa Eduardo Marques, Vasco Thudichum Vasconcelos

Grammar-based testing Grammar-based testing Inputs for the SUT are many times (or can be seen as) defined by a grammar . Testing the SUT with valid inputs: exercise productions of the grammar according to some criterion. Testing the SUT with invalid inputs: consider grammar-based mutations are also considered to test the SUT with invalid inputs. Examples XML program data (in particular XML data with associated XML schemas) Programs written in a programming language (to test compilers) ! Assorted data formats and instances of them (JSON, ASN.1, …) … 2

Example grammar A simple grammar for a toy language of arithmetic expressions in BNF notation production rule root symbol expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num terminal op ::= + | - | * | / symbols letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 non-terminal 6 non-terminal symbols symbols 40 (4+26+10) terminal symbols 47 production rules (3+2+2+4+26+10) 3

Derivations and syntax tree Syntax tree for expr ::= id | num | expr op expr id ::= letter | letter id ab+12 num ::= digit | digit num op ::= + | - | * | / expr letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 op expr expr Sample derivations: + id num a letter id digit num expr => id => letter => a 43 letter 1 digit a expr => num => digit num => 4 num => 4 digit => 43 b 2 ab+12 expr => expr op expr => expr + expr => ... => ab+12 4

Coverage criteria Terminal Symbol Coverage (TSC) TR contains each terminal in the grammar. One test case per terminal. Toy language example: 40 test requirements Production Coverage (PDC) TR contains each production rule in the grammar. One test case per production (hence PDC subsumes TSC ). Toy language example: 47 test requirements Derivation Coverage (DC) TR contains every possible string that can be derived from the grammar. One test case per derivation required. Not practical - TR usually infinite as in our toy language. When applicable, DC subsumes PDC. 5

Exercise 1 expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num op ::= + | - | * | / letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 Suppose we wish to test a parser or interpreter for the example toy language. 1. Define a test set (i.e., a set of grammar derivations) that satisfies TSC but not PDC. 2. Complete the previous test set to satisfy PDC. 6

Exercise 1 (solution) expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num op ::= + | - | * | / letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 1. We can consider the following tests for TSC: a b … z (26 tests) 0 1 … 9 (10 tests) a+b a-b a*b a/b (4 tests) PDC is not satisfied since no test case covers productions id ::= letter id or num ::= digit num 2. To satisfy PDC we can, for instance, additionally consider: ab 12 7

Grammar-based mutation Mutations over a grammar can be considered with the purpose of generating invalid derivations. Invalid derivations must be recognized as errors by the software under test. Example mutation operators : Terminal and nonterminal deletion : remove a terminal or nonterminal symbol from a production. Terminal and non-terminal duplication : duplicate a terminal or nonterminal symbol in a production. Terminal replacement : replace a terminal by another terminal. Non-terminal replacement 8

Grammar mutations - example empty string expr ::= id | num | expr op expr | ε | op expr | expr expr | expr op id ::= letter | letter id num ::= digit | digit num mutant productions op ::= + | - | * | / resulting from nonterminal letter ::= a | … | z deletion for expr productions digit ::= 0 | 1 | 2 | … | 9 Example mutant derivations: 1) ε , 2) +a , 3) aa , 4) 12+ Exercise 2 : Consider nonterminal duplication over expr . Describe the mutated grammar and provide one mutant derivation per each mutated production rule. 9

Nonterminal duplication expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num op ::= + | - | * | / letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 Exercise 2 : Consider nonterminal duplication over expr . Describe the mutated grammar and provide one mutant derivation per each mutated production rule.

Exercise 2 (solution) expr ::= id | num | expr op expr | id id | num num | expr expr op expr | expr op op expr | expr op expr expr id ::= letter | letter id num ::= digit | digit num mutant productions op ::= + | - | * | / resulting from nonterminal letter ::= a | … | z duplication digit ::= 0 | 1 | 2 | … | 9 Example strings from mutated rules (not all of them are invalid inputs): 1) aa (valid) , 2) 1213 (valid) , 3) a +1a+b , 4) a++10 , 5) a+1a The “ id id ” and “ num num ” rules always produce valid inputs. Some derivations of “ expr expr op expr ” and “ expr op expr expr” can be valid (e.g. ab+c, a+bc). Derivations of “ expr op op expr ” are always invalid. 11

Specification-based Mutation • Address specifications as finite state machines • A finite state machine is essentially a graph G, as defined in the Graph Coverage lecture, with a set of states (nodes), a set of initial states (initial nodes), and a transition relation (the set of edges) • Finite state descriptions can capture system behavior at a very high level – suitable for communicating with the end user. FSM are useful for system testing. • There are powerful analysis tools for FST: Model Checkers. Produce counterexamples for properties that do not hold in the FSM

FSM _ Example • States described by the truth values of two variables, x and y A&O, Figure 5.4

FSM Mutation Operators • Constant replacement: Replace the “next” value of a variable by a different constant • E.g., In state FT, turn into T the next value of y A&O, Figure 5.5

Killing a mutant • We need a sequence of states that is allowed by the transition relation of the original state machine, but not by the mutated state machine. Such a sequence is precisely a test case that kills the mutant • E.g., FF → TT → FT → TF • Finding a test to kill a mutant A&O, Figure 5.5 of a FSM can be automated using a model checker

Model Checking • A model checker takes two inputs: A FSM and a statement of some property, expressed in a temporal logic. • In this case we are interested in: It is always the case that if x=false and y=true then eventually y=false A&O, Figure 5.5 • ☐ ( ¬ x ⋀ y → ♢ ¬ y )

Counterexample • The model checker produces the following computation (a sequence of states): 1. FF 2. TT 3. FT 4. Goto 2 • If the model checker does A&O, Figure 5.5 not produce a counterexample, we know ☐ ( ¬ x ⋀ y → ♢ ¬ y ) that the mutant is equivalent

Exercise 3 • Consider the mutant “wantP = false; 4”. Write a temporal logic formula that kills the mutant

Input Space Grammars • One common use of grammars is to define the syntax of the inputs to a program, method, or software component • We will look at - regular expressions - XML

Regular expressions (deposit Account Amount | debit Account Amount) ∗ where Account is Digit Digit Digit Digit is 0|1|2|3|4|5|6|7|8|9 Amount is $ Digit+ . Digit Digit • Valid inputs: - deposit 739 $12.35 deposit 644 $12.35 debit 739 $19.22 • Invalid inputs: - deposit 644 $.35 - deposit 23 $12.35 - withdraw 739 $12.35

Mutation for regular 
 
 
 
 expressions 1. Subexpression replacement 
 Digit is 0|1|2|3|z|5|6|7|8|9 Account is Digit Digit Amount 2. Subexpression deletion 
 Digit is 0|1|2|3|5|6|7|8|9 Account is Digit Digit 3. Subexpression duplication Digit is 0|1|2|3|55|6|7|8|9 Account is Digit Digit Digit Digit

Exercise 4 (deposit Account Amount | debit Account Amount) ∗ where Account is Digit Digit Digit Digit is 0|1|2|3|4|5|6|7|8|9 Amount is $ Digit+ . Digit Digit For each of Digit is 0|1|2|3|z|5|6|7|8|9 the six Account is Digit Digit Amount mutants find a Digit is 0|1|2|3|5|6|7|8|9 string that is Account is Digit Digit in the original Digit is 0|1|2|3|55|6|7|8|9 RE but not in the mutated Account is Digit Digit Digit Digit one

XML • Commonly used in Web applications (JSON also gaining popularity) • XML documents (or messages) must be well formed: opening tags matched by corresponding closing tags A&O, Figure 5.8

XML Schemas • XML documents can be constrained by grammar definitions, written in XML Schemas • In example: unbounded number of books; 6 pieces of info per book; ISBN optional; price is decimal w/ 2 digits after the ‘,’ and min value 0; … A&O, Figure 5.9