Constraint-based Testing Software Engineering Gordon Fraser • Saarland University Based on slides by Arnaud Gotlieb & Koushik Sen So many different test objectives... Path testing 0,*!#@0A.(#-.(4#$%&!'(,#-.(.L!#$%&!'.(#-.(.7 subsumes Theoretical Criteria CoC ! Path testing !" #$%&!'()*&!+!(,#-.(./ #$%&!'.)*&!+!.(#-.(./ 0,*!-1!+!2/ All-DU paths Boundary Compound RACC RICC 9$0:(!4'()*&7!" # interior testing condition testing >%:?( ;&<( #$%&!#/ " All uses #!+!'()*&/ 03!4#!++!5657!"!! >%:?( ;&<( MCDC testing $ % !(:?(03!4#!++!5K57!" '.)*&!+!5!5/ CACC GICC All-p-some-c 8! >%:?( ;&<( Branch and Practical Criteria (:?( & 0,*!.0@0*A$0@$!+!B(CAD%:<(?E'466()*&7F/ ' '.)*&!+!'()*&/ LCSAJ testing 0,*!.0@0*A:-9!+!B(CAD%:<(?E'466()*&7F/ condition testing 8 03!4.0@0*A$0@$!++!GH!II!.0@0*A:-9!++!GH7!" All-c-some-p All-p uses GACC >%:?( ;&<( (:?(!" ( -1!+!H/ ) Branch testing '.)*&!+!HJ!'!.0@0*A$0@$!6!.0@0*A:-9/ 8 8 All-c uses All-defs Branch testing Loop boundary Basic CC PC Statement testing 66.)*&/ * testing condition testing 66()*&/ '.)*&!+!5=25/ + 8 &(*<&,!-1/ Statement testing 8 But how to create the tests? We are still looking at the problem of deriving input data that will lead Test Data Generation execution to some particular point in the control flow that is interesting Given a function and a location we want to for testing reasons. Today, we will reach, how do we derive inputs to the consider constraint based testing, function that lead the control flow to the which allows us to reason about desired statement? the precise conditions under which a test goal is satisfied, and allows us to deduce test data satisfying the test goals. Think Guess Search Deduce
Looking at the control flow graph of 0,*!#@0A.(#-.(4#$%&!'(,#-.(.L!#$%&!'.(#-.(.7 the cgi_decode example, we see A ! !" #$%&!'()*&!+!(,#-.(./ that in order to reach node E #$%&!'.)*&!+!.(#-.(./ 0,*!-1!+!2/ several conditions have to hold - B conditions on variables that are 9$0:(!4'()*&7!" # c has to be “+” >%:?( ;&<( *eptr has to be altered during the program C #$%&!#/ " #!+!'()*&/ true 03!4#!++!5657!"!! execution. >%:?( ;&<( $ D % E '.)*&!+!5!5/ !(:?(03!4#!++!5K57!" 8! >%:?( ;&<( (:?( & F G ' 0,*!.0@0*A$0@$!+!B(CAD%:<(?E'466()*&7F/ '.)*&!+!'()*&/ 0,*!.0@0*A:-9!+!B(CAD%:<(?E'466()*&7F/ 8 03!4.0@0*A$0@$!++!GH!II!.0@0*A:-9!++!GH7!" >%:?( ;&<( Under which H I ( ) (:?(!" -1!+!H/ conditions do we '.)*&!+!HJ!'!.0@0*A$0@$!6!.0@0*A:-9/ 8 8 reach this point? L 66.)*&/ * 66()*&/ M '.)*&!+!5=25/ + 8 &(*<&,!-1/ 8 Constraint-based Testing • Constraint generation Extract a constraint system from the program and a testing objective • Constraints on inputs If inputs satisfy constraints, then testing objective will be satisfied • Constraint solving: Solve the constraint system to generate test data • Static analysis aims at finding runtime errors (e.g. division-by-zero, overflows, ...) at compile time • CBT aims at finding functional faults (e.g. P returns 3 while 2 was expected)
• Model-checking tools explore paths of software models for proving properties • CBT looks only for counter-examples • Dynamic analysis approaches extract likely invariants • CBT exploits symbolic reasoning to find counter-examples to given properties The ida of constraint based testing is to transform a program and a Overview test goal for that program to a constraint system... Program Test Goal Constraint system 0,*!#@0A.(#-.(4#$%&!'(,#-.(.L!#$%&!'.(#-.(.7 A ! !" #$%&!'()*&!+!(,#-.(./ #$%&!'.)*&!+!.(#-.(./ (a>b ∧ c = 2) 0,*!-1!+!2/ B 9$0:(!4'()*&7!" # ∨ (a>5 ∧ >%:?( ;&<( C #$%&!#/ " #!+!'()*&/ 03!4#!++!5657!"!! >%:?( ;&<( D b>a) ∨ (a<b $ D E % '.)*&!+!5!5/ !(:?(03!4#!++!5K57!" 8! >%:?( ;&<( (:?( F & 0,*!.0@0*A$0@$!+!B(CAD%:<(?E'466()*&7F/ G ' '.)*&!+!'()*&/ 0,*!.0@0*A:-9!+!B(CAD%:<(?E'466()*&7F/ ∧ b<5 ∧ a > 8 03!4.0@0*A$0@$!++!GH!II!.0@0*A:-9!++!GH7!" >%:?( ;&<( H I (:?(!" ( -1!+!H/ ) '.)*&!+!HJ!'!.0@0*A$0@$!6!.0@0*A:-9/ 8 0) 8 L 66.)*&/ * 66()*&/ '.)*&!+!5=25/ M + 8 &(*<&,!-1/ 8
...and then use a constraint solver to automatically derive a solution to Overview the constraint system - which is our test data. Constraint system Constraint solver Test data (a>b ∧ c = 2) ∨ (a>5 ∧ a = 1 b>a) ∨ (a<b b = 2 c = 3 ∧ b<5 ∧ a > 0) Constraint Solving Relevant questions: Does the constraint system (CS) have a What is a constraint? solution? Can we generate a solution to CS? Can we generate the best solution to CS? • A constraint is a condition that a solution to an optimization problem must satisfy • x > 5, y < 10 • a > b ∧ b > 10 • Constraint satisfaction: Finding value assignments to variables such that constraints are satisfied
Relevant Questions • Does the constraint system (CS) have a solution? To decide whether the testing objective is reachable or not • Can we generate a solution to CS? Test data generation • Can we generate the best solution to CS? Test data generation that optimizes a cost function Constraint solving • Computational domain, constraint language results from the choice of programs and properties to be considered • Booleans - Boolean formula (A&&B&&!C)||(...) • Integers • Bounded Integers • Rationals • Reals • Floating-point numbers • The landscape of complexities for Decidability and different domains (rows) and types of constraints (columns) looks complexities scary - in practice, however, we can handle anything involving non- linear constraints quite well using Polynomial Non-linear Boolean Formula Linear constraints heuristics. constraints constraints 2-SAT in P 0-1 programming Booleans 3-SAT is NP- ? ? is NP-complete complete Bounded integers - NP-complete NP-complete NP-complete Integer Integers - programming is Undecidable Undecidable NP complete Nonlinear Linear Rationals and reals - programming is Undecidable programming in P NP-complete
Decision procedures (best practices) Polynomial Non-linear Boolean Formula Linear constraints constraints constraints Davis & Putnam Booleans (DPLL) Cooper’s procedure fir Bounded integers Oresburger algorith. Constraint Constraint Constraint Integers satisfaction satisfaction satisfaction Simplex Groebner basis Interval Rationals and reals Fourier (Buchberger alg) propagation Elimination Constraint Satisfaction • A constraint system involves a set of variables V, a set of finite domains D, and a set of constraints C • A solution is an assignment of V to values in D that satisfies C • A constraint system is unsatisfiable when it has no solutions • Constraint satisfaction involves 3 interleaved processes: Constraint propagation Constraint Variable filtering labeling Constraint filtering • Given a single constraint, filter the domains of variables by removing inconsistent values • Depends on a level of consistency to be achieved Domain consistency - bound consistency - and many more • Example: X in {2,3,4,6,10}, Y in {1,2,3,4,6,8}, Z in {6}, X*Y=Z
Domain consistency checks for each variable of a constraint for Domain Consistency which of its values there is support in the values of the other variables. For each value in Dx, find a support in Dz and Dy {2,3,4,6,10} {6} X Z X*Y=Z Y {1,2,3,4,6,8} Bound consistency only reduces the domain bounds, and is Bound Consistency therefore cheaper. For each bound in Dx, find a support in Dz and Dy {2,3,4,6,10} {6} X Z X*Y=Z Y {1,2,3,4,6,8} After constraint filtering, the reduced value domains (result of Constraint Propagation function narrow in above source code) are propagated to other constraints that involve the same • Propagates prunings throughout the constraint system variables. This process is • Implemented as a fixpoint algorithm: implemented as a fixpoint algorithm that iterates until no more changes Agenda := C; can be propagated. while(!Agenda.isEmpty()) { c := POP(Agenda); D’ := narrow(c,D); if(D’ != D) Agenda := Agenda ∪ {c’ in C | vars(c’) ∩ vars(c)!= ∅ } D := D’ } return D’;
Constraint Propagation 15 10 5 0 0 5 10 15 X, Y in 0..10 , X*Y=6, X+Y=5 Constraint Propagation 15 12 9 6 3 0 0 3 6 9 12 15 X, Y in 0..10, X*Y=6 , X+Y=5 Constraint Propagation 15 12 9 6 3 0 0 3 6 9 12 15 Y in 0..10, X*Y=6, X+Y=5 X,
Recommend
More recommend
