Testing Programs Programs Testing with Symmetry Symmetry with and why why not not Java Java Card Card applets applets and and APIs APIs ? ? and Arnaud Gotlieb Gotlieb Arnaud IRISA / INRIA IRISA / INRIA Rennes, FRANCE Rennes, FRANCE Cassis Cassis -- Mar 2004 -- Mar 2004 1 1
Outline Outline Motivations Motivations A Generalized Generalized definition definition of of symmetry symmetry relation relation A Symmetric Testing Testing Symmetric First experimental experimental results results First Related and and further further works works Related Cassis Cassis -- Mar 2004 -- Mar 2004 2 2
A diagrammatic diagrammatic view view of of Program Program Testing Testing A A sequential program computing Input test data Outputs Oracle checking verdict (pass, fail, ?) Cassis Cassis -- Mar 2004 -- Mar 2004 3 3
Non- -testable testable programs programs [Weyuker Weyuker 82] 82] Non [ � No ( � No (complete complete and and correct) oracle correct) oracle available available Because Because No formal formal model model available available No Only informal informal and and partial partial specifications specifications Only Expected results results too too difficult difficult to to compute compute by hand by hand Expected … … Typical examples examples: : Typical APIs, , third third- -party party libraries libraries (no no source code) source code) APIs ( COTS (no no source code) source code) COTS ( complex mathematical mathematical functions functions complex Cassis Cassis -- Mar 2004 -- Mar 2004 4 4
Testing with with symmetry symmetry : a : a very very first first example example Testing P : a program program that that implements implements the the gcd gcd of of 2 2 integers integers P : a Problem: P(1309, 693) = ? : P(1309, 693) = ? Problem relation: ∀ ∀ u, ∀ v, u, ∀ Symmetry relation: v, gcd gcd (u,v) = (u,v) = gcd gcd (v,u) (v,u) Symmetry ≠ P(693, 1309) , if P(1309, 693) ≠ Hence, if P(1309, 693) P(693, 1309) Hence then verdict = fail fail then verdict = Cassis Cassis -- Mar 2004 -- Mar 2004 5 5
Outline Outline Motivations Motivations A Generalized Generalized definition definition of of symmetry symmetry relation relation A Symmetric Testing Testing Symmetric First experimental experimental results results First Related and and further further works works Related Cassis Cassis -- Mar 2004 -- Mar 2004 6 6
Background on Group Theory Theory Background on Group iff ∃ ∃ neutral, ∀ x ∃ inverse(x neutral, ∀ x ∃ Group (E,o) o) iff inverse(x), o associative ), o associative Group (E, Symmetric Group S S n : set of permutations over {1,..,n} Symmetric Group n : set of permutations over {1,..,n} θ .x denotes (x ) θ if x = (x 1 ,..,x n .x denotes (x θ , .., x x θ ) if x = (x 1 ,..,x n ) (1) , .., ) ) θ (1) θ (n (n) τ = (12) and σ σ = (12..k) can be generated by τ S n = (12) = (12..k) S n can be generated by and Group homomorphism from from S S k to S S l Group homomorphism k to l ϕ : ϕ ( θ o θ ’ ϕ ( θ ) o ϕ ( θ ’ ϕ such as ϕ ( θ o θ ) = ϕ ( θ ) o ϕ ( θ : S S k � S S l ’) = ’) ) k � such as l Cassis Cassis -- Mar 2004 -- Mar 2004 7 7
Symmetry relation relation [ Symmetry [Gotlieb Gotlieb ISSRE 03] ISSRE 03] Program p : p : D D 1 ... x D k � D 1 ’ x ... x D l ’ Program x ... x D � D 1 ’ x ... x D l ’ 1 x k ψ k ψ is a a symmetry symmetry relation relation for p for p over over D D 1 ... x D k iff : : ,l is x ... x D k iff 1 x k,l 1) ∀ ∀θ θ ∈ ∈ S ∃η ∈ ∈ S ∀ x θ .x η .p , ∃η as ∀ p( θ ) = η S k S l , such such as x p( .x ) = .p( x ) ( x ) 1) k , l , 2) ψ k,l : : S S k � S S l is a group a group homomorphism homomorphism is k � l θ � � η η θ ψ 2,1 a ψ over ℕ ℕ x ℕ x ℕ Ex: gcd gcd satisfies satisfies a symmetry relation relation over Ex: 2,1 symmetry Cassis Cassis -- Mar 2004 -- Mar 2004 8 8
Symmetry relation : relation : examples examples Symmetry Methods from from java.util.Collections Perm. Per. Symm Methods Perm. Per. Symm java.util.Collections inputs outp inputs outp relation relation (12 symmetric methods over 19 distincts methods) ψ |A|,|A| boolean replaceAll replaceAll (List A, (List A, ψ boolean A A A A |A|,|A| Object oldVal oldVal, ,Object Object newVal newVal) ) Object ψ |A|,1 Object max (Collection A) (Collection A) ψ Object max A Ret A Ret |A|,1 ψ |A|,|B| void copy (List B, (List B, List List A) A) ψ void copy A B A B |A|,|B| ψ |A|,|A| ψ void sort (List A) (List A) void sort A A A A |A|,|A| ψ 1,n ψ List nCopies nCopies ( (int int n, n,Object Object O) O) O Ret List O Ret 1,n Cassis Cassis -- Mar 2004 -- Mar 2004 9 9
Finding symmetry symmetry violations violations Finding � The symmetry relation has to be given by the tester: in extension { ( θ , η ) } ∀θ∈ Sk � If p( θ .x) ≠ η .p(x) for any x ∈ D 1 x .. x D k then verdict = fail � Any test data generator can be employed (random, pair-wise, boundary-value, …) But, how to find all the symmetry violations ? 1 0 1 0 Cassis Cassis -- Mar 2004 -- Mar 2004
Outline Outline Motivations Motivations A Generalized Generalized definition definition of of Symmetry Symmetry relation relation A Symmetric Testing Testing Symmetric First experimental experimental results results First Related and and further further works works Related Cassis Cassis -- Mar 2004 -- Mar 2004 11 11
Local exhaustive testing testing [Wood,Miller, Local exhaustive [Wood,Miller,Noonan Noonan 92] 92] tuned for for testing testing symmetry symmetry relations relations tuned - Tries exhaustively all the input values into a restricted finite domain D of D 1 x .. x D k � in Symmetric Test., a Cartesian Product iterator Ex: {a,b} x {c,d,e} gives (a,c),(a,d),(a,e),(b,c),(b,d),(b,e) - Proves that p( θ .x) = η .p(x) holds ∀ x ∈ D when both the executions of p( θ .x) and p(x) terminate Cassis Cassis -- Mar 2004 -- Mar 2004 12 12
Comparison checks checks Comparison ∀ x ∈ D ∀θ θ ∈ ∈ S ∀ x ∈ , ∀ S k ST checks checks: : D , , ST k, θ θ x x θ .x θ x p p p(x) p(x) o θ θ (x ≠ ψ ψ k θ ) ,l ( θ p o ) o o p p (x) x) ≠ p (x) k,l η = ψ ψ k ,l ( θ θ ) ) k,l � but but there are k! permutations in there are k! permutations in S S k � k ψ k θ ) θ ∈ ∈ S needs to know ψ ,l ( θ ) for all θ � needs to know S k � k,l k Cassis Cassis -- Mar 2004 -- Mar 2004 13 13
Checking only only two two permutations: permutations: Checking Symmetric Testing requires only to check τ = (12) and σ = (12..k) p o τ = ψ k,l ( τ ) o p Proposition : ∀θ ∈ S k , p o θ = ψ k,l ( θ ) o p ⇔ p o σ = ψ k,l ( σ ) o p Sketch of proof: ( ⇒ ) trivial ( ⇐ ) p o θ = p o ( τ o σ …) = ψ k,l ( τ ) o p o ( σ …) = ( ψ k,l ( τ ) o ψ k,l ( σ ) …) o p = ψ k,l ( θ ) o p (because ψ k,l is an homomorphism) Cassis Cassis -- Mar 2004 -- Mar 2004 14 14
A semi semi- -correct correct procedure procedure for ST for ST A ψ k τ ), ψ k σ ) , ψ ( τ ), ψ ( σ In: program program p , finite finite domain domain D ) In: p , D , ,l ( ,l ( k,l k,l that ψ ψ k Out: a a symmetry symmetry violation or a violation or a proof proof that holds over over D Out: ,l holds D k,l ≠ ∅ ∅ ) ( D ≠ while ( D while ) pick up up x in D and D := D pick x in D and D := D \ \ {x} {x} τ .x σ .x := p( τ := p( σ let r:= p(x), let r:= p(x), r r τ .x), ), r r σ .x) ) τ := p( σ := p( ≠ ψ ψ k τ ) . τ ≠ ( τ if ( ) . r) then return violation ( if ,l ( r) then return violation ( r r τ (x,r x,r, ,r r τ ) τ ) k,l ≠ ψ ψ k σ ) . σ ≠ ( σ if ( ) . r) then return violation violation (x,r if ,l ( r) then return ( r r σ (x,r, ,r r σ ) σ ) k,l return ( return (« «Q.E.D. Q.E.D.» ») ) Cassis Cassis -- Mar 2004 -- Mar 2004 15 15
Limitations of of Symmetric Symmetric Testing Testing Limitations Terminaison not not guaranteed guaranteed, but , but Terminaison # comparison comparison checks checks is is O(d) in place O(d) in place of of O(k! . d) O(k! . d) # where d = # test data d = # test data where Impossible to know know which which inputs inputs among among Impossible to τ .x σ .x x, τ , σ .x is is responsible responsible of of the the symmetry symmetry violation violation .x, x, Incorrect versions of of p p may may be be symmetric symmetric too too ! ! Incorrect versions But, But, No oracle is is required required, ST , ST is is fully fully automatic automatic No oracle Cassis Cassis -- Mar 2004 -- Mar 2004 16 16
Outline Outline Motivations Motivations A Generalized Generalized definition definition of of symmetry symmetry relation relation A Symmetric Testing Testing Symmetric First experimental experimental results results First Related and and further further works works Related Cassis Cassis -- Mar 2004 -- Mar 2004 17 17
Recommend
More recommend
