An Integrated Approach to Assertion-Based Random Testing an 1,4 e F. Morales 1 Ignacio De Casso San Rom´ Jos´ ıa 1,2 Manuel V. Hermenegildo 1,4 Pedro L´ opez Garc´ 1 IMDEA Software Institute, 2 Spanish Research Council (CSIC), and 4 Technical University of Madrid (UPM) MFoC – November 26, 2019 De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 1 / 19
Introduction Introduction Objective Random testing based on assertion properties. ◮ Integrated in an assertion-based verification and debugging framework. Combined with static analysis. Unified assertion language. ◮ Mostly automatic. Language agnostic: ◮ Translate programs into an IR: (Constraint) Horn Clauses. ◮ Perform analysis and assertion-based testing on that IR. Tools The Ciao System: ◮ Built up from a logic-based (CHC) kernel. ◮ Rich assertion language and analysis/testing framework. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 2 / 19
Horn Clauses as IR Intermediate Repr.: (Constraint) Horn Clauses Analysis / Testing Transformation Java Source Java parser Domains javac Shapes/sizes Sharing Assertions Java Bytecode ... Resources CHA soot + Ciao transform. Ciao Source IR − CFG (Horn clauses) Analysis Testing ISA / LLVM / ... xcc / clang / Ciao XC Source Transformation: ◮ Source: Program P in L P + Semantics of L P ◮ Target: A (C) Horn Clause program capturing [[ P ]]. Block-based CFG. Each block represented as a Horn clause . Allows supporting multiple languages. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 3 / 19
Horn Clauses as IR Transformation Example - Source Adding numbers from 1 to n source public static int r(int n) { int ans=0; while(n>0) { ans+=n; n-=1; } return ans; } De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 4 / 19
Horn Clauses as IR Transformation Example - Source Adding numbers from 1 to n ✞ source Horn clause IR public static int r(int n) :- pred r/2 : num * var. { int ans=0; r(N,Ret) :- while(n>0) Ans=0, { r_1(N, Ans, Ret). ans+=n; n-=1; r_1(N, Ans, Ret) :- } N>0, return ans; add(Ans, N, Ans1), } N1 is N - 1, r_1(N1, Ans1, Ret). r_1(N, Ans, Ret):- N=<0, Ret=Ans. ✝ ✆ De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 4 / 19
Horn Clauses as IR Example: XC source, ISA (left), and IR (right) int fact( int N) { if (N <= 0) return 1; return N * fact(N - 1); } 1 <fact >: 1 fact(R0,R0_3):- 2 0x01: entsp 0x2 2 entsp(0x2), 3 0x02: stw r0, sp[0x1] 3 stw(R0,Sp0x1), 4 0x03: ldw r1, sp[0x1] 4 ldw(R1,Sp0x1), 5 0x04: ldc r0, 0x0 5 ldc(R0_1,b0x0), 6 0x05: lss r0, r0, r1 6 lss(R0_2,bR0_1,R1), 7 0x06: bf r0, <0x08> 7a bf(R0_2 ,0x8), 7b fact_aux(R0_2,Sp0x1,R0_3,R1_1). 9 fact_aux(1,Sp0x1,R0_4,R1):- 10 0x07: bu <0x10> 10 bu(0x10), 11 0x0a: ldw r0, sp[0x1] 11 ldw(R0_1,Sp0x1), 12 0x0b: sub r0, r0, 0x1 12 sub(R0_2,R0_1 ,0x1), 13 0x0c: bl <fact> 13a bl(fact), 13b fact(R0_2,R0_3), 15 15 0x0d: ldw r1, sp[0x1] ldw(R1,Sp0x1), 16 16 0x0e: mul r0, r1, r0 mul(R0_4,R1,R0_3), 17 17 0x0f: retsp 0x2 retsp(0x2). 18 fact_aux(0,Sp0x1,R0,R1):- 19 19 0x08: mkmsk r0, 0x1 mkmsk(R0,0x1), 20 20 0x09: retsp 0x2 retsp(0x2). De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 5 / 19
The Ciao Model The Ciao Assertion Language (subset) :- pred Pred [ :Precond ] [= > Postcond ] [ + Comp-formula ] . Each typically a “mode” of use; the set covers the valid calls . :- pred quicksort ( X , Y ) : ( list ( X , int ), var ( Y )) = > sorted l ( Y ) + ( is det , not fails ) . :- pred quicksort ( X , Y ) : ( var ( X ), list ( Y , int )) = > ground ( X ) + non det. Properties; from libraries or user defined (in the source language): :- regtype list ( X , L ) . :- prop sorted l ( L ) . list ( , [] ). sorted l ( [] ) . sorted l ( [ ] ) . list ( X ,[ H | T ]) :- X ( H ), list ( X , T ). sorted l ([ X , Y | Z ]) :- X > Y , sorted l ([ Y | Z ]) . Types/shapes, cost, data sizes, aliasing, termination, determinacy, non-failure, ... Program-point Assertions: Inlined with code: ..., check ( int(X), X>0 ), Z is Y/X, ... . Assertions optional, can be added at any time. Provide partial spec. Part of the programming language and “runnable.” Used everywhere: itf with analyzer, doc gen., foreign itf, debug ... De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 6 / 19
The Ciao Model Debugging in the Ciao Model possible run−time error Program Static Analysis RT Check P Analysis Info verification warning Unit Test [[P]] :− check compile−time error :− trust :− texec :− test verified I :− check Assertion Comparator Normalizer :− false & Lib Itf. (Incl. VCgen) Builtins/ certificate (optimized) + :− checked (ACC) code Libs PREPROCESSOR The Ciao Debugging Framework Abstract interpretation-based static analysis tries to verify assertions. ◮ Compile-time errors if violated, warnings if unable to verify. Run-time checks generated for parts of asserts not verified statically. ◮ Run-time errors (exceptions) if violated. User-defined unit tests. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 7 / 19
The Ciao Model Automatic test generation: Assertion-based Testing possible run−time error Program Static Analysis RT Check P Analysis Info verification warning Unit Test [[P]] :− check compile−time error :− trust :− texec :− test verified I :− check Assertion Comparator Normalizer :− false & Lib Itf. (Incl. VCgen) Builtins/ certificate (optimized) + :− checked (ACC) code Libs PREPROCESSOR Goal: random unit tests derived automatically from assertions. Components and ideas already present in Ciao model: ◮ Assertion language. ◮ Unit tests and runtime-checking framework. ◮ (Runnable) properties as generators → test cases from assrt. precondition. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 8 / 19
Test Case Generation Objective: Test case generation Objective: generating test cases from calls fields of assertions pred q(X 1 ,...,X n ) : (pre 1 (X 1 ,...,X n ),...,pre r (X 1 ,...,X n )) [=> Postcond(X 1 ,...,X n )] [+ Comp-formula] In principle, properties can be run as efficient generators. ◮ Using free variables as arguments, ◮ and a suitable resolution strategy. E.g.: ✞ list([]). list([_|X]) :- list(X). ✝ ✆ − → ?- list(X). X=[];X=[_];X=[_,_]; X=[_,_,_];... In practice, not that simple: ◮ Termination/efficiency trade-offs. ◮ Inappropriate test case distribution. We study several cases. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 9 / 19
Test Case Generation Generation for HC + Herbrand properties (structures) HC+Herbrand properties − → already generators if run ✞ ◮ e.g., ?- list(X) − → X=[];X=[_];X=[_,_];... list([]). list([_|Xs]) :- list(Xs). ✝ ✆ Depth-first resolution strategy not always well suited: ◮ Efficient, but termination problems. ◮ Skewed distribution of values. ◮ Incomplete ( list_peano(X) → [],[0],[0,0] ... but [s(0)] never generated). Solution: alternative search strategies. ◮ E.g., breadth-first, iterative deepening, etc. ◮ Already available in Ciao using ’packages’ mechanism. Our tool → added also randomized search strategy ◮ Clauses chosen at random. ✞ solve_goal(G) :- random_clause(G,Body), solve_body(Body). ... solve_body([G|Gs]) :- solve_goal(G), solve_body(Gs). ✝ ✆ ◮ Growth control, not repeated solutions, size parameter ... Already enough for regular types. De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 10 / 19
Test Case Generation Generation for HC + Other domains Proposal: Constraint-based random search Random search collecting constraints for each derivation ◮ E.g., X<Y interpreted as X.<.Y in CLP(Q). ◮ Solvers ensure constrains are satisfiable during all steps of generation → failure and backtracking otherwise. E.g., sorted_l([N,M|Ms]):-N =<M, sorted_l([M|Ms]). sorted_l([_]). sorted_l([]). ?- sorted_l(L) → ...; L = [X,Y,Z], X.=<.Y, Y.=<.Z;.... Last step: random sampling or enumeration of constraints. E.g., L = [X,Y,Z], X.=<.Y, Y.=<.Z;... � X=1, Y=4, Z=9 � L=[1,4,9] Supported domains: ◮ Arithmetic → CLP(Q) and CLP(FD) domains + random sampler: ◮ Modes and sharing → new constraints domain + random sampler: e.g., (list(X,var), list(Y,var), indep(X,Y)) � X=[A,B,A],Y=[C,D] ◮ Generate-and-check or user-defined generators for other domains. ◮ Random instantiation of unconstrained free variables. e.g., list(X) � X=[_,_,_,_] � X=[f(a),0,_,[1,2]] De Casso et al. (IMDEA Software/UPM/CSIC) Assertion-Based Random Testing MFoC – November 26, 2019 11 / 19
Recommend
More recommend
