Acceptance Test Optimization Optimization Mohamed Mussa, Ferhat Khendek SAM’2014
Outline Outline • Background • Problem Statement • Overall Approach – Integration test cases selection – Integration test cases selection – Comparing test models • Conclusion 2
Background Test process consists of several phases Acceptance Acceptance Integration Integration Unit Unit Testing Testing Testing Test Test Test Test Test Test Test Test Test Test Test Test Test Test Models Models Model Model Models Models Model Model Model Model Model Model Model Model �
Background - Model Based Testing Development Process Development Process Model CODE Execution Execution Test Test Test Model CODE Testing Process Testing Process �
Background - Model Based Testing • Different approaches for different test phases – Unit, integration, acceptance • Different notations/languages – Different subsets of the same language – Different subsets of the same language • Many test models are produced during different phases – Redundancy – Test generation/planning: no reusability – Test Execution: no optimization 5
Background - Model Based Test Framework • Goals – Provide a systematic transition between the test phases • Framework Framework – Strengthen the collaboration between the development and the testing teams – Well know standards & reuse – Improve the test process • Enable reusability & optimization
Background - Model Based Test Framework Unit Unit Integration Integration Acceptance Acceptance Unit test Integration test model cn model int(n-1) Optimized acceptance Integration test test model Unit test model int1 model c1 Acceptance test model �
Problem Statement • Test cases may be exercised several times across the testing phases – Integration vs. Acceptance • Goal: remove redundant acceptance test cases – Reduce test execution time Reduce test execution time �
Problem Statement • Obvious solution – Compare integration test cases and acceptance test cases • Problem – Some integration test cases may include stubs for subsequent system components system components – Cannot be substituted to acceptance test cases Test Test Test Test Model Model 1 1 Model 2 Model Comp 1 2 Test Test SbSys 1 SbSys 2 Model Model n Test Test Comp 2 n- -1 1 Model Model n Comp 3 n Comp 4 SbSys n-1 System Comp n �
Overall Approach • Integration test cases selection • Compare integration test cases to • Compare integration test cases to acceptance test cases 10
Integration test cases selection • Test cases of last integration round are applied on complete system • Compare the behavior of test stubs of each test case to the behavior of CUTs of test cases of subsequent integration rounds • No additional information beside the test models • No additional information beside the test models Test Test Model 1 Model 1 Test Test Model 2 Model Comp 1 2 Test Test SbSys 1 Model Model n Test Test SbSys 2 Comp 2 n- -1 1 Model Model n n Comp 3 Comp 4 SbSys n-1 System Comp n ��
Integration test cases selection Test stubs can be specified explicitly, or «TestContext» «SUT» «SUT» «TestComponent» TC k SbSys k CUT k+1 stub k m1 m2 m3 m4 m5 m5 m6 m6 specified implicitly «TestContext» «SUT» «SUT» TC k SbSys k CUT k+1 m1 m2 m3 m4 m5 m6 12
Integration test cases selection • Event based comparison • Not instance based comparison – Instances are different • Not event name based – but message – event types: message, time, miscellaneous 13
Integration test cases selection Selection condition – L et • T k = {I k , E k , R k } be an integration test case at integration round k , • T i = {I i , E i , R i } be an integration test case at integration round i , i i i i • i > k – T k does not use a test stub for the CUT of T i if and only if ( ) ( ) ( ( ) ( ) ) . ∀ e , e ⋅ e ∈ E , e ∈ E | e ≠ e ∨ e = e ∧ e . owner . st ≠ SUT i k i i k k i k i k i 14
Integration test cases selection Comparing integration test cases Round k+1 Subsequent rounds «TestContext» «SUT» «SUT» TC m TC m SbSys m SbSys m CUT m+1 CUT m+1 «TestContext» «SUT» «SUT» s1 s2 TC k SbSys k CUT k+1 s3 m1 s4 m2 m3 m4 «TestContext» «SUT» «SUT» m5 TC n SbSys n CUT n+1 m6 m1 m3 m4 m6 ��
Integration test cases selection Test cases, which their stubs do not match match with with subsequent subsequent CUTs, CUTs, are are compared to acceptance test cases 16
Comparing Test Models Comparing Test Models • A lot of work has been done – Compared models are evolved from the same source – Two-Way vs. Three-Way – Look up for differences (Add/Delete/Modify) – Look up for differences (Add/Delete/Modify) – Structure vs. Behavior • Our case – Models did not necessary evolve from the same source 17
Comparing Test Models Comparing Test Models • Comparing MSCs or Sequence Diagrams is not straightforward – Several researchers have tackled this issue • But this is not difficult for test cases – Finite behaviors – Finite behaviors 18
Comparing Test Models Comparing Test Models • A test case T is a tuple (I, E, R), where – I : a set of instances – E : a set of events – R ⊆ (E x E): a partial order reflecting the transitive closure of the order relation between events on the same axis and the sending and reception events of the same message the sending and reception events of the same message • Test case inclusion – T acc = {I a , E a , R a } and T int = {I i , E i , R i } – T acc is included in T int iff • E a ⊆ E i • R a ⊆ R i 19
Comparing Test Models Comparing Test Models Comparing test cases Acceptance test case Integration test cases «TestContext» «SUT» «SUT» TC m TC m SbSys m SbSys m CUT m+1 CUT m+1 «TestContext» «SUT» s1 s2 TC a Sys s3 s4 m1 m6 «TestContext» «SUT» «SUT» TC n SbSys n CUT n+1 m1 m3 m4 m6 ��
Conclusion • We proposed an optimization approach that reduces the acceptance test suite length – already done at integration phase • Implemented and completed the framework • Implemented and completed the framework • What kind of systems would benefit ? • Requires evaluation of the gain 21
Recommend
More recommend
