1 University of Geneva, Switzerland 2 CERN, Geneva, Switzerland Automated Test Case Generation for CTRL using Pex: Lessons Learned David PY Lawrence 1 Stefan Klikovits 1 , 2 Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1 s.k@cern.ch
What are we doing? • 1 MLOC code • no automated unit testing until two years ago • frequent changes in execution environment • (mostly) manual verification • big expenses (time) on QA side Stefan Klikovits 2 s.k@cern.ch
What are we doing? • 1 MLOC code • no automated unit testing until two years ago • frequent changes in execution environment • (mostly) manual verification • big expenses (time) on QA side test cases code Stefan Klikovits 2 s.k@cern.ch
How are we doing it? TI generator CTRL code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. test cases translator ITEC workflow Stefan Klikovits 3 s.k@cern.ch
How are we doing it? TI generator CTRL code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. test cases translator ITEC workflow Considering Execution Environment Resilience: A White-Box Approach Klikovits et. al. SERENE 2015 Stefan Klikovits 3 s.k@cern.ch
Recap semi-purification • replace dependencies with parameters 1 f ( x ){ 1 f_sp ( x , a ,b){ 2 GLOBAL_VAR : 2 a : i f i f 3 return dbGet(x) 3 return b 4 e l s e : 4 e l s e : 5 return − 1 5 return − 1 6 } 6 } A non-pure function Semi-purified f(x) 1 test_f_sp (){ 2 x = f ( " t e s t " , True , 5 ) // act 3 a s s e r t ( x == 5) // a s s e r t 4 } Test case Stefan Klikovits 4 s.k@cern.ch
From Pex to test cases Test case // Mock( func ) r e t u r n (. . . ) Test Input Set void test_case_1 (){ param1 = . . . SP parameter vals mock spec param2 = . . . parameter vals Pex call to CUT r e s = CUT( param1 , param2 ) observation vals asserts a s s e r t ( " check " , r e s == . . . ) } Test case generation from Pex output Stefan Klikovits 5 s.k@cern.ch
Test case generation: results 166 Unsupported C# Code CTRL Functions SP Functions Translation Semi-purification 602 1111 786 159 184 SP Err Translation Err ATCG (Pex) 294 1184 TCGen Err mismatching oracles Test Inputs Test Cases 2465 TCGen 3678 3972 matching oracles 29 Exec Error Stefan Klikovits 6 s.k@cern.ch
Test case generation: update ✟ 249 ✟ 166 Unsupported CTRL Functions C# Code SP Functions Translation Semi-purification 1191 ✘✘ ✟ ✟ 676 ✟ 875 ✟ 602 1111 786 ✟ 67 ✟ ✟ 199 ✟ 159 184 SP Err ATCG (Pex) Translation Err 489 ✘✘ 1184 ✟ 493 ✟ 294 mismatching oracles TCGen Err 2613 ✘✘ Test Inputs 2464 Test Cases TCGen 3610 ✘✘ 3117 ✘✘ 3972 matching oracles 3678 15 ✚ ✚ 29 Exec Error Stefan Klikovits 7 s.k@cern.ch
Number of test cases 295 231 200 # Functions 150 100 87 44 50 19 0 1 - 3 4 - 7 8 - 15 > 15 # Test cases Stefan Klikovits 8 s.k@cern.ch
Coverages 230 200 150 # Functions 129 118 110 100 50 15 100% 99% - 75% 74% - 50% 49% - 1% 0% Coverage Stefan Klikovits 9 s.k@cern.ch
Coverages: updated 406 # Functions 230 200 150 129 118 110 88 100 70 58 54 50 15 100% 99% - 75% 74% - 50% 49% - 1% 0% Coverage Stefan Klikovits 9 s.k@cern.ch
Coverages: matching oracles 406 377 # Functions 200 150 90 88 100 83 74 70 58 54 52 50 100% 99% - 75% 74% - 50% 49% - 1% 0% Coverage Stefan Klikovits 9 s.k@cern.ch
Lessons learned • not everything can be translated (easily) Stefan Klikovits 10 s.k@cern.ch
Lessons learned • not everything can be translated (easily) • not all features should be supported Stefan Klikovits 10 s.k@cern.ch
Lessons learned • not everything can be translated (easily) • not all features should be supported • C# is no silver bullet Stefan Klikovits 10 s.k@cern.ch
Lessons learned • not everything can be translated (easily) • not all features should be supported • C# is no silver bullet • improving the quality of test cases ? Stefan Klikovits 10 s.k@cern.ch
Lessons learned • not everything can be translated (easily) • not all features should be supported • C# is no silver bullet • improving the quality of test cases ? • tools have “features” Stefan Klikovits 10 s.k@cern.ch
How are we doing it? TI generator CTRL code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs CTRL test test input driver test gen. translator test cases ITEC workflow Stefan Klikovits 11 s.k@cern.ch
How are we doing it? TI generator CTRL code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs CTRL test test input driver test gen. translator test cases ITEC workflow Stefan Klikovits 11 s.k@cern.ch
Why test the translation? • DySyEx: execute code, cover max. paths • small differences – big impacts Stefan Klikovits 12 s.k@cern.ch
Why test the translation? • DySyEx: execute code, cover max. paths • small differences – big impacts http://samcnitt.tumblr.com/ Stefan Klikovits 12 s.k@cern.ch
Why not test the translator? • No formal semantics? • Changing language? • Effort to adapt the solution? Stefan Klikovits 13 s.k@cern.ch
Why not test the translator? • No formal semantics? • Changing language? • Effort to adapt the solution? http://asterix.wikia.com/wiki/Asterix_and_Cleopatra Stefan Klikovits 13 s.k@cern.ch
How to test? Stefan Klikovits 14 s.k@cern.ch
How to test? Divide http://chapleau.us/Img/caesar_asterix.gif Stefan Klikovits 14 s.k@cern.ch
How to test? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Stefan Klikovits 14 s.k@cern.ch
How to test? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Analyse Blocks https://en.gamigo.com/game/asterix Stefan Klikovits 14 s.k@cern.ch
How to test? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 14 s.k@cern.ch
How to test? 1 func ( i n t a , b ) { i n t i n t 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 } Anonymise Divide https://www.youtube.com/watch?v=UF6E-4G4n_M Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 14 s.k@cern.ch
How to test? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 return a % b 6 return i n t % i n t 7 } e l s e { 7 } e l s e { 8 return a + b 8 return i n t + i n t 9 } 9 } 10 } 10 } Divide Anonymise Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 14 s.k@cern.ch
How to test? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 return a % b 6 return i n t % i n t 7 } e l s e { 7 } e l s e { 8 return a + b 8 return i n t + i n t 9 } 9 } 10 } 10 } Divide Anonymise 1 func ( i n t a , b ) { i n t i n t 2 i n t ++ 1 3 i n t ++ 1 4 i n t = i n t + i n t 1 5 i f ( i n t > i n t ){ 0 6 i n t % i n t return 0 7 } { e l s e 8 i n t + i n t return 1 9 } 10 } Conquer Analyse Blocks https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 14 s.k@cern.ch
How to test? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 a % b 6 i n t % i n t return return 7 } { 7 } { e l s e e l s e 8 a + b 8 i n t + i n t return return 9 } 9 } 10 } 10 } Divide Anonymise 1 i n t func ( i n t a , i n t b ) { 2 i n t ++ 1 � φ ( L i ) 3 i n t ++ 1 4 i n t = i n t + i n t 1 φ = 5 i f ( i n t > i n t ){ 0 | L | 6 return i n t % i n t 0 7 } e l s e { 8 return i n t + i n t 1 9 } 10 } Conquer Analyse Blocks Stefan Klikovits 14 s.k@cern.ch
Anonymisation & basic blocks • Equivalence classes • exhaustive testing of features • confidence in basic blocks • calculate confidence for CUT Stefan Klikovits 15 s.k@cern.ch
The quality metric For translated source code n φ ( anon i ) � i =1 conf ( c ) = , anon i ∈ { Anon } n Stefan Klikovits 16 s.k@cern.ch
Recommend
More recommend
![Real-Time Modeling and Test Case Generation Konrad Krentz A Convincing Safety Case [1] 2 A](https://c.sambuz.com/1037815/real-time-modeling-and-test-case-generation-s.jpg)
