1 University of Geneva, Switzerland 2 CERN, Geneva, Switzerland Towards Language Independent (Dynamic) Symbolic Execution Manuel Gonzalez-Berges 2 Stefan Klikovits 1 , 2 Didier Buchs 1 Stefan Klikovits 1 stefan.klikovits@unige.ch
What are we doing? • 1 MLOC code (Control) • no automated unit testing until three ago • frequent changes in execution environment • (mostly) manual verification • big expenses (time) on QA side Stefan Klikovits 2 stefan.klikovits@unige.ch
What are we doing? • 1 MLOC code (Control) • no automated unit testing until three ago • frequent changes in execution environment • (mostly) manual verification • big expenses (time) on QA side test cases code Stefan Klikovits 2 stefan.klikovits@unige.ch
Language Independent Test Case Generation https://thriftytraveller.files.wordpress.com/2013/11/asterix_obelix3.gif Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation https://thriftytraveller.files.wordpress.com/2013/11/asterix_obelix3.gif Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation 1. Develop generic tool Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation 1. Develop generic tool http://asterix.wikia.com/wiki/Asterix_and_Cleopatra Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation 1. Develop generic tool 2. Modify parser and execution Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation 1. Develop generic tool 2. Modify parser and execution https://www.efiliale.de/efiliale/images/aktionen/asterix/5624_Troubadix.png Stefan Klikovits 3 stefan.klikovits@unige.ch
Language Independent Test Case Generation 1. Develop generic tool 2. Modify parser and execution 3. Translate to existing tool language http://www.asterix.com/asterix-de-a-a-z/les-personnages/perso/a43b.gif Stefan Klikovits 3 stefan.klikovits@unige.ch
Semantics, semantics, semantics • small differences – big impacts http://samcnitt.tumblr.com/ Stefan Klikovits 4 stefan.klikovits@unige.ch
How do we generate TCs? TI generator CTRL code SP CTRL source code IDE SP engine SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. translator test cases ITEC workflow Stefan Klikovits 5 stefan.klikovits@unige.ch
How do we generate TCs? TI generator CTRL code SP CTRL source code IDE SP engine SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. translator test cases ITEC workflow Considering Execution Environment Resilience: A White-Box Approach Klikovits et. al. , Proc. SERENE 2015, Paris Stefan Klikovits 5 stefan.klikovits@unige.ch
Semi-purification • replace dependencies with parameters 1 f ( x ){ 1 f_sp ( x , a ,b){ 2 i f GLOBAL_VAR : 2 i f a : 3 return dbGet(x) 3 return b 4 e l s e : 4 e l s e : 5 return − 1 5 return − 1 6 } 6 } A non-pure function Semi-purified f(x) 1 test_f_sp (){ 2 x = f ( " t e s t " , True , 5 ) // act 3 a s s e r t ( x == 5) // a s s e r t 4 } Test case Stefan Klikovits 6 stefan.klikovits@unige.ch
CUT translation & TC Gen • Pex (Microsoft Research) • Dynamic Symbolic Execution • translate CUT, generate PUT • manually create Pex factories, data types, built-in functions https://sites.google.com/site/diedruidenmt/_/rsrc/1367838067499/miraculix/Miraculix.jpg Stefan Klikovits 7 stefan.klikovits@unige.ch
From Pex to test cases Test case // Mock( func ) r e t u r n (. . . ) Test Input Set void test_case_1 (){ param1 = . . . SP parameter vals mock spec param2 = . . . parameter vals Pex call to CUT r e s = CUT( param1 , param2 ) asserts observation vals a s s e r t ( " check " , r e s == . . . ) } Test case generation from Pex output Stefan Klikovits 8 stefan.klikovits@unige.ch
How are we doing it? CTRL TI generator code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. translator test cases ITEC workflow Stefan Klikovits 9 stefan.klikovits@unige.ch
How are we doing it? CTRL TI generator code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. translator test cases ITEC workflow Stefan Klikovits 9 stefan.klikovits@unige.ch
How are we doing it? CTRL TI generator code SP CTRL source code SP engine IDE SP tool code translator SP data results TC Gen test inputs CTRL test inputs test CTRL test input driver test gen. translator test cases ITEC workflow Automated Test Case Generation for CTRL using Pex: Lessons Learned Klikovits et. al. , Proc. SERENE 2016, Gothenburg Stefan Klikovits 9 stefan.klikovits@unige.ch
How to test translation? Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? Divide http://chapleau.us/Img/caesar_asterix.gif Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Analyse Blocks https://en.gamigo.com/game/asterix Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? Divide Anonymise http://chapleau.us/Img/caesar_asterix.gif https://www.youtube.com/watch?v=UF6E-4G4n_M Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? 1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 } Anonymise https://www.youtube.com/watch?v=UF6E-4G4n_M Divide Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 return a % b 6 return i n t % i n t 7 } e l s e { 7 } e l s e { 8 return a + b 8 return i n t + i n t 9 } 9 } 10 } 10 } Divide Anonymise Analyse Blocks Conquer https://en.gamigo.com/game/asterix https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 return a % b 6 return i n t % i n t 7 } e l s e { 7 } e l s e { 8 return a + b 8 return i n t + i n t 9 } 9 } 10 } 10 } Divide Anonymise 1 i n t func ( i n t a , i n t b ) { 2 i n t ++ 1 3 i n t ++ 1 4 i n t = i n t + i n t 1 5 i f ( i n t > i n t ){ 0 6 i n t % i n t return 0 7 } { e l s e 8 i n t + i n t return 1 9 } 10 } Conquer Analyse Blocks https://www.pinterest.com/pin/336784878358770673/ Stefan Klikovits 10 stefan.klikovits@unige.ch
How to test translation? 1 i n t func ( i n t a , i n t b ) { 1 i n t func ( int , i n t ){ 2 a++ 2 i n t ++ 3 a++ 3 i n t ++ 4 b = b+2 4 i n t = i n t + i n t 5 i f ( a > b ){ 5 i f ( i n t > i n t ) { 6 return a % b 6 return i n t % i n t 7 } e l s e { 7 } e l s e { 8 return a + b 8 return i n t + i n t 9 } 9 } 10 } 10 } Divide Anonymise 1 i n t func ( i n t a , i n t b ) { 2 i n t ++ 1 � φ ( L i ) 3 i n t ++ 1 4 i n t = i n t + i n t 1 5 i f ( i n t > i n t ){ 0 φ = 6 i n t % i n t return 0 | L | 7 } { e l s e 8 i n t + i n t return 1 9 } 10 } Analyse Blocks Conquer Stefan Klikovits 10 stefan.klikovits@unige.ch
Test case generation: results 499 Unsupported Feature CTRL Functions SP Functions C# Code Translation Semi-purification 1521 977 791 45 186 SP Error Translation Error ATCG (Pex) 898 mismatching oracles 4138 Test Cases Test Inputs TCGen 5060 5060 matching oracles 24 Exec Error Stefan Klikovits 11 stefan.klikovits@unige.ch
Number of Test Cases # TCs per function (n = 791) 45 42.1% 40 35 32.9% 30 % Functions 25 20 17.1% 15 10 5.9% 5 1.3% 0.8% 0 0 1 - 3 4 - 7 8 - 14 15 - 30 > 30 Box plot distribution 10 0 10 20 30 40 50 60 70 80 90 100 110 120 Stefan Klikovits 12 stefan.klikovits@unige.ch
Coverage n = 791 76.0% 76.0% % Functions 9.9% 7.2% 5.8% 1.0% 0.1% no report 0% 1% - 49% 50% - 75% 75% - 99% 100% Coverage Stefan Klikovits 13 stefan.klikovits@unige.ch
Lessons learned • not everything can be translated (easily) • nor should it ... (?) • C# is no silver bullet • improving the quality of test cases ? • tools have “features” Stefan Klikovits 14 stefan.klikovits@unige.ch
Recommend
More recommend
