Haven’t We Solved This Problem Yet? Automated Test Data Generation for Coverage: Lakhotia et al. Outline Haven’t We Solved This Problem Yet? Motivation Background Search Based Kiran Lakhotia 1 Phil McMinn 2 Mark Harman 1 Testing Concolic Testing Pointers in CUTE and AUSTIN 1 CREST Empirical Department of Computer Science Study King’s College London Experimental Setup Test Subjects 2 Department of Computer Science Results University of Sheffield Answers Summary CREST, London, 2009
Haven’t We Solved This Motivation 1 Problem Yet? Lakhotia et al. Background on AUSTIN and CUTE 2 Outline Search Based Testing Motivation Concolic Testing Background Pointers in CUTE and AUSTIN Search Based Testing Concolic Testing Pointers in Empirical Study CUTE and 3 AUSTIN Experimental Setup Empirical Study Test Subjects Experimental Setup Results Test Subjects Results Answers To Research Questions Answers Summary Summary 4
Motivation and Research Questions Haven’t We Solved This Problem Yet? Lately two very different automated test data generation Lakhotia et al. techniques have gained a lot of attention: Outline Search based testing ( e.g. EvoTest www.evotest.eu, 3 year Motivation project worth $4m) Background Search Based Dynamic symbolic execution ( e.g. Pex, CUTE, CREST) Testing Concolic Testing Pointers in But, CUTE and AUSTIN Empirical How effective are concolic and search based test data Study generation for real world programs? Experimental Setup Test Subjects What is the relative efficiency of both approaches? Results Answers Which types of program structure does each technique fail Summary to cover?
Search Based Structural Testing Haven’t We Solved This We transform the testing problem into an optimization Problem Yet? problem; the assumption is that comparing two potential Lakhotia et al. solutions is easier than generating a correct solution from Outline scratch. Motivation Background Example Candidate Solutions Search Based Testing Concolic Testing void testme(int a, int b) Pointers in CUTE and { AUSTIN a = 0, b = 10 random start if( a * b > 100 ) Empirical //target Study Experimental } Setup Test Subjects Branch Distance Function: Results Answers if( 100 - (a * b) < 0 ) Summary return 0; else return ( 100 - (a * b) ) + constant;
Search Based Structural Testing Haven’t We Solved This We transform the testing problem into an optimization Problem Yet? problem; the assumption is that comparing two potential Lakhotia et al. solutions is easier than generating a correct solution from Outline scratch. Motivation Background Example Candidate Solutions Search Based Testing Concolic Testing void testme(int a, int b) Pointers in CUTE and { AUSTIN a = 0, b = 10 random start if( a * b > 100 ) Empirical a = -10, b = 30 //target Study Experimental } Setup Test Subjects Branch Distance Function: Results Answers if( 100 - (a * b) < 0 ) Summary return 0; else return ( 100 - (a * b) ) + constant;
Search Based Structural Testing Haven’t We Solved This We transform the testing problem into an optimization Problem Yet? problem; the assumption is that comparing two potential Lakhotia et al. solutions is easier than generating a correct solution from Outline scratch. Motivation Background Example Candidate Solutions Search Based Testing Concolic Testing void testme(int a, int b) Pointers in a = 0, b = 10 random start CUTE and { AUSTIN a = -10, b = 30 if( a * b > 100 ) Empirical //target Study a = 5, b = 9 Experimental } Setup Test Subjects Branch Distance Function: Results Answers if( 100 - (a * b) < 0 ) Summary return 0; else return ( 100 - (a * b) ) + constant;
Search Based Structural Testing Haven’t We In search based testing we are not concerned about which path Solved This Problem Yet? leads to the target. The path taken up to a target is an Lakhotia et al. emergent property of the search process. Outline Motivation void testme(int a, int b, int c) Background { Search Based Testing if(a == 3) Concolic Testing Pointers in { CUTE and AUSTIN ... Empirical } Study Experimental Setup if(b == 2) Test Subjects Results if(c == 1) Answers //target Summary }
Search Based Structural Testing Haven’t We In search based testing we are not concerned about which path Solved This Problem Yet? leads to the target. The path taken up to a target is an Lakhotia et al. emergent property of the search process. Outline Motivation void testme(int a, int b, int c) Background { Search Based Testing if(a == 3) Concolic Testing Pointers in { CUTE and AUSTIN ... Empirical } Study Experimental Setup if(b == 2) Test Subjects Results if(c == 1) Answers //target Summary }
Search Based Structural Testing Haven’t We In search based testing we are not concerned about which path Solved This Problem Yet? leads to the target. The path taken up to a target is an Lakhotia et al. emergent property of the search process. Outline Motivation void testme(int a, int b, int c) Background { Search Based Testing if(a == 3) Concolic Testing Pointers in { CUTE and AUSTIN ... Empirical } Study Experimental Setup if(b == 2) Test Subjects Results if(c == 1) Answers //target Summary }
Search Based Structural Testing Haven’t We In search based testing we are not concerned about which path Solved This Problem Yet? leads to the target. The path taken up to a target is an Lakhotia et al. emergent property of the search process. Outline Motivation void testme(int a, int b, int c) Background { Search Based if(a == 3) Testing Concolic Testing { Pointers in CUTE and ... AUSTIN } Approach Level: 1 Empirical Study Experimental if(b == 2) Setup Approach Level: 0 Test Subjects if(c == 1) Results Answers //target Summary }
Search Based Structural Testing In search based testing we are not concerned about which path Haven’t We Solved This leads to the target. The path taken up to a target is an Problem Yet? Lakhotia et al. emergent property of the search process. Outline Motivation void testme(int a, int b, int c) Background { Search Based if(a == 3) Testing Concolic Testing { Pointers in CUTE and ... AUSTIN } Approach Level: 1 Empirical Study Experimental if(b == 2) Approach Level: 0 Setup Test Subjects if(c == 1) Results Answers //target Summary } Fitness Function: Approach Level + normalized( Branch Distance )
AUSTIN Haven’t We Solved This Problem Yet? Lakhotia et al. AUSTIN: Cover all goals ( i.e. edges in a control flow graph) Outline Execute program with a concrete input vector Motivation Background Symbolically execute concrete path taken by said input in Search Based parallel Testing Concolic Testing Pointers in When execution follows an infeasible path w.r.t. current CUTE and AUSTIN goal Empirical Study Apply a hill climb algorithm to solve arithmetic constraints Experimental Apply custom pointer rules to solve constraint over Setup Test Subjects memory locations Results Answers Summary
Concolic Testing Haven’t We Solved This Problem Yet? Lakhotia et al. The term concolic stands for conc rete execution with symb olic Outline execution. It is similar to dynamic symbolic execution: Motivation Explore all feasible execution paths Background Search Based Testing Execute program with a concrete input vector Concolic Testing Pointers in CUTE and Symbolically execute concrete path taken by said input in AUSTIN parallel Empirical Study Use concrete values to simplify symbolic expressions when Experimental Setup Test Subjects necessary ( i.e. regard them as constant values) Results Answers Summary
Concolic Testing Haven’t We Solved This Problem Yet? Lakhotia et al. Suppose you want to find input values for a and b to traverse Outline the path to the target. Motivation Background Example Concrete Values Path Condition Search Based Testing Concolic Testing void testme(int a, int b) a = -10, b = 50 Pointers in CUTE and { AUSTIN ( a 0 ∗ b 0 ) ≤ 100 if( a * b > 100 ) Empirical //target Study } Experimental Setup Test Subjects Results Answers Summary
Concolic Testing Haven’t We Solved This Problem Yet? Lakhotia et al. Suppose you want to find input values for a and b to traverse Outline the path to the target. Motivation Background Example Concrete Values Path Condition Search Based Testing Concolic Testing void testme(int a, int b) a = -10, b = 50 Pointers in CUTE and { AUSTIN ( a 0 ∗ b 0 ) > 100 if( a * b > 100 ) Empirical //target Study } Experimental Setup Test Subjects Results Answers Summary
Concolic Testing Haven’t We Solved This Problem Yet? Lakhotia et al. Outline But your constraint solver cannot handle non–linear constraints. Motivation Example Concrete Values Path Condition Background Search Based Testing void testme(int a, int b) a = -10, b = 50 Concolic Testing { Pointers in CUTE and if( a * b > 100 ) ( a 0 ∗ b 0 ) > 100 AUSTIN //target Empirical } Study Experimental Setup Test Subjects Results Answers Summary
Recommend
More recommend
