CUTE: A Concolic Unit Testing Engine for C (ACM SIGSOFT Impact - PowerPoint PPT Presentation

CUTE: A Concolic Unit Testing Engine for C (ACM SIGSOFT Impact Award 2019) Koushik Sen, UC Berkeley Darko Marinov, UIUC Gul Agha, UIUC

Programs Have Bugs 2

Why Program Testing? ü Programmer familiarity ü Concrete input for debugging ü No false positives ü Easy regression 3

Why Automated Testing? 4

Automated Testing Hits the Mainstream 5

Automated Testing Hits the Mainstream 6

Automated Testing Hits the Mainstream 7

Automated Testing Hits the Mainstream 8

Automated Test Generation Trend • 1976: King’76, Clarke’76, Howden’77 • 2000: Java PathFinder • 2001: Started my PhD UIUC • 2001: SLAM/Blast: Automatic predicate abstraction • 2001: Java PathExplorer: Runtime Verification • 2003: Runtime monitoring with Eagle (Internship) • 2003: Generalized Symbolic Execution

Automated Test Generation Trend • 1976: King’76, Clarke’76, Howden’77 • 2000: Java PathFinder • 2001: Started my PhD UIUC • 2001: SLAM/Blast: Automatic predicate abstraction • 2001: Java PathExplorer: Runtime Verification • 2003: Runtime monitoring with Eagle (Internship) • 2003: Generalized Symbolic Execution • 2005: DART: Directed Automated Random Testing (Internship) • 2005: CUTE: A Concolic Unit Testing Engine for C • 2006: jCUTE: Concolic Testing for Multi-threaded programs Symbolic JPF, KLEE, CREST, S 2 E, Angr, Veritesting, Mayhem, Triton, Jalangi, CATG

What is Concolic testing? • Combine concrete execution and symbolic execution Conc rete + Symb olic = Concolic 12

Goal • Automated Unit Testing of real-world C and Java Programs • Generate test inputs • Execute unit under test on generated test inputs • so that all reachable statements are executed • Any assertion violation gets caught 14

Goal • Automated Unit Testing of real-world C and Java Programs • Generate test inputs • Execute unit under test on generated test inputs • so that all reachable statements are executed • Any assertion violation gets caught • Concolic Testing Approach: • Explore all execution paths of an unit for all possible inputs 15

Computation Tree • Can be seen as a binary tree with possibly infinite depth • Computation tree 1 0 • Each node represents the execution of a “ if then else ” statement 1 0 1 0 • Each edge represents the execution of a sequence of non-conditional statements 0 1 1 • Each path in the tree represents an equivalence class of inputs 1 0 1 16

Concolic Testing Approach int double (int v) { return 2*v; n Random Test Driver: } q random values for x and y void testme (int x, int y) { n Probability of reaching z = double (y); ERROR is extremely low if (z == x) { if (x > y+10) { ERROR; } } } 17

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } state state condition void testme (int x, int y) { x = 22, y = 7 x = x 0 , y = y 0 z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 18

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { z = double (y); x = 22, y = 7, x = x 0 , y = y 0 , if (z == x) { z = 14 z = 2*y 0 if (x > y+10) { ERROR; } } } 19

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { z = double (y); if (z == x) { 2*y 0 != x 0 if (x > y+10) { ERROR; } } x = 22, y = 7, x = x 0 , y = y 0 , } z = 14 z = 2*y 0 20

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { Solve: 2*y 0 == x 0 z = double (y); Solution: x 0 = 2, y 0 = 1 if (z == x) { 2*y 0 != x 0 if (x > y+10) { ERROR; } } x = 22, y = 7, x = x 0 , y = y 0 , } z = 14 z = 2*y 0 21

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { x = 2, y = 1 x = x 0 , y = y 0 z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 22

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { z = double (y); x = 2, y = 1, x = x 0 , y = y 0 , if (z == x) { z = 2 z = 2*y 0 if (x > y+10) { ERROR; } } } 23

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { z = double (y); if (z == x) { 2*y 0 == x 0 x = 2, y = 1, x = x 0 , y = y 0 , if (x > y+10) { z = 2 z = 2*y 0 ERROR; } } } 24

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { z = double (y); if (z == x) { 2*y 0 == x 0 if (x > y+10) { x 0 ≤ y 0 +10 ERROR; } } } x = 2, y = 1, x = x 0 , y = y 0 , z = 2 z = 2*y 0 25

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { Solve: (2*y 0 == x 0 ) ∧ (x 0 > y 0 + 10) z = double (y); Solution: x 0 = 30, y 0 = 15 if (z == x) { 2*y 0 == x 0 if (x > y+10) { x 0 > y 0 +10 ERROR; } } } x = 2, y = 1, x = x 0 , y = y 0 , z = 2 z = 2*y 0 26

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { x = 30, y = 15 x = x 0 , y = y 0 z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 27

Concolic Testing Approach Concrete Symbolic int double (int v) { Execution Execution concrete symbolic path return 2*v; } condition state state void testme (int x, int y) { Program Error z = double (y); if (z == x) { 2*y 0 == x 0 if (x > y+10) { x 0 > y 0 +10 x = 30, y = 15 x = x 0 , y = y 0 ERROR; } } } 28

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 29

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 30

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 31

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 32

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 33

Explicit Path (not State) Model Checking n Traverse all execution paths one by one to detect T F errors q assertion violations T q program crash F T F q uncaught exceptions n combine with address F T T sanitizer to discover memory errors T F T 34

Novelty : Simultaneous Concrete and Symbolic Execution Concrete Symbolic int foo (int v) { Execution Execution concrete symbolic path return (v*v) % 50; } condition state state void testme (int x, int y) { x = 22, y = 7 x = x 0 , y = y 0 z = foo (y); if (z == x) { if (x > y+10) { ERROR; } } } 35

Novelty : Simultaneous Concrete and Symbolic Execution Concrete Symbolic int foo (int v) { Execution Execution concrete symbolic path return (v*v) % 50; } condition state state void testme (int x, int y) { Solve: (y 0 *y 0 )%50 == x 0 z = foo (y); Don ’ t know how to solve! Stuck? if (z == x) { (y 0 *y 0 )%50 !=x 0 if (x > y+10) { ERROR; } } x = 22, y = 7, x = x 0 , y = y 0 , } z = 49 z = (y 0 * y 0 )%50 36

Novelty : Simultaneous Concrete and Symbolic Execution Concrete Symbolic Execution Execution concrete symbolic path condition state state void testme (int x, int y) { Solve: foo (y 0 ) == x 0 z = foo (y); Don ’ t know how to solve! Stuck? if (z == x) { foo (y 0 ) !=x 0 if (x > y+10) { ERROR; } } x = 22, y = 7, x = x 0 , y = y 0 , } z = 49 z = foo (y 0 ) 37