Construction Matt Staats KAIST Shin Hong, Moonzoo Kim, Gregory - PowerPoint PPT Presentation

Supporting Test Oracle Construction Matt Staats KAIST Shin Hong, Moonzoo Kim, Gregory Gay, Mats Heimdahl Gregg Rothermel University of Minnesota Twin-Cities KAIST / University of Nebraska-Lincoln 1

The Testing Problem Test Oracle Test Oracle Check behavior Check behavior ? ? ? Program Yes/No Program Program Program Program Program Program Program Program Behavior Behavior Behavior Executed over Execute over Executed over Executed over Talented Developers? Talented Developers? Talented Developers? Talented Developers? Talented Developers? Talented Developers? Test Input Test Input Test Input Test Input 2

The Automated Testing Problem Test Oracle Test Oracle Program Program Executed over Generate from Executed over Talented Developers? Talented Developers? Test Input Test Input 3

The Automated Testing Problem Program Problems Test Input Generation Tools Test Input Generation Tools Symbolic Execution Concolic Execution Symbolic Execution Concolic Execution Test Inputs Symbolic Java DART More Work CUTE Symbolic Java Finder DART CUTE Finder ? ? jCUTE SAGE ? ? jCUTE SAGE ? ? ? ? KLEE KLEE SCORE SCORE Random Testing Genetic Algorithm- Based Testing Random Testing Genetic Algorithm- Based Testing Randoop Testful Randoop Testful Adaptive Random AUSTIN Testing Adaptive Random AUSTIN Testing Naïve Random Iguana Testing Naïve Random Iguana Testing What now? What now? • Huge numbers of tools for this! 4

The Automated Testing Problem , n o i t a t n e s e r ? p s e e r k . Goal: get developer to state what the t a d n t r e s i r i e m e w f f d a i D n d i f n o i k t program should do (but correctly this time) ? s r e i r e s i s d i h a r T e a h r O Must check result! Sameish Test Oracle Test Oracle Test Oracle Test Oracle Fix Generate from? Generate from? Generate from? Program Program Program Program Generate from Generate from Generate from Executed over Executed over Executed over Generate from Executed over Talented Developers? Talented Developers? Talented Developers? Talented Developers? Test Input Test Input Test Input Test Input 5

Two Big Research Questions • What types of test oracles can developers easily/correctly understand and build? – What tasks are people actually good at? – What should be trying to deliver? • How can we help construct such test oracles? – Techniques, algorithms, tools, etc. – Empirical studies (with users, necessarily) – Industrial case studies 6

Automatic Invariant Generation • Idea: automatically generate Program invariants from the system • User then (necessarily) Automated Property / Behavior Extraction evaluates result • Remaining invariants represent Program Behavior test oracle • Several approaches, varying result User • Daikon , AutoInfer , Xie/Notkin work Reject as Accept as correct incorrect 7

Automatic Invariant Generation • Unclear how effective users Program are at classifying results – Problems if poor Randoop (Test Input Generation) + Daikon – Little evidence in favor of use • Study: Daikon dynamic Program + Program invariant generator Invariants – 2 case studies, approx. 30 students total – 3 programs – Each student classified an User invariant as true, false, or Correct Incorrect unknown (unclassified) Invariants Invariants 8

Automatic Invariant Generation KAIST User Study Classification Results Our classification User classification 9

Automatic Invariant Generation • Questions: – Why does this occur? – Impact of this on actual testing process? • Answers: – Why? Not really sure – Impact? No idea at all (but we guess negative) To be presented at ISSTA 2012 Coauthors: Shin Hong, Moonzoo Kim, Gregg Rothermel 10

Test Oracle Generation Support • Uncomfortable with complete automation for oracles – Evidence is suspect – Requires change in user behavior • As an alternative to complete construction, we thought we could support users in making oracles • Select oracle data : part of system oracle defined over • User still has to define oracle Program Test Input Generation Tools Oracle Symbolic Execution Concolic Execution Test Inputs Symbolic Java DART CUTE Finder jCUTE SAGE KLEE Oracle Data SCORE Selection Random Testing Genetic Algorithm- Based Testing Randoop Testful Test Inputs + Adaptive Random Oracle Data AUSTIN Testing Naïve Random Iguana Testing Must evaluate test results 11

Test Oracle Generation Support • Mutation testing was used to determine where and when we can detect changes • Result is that for a set of test inputs, person has a list of useful variables – Note: domain is critical avionics, so problems of heap, etc. go away • Goal: do better than other methods of selecting oracle data Variable Measure Program Mutants Effectiveness Generate Variable Effectiveness Ranking Run Against Oracle Data Specifies expected value Test Inputs (Generated Expected Value Tester Externally) Test Oracle 12

Test Oracle Generation Support To be presented at ICSE 2012 Latctl Person Coauthors: Gregory Gay, Mats Heimdahl DWM_2 DWM_1 13

Questions 14

Future Work Program: Gatekeeper Input: “ Keymaster ” Expected Value: Automatically extract test oracle “ Zuel ” 15