1 Automated Test Oracles Automated Test Oracles for GUIs for GUIs Eighth International Symposium Eighth International Symposium on the Foundations of Software on the Foundations of Software Engineering, San Diego, CA, Engineering, San Diego, CA, Nov. 6- Nov. 6 -10, 2000. 10, 2000. A Test Case for WordPad A Test Case for WordPad 2 This is the text. SelectText SelectText Format Font 18 OK (“This”) (“text”) Underline Format Font OK This is the text. 1
What Is Correct Behavior What Is Correct Behavior 3 ?! This is the text. SelectText Format Font 18 OK This is the text. (“This”) SelectText Format Font Underline OK (“text”) This is the text. Check State, not only Output !! Research Focus Research Focus 4 • Goal • Goal – To check the To check the GUI’s state GUI’s state after each event after each event – • • Approaches Approaches – – Manual Manual – – Automated Automated Actual State Expected State Automated Verifier • • Challenges Challenges – Generating expected state Generating expected state – Verdict – – Extracting actual state Extracting actual state – Comparing expected & actual states – Comparing expected & actual states 2
Outline Outline 5 • Overview of GUI Oracle Overview of GUI Oracle • • Generating Expected State Generating Expected State • – – Modeling the GUI’s State Modeling the GUI’s State • Objects • Objects • Properties Properties • – – Modeling the Events Modeling the Events • Obtaining Actual GUI’s State Obtaining Actual GUI’s State • • Comparing Actual & Expected States Comparing Actual & Expected States • • Case Study: MS WordPad Case Study: MS WordPad • • Concluding Remarks Concluding Remarks • Overview of GUI Oracle Overview of GUI Oracle 6 Test Case Run-time Formal Expected-state information from GUI executing GUI Generator Model Expected State Execution Actual Verifier State Monitor Oracle Oracle Oracle Oracle Verdict 3
Modeling the GUI Modeling the GUI 7 A GUI consists of Objects Form Window State wsNormal Width 1088 AutoScroll TRUE Button Label Caption Cancel Align alNone Enabled TRUE Caption Files of type: Visible TRUE Color clBtnFace Height 65 Font (tFont) 8 All Properties of All Properties of Cancel Cancel 4
Determining Properties Determining Properties 9 • Manual Examination of GUI Manual Examination of GUI • • Specifications (Reduced Set) Specifications (Reduced Set) • – – GUI being tested GUI being tested • Toolkit/Language (Complete Set) Toolkit/Language (Complete Set) • – All available properties All available properties – Now we know how to represent the GUI’s state Modeling Events Modeling Events 10 • • Events are State Transducers Events are State Transducers State: S i This is the text. SelectText (“This”) Event: e This is the text. This State: S j Notation: S j = [S i , e] 5
Representing Events Representing Events 11 • We define an event as: We define an event as: • State j = [State i , event] • For example: For example: • State j = [State i , cut] • Need a compact representation Need a compact representation • Operators Operators 12 Operator :: CUT Preconditions : File Edit View Ins isCurrent(Menu2). Menu1 Effects : FORALL Obj in Objects Selected(Obj) ⇒ Cut ⇒ ⇒ ⇒ ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). Menu2 Obtaining next state 6
Deriving Expected State Deriving Expected State 13 • Given Given S S 0 , the initial state, • 0 , the initial state, • A sequence of events A sequence of events • e 1 e 2 e 3 e n • Obtain Obtain S S 1 = [S 0 , e 1 ] • 1 = [S 0 , e 1 ] • And And S S i = [S S i , e e i ] • i = [ 1 , i ] i- -1 e n e 1 e 2 e 3 S 0 S 1 S 2 S 3 S n S 0 S 1 S 2 S 3 S n Obtaining Actual GUI’s State Obtaining Actual GUI’s State 14 • • Execution Monitor Execution Monitor – Screen Scraping Screen Scraping – – – Queries Queries – – Compatible with Expected State Compatible with Expected State – Returns <Object, Property, Value> – Returns <Object, Property, Value> <Button1, “Caption”, “Cancel”> <Button1, “Caption”, “Cancel”> 7
Automated Execution Automated Execution 15 Test Executor GUI Under Test Test Cases Test Cases Execution Monitor ACTUAL STATE: (isCurrent ROOT) Verifier (Contains ROOT D (Contains ROOT D Expected State Expected State Comparing Comparing 16 Actual and Expected States Actual and Expected States • Verifier • Verifier • • Three Levels of Testing Three Levels of Testing – Changed Property Set Changed Property Set (Operators) (Operators) – – – GUI Relevant Property Set GUI Relevant Property Set (Specifications) (Specifications) – – Complete Property Set Complete Property Set (Toolkit/Language) (Toolkit/Language) • • Hybrid Approach Hybrid Approach – Use all 3 – Use all 3 8
Case Study Case Study 17 • Purpose: Determine Purpose: Determine • – Time to Derive Expected State Time to Derive Expected State – – – Time to Execute Monitor and Verifier Time to Execute Monitor and Verifier • Experimental Design Experimental Design • – GUI GUI : : Our Version of MS WordPad (36 Modal Our Version of MS WordPad (36 Modal – Windows, 362 events) Windows, 362 events) – – Test Cases Test Cases : : Generated 290 Test Cases (6 Generated 290 Test Cases (6- -56 56 events) using an AI Planner events) using an AI Planner – Hardware Platform Hardware Platform : : 350 MHz Pentium based 350 MHz Pentium based – Machine, 256 MB RAM Machine, 256 MB RAM – Properties – Properties : : Reduced Set Reduced Set – Level of Testing Level of Testing : : GUI Relevant Property Set GUI Relevant Property Set – Deriving Expected State Deriving Expected State 18 Generating Test Cases and Deriving Expected State 0.9 Test Case + Expected State 0.8 0.7 Time (sec.) 0.6 0.5 Test Case 0.4 0.3 Expected State 0.2 0.1 0 1 6 11 16 21 26 31 36 41 46 51 56 Test-Case Length Total CPU time (test case and expected state) 75.84 sec. 9
Execution Execution 19 Executing Test Cases, Verifier and Execution Monitor 5 Test Case + Verifier + Execution Monitor 4 Time (sec.) 3 Verifier + Execution Monitor 2 1 Test Case 0 1 6 11 16 21 26 31 36 41 46 51 56 Test-Case Length Relevant-properties verification Total running time < 10 minutes 10
Recommend
More recommend
