Automatically Discovering, Reporting Kevin Moran , Mario - PowerPoint PPT Presentation

& College of William & Mary - SEMERU - Department of Computer Science Automatically Discovering, Reporting Kevin Moran , Mario Linares-Vásquez, and Reproducing Android Carlos Bernal-Cárdenas, Christopher Vendome, Application Crashes & Denys Poshyvanyk ICST 16 Chicago, IL Tuesday, April 12th, 2016

2

2

MANUAL TESTING

MANUAL TESTING

AUTOMATED TESTING

AUTOMATED TESTING

AUTOMATED TESTING

CATEGORIES OF AUTOMATED TESTING APPROACHES FOR MOBILE APPS • Model-based input generation • Random-based input generation • Record and replay • Others (Manual Testing Frameworks)

THE CURRENT STATE OF AUTOMATED MOBILE APPLICATION TESTING GUI Types of Crash Replayable Test NL Crash Emulators, Tool Name Instr. Exploration Events Resilient Cases Reports Devices Yes Yes No No No Guided/Random Dynodroid System, GUI, Text No System/Evo GUI No No No N/A EvoDroid Yes Systematic GUI, Text No No No N/A AndroidRipper Yes Model-Based GUI, Text No Yes No N/A MobiGUItar Yes Systematic GUI No No No Yes A3E DFS Yes Model-Based GUI No No No Yes A3E Targeted [20] Yes Model-Based GUI, Text N/A No No Yes Swifthand Yes Programmable N/A No No Yes PUMA System, GUI, Text Yes Systematic GUI N/A No No Yes ACTEve Yes Random Yes Yes No N/A VANARSena System, GUI, Text Yes Test Cases N/A N/A No No Thor Test Case Events Yes Model-Based System, GUI N/A Yes No N/A QUANTUM Yes Multiple Yes Yes No N/A AppDoctor System, GUI, Text No Model-Based GUI N/A No No N/A ORBIT No Record/Replay GUI N/A N/A No No SPAG-C No Scripting GUI N/A Yes No N/A JPF-Android No Model-based GUI, Text No Yes No Yes MonkeyLab No GUI, Text Manual Yes Yes Yes CrashDroid Manual Rec/Replay No Symbolic GUI, Text N/A Yes No N/A SIG-Droid No Systematic Yes Yes Yes Yes CrashScope GUI, Text, System

THE CURRENT STATE OF AUTOMATED MOBILE APPLICATION TESTING GUI Types of Crash Replayable Test NL Crash Emulators, Tool Name Instr. Exploration Events Resilient Cases Reports Devices Yes Yes No No No Guided/Random Dynodroid System, GUI, Text No System/Evo GUI No No No N/A EvoDroid Yes Systematic GUI, Text No No No N/A AndroidRipper Yes Model-Based GUI, Text No Yes No N/A MobiGUItar Yes Systematic GUI No No No Yes A3E DFS Yes Model-Based GUI No No No Yes A3E Targeted [20] Yes Model-Based GUI, Text N/A No No Yes Swifthand What are the limitations of current Yes Programmable N/A No No Yes PUMA System, GUI, Text Yes Systematic GUI N/A No No Yes ACTEve Yes Random Yes Yes No N/A VANARSena automated approaches? System, GUI, Text Yes Test Cases N/A N/A No No Thor Test Case Events Yes Model-Based System, GUI N/A Yes No N/A QUANTUM Yes Multiple Yes Yes No N/A AppDoctor System, GUI, Text No Model-Based GUI N/A No No N/A ORBIT No Record/Replay GUI N/A N/A No No SPAG-C No Scripting GUI N/A Yes No N/A JPF-Android No Model-based GUI, Text No Yes No Yes MonkeyLab No GUI, Text Manual Yes Yes Yes CrashDroid Manual Rec/Replay No Symbolic GUI, Text N/A Yes No N/A SIG-Droid No Systematic Yes Yes Yes Yes CrashScope GUI, Text, System

LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING • Lack of detailed, easy to understand testing results for faults/ crashes 1 • No easy way to reproduce test scenarios 1 • Not practical from a developers viewpoint • Few approaches enable different strategies capable of generating text and testing contextual features 1 S. R. Choudhary, A. Gorla, and A. Orso. Automated Test Input Generation for Android: Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015

PAST STUDIES OF MOBILE CRASHES AND BUGS • Many crashes can be mapped to well-defined , externally inducible faults 1 • Contextual features , such as network connectivity and screen rotation, account for many of these externally inducible faults 12 • These dominant root causes can affect many different user execution paths 1 1 L. Ravindranath, S. Nath, J. Padhye, and H. Balakrishnan. Automatic and scalable fault detection for mobile applications. MobiSys ’14 2 R. N. Zaeem, M. R. Prasad, and S. Khurshid. Automated generation of oracles for testing user-interaction features of mobile apps, ICST ’14

OUR SOLUTION: CRASHSCOPE • Completely automated approach • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators • Differing execution strategies able to test contextual features

CRASHSCOPE DESIGN 1 3 Static Analysis (Contextual Feature Report Generation Android Extraction) Application . apk CrashScope Database or 4 5 2 app src Crash-Execution Crash-Execution GUI-Ripping Engine Script Generator Script Replayer Physical Device or Emulator

CRASHSCOPE DESIGN 1 3 Static Analysis (Contextual Feature Report Generation Android Extraction) Application . apk CrashScope Database or 4 5 2 app src Crash-Execution Crash-Execution GUI-Ripping Engine Script Generator Script Replayer Physical Device or Emulator 1

CRASHSCOPE DESIGN 1 3 Static Analysis (Contextual Feature Report Generation Android Extraction) Application . apk CrashScope Database or 4 5 2 app src Crash-Execution Crash-Execution GUI-Ripping Engine Script Generator Script Replayer Physical Device or Emulator II 1

CRASHSCOPE: ANALYSIS 1 Contextual Feature Extractor .apk decompiler 2 GUI Ripping Engine (if necessary) Crash after last step? Manifest File API Extractor Parser No Yes —Touch Event —GUI Component Android Information Application Execution CrashScope App and —Screenshots Finished? Rotatable Database Activity Level Save Activities . apk 3 Contextual Execution Features Information or No Yes Continue Execution Physical Device or Emulator app src Android Decision Engine App and UIAutomator Activity Level Determine next Contextual <Action, GUI> Features Event to Execute Event Execution Engine Enable/Disable ( adb input & Activity/App telnet ) Features

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION uiautomator

CRASHSCOPE: EXPLORATION uiautomator

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION • Activity • Checkable, Checked, Clickable, Long Clickable? • Component Index • Current Window • Enabled? • XML_ID • Component Type • Position (Absolute and Relative) • Text • Screenshot →

CRASHSCOPE: EXPLORATION

CRASHSCOPE: EXPLORATION CrashScope Database

CRASHSCOPE STRATEGIES • GUI-Traversal: Top-Down & Bottom Up • Text Entry: Expected, Unexpected, No Text • Contextual Features: Enabled or Disabled

CRASHSCOPE: REPORT AND SCRIPT GENERATION Augmented Natural Language Report Generator 4 App 5 CrashScope Report http://cs.wm.edu/semeru Google Database Executions Containing Parser Crashes Step Processor Web Based Application Bug Report ( JSP , MySQL , and Bootstrap ) CrashScope Database 3 7 6 Crash Execution Script Generator Crash Execution Script Replayer Contextual App Replay CrashScope Database Event Executions Script Script Parser Containing Interperter / Parser Generator adb Crashes Replayer Physical Device Contextual or Emulator Event Execution (telnet Replay Script Tuples commands) <adb shell input tap 780 1126> Event Execution <adb shell input text ‘abc!@#’> Engine <Disable_Network> ( adb sendevent & <Disable_GPS> adb input )

CRASHSCOPE DEMO

CRASHSCOPE DEMO

CRASHSCOPE: REPORTS

CRASHSCOPE: REPORTS

CRASHSCOPE: REPORTS

CRASHSCOPE: REPORTS

CRASHSCOPE: REPORTS

CRASHSCOPE: REPORTS

EVALUATION • Two Empirical Studies • Study 1: Crash Detection Capabilities • Study 2: Crash Report Reproducibility and Readability

STUDY 1: CRASH DETECTION & COVERAGE • RQ 1 : Crash Detection Effectiveness? 
 • RQ 2 : Orthogonality of Crashes? 
 • RQ 3 : Effectiveness of Individual Strategies? 
 • RQ 4 : Does Crash Detection Correlate with Code Coverage?

STUDY 1: EXPERIMENTAL SETUP Tool Name Android Version Tool Type TOOLS USED IN Monkey Any Random THE COMPARATIVE A3E Depth-First Any Systematic FAULT FINDING Any Model-Based GUI-Ripper STUDY Dynodroid v2.3 Random-Based PUMA v4.1+ Random-Based • 61 subject applications from the Androtest 1 toolset • Each testing tool was run 5 separate times for 1 hour, whereas CrashScope ran through all strategies • Monkey was limited by the number of events 1 S. R. Choudhary, A. Gorla, and A. Orso. Automated Test Input Generation for Android: Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015