Mobile Testing-as-a-Service (Mobile TaaS or MTaaS) Jerry Gao, Ph.D. Professor Computer Engineering Department San Jose State University W. T. Tsai, Ph.D. Professor Sch Compt Infor & Dec Sys Engr Arizona State University
Table of Contents • Introduction • What is Mobile TaaS? • Why is Mobile TaaS Important? • Mobile Testing Process • Major Players and Tools • Mobile Testing Approaches, Infrastructures, and Comparisons • Mobile TaaS Approaches, Infrastructures, and Comparisons • Expected Benefits and Features • Test Models and Test Coverage Criteria • User View of TaaS • Cloud Computing View of TaaS • Issues and Needs for Future Research
Introduction – Market Trends on Mobile According to the latest study from Juniper Research, the market for cloud-based mobile applications will grow 88% from 2009 to 2014 . The market was just over $400 million this past year, says Juniper, but by 2014 it will reach $9.5 billion . Driving this growth will be the adoption of the new web standard HTML5, increased mobile broadband coverage and the need for always-on collaborative services for the enterprise.
Introduction – Market Needs on Mobile
Knowledge Background for Mobile TaaS Research Software Cloud Testing and Mobile Computing Test Computing & SaaS Automation Engineering
What Is Mobile TaaS? No existing definition for Mobile TaaS. Our first definition: (Mobile TaaS in a cloud infrastructure): “Mobile Testing as a Service (known as Mobile TaaS) provides on-demand testing services for mobile applications and SaaS to support software validation and quality engineering processes by leveraging a cloud-based scalable mobile testing environment to assure pre-defined given service-level-agreements (SLAs). Mobile TaaS on clouds offers a new business model for diverse mobile software validation services using the pay-as-you-test model to achieve cost-sharing and cost-reduction in mobile computing resources, networks, cloud computing and storage infrastructures.”
Mobile SaaS Features Mobile Client Software (Thin/Smart/Thick Client) Multi-Tenancy and Searchable, Discoverable, Elastic Scalability Deployable, and Executable Mobile Customization and On-Demand Function Configuration Services with APIs SaaS Online Distributed Mobile Pay-as-you-go Access at Anywhere & Anytime Billing Model Mobility and Connectivity Over wireless Internet
Mobile Testing Scope Function and Compatibility, Behavior Testing Multi-Tenancy Testing Interoperation ability On Mobile Clouds and Usability Testing Mobile QoS Testing On-Demand Mobile (Performance, load, Test Run & Control Testing Scalability..) Internationalization Mobile Test Simulation Testing on Mobile and Virtualization Mobility and Connectivity Testing
Mobile TaaS Requirements and Needs Testing at Anywhere & Anytime (365/7/24) for Mobile App/Web App. Test Project-Oriented Contracting, Multi-Tenancy & Utilization Billing Customization Support and Reporting Mobile On-Demand Scalable Large-Scale On-demand Mobile Test Infrastructures Mobile Test Services TaaS and Environments Control and Configuration Test Tracking, Monitoring, Of Mobile Test Environments and Coverage Analysis Easy Interoperation ability, Mobility & Connectivity
Why Is MTaaS Important? High costs on mobile test infrastructures due to: • Diverse mobile devices and HW appliances &APIs • Diverse mobile operation environments (platforms, connectivity, and configurations) • Diverse wireless connectivity and configurations Frequent changes and upgrades on: • Mobile devices, appliances, and their APIs • Mobile applications, features, and service plans • Mobile platforms, technologies, input solutions Complex mobile user interfaces • Diverse mobile browsers and technologies Complexity • Input approaches, display screens • Multi-language mobile contents Large-scale on: • On-demand mobile test service requests Large-Scale • Mobile test simulation and traffic loads • Virtualization of mobile devices and environments Multi-tenancy of mobile app/web applications • Functions, behaviors, and QoS REQs Multi-Tenancy • Data, security features, user interfaces
Why Is MTaaS Important?
Mobile Web vs. Native Apps: Differences in Testing
Mobile Testing Process for MSaaS/Mobile APP Tenant-Based Fault/Disaster Live Upgrading Live State Live Re-Test Continuous Testing Recovery Testing Testing Scenario Testing Testing Usability Mobile Feature Connectivity Compatibility Mobility Testing Testing Testing Feature Testing Testing Mobile Security Internationalization Interoperability Testing Testing Testing Tenant-Based APP Installation Security System QoS Multi-tenancy System Testing & Deployment Testing Testing Testing Scalability Reliability Performance & Availability Testing Testing Load Testing Testing Tenant-Based System Function GUI-Based Business Testing Function Behavior /Scenario Intelligence Testing Testing Testing Testing Service Component Integration Black-Box White-Box Performance Component Service Component Testing Testing Testing API Testing Unit Testing
Major Players and Tools in Mobile Testing seeTestMob MobileClou MonkeyTal SandStorm Robotiu WebDriver MonkeyRu Selenium Calabash Dollop Android eggPlant Neoload MITE Sikuli nner ile k d m Testing Focus Functional Testing R e-Testing Performance Testing Load Testing Platform Support Linux Windows Mac Tests on Mobile O.S Android iOS Windows Testing Mobile Web Apps Testing ative Apps Source code based testing Tests done using Emulator Real Device Scrip languages Java Python Jython Ruby Perl Record & Replay Image based Service Cost License / Subscription Open Source
Mobile Testing Environments and Infrastructures Emulation-Based Testing Mobile App server Emulation-Based Mobile GUI Simulation-Based Mobile Testing Mobile App server Simulation-Based Mobile GUI Wireless Device-Based Mobile Testing Internet Mobile App server Mobile Device in a Lab. Remote Device-Based Mobile Testing Wireless Internet Mobile App server Remote Mobile Device in a third-party mobile Lab.
Conventional Mobile Testing Approaches Perspectives/ Different Emulation-Based Simulation-Based Mobile Device Remote Device Approaches Testing Testing Based Testing in a Based Mobile Lab. Testing in a Lab. System Function Testing Limitations on Limitations on Support Support device-specific device-specific functions functions End-to-End System w/o real mobile w/o real mobile Support Support Behavior Testing device behaviors device behaviors Mobile Compatibility & Very limited Very limited Require diverse Require diverse Interoperability Testing mobile devices and mobile devices and platforms platforms Mobile GUI and Usability Limited w/o Limited w/o Good for real Good for real device Testing devices devices device experience experience Load Testing Require server- Easy to generate Costly to generate Costly to generate oriented load simulation-based real mobile traffic real mobile traffic testing solutions system loads loads loads Wireless Connectivity Limited coverage Limited coverage Good coverage but Good coverage, but Testing require local require remote wireless network wireless network connection control connection control Operation Costs Low Low High High Download & deployment Not support for Not support for Support Support Mobile Native App Mobile Native App
Conventional Mobile Testing Approaches Perspectives/ Emulation-Based Simulation-Based Mobile Device Remote Device Different Testing Testing Based Testing Based Mobile Approaches Testing Scalability Testing Limited Support Limited Support Not Support Not Support Performance Testing Check emulation- Check simulation- Check mobile device Check remote based mobile system based mobile system user and system mobile device and performance performance performance system performance Multi-Tenancy Limited coverage on Limited coverage on Supported with very Supported with very Testing multi-tenanted GUI multi-tenanted GUI high costs high costs features for specific features for specific devices devices Mobility and Limited testing Limited testing Limited to signed Limited to signed Location-based based on configured based on simulated device testing device testing Testing locations locations Mobile Security Limited coverage on Limited coverage on Support testing on Support testing on Testing real mobile OS/ mobile OS/ very limited scale very limited scale platforms and platforms and mobile clients mobile clients Test Automation Easy for GUI testing Easy for GUI testing Required integrated Required integrated and load testing mobile test beds with mobile test beds with wireless connectivity wireless connectivity
Different Mobile TaaS Infrastructure Emulation-Based Wireless Internet Mobile Testing on Clouds Emulation-Based Mobile App Mobile clients Instances Simulation-Based Mobile Wireless Testing on Clouds Mobile App Internet Instances Simulation-Based Mobile clients Wireless Device-Based Mobile Mobile App Internet Testing on Clouds Instances Mobile Device in a private cloud
Recommend
More recommend
