Testing the Mobile Application's Performance Testing the Mobile - PowerPoint PPT Presentation

Testing the Mobile Application's Performance Testing the Mobile Application's Performance Case Study on Communicator Mobile Case Study on Communicator Mobile Rama Krishna Pagadala Rama Krishna Pagadala Microsoft Corporation Microsoft Corporation

Agenda • Performance Metrics for Mobile Applications • Communicator Mobile 2007 R2 • Product Performance Strategy • Performance Testing Strategy • Architectural improvements • Takeaways from this case study

Performance Time - UI Response Time • Metrics (Mobile) Users tend to be less patient – Very demanding –

User: “Download is too slow!” How can we make the download faster ? “ Operational Management” Better compression, faster connection, Most compelling and tangible, smaller file size, etc. but expensive How can we make the download appear faster? Provide progress indication, diversion, “ Perception management”. Inexpensive, effective for etc. mainstream usage context How can the download be more tolerable ? Offer completion notification, provide “ Tolerance management”. Best remedy when perception “emergency exits”, etc. is rigid. Inexpensive Designing and Engineering Time by Steven C. Seow

Performance Battery – critical • Metrics Drains non-linear • Mitigations • Minimize the radio wakeups – System events instead of polling –

Cell Radio (Data)

Performance Network Bandwidth • Metrics Low speed networks – Pay-per-use ($) – Roaming ($$$) – Impacts • Time – User Experience – Battery –

Performance Metrics (no Virtual) Memory & CPU • Device sluggishness  Responsiveness – Application crashes –

Performance Metrics Binary size (footprint) • Application load time – RAM usage – Download/Install time –

Communicator Mobile 2007 R2 • Communications application for enterprise users • Always-on • Completes UC* story and complements other UC endpoints • Contacts, Presence, Work calls, Conferencing, Search, … • Windows marketplace *Microsoft Unified Communications

Product Strategy • Customer feedback for CoMo 2007 • Other communication apps • Personas • 3 Power Users • Performance goals • Ex: Login < 10 seconds • Design for Performance

User Profiles • Testing Baseline/Email Power User Strategy –

User Profiles • Testing Communicator Mobile Power User Strategy –

Testing Scenarios • Strategy End-to-End scenarios (User centric) – Script – Specific scenarios (Dev/Test teams) –

3 rd party Tools • Testing Strategy PowerMon (device level) – GPRSMon (network/application level) – In house Tools • NetMon (network level) – Memory/CPU (per process) –

Testing Strategy Test Execution • Daily/Weekly? – Manual vs. Automated? – Test configuration: PocketPC vs. SmartPhone? 3G vs. – Edge?, … Debugging • Is hard – Inconsistent issues – Logging impacts the performance – Standard variation –

Testing Reporting • Strategy Weekly report (Email) – Trends, charts – Automated results (daily) – Product team analysis –

Test Matrix • Wide array of devices • Carriers • Networks - 2.5/Edge/3G • Cellular Data vs. WiFi • Platform intricacies (WM 6.x power management)

Architectural improvements • Poll vs. Push – Presence updates – Contacts in view • Collapse Contact Groups – improves sign-In time! • Piggy back the data – Keep-alives with Presence updates – ActiveSync (Email/Calendar)

Takeaways • Performance is key to mobile application's adoption – Think/Design/Test early on – Set clear goals for select-few targeted users • Performance testing is challenging on mobile platforms – Metrics are inter dependent – Tools, variety of platforms, … – Usage patterns are quite different – +/- 10-20% Variation? – Consider External factors affecting perf: signal strength, carrier networks, Network type (3G/HSPDA), … • Manual (UI) vs. Automated (UI/API) • Absolute (engineers) vs. Perceived (users!)

Credits • Communicator Mobile Product Team • Windows Live Messenger Team • Designing and Engineering Time: The Psychology of Time Perception in Software by Steven C. Seow • Power Monitor • SPB GPRSMon

Thanks! Questions