AD30 - 5 Common Types of Mobile App Bugs Found Using - PDF document

¡ ¡ ¡ ¡ AD30 ¡ DevOps ¡Practices ¡ 10:00 ¡AM ¡ ¡ ¡ ¡ ¡ ¡ AD30 ¡-­‑ ¡5 ¡Common ¡Types ¡of ¡Mobile ¡App ¡ Bugs ¡Found ¡Using ¡AI ¡ ¡ Presented ¡by: ¡ ¡ ¡ ¡ Sandy ¡Park ¡ ¡ ¡Apptest.ai ¡ ¡ Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ 888 -­‑-­‑-­‑ 268 -­‑-­‑-­‑ 8770 ¡ ·√·√ ¡904 -­‑-­‑-­‑ 278 -­‑-­‑-­‑ 0524 ¡-­‑ ¡info@techwell.com ¡-­‑ ¡ https://agiledevopswest.techwell.com/ ¡ ¡

¡ Sandy ¡Park ¡ ¡ Sandy ¡Park ¡is ¡a ¡co-­‑founder ¡of ¡Apptest.ai, ¡an ¡emerging ¡startup ¡in ¡exploratory ¡mobile ¡ testing ¡located ¡in ¡the ¡bay ¡area ¡and ¡Seoul. ¡She ¡recently ¡automated ¡Facebook's ¡fiber ¡ connectivity ¡projects ¡while ¡partnering ¡with ¡MNOs ¡in ¡EMEA ¡and ¡APAC. ¡Prior ¡to ¡that, ¡she ¡ was ¡a ¡Director ¡of ¡mobile ¡partnerships ¡in ¡Samsung ¡USA. ¡She ¡also ¡contributed ¡to ¡the ¡ acquisition ¡of ¡EdgeSpring ¡to ¡Salesforce, ¡which ¡is ¡now ¡called ¡Salesforce ¡Wave. ¡She ¡holds ¡ a ¡Ph.D. ¡of ¡Energy ¡Resources ¡Engineering, ¡and ¡a ¡Master's ¡of ¡Electrical ¡Engineering ¡both ¡ from ¡Stanford ¡University. ¡

4/29/19 5 Common Types of Mobile App Bugs Found Using AI June 2019 Sandy Park So Many Bugs, So Little Time 2 1

4/29/19 AGENDA App Errors, Overview What is AI Testbot 5 Types of Common Errors Principles Behind AI Testbot vs. Static Shift Left with CI Integrations 3 AI TESTING AS MAJOR TREND Gartner: “By 2022, at least 40% of new projects will have AI co-developers on the team.” What’s it like to be L5 for testing? https://medium.com/@giltayar/not-only-cars-the-six-levels-of-autonomous-testing-a22611ab1d7f 4 2

4/29/19 BUSINESS IMPACT via AI TESTBOT 50% Cost Saving Simple to Use 400% Increase in Release Frequency Test More! 300% Increase in Testing Capacity More Errors Caught 10% Increase in Customer Retention Higher Quality 10% Increase in Conversion Rate Less Poor Reviews! 5 ERROR UNIVERSE from MACHINE FOUND Detectable by Logs - Crashes - Webpage not available We’ll come back later! OTHERS - Context-specific - UI Aesthetics 6 3

4/29/19 CATEGORIZATION – PREVIOUS STUDIES Sapienz: Multi-objective Automated Testing Characterizing Android Specific Crash Bug, ‘19 Large-Scale Analysis of Framework-Specific Exceptions for Android Applications, ’12 in Android Apps, ‘18 7 AI TESTBOT SUMMARY STATS ETC 18% Broken URL Link 25% RuntimeException 2% BadTokenException 2% UnsatisfiedLinkError 3% 16,260 Test Runs Out of memory 3% 1,107 Apps Tested IllegalStateException 3% Native 13% 214 Buggy Apps (19.3%) InflateException 4% Resource Limitation Error 4% NullPointerException FileUriExposedException 9% 5% IndexOutOfBoundsException 9% 8 4

4/29/19 PERFORMANCE METRIC 1 - COVERAGE 9 PERFORMANCE METRIC 2 – ERROR HIT RATE Apptest.ai Detected More Errors 15 vs. 2 among US Top 100 Apps (Q2 ‘18) Unique Unique Applications Actions Result Actions Result Screenshots Screenshots AccuWeather 214 97 Error(s) Found 116 65 Clean/No Errors MLB At Bat 269 119 Error(s) Found 12 11 Clean/No Errors Bitmoji 301 118 Error(s) Found 16 12 Clean/No Errors The Honest Co 360 90 Error(s) Found 98 44 Error(s) Found Smiley Emoji Keyboard 2018 317 79 Error(s) Found 17 11 Clean/No Errors BK Booster 72 34 Error(s) Found 34 24 Error(s) Found AW Security Cleaner 79 19 Error(s) Found 46 29 Clean/No Errors NBC Sports Scores 275 125 Error(s) Found 48 45 Clean/No Errors News Break: Local & Breaking 302 135 Error(s) Found 20 15 Clean/No Errors Pinterest 293 31 Error(s) Found 16 8 Clean/No Errors Super Speed - Clean & Booster 160 28 Error(s) Found 23 18 Clean/No Errors Tubi TV - Free Movies & TV 303 146 Error(s) Found 95 28 Clean/No Errors King James Bible 191 124 Error(s) Found 247 224 Clean/No Errors Reddit 309 176 Error(s) Found 171 71 Clean/No Errors CNBC 142 49 Error(s) Found 18 12 Clean/No Errors 10 5

4/29/19 Deep Learning NLP NER Ontology UI Detection Virtual Persona Reinforcement Learning HOW IT WORKS 11 Step1: Object Detection Step2: Layout – Grouping and Labelling Group 1 Image: Image 1 Image 1 Text: Text 1 Text 1 Group 2 Option A : Text A Option A Text A Option B: Text B Option B Text B Group 3 Button 1 Button 2 Button2 : “Save” Button1 : “Play” Step3: Action Path Type: Image Type: Option Type: button Label: Image 1 Label: Text A Label: “Play” Order: 1 Order: 1 Order: 1 Value: N/A Value: Unchecked Value: N/A Group 1 Group 2 Group 3 Type: Text Type: Option Type: button Label: Text 1 Label: Text B Label: “Save” Order: 2 Order: 2 Order: 1 Value: N/A Value: Unchecked Value: N/A 12 6

4/29/19 PRODUCT SNAPSHOT COMMON ERROR TYPES Out of 39,000+ tests & 3,300+ bugs Broken Link Null Pointer Exception Index Out of Bounds Exception Out of Memory Bad Token Exception 14 7

4/29/19 Broken Link 1 Tubi TV 2.12.9 / LG Nexus 5 / Android 6.0 15 Broken Link 2 MLB At Bat 7.5.0 / Google Pixel / Android 9.0 16 8

4/29/19 Null Point Exception 1 Auto Reply for WA 7.5.3 / LG Nexus 5 / Android 6.0 17 Null Point Exception 2 Heart Radio 4.10.14 / LG Nexus 5X / Android 8.0 18 9

4/29/19 Index Out of Bounds 1 Lyft 5.23.3 / LG Nexus 5 / Android 6.0 19 Index Out of Bounds 2 DuoLingo 5.23.3 / LG Nexus 5 / Android 6.0 20 10

4/29/19 Out of Memory 1 End Game Wallpaper 1.0 / Samsung Galaxy S5 / Android 6.0 21 Out of Memory 2 Emoji Keyboard 1.1.7 / Samsung Galaxy S7 Edge / Android 7.0 22 11

4/29/19 Bad Token Exception 1 UAN e-Sewa 1.1 / Samsung Galaxy S7 Edge / Android 7.0 23 Bad Token Exception 2 Snake Screen Joke 1.0 / LG G6 / Android 8.0 24 12

4/29/19 Dynamic Testing vs. Static Testing 25 Limitation on Code Based Automation 1 Unidentified Resource IDs and Descriptions → Lack of information to automate 26 13

4/29/19 Limitation on Code Based Automation 2 Incorrect Clickable Properties 27 Multi-layer Detection Former technology remains taking actions on background which is irrelevant. 28 14

4/29/19 Sequencing Correct Input Order (1) (1) (1) (2) (3) (2) (2) (4) (3) (5) (3) (4) (4) (5) (6) 29 Input Relevant Value Strings Correct input Random string format detection for input slots for location, number, profile data, etc. 30 15

4/29/19 Mobile UI Recognition Deep Learning and Training for UI Element recognition 31 Named Entity Recognition Predicting the login based on certain patterns and using relevant name as an input for search. https://www.analyticsvidhya.com/blog/2019/02/flair-nlp-library-python/ 32 16

4/29/19 My app is updated every day. Can I test my app every day ? Hell No! ← Manual Testing Maybe. ← QA Scripting Yes! ← 33 Conventional MANUAL Functional Testing MANUAL MANUAL MANUAL TEST MANUAL TEST MANUAL TEST MANUAL TEST CODE BUILD MANUAL DEPLOY TEST TEST TEST Inconsistent and time consuming experience 34 17

4/29/19 Conventional AUTOMATED Functional Testing CODE CODE BUILD TEST DEPLOY TEST Time consuming and requires learning tools or expertise 35 Continuous Testing with AI TESTBOT Sync with your code update CODE CODE TEST BUILD DEPLOY TEST Test More! Codeless, AI based, continuous automated testing. Zero coding and knowledge required. Detect issues much earlier in the pipeline. 36 18

4/29/19 Integrate with your Favorite CI Tool 37 Jenkins with AI Testbot 38 19

4/29/19 Configure CI Pipeline (Jenkins) 39 Access Test Results (Jenkins) 40 20

4/29/19 Circle CI with AI Testbot • Just Add 5 lines to the configuration. orbs: testbot: saucelabs/ios-app-test@volatile … - testbot/run-test:: binary_path: app/build/outputs/apk/app-debug.apk project_id: '818 ‘ … .circleci/config.yml Bitrise with AI Testbot • Just Add 4 lines to the configuration. … - saucelabs-testbot: inputs: - binary_path: HackerNews.ipa - project_id: 825 … bitrise.yml 21

4/29/19 Microsoft Azure with AI Testbot • Just Add 7 lines to the configuration. … - bash: | git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY)… azure-pipelines.yml For the last eight months, NBC Owned Television Station was a pioneer to run its AI Bot to test their Android and iOS apps in production. Now it runs in all stages, everyday! 22

4/29/19 THANK YOU Confidential 23