¡ ¡ W2 ¡ AI ¡and ¡Data ¡Analytics ¡ Wednesday, ¡October ¡17th, ¡2018 ¡10:15 ¡AM ¡ ¡ ¡ ¡ ¡ ¡ ¡ What's ¡Our ¡Job ¡When ¡the ¡Machines ¡Do ¡ Testing? ¡ ¡ Presented ¡by: ¡ ¡ ¡ Geoff ¡Meyer ¡ ¡ ¡ ¡ ¡ Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ 350 ¡Corporate ¡Way, ¡Suite ¡400, ¡Orange ¡Park, ¡FL ¡32073 ¡ ¡ 888 -‑-‑-‑ 268 -‑-‑-‑ 8770 ¡ ·√·√ ¡904 -‑-‑-‑ 278 -‑-‑-‑ 0524 ¡-‑ ¡info@techwell.com ¡-‑ ¡http://www.starwest.techwell.com/ ¡ ¡ ¡ ¡ ¡ ¡ ¡
¡ ¡ ¡ ¡ Geoff ¡Meyer ¡ ¡ A ¡test ¡architect ¡in ¡the ¡Dell ¡EMC ¡infrastructure ¡solutions ¡group, ¡Geoff ¡Meyer ¡has ¡ more ¡than ¡thirty ¡years ¡of ¡industry ¡experience ¡as ¡a ¡software ¡developer, ¡manager, ¡ program ¡manager, ¡and ¡director. ¡Geoff ¡oversees ¡the ¡test ¡strategy ¡and ¡architecture ¡for ¡ more ¡than ¡four ¡hundred ¡software ¡and ¡hardware ¡testers ¡across ¡India, ¡Taiwan, ¡and ¡ the ¡United ¡States. ¡He ¡leads ¡initiatives ¡in ¡continuous ¡testing, ¡predictive ¡analytics, ¡and ¡ infrastructure ¡as ¡a ¡service ¡(IaaS). ¡Outside ¡of ¡work, ¡Geoff ¡is ¡a ¡member ¡of ¡the ¡Agile ¡ Austin ¡community, ¡contributor ¡to ¡the ¡Agile ¡and ¡STAR ¡conferences, ¡and ¡an ¡active ¡ mentor ¡to ¡veterans ¡participating ¡in ¡the ¡Vets4Quality.org ¡program, ¡which ¡provides ¡ an ¡on-‑ramp ¡to ¡a ¡career ¡in ¡software ¡quality ¡assurance. ¡ ¡
What’s Our Job When The 17 Oct 2018 Geoff Meyer, Test Architect Machines Do Testing? geoff_meyer@dell.com
Navigating the Age of the Machine Machine Partnerships in Test A Journey in the age of Smart Assistants What’s our job?
Beware!
Abundance Number of People living in Extreme Poverty
Budding Effect
Convenience
Industrial Revolutions are not new 1 st Industrial Revolution 2 nd IR 3 rd IR 4 th IR 2000 2018 1800 1900
Analytics and AI in Business Human Resources Legal Discovery Global Audit Global Services & Support Financial Services Sales & Pricing Marketing Analytics Operations …Identify what doesn’t work well in a process, service or product and make it go away ~ Malcolm Frank, Cognizant Future of Work
But it’s OK Almost 90%! Enhanced Invented Replaced 75% 12% 13% "Think about a job as the sum of it’s tasks.” ~ What to do when Machines do everything Cognizant Center for the Future of Work
Context at DellEMC Servers 465 Trillion Test Configurations!! Server Configuration Elements Chassis Processor Memory DIMM Memory Configuration Hard Disk Drive (HDD) Non-Volatile Memory (NVM) Embedded Systems Management Power Management BIOS Power Supply Bezel Network Daughter Card RAID Controller Network Interface Card (NIC) Host Bus Adapter (HBA) Additional PCIe Cards Cooling
DellEMC Server: Evolution of Testing Waterfall Agile Scripted Automation Shift-left testing DevOps Automated UI Testing Axon Workflow Automation Cognitive Automation Continuous 1997 Deployment Integration Non-Stop Test Ops Configuration Portability Continuous Test Assistant (Zeno) Reactor Reuse Marketplace 2010 Test Suite Assistant Continuous HW-Aware Testing AI-Assisted UI Deployment Priority-driven testing Automation 2013 SDETs Continuous Monitoring Lab-as-a-Service TMMi 2015 2017 2019
The Analytics Continuum How can we Self-Learning make it What will Happen? Happen? Artificial Intelligence Why did it Prescriptive Business Value Happen? Predictive What Diagnostic Happened? Descriptive Report Correlate Predict Recommend Autonomous Source: Gartner Data Analytics Sophistication
Data Analytics Modeling Data Data Sources Cleanse Insights, Predictions, Recommendations Domain Analytics Knowledge Engine (i.e. Rules) Feedback
What goes into the Model? “Risk -based Testing” Rules and heuristics “Prior Failed “Only deploy Test Cases” BVT- verified builds” Best- practices Rules of Thumb Rules Positive/Negative Patterns Tribal Knowledge Common “Related historical Sense “Test the defect test failures” that got fixed”
It’s all about the data “Nobody really goes out of their way to point out the importance of data…” ~ Brian Sletten, Bosatsu Consulting
What if we had a Smart Assistant?
What is the The Smart Assistant release risk given the testing that’s been completed? What are the high-value SUT What test scripts configurations? should be retired rather than be re-factored? What’s the optimal What tests can detect coverage for this the maximum number of defects build/test cycle? What automated test given the changes in the failures appear to be current build duplicates?
Data Sources - Product Engineering
Selecting our Technology Partners
Operationalizing • Standardize your Data Sources • Develop Connectors to extract data • Design/Develop your data lake • Workflow Automation • Visualization Dashboards
SUT Configuration Model Team “Q” - System Under Test Current Testing Scenario Challenges ➢ 465 Trillion possible server configs!!! ➢ Which are the High Value Configs? ➢ How to ensure Optimal Configs Coverage? Data Sources Objectives ➢ Quickly predict “best - available” SUT configurations during planning and test execution ➢ Ensure Optimal Configs Coverage ➢ Prioritizes High-Value Configs Q Technology Partner: Dell Performance Analytics Group
Test Planning Model “JARVIS” Objectives • Use historical test data and defects as predictors and to expose patterns • Automate deep-think testing tasks • Codify Subject-Matter Expertise • Real-time access to active Increased Test repositories Capacity Reduced test Which manual tests are most effective, and should be automated? cycle time How can I accelerate discovery of break/fix? Re-allocate to Which of my test cases appear to be obsolete? Exploratory Testing Am I over-testing or under-testing? Fast Find of Automation Break/Fix candidates Technology Partner: De-prioritization candidates
Testbots are here AI-assisted UI Automation • Increases UI test coverage • At substantially less cost of creation and maintenance
Duplicate Defect Prediction DellEMC XtremIO
What’s our job?
What can Humans do Better? • Creative, Collaborative, Problem Solving • Contextualized Intelligence • Empathy • Storytelling
Key Skills/Attributes of a Tester Janet Gregory
Testing ~ Value Creation Testing vs. Checking • Establishes Expected Behavior • Collaborative • Curious Sprints • New Feature Exploratory 1 2 3 4 5 6 Development/Test • Cognitive • Analyze Potential Risks Legacy test • Requires Thinking Legacy Regression Checking automation maintenance Checking ~ Value Sustainment 1 1 1 1 1 • Confirms Expected Behavior New feature test 2 2 2 2 • Robotic automation development New Feature Regression Checking • Tedious & maintenance 3 3 3 • Scripted 4 4 • Vigilance for deviations 5 • Monitor Known Risks “No one confuses spellcheckers with editing, so why confuse automated checks as testing?” • Requires Processing ~ Michael Bolton
Envisioning the Future of Testing Dependencies Requirements Product roadmap Technologies Technical Debt Value Creation Collaborative Value Resources Sustainment
Envisioning the Future of Testing Automation Dependencies Requirements • Unit tests Product roadmap Technologies • Code Complexity Technical Debt • Build Verification Testing • Regression Testing • Non-Functional Testing • Value Simulation/Emulation • DevOps Creation • Continuous Integration • Environment Provisioning • Continuous Deployment • Continuous Testing • Cognitive Tasks (AI/ML) Collaborative • Process Orchestration Value Resources • Non-Step Test Operations Sustainment • Continuous Monitoring
What about Data Science skills? Creative Commons
What are our roles? Test Engineer Test Strategy Strategy Predictive Modeling Requirements Elaboration Test Case Analysis Planning Test Case Development Test Suite Planning AI Trainer Test Environment Mgmt Test Configuration Planning Provisioning SW/FW Provisioning(DevOps) Test Data Mgmt Test Case Execution Test Automation Development Exploratory Testing Dev/Testing AI Maker Data Connectors Data Scientist SDET Test Failure Triage Tracking Defect Tracking
Recommend
More recommend
