"Test Automation Success: Choosing the Right People and - PDF document

W8 Track 4/28/2010 1:45 PM "Test Automation Success: Choosing the Right People and Process" Presented by: Kiran Pyneni Aetna, Inc. Brought to you by: 330 Corporate Way, Suite 300, Orange Park, FL 32073 888 ‐ 268 ‐ 8770 ∙ 904 ‐ 278 ‐ 0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Kiran Pyneni As senior QA manager at Aetna, Kiran Pyneni is responsible for improving test organization efficiencies through automation. Kiran leads a team of innovative automation enablers responsible for driving automation across Aetna. Prior to joining Aetna, Kiran had several roles: software developer, senior tester, testing architect, QA lead, and delivery manager for Fortune 500 organizations. Kiran was an invited speaker at the HP Software Universe 2009 USA Conference speaking about automation testing.

Test Automation Success: Choosing the Right People & Process Kiran Pyneni, Automation Manager Aetna, Inc. Agenda Company Background Automation Team Automation Objectives Automation Objectives Our Journey… Strategy Team Delivery Team Benefits of Teams Strategy Automation Lifecycle Automation Lifecycle Benefits of Process Integration Metrics Of Our Success Automation Coverage, Efforts Automated, Utilization & Cost Avoidance Questions 1

Aetna Background • Aetna is one of the nation's leading diversified health care benefits companies, serving members with information and resources to help them make better informed decisions about their health care. • P Products and management Programs: d t d t P – Aetna offers a broad range of insurance and employee benefits products. – The first national, full-service health insurer to offer a consumer-directed health plan, Aetna continues to lead the way with its Aetna HealthFund2 line of products, including HSA, HRA and RRA options. – Aetna offers a wide array of programs and services that help control rising employee benefits costs while striving to improve the quality of health care, such as case management; disease and patient safety programs; integrated medical, dental, pharmaceutical, behavioral health and disability information. – Aetna provides members with access to convenient tools and easy-to-understand information that can help them make better-informed decisions about their health information that can help them make better informed decisions about their health and financial wellbeing. 3 Automation Team Improves efficiencies and quality at our organization by automating consistent and repeatable manual tasks using industry standard technologies / tools / frameworks. Standards & Best Practices Automation Assessments & Reports & Metrics Team Designs Automation Development & Maintenance 2

Automation Objectives Automation Objectives Automating Right Tasks Utilization of existing automation Sustaining and Growing Automation Suite Our Journey… Decentralized Approach Centralized Approach • Benefits • Benefits – Easy identification of – Standardization of automation automation opportunities automation opportunities tool set at enterprise level tool set at enterprise level – Higher collaboration between – Sharing of coding best practices, application & automation teams templates and standards • Challenges – Cost of automation development is lowered – Automation development has • Challenges become expensive – Usage of different tools & – Slower automation coverage solutions progression – Standards & Best practices are – Lower automated scripts not shared across applications utilization – Funding Challenges 3

Our Journey Cont’d. – Challenges & Solutions Challenges Challenges • Identification of automation opportunities • • Early completion of maintenance for Early completion of maintenance for • Prioritization of automation projects automation scripts based on ROI • Funding for automation projects • Cost of automation • Collaboration between automation & • Implementation of optimized designs business teams Solution Solution • Structuring automation as strategy & • Create automation life cycle process and delivery team integrate with company's overall software development life cycle Automation Team Structure Automation Team Automation Automation St Strategy Team t T Delivery Team D li T 4

Automation Strategy Team • Definition: – To provide automation strategic direction and identify potential new automation opportunities • Characteristics: – Project management experience – Various automation frameworks/tools knowledge – Ability to manage multiple projects in parallel – Very strong written and verbal communication skills – Very strong leadership and analytical skills • Activities: – Process & Best Practices – Assessment & Strategies – Design Solutions & Techniques g q – Estimations & Planning – ROI Cost/Benefit Analysis – Reports & Metrics – Tool Technical Support & Analysis Automation Delivery Team • Definition: – To develop, deliver, and maintain automation solutions • Characteristics: – Understanding of programming principles and best U de sta d g o p og a g p c p es a d best practices – Knowledge of various automation tools – Knowledge of multiple programming languages – Strong communication skills • Activities: – Scripts Development – Scripts Maintenance – Coding Standards – Scripts Planning & Execution p g – Day to Day interactions w/End User 5

Benefits of Teams Strategy Benefits • Easy identification of automation opportunities • Easy identification of automation opportunities • Prioritization of automation projects based on ROI • Improved automation design • Optimized the cost of automation • Clearly defined roles & responsibilities y • Better communication process • Optimized automation project planning Automation Lifecycle Phases Automation Evaluation Automation Preparation Automation Generation Execution & Acceptance • Identify the resources • Create Business Process needed. • Understand functionality of Test (BPT) components the project / application • Understand the • Parameterize test data in • Unit testing of requirements with the help requirements with the help • Understand testing • Understand testing BPT components BPT components automation automation application demos. processes • Review components for • Deliver automation to Activities • Create automation • Understand target quality and correctness application team framework / design environments • String the BPT • Execute automation by • Identify timelines and • Understand test data components together to application team milestones details create a test script • Validation of ROI • Training required for • Verify automation tool • Develop automation scripts • Retrospection automation tools compatibility for BPT components • Sign off • Tool installation if required • Identify Automation • Identify test data to opportunities conduct unit & acceptance • Access & Environment testing Setup G2 G2 G3 G3 G1 G1 Deliverables • Preliminary Estimations • Tested automation • Business Process Test • Automation Project Plan scripts • Automation Test Strategy (BPT) components • Framework documentation – • Business Case - ROI • Automated test • Automation Scripts Re-usable components, execution results • Selected automation tool Portability, test data grid • Test Data • Lessons Learned • Detailed Estimations design, etc. • Sign Off • Go, No Go 6

Team Collaboration Automation Automation Strategy Team Delivery Team Automation Automation Automation Execution Evaluation Preparation Generation & Acceptance Integration of Automation Lifecycle & Software Development Lifecycle Automation Feasibility/Analysis Evaluation Automation Design Preparation Automation Implementation Generation Execution & Testing & Acceptance Maintenance 7

Benefits of Process Integration Benefits • Improved understanding on automation process across • Improved understanding on automation process across the entire organization • Better collaboration between business and automation teams • Funding challenges are removed • Improved on-time delivery of automation scripts • Helped to promote the automation across entire organization • Formal escalation process developed • Initialized the creation of automation reports & metrics Automation Coverage – Regression Testing Year 2008 Year-2008 35% 35% Q1-2009 42% Q2-2009 45% Q3-2009 47% Q4-2009 52% 8

Manual Testing Efforts Automated 13,803 hrs 11,750 hrs 10,200 hrs 8,570 hrs 7,047 hrs Year-2008 Q1-2009 Q2-2009 Q3-2009 Q4-2009 Year As of 2008 As of 2009 2009 only efforts MR1 Total Efforts Automated 7047 hrs 13803 hrs 6756 hrs Utilization & Cost Avoidance Utilization 150% 134% 130% 87% 45% Year-2008 Q1-2009 Q2-2009 Q3-2009 Q4-2009 Cost Avoidance 10,810 hrs , 9,867 hrs 9,618 hrs 7,200 hrs Q1-2009 Q2-2009 Q3-2009 Q4-2009 9

Slide 17 MR1 Deleted "I&T" from chart Matt Rosenfeld, 2/1/2010

Questions 10