W8 5/16/2007 1:45:00 PM "A P ROCESS FOR S COPING AND P LANNING T - PDF document

BIO PRESENTATION W8 5/16/2007 1:45:00 PM "A P ROCESS FOR S COPING AND P LANNING T EST A UTOMATION P ROJECTS " Jennifer Seale Nationw ide Insurance International Conference On Software Test Analysis And Review May 14-18, 2007 Orlando, FL USA

Jennifer Seale Jennifer Seale Jennifer Seale holds Master’s degrees in English and Computer Science as well as active CSTE, PMP, and Mercury CPC-QTP designations. She has seven years of experience in the software industry as a developer, tester, test coordinator, and test automation solution architect. Jennifer is a top-rated adjunct faculty member at Franklin University and has taught a variety of undergraduate programming classes over the past four years. For the past two years Jennifer has contributed to the development of the Test Automation Center of Excellence at Nationwide Insurance.

Test Automation Centers of Excellence Jennifer Seale

Impediments to Automation Projects • Lack of experience and knowledge • Lack of resources • Environment instability • Project Process Immaturity • Costs – Initial investment may be large (tools, knowledge, training) – Return on Investment may be long term – Maintenance required 2

Nationwide’s Solution • Centralized Test Automation Team • Experts in designing automation solutions and using automation tools • Partner with projects • Provide various services 3

Developing a Center of Excellence • Set up Shop • Document Review Process • Process for engagements • Templates to support process • Process to manage requests for services • Process to track status • Continuous Improvement • Best Practices and Standards 4

Customer Engagement Process (CEP) Six Phase Process 1. Scoping 2. Planning 3. Development 4. Execution 5. Wrap Up 6. Maintenance 5

Templates Supporting our Process • Initial Screening Survey • Readiness Assessment • Complexity Assessment • Statement of Work (SOW) – Schedule – Cost Benefit Analysis (CBA) – Training and Knowledge Transfer Plan • Proof of Concept (POC) • Customer Development Plan • Test Automation Design Document • Test Automation User Guide • Project Log 6

CEP Phase 1: Scoping • Learn about project and needs • Complete Assessments • Determine scope of engagement • Develop preliminary schedule • Conduct POC • Get approval to continue to planning 7

Scoping: Assessments • Initial Screening Survey – Does automation make sense for the project? – Focus on goals, application, environments, builds • Readiness Assessment – Is the project ready for a test automation effort? – Focus on resources, processes, artifacts, funding • Complexity Assessment – How much effort will it take to automate this project? – Focus on scope, dependencies 8

Scoping: Determine Scope We typically use the following method to determine the scope of the project: 1. Determine automation needs. 2. Break down automation needs into a list of well- defined, manageable units of work. 3. Prioritize the list in terms of functional dependencies. 4. If the prioritized list is long, divide the list into logical sections. These sections will be used during planning to allow a phased approach to automation development. 9

Scoping: Preliminary SOW • Using template as base, add details as they are revealed through conversations with the customer. • Helps to have information in writing • Will become a complete description of all the work involved in the automation effort. • Need to determine preliminary schedule to help in resource planning. 10

Scoping: Proof of Concept (POC) • A POC is a prototype of the automation that demonstrates the technical feasibility of the automation project. • Use Complexity Assessment results to help determine an interesting subset of scenarios to cover in POC. • Automate this subset of test scenarios using the selected automation tool(s) and report results. 11

Scoping: Benefits of a POC • Increased confidence in the project • Early identification of challenges and risks • Early set up of automation environment • Provides useful input to estimating effort 12

Scoping: Customer Development Plan When we say “No” to an automation project, we offer to provide the customer with a document that outlines what needs to be addressed on their project before an automation effort can begin. Most useful when: • the project is interested in automation • automation makes sense for the project • other factors exist that put the success of an automation effort at risk 13

CEP Phase 2: Planning • Refine SOW • Refine Schedule • Perform Cost Benefit Analysis • Training and Knowledge Transfer Plan • Test Automation Design Document • Approval for SOW 14

Planning: Refining SOW • Finalize communication plan • Finalize details of objectives and deliverables • Finalize details of schedule • Finalize cost estimates 15

Planning: Cost Benefit Analysis • Using template as a base, tailor to fit project specifics. • Estimates all costs associated with automation effort. • Predicts benefits of automation effort based on how often the project will run the automation once it is complete. • Predicts the breakeven point, when the benefit is equal to the cost. 16

Planning: Automation Costs When planning for an automation, consider the following costs: – Staff Resources – Training – Learning curve – Initial Development – Execution and Analysis – Maintenance – Enhancements – Hardware – Support 17

Planning: Automation Benefits There are three types of benefit that can be realized through Test Automation: • Reduction –benefit realized when a test suite that is currently being executed manually is replaced by an automated suite. • Coverage –benefit realized when an automated suite is created or extended to perform testing that is not currently being executed manually due to lack to resources or funds. • Quality –benefit realized by detecting defects earlier in the software development life cycle due to the automation. 18

Return on Investment Generic Automation Savings Profile 4000 3500 3000 Break Initial 2500 Even Coverage Hours 2000 Investment 1500 1000 Reduction 500 Quality 0 -500 -1000 Jan-06 Mar-06 May-06 Jul-06 Nov-06 Jan-07 Mar-07 May-07 Jul-07 Nov-07 Jan-08 Mar-08 May-08 Jul-08 Nov-08 Jan-09 Mar-09 May-09 Jul-09 Nov-09 Sep-06 Sep-07 Sep-08 Sep-09 Months

Planning: Training and Knowledge Transfer Plan • It is important that we prepare key members of the project team to take over the automation. • We tailor this plan specifically to meet the needs of the project. • Often this plan contains its own schedule. • Typically produce a Test Automation User Guide. 20

Planning: Test Automation Design Document • Documents the automation design for the project. • Provides an overview of exactly what was automated for Test leads to use when planning future testing efforts. • Documents all design decisions and workarounds that were employed. 21

Approval of SOW • The completed SOW is sent to the customer stakeholders for review. • We conduct a face-to-face meeting where we walk the customer through the SOW to ensure we are in agreement. • Once the SOW is approved, development begins. 22

CEP Phase 5: Wrap Up • Complete Training and Knowledge Transfer and Plan • Update SOW with results of engagement • SOW Complete approval meeting • Customer Satisfaction Survey • Engagement Retrospective 23

CEP Phase 6: Maintenance • Follow up with projects every 6 months for life of project • Update CBA with actuals • Address noteworthy issues 24

Managing Requests for Services • Spreadsheet of requests established • Tab per request with overview page • Log all pre-engagement contact • Used for tracking and forecasting • Available on shared drive, all team members responsible for keeping up- to-date. 25

Tracking Status: The Project Log • Spreadsheet with multiple tabs • Tab per document to track change requests • Tab to track technical issues • Tab to track resource issues • Tab for lessons learned • Always available for review • Wealth of historic data for future projects 26

Continuous Improvement • Document Change Request (CR) System – Survey Results – Lessons Learned – Retrospectives • Quarterly reviews to address CRs and improve the process and associated templates 27

Best Practices and Standards • Perfectly positioned to tailor theoretical best practices to specific business • Reuse solution models across applications • Develop standards for specific technologies used by the business 28