T3 Concurrent Session Thursday 10/25/2007 9:45 AM JUMP TO: Biographical Information The Presentation The Ten Most Important Automation The Ten Most Important Automation Questions—and Answers Questions—and Answers Presented by: Mukesh Mulchandani and Krishna Iyer, ZenTEST Labs Presented at: The International Conference on Software Testing Analysis and Review October 22-26, 2007; Anaheim, CA, USA 330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 � 904-278-0524 � sqeinfo@sqe.com � www.sqe.com
Mukesh Mulchandani Mukesh Mulchandani Mukesh Mulchandani (co-author), is the CTO of ZenTEST Labs, an independent software testing solutions company based out of India. Mukesh has 7 years of work experience in Software Testing and automation consulting. Mukesh has experience in servicing outsourced testing having worked on test automaton projects with clients such as S1 Corporation, Kanbay, Capgemini Consulting and some of the Fortune 500 clients like Morgan Stanley and Merrill Lynch. He has played a major role in designing functional automation processes at company level. Mukesh has also provided testing consulting services to 2 of the top 5 IT consulting companies in the world. Mukesh is a certified Mercury Product consultant for WinRunner and QTP and also a Certified Software Test Engineer from QAI USA. Mukesh has co- authored whitepaper for various conference including STAREAST and STARWEST Krishna Iyer Krishna Iyer Krishna Iyer is the CEO of ZenTEST Labs, an independent software testing solutions company based out of India. Krishna has 9 years of Quality management and Process consulting experience. Krishna has experience in test outsourcing of large multi-site projects having worked with clients such as GE, CitiFinancial, IBM, HSBC, etc both in India and the United States. He also has participated in CMM assessments and has been one of the key players in taking a large software integrator to CMMi Level 5. Krishna is a qualified accountant by education and a certified software quality analyst as well. He is a prolific speaker and writer and has presented in various conferences including STARWEST and STAREAST. Krishna has made various presentations on automation frameworks.
The Ten Most Important Automation Questions and Answers Krishna Iyer and Mukesh Mulchandani Star WEST 2007
Objective of this presentations � To present choices every test automation project must exercise (the first 5 questions) � To project future trends in automation (the last 5 questions) � To discuss unique concepts such as vertical vs. horizontal automation � To demonstrate advanced automation concepts � To share ideas for increasing ROI of your automation project based on ZenTEST Labs’ experience 2 StartWest 2007
The ten most important automation questions Five Choices Five Trends 1. To automate now or automate 1. Keyword or functional later? (the time choice) decomposition? 2. To automate this or automate (The combined trend) that? (the test case choice) 2. Building extensibility or not? 3. Vertical Automation or (The user trend) Horizontal Automation? (the 3. Whether to introduce flow choice) automated test driven 4. Test data hard coded or Test development (The ‘ time to data kept reusable? (the obj ect market’ trend) oriented choice) 4. Whether to automate and 5. How To handle dynamic obj ects? execute early (The ‘ unit test’ (the obj ect map choice) trend) 5. Whether to offer automation scripts to the client (the selling trend) 3 StartWest 2007
Q#1: To automate now or automate later… ▲ Key to successful test automation is knowing WHEN to automate. ▲ Often overlooked and sometimes under valued. ▲ Even best automation approach fails to give higher ROI because of wrong timing of test automation. As it is � Pointless if it is at the fag end of a product lifecycle � Painful if the product is unstable 4 StartWest 2007
Q#1: To automate now or automate later… contd. � Most important factors to be considered before starting automation. � Is the application/module that is planned for automation stable enough for automation. � Is manual testing process in place � Detailed test conditions and pre conditions. � Accurate Test data and expected result. � When your sure of ROI paying off 5 StartWest 2007
Q#1: To automate now or automate later… contd. What defines “Stable Enough” � Core functionality & navigation flow is approved and accepted by end client. � No critical defects to be fixed around the functionality which will change the application from its current state. � No planned major enhancements in the functionality for min next 3 regression rounds. 6 StartWest 2007
Q#2: To automate this or automate that � After knowing when to automate, its critical to know what to automate and what not to automate. � Remember: Its not feasible to automate all the test cases. � Clear test cases selection criterion should be available with test automation team to improve effectiveness of automation suite . 7 StartWest 2007
Q#2: Test Case Selection Criterion ▲ Is the test case repeatable? ▲ Does this test case require manual intervention? ▲ Has the test case passed manual verification? ▲ Are all the preconditions for the test case taken care-of? ▲ Are the execution steps very clear? ▲ Do we have test data for this test case? ▲ Is the expected result clear enough to decide the test case status (PASS & FAIL)? ▲ Will the test case survive the functional changes around it? ▲ Is the test case straightforward for automation? ▲ Can I trust this script to really test this part of the feature? 8 StartWest 2007
Q#3: Vertical OR Horizontal Automation. Horizontal Automation Automating End to End Flow where each test cases is independent of each other such as Logout->Login(user2) Approve payment1 Create payment1 TC1: Login(user1) Logout->Login(user2) Reject payment1 Create payment2 TC2: Login(user1) Vertical automation is executing test cases feature wise to save Automation execution time. e.g. Vertical Approve payment1 Create payment1 Logout->Login(user2) TC1&2: Login(user1) Reject payment2 Create payment2 9 StartWest 2007
Q#3: Vertical OR Horizontal Automation. Advantages of Horizontal Automation ▲ Flexibility in running any test case as each test case is independent of other. ▲ Easy to organize and automate. ▲ Test log is created the movement each test case execution is over Advantages of Vertical Automation ▲ Faster test execution as navigation is minimized, (such as login -logout) ▲ Effective when same functional flow is tested with different data sets 10 StartWest 2007
Q#4: Test data hard coded or test data kept reusable ▲ An good automation architect ensures that scripts are reusable. ▲ And a great automation architect ensures that scripts and test data, both, are reusable. 11 StartWest 2007
Q#4: Test data hard coded or test data kept reusable ▲ Generally , there is more emphasis on reusing test script than reusing test data. ▲ Generally one test set is used per test cases. For e.g. approve payment test case and reject payment test case may use same user (i.e. user1) but the test data sheet is stored within the test case and gets repeated again and again for each test case. ▲ On the other hand to keep the test data reusable: � Store data screen wise and not test case wise � Assign reference id to each test data. � Pass reference id to each test case for accessing the test data. 12 StartWest 2007
Q#5: How to handle dynamic objects; object map or advance scripting? ▲ Tip1. Store different unique properties than what is normally stored. e.g. if a button label is dynamically changing again and again, then store button class and msw_id instead of storing class and label . ▲ Tip2. Don’t store the object in object map/ repository. Create the object dynamically through scripting and use it. ▲ When to use advance scripting for dynamic objects. � When objects are dynamically changing. � When similar types of objects are available with minor logical differences. e.g. pagination, where list of records are displayed on diff pages, links to these pages are similar objects. The page number is different and gets incremented by 1. Instead of storing all the 100 page links, here the engineer needs to store one object and change the page number property dynamically, this reduces the object count in the repository, which saves lot of RAM. 13 StartWest 2007
Q#6: Keyword driven or Functional decomposition approach Functional decomposition (FD) Keyword driven (KD) In the functional decomposition In the Keyword driven approach, business processes approach, each business are created first and while process is mapped into actions creating the test scripts, each and further each operation is business process is called in a mapped as a keyword. It is sequence. easy for non technical users to This approach is modular but for create test scenarios without every test case, a test script is knowing much of the testing required. In this approach tool. Scripts are not modular scripts are maintainable to the and major advantages of extent that implementation of the business process is not functional decomposition are changed. lost. 14 StartWest 2007
Recommend
More recommend
