agilegurgaon com
play

www.agilegurgaon.com Meetu Arora Sr V.P. Quality Assurance - PowerPoint PPT Presentation

Oct 04, 2023 •433 likes •760 views

www.agilegurgaon.com Meetu Arora Sr V.P. Quality Assurance Naukri.com www.agilegurgaon.com Testing Team Journey @Naukri.com Automation WATIR 2007 2007 2006 2006 2008 2008 Agile testing team 2010 2010 Inception of 2012 2012

  1. www.agilegurgaon.com

  2. Meetu Arora Sr V.P. Quality Assurance Naukri.com www.agilegurgaon.com

  3. Testing Team Journey @Naukri.com Automation WATIR 2007 2007 2006 2006 2008 2008 Agile testing team 2010 2010 Inception of 2012 2012 Automation Testing Team 2015 QTP Automatio Aut ion Sel Seleniu ium Entir En ire team eam: Ma Manual l & & www.agilegurgaon.com Aut Automatio ion (Sele (Seleniu ium)

  4. • Identify Need • Identify Metrics/What • Identify Path/How • Prerequisites • Implementation • Our Measurements • Results www.agilegurgaon.com

  5. www.agilegurgaon.com

  6. Identify Metrics/What www.agilegurgaon.com

  7. Identify Path/How www.agilegurgaon.com

  8. Prerequisites • Testing Team capable of doing automation • Test case consolidation & management • Dedicated scrum teams v/s shared resources • Metric Baselines  Post live defect seepage  Test case coverage  Automation coverage – UT, IT, FT  Automation test flakiness  Automation Execution time  Velocity/Delivery  Planning Efficiency  Build Quality www.agilegurgaon.com

  9. Implementation Implementation www.agilegurgaon.com

  10. Transition to Automation 35 Manual testers & 3 Automation testers TO 38 Manual + Automation = POWER testers www.agilegurgaon.com

  11. Transition to Automation - Challenges • Skill mismatch • Team dynamics • Lack of inclination to move towards automation • High investment in terms of time and effort www.agilegurgaon.com

  12. Transition to Automation - Path • Perseverance • Don’t fall back • Very Small Steps • Low hanging fruit first - ROI • Tester empowerment through automation • Focus on frameworks • We created Selenium POM code generator, which has been open sourced: https://github.com/naukri- engineering/SeleniumCodeGenerator • TestNG and XSLT for reporting • Generic Function library • Coding Guidelines and Code Review Process • Contests for motivation • Provide migration channels www.agilegurgaon.com

  13. Test Case Consolidation and Management To know more about how to integrate automation scripts with Jenkins visit our blog: http://engineering.naukri.com/2015/05/integrate-your-automation-with-jenkins/ www.agilegurgaon.com

  14. Testing Throughout • Testing individual stories and Integration testing as you go along • Progressive/Parallel automation testing • Minimizing Automated Tests Flakiness • Reducing our automated tests execution time • Run automated regression suites periodically using Jenkins To know more about Continuous Testing @ Naukri read our blog: http://engineering.naukri.com/2016/03/continuous-testing-naukri/ www.agilegurgaon.com

  15. Automation Coverage Naukri India 96.65% 92.10% 95.96% 96.99% 92.72% 16000 100% 75.67% 90.82% 86.29% 84.55% 81.56% 14000 74.32% 68.76% Automation Coverage 80% 12000 No. of Test cases 10000 60% 10641 8000 10542 10846 10491 10283 40% 10062 6000 9313 8925 8679 8318 8153 7746 4000 20% 2000 3047 2636 2249 1859 1508 1228 1354 929 876 596 554 611 0 0% April May June July August Septemebr October November December January February March Not Automatable Automatable %Automation Coverage out of Automatable Naukri Gulf 93.20% 93.00% 92.90% 88.10% 7000 100% 84.60% 76.80% 6000 Automation Coverage 71.50% 80% No. of Test cases 5000 58.00% 46.30% 60% 4000 29.29% 6242 5885 5784 5861 5658 27.80% 3000 5145 40% 25.00% 4576 4436 3859 2000 3077 3072 20% 1000 1818 309 347 317 336 339 422 417 402 410 415 417 0 124 0% April May June July August Septemebr October November December January February March Not Automatable Automatable %Automation Coverage out of Automatable www.agilegurgaon.com

  16. Automation Flakiness Reduction • 15% to 5% in Naukri • 45% to 7% in Mobile Apps • 17% to 7% in NaukriGulf To know more about how we were able to optimize our tests and reduce flakiness visit our blog: http://engineering.naukri.com/2016/03/reduce- test-automation-flakiness/ www.agilegurgaon.com

  17. Automation Execution Time Reduction By selenium grid implementation we were able to reduce our execution time to 1/5 th – From 25Hrs to 5Hrs To know more about Selenium Grid implementation visit our blog: http://engineering.naukri.com/2015/10/parallel-testing-at-naukri/ www.agilegurgaon.com

  18. Preventing Bugs • Tester, developer, product owner, architect are all part of the backlog grooming • Entire team focuses on defining what and how • Product backlog grooming is done one iteration in advance • Testers contribute test cases upfront during this period and add them to the user stories in form of acceptance criteria or alternate paths www.agilegurgaon.com

  19. Preventing Bugs • Peer testing at developer end • Automated Build verification tests have been created and are run before providing builds to testers • Progressive Automation testing approach is used. www.agilegurgaon.com

  20. Testing Understanding • Put yourself in the customer shoes • We encourage our scrum teams to have direct interaction with actual customers – proactive & reactive • Effective feedback loops • Our tech support team regularly shares reports on issue patterns which are used by scrum teams as inputs to design/test/improve systems • Measure the customer usage pattern and use it to design test cases • We regularly analyse user data patterns to come up with and refine our test strategy www.agilegurgaon.com

  21. Building the best system • Build Implicit Requirements • Focus more on building positive product scenarios • Focus on Bug Causal Analysis • Peripheral testing: Focus v/s Defocus To know more about peripheral testing please visit our blog http://engineering.naukri.com/2016/03/peripheral-testing/ www.agilegurgaon.com

  22. Team responsibility for quality • Everybody tests as and when needed • Measure Quality at various levels • Build Quality Meter • Post Production Issue Seepage www.agilegurgaon.com

  23. Build Quality Meter Bug Severity Bug Type: Functional, UI, product design, Implicit, Validation, Incomplete requirement, Insufficient impact analysis, Integration Environment, Configuration, Inadequate testing,DOA www.agilegurgaon.com

  24. Our Measurements Indicative Data has been used for illustration purpose Velocity Report 50 45 40 35 30 25 20 15 10 5 0 Sprint15-28Dec Sprint29Dec-11Jan Sprint12-25Jan Sprint26Jan-8Feb Sprint9-22Feb Sprint23Feb-7March Sprint8-21Mar Commitment 39 40 44 42 44 42 46 Completed Unplanned P1/P2 2 3 2 5 3 4 2 Completed Unplanned 6 5 1 2 4 2 5 Planned Live/Staging 6 1 2 10 5 3 4 Completed Planned 19 27 39 25 30 32 34 Target 41 41 41 41 41 41 41 www.agilegurgaon.com

  25. Our Measurements Indicative Data has been used for illustration purpose Build Quality Trend 100 100 Excellent 93 [90 +] 90 90 Good 80.86 Excellent [80-90] 80 80 70.68 Average [70-80] Good 70 70 BUILD QUALITY % Bad Average 60.76 60 60 [Less than 70] Bad BUGS 50 50 40 40 30 30 20 20 11 9 7 10 10 5 5 4 4 3 2 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16 ITERATION Functional Bugs UI Bugs Validation Live Implicit Integration Build Quality www.agilegurgaon.com

  26. Our Measurements Indicative Data has been used for illustration purpose Functional Automation Coverage 18 17 16 14 14 13 13 12 11 11 NO. OF STORIES 10 9 8 6 4 4 4 2 2 1 1 0 26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16 Total Stories Automatable Automated www.agilegurgaon.com

  27. Our Measurements Bugs Reported on Live 10 9 8 7 6 BUG COUNT 5 4 3 1 4 2 1 3 0 1 2 2 3 3 1 2 1 0 26 Jan - 8 Feb 9 -22 Feb 23 Feb-7 Mar 8-21 March'16 Client Reported Issues (Data Fixes) 2 4 3 1 Client Reported Issues (Code Fixes) 1 2 0 1 Live issues (Except client issues) 2 1 3 3 Indicative Data has been used for illustration purpose www.agilegurgaon.com

  28. Results  Post live defect seepage: 50% reduction  Test Cases: 10K increase  Automation Coverage: 44% increase  Automation Scripts Execution time: 20% reduction  Automation flakiness: 20% reduction  Velocity: 25% increase  15% Build quality improvement  30 % Planning Efficiency improvement www.agilegurgaon.com

  29. Key Take Away’s • Defining the “ Right Metric ” • Measure, Review, Improve …. • For Agile testing through out focus should be on progressive/parallel automation testing along with creating reliable tests that take minimal time to execute . • Defect Prevention is the Key • Team v/s Individual mind-set www.agilegurgaon.com

  30. www.agilegurgaon.com

  31. www.agilegurgaon.com

More recommend