Improving Your Quality Process A Practical Example John Balza johnjbalza@comcast.net John Balza Page 1 Pacific Northwest Software Quality Conference
Objectives To provide an example of how to improve your software quality process Agenda: • Initial Situation • Defect Analysis Targets Improvements • Justify the Investment • Creating a Test Model • Results John Balza Page 2 Pacific Northwest Software Quality Conference
HP-UX Statistics/Organization • Approximately 18 Million lines of code total • New or changed code – 1.4 Million in 11.00 – 3.0 Million in 11.11 – 6.0 Million in 11.23 • HP-UX Code is developed by ~ 1500 people in 13 labs in 7 geographical locations. • Each Lab responsible for – maintaining existing code – developing new functionality – quality of their code. • One lab responsible for end to end quality – System testing, solution testing, beta test John Balza Page 3 Pacific Northwest Software Quality Conference
Customer Situation Results from Interex Engineering Investment Survey, 1999 (HP user group) • Most important strategic directions for HP in next 5 years – 1. Keeping customer costs down – 2. Developing higher quality software John Balza Page 4 Pacific Northwest Software Quality Conference
HP-UX Quality Improvement Program “10X in 5 Years” GOAL: Decrease customer found defects by a factor of 10 Internal Goals • Drive up the defects prevented or found before check-in – Reduce defects found after check-in by 2x every 18 months • Increase the defect removal rate between check in and customer release – Have our system testing better reflect our customers environments John Balza Page 5 Pacific Northwest Software Quality Conference
Defect Analysis Know what defects are escaping and remove the causes John Balza Page 6 Pacific Northwest Software Quality Conference
Required a Defect Analysis What are the most common types What is the root cause? of defects escaping? How could we best remove these types of defects 100 50 Get beyond best practice 0 What specifically needs to Req Code be done in reviews, training, tests, etc . Required a defect analysis on all defects that had escaped a development team. Typical recommendations: • HP-UX Internals training • Better communication of dependencies with other teams • Specialized peer review for ‘locking’ issues • Testing improvements John Balza Page 7 Pacific Northwest Software Quality Conference
Testing Themes From defect analysis we found several common themes: • Over 60% of our defects were found in the ‘core’ of the operating system (networking, kernel, commands, libraries) • Over 50% of escaped defects were ‘functional defects’ • Another 10% were ‘configuration defects’ – problems that would occur only in certain configurations, typically high end SPUs Developers only found a fraction of their defects • 50% of defects were found in testing by other teams (25% by system test) • 12% were found by customers John Balza Page 8 Pacific Northwest Software Quality Conference
ROI Justification Provide management with a solid cost-saving argument using your or industry data John Balza Page 9 Pacific Northwest Software Quality Conference
Defects injected per KNCSS Phase Defects Introduced Requirements 10.1 Design 12.8 Code 17.2 Documentation 4.0 Bad Fixes 5.0 Total 49.1 ���������������������������������������������������������������� �������������������� ���!"#�$%�&�%�%%%�' ����(�����)����* John Balza Page 10 Pacific Northwest Software Quality Conference
Defect removal efficiencies Lowest Modal Highest Removal Efficiency Efficiency Efficiency Steps 25% 35% 50% Requirements inspection 25% 30% 40% Informal design reviews Formal design inspections 45% 65% 85% 20% 30% 45% Informal code reviews 45% 65% 85% Formal code inspections 20% 40% 60% Code desk check 15% 30% 55% Unit Testing 20% 30% 40% New Function Testing 10% 15% 30% Regression Test 25% 35% 45% Integration test 15% 20% 30% Performance Test 25% 35% 55% System Test 20% 30% 40% Beta Test <10 sites ������������������������������������������������������������������������������� John Balza Page 11 Pacific Northwest Software Quality Conference
Defect Detection Costs: hours/defect ~18 Requirements inspection ~18 Informal design reviews ~10 Formal design inspections 9 Informal code reviews +����������������� 15 ��,����+-������������"����.�,��� Formal code inspections ���� 9 Code desk check 7.5 Unit Testing (by developer) +��������������� 12.5 New Function Testing ��������������� 5 ���������/���0����� Regression Test ���� ~15 Integration test ~15 Performance Test 15 System Test 10 Beta Test <10 sites John Balza Page 12 Pacific Northwest Software Quality Conference
Defect Fix Costs: hours/defects 1��������+��� .5 Inspections/reviews ~.5 Code desk check 1������������ 2.5 Unit Testing (by developer) 5.0 New Function Testing 1��������+��� 12 Regression Test 12 Integration test 2������������������������� 20 Performance Test ,���������3%#$%�4���� 20 System Test ~20 Beta Test <10 sites 40 Customer reported defect (fix and patch once) John Balza Page 13 Pacific Northwest Software Quality Conference
Our Typical Defect Removal with design walk through, code desk check, tests for new functionality and regression system tests Today's typical practice Customer sees 5.7 defects/KNCSS Total cost 619 hours/KNCSS 50 45 40 35 Defects injected 30 Defects removed 25 Cumm injected 20 Cumm removed 15 10 5 0 s t t e n s s t d g e n e o i T T e s C m e r m D e e e n r t i g u s i q s y S e e D R John Balza Page 14 Pacific Northwest Software Quality Conference
Improvement With Inspections Add in rigorous inspections Customer sees 2.8 defects/KNCSS Total Cost 356 hours/KNCSS 50 45 40 35 Defects injected 30 Defects removed 25 Cumm injected 20 Cumm removed 15 10 5 0 s t t e n s s t d g e n e i o e T T s C m e r m D e e n e r g t i u s i y q s S e e D R 2X improvement with 42% less labor John Balza Page 15 Pacific Northwest Software Quality Conference
Inspections/Peer Reviews • Retrained every engineer on the Inspection Process • Set-up a database to record the peer review results • Required inspections of all design documents and some form of peer review on all code • Started development of checklists for the peer reviews • Many labs established a Plan-Do-Check-Act cycle with quarterly reviews by management John Balza Page 16 Pacific Northwest Software Quality Conference
With Inspections & More tests Inspections, Unit, Subsystem Integration, Beta Test Customer sees 1.0 defect/KNCSS Total Cost: 304 hours/KNCSS 50 45 40 35 Defects injected 30 Defects removed 25 Cumm injected 20 Cumm removed 15 10 5 0 s e t t n s s t d g e n e i o T e s T C m e r m D e e e n r i g t u s i q s y S e e D R Another 3X quality improvement with 15% less labor John Balza Page 17 Pacific Northwest Software Quality Conference
Changed Internal Goals Combination of defect analysis and this ROI model resulted in management endorsement of new internal goals • Development team finds 90% of their own defects (70% through peer review, 20% through testing) • Partner/System Test finds 90% of remaining defects • Customer only finds 1% John Balza Page 18 Pacific Northwest Software Quality Conference
Creating a Test Model Create a process model: the current model and target model John Balza Page 19 Pacific Northwest Software Quality Conference
Original Test Model Tests of subsystem Tests of entire product White Box Subsystem IC System Solution Test Test Test Test Test No criteria >95% pass on >95% pass on >95-99% 0 defects Functional functional test pass on running with test (usually functional test key layered on a couple on 1 SPU) (more varied applications configurations and larger config) Run stress 96 continuous tests on small hours of configuration operation on stress tests John Balza Page 20 Pacific Northwest Software Quality Conference
Recommend
More recommend
