BIO PRESENTATION PAPER F1 11/18/2005 10:00 AM L ET ' S M AKE B UGS M ISERABLE Anibal Sousa Microsoft Corporation International Conference On Software Testing Analysis & Review November 14-18, 2005 Anaheim, CA USA
Anibal Sousa Anibal Sousa is a Test Manager for Microsoft in the Microsoft Business Solutions division, working in the Business Contacts Manager for Outlook product. He has about 15 years of experience working in many IT departments, from software development to customer support. Anibal joined Microsoft in 1998, and since then has worked in the Testing discipline shipping many different products, like Exchange Conferencing and Instant Messaging Servers, GDI+ and Business Contact Manager for Outlook Versions 1 and 2. He is passionate about Testing and Quality Assurance, always looking for ways to improve the software development and testing processes, with focus on new methodologies and practices, like Test Templates, Risk Based Testing, Model Based Testing, etc. Anibal has a Master of Science degree in Computer Science from PUC/RJ in Brazil, and loves his family and soccer.
Let’s Make Bugs Let’s Make Bugs Miserable Miserable Anibal Sousa Anibal Sousa Anibal.Sousa@microsoft.com Anibal.Sousa@microsoft.com Test Manager Test Manager Microsoft Corp. Microsoft Corp.
Scope of this presentation - Agenda Scope of this presentation - Agenda � Defining a bug database Defining a bug database � � Spec critique process Spec critique process � � Pre check in tools Pre check in tools � � Looking for more and different bugs Looking for more and different bugs � � Bug bar and Triage process Bug bar and Triage process � � Bug charts and data analysis Bug charts and data analysis � � Making best use of bugs found by customers Making best use of bugs found by customers �
Who am I? Who am I? � 8 years working for Microsoft in Test area 8 years working for Microsoft in Test area � � Products I worked on: Products I worked on: � � Instant Messaging Server 2000 Instant Messaging Server 2000 � � Conferencing Messaging Server 2000 Conferencing Messaging Server 2000 � � GDI+ versions 1.0 and 1.1 GDI+ versions 1.0 and 1.1 � � Business Contact Manager versions 1.0 and 2.0 Business Contact Manager versions 1.0 and 2.0 � � I will be talking about ideas, processes and practices I will be talking about ideas, processes and practices � developed and tested during these 8 years developed and tested during these 8 years
Motivation Motivation � Bugs will always be there! Bugs will always be there! � � Can we prevent the creation of bugs? Can we prevent the creation of bugs? � � How do we keep control of bugs? How do we keep control of bugs? � � Are we fixing the right bugs? Are we fixing the right bugs? � � Are they being fixed correctly? Are they being fixed correctly? � � Where should I look for more bugs? Where should I look for more bugs? � � What bugs did I miss? What bugs did I miss? �
First, you need a bug database, but ... First, you need a bug database, but ... � Can you afford to buy one? If not, you could use Can you afford to buy one? If not, you could use � Excel, SQL, etc. Excel, SQL, etc. � After you have it, what kind of data will you After you have it, what kind of data will you � store? store? � It should simplify the bug management process, It should simplify the bug management process, � for you and your team members for you and your team members � And after you start colleting data, what can you And after you start colleting data, what can you � do with it? do with it?
Life cycle of a bug Life cycle of a bug Triage inspects bug and Bug gets into Bug is verified Bug gets resolved assigns to developer the system then closed Bug is open Active bug Bug is but not assigned to Resolved bug Closed bug found assigned developer Bug was not resolved correctly Bug goes back to Triage for orientation
Some key fields the DB should have: Some key fields the DB should have: � Title Title � Assigned to (owner) Assigned to (owner) � � � Repro Steps Repro Steps � Priority Priority � � � Feature area Feature area � Resolution Resolution � � � Severity Severity � History log History log � � � Status Status � Opened by and date Opened by and date � � � Type Type � Resolved by and date Resolved by and date � � � How found How found � Closed by and date Closed by and date � � � Test Case ID Test Case ID � UA Impact UA Impact � � � Release Release � Triage status Triage status � �
Spec Critique/Review, driven by Test Spec Critique/Review, driven by Test 4b - Yes Spec Draft - Reviewed Spec - Are there open 2 - Feature team provides Ready for Review Ready for Critique issues with Spec input to the Spec document document? 1 - PM creates draft of Spec based on requirements 3 – Team members review the spec, and additional information and critique meeting occurs 4a - No Spec Ready - Ready for Coding Requirements, feedback, surveys, bugs, etc.
Killing bugs before they get you Killing bugs before they get you � Code Review (dev and test) Code Review (dev and test) � � Buddy Build and Buddy Test Buddy Build and Buddy Test � � Pre Check in tools Pre Check in tools � � Automated daily builds Automated daily builds � � Automated Code Coverage builds Automated Code Coverage builds � � Automated tests and tools execution Automated tests and tools execution �
Looking for different bugs Looking for different bugs (and the bugs you missed) (and the bugs you missed) � Focus and Ad hoc days Focus and Ad hoc days � � Bug hunts Bug hunts � � Bug bashes Bug bashes � � Feature rotation between testers Feature rotation between testers � Different eyes will find different problems!
Bug Bar – motivation (1/2) Bug Bar – motivation (1/2) � 5000+ bugs during product development life 5000+ bugs during product development life � cycle cycle � Subjective and not deterministic process for Subjective and not deterministic process for � triaging bugs triaging bugs � Testers were confused about what kind of bugs Testers were confused about what kind of bugs � were still being accepted – they wanted to be able were still being accepted – they wanted to be able to focus on the right set of problems to to focus on the right set of problems to investigate investigate � Bad for morale and team engagement Bad for morale and team engagement �
Bug Bar – motivation (2/2) Bug Bar – motivation (2/2) � Bugs will occur during the whole project Bugs will occur during the whole project � � Quality should go up and not down Quality should go up and not down � � Risk of regression gets higher with time Risk of regression gets higher with time � � Not all features get ready at the same time Not all features get ready at the same time �
Bug Bar is the proposal! Bug Bar is the proposal! Define the bug categories: Impact Name Definition Reasonable region of a feature is not working as expected because of the bug, and 1 Ship Stopper there is no workaround. We can not ship the product with this bug! 2 QFE Serious bug. If a customer finds this bug, we will have to issue a QFE (patch). This bug will cause Customer Support calls. Publishing KB article is not enough, 3 PSS since workaround may not exist or be too complicated. Knowledge Base (KB) This bug might be very visible and affect functionality significantly. In case it has a 4 article workaround, it is not so obvious or simple. This is a bug, but might be obscure, rare or have small impact. Normally it has an easy 5 Bug workaround. This bug is hard to find, not noticeable, causes minor problems (or none) and can be 6 Improvement ignored. 7 Wish These might not be even considered bugs.
Bug Bar @ Alpha stage Bug Bar @ Alpha stage Import SBA User Areas Reports Export Forms Performance PDA Integration Assistance � � � � � � � 1 Ship Stopper � � � � � � � 2 QFE � � � � 3 PSS � � � 4 KB � 5 Bug � 6 Improvement 7 Wish � Each feature is considered independently Each feature is considered independently � � Each feature will have its bar changed thru time Each feature will have its bar changed thru time � � When deciding bar, consider: new/old feature, risk and When deciding bar, consider: new/old feature, risk and � impact of changes, development stage, coverage, etc. impact of changes, development stage, coverage, etc.
Bug Bar @ Beta and RTM stages Bug Bar @ Beta and RTM stages Import SBA User Areas Reports Export Forms Performance PDA Integration Assistance � � � � � � � 1 Ship Stopper � � � � � � � 2 QFE � � � � 3 PSS � � 4 KB 5 Bug Import SBA User Areas Reports Export Forms Performance PDA Integration Assistance � � � � � � � 1 Ship Stopper � � 2 QFE 3 PSS 4 KB
Recommend
More recommend