Learning objectives View the big picture'' of software quality in - PowerPoint PPT Presentation

Learning objectives • View the “big picture'' of software quality in the context of a software development project Software Test and Analysis and organization: in a Nutshell • Introduce the range of software verification and validation activities • Provide a rationale for selecting and combining them within a software development process. (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 1 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 2 Engineering processes Verification and design activities • Sophisticated tools • Verification and design activities take various forms – amplify capabilities – suited to highly repetitive construction of non- – but do not remove human error critical items for mass markets • Engineering disciplines pair – highly customized or highly critical products. – construction activities with • Appropriate verification activities depend on – activities that check intermediate and final products – engineering discipline • Software engineering is no exception: – construction process construction of high quality software requires – final product – construction and – quality requirements. – verification activities (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 4

Peculiarities of software Impact of new technologies Software has some characteristics that make • Advanced development technologies V&V particularly difficult: – can reduce the frequency of some classes of errors – Many different quality requirements – but do not eliminate errors – Evolving (and deteriorating) structure • New development approaches can introduce – Inherent non-linearity – Uneven distribution of faults new kinds of faults examples Example – deadlock or race conditions for distributed software If an elevator can safely carry a load of 1000 kg, – new problems due to the use of polymorphism, it can also safely carry any smaller load; dynamic binding and private state in object-oriented If a procedure correctly sorts a set of 256 elements, it may fail on a set of 255 or 53 or 12 elements, software. as well as on 257 or 1023. (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 5 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 6 Variety of approaches Five Basic Questions • There are no fixed recipes 1. When do verification and validation start? When are they complete? • Test designers must 2. What particular techniques should be applied – choose and schedule the right blend of techniques during development? • to reach the required level of quality • within cost constraints 3. How can we assess the readiness of a product? – design a specific solution that suits 4. How can we control the quality of successive • the problem releases? • the requirements • the development environment 5. How can the development process itself be improved? (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 7 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 8

1: When do V&V start? Early start: from feasibility study When are they complete? • Test is not a (late) phase of software • The feasibility study of a new project must take development into account the required qualities and their impact on the overall cost – Execution of tests is a small part of the verification and validation process • At this stage, quality related activities include • V&V start as soon as we decide to build a – risk analysis software product, or even before – measures needed to assess and control quality at each stage of development. • V&V last far beyond the product delivery as long as the software is in use, to cope with – assessment of the impact of new features and new quality requirements evolution and adaptations to new conditions – contribution of quality control activities to development cost and schedule. (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 9 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 10 2: What particular techniques should Long lasting: beyond maintenance be applied during development? • Maintenance activities include No single A&T technique can serve all purposes The primary reasons for combining techniques are: – analysis of changes and extensions – Effectiveness for different classes of faults – generation of new test suites for the added example: analysis instead of testing for race conditions functionalities – Applicability at different points in a project – re-executions of tests to check for non regression of example: inspection for early requirements validation software functionalities after changes and – Differences in purpose extensions example: statistical testing to measure reliability – Tradeoffs in cost and assurance – fault tracking and analysis example: expensive technique for key properties (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 11 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 12

Staging A&T techniques 3: How can we assess the readiness of a product? Requirements Requirements Architectural Detailed Integration & Unit Coding Maintenance Elicitation Specification Design Design Delivery Planning & monitoring Identify qualites Plan acceptance test • A&T during development aim at revealing faults Plan system test Plan unit & integration test Monitor the A&T process • We cannot reveal are remove all faults Verification of specs Validate specifications Analyze architectural design • A&T cannot last indefinitely: we want to know Inspect architectural design Inspect detailed design Code inspection if products meet the quality requirements Generation of tests Generate system test Generate integration test Generate unit test • We must specify the required level of Generate regression test Update regression test dependability test case execution and sw validation Design scaffolding Design oracles Execute unit test • and determine when that level has been Analyze coverage Generate structural test attained. Execute integration test Execute system test Execute acceptance test Execute regression test improvement Process Collect data on faults analyze faults and improve the process (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 13 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 14 Different measures of dependability Example of different dependability measures Web application: • Availability measures the quality of service in • 50 interactions terminating with a credit card charge. terms of running versus down time • The software always operates flawlessly up to the point • Mean time between failures (MTBF) measures that a credit card is to be charged, but on half the the quality of the service in terms of time attempts it charges the wrong amount. between failures What is the reliability of the system? • If we count the fraction of individual interactions that • Reliability indicates the fraction of all are correctly carried out, only one operation in 100 attempted operations that complete fail: The system is 99% reliable. successfully • If we count entire sessions, only 50% reliable, since half the sessions result in an improper credit card charge (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 15 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 16

4: How can we control the quality of Assessing dependability successive releases? • Software test and analysis does not stop at the • Randomly generated tests following an first release. operational profile • Software products operate for many years, and • Alpha test: tests performed by users in a undergo many changes: controlled environment, observed by the – They adapt to environment changes development organization – They evolve to serve new and changing user • Beta test: tests performed by real users in their requirements. own environment, performing actual tasks • Quality tasks after delivery without interference or close monitoring – test and analysis of new and modified code – re-execution of system tests – extensive record-keeping (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 17 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 18 5: How can the development process A four step process to improve fault itself be improved? analysis and process • The same defects are encountered in project 1. Define the data to be collected and after project implementing procedures for collecting them • A third goal of the improving the quality 2. Analyze collected data to identify important process is to improve the process by fault classes – identifying and removing weaknesses in the 3. Analyze selected fault classes to identify development process weaknesses in development and quality – identifying and removing weaknesses in test and measures analysis that allow them to remain undetected 4. Adjust the quality and development process (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 19 (c) 2007 Mauro Pezzè & Michal Young Ch 1, slide 20