Learning objectives • Understand the main purposes of automating software analysis and testing • Identify activities that can be fully or partially Automating Analysis and Test automated • Understand cost and benefit trade-offs in automation • S eparate publicity from important features in descriptions of commercial A&T tools (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 1 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 2 Three Potential Roles of Automation Approaching Automation • Necessary for introducing a task • Prioritize automation steps based on – example: coverage tools enable measuring structural – variations in impact, maturity, cost, scope of the coverage of test suites technology – fit and impact on the organization and process • Useful to reduce cost • Three (non-orthogonal) dimensions for – example: capture and replay tools reduce the costs of reexecuting test suites automation • Useful to increase (human) productivity – value and current cost of the activity – example: software inspection is a manual activity, – extent to which the activity requires or is made less expensive by automation but tools to organize and present information and manage communication increase the productivity of – cost of obtaining or constructing tool support people (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 4
Automation Costs Vary Enormously Costs May Depend on Scope • S ome tools are so simple to develop that they are • S ometimes a general-purpose tool is only marginally justifiable even if their benefits are modest more difficult to produce than a tool specialized for one project – example: generate test cases from finite state machine models – example: general capture and replay for Windows applications • S ome tools that would be enormously valuable are vs capture and replay for a specific Windows application simply impossible – Investment in the general-purpose tool, whether to build it or – example: identify exactly which parts of a program can never to buy it, can be amortized across projects be executed (a provably undecidable problem) • In other cases, simple, project-specific tools may be more cost effective – Tool construction is often a good investment in a large project – example: simulators to permit independent subsystem testing (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 5 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 6 Focusing Where Automation Pays Planning: The Strategy Level • S imple repetitive tasks are often straightforward to • Prescribes tools for key elements of the quality process automate • Can include detailed process and tool prescriptions – humans are slow and make errors in repetitive tasks • Recommends different tools contingent on aspects of a • But ...j udgment and creative problem solving remain proj ect outside the domain of automation – (application domain, development languages, size, overall • Example: Humans are quality,...) • Often included in the A&T strategy: tools for – Very good at identifying relevant execution scenarios that correspond to test case specifications – Organizing test design and execution – Very inefficient at generating large volumes of test cases or – Generating quality documents identifying erroneous results within a large set of outputs from – Collecting metrics regression tests – Managing regression test suites • Automating the repetitive portions of the task reduces • Less often included: tools for costs, and improves accuracy as well – Generating test cases – Dynamic analysis (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 7 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 8
Planning: The Project Level • The A&T Plan Indicates – Tools inherited from the strategy – Additional tools selected for that proj ect Process Support: For new or customized tools, the A&T plan must include • Costs (including training) Planning & Monitoring • Implied activities • Potential risks • The plan positions tools within the development process and the analysis and test methodology – Avoid waste of cost and effort from lack of contextualization of the tools – Example: tools for measuring code coverage • simple and inexpensive • (if not properly contextualized) an annoyance, producing data not put to productive use (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 9 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 10 Automation in Process Management Classic Planning Tools • Facilitate task scheduling, resource allocation, and cost • Managing a process involves ... estimation by arranging tasks according to resource and – planning a set of activities with appropriate cost and quality time constraints trade-offs • Can be specialized to A&T management with features – monitoring progress to identify risks as early as possible for deriving relations among tasks, launching tasks, and – avoiding delays by adjusting the plan as needed monitoring completion of activities • ... and requires ... • Examples: tools to – human creativity and insight for which no tool can substitute – recognize delivery of a given artifact • Tools can support process management and improve – schedule execution of a corresponding test suite decision making by – notify test designer of test results – organizing and monitoring activities and results – record the actual execution time of the activity – signal schedule deviations to the quality manager – facilitating group interaction • Most useful when integrated in the analysis and test – managing quality documents environment – tracking costs (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 11 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 12
Version and Configuration Control Tools Monitoring • Integrated quality tracking • Analysis and testing involve complex relations among a large number of artifacts – improves efficiency in a well-structured process, – does not by itself bring order out of chaos • Version and configuration management tools • Progress must be monitored in terms of – relate versions of software artifacts – schedule (actual effort and completion times vs – trigger consistency checks and other activities plan) – support analysis and testing activities like they – level of quality control assembly and compilation of related modules • Quality of the final product • example: trigger execution of the appropriate test suites – cannot be directly measured before its completion for each software modification – but we can derive useful indications • Improve efficiency in well-organized processes • example: orthogonal defect classification [see chapter 20] – not a substitute for organization (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 13 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 14 Quality Tacking Managing People • Essential function: recognize deviations from expectation as early • People may work as possible to reduce consequences – in different groups • Proxy measures – must be computed early – in different companies – must be interpreted in a way that avoids misleading conclusions or – distributed across time zones and continents distorted incentives • Example: lines of code • A large proportion of a software engineer's time • useful as a simple proxy for productivity is devoted to communication • must be carefully interpreted to avoid creating both an incentive for verbosity and a disincentive for effective reuse • We need to • Example: number of faults detected • useful to detect deviations from the norm – facilitate effective communication • one should be as concerned about the causes of abnormally low numbers as high – limit disruptions and distractions of unmanaged • Collection, summary, and presentation of data can be automated communication • Design and interpretation cannot be automated (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 15 (c) 2007 Mauro Pezzè & Michal Young Ch 23, slide 16
Recommend
More recommend
