Software Measurement Massimo Felici and Conrad Hughes - PowerPoint PPT Presentation

Software Measurement Massimo Felici and Conrad Hughes mfelici@staffmail.ed.ac.uk conrad.hughes@ed.ac.uk http://www.inf.ed.ac.uk/teaching/courses/sapm/ Slides: Dr James A. Bednar SAPM Spring 2009: Software Measurement 1

Why Measure? If we want to make reasonable decisions about projects, we have to measure some sort of data on which to base those decisions. However, making such measurements is extremely time consuming, which is expensive. It is also extremely difficult (impossible?) to get unambiguous data. Thus it is crucial to decide what to measure about your projects, and what you will do with that information. SAPM Spring 2009: Software Measurement 2

Identifying Issues How do you figure out what to measure? Some are obviously things that you must measure to have any idea what you are doing: • Project constraints (e.g. you need to know if you are going over budget) • External requirements/Product acceptance criteria (you need to demonstrate that requirements are met) Others are based on analysis of what risks you face in this project, what has gone wrong in previous projects, etc. SAPM Spring 2009: Software Measurement 3

Issues That Can Be Measured 1. Schedule : Can we expect it to be done on time? 2. Cost : Can we afford to finish this project, or will it end up costing more than it is worth? 3. Size : How big is the product so far? Is the scope stable? 4. Quality : Is the product being made well, with few bugs? 5. Ability : How much design/coding/debugging/etc. can this team do per month? 6. Performance : Is the program fast enough, using reasonable resources? Most of these interact strongly with the others. SAPM Spring 2009: Software Measurement 4

Issues 1. Schedule What you want to know: What you can measure: Is progress being made? Dates of milestone delivery Is work being done? Components completed Requirements met Paths tested Problem reports resolved Reviews completed Change requests completed SAPM Spring 2009: Software Measurement 5

Issues 2. Cost What you want to know: What you can measure: How much is it demanding Total effort of our staff? Number of staff involved Staff experience levels Staff turnover Are we getting our money’s Earned value worth? Cost Is project making good use Availability dates (too early, late?) of external resources? Resource utilization SAPM Spring 2009: Software Measurement 6

Issues 3. Size What you want to know: What you can measure: How large is this program Lines of code so far? Number of components Words of memory Database size How much does this Requirements met program accomplish so far? Function points Change requests completed SAPM Spring 2009: Software Measurement 7

Issues 4. Quality What you want to know: What you can measure: How reliable is the software? Problem reports Defect density Failure interval How hard was it to fix Rework size the bugs? Rework effort SAPM Spring 2009: Software Measurement 8

Issues 5. Ability What you want to know: What you can measure: Is the development process Capability Maturity Model level well managed? How productive is this team? Code size / effort Functional size / effort SAPM Spring 2009: Software Measurement 9

Issues 6. Performance What you want to know: What you can measure: Is the program fast enough? Cycle time Are the resources required by CPU utilization the program acceptable? I/O utilization Memory utilization Response time SAPM Spring 2009: Software Measurement 10

Prioritizing Issues Example Issue Probability Relative Project of occurrence impact exposure Aggressive schedule 1.0 10 10 Unstable requirements 1.0 8 8 Staff experience 1.0 5 8 Reliability requirements 0.9 3 4 COTS performance 0.2 9 1 Only do significant work to measure those issues to which the exposure is high; any extra work you give your coders for useless quantities will put all your data at risk. SAPM Spring 2009: Software Measurement 11

Making a Measurement Plan • Issues and measures • Data sources • Levels of measurement • Aggregation structure • Frequency of collection • Method of access • Communication and interfaces • Frequency of reporting SAPM Spring 2009: Software Measurement 12

Limitations 1 • Most measures are misleadingly precise, yet not very accurate • Size doesn’t map directly to functionality, complexity, or quality • Incremental design requires measuring of incomplete functions • The most meaningful software statistics are time consuming to collect SAPM Spring 2009: Software Measurement 13

Limitations 2 • Many measures only apply after coding has been done • Performance and resource utilization may only be known after integration and testing • Often no distinction between work and re-work • Milestones don’t measure effort, only give critical paths • Time lag between problems and their appearance in reports SAPM Spring 2009: Software Measurement 14

Limitations 3 • Difficult to compare relative importance of measures • Important measures may be spread across components • Hard to find reliable historical data to compare with • Changes suggested by one performance indicator may affect others • Overall Capability Maturity Model level may not predict performance on a specific project SAPM Spring 2009: Software Measurement 15

Checking Your Data • Are units of measure comparable (e.g. lines of code in Java versus Python)? Normalization? • What are acceptable ranges for data values? • Can we tolerate gaps in data supplied? • How far does a value have to be from our plan for us to need a new plan? SAPM Spring 2009: Software Measurement 16

Indicator 1. Design Progress Number of units completing design Planned Actual Project time With an indicator and a plan, you can see if you are on track. SAPM Spring 2009: Software Measurement 17

Indicator 2. Effort Actual Staff hours per month Planned Project time SAPM Spring 2009: Software Measurement 18

Estimator: Size-Effort Upper 95% Staff months (log scale) Lower 95% Number of lines of source code (log scale) With enough data, you can try to predict future performance. SAPM Spring 2009: Software Measurement 19

Summary • Measurement is time-consuming, difficult, and impossible to do perfectly • You need to choose what you want to find out, and how to approach measuring that • Always be aware of the limitations of the measurement and of how it relates to what you really want to know • Be careful when trying to relate past performance to the future SAPM Spring 2009: Software Measurement 20

Required Readings • S.L. Pfleeger, R. Jeffery, B. Curtis, B. Kitshenham. Status Report on Software Measurement. IEEE Software, March/April 1997. • B. Clark. Eight Secrets of Software Measurement. IEEE Software, September/October 2002. • J. Boegh, S. De Panfilis, B. Kitchenham, A. Pasquini, A Method for Software Quality Planning, Control, and Evaluation. IEEE Software, March/April 1999. • J. Clapp. Getting Started on Software Metrics. IEEE Software, January 1993. SAPM Spring 2009: Software Measurement 21

Suggested Readings • S.L. Pfleeger. Software Metrics: Progress after 25 Years?, IEEE Software, November/December 2008. • R.J. Offen, R. Jeffery. Establishing Software Measurements Programs. IEEE Software, March/April 1997. SAPM Spring 2009: Software Measurement 22