9 Digit Stakes and the Measurem ent Stack Dr. Bill Curtis SVP and - PDF document

9 Digit Stakes… …and the Measurem ent Stack Dr. Bill Curtis SVP and Chief Scientist, CAST Research Labs Director, Consortium for IT Software Quality Bill’s Decem ber 20 11 Trip CAST Confidential 1 1

It’s 10 AM, Do You Know Where Your Money Is ? No man’s property is safe w hile Wall Street is in session ! CAST Confidential 2 Code Unit Level  Pre-Build Analysis Code Unit Level Code Unit Level 1  Code style & layout  Expression complexity  Code documentation  Class or program design  Basic coding standards  Developer level IDE Unit Test & Static Analysis tools Developer level code unit analysis CAST Confidential 3 2

Technology Level  Post-Build Analysis Technology Level Code Unit Level 1  Code style & layout  Expression complexity  Code documentation  Class or program design  Basic coding standards  Developer level 2 Java Java Java Java Java Technology Level  Single language/technology layer Web Services Java Java  Intra-technology architecture  Intra-layer dependencies  Design & structure  Inter-program invocation  Security vulnerabilities  Development team level Single language static analysis tools Quality Assurance CAST Confidential 4 System Level  System Integration Analysis System Level Code Unit Level 1  Code style & layout  Expression complexity  Code documentation  Class or program design  Basic coding standards  Developer level APIs JSP ASP.NET 2 Java Technology Level C++  Single language/technology layer Web VB Java C# Services COBOL  Intra-technology architecture  Intra-layer dependencies Hibernate Messaging  Design & structure Struts  Inter-program invocation .NET Spring  Security vulnerabilities  Development team level COBOL 3 PL/SQ T/SQL L EJB Application Stack Level SQL  Integration quality  Function point, Server Oracle  Architectural  Effort estimation  Data access control DB2 compliance Sybase IMS  Risk propagation  SDK versioning  Application security  Calibration across  Resiliency checks technologies  Transaction integrity  IT organization level CAST Confidential 5 3

The QA Gap Functional Testing Structural Analysis (functional defect removal) (Non-functional Defect Removal—Reliability, Performance, Security, Maintainability) System Level (Quality Assurance) Integration & System Test Build and Integration Functional Unit Tests Coding Best Practices (code unit correctness) (readability, code unit reliability) Code Unit Level (Developer) IDE Unit Testing IDE Static Analysis CAST Confidential 6 Analyzing System Level Structural Quality Attribute Parsing Analysis Violations Measures Oracle PL/SQL Expensive operation in loop Sybase T-SQL Static vs. pooled connections Performance SQL Server T-SQL Complex query on big table IBM SQL/PSM Large indices on big table C, C++, C# Pro C Empty CATCH block Cobol Evaluation of Uncontrolled data access CICS Reliability 1200+ coding & Poor memory management Visual Basic architectural rules VB.Net Opened resource not closed ASP.Net Java, J2EE SQL injection JSP Application Cross-site scripting Security XML meta-data Buffer overflow HTML Uncontrolled format string Javascript VBScript Unstructured code PHP Misuse of inheritance PowerBuilder Transferability Oracle Forms Lack of comments PeopleSoft Violated naming convention SAP ABAP, Netweaver Highly coupled component Tibco Duplicated code Business Objects Changeability Index modified in loop Universal Analyzer for other languages High cyclomatic complexity CAST Confidential 7 4

Architecturally-Com plex, Multi-Com ponent Defects Study of defects across 1 open source, 2 large NASA applications Observation % of cases  60% Fixes mapping to > 2 files  30-40% Fixes mapping to > 3 files  10-36% Fixes mapping to > 2 components  10-20% Fixes mapping to > 2 subsystems Spread of faults 80% of faults in 20% of files M. Hamill & K. Goseva-Popstojanova (2009). Common faults in software fault and failure data. IEEE Transactions of Software Engineering , 35 (4), 484-496. CAST Confidential 8 2) Detect Architecturally Com plex Defects A structural flaw involving interactions Architecturally among multiple components, often Complex Defect residing in different subsystems % of total % of total repair effort app defects 48% 92% Code unit-level 20x as violations many 52% Primary cause fixes to of operational correct problems Architecturally 8% Complex Defects CAST Confidential 9 5

Productivity and Rework  Detroit Was Better Mass-Production Auto Assembly defects defects defects Rew ork = 25% of effort Expected path Expected path defects defects defects Rew ork = 40% of effort Recode Retest Recode Classic Softw are Development Retest CAST Confidential 10 Five Purposes for Software Measurem ent Govern 5) Improve Business risk executive visibility 1) Reduce  Reliability business 3) Control out- risk  Performance sourced work  Security Plan Develop Release Operate  Changeability 4) Improve  Understand- development productivity ability 2) Reduce maintenance cost IT Cost CAST Confidential 11 6

Structural Quality in Business Risk Term s Quality Operational Source of Characteristic change benefit $ Few er outages, Reduction in r 2 More stable, faster recovery lost revenue resilient code Reliability $ Less degraded Reduction in r 2 response time productivity loss Faster, more efficient code r 2 Performance $ Faster response Reduction in to customers lost customers Few er hackable r 2 w eaknesses $ Security Less risk of Value of reduced breach breach risk CAST Confidential 12 Case Study 1  Major US Consum er Bank Defects per 100 Resource Hours Situation SW Integration Test User Acceptance Test Retirement services, >$100B in assets  Production 75 supported application   Complex technology environment  IT-intensive business process Initiated structural quality analysis 4Q07  Cost of Defects per 100 Resource Hours Result Sustained reduction in test and  production defects  7X reduction in defect costs CAST Confidential 13 7

Case Study 2  Large Telco Reduces Defect Costs  Order Management System (OMS)  J2EE, VB, ASP, OMS Oracle, XML, Amdocs Enabler  Multi-year development, >$100m per year, 6 releases PY, runaway costs, 700 600 Defect Volume in QA Code 500 Non Code 400 300 200 100 0 R8 – Structural Quality Analysis starts here CAST Confidential 14 Rethinking Productivity Measurem ent Release Volume of code developed, modified, or deleted = Productivity Total effort expended on the release Productivity baseline  a value in a monotonically declining function that compares the amount of product produced to the effort required to produce it … unless you take action Original productivity baseline Incremental increases in technical debt Continuing decrease in productivity CAST Confidential 15 8

Technical Debt = Carry-forward Rework Release N Release N+1 Release N+2 Develop N Develop N+1 Develop N+2 Rework N Rework N+1 Rework N+2 Rework N Rework N Unfixed defects Rework N+1 release N Unfixed defects release N Unfixed defects release N+1 CAST Confidential 16 Adjust Productivity for Technical Debt Release N Productivity for Release N Develop N Rework N Volume of code developed, modified, Unfixed defects deleted, and rework carried forward release N Total effort expended on Release N CAST Confidential 17 9

What Predom inates Software Variation “ After product size, people factors have the strongest influence in determining the amount of effort required to develop a software product.” (P. 46) “Personnel attributes and human resource activities provide by far the largest source of opportunity for improving software development productivity.” (Boehm, 1981, p.666) Boehm, et. al (2000) CAST Confidential 18 Program m er Variation Swam ps Everything 50 Individuals Programs Symbology Percent of variance Spacial arr. 40 30 20 10 0 Forward Backward Dataflow Coding time Editor trans. Maintenance time Comprehension Experiment Coding Experiment CAST Confidential 19 10

Com plexity Profiles for Individual Developers Slopes .16 - .73 80 2 r .48 - .87 60 Changes 40 20 40 80 120 160 Syntactic complexity Basili & Hutchens (1983) CAST Confidential 20 The Measurem ent Stack Level Guidelines MBNQA, ??? Business / Customer 9 ROI Risk Cost, 7-8 Incidents ITIL, COBIT, IT-CMF Engineering / IT Availalability Productivity, 6 Team / Project TSP, CMMI Schedule, Budget Developer 5 Hours, Size, Defects PSP Measures CAST Confidential 21 11

Value Transitions in the Measurem ent Stack Profit Revenue Monetization Availability Cost Incidents Prediction Correlation Budget Defects Rework Schedule Aggregation Defects Hours Process Ability Size CAST Confidential 22 Consortium for IT Software Quality CISQ Quality Characteristic Specifications  Co-sponsored by SEI and the Object Management Group (OMG)  24 original member companies  Objective to standardize code level measurement of software attributes  Automated Function Points now a supported specification of OMG CAST Confidential 23 12

www.it-cisq.org  Mem bership Is Free CAST Confidential 24 13