5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ) Nanjing, Dec 4, 2017 Pitfalls and Countermeasures in Software Quality Measurements and Evaluations Hironori Washizaki washizaki@waseda.jp Prof., Global Software Engineering Laboratory, Waseda University Visiting Prof., National Institute of Informatics Director, SYSTEM INFORMATION CO., LTD. Vice-Chair, IEEE CS Japan Chapter Chair, SEMAT Japan Chapter Convenor, ISO/IEC/JTC1/SC7/WG20 PC Co-Chair, APSEC 2018 Nagoya! H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Metric and Measurement • Mapping attributes to values/names on scales Cyclomatic complexity – Quality control by single metric – Estimating metric values by other metrics • You cannot control what you cannot 1 Reuse rate 0.8 measure! 0.6 0.4 0.2 0 #Header files included 0 0.05 0.1 0.15 0.2 0.25 0.3 N. tests Complexity Defect Function Coupling passed LOC density point Module Source Defect Test Func. Spec. design code report case Req. Design Impl. Testing Def. 2 Effort, time
Pitfalls and Countermeasures Pitfall Countermeasure Negative Goal-orientation Hawthorne effects Multidimensional measurements Visualization of relationships among Organization organizational goals, strategies, and misalignment measurements Exhaustive identification of rationales Prediction incorporating uncertainty Uncertain future Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation 3 H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Pitfalls and Countermeasures Pitfall Countermeasure Negative Goal-orientation Hawthorne effects Multidimensional measurements Visualization of relationships among Organization organizational goals, strategies, and misalignment measurements Exhaustive identification of rationales Prediction incorporating uncertainty Uncertain future Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation 4 H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Hawthorne Effect 5 nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attention http://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg
Goal-Question-Metric (GQM) Paradigm • Goal-oriented framework for identifying goals and necessary corresponding metrics • Goal: measurement goals • Question: questions for evaluating goal achievement • Metric: objective or subjective metrics for obtaining necessary quantitative data to answer questions Goal achievement evaluation Goal Question Answer Identifying Interpreting Metric Measured metrics measurement value results Collected data R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw -Hill Education, 1999
GQM-based Multidimensional Measurements Sub- Goal Question Metric Question Reliability Are functions Is call- Depth of not nesting call graph T Efficiency complicated? not too deep? Cyclomatic Analysability Number min Is logic Maintain Changeability not too Are entities Estimated static -ability complex? named Testability path length min properly? Portability Measured by Measured by Independent Independent Specific Specific Reusability some tools some tools (Scale type: T threshold, min smaller is better, max larger is better) H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007) 7
SEMAT-based Multidimensional measurements Opportunity Stakeholders Customer Software System Requirements Solution Team Work Endeavour Way of Working Ivar Jacobson, Pan- Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering: 8 The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42 -49, December 2012.
alpha as Project Measurement Checklist State 1 XXXXXXXXXXXXXXXXXX XXXXXXXXXXX ４ 1 ５ 2 3 XXXXXXXXXXXX State 2 XXXXXXXXXXXXXXXXXX XXXXXXXXXXX XXXXXXXXXXXX State 3 XXXXXXXXXXXXXXXXXX XXXXXXXXXXX XXXXXXXXXXXX …….. ……. Indicator alpha 9 Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software Engineering Practices, ICSE'13
Example from ITA WG on project failuers • An employee in charge of Bank office inquires "registered customer information cannot be browsed from the terminal“. • It was because a batch processing for the previous day has ended abnormally due to wrong data input. • It took about two hours to recover the data, and employees at each office had to handle customer inquiry manually. • For that reason, we had received plenty of complaints from customers who had been waiting for a long time! 10 H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence Conference in Seoul, 2017.
Problem Delay in service Delay in service Opportunity: Stakeholders: deployment deployment NG “Benefit NG “Satisfied Accrued” in Use” Cause Misunderstanding Misunderstanding Wrong data input Wrong data input of abnormal state of abnormal state as normal state as normal state Root cause Requirement Software Defects in Defects in s: NG System: NG Huge data inputs Huge data inputs operation manual operation manual “Coherent” “Demonstrable” manually manually Countermeasure Frequent rule- Frequent rule- Automate data Automate data Team: Work: violations violations input input NG NG “Under “Collaborating” Control” Monitor and Monitor and Review and modify Review and modify enforce compliance enforce compliance Way of Working: rules rules with rules with rules NG “In Use” 11
Pitfalls and Countermeasures Pitfall Countermeasure Negative Goal-orientation Hawthorne effects Multidimensional measurements Visualization of relationships among Organization organizational goals, strategies, and misalignment measurements Exhaustive identification of rationales Prediction incorporating uncertainty Uncertain future Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation 12 H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Organization misalignment We increase customer satisfaction by quality and usability improvement! Top management We reduce customer- reported defects by improving testing process and product maintainability. Development We track defect data and team code quality metrics. Quality assurance team 13
GQM+Strategies • Alignment and tracing among goal, strategy and data Measurement Data Goals and Strategies Goal: Increase Goal: Increase M1: Customer M1: Customer customer customer Business Level satisfaction survey satisfaction survey satisfaction by 10% satisfaction by 10% Context & Assumption Strategy: Improve Strategy: Improve Strategy: Improve Strategy: Improve No sub-level usability of usability of product quality product quality goal defined product product Goal: Reduce Goal: Reduce Software Level customer- customer- M2: Field defect M2: Field defect reported defects reported defects data data by 20% by 20% M3: Code quality M3: Code quality metrics (McCabe, metrics (McCabe, Context & Assumption coupling, cohesion) coupling, cohesion) Strategy: Improve Strategy: Improve Strategy: Improve Strategy: Improve Isolated Isolated efficiency of efficiency of maintainability of maintainability of data strategy 14 system testing system testing software software
Context-Assumption-Matrix [IEICE’16] Health Target Insuran ance Hospital al senso sor ・・・ compan any View View maker Hospital al Insuran ance compan any Health sensor sor maker ・・・ Resource Context / Assumption Personal depends medical Insurance company can info. Medical offer personalized products Insurance info by having medical info… company Goal provider Customized product Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with 15 stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99 -D, No.9, pp.2219-2228, 2016.
Expectation from the Medical view of insurance company info. Insurance provider company Goal Sales of new Secure management products Goal of medical info. Personalized Personal Conflict, redundancy, Conflict, redundancy, Context / Assumption products developed fitness Confirming Must ensure complement complement Insurance management personal info. company Medical Context / Assumption systems guideline info. Insurance product and provider Pati Getting fee can be personalized Strategy ent consent based on personal Confirm personal info. medical info. Management system of Strategy receiver of info. Health Receive medical info. sensor maker G Expectation from the view of health A sensor maker S S 16
Recommend
More recommend
