Software Qualities Maintainer Good Documentation Quality Assurance Readable Code Good Design Reliability Low Cost Correctness Portability Efficiency Increased Functionality productivity Ease of use Ease of learning Customer User 11-QA 1 2 11-QA Software Quality Assurance Costs of Poor Quality ● Use of analysis to validate artifacts ● Increased time to find and fix problems ■ requirements, designs, code, test plans ● Increased cost to distribute modifications ● Technical reviews ● Increased customer support ● Document reviews ● Product liability ● Compliance to standards ● Failure in the market place ● Control of changes 3 4 11-QA 11-QA Software Reviews Software Reviews (cont.) ● Individuals read and comment on the software ● Applicable to all software artifacts artifacts ■ code inspections ● Very human intensive ■ requirements and design reviews ■ walk-throughs ● Overriding evidence shows that it ● Recent research shows that ■ improves quality and productivity ■ particular kind of review, size of team, etc. doesn’t matter ■ reduces cost ■ need at least one good, dedicated person doing the review ● It is usually one of the first activities to be dropped when schedules get tight 5 6 11-QA 11-QA 1
Typical Review Team Software Review Guidelines ● Developer -- presents the material ● Review the artifact ● Moderator -- keeps the review on track ■ don’t attack the developer ■ makes sure everyone abides by the process ● Stick to an agenda ● Secretary -- takes minutes, documents problems ● Limit debate found ■ watch out for “religious” issues ● Optionally ■ watch out for stylistic issues that don’t affect maintainability ■ Apprentice -- learning about the project ■ Domain expert -- familiar with the domain and can verify ● Identify problems, not solutions assumptions ● Keep accurate notes ● Establish and follow evaluation guidelines ● Limit number of participants 7 8 11-QA 11-QA Sample evaluation Guidelines: Technical Review Guidelines (cont.) Code Inspection ● Prepare beforehand ● Has the design been correctly translated to code? ■ both developers and reviewers ● Are language features used appropriately? ● Allocate resources for reviews ● Are coding standards followed? ■ people and time ■ be careful! make sure the standard makes a difference ● Possible outcomes ● Are documentation standards followed? ■ accept product as is ● Are there misspellings or typos? ■ reject until modified ■ reject product outright ● Are the comments accurate and unambiguous? ■ accept product provisionally 9 10 11-QA 11-QA Sample evaluation Guidelines: QA Terminology Code Inspection (cont.) ● Are data types and declarations appropriate? ● Correctness ● Fault ● Are all constants correct? ● Reliability ● Error ● Are all variables initialized before being used? ● Testing ● Are there overly complex conditions? ● Verification ● Debugging ● Is there unreachable code? ● Validation ● Failure ● Are there obvious inefficiencies? ● V&V 11 12 11-QA 11-QA 2
Terminology More terminology ● Correctness: artifact is consistent with its specification ● Testing: entails executing the software on selected ■ Specification could be wrong or incomplete test cases ■ Rarely is software known to be correct ■ Evaluate the results (oracle) ■ Rarely is the specification correct ■ Evaluate the performance ● Reliability: probability that the software is correct ■ Evaluate the ease of use ■ Statistical measure based on past performance ● Common Wisdom: Testing reveals bugs but does ◆ e.g., mean time to failure not guarantee the absence of bugs ■ How should you select test cases? ■ How do you know when to stop testing? 13 14 11-QA 11-QA More terminology More terminology ● Failure: an erroneous result ● Debugging: the process of finding the cause of a “bug” and a way to fix it ■ incorrect outputs/response for given inputs/stimuli ■ w/o introducing additional bugs! ■ fails to meet real-time constraints ● Verification: process of proving, using ● Error: incorrect concept mathematical reasoning, that a program is ■ may cause failures if not corrected “correct” ● Fault: the cause of one or more failures ■ proofs vs. modelchecking ■ discovered after release ■ is expensive and is not always possible ■ is not foolproof 15 16 11-QA 11-QA More terminology Many different kinds of testing ● Unit testing: test individual components ● Validation: process associated with showing that ■ test stubs simulate called components the software performs reasonably well ■ test harness simulates “outer” context and maintains stubs ● V & V: verification & validation? ● Integration testing: combine components and test them ■ more typically equated with validation ■ follows build plan ● System testing: test whole system 17 18 11-QA 11-QA 3
More kinds of testing More kinds of testing ● Acceptance testing: testing to determine if the ● Black box / functional testing: product is acceptable ■ testing based on specifications ● Regression testing: retesting after the system ● White box / structural testing: has been modified ■ determine “old” test cases that must be re-executed ■ testing based on looking at the artifact ■ determine what new test cases are required ● Both black box and white box testing are needed 19 20 11-QA 11-QA Testing is hard work Testing is hard work (cont.) ● Typically 50% of software development effort ● Psychologically difficult for a programmer to test goes into testing his/her own code thoroughly ■ up to 85% for life-critical software ● Exhaustive testing requires testing all combinations of input values ● How to identify “good” test cases? ■ high probability of finding a new error ■ Sorting an array of size 10 containing integers in the range 1 . . 10 has 10! combinations (3,628,800 cases) ■ hits “boundary” conditions ■ “weirdo” cases ◆ often reveal bad assumptions and/or lack of rigor ● Objective is to find errors ■ test case is “successful” if it finds a new error 21 22 11-QA 11-QA Testing Testing Principles ● CAN: ● Tests should be traceable to requirements ● Tests should be planned long before testing begins ■ Uncover errors ● Exhaustive testing is not possible ■ Show specifications are met for specific test cases ■ 80% of all errors typically occur in 20% of the modules ■ Be an indication of overall reliability ■ test cases should be chosen to maximize likelihood of finding ■ Increase reliability (why??) an error ● CANNOT: ■ Prove that a program is error-free ■ Serve as verification (why??) 23 24 11-QA 11-QA 4
Testing Principles (cont.) Testability ● Testing should be done by someone other than ● Simple software is easier to test the developers ■ minimize coupling, maximize cohesion ● Output is sufficient to determine correct behavior ■ Developers do original testing ● Performs its own tests for internal errors ■ SQA does independent testing ■ raises meaningful exceptions ◆ usually black box testing ● All code is reachable ● Automated testing tools should be used ● Independent modules can be tested in isolation ■ Reduce testing costs ● Documentation is complete and accurate ■ Reduce likelihood of human error 25 26 11-QA 11-QA Debugging Quality is an on-going concern ● You can’t build quality into a system after the fact ● Find the cause of a failure and fix it ● Quality should be a consideration during every phase ■ an art, not a science of development ● Debugging is difficult because ● Plan for testing / validation in all phases ■ symptom may appear long after the fault occurs ■ requirements -> functional test cases ■ design -> functional and structural test cases ■ symptom may be difficult to reproduce ■ code -> enhanced func & struc test cases ■ symptom may be intermittent ■ maintenance -> further enhanced func & struc test cases ● Unit testing helps localize errors 27 28 11-QA 11-QA Introduction to SQA Summary Formal Verification How many tests do you have to do 0 ● U.S. software costs $200 billion/year i = to show d=nx , always?? (or that 0 d = the loop even works…) ● Need to do i ≠ n ■ improve software quality 1 i = i + Need a way to “prove” ■ reduce costs properties in general. d = d + x ◆ V&V is over 50% of the cost od ● Improving V&V should reduce costs significantly Proofs Model Checking while improving quality Formal Mathematically based 29 30 11-QA 11-QA 5
Recommend
More recommend