Stuff SKIPPED Power Lecture Lab is due by 5pm today Software - PDF document

Stuff SKIPPED Power Lecture � Lab is due by 5pm today � Software perspective on power and energy management � Exam 1 next Tues � Mechanisms are provided by the HW people � I’ll be out of town so Zhe will give the exam � Frequency scaling � Voltage scaling � New lab and HW assignments after the exam � Sleep modes � Analysis of HW + workload can give us ballpark estimate of whether there is a good match � Policies are up to software � But it’s often difficult to balance power, performance, and users’ convenience Today � Testing embedded software � Fact: Most multithreaded Java programs run all of their test cases perfectly well when all locking is � Kinds of tests removed � When to test � How to test � What does this mean? � Test coverage Testing The Testing Mindset � Testing is the fundamental way that reliable � Creating good tests for your own software is hard embedded software is created � At least three reasons for this � This is why we can build safety-critical applications using � Microsoft (and other companies) separate testers buggy compilers! from developers � However, good testing techniques are neither easy � Different skill sets or intuitive � Good testers are adversarial � Lots of basic questions: � Goal is to break the software � When to test? � This can lead to strained relations between developers and � Who tests? testers � Where do test cases come from? � The best developers truly attempt to break their own � How to evaluate the result of a test? code � How much testing is enough? 1

Kinds of Tests Test Levels � Functionality – testing functional behavior � Hardware unit test � Interfaces – testing interaction with other systems � Hardware integration test � Security – test for robustness to intrusion � Software unit test � Standards – check for compliance � Software integration test � Regression � HW/SW integration test � Testing whether everything works after the system has been � System test changed � Acceptance test � Test cases derived from prior failures � Field test � Resources – measuring required resources such as CPU time, memory, network bandwidth � Load and stress – trying to overload the system Where do tests come from? When to Test � Use cases � Every combination of kind of test and test level should be run as early as is feasible � Developer intuition � Basic fact: Cost to fix a bug increases greatly as � Previous failures development progresses � Boundary cases from specification � Stress tests � Random inputs � Directed random / analysis-driven inputs Testing by Developers Unit Testing � Why? � Defects cheaper to fix when found earlier Driver � High-quality parts make integration possible � Defects found late are hard to map back to the source code Test Harness � Some kinds of tests are only possible at the unit level � Developers understand the implementation, which can lead to better test cases Unit under test � Quality cannot be added at the end of development � Has to be there from the start Stubs 2

Integration Testing Strategies Design for Test � Bottom-up � Term most often used in context of hardware � Start with low-level modules with few dependencies � Also applies to software � Exercise them using drivers � How to do this? � Top-down � Lots of assertions for preconditions and postconditions � Overall control structure drives tests � Implement self-tests � Stubs provided for nonexistant modules � Provide test scaffolding along with code � “Look and feel” of the system established early � Expose all interfaces for testing � Big-bang � Examples – how would you design these for test? � Only works for small systems � Code to set PLL � Useful for tightly coupled systems where top-down and � Code responding to an external interrupt source bottom-up are difficult Test Oracles Test Coverage � Test oracle – Code that tells us if the system is � Coverage metrics try to answer the question: How responding properly to tests can we know when to stop testing? � Some oracles are easy � Example metrics: � Not working if the software crashes � Function coverage – are all functions executed? � Not working if the software stops responding normally to � Statement coverage – are all statements executed? inputs � Branch coverage – is every possible decision executed at � Not working if an assertion is violated every branch? � Path coverage – is every path through the code executed? � Some oracles are very difficult � Value coverage – is the full range of every variable covered? � E.g. is the aircraft responding properly to crosswind? � Mutation coverage – are all variants of the program � Manual interpretation of the specification and test results covered? typically required � Exception coverage – are all exceptions signaled? � In most cases goal is 100% coverage Evaluating Coverage Metrics Coverage of Concurrent SW � Coverage metric must be understood by the user � Problem: � Near-complete coverage must be achievable � Traditional test coverage metrics are in terms of sequential software � Exceptions require fixing or manual review � Embedded software is concurrent � Some action should be taken upon reaching 100% � What are some plausible metrics for concurrent coverage software? � Interrupt nesting coverage � Interrupt preemption coverage � Thread preemption coverage � Synchronization coverage • Each lock “does interesting things” 3

Stress Testing Stress Testing for Interrupts � Test system at the limits of (and outside) its load � What bugs are we trying to find? parameters � How to do it? � Intuition: This exposes different kinds of problems than � What if data comes along with the interrupt? regular test cases do � How to tell when we’re done? � Examples – how would you stress test: � Embedded web server � An RTOS � A cell phone � Tricky problem: Thinking of as many sources of stress as possible Summary � Embedded software is only as good as its test cases � You should assume any conditions not tested will fail � … because they will � Developers perform early testing of components � Requires adversarial mindset � Requires wishful thinking to be ruthlessly suppressed � Integration cannot possibly succeed without reliable components � Summary: � Test early � Test often � Test creatively 4