Testability Reid Holmes Testability The degree to which a system - PowerPoint PPT Presentation

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf https://www.youtube.com/watch?v=1keyEiJHqPw https://www.youtube.com/watch?v=XcT4yYu_TTs Testability Reid Holmes

Testability The degree to which a system or component facilitates the establishment of test objectives and the execution of tests to determine whether those objectives have been achieved. REID HOLMES - CPSC 410: Advanced Software Engineering

Given a finite amount of time and resources, how can we validate that the system has an acceptable risk of costly or dangerous defects. —Bob Binder REID HOLMES - CPSC 410: Advanced Software Engineering

Develop Check- In Test REID HOLMES - CPSC 410: Advanced Software Engineering

Why not test? ‣ Good reasons: ‣ I don’t know how! ‣ Legacy code ‣ Bad reasons: ‣ Bad design ‣ Doesn’t catch bugs (now) ‣ Slow ‣ Boring ‣ Hard to change ‣ That’s QA’s job REID HOLMES - CPSC 410: Advanced Software Engineering

Common assumption ‣ “The cost of fixing faults rises exponentially with how late [e.g., requirements, design, implementation, deployment] they are detected.” ‣ This is commonly stated but is based on evidence from 20+ years ago. ‣ This assumption does not seem to hold for modern processes, tools, and languages. ‣ That said, it is still necessary to validate that the system works. ‣ Also important that it continues to work as the system evolves. REID HOLMES - CPSC 410: Advanced Software Engineering

Terminology ‣ E ffi ciency: number of tests per unit of e ff ort. ‣ E ff ectiveness: the probability of detecting a bug per unit of e ff ort. ‣ Higher testability: more e ff ective tests, same cost. ‣ Lower testability: fewer weaker tests, same cost. ‣ Repeatability: the likelihood that running the same test twice will yield the same result. ‣ SUT/CUT: System/Code Under Test ‣ White-box: tests consider internals of CUT. ‣ Black-box: tests are oblivious of internals of CUT. http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf REID HOLMES - CPSC 410: Advanced Software Engineering

Anatomy of a test ‣ To reveal a fault, a test must: ‣ Reach some code ‣ Trigger a defect ‣ Propagate an incorrect result ‣ The result must be observed ‣ The result must be interpreted as incorrect ‣ Test threats: ‣ Non-deterministic dependencies ‣ Threading/Race Conditions/Deadlock ‣ Shared data REID HOLMES - CPSC 410: Advanced Software Engineering

Properties of Testability ‣ Controllability ‣ The extent to which the SUT can be made to perform specific actions of interest. ‣ Observability ‣ The extent to which the response of the SUT to a test can be verified. ‣ Isolateability ‣ The degree to which the element under test can be validated on its own. ‣ Automatability ‣ The ability to execute the test programmatically. REID HOLMES - CPSC 410: Advanced Software Engineering

Controllability ‣ What do we have to do to run a test? ‣ How expensive is it? ‣ Does the SUT make running a test impractical ? ‣ Give a test goal, do we have enough information to create an adequate suite? ‣ How much tooling can we a ff ord? REID HOLMES - CPSC 410: Advanced Software Engineering

Observability ‣ What do we have to do to identify pass/fail? ‣ How expensive is it to do this? ‣ Can we extract the result from the SUT? ‣ Do we know enough to identify pass/fail? REID HOLMES - CPSC 410: Advanced Software Engineering

Isolateability ‣ Can the element being tested be isolated? ‣ What is the cost to do this? ‣ If an element cannot be naturally isolated, can we simulate it (e.g., with mocks / stubs)? ‣ Why bother? ‣ Isolated components are: ‣ Simpler to reason about (e.g., root cause analysis) ‣ Less prone to non-determinism ‣ Faster ‣ Simulated dependencies can also more enable validating unusual states. REID HOLMES - CPSC 410: Advanced Software Engineering

Automatability ‣ Can tests be executed without human intervention? ‣ Huge economic advantages: ‣ Setting up automation: 5 hours ‣ Running manual test: 30 minutes. ‣ Automation pays for itself after just 10 iterations. ‣ What is the cost of automated infrastructure? ‣ What is the benefits of using a test infrastructure? ‣ Executions can be batched. ‣ Run on same configuration / hardware. ‣ Global visibility of results. ‣ Enables regression testing. REID HOLMES - CPSC 410: Advanced Software Engineering

Challenges ‣ Tests are code too. ‣ Also subject to their own faults. ‣ Not all test failures uncover faults: ‣ Defect in test itself ‣ Flaky test (due to some form of non-determinism) ‣ Requirements shortcoming (undefined behaviour) ‣ Implicit assumptions often surfaced by tests ‣ How to retrospectively recognize a ‘true’ failure? ‣ Developer changed the source (not test), test passed on the next iteration. REID HOLMES - CPSC 410: Advanced Software Engineering

Develop Check- In Test Must: Be Fast Be Reliable Isolate Failures REID HOLMES - CPSC 410: Advanced Software Engineering

Kinds of tests REID HOLMES - CPSC 410: Advanced Software Engineering

Unit tests isolate failures . Even if a product contains millions of lines of code, if a unit test fails, you only Kirkland (1) need to search that small unit under test to find the bug. Kurt Alfred Kluever (1) Writing effective unit tests requires skills in areas such as dependency management, mocking, and hermetic testing. I Mark Ivey (1) won't cover these skills here, but as a start, the typical example offered to new Googlers (or Nooglers) is how Google This site uses cookies from Google to deliver its services, to personalise builds and tests a stopwatch. Mark Striebeck (1) ads and to analyse traffic. Information about your use of this site is shared with Google. By using this site, you agree to its use of cookies. Marko Ivanković (1) Unit Tests vs. End­to­End Tests LEARN MORE GOT IT With end­to­end tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally for Michael Bachman (1) all end­to­end tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, that Mike Wacker (1) bug could be anywhere in the product. Mona El Mahdy (1) Although end­to­end tests do a better job of simulating real user scenarios, this advantage quickly becomes outweighed by all the disadvantages of the end­to­end feedback loop: Noel Yap (1) Test value Patricia Legaspi (1) Unit End­toEnd Peter Arrenbrecht (1) Phil Rollet (1) Fast Pooja Gupta (1) Radoslav Vasilev (1) Rajat Dewan (1) Rajat Jain (1) Reliable Rich Martin (1) Richard Bustamante (1) Roshan Sembacuttiaratchy (1) Ruslan Khamitov (1) Isolates Failures Sean Jordan (1) Sharon Zhou (1) Stephen Ng (1) Tejas Shah (1) Simulates a Real User Test Analytics (1) Vojta Jína (1) REID HOLMES - CPSC 410: Advanced Software Engineering Integration Tests Unit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work well Archives together. But even then, you do not necessarily need end­to­end tests. For that, you can use an integration test. An integration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that they October 2015 (1) coherently work together. August 2015 (1) If two units do not integrate properly, why write an end­to­end test when you can write a much smaller, more focused June 2015 (1) integration test that will detect the same bug? While you do need to think larger, you only need to think a little larger to verify that units work together. May 2015 (2) Testing Pyramid April 2015 (2) Even with both unit tests and integration tests, you probably still will want a small number of end­to­end tests to verify March 2015 (1) the system as a whole. To find the right balance between all three test types, the best visual aid to use is the testing February 2015 (1) pyramid. Here is a simplified version of the testing pyramid from the opening keynote of the 2014 Google Test Automation Conference: January 2015 (2) December 2014 (2) November 2014 (1) October 2014 (2) September 2014 (2) August 2014 (2) July 2014 (3) June 2014 (3) May 2014 (2) April 2014 (2)

Test pyramid http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html REID HOLMES - CPSC 410: Advanced Software Engineering

Continuous Integration ‣ Every project will build and execute automated tests. ‣ TravisCI (https://travis-ci.com/ (NOT ORG)) has provided the class with large numbers of commits ‣ You can request access online, we will grant it REID HOLMES - CPSC 410: Advanced Software Engineering

Activity ‣ In your groups: ‣ Choose one of your use cases ‣ Describe one end to end test for this use case ‣ Describe 2 integration tests for this use case ‣ Describe 3 unit tests for this use case REID HOLMES - CPSC 410: Advanced Software Engineering