ICS 121: Design for Testability

Testability is the quality of a system's design and implementation that makes it easy to fully test.
Examples of poor testability:
- The system can only be tested as a whole, there is no programmatic API to test parts of it
- The system can only be tested manually, which is slow and error prone
- The system only runs on one server or database, and the developer are making changes while we are testing
- The same inputs can generate different outputs, so there can never be one expected output
- The output looks ok, but the program corrupts a database that the tester cannot see
- The system has so many defects that we cannot even run our tests
- The system is changing so fast that we have to keep rewriting our tests.
Design qualities that make testing easier:
- Testing works best when there are only a few defects. If there are too many defects, use reviews first.
- The tester needs to control the unit under test: provide exactly the desired test data, no other data is used.
- The tester needs to evaluate the resulting state of the system including all output and side-effects
- The testing process needs to be automated so that it is fast and reliably repeatable

Expose an interface for the needed testing scope
- Test public and package-scoped methods. Private methods are harder to test.
- Add constructors that are useful for constructing test data
- Highly testable methods get all their data in parameters, not by making system calls to check the system clock, read or write files, or access the database
Make correctness automatically verifiable
- Correctness must be objective, not "looks OK" to a human
- Format output and expected output in a way that is easy to compare or verify, regardless of how it looks to the user
Make units individual testable
- Consider flattening the call tree.
- Avoid dependencies on overall system initialization proceedures. You should not need your whole application to be running to just test one datastructure.
Separate code by testability
- Some parts of the system are harder to test (e.g., database access and updates)
- Some parts are easier to test (e.g., basic algorithms and datastructures).
- Separate out those parts so that the easy parts can be tested easily. If they are well tested, the others can be covered in system test.
Establish a testing order
- Test lower-level functions first, and repair defects before testing higher-level functions that depend on them

A test fixture is a special environment that makes it easier to test one unit of code. Fixtures are needed to test a unit of code without an entire working system.
Test harness
- A test harness calls the unit under test and compares the actual result to the expected result.
- In JUnit, your test classes are test harnesses.
Stubs
- Useful when testing function f() that calls g(), but g() is not written yet or is too hard to test.
- Function g() can be replaced with a stub of g() that is just used for testing. Often the stub of g() will simply return a constant.
Test wrappers
- Useful when testing function f() that calls g(), but g() is too hard to test.
- Function g() can be replaced with a new function g2() that calls the real g(), but also tests assertions or records diagnostic information.
Mock Objects (endotesting)
- Mock objects are an object-oriented version of stubs and test wrappers
- Don't just verify the output of a function, instead pass in a special "mock object" as an argument.
- The mock object consists solely of stub methods or test wrappers around a real object
- Mock objects are useful for testing that the system works the same way that was specified in a sequence diagram
- Called "endotesting" because it tests a function, f(x), from the inside out. Imagine that your doctor gave you a pill with a tiny video camera inside.