ICS 125: Test Suites

Think of this as a design document for your testing. Focus just on the tests to be done. Staffing and scheduling belong in main project planning documents.
Introduction: Approach, Goals (coverage), Assumptions
Test suite
- Functional
- Scalability/Performance/Volume
- Robustness/Error handling
- Usability
Conclusion: Results, Reports

Test coverage is a metric that helps evaluate how well a test suite assures a quality goal.
Specification-based coverage metrics:
- Specification coverage: Is every requirement tested?
- UI coverage: Is every screen seen? Every option selected? Every button pressed? Every link followed?
- API coverage: Is every public function called? Every parameter exercised?
Implementation-based coverage metrics:
- Statement coverage: Is every source code statement exercised at least once?
- Branch coverage: Is every branch taken both ways? Consider short-circuit boolean operators.
- Path coverage: Is every combination of branches exercised?

First, decide on the overall goals and plan.
Design unit test suite to match the API of each unit. E.g., define test cases for each public method in your class. Add unit test cases to improve coverage.
Draw functional system test suite from requirements use cases, then add test cases to improve coverage.
Focus on usability review rather than controlled usability testing, unless you have the resources and experience needed. Gather and act on informal feedback from users.
Design performance and scalability test suites by identifying key parameters and scaling them up or down, in isolation or combination. Review design and code to estimate performance in terms of parameters, then test the accuracy of your estimates.

Try to cover all possible input values:
- Boolean or enumeration: test all possibilities
- Integer: identify equivalence classes, test one value in each class, or test both sides of each class boundary
- Reals: equivalence classes, plus test minor deltas for accuracy
- String: empty, multiple character classes, short and long, same as other strings or different
- Dates: boundaries, past/present/future, multiple formats
- Complex objects: break them down into smaller values, e.g., consider a Point to be an X and a Y
- Anything: make sure to include both valid and invalid values
- Anything: make sure to include both reasonable and unreasonable values
- Combinations of parameters: equal, unequal, related, unrelated, pointers to the same objects
It can also be useful to try to force the system to generate specific outputs: maximum, minimum, valid, invalid, empty, small, large
Make sure that you know the corrected expected output. A "test oracle" is code that computes the expected result in a known-good way, it may work backwards.
Try to cover all possible statements/branches:
- Look at the condition of each branch
- Decide what variable value would be needed to make it true or false
- Work backwards to find a test input that achieves those variable values