ICS 121: Quality Concerns & Approaches

Normally, the term "quality" by itself refers to correctness of the program with respect to its specification.
Other external product qualities: reliability, accuracy, usability, learnability, understandability, efficiency, scalability, robustness, security
Internal product qualities: maintainability, flexibility, portability, testability, reusability

Quality directly affects time-to-market
- Development progress cannot be objectively assessed without quality measures
- The length of the testing process depends on the number of defects when testing begins, and the rates of discovery and repair.
- When testing comes too late in the process, testing overruns due to low quality mean missed ship dates
Products always ship with some defects
- You need to know what they are, and how severe
- Defects drive up support costs
- Consumers are less tolerant of defective software than they were 10 years ago

Assuring quality is different from achieving it
Assurance is measuring how sure you are and making yourself more sure
Form a QA plan with measurable goals
Measure quality
- How many defects do you have? How many have you found? not found yet?
- Severity: How would these defects affect customers and users?
- Establish effective feedback channels for customers/users
- Track and measure defects with an issue tracking system
- Estimate defects based on historical data, scope of changes, risks
- How much time/effort/money is defect discovery costing you?
- How much time/effort/money is defect repair costing you?
- What percentage of repairs fail or introduce new defects?
Communicate the plan and the measurements to other team members
- A plan is useless unless people follow it
- Up-to-date quality measures are a key part of project status
Evaluate quality with respect to goals and make process improvements
- Are you achieving the stated quality goals?
- If not, improve the process by adjusting effort level, release plans, adding or changing QA activities.
- E.g., a QA plan that focused entirely on testing is likely to uncover a large number of defects late in the release cycle. It should be replaced with a new QA plan that includes more activities that reduce defect creation.

Early and often
- "Early" saves money because less is built on top of faulty code, requirements, or design
- Early finding helps estimate number of defects before too late
- Early fixing helps estimate effort to fix before too late, creates and uncovers
- Often helps catch individual defects before they can interact, or mask each other
When not to work work on testing?
- When you don't know what to test for. You need better requirements
- When you find too many defects. Go back to the requirements and design and find out what went wrong
- When you are getting tired of manual testing. You need automation
- When regressions happen repeatedly. It's time to fix the process, add more change controls
- When too many major defects are found but not yet fixed. Current testing will be invalidated by all the changes
Where to put your effort instead?
- Better requirements
- Test planning and infrastructure
- Reviews and assertions

Who: Unit tests were traditionally done by the QA team: they did unit tests, integration tests, and system tests. Now the trend is to have developers write and maintain unit tests throughout the project. In some cases, unit tests serve as light-weight replacements for detailed requirements.
What: A unit test is a simple, automated test that verifies one aspect of one unit of code.
- A test suite contains many tests
- A unit test typically tests one class in the system
- A unit test consists of test cases
- Each test case typically tests one method in the system
- There can be many test cases for each method in the system
- Each test case either succeeds or fails, there is no gray area
- If a test case has an error, that is also a failure
- A test or test suite can be said to succeed to a certain percentage
Why:
- Running unit tests is practical because they are automated
- Debugging defects found by unit tests is easy because they are narrowly scoped
- Adopting a practice of writing unit tests is easy because they are very incremental
- Unit tests can fully cover a piece of code, giving good assurance.
- Writing unit tests can help clarify a developer's thinking

It is useful to run these tools nightly so that everyone on the development team sees status and so that problems are highlighted early so they can be fixed.
Checkstyle checks java coding style rules and common errors: example from Maven.
- You can see where you need to clean up your code and fix some defects
JUnit runs unit test cases and generates test results: example from Maven.
- You can see where you need to fix defects
Clover runs unit test cases and measures coverage: example from Maven.
- You can see where you need to write more tests