example use case templatesample test plan templateProject plan template
ICS 121: Quality Concerns & Approaches
- What is quality?
- How to achieve it?
- How to assure it?
- Unit Tests > Who, What, and Why?
- Automated Style Checks and Test Execution
What is Quality?
- Basically, quality is all the things that can be good about a piece of software
- Normally, the term "quality" by itself refers to correctness
of the program with respect to its specification.
- Other external product qualities: reliability,
accuracy, usability, learnability, understandability, efficiency,
scalability, robustness, security
- Internal product qualities: maintainability, flexibility,
portability, testability, reusability
- Quality directly affects time-to-market
- Development progress cannot be objectively assessed
without quality measures
- The length of the testing process depends on
the number of defects when testing begins, and the
rates of discovery and repair.
- When testing comes too late in the process, testing
overruns due to low quality mean missed ship dates
- Products always ship with some defects
- You need to know what they are, and how severe
- Defects drive up support costs
- Consumers are less tolerant of defective software
than they were 10 years ago
Quality > How to achieve it?
- First: Build quality into the product
- Have stated quality goals, plan for them, measure progress
- Have good requirements, design, and implementation
- Start with high quality components
- Use checklists and guidelines
- Manage and control changes
- Take the time to think through decisions
- Document and/or build in assertions and invariants
- Second: Search for and eliminate poor quality
- Reviews: Developers go over specific aspects of the system
- Static analysis: automatically look for common coding errors, prove assertions
- Testing: Exercise the system with test inputs and expected outputs, check assertions
- Repair defects: Make sure you understand the problem. Verify that the fix worked and
did not introduce new defects.
- Quarantine releases: near release dates allow only defect repair to improve quality,
not new features that have new defects.
- Software process improvement: so it doesn't happen again
How to Assure Quality?
- Assuring quality is different from achieving it
- Assurance is measuring how sure you are and making yourself more
- Form a QA plan with measurable goals
- Measure quality
- How many defects do you have? How many have you found? not
- Severity: How would these defects affect customers and users?
- Establish effective feedback channels for customers/users
- Track and measure defects with an issue tracking system
- Estimate defects based on historical data, scope of changes, risks
- How much time/effort/money is defect discovery costing you?
- How much time/effort/money is defect repair costing you?
- What percentage of repairs fail or introduce new defects?
- Communicate the plan and the measurements to other team
- A plan is useless unless people follow it
- Up-to-date quality measures are a key part of project status
- Evaluate quality with respect to goals and make process improvements
- Are you achieving the stated quality goals?
- If not, improve the process by adjusting effort level,
release plans, adding or changing QA activities.
- E.g., a QA plan that focused entirely on testing is likely to
uncover a large number of defects late in the release cycle. It
should be replaced with a new QA plan that includes more activities
that reduce defect creation.
When to work on Quality?
- Early and often
- "Early" saves money because less is built on top of faulty code,
requirements, or design
- Early finding helps estimate number of defects before too late
- Early fixing helps estimate effort to fix before too late,
creates and uncovers
- Often helps catch individual defects before they can
interact, or mask each other
- When not to work work on testing?
- When you don't know what to test for. You need better requirements
- When you find too many defects. Go back to the requirements and design and find out what went wrong
- When you are getting tired of manual testing. You need automation
- When regressions happen repeatedly. It's time to fix the process, add more change controls
- When too many major defects are found but not yet fixed. Current testing will be invalidated by all the changes
- Where to put your effort instead?
- Better requirements
- Test planning and infrastructure
- Reviews and assertions
Unit Tests > Who, What, and Why?
- Who: Unit tests were traditionally done by the QA team: they did
unit tests, integration tests, and system tests. Now the trend is
to have developers write and maintain unit tests throughout the
project. In some cases, unit tests serve as light-weight
replacements for detailed requirements.
- What: A unit test is a simple, automated test that verifies one
aspect of one unit of code.
- A test suite contains many tests
- A unit test typically tests one class in the system
- A unit test consists of test cases
- Each test case typically tests one method in the system
- There can be many test cases for each method in the system
- Each test case either succeeds or fails, there is no gray area
- If a test case has an error, that is also a failure
- A test or test suite can be said to succeed to a certain percentage
- Running unit tests is practical because they are automated
- Debugging defects found by unit tests is easy because they are narrowly scoped
- Adopting a practice of writing unit tests is easy because they are very incremental
- Unit tests can fully cover a piece of code, giving good assurance.
- Writing unit tests can help clarify a developer's thinking
- The test-first process
- Developer has a working system that is partly implemented
- Product management selects a feature to add to this release. They write short spec.
- Developer reads spec, understands feature
- Developer writes test cases, further understands feature
- If needed, developer and product management discuss/revise feature
- Developer runs test cases, they must fail
- Developer implements code, tests, revises. Keeps going until all test cases pass
- Test cases are kept for daily regression testing
- If the requirement ever changes, the tests are updated
- PROs and CONs:
- PRO: Testing actually gets done rather than ignored
- PRO: Developers refine their detailed understanding of the requirements before coding
- PRO: Requirement is formally stated
- PRO: Formal requirement and unit test are always synced (because they are the same thing)
- PRO: Percentage of passed tests indicates project progress
- CON: Formal statement of requirement is in code, not accessible by non-developers
- COM: Formal statement of requirement may be wrong, it is hard to validate
- CON: False sense of security that unit tests are all of QA
Automated Style Checks and Test Execution
- It is useful to run these tools nightly so that everyone on the
development team sees status and so that problems are highlighted
early so they can be fixed.
- Checkstyle checks java coding style rules and common errors: example
- You can see where you need to clean up your code and fix some defects
- JUnit runs unit test cases and generates test results: example
- You can see where you need to fix defects
- Clover runs unit test cases and measures coverage: example
- You can see where you need to write more tests