ICS 125: Quality Concerns & Approaches
Overview
- What is quality?
- How to achieve it?
- How to assure it?
- Unit Tests > Who, What, and Why?
- Test-first
- Automated Style Checks and Test Execution
What is Quality?
- Basically, quality is all the things that can be good about a piece of software
- Some external product qualities
- Correctness/reliability: Lack of defects
- Accuracy: numeric calculations are correct to many decimal places
- Usability: learnability, understandability, fitness for user
tasks, efficient for users to use, not error prone for users to
use
- Efficiency: Produces results quickly
- Scalability: Works well for large N, where N is ...
- Robustness: handles invalid inputs well
- Security/Integrity: it cannot be hacked or exploited
- Some internal product qualities
- Maintainability
- Flexibility
- Portability
- Testability
- Reusability
Quality > How to achieve it?
- Build quality into the product
- Have stated quality goals, plan for them, measure progress
- Have good requirements, design, and implementation
- Start with high quality components
- Use checklists and guidelines
- Manage and control changes
- Take the time to think through decisions
- Document and/or build in assertions and invariants
- Search for and eliminate poor quality
- Reviews: Developers go over specific aspects of the system
- Static analysis: automatically look for common coding errors, prove assertions
- Testing: Exercise the system with test inputs and expected outputs, check assertions
- Software process improvement: so it doesn't happen again
How to Assure Quality?
- Assuring quality is different from achieving it
- Assurance is measuring how sure you are and making yourself more
sure
- Form a QA plan
- Are you doing everything you can to build in quality? If not,
where are defects creeping in?
- What types of analysis will you do?
- What types of review will you do?
- What types of testing will you do?
- Are your requirements and design good enough to even worry about testing?
- What testing coverage will you need? Not all components need
the same level of testing.
- What testing tools/methods will you use?
- Is your system being built in a way that supports that testing
method/tool? (testability)
- How will you defend against regressions?
- Measure quality
- How many defects do you have? How many have you found?
- How much time/effort/money is defect repair costing you?
- How is product quality affecting customers and users?
- Estimate defects based on historical data, scope of changes, risks
- Track and measure defects with an issue tracking system
- Establish effective feedback channels for customers/users
- Communicate the plan and the measurements to other team members
When to work on Quality?
- Early and often
- "Early" saves money because less is built on top of faulty code,
requirements, or design
- Early finding helps estimate number of defects before too late
- Early fixing helps estimate effort to fix before too late,
creates and uncovers
- Often helps catch individual defects before they can
interact, or mask each other
- When not to work work on testing?
- When you don't know what to test for. You need better requirements
- When you find too many defects. Go back to the requirements and design and find out what went wrong
- When you are getting tired of manual testing. You need automation
- When regressions happen repeatedly. It's time to fix the process, add more change controls
- When too many major defects are found but not yet fixed. Current testing will be invalidated by all the changes
- Where to put your effort instead?
- Better requirements
- Test planning and infrastructure
- Reviews
Unit Tests > Who, What, and Why?
- Who: Unit tests were traditionally done by the QA team: they did
unit tests, integration tests, and system tests. Now the trend is
to have developers write and maintain unit tests throughout the
project. In some cases, unit tests serve as light-weight
replacements for detailed requirements.
- What: A unit test is a simple, automated test that verifies one
aspect of one unit of code.
- A test suite contains many tests
- A unit test typically tests one class in the system
- A unit test consists of test cases
- Each test case typically tests one method in the system
- There can be many test cases for each method in the system
- Each test case either succeeds or fails, there is no gray area
- If a test case has an error, that is also a failure
- A test or test suite can be said to succeed to a certain percentage
- Why:
- Running unit tests is practical because they are automated
- Debugging defects found by unit tests is easy because they are narrowly scoped
- Adopting a practice of writing unit tests is easy because they are very incremental
- Unit tests can fully cover a piece of code, giving good assurance.
- Writing unit tests can help clarify a developers thinking
Test-First
- The test-first process
- Developer has a working system that is partly implemented
- Product management selects a feature to add to this release. They write short spec.
- Developer reads spec, understands feature
- Developer writes test cases, further understands feature
- If needed, developer and product management discuss/revise feature
- Developer runs test cases, they must fail
- Developer implements code, tests, revises. Keeps going until all test cases pass
- Test cases are kept for daily regression testing
- If the requirement ever changes, the tests are updated
- PROs and CONs:
- PRO: Testing actually gets done rather than ignored
- PRO: Developers refine their detailed understanding of the requirements before coding
- PRO: Requirement is formally stated
- PRO: Formal requirement and unit test are always synced (because they are the same thing)
- PRO: Percentage of passed tests indicates project progress
- CON: Formal statement of requirement is in code, not accessible by non-developers
- COM: Formal statement of requirement may be wrong, it is hard to validate
- CON: False sense of security that unit tests are all of QA
Automated Style Checks and Test Execution
- It is useful to run these tools nightly so that everyone on the
development team sees status and so that problems are highlighted
early so they can be fixed.
- Checkstyle checks java coding style rules and common errors: example
from Maven.
- You can see where you need to clean up your code and fix some defects
- JUnit runs unit test cases and generates test results: example
from Maven.
- You can see where you need to fix defects
- Clover runs unit test cases and measures coverage: example
from Maven.
- You can see where you need to write more tests
sample use cases templateexample software testing plan templateProject plan template