Essential testing metrics

Organisations are under increasing pressure to reduce the cost and increase the efficiency of their testing programmes. As a response to this, many organisations are either looking to optimise their in-house teams or to outsource all or part of their testing projects.

The challenge for an organisation is how do they know if their testing provider is delivering the cost savings and efficiency required. To determine this it is necessary to look beyond the "big talk", "large offshore savings", "technical collateral', "tools" and "compelling resource costs" to the fundamental reasons why a test team is engaged. My view is that the sole purpose of software testing is to "find and remove defects as early and as efficiently as possible during the project lifecycle" – anything else should support this or it is peripheral. 

Taking the above premise, there are a number of simple metrics that can be used to quantifiably measure the effectiveness of a test team.

Quality

This is the most important testing metric as it relates to the primary purpose of testing.

• Defect Removal Efficiency – This is the percentage of “defects found and removed during the project delivery lifecycle” compared with “the total number of defects within an system”. This testing metric can also be used to determine the quality of the code and documentation deliverables produced by the project.

Efficiency

The cost of testing is easily buried in a lot of talk and frantic activity. Cutting through this provides a real measure of efficiency.

• Cost per Defect – This can be a difficult measure on which to get agreement, but, if done diligently, will provide an accurate in-sight into the true cost of the service being provided.
• Cost per Requirement Covered – This measure provides balance with the metric above; as any experienced tester would know it is easy to manipulate either the ‘number of defects raised’ or ‘number of test cases executed’.

Adherence to Required Standards

Cutting corners with agreed standards can exposure the hiring organisation to unanticipated levels of delivery risk.

• Test Coverage – Measuring test coverage (including regression testing) requires strong traceability of test deliverables across the solution.
• Automation Coverage – Often a contractual deliverable for a team.
• Defect Quality – What proportion of defects actually result in a change to either code, a document or a process.
• Standard of the Test Collateral produced – Are the organisational/project standards being followed, and is the collateral reusable?

Over the next few posts I will explore each of the above test metrics and discuss how they may be applied within the testing life-cycle.