Lesson 3 of 15 ~40 min
Course progress
0%

Quality Metrics That Matter

Implement meaningful quality metrics and KPIs

1 Quality Metrics That Matter 2 Metrics & Reporting Quiz

Quality Metrics and KPIs

Learn to measure quality effectively and communicate it to stakeholders.

Why Metrics Matter

Purpose of Metrics:

✓ Measure progress toward quality goals
✓ Identify trends and patterns
✓ Drive data-driven decisions
✓ Demonstrate QA value
✓ Predict quality issues
✓ Optimize resource allocation

Bad Metrics vs Good Metrics:

Bad: Number of test cases executed
Good: Test coverage of critical paths

Bad: Bugs found
Good: Defect detection effectiveness

Bad: Test execution time
Good: Mean time to detect (MTTD)

Essential Quality Metrics

1. Test Coverage Metrics

Code Coverage:

Line Coverage = (Lines Executed / Total Lines) × 100%
Branch Coverage = (Branches Executed / Total Branches) × 100%
Function Coverage = (Functions Tested / Total Functions) × 100%

Target: 80% line coverage, 70% branch coverage

Requirements Coverage:

Coverage = (Requirements Tested / Total Requirements) × 100%

Track:
- Untested requirements
- Partially tested requirements
- Fully tested requirements

Risk Coverage:

High-Risk Coverage = (High-Risk Items Tested / Total High-Risk) × 100%

Prioritize: 100% high-risk, 80% medium-risk, 60% low-risk

2. Defect Metrics

Defect Density:

Defect Density = Total Defects / Size

Where Size can be:
- Lines of Code (KLOC)
- Function Points
- Story Points
- Features

Industry Benchmark: 1-25 defects per KLOC

Defect Detection Effectiveness (DDE):

DDE = (Defects Found in Testing / Total Defects) × 100%

Total Defects = Pre-release Defects + Post-release Defects

Target: > 85% (catch most bugs before release)

Defect Leakage:

Leakage = (Production Defects / Total Defects) × 100%

Lower is better. Target: < 10%

Defect Removal Efficiency:

DRE = (Defects Removed / Defects Injected) × 100%

Track by phase:
- Unit Testing DRE
- Integration Testing DRE
- System Testing DRE

3. Test Execution Metrics

Test Pass Rate:

Pass Rate = (Passed Tests / Total Tests) × 100%

Track trends:
- Sprint over sprint
- Build over build
- Environment comparison

Test Automation Coverage:

Automation % = (Automated Tests / Total Tests) × 100%

Track:
- By test type (unit, integration, E2E)
- By feature area
- Growth over time

Test Execution Time:

Avg Execution Time = Total Time / Number of Test Runs

Goals:
- Unit tests: < 10 minutes
- Integration: < 30 minutes
- E2E suite: < 2 hours

4. Time-Based Metrics

Mean Time to Detect (MTTD):

MTTD = Time Bug Introduced → Time Bug Detected

Lower is better. Target: < 1 sprint

Mean Time to Resolve (MTTR):

MTTR = Time Bug Reported → Time Bug Fixed & Verified

Track by severity:
- P0: < 4 hours
- P1: < 24 hours
- P2: < 1 week

Test Cycle Time:

Cycle Time = Start of Testing → End of Testing

Optimize to:
- Reduce feedback loops
- Enable faster releases
- Improve time to market

5. Quality Gate Metrics

Release Readiness:

Readiness Score = Weighted Average of:
- Test coverage (30%)
- Defect density (25%)
- P0/P1 bugs (25%)
- Automation coverage (20%)

> 80% = Ready to release

Build Stability:

Stability = (Successful Builds / Total Builds) × 100%

Green Build Rate Target: > 90%

Mermaid Diagram: Metrics Dashboard Flow

graph TD
    A[Raw Data] --> B[Metrics Calculation]
    B --> C[Dashboard Visualization]
    C --> D{Threshold Met?}
    D -->|Yes| E[Green Status]
    D -->|No| F[Alert & Action]
    F --> G[Root Cause Analysis]
    G --> H[Improvement Actions]
    H --> A
    E --> I[Continue Monitoring]
    I --> A

Dashboard Design

Executive Dashboard

High-Level KPIs:

┌─────────────────────────────────────┐
│  Quality Health Score: 85%  ✓       │
├─────────────────────────────────────┤
│  Defect Trend         ↓ 12%         │
│  Test Coverage        ↑ 78%         │
│  Automation %         → 65%         │
│  Release Readiness    ✓ 88%         │
└─────────────────────────────────────┘

Trend Visualization:

Quality Score (Last 6 Months)
100% ┤                            ╭─
 75% ┤                    ╭───────╯
 50% ┤        ╭───────────╯
 25% ┤────────╯
  0% └─────────────────────────────────
     Jan  Feb  Mar  Apr  May  Jun

Team Dashboard

Detailed Metrics:

Test Execution:
- Total Tests: 1,247
- Passed: 1,189 (95.3%)
- Failed: 42 (3.4%)
- Blocked: 16 (1.3%)

Automation:
- Automated: 812 (65%)
- Manual: 435 (35%)
- In Progress: 23

Defects:
- Open: 15 (P0: 0, P1: 2, P2: 13)
- Fixed: 127
- Verified: 98

Reporting Best Practices

1. Actionable Insights

Bad Report:

"We executed 500 test cases this week."

Good Report:

"We achieved 92% test coverage of critical user paths.
Regression suite execution time reduced by 30% through
automation. 3 high-priority defects remain blocking
release - addressing with dedicated resources."

2. Trend Analysis

Show Progression:

Sprint Comparison:
Sprint N-2: 78% pass rate
Sprint N-1: 85% pass rate
Sprint N:   92% pass rate  ↑ 14% improvement

Action: Automation investment paying off

3. Risk Communication

Risk-Based Reporting:

HIGH RISK:
✗ Payment flow: 2 P1 defects open
✗ API integration: Performance degradation

MEDIUM RISK:
⚠ User profile: UI inconsistencies
⚠ Search: Edge cases uncovered

LOW RISK:
✓ Admin panel: All tests passing
✓ Reporting: Stable

Metric Implementation

Data Collection

Automated Collection:

# CI/CD Pipeline Metrics
- test_results:
    total: ${TOTAL_TESTS}
    passed: ${PASSED_TESTS}
    failed: ${FAILED_TESTS}
    duration: ${EXECUTION_TIME}

- code_coverage:
    line_coverage: ${LINE_COV}
    branch_coverage: ${BRANCH_COV}

- defects:
    total_open: ${OPEN_BUGS}
    by_severity: ${BUGS_BY_SEV}

Integration Points:

Test Management Tool → Data Warehouse → BI Dashboard
                    ↓
              Slack/Email Alerts

Metric Governance

Review Cadence:

Daily:
- Build success rate
- Critical defects
- Test execution status

Weekly:
- Test coverage trends
- Defect density
- Automation progress

Monthly:
- Quality score
- DDE/MTTR trends
- Strategic metrics

Quarterly:
- ROI analysis
- Tool effectiveness
- Process improvements

Common Pitfalls

❌ Vanity Metrics

Problem: Tracking metrics that look good but don't drive action
Example: "1000 test cases written" (without coverage context)

Solution: Focus on outcome metrics

❌ Too Many Metrics

Problem: Overwhelming stakeholders with data
Example: 50-metric dashboard

Solution: 5-7 key metrics max

❌ Lack of Context

Problem: Numbers without interpretation
Example: "Defect density: 5.2"

Solution: "Defect density: 5.2 (↓ from 7.1, target: <4.0)"

❌ Gaming Metrics

Problem: Teams optimize for metric, not quality
Example: High test count, low meaningful coverage

Solution: Balanced scorecard with quality outcomes

Metrics Template

# Quality Metrics Report - [Sprint/Release]

## Executive Summary
Quality Score: XX% (↑/↓ from last period)
Release Readiness: Yes/No
Key Risks: [List top 3]

## Test Coverage
- Requirements: XX%
- Code Coverage: XX%
- Critical Paths: XX%

## Defect Analysis
- Total Defects: XX
- By Severity: P0(X), P1(X), P2(X)
- Defect Leakage: XX%
- MTTR: XX hours

## Test Execution
- Pass Rate: XX%
- Automation %: XX%
- Execution Time: XX hours

## Trends
[Charts showing 3-month trends]

## Action Items
1. [Critical action]
2. [Improvement area]
3. [Next steps]

Key Takeaways

✅ Focus on actionable metrics, not vanity metrics
✅ Communicate trends, not just snapshots
✅ Align metrics with business objectives
✅ Keep dashboards simple (5-7 key metrics)
✅ Automate data collection
✅ Review and adjust metrics regularly
✅ Tell a story with data

Effective metrics drive better decisions and demonstrate QA’s business value!

Knowledge Check

What is Defect Detection Effectiveness (DDE)?

The rate at which defects are fixed
The density of defects per KLOC
The percentage of defects found in testing vs total defects
The time taken to detect a defect

What is a good target for defect leakage?

< 25%
< 10%
< 50%
< 5%

Which is an example of a vanity metric?

Number of test cases executed
Defect detection effectiveness
Test coverage of critical paths
Mean time to detect (MTTD)