Quality metrics automation vs manual testing coverage: what's the right balance for agile teams?

ALM Rally (Broadcom Agile Central)
, , , , , , , ,
1

We’re using Rally 2023 for quality management across 8 agile teams. There’s an ongoing debate about how much to invest in automated metric collection versus maintaining detailed manual test coverage tracking.

Our automated dashboards pull defect density, test pass rates, and velocity metrics. But some teams argue we’re missing critical quality insights that only come from manual test result analysis and exploratory testing documentation.

How are other organizations balancing automated quality metrics with manual testing coverage in Rally? What’s proven most effective for risk-based test selection and overall quality assurance?

2

That makes sense - we probably shouldn’t apply the same approach across all 8 teams since they’re working on products at different maturity levels.

How do you handle the manual test result integration with your automated dashboards? Do you keep them separate or try to create unified quality views in Rally?

3

The key is recognizing they serve different purposes. Automated metrics give you trend visibility and early warning signals - things like defect escape rate, cycle time, and regression coverage. Manual test tracking captures domain knowledge, edge cases, and areas where automation isn’t cost-effective yet.

We typically aim for 70% automated metric collection for repeatable quality indicators, and 30% manual documentation for exploratory testing and complex integration scenarios. The manual work should focus on risk areas that automated metrics can’t adequately assess.

4

Don’t forget about the feedback loop speed. Automated metrics update in near real-time, giving teams immediate quality signals. Manual test documentation often lags by days or even sprints.

For continuous delivery, you need fast feedback on quality. That’s where automated metric collection really shines. Manual testing should focus on periodic deep dives and risk validation, not day-to-day quality assessment.

5

After seeing this discussion and reflecting on implementations across multiple organizations, here’s what I’ve observed works best:

Automated Metric Collection Configuration: Set up Rally dashboards to automatically track quantitative quality indicators:

  • Defect density per sprint/release
  • Test execution pass rate trends
  • Defect escape rate from each testing phase
  • Mean time to detect/resolve defects
  • Code coverage from CI/CD integration
  • Regression test stability metrics

These metrics should update automatically through Rally’s integration with your test automation framework and CI/CD pipeline. Configure alerts when metrics cross thresholds (e.g., defect density >15 per sprint, pass rate <85%).

Manual Test Result Integration: Maintain manual test documentation in Rally for:

  • Exploratory testing sessions (time-boxed)
  • Complex integration scenarios requiring human judgment
  • Usability and user experience validation
  • Risk-based testing of critical business flows
  • Edge cases discovered during development

Structure manual test cases with consistent fields: Risk Level, Test Type, Expected Result, Actual Result, Defects Found. This allows dashboard aggregation alongside automated metrics.

Dashboard Aggregation Strategy: Create role-specific quality dashboards:

Team Dashboard - Real-time automated metrics (defect trends, test pass rates, velocity) with links to supporting manual test details. Updated continuously.

Leadership Dashboard - Aggregated quality health across all teams, combining automated KPIs with manual risk assessments. Updated weekly.

Product Dashboard - Feature-level quality view showing both automated test coverage and manual validation status. Updated per sprint.

The aggregation should show automated metrics as the primary view with drill-down capability into manual test insights for context.

Risk-Based Test Selection: This is where manual and automated approaches must work together. Use Rally’s custom fields to tag features/stories with risk levels (High/Medium/Low).

High-risk items get both automated regression coverage AND manual exploratory testing. Medium-risk items rely primarily on automated tests with periodic manual validation. Low-risk items use automated tests only.

Configure your Rally dashboards to show test coverage by risk level, highlighting gaps where high-risk features lack sufficient manual validation despite passing automated tests.

Practical Balance Recommendation: For mature agile teams:

  • 80% of ongoing quality assessment through automated metrics
  • 20% through manual testing documentation and analysis

For new products/teams:

  • 60% automated metrics (still essential for trend visibility)
  • 40% manual testing (higher exploration and learning phase)

The balance should shift over time as product maturity increases and automated test coverage expands. Review quarterly and adjust based on defect escape trends and team feedback.

Most importantly: Don’t let perfect be the enemy of good. Start with basic automated metric collection in Rally, then incrementally add manual test integration as you identify gaps in quality visibility. The goal is actionable insights, not comprehensive documentation.

6

We integrate them through custom Rally fields and dashboard widgets. Manual test results feed into the same defect tracking workflow as automated tests, so our quality dashboards show a combined view.

The trick is standardizing your manual test result format so it can be aggregated with automated metrics. Use consistent pass/fail criteria, risk classifications, and coverage tags. Then your dashboard can slice quality data by test type (automated vs manual) while still showing overall product health.

7

I’d push back on fixed percentages. It depends heavily on your product maturity and team capability. Early-stage products need more manual exploration and risk-based testing. Mature products with stable APIs can rely more heavily on automated metric dashboards.

In Rally, we configure different dashboard aggregations for different product stages. New features get manual test case tracking with detailed pass/fail criteria. Established components rely on automated test execution metrics and defect trend analysis.