Implementing sprint test velocity tracking in codebeamer cb-24

ALM PTC Codebeamer
, , , , , , , ,
1

We successfully implemented a test velocity tracking dashboard in codebeamer cb-24 for our portfolio planning needs. This use case shares our approach to measuring sprint test execution velocity and predicting testing capacity.

Business Context: Our organization runs 15 concurrent scrum teams with varying test execution rates. Portfolio planning required visibility into each team’s test velocity to forecast release readiness accurately.

Implementation Overview: Built custom reports using cb-24’s sprint-mgmt module to calculate test velocity metrics:

SELECT sprint_id, COUNT(test_id) as tests_executed,
       SUM(execution_time) as total_time
FROM test_execution
WHERE sprint_id IN (last_5_sprints)

The dashboard aggregates test case execution data per sprint, calculates average velocity, and projects future sprint capacity. Integration with the custom reports module enables automated velocity trend analysis across the portfolio.

2

Thank you for the excellent questions. Here’s the comprehensive breakdown of our test velocity tracking implementation in codebeamer cb-24:

Velocity Calculation Methodology:

We use a weighted velocity model that accounts for test complexity:

  • Unit tests: 1 velocity point
  • Integration tests: 3 velocity points
  • End-to-end tests: 5 velocity points
  • Manual exploratory tests: 8 velocity points

This weighting reflects actual execution time differences and provides realistic capacity forecasting.

Multi-Sprint Test Handling: Tests are counted in the sprint where execution completes, not where they start. This aligns with our “done” definition and prevents double-counting. The dashboard tracks in-progress tests separately to provide visibility into work spanning sprint boundaries.

Velocity Segmentation: We maintain separate velocity tracks for automated and manual testing:

-- Pseudocode for segmented velocity query:
1. Query test executions grouped by sprint_id and test_type
2. Calculate weighted points per test based on complexity
3. Aggregate automated vs manual velocity separately
4. Store results in custom dashboard tables
5. Generate trend charts for 12-sprint rolling window

This segmentation is critical for portfolio planning since automated test velocity scales differently than manual test velocity as team size changes.

Dashboard Refresh and Data Pipeline: The dashboard updates via scheduled job every 4 hours during business hours, with manual refresh available. We chose batch updates over real-time for performance reasons - calculating velocity across 15 teams and 5 sprints of historical data is computationally expensive. The 4-hour refresh provides sufficient accuracy for portfolio planning while maintaining system performance.

Outlier Handling and Statistical Smoothing: We implemented a robust outlier detection mechanism:

  • Calculate 12-sprint moving average velocity per team
  • Identify sprints where velocity deviates >30% from moving average
  • Flag outliers for manual review (holidays, team transitions, tooling issues)
  • Use median velocity instead of mean for forecasting to reduce outlier impact

The system also tracks velocity variance as a quality metric - teams with high variance indicate unstable testing processes that need attention.

Test Status and Re-execution Impact: Failed tests are handled through a re-execution penalty factor. When a test fails and requires re-execution:

  • Initial execution counts toward velocity
  • Re-execution adds 0.5x velocity points (reflects rework overhead)
  • This penalizes velocity for quality issues, encouraging teams to improve test stability

Portfolio Planning Integration: The velocity dashboard feeds directly into our release planning tools. Portfolio managers can:

  • View aggregated velocity trends across all teams
  • Forecast testing capacity for upcoming releases
  • Identify teams with declining velocity (early warning for capacity issues)
  • Allocate test workload based on historical velocity data

Technical Implementation Details: Built using cb-24’s custom report framework with SQL-based data aggregation. The dashboard includes:

  • Velocity trend charts (line graphs per team)
  • Capacity heatmap (shows team availability vs planned test load)
  • Predictive forecasting (projects completion dates based on velocity trends)
  • Drill-down capability (click team to see individual sprint details)

Lessons Learned:

  1. Start with simple velocity calculation, add complexity incrementally
  2. Involve teams in defining velocity weights - they know test effort best
  3. Make velocity data transparent - teams use it to improve their processes
  4. Don’t use velocity for team comparison - context differs too much
  5. Combine velocity with quality metrics (defect escape rate) for complete picture

Results After 6 Months:

  • Portfolio planning accuracy improved 40% (fewer release date slips)
  • Test capacity forecasting within 15% of actual (previously 35% variance)
  • Teams proactively address velocity declines before they impact releases
  • Automated test coverage increased 25% (teams see velocity benefit)

The implementation required about 80 hours of initial development and 4 hours monthly maintenance. ROI became positive within 3 months through improved release predictability and reduced planning overhead.

3

How do you handle velocity outliers? Some sprints have abnormally high or low test execution due to holidays, team changes, or infrastructure issues. We use a moving average with outlier detection to smooth velocity trends for more reliable portfolio planning predictions.

4

This is exactly what we need for our scaled agile setup. How did you handle test cases that span multiple sprints? Did you count them in the sprint where they started or where they completed?

5

Impressive implementation. I’m curious about the velocity calculation formula. Did you use simple test count per sprint, or did you factor in test complexity? We’ve found that counting all tests equally skews velocity metrics since integration tests take 10x longer than unit tests. Also, how frequently does the dashboard refresh - is it real-time or batch updated?

6

Great use case. One challenge we face is distinguishing between manual and automated test velocity. Automated tests execute much faster, so mixing them in velocity calculations creates misleading projections. Did you segment velocity by test type in your cb-24 implementation?