Test execution metrics in Test Plans not updating after pipeline runs

ALM Azure DevOps
, , , , , , , ,
1

We’re running automated tests through Azure Pipelines and publishing results to Test Plans, but the metrics dashboard isn’t reflecting the latest execution data. The pipeline completes successfully and shows test results in the build summary, but when we open the Test Plans metrics view, the pass/fail counts remain stale from 3 days ago.

Our publishing task looks like this:

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-results/*.xml'
    testRunTitle: 'API Integration Tests'

The Test Plan is associated with the build definition, and pipeline service account has Contributor access to the project. We’ve tried manually refreshing the dashboard and clearing browser cache. The Test Runs tab shows recent executions, but aggregate metrics like trend charts and pass rate percentages don’t update. This is blocking our sprint review presentations where we need current quality metrics. Any ideas what could cause the Test Results publishing task to complete without updating Test Plan metrics?

2

I’ve seen similar behavior when the Test Plan association isn’t configured at the test case level. The publishing task sends results to Azure DevOps, but if test cases in your Test Plan don’t have the automated test method mapped, metrics won’t aggregate properly. Check if your test cases have the ‘Associated Automation’ field populated with the correct test method identifier that matches what’s in your test results XML.

3

Update: We got the permissions sorted and modified our publishing task. Here’s what actually fixed the metrics dashboard refresh issue:

1. Pipeline Service Account Permissions The account running the pipeline needs ‘Manage test plans’ permission in Test Plans security settings. Project-level Contributor access isn’t sufficient because Test Plans has its own permission scope for updating metrics and aggregations.

2. Test Plan Association Configuration In your pipeline YAML, explicitly specify the test plan and configuration IDs:

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-results/*.xml'
    testRunTitle: 'API Tests - Build $(Build.BuildNumber)'
    publishRunAttachments: true
    configuration: 'Windows 10'
    testPlanId: 12345
    testConfigurationId: 67890

3. Metrics Dashboard Refresh Timing The dashboard doesn’t update in real-time. Azure DevOps processes test results asynchronously and recalculates aggregate metrics every 15-30 minutes. After fixing permissions, we saw metrics update within 20 minutes of pipeline completion.

4. Test Case Automation Mapping Ensure test cases in your Test Plan have the ‘Associated Automation’ field populated with the exact test method name from your JUnit XML results. The format should match: namespace.classname.methodname. Without this mapping, results publish but don’t link to Test Plan cases for metrics rollup.

5. Unique Test Run Titles Appending build number or timestamp to testRunTitle ensures each execution creates a new test run rather than updating an existing one, which prevents metrics aggregation conflicts.

After implementing all five changes, our metrics dashboard now updates reliably within 25 minutes of pipeline completion. The trend charts, pass rate percentages, and quality metrics all reflect current test execution data. The key was the Test Plans-specific permission - that was the blocker preventing metrics from being written back to the dashboard.

4

Another thing to verify - is your testRunTitle parameter unique for each execution? I’ve noticed that if you reuse the same test run title, Azure DevOps sometimes treats it as an update to an existing run rather than creating a new one, which can mess up the metrics aggregation logic. Try appending a timestamp or build number to make each run distinct. Also, the mergeTestResults parameter might affect how results roll up into Test Plan metrics.

5

Thanks Sarah. I checked and about 60% of our test cases do have Associated Automation configured. Those should at least show updated metrics, right? The weird part is even those aren’t reflecting in the dashboard trends. The Test Runs tab shows all executions correctly, so the data is getting into Azure DevOps somewhere.

6

Tom, that was helpful - I checked and the service account only had ‘View test runs’ permission in Test Plans. I’ve requested the Manage permission from our admin. Jen, we are using a static title, so I’ll add the build ID to make it unique. Will update once permissions are changed and we run another pipeline.