Defects auto-created multiple times for same failed test execution

ALM Azure DevOps
, , , , , , , ,
1

Our CI/CD pipeline is creating duplicate defects for the same test failures. We have a YAML pipeline configured to run automated tests and auto-create bugs when tests fail. The problem is that if a test fails multiple times across different pipeline runs, we get multiple Bug work items created for what is essentially the same issue.

For example, test ‘LoginValidation_InvalidCredentials’ failed in pipeline run #452, #455, and #458. We now have three separate Bug work items (#8901, #8934, #8967) all describing the same test failure.

Current pipeline configuration:

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'
    failTaskOnFailedTests: true

The test result publishing seems to lack any duplicate detection logic. Is there a way to configure Azure Pipelines to check if a Bug already exists for a specific test case before creating a new one? Our backlog is getting clogged with duplicate defects that require manual cleanup.

2

That tagging approach makes sense. How do you handle the initial Bug creation? Do you still use PublishTestResults or do you create Bugs manually via REST API based on test results?

3

Consider using Azure DevOps work item query results as your duplicate check. Before creating a Bug, run a WIQL query filtering by Title (containing test name) and State (not Closed). If results return > 0, skip Bug creation. This is simpler than custom tagging and works with existing work items.

4

The PublishTestResults task doesn’t have built-in duplicate detection. It’s designed to publish results, not manage work item creation. You’ll need a separate task or custom script to handle the deduplication logic.

5

We solved this by implementing a custom deduplication service. Before creating a Bug, we query existing Bugs with a specific tag pattern (e.g., ‘AutoTest:LoginValidation_InvalidCredentials’). If a Bug with that tag exists and is in an open state (New, Active, Committed), we update it with the new failure information instead of creating a duplicate. If the Bug is Resolved or Closed, we create a new one since it’s a regression. The logic runs in a PowerShell task after test execution but before result publishing.

6

Here’s a comprehensive solution addressing all aspects of duplicate defect prevention:

Test Result Publishing Configuration: Keep your PublishTestResults task as-is for visibility, but disable automatic Bug creation. The built-in auto-create feature has no duplicate detection and will always create new work items.

Duplicate Detection Logic Implementation: Add a PowerShell task after test publishing that implements smart duplicate detection:

$failedTests = Get-FailedTests # Parse test results
foreach ($test in $failedTests) {
  $query = "SELECT [System.Id] FROM WorkItems WHERE [System.WorkItemType] = 'Bug' AND [System.Title] CONTAINS '$($test.Name)' AND [System.State] <> 'Closed'"
  $existing = Invoke-RestMethod -Uri $wiqlEndpoint -Method POST -Body $query
  if ($existing.workItems.Count -eq 0) {
    New-Bug -TestName $test.Name -StackTrace $test.Error
  }
}

Pipeline Task Configuration: Structure your YAML to separate concerns:

  1. Run tests
  2. Publish results (for reporting)
  3. Parse failed tests (custom script)
  4. Check for existing Bugs (WIQL query)
  5. Create Bugs only for new failures (REST API)

Custom Deduplication Strategy: Implement a three-tier matching approach:

  • Tier 1: Exact test name match in Bug title
  • Tier 2: Test name + similar error message (use fuzzy matching)
  • Tier 3: Test suite + failure category

This prevents both exact duplicates and near-duplicates where test names vary slightly.

Handling Regressions: Your deduplication logic should differentiate between:

  • Active failures: Update existing Bug with new run info
  • Regressions: Create new Bug if previous was Resolved/Closed

Add a comment to existing Bugs when updating:


Failed again in pipeline run #{buildId}
Timestamp: {timestamp}
Branch: {sourceBranch}

Performance Optimization:

  • Cache WIQL query results at the start of the deduplication task
  • Batch REST API calls for multiple Bug creations
  • Use parallel processing for large test suites (100+ tests)

Backlog Cleanup: For your existing duplicate Bugs, create a one-time cleanup script:

  1. Query all Bugs with ‘AutoTest’ tag or similar identifier
  2. Group by test name
  3. Keep the most recent Bug, close others as ‘Duplicate’
  4. Link closed Bugs to the kept Bug for traceability

Alternative Approach: If you want minimal pipeline changes, use Azure DevOps Analytics to create a duplicate detection dashboard. Flag potential duplicates daily and handle them in a manual review process. Less automated but requires no pipeline modifications.

The WIQL-based duplicate check is the most reliable approach and integrates cleanly with existing Azure Pipelines infrastructure.

7

We use the CreateWorkItem task from the Azure DevOps extension marketplace, but wrapped it in a conditional check. The condition queries for existing Bugs matching the test case name. Only if the query returns empty does the CreateWorkItem task execute. Works well but adds about 15 seconds to pipeline runtime for the query execution.