Test execution metrics dashboard missing automated test results from CI/CD pipeline

ALM Atlassian Jira Software
, , , , , , , ,
1

Our test execution dashboard in Jira 8 is not capturing automated test results from our Jenkins CI/CD pipeline. Manual test executions appear correctly, but automated runs from Jenkins aren’t updating test case statuses or metrics.

We’ve configured the Jenkins plugin to post results via webhook, and I can see successful POST requests in Jenkins logs. However, the Jira dashboard shows incomplete test coverage - only manual executions are counted.

Webhook configuration looks correct:

{
  "url": "https://jira.company.com/rest/api/2/issue/bulk",
  "auth": "Bearer ${JIRA_TOKEN}"
}

The test result payload schema includes test case keys, status, and execution timestamps. API credentials are validated and working for other integrations. What’s preventing automated test results from appearing in the quality dashboard?

2

Good point Alex. I checked and the webhook is targeting issue type “Test Case” which matches our manual executions. However, I’m wondering if the payload format is the issue. The dashboard might be looking for specific fields that the automated results aren’t populating. Is there a schema validation I can run to compare manual vs automated result formats?

3

I bet it’s the test case key mapping. When Jenkins posts results, it needs to match test case keys exactly as they exist in Jira. If your automated tests use different identifiers (like test method names) instead of Jira issue keys, the webhook won’t know which test cases to update. Check your Jenkins test result parser configuration to ensure it’s extracting and sending actual Jira issue keys in the payload.

4

Check if your Jenkins plugin is sending results to the correct issue type. Jira’s test execution tracking usually requires specific issue types (like “Test” or “Test Execution”) rather than generic issues. Your webhook might be creating or updating the wrong issue type that’s excluded from dashboard filters.

5

The issue is likely in how test executions are linked to test cases. Jira quality dashboards typically track test execution records, not direct updates to test case issues. Your webhook might be updating test case status directly instead of creating linked test execution records that the dashboard aggregates.

6

I’ve implemented this exact integration successfully. The problem is multi-layered and requires addressing all four focus areas:

1. CI/CD Webhook Configuration: Your webhook endpoint is incorrect. The /rest/api/2/issue/bulk endpoint doesn’t handle test execution creation. Use the test execution-specific endpoint:

POST /rest/api/2/issue
Content-Type: application/json

Configure Jenkins to create Test Execution issues (not update Test Case issues directly). Each automated run should create a new execution record linked to the test case.

2. Test Result Payload Schema Validation: Your payload must include the test execution link. Here’s the correct structure:

{
  "fields": {
    "project": {"key": "QA"},
    "issuetype": {"name": "Test Execution"},
    "issuelinks": [{"type": {"name": "Tests"}, "inwardIssue": {"key": "TC-123"}}]
  }
}

The issuelinks array connects the execution to the test case. Without this, executions appear orphaned and don’t contribute to dashboard metrics.

3. API Credential Management: Verify your service account has these specific permissions:

  • Create Issues (Test Execution type)
  • Link Issues
  • Edit Issues in the target project
  • View Test Reports

Use a dedicated service account rather than personal tokens. Check Application Links if using OAuth instead of Bearer tokens.

4. Test Case Key Mapping Between Systems: This is critical. Your Jenkins tests must output Jira test case keys. Modify your test framework to include Jira keys in test annotations or metadata:

@Test
@JiraTestKey("TC-123")
public void testUserLogin() { ... }

Configure your Jenkins result parser (JUnit XML, TestNG, etc.) to extract these keys and include them in the webhook payload. Without accurate key mapping, the webhook can’t link executions to cases.

Dashboard Configuration: Ensure your quality dashboard filter includes Test Execution issue type and uses the “Tests” link relationship to aggregate results. The dashboard should query: issueType = "Test Execution" AND issue in linkedIssues("TC-*", "tests").

The root issue is that you’re trying to update test cases directly instead of creating linked test execution records. Jira’s test management model separates test case definitions from execution results for traceability and historical tracking.

7

Also verify the webhook authentication scope. Your Bearer token might have permissions to create issues but not update existing test executions. I’ve seen cases where the API credentials work for some operations but lack the specific permission needed to transition test case status or create execution records. Try using a project admin token temporarily to rule out permissions issues.