Test automation framework fails to sync test results back to Rally

ALM Rally (Broadcom Agile Central)
, , , , , , , ,
1

We’ve implemented a custom Jenkins pipeline that executes our Selenium test suite and attempts to sync results back to Rally using the REST API plugin. The test execution completes successfully in Jenkins, but the results aren’t appearing in Rally test case execution records.

I’ve verified the Jenkins REST API plugin configuration and the Rally API token has read/write permissions. The issue seems related to test result payload schema mapping - our parameterized builds pass additional metadata that might not align with Rally’s expected format.

{
  "testCaseResult": "Pass",
  "buildNumber": "${BUILD_ID}",
  "environment": "${TEST_ENV}"
}

The traceability between automated tests and Rally test cases is broken, making it impossible to track test coverage accurately. Has anyone successfully integrated parameterized Jenkins builds with Rally test case execution?

2

You need to troubleshoot each focus area systematically:

Jenkins REST API Plugin Configuration: Verify the endpoint URL format matches Rally’s API v2.0 specification: https://rally1.rallydev.com/slm/webservice/v2.0/testcaseresult/create. Ensure authentication headers include the ZSESSIONID token.

Rally API Token Permissions: Beyond TestCaseResult creation, verify your token has permissions for the specific workspace and project. Test with a simple curl command:

curl -H "ZSESSIONID: ${API_KEY}" \
  https://rally1.rallydev.com/slm/webservice/v2.0/testcaseresult

Test Result Payload Schema Mapping: Rally requires specific fields - TestCase (reference), Build, Verdict, Date. Your custom fields must be pre-defined in Rally workspace settings. Here’s a corrected payload structure:

{
  "TestCaseResult": {
    "TestCase": "/testcase/12345",
    "Build": "Jenkins-${BUILD_NUMBER}",
    "Verdict": "Pass",
    "Date": "2025-08-14T10:23:00Z",
    "Notes": "Environment: ${TEST_ENV}"
  }
}

Parameterized Build Handling: Use Jenkins Pipeline’s withEnv block to inject parameters before API calls. Avoid direct variable substitution in JSON strings - it causes type mismatches.

Implement these changes and enable debug logging in Jenkins (add -Dcom.rallydev.rest.client.debug=true to JVM args). The logs will show exact API request/response cycles, making it easier to identify schema validation failures. If you’re still seeing issues after fixing the payload structure, the problem might be with TestCase reference resolution - ensure your TestCase FormattedID exists in the target Rally workspace.

3

The issue is your custom fields in the payload. Rally’s TestCaseResult object has a strict schema and won’t accept arbitrary fields like environment or buildNumber unless they’re defined as custom fields in your workspace. You need to either map these to existing Rally fields or create custom fields first. I’d recommend using the Notes field for build metadata instead of trying to create new schema fields.

4

Thanks for the suggestions. I checked the API token and it has full TestCaseResult permissions. Plugin version is 2.4.1. I found some 400 errors in Jenkins logs mentioning ‘invalid field mapping’ but the error messages aren’t specific about which fields are problematic. Our parameterized builds pass custom fields that might not exist in Rally’s schema.

5

Can you verify your API token permissions include TestCaseResult creation rights? Go to Rally workspace settings and check the API key permissions. Also, the REST API plugin version matters - older versions had issues with complex JSON payloads. What version are you running?

6

I ran into similar issues last quarter. The problem is usually with how Jenkins handles parameterized build variables in the API payload. Rally expects specific field names and the environment variables you’re passing might be causing validation failures. Check your Jenkins console logs for any API response errors - they’re often hidden in the verbose output.

7

I’ve seen this exact scenario. The Jenkins REST API plugin doesn’t automatically handle parameter substitution in JSON payloads the way you’d expect. When you use ${BUILD_ID}, Jenkins might be sending that literal string instead of the resolved value. Try using environment variable injection in your pipeline script before constructing the payload, or use a Groovy script to build the JSON dynamically with proper variable resolution.