Your HTTP 400 error is caused by test result format mapping issues combined with incomplete post-build action configuration. Here’s the complete solution:
1. Jenkins Rally Plugin Installation
First, verify your plugin installation is complete:
- Jenkins > Manage Jenkins > Plugin Manager
- Installed tab > Search “Rally”
- Confirm version 2024.2 is active (you have this)
- Check for dependency plugins:
- Credentials Plugin (required)
- Plain Credentials Plugin (required)
- Junit Plugin (for test result parsing)
If dependencies are missing, install them and restart Jenkins.
2. API Credential Configuration
Your credentials can read Rally but can’t write results - this indicates a permission issue:
// Pseudocode - Verify API credentials:
1. Jenkins > Credentials > System > Global credentials
2. Find Rally API Key credential
3. Test permissions using Rally REST API:
GET /api/v2.0/testcaseresult (should return 200 OK)
POST /api/v2.0/testcaseresult (should return 200, not 403)
4. If POST fails, API key lacks "Create Test Results" permission
Fix Rally API permissions:
- Rally > Admin > User Permissions
- Find the API user account (service account for Jenkins)
- Grant these permissions:
- Test Management > Create Test Case Results
- Test Management > Edit Test Case Results
- Project > Editor role (minimum)
- Save and test API call again
3. Test Result Format Mapping
The HTTP 400 error indicates payload format issues. Rally expects this structure:
{
"TestCase": "/testcase/12345",
"Build": "Jenkins-#456",
"Verdict": "Pass",
"Date": "2025-11-03T08:45:00Z",
"Duration": 2.5,
"Notes": "Executed via Jenkins"
}
But Jenkins JUnit output looks like:
<testcase classname="com.example.UserLoginTest"
name="testSuccessfulLogin"
time="2.5"/>
Configure format mapping in Jenkins:
- Job Configuration > Post-build Actions > Rally Test Result Publisher
- Enable “Custom Test Result Mapping”
- Set mapping rules:
// Pseudocode - Test result mapping configuration:
1. Source Format: JUnit XML
2. Test Case ID Resolution:
- Pattern: Extract TC-#### from test case annotation or name
- Fallback: Use classname + method name lookup table
3. Verdict Mapping:
- JUnit success → Rally "Pass"
- JUnit failure → Rally "Fail"
- JUnit skipped → Rally "Inconclusive"
4. Build Reference: Use Jenkins ${BUILD_NUMBER}
5. Timestamp: Use test execution time from JUnit XML
Test Case ID Resolution Options:
Option A: Use annotations in test code (recommended)
@Test
@RallyTestCase("TC-1234")
public void testSuccessfulLogin() {
// test code
}
Jenkins plugin extracts TC-1234 from annotation.
Option B: Use mapping file
Create rally-testcase-mapping.csv:
com.example.UserLoginTest.testSuccessfulLogin,TC-1234
com.example.UserLoginTest.testFailedLogin,TC-1235
Upload to Jenkins job workspace and reference in plugin config.
Option C: Use naming convention
Rename tests to include Rally ID:
public void testTC1234_UserLogin() {
// Plugin extracts TC-1234 from method name
}
4. Post-Build Action Configuration
Configure the Rally Test Result Publisher post-build action:
// Pseudocode - Complete post-build configuration:
1. Add Post-Build Action: "Publish Test Results to Rally"
2. Rally Connection:
- Server URL: https://rally1.rallydev.com
- Workspace: Your workspace name
- Project: Your project name
- Credentials: Select API key credential
3. Test Result Source:
- Result Files: **/target/surefire-reports/*.xml
- Format: JUnit XML
4. Test Case Mapping:
- Mapping Mode: Annotation-based (or File-based)
- Mapping File: rally-testcase-mapping.csv (if using Option B)
5. Result Options:
- Create Missing Test Results: Enabled
- Attach Build Log: Enabled (optional)
- Build Reference Format: Jenkins-${BUILD_NUMBER}
6. Failure Handling:
- On Mapping Failure: Log warning and continue
- On Upload Failure: Mark build unstable
Critical Configuration Point:
The “Create Missing Test Results” option must be enabled. This tells the plugin to create new TestCaseResult objects in Rally even if none exist yet for this build. Without this, the plugin expects existing results and fails.
Debugging Your Specific Error
Your HTTP 400 error with “Failed to create test case results” typically means:
-
Test Case doesn’t exist: Plugin trying to reference TC-#### that doesn’t exist in Rally
- Fix: Verify all test case IDs in mapping exist in Rally project
- Query Rally: `GET /api/v2.0/testcase?query=(FormattedID = “TC-1234”)
-
Invalid verdict value: Mapping produced invalid status
- Fix: Ensure verdict mapping uses exact Rally values: “Pass”, “Fail”, “Inconclusive”, “Blocked”
- Check for typos: “Passed” ≠ “Pass”
-
Missing required fields: Rally test result missing mandatory fields
- Fix: Ensure mapping includes TestCase reference and Date
- TestCase must be full URL:
/testcase/12345 not just `12345
-
Build reference format: Rally doesn’t accept the build identifier format
- Fix: Use alphanumeric format:
Jenkins-456 not Jenkins #456 (no special chars)
Testing the Configuration
- Create a simple test job with one test case
- Ensure test case TC-9999 exists in Rally (create manually if needed)
- Add annotation to test: `@RallyTestCase(“TC-9999”)
- Run build and check Jenkins console output for detailed error messages
- If successful, expand to full test suite
Enable Debug Logging
For detailed troubleshooting:
- Jenkins > Manage Jenkins > System Log
- Add new log recorder: “Rally Integration”
- Add logger:
com.rallydev.jenkins at DEBUG level
- Run build and review logs for exact API payloads and responses
Common Solutions Summary
- If test cases don’t exist: Create them in Rally first, or enable auto-creation in plugin (if available in your version)
- If mapping fails: Use annotation-based approach instead of naming conventions
- If permissions fail: Grant API user “Editor” role on project
- If format fails: Verify JUnit XML is valid (
xmllint --noout test-results.xml)
Once configured correctly, your Jenkins builds will automatically create test case results in Rally with proper verdict, timestamp, and build reference.