Here’s a complete solution addressing all focus areas:
Xray Test Coverage and Execution Entities:
Xray uses three key entities:
- Test (issue type): The test case definition
- Test Execution (issue type): A container for test runs
- Test Run: The actual execution result (Pass/Fail/etc.) linking a Test to a Test Execution
Requirements link to Tests via “tests” link type. Tests are executed within Test Executions. The challenge is that requirement status needs to reflect the aggregate of all Test Runs across potentially multiple Test Executions.
Automation Rules Triggered from Test Executions:
Your current approach triggers on individual test execution changes, which can’t evaluate aggregate status. Here’s a better trigger strategy:
-
Use Xray’s built-in “Requirement Test Summary” custom field (enable in Xray > Settings > Test Coverage)
-
This field automatically aggregates all test results for a requirement
-
Trigger your automation rule on field value changes:
WHEN: Field value changed
FIELD: Requirement Test Summary
CONDITION: New value = "Failed" OR "Partial"
ACTION: Transition issue to "In Progress"
-
Create a second rule for successful verification:
WHEN: Field value changed
FIELD: Requirement Test Summary
CONDITION: New value = "Passed"
ACTION: Transition issue to "Verified"
Link-Based Evaluation of All Related Tests:
If you can’t use the Xray summary field, implement a link-based evaluation using a scheduled rule:
TRIGGER: Scheduled (every 2 hours)
JQL: project = REQ AND status = Verified AND issueFunction in hasLinks("tests")
FOR EACH: Matching issues
BRANCH: Related issues where linkType = "tests"
CONDITION: Advanced compare - Check if ANY linked test has latest execution status = FAIL
ACTION: Transition parent requirement to "In Progress"
COMMENT: "Requirement rolled back due to failed test: {{issue.key}}"
This queries all requirements in Verified status, examines their linked tests, and rolls back any that have failed executions.
Requirement Workflow Status Rollback:
Implement a two-tier workflow for requirements:
-
Draft → Ready for Test → In Test → Verified → Approved
-
Add a transition “Roll Back” from Verified to In Test with these conditions:
- Requirement Test Summary != “Passed”
- OR any linked test execution status = Failed
-
Make this transition available to automation rules (not just manual users)
-
Add a post-function that logs the rollback reason:
Rolled back due to test failure
Failed tests: [list of test keys]
Timestamp: {{now}}
Quality Gate Custom Field Integration:
Integrate the Xray quality gate field into your workflow:
-
Enable “Requirement Test Summary” in Xray settings
-
Add this field to your requirement screen layout
-
Create a condition on the “Verify” transition:
Requirement Test Summary = "Passed"
This prevents manual verification if tests haven’t all passed
-
Create an automation rule that monitors this field:
TRIGGER: Field value changed (Requirement Test Summary)
CONDITION:
IF new value = "Failed" OR "Partial"
AND current status = "Verified"
THEN: Transition to "In Test"
ADD COMMENT: "Automated rollback: Test coverage incomplete or failed"
-
Add a dashboard gadget showing requirements with mismatched status:
JQL: project = REQ
AND status = Verified
AND "Requirement Test Summary" != "Passed"
This helps identify requirements that slipped through before the automation was fixed
Complete Automation Rule Configuration:
Replace your current rule with these two rules:
Rule 1: Rollback on Test Failure
Name: Requirement - Rollback on Test Failure
Trigger: Field value changed
Field: Requirement Test Summary
IF Conditions:
- New value = "Failed" OR "Partial"
- Current status = "Verified" OR "Approved"
THEN Actions:
- Transition issue: "Roll Back to In Test"
- Add comment: "Automated rollback due to test failure. Summary status: {{issue.Requirement Test Summary}}"
- Send notification to: {{issue.Assignee}}, {{issue.Reporter}}
Rule 2: Verify on All Tests Passed
Name: Requirement - Auto-Verify on Test Success
Trigger: Field value changed
Field: Requirement Test Summary
IF Conditions:
- New value = "Passed"
- Current status = "In Test"
- All linked tests have status "Done" (optional additional check)
THEN Actions:
- Transition issue: "Verify"
- Add comment: "All tests passed. Requirement automatically verified."
- Send notification to: Stakeholders
Handling Timing and Edge Cases:
- Xray updates the summary field within 5-10 seconds of test execution completion
- If you need immediate rollback, add a third rule triggered directly by test execution status changes that sets a “Needs Review” flag
- The summary field-based rule then clears the flag and performs the actual transition
- For requirements with 100+ linked tests, the summary field update may take up to 30 seconds
Verification and Monitoring:
Create a saved filter to monitor the system:
project = REQ
AND status = Verified
AND "Requirement Test Summary" in ("Failed", "Partial", "Not Tested")
ORDER BY updated DESC
Run this daily to catch any requirements that weren’t properly rolled back. This gives you confidence that the automation is working correctly and stakeholders are seeing accurate requirement status.