Automated compliance audit reports from test automation results

ALM Atlassian Jira Software
, , , , , , , ,
1

We successfully automated our SOC2 compliance evidence generation using Xray test execution results and ScriptRunner scheduled jobs. Previously, our audit preparation required 40+ hours of manual work each quarter to compile test evidence, map requirements to test cases, and generate coverage reports.

Our implementation uses ScriptRunner’s scheduled jobs to query Xray test execution data via REST API, extract traceability links between requirements and test cases, and generate formatted audit reports in PDF format. The automation runs weekly, maintaining a continuous evidence trail that auditors can review at any time.

Key benefits we’ve achieved:

  • Zero manual effort for SOC2 evidence compilation
  • Real-time visibility into requirement coverage gaps
  • Automated traceability matrix generation linking security controls to test cases
  • Historical trend analysis showing testing consistency over time

The solution handles our 500+ test cases across 12 security control domains, with complete traceability to Jira requirements tickets. Happy to share implementation details and lessons learned.

2

What about the PDF generation? Are you using a Jira plugin or external service for formatting the audit reports?

3

Great questions. For PDF generation, we use ScriptRunner’s built-in HTML rendering combined with a lightweight external service (wkhtmltopdf) running on our Jira DC infrastructure. The ScriptRunner job generates an HTML report with CSS styling matching our audit templates, then calls the PDF converter via local subprocess.

Scheduling Strategy and SOC2 Evidence Export: We run the main evidence collection job weekly on Saturday nights at 2 AM. This timing ensures zero business hour impact while maintaining fresh audit trails. The job takes approximately 12 minutes to process our 500+ test cases, generate traceability matrices, and export formatted reports to our compliance file share.

For SOC2 evidence export specifically, we create three artifacts:

  1. Complete Traceability Matrix - CSV mapping every security control to test cases with execution status
  2. Executive Summary PDF - High-level coverage metrics, pass/fail trends, and gap analysis
  3. Detailed Evidence Package - Individual test execution reports with screenshots and logs

Xray Traceability Deep Dive: The traceability magic happens through Xray’s requirement link types. We use custom link types ‘tests_control’ and ‘validates_requirement’ to create explicit connections. Our ScriptRunner job traverses these links bidirectionally - starting from control tickets to find tests, and from test executions back to requirements. This dual verification catches orphaned tests and unlinked controls.

Coverage Gap Analysis Implementation: Beyond the basic gap detection I mentioned earlier, we implemented severity-based alerting. Critical controls (authentication, encryption, access management) trigger immediate Slack notifications if gaps exceed 30 days. Standard controls allow 60-day windows. The system also tracks gap trends - if a control repeatedly falls into gap status, it’s flagged for test suite improvement.

Performance-wise, we optimized the API calls by batching requests and caching requirement metadata. Initial runs took 45+ minutes, but after optimization we’re down to 12 minutes for our full dataset. The key was reducing individual REST calls and using bulk query endpoints where Xray supports them.

One lesson learned: maintain a separate ‘audit archive’ project in Jira where historical reports are attached to dated tickets. This creates an immutable audit trail that auditors love - they can see exactly what evidence existed at any point in time.

The entire solution runs autonomously now. Our compliance team receives weekly email summaries with the PDF attached, and they only intervene when gap alerts appear. Auditors have praised the continuous evidence approach versus our old ‘scramble before audit’ methodology.

4

This is exactly what we need. Currently spending entire weeks preparing for quarterly audits. How do you handle the Xray API queries? Are you using GraphQL or REST endpoints to pull test execution history?

5

Coverage gap analysis happens in two stages. First, we maintain a master list of all SOC2 control requirements as Jira tickets with a custom field ‘SOC2_Control_ID’. Our ScriptRunner job queries all control tickets, then cross-references them against Xray test cases using the requirement links.

Any control ticket without linked test cases triggers an alert. We also check execution frequency - controls requiring quarterly testing that haven’t been executed in 90+ days show as gaps. The report includes a dedicated ‘Coverage Gaps’ section with control IDs, descriptions, and days since last execution. This proactive alerting reduced our audit findings from 8 last year to zero this quarter.

6

We use Xray’s REST API exclusively. The key endpoints are /rest/raven/1.0/api/testexec for execution details and /rest/raven/1.0/api/test for test case metadata including requirement links. Our ScriptRunner job runs this query structure:

// Query test executions for date range
result = get("/rest/raven/1.0/api/testexec?startDate=${startDate}&endDate=${endDate}")
executions = parseJson(result)
executions.each { exec ->
  testLinks = get("/rest/raven/1.0/api/test/${exec.testKey}/requirement")
}

The response includes all traceability data we need. We then transform it into audit-friendly formats with requirement IDs, test results, and execution timestamps.