Risk management flags blocking automated test execution in production deployment pipeline

ALM IBM Engineering Lifecycle Management
, , , , , , , ,
1

We’ve integrated ELM’s risk management module with our Jenkins deployment pipeline, but high-risk test cases are now blocking production deployments even when they pass successfully.

The risk management module flags certain test cases based on historical failure rates and complexity scores. When a flagged test is included in the deployment validation suite, Jenkins fails the pipeline with:


Risk Assessment: FAILED
High-risk test cases detected: 12
Risk score threshold exceeded: 85/75

These tests all passed, but the pipeline won’t proceed because the cumulative risk score is above our configured threshold of 75. We need these tests in our validation suite - they cover critical payment processing functionality. The issue is delaying our deployments by days while we manually review and approve risk exceptions. How do others handle risk exception rules in automated pipelines? Can we configure pipeline-specific thresholds or is there a way to update the scoring model to be less conservative?

2

I can provide a comprehensive solution that addresses all aspects of risk management in automated pipelines:

1. Risk Exception Rules Configuration: Navigate to ELM Admin Console > Risk Management > Exception Rules and create targeted exceptions:

  • Category-Based Exceptions: Create exceptions for “Critical Payment Tests” category
  • Tag-Based Exceptions: Tag tests with auto-approve-risk for automated pipeline bypass
  • Conditional Exceptions: Set rules like “IF test_status=PASSED AND risk_score<90 THEN approve”

This allows critical tests to proceed while maintaining risk visibility.

2. Pipeline-Specific Thresholds: Implement dynamic threshold configuration in your Jenkins pipeline:

stage('Risk Assessment') {
    script {
        def threshold = env.ENVIRONMENT == 'production' ? 75 : 85
        def riskScore = elmRiskAPI.calculateScore(testSuite)
        if (riskScore > threshold) {
            error("Risk threshold exceeded: ${riskScore}/${threshold}")
        }
    }
}

This provides environment-appropriate risk tolerance without manual intervention.

3. Scoring Model Updates: Recalibrate your risk scoring model to weight recent outcomes more heavily:

  • Access Risk Management > Scoring Configuration
  • Adjust “Historical Failure Weight” from default 0.7 to 0.4
  • Increase “Recent Success Weight” from 0.3 to 0.6
  • Set “Success Decay Period” to 30 days

This makes the model more responsive to improving test stability.

4. Override Workflows for Manual Review: Implement a structured override process:

  • Configure approval gates in Jenkins for risk scores between 75-85
  • Scores above 85 require change board approval
  • Auto-approve scores below 75
  • Log all overrides to ELM audit trail

Example Jenkins configuration:

if (riskScore > 75 && riskScore < 85) {
    timeout(time: 2, unit: 'HOURS') {
        input message: "Risk score ${riskScore}. Approve deployment?",
              submitter: 'release-managers'
    }
}

5. Risk Score Calculation Optimization: Refine how risk scores are calculated for test suites:

  • Use weighted average instead of cumulative score
  • Factor in test pass rate over last 10 runs
  • Consider test execution time (longer tests = higher risk of flakiness)
  • Apply domain-specific multipliers (payment tests weighted 1.5x)

6. Monitoring and Continuous Improvement: Set up risk management dashboards:

  • Track false positive rate (tests flagged high-risk but consistently passing)
  • Monitor exception usage patterns
  • Review threshold breach frequency
  • Quarterly scoring model recalibration based on data

7. Best Practices for Your Payment Tests: For critical payment processing tests specifically:

  • Create a “Verified Critical” designation that bypasses standard risk scoring
  • Require these tests to pass in both staging and production-like environments
  • Implement additional validation layers (security scans, compliance checks)
  • Document risk acceptance in deployment records

8. Integration Architecture: Ensure your Jenkins-ELM integration supports risk management:

elmRiskAPI.evaluateTestSuite([
    suiteId: testSuiteId,
    environment: env.ENVIRONMENT,
    threshold: dynamicThreshold,
    exceptionTags: ['critical-payment', 'auto-approve-risk'],
    approvalWorkflow: true
])

Implementation Roadmap:

  1. Week 1: Configure exception rules for critical tests
  2. Week 2: Implement pipeline-specific thresholds
  3. Week 3: Update scoring model parameters
  4. Week 4: Deploy override workflows and monitoring

This balanced approach maintains governance while enabling automated deployments. The key is making risk management context-aware rather than applying blanket restrictions. Your payment processing tests can proceed automatically when passing, while still maintaining audit trails and escalation paths for genuine risk scenarios.

3

You can configure risk exception rules in ELM’s risk management settings. Create exceptions for specific test case categories or tags. This way, critical tests can bypass the threshold check while still being tracked for risk metrics. It’s under Admin > Risk Management > Exception Rules.

4

Don’t forget about the scoring model updates. ELM’s risk scoring should be recalibrated quarterly based on actual deployment outcomes. If your payment processing tests consistently pass despite high risk scores, the model needs to learn from that. You can adjust the weighting factors for historical failure rate vs. current test stability in the risk configuration.

5

We had the same issue. The problem is that the default scoring model in ELM 7.0.1 is too aggressive for automated pipelines. It treats all high-risk tests equally, but in reality, a passing high-risk test should carry less weight than a failing low-risk test. You need to adjust the risk scoring model parameters to account for actual test outcomes, not just historical patterns.