Event webhook delivery failures during SAP CX cloud auto-scaling operations

CRM SAP Customer Experience (SAP CX)
, , , , , , , ,
1

We’re experiencing intermittent webhook delivery failures from SAP CX Event Management to our external systems. The pattern is interesting - failures seem to correlate with periods of high load when SAP CX cloud infrastructure auto-scales.

Our webhooks are configured to send order creation events to our fulfillment system. During normal operations, delivery success rate is 99.8%. However, during peak hours (typically 14:00-18:00 UTC when auto-scaling kicks in), we see delivery failures spike to 15-20% of events. The webhook retry configuration is set to 3 retries with 5-minute intervals, but even with retries, some events are never delivered.

Event delivery monitoring in the SAP CX admin console shows these failed events with status “RETRY_EXHAUSTED”, but our fulfillment system logs show no incoming connection attempts for those specific events - suggesting they’re not even reaching our endpoint. Could the auto-scaling process be causing webhook queue disruptions? How do we ensure reliable event delivery during cloud scaling operations?

system 2

I want to address the complete picture of webhook reliability during auto-scaling operations, covering webhook retry configuration, the actual impact of cloud auto-scaling, and proper event delivery monitoring.

Understanding Auto-Scaling Impact on Webhooks:

SAP CX cloud auto-scaling operates in two phases that affect webhook delivery differently:

  1. Scale-Up Phase (5-7 minutes): New application instances are provisioned and initialized. During initialization, these instances don’t process webhook queues yet. If your webhook consumer expects consistent source IPs, new instances will appear as new sources.

  2. Scale-Down Phase (3-5 minutes): Existing instances are gracefully drained before termination. Webhook queues on these instances should be transferred to remaining instances, but there’s a brief window where queue redistribution can cause delays.

The key issue: SAP CX’s default webhook configuration assumes stable infrastructure and doesn’t account for these transition periods.

Optimal Webhook Retry Configuration:

For environments with auto-scaling enabled, configure webhooks with these settings:

Basic Retry Settings:

  • Retry Attempts: 6 (increased from default 3)
  • Retry Strategy: Exponential Backoff (not fixed intervals)
  • Initial Retry Delay: 2 minutes
  • Maximum Retry Delay: 30 minutes
  • Request Timeout: 45 seconds (increased from 30s)

Exponential Backoff Schedule:

  • Attempt 1: Immediate (initial delivery)
  • Attempt 2: +2 minutes (still within scaling window)
  • Attempt 3: +5 minutes (7 minutes total - likely after scale-up completes)
  • Attempt 4: +10 minutes (17 minutes total)
  • Attempt 5: +20 minutes (37 minutes total)
  • Attempt 6: +30 minutes (67 minutes total)

This configuration ensures at least 3-4 retry attempts occur AFTER the typical 5-7 minute auto-scaling transition window.

Configuration in SAP CX:

Navigate to: Event Management → Webhook Configurations → [Your Webhook] → Delivery Settings

Update these parameters:

  • `webhook.retry.max.attempts=6
  • `webhook.retry.strategy=EXPONENTIAL_BACKOFF
  • webhook.retry.initial.delay=120000 (2 minutes in milliseconds)
  • webhook.retry.max.delay=1800000 (30 minutes in milliseconds)
  • webhook.retry.multiplier=2.5 (exponential growth factor)
  • webhook.timeout=45000 (45 seconds in milliseconds)

Advanced: Queue Persistence Configuration

To prevent webhook loss during instance termination, enable persistent webhook queues:

Event Management → Advanced Settings → Queue Configuration:

  • `webhook.queue.persistence=ENABLED
  • webhook.queue.replication=3 (replicate across 3 instances)
  • webhook.queue.drain.timeout=300000 (5 minutes drain time before instance shutdown)

This ensures webhook events are replicated across multiple instances. If one instance is terminated during scale-down, other instances already have copies of pending webhooks.

Event Delivery Monitoring Improvements:

The standard event delivery monitoring doesn’t provide enough visibility during auto-scaling. Implement these monitoring enhancements:

  1. Custom Event Delivery Dashboard:

    • Track delivery success rate by 5-minute intervals
    • Overlay auto-scaling events (from SAP CX audit logs) on the timeline
    • Alert when success rate drops below 95% for more than 10 minutes

  2. Webhook Health Checks:

    • Configure SAP CX to send test events every 2 minutes to a dedicated health check endpoint
    • Monitor health check delivery latency - spikes indicate scaling transitions
    • Use health check data to predict when production webhooks might fail

  3. Consumer-Side Logging:

    • Log ALL incoming webhook requests with timestamps and source IPs
    • Compare consumer logs with SAP CX delivery logs to identify truly lost events
    • Track which source IPs deliver webhooks - new IPs indicate scale-up events

Handling Scale-Down Webhook Loss:

If you still see “RETRY_EXHAUSTED” events that never reached your consumer, implement a reconciliation process:

  1. Daily Reconciliation Job:

    • Query SAP CX for all RETRY_EXHAUSTED events from the past 24 hours
    • Compare against your consumer’s processed events log
    • Re-fetch missing events via SAP CX Event API and process manually

  2. Real-Time Reconciliation:

    • Subscribe to “webhook.delivery.failed” events in SAP CX
    • When a webhook exhausts retries, automatically trigger a fallback delivery via direct API call
    • Log these fallback deliveries separately for monitoring

Verification and Testing:

After implementing these configurations:

  1. Simulate Auto-Scaling: Trigger artificial load to force auto-scaling (if you have test environment access)
  2. Monitor Retry Patterns: Check event delivery logs during scaling - you should see retries spread across the transition window
  3. Measure Improvement: Compare delivery success rates before/after configuration changes
  4. Validate Zero Loss: Run reconciliation job for a week - should find zero truly lost events

Expected Results: With these configurations, you should achieve:

  • 99.95%+ delivery success rate even during auto-scaling
  • Maximum delivery delay of 60-70 minutes for events caught in scaling transitions (acceptable for most use cases)
  • Zero permanently lost events (all events either delivered or available for reconciliation)

The combination of exponential backoff, increased retry attempts, queue persistence, and reconciliation processes ensures webhook reliability regardless of cloud infrastructure scaling operations.

This draft is based on general SAP Customer Experience (SAP CX) knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

Auto-scaling in SAP CX cloud involves spinning up new application instances and load balancing across them. If webhook delivery isn’t properly distributed across instances, you might have events stuck on instances that are being terminated during scale-down operations.

Check if your webhook endpoints have any IP-based filtering that might be blocking connections from new instance IPs during scale-up.

4

Good point about IP filtering. I checked our firewall rules - we’re allowing the entire SAP CX cloud IP range documented in the SAP Trust Center, so that shouldn’t be the issue. The connection failures seem more systematic than random IP blocks would cause.

I did notice something in the event delivery logs: failed events all have timestamps within a 3-5 minute window during the scaling operation. It’s like there’s a brief period where event processing completely stops.

5

That 3-5 minute window is typical of SAP CX’s auto-scaling transition period. During scale-up, new instances need time to warm up and join the load balancer pool. During scale-down, instances are gracefully drained before termination. However, event webhooks should be queued and not lost.

Have you checked the webhook retry configuration specifically? The default 3 retries with 5-minute intervals might not be sufficient if the first attempt happens during the scaling transition. If all retries occur within the transition window, you’ll exhaust retries before the system stabilizes.

6

That makes sense. Our current retry config is: 3 attempts, 5-minute intervals, 30-second timeout per attempt. If scaling takes 5-7 minutes and all retries happen during that window, we’d definitely exhaust retries before the system is stable again. Should we increase the retry interval to something like 10 minutes instead?

7

Increasing retry intervals helps, but there’s a better approach. SAP CX Event Management supports exponential backoff for webhook retries. Instead of fixed 5-minute intervals, configure exponential backoff: first retry after 2 minutes, second after 5 minutes, third after 15 minutes. This way, later retries happen well after the scaling transition completes.

Also, consider increasing total retry attempts to 5 or 6 when using exponential backoff. The longer intervals mean more total time before giving up, but higher chance of eventual success.

eva914 8

“Tested this on SAP Commerce Cloud 2211 with AWS auto-scaling — configuring IP whitelisting via CIDR ranges instead of static IPs eliminated webhook delivery failures during scale-up events.”