Incident webhook integration fails with 502 Bad Gateway when sending large attachments to external ticketing system

QMS ETQ Reliance
, , , , , , , ,
1

Our incident-mgmt module is configured to send webhooks to ServiceNow whenever a new incident is created. This works fine for incidents without attachments, but when users attach investigation photos or PDF reports, the webhook fails with 502 Bad Gateway errors.

Error from ETQ logs:


Webhook POST failed: 502 Bad Gateway
Endpoint: https://company.service-now.com/api/incident
Payload size: 8.7MB
Timeout: 30 seconds

The webhook error handling just retries three times then gives up, leaving incidents unsynced in ServiceNow. Our reverse proxy config might be limiting payload sizes but I’m not certain where the bottleneck is. Anyone experienced similar webhook payload size issues with ETQ integrations?

2

Let me provide a complete solution addressing all three focus areas:

Payload Size Limits: The 502 error is definitely related to payload size constraints at the reverse proxy layer. Here’s a multi-layer approach:

  1. Immediate Fix - Increase Proxy Limits: Update your nginx configuration:
client_max_body_size 25M;
proxy_read_timeout 120s;
proxy_connect_timeout 120s;
proxy_send_timeout 120s;

However, this is just a temporary solution. Sending 8.7MB payloads in webhooks is not a sustainable pattern.

  1. Better Architecture - Separate Attachments: Modify your ETQ webhook to send incident metadata only, then handle attachments asynchronously. Configure the webhook payload to exclude binary attachment data but include attachment metadata (filenames, sizes, ETQ attachment IDs). After the webhook succeeds, trigger a separate integration job that retrieves attachments from ETQ via its API and uploads them to ServiceNow using the attachment upload endpoint.

Reverse Proxy Configuration: Beyond just increasing limits, implement proper error handling and logging at the proxy level. Add custom error pages for 502 responses that provide more diagnostic information. Configure nginx to log the actual payload sizes and timeout events:

log_format webhook '$remote_addr - $request_length bytes - '
                   '$upstream_response_time seconds';
access_log /var/log/nginx/webhook.log webhook;

This helps you identify exactly which requests are hitting limits and how long they’re taking.

Also consider implementing a separate nginx location block specifically for webhook endpoints with higher limits than your general API endpoints. This prevents webhook requirements from forcing you to open up payload limits across your entire infrastructure.

Webhook Error Handling: ETQ’s default retry logic (3 attempts then fail) is too simplistic for this scenario. Implement a more sophisticated error handling strategy:

  1. Configure ETQ Webhook Settings: In ETQ admin console, increase the timeout from 30s to at least 90s for incident webhooks. Adjust retry settings to use exponential backoff - first retry after 30s, second after 2 minutes, third after 5 minutes.

  2. Implement Circuit Breaker Pattern: If webhooks consistently fail, temporarily disable them and queue incidents for batch processing. This prevents overwhelming ServiceNow with retry attempts.

  3. Add Monitoring and Alerting: Set up monitoring that tracks webhook success rates, payload sizes, and response times. Alert when success rate drops below 95% or when average payload size exceeds 5MB (your threshold for needing attachment separation).

  4. Fallback Mechanism: Create a scheduled integration job that runs every hour to check for incidents in ETQ that don’t have corresponding ServiceNow tickets. This catches any webhooks that failed all retries and ensures no incidents are lost.

  5. Detailed Error Logging: Enhance ETQ’s webhook error logging to capture the full response headers and body from failed requests. 502 errors can have different root causes (gateway timeout vs. payload too large vs. backend unavailable) and you need to distinguish between them for proper troubleshooting.

For your specific situation with 8.7MB payloads, I strongly recommend implementing the attachment separation approach rather than just increasing limits. This provides better scalability and reliability. The webhook should create the incident record in ServiceNow and return the ticket ID, then a follow-up API call handles attachment uploads. This way your webhook payloads stay under 100KB and complete in under 5 seconds, making them much more reliable.

3

Don’t forget about ETQ’s own timeout settings for outbound webhooks. Even if your proxy allows large payloads, ETQ might have its own limits. Check the webhook configuration in ETQ admin console - there should be timeout and retry settings you can adjust. We had to increase ours from 30s to 90s for similar scenarios.

4

502 errors typically indicate the reverse proxy or load balancer is timing out or rejecting the request before it reaches ServiceNow. Check your nginx or Apache config for client_max_body_size or similar payload limits. Default is often 1MB which would definitely block your 8.7MB payloads.

5

Yes, separating attachments from the main webhook payload is the recommended pattern. Send the incident metadata first via webhook, get the ServiceNow ticket ID back, then use a separate API call to upload attachments to that ticket. This keeps your webhook payloads small and fast. ETQ’s webhook configuration allows you to customize the payload structure, so you can exclude attachment data and just include attachment metadata like filename and size.

6

Thanks for the suggestions. We do have nginx as a reverse proxy. I’ll check the client_max_body_size setting. But if we can’t increase payload limits easily due to security policies, what’s the best approach? Should we modify the webhook to exclude attachments and sync them separately?