The webhook delivery delays you’re experiencing are a known characteristic of Qualio’s audit event processing pipeline in qual-2023.1. Here’s what’s happening and how to address it:
Webhook Queue Monitoring:
Qualio’s webhook system uses a multi-tier queue architecture. Audit events go through an additional validation layer before entering the webhook queue because they need to ensure data consistency across related audit records (findings, observations, CAPAs, etc.). This pre-processing adds 2-5 minutes of latency before the webhook even enters the delivery queue.
You can monitor the actual queue depth by calling:
`GET /api/v1/webhooks/queue-status?event_type=audit
This will show you how many audit webhooks are pending delivery versus already dispatched.
API Rate Limiting:
Here’s the critical piece: webhook deliveries count against your organization’s overall API rate limit, but they use a separate sub-limit specifically for outbound webhook calls. In qual-2023.1, the default webhook delivery rate is capped at 10 requests per minute per event type, regardless of your overall API quota.
During business hours when you’re creating multiple audit findings simultaneously, these webhooks queue up and deliver at the rate-limited pace. This explains why you see 15-20 minute delays during peak times and overnight catch-up.
To verify this is your issue, check the webhook delivery metrics:
GET /api/v1/webhooks/delivery-metrics?hours=24 Look for the throttled_deliveries` count - if this is non-zero, you’re hitting the rate limit.
Dashboard Data Freshness:
For real-time compliance dashboards, I recommend a three-pronged approach:
-
Request Rate Limit Increase: Contact Qualio support to increase your webhook delivery rate limit for audit events from 10/min to 30/min. This is a configuration change they can make at the tenant level.
-
Implement Webhook Batching: Instead of triggering individual webhooks for each audit event, configure Qualio to batch audit events and deliver them in groups every 2 minutes. This reduces the total number of webhook calls while improving overall latency.
-
Supplement with Polling: For critical audit events, implement a lightweight polling mechanism that checks for new audit findings every 60 seconds using:
`GET /api/v1/audits/findings?created_after={last_check_timestamp}
This ensures your dashboard never falls more than 1 minute behind, even if webhooks are delayed.
-
Enable Webhook Priority Override: In your webhook configuration, add the header X-Webhook-Priority: high to your audit event webhook subscriptions. This moves audit webhooks into the high-priority queue, reducing delays to under 2 minutes even during peak usage.
The combination of rate limit increase, batching, and priority override should reduce your webhook delays from 15-20 minutes down to under 2 minutes, which is acceptable for most real-time compliance dashboards. The polling mechanism acts as a safety net for any edge cases where webhooks still experience delays.
Note that qual-2023.2 and qual-2024.1 have improved webhook delivery performance with dynamic rate limiting that scales based on queue depth, so upgrading would also resolve this issue.