Your issue requires a multi-layered solution addressing webhook retry logic, idempotency implementation, and event payload structure. Here’s the comprehensive fix:
Understanding the Problem: Creatio 7.18’s webhook system generates timestamps at send-time, not event-time. This is intentional - it helps receiving systems track retry attempts and implement age-based filtering. However, it complicates hash-based idempotency.
Proper Idempotency Implementation:
- Use eventId as Primary Key: Never use payload hash for idempotency with webhooks. The eventId is specifically designed for this purpose and remains constant across all retries:
// Your idempotency check should be:
SELECT * FROM processed_events
WHERE event_id = 'abc123'
If the record exists, skip processing regardless of other payload differences.
- Webhook Retry Logic Configuration: Modify your integration hub webhook settings to optimize retry behavior:
- MaxRetryAttempts: 3 (default is fine)
- InitialRetryDelay: 60 seconds (increase from default 30)
- RetryMultiplier: 2 (exponential backoff)
- MaxRetryDelay: 300 seconds
This gives your endpoint more recovery time between retries.
- Event Payload Structure Standardization: The payload variance you’re seeing (different field sets on retry) indicates timing issues. Configure your webhook trigger properly:
In Integration Hub, for your webhook configuration:
- Trigger Type: “On Record Saved” (not “On Field Changed”)
- Include Delay: 2 seconds (allows transaction to complete)
- Payload Mode: “Complete Record” (ensures all fields included)
This ensures the payload is consistent from the first send.
Advanced Idempotency Pattern:
For robust duplicate prevention, implement this pattern:
// Pseudocode - Two-tier idempotency:
1. Check if eventId exists in processed_events table
2. If not exists, check if similar event processed recently:
- Same entity (Contact)
- Same record ID
- Same eventType (Created/Updated)
- Within last 60 seconds
3. If both checks pass, process event and store eventId
4. If either check fails, log duplicate and skip
This catches both exact duplicates (same eventId) and functional duplicates (rapid-fire events for same record).
Handling Partial vs Complete Payloads:
The issue where retries contain different data suggests webhook firing mid-transaction. Solutions:
- Enable “WaitForTransactionComplete” in webhook config (if available in your version)
- Implement eventual consistency on your side: Accept the first event, then update with data from retries if eventId matches but payload is more complete
- Use webhook signature verification (as mentioned) to detect legitimate retries vs duplicate triggers
Webhook Signature Verification:
Creatio includes an X-Webhook-Signature header. Use it:
// Signature verification
expectedSignature = HMAC_SHA256(payload, webhookSecret)
if (receivedSignature == expectedSignature) {
// Valid webhook from Creatio
}
The signature is calculated on the canonical payload (excluding timestamp), so it remains constant across retries. You can use this for idempotency instead of eventId if you prefer.
Monitoring and Debugging:
Add these checks to identify duplicate sources:
- Log eventId, timestamp, and payload hash for every received webhook
- Track retry attempts using the X-Retry-Count header (if present)
- Monitor for eventIds appearing with different payloads - this indicates configuration issues
After implementing these changes, your system should handle retries gracefully without duplicate processing, even when payloads vary slightly due to timing.