Work order status update webhooks fail to deliver when transitions occur rapidly

MES Rockwell FactoryTalk MES
, , , , , , , ,
1

Our downstream systems are missing critical work order status updates because webhooks from FT 10.0 fail to deliver during rapid state transitions. When work orders move quickly through multiple states (Released → Started → Paused → Resumed → Completed within 10 minutes), only the first and last webhook notifications reach our external WMS.

Webhook endpoint configured:


POST https://wms.company.com/api/mes-events
Payload: {"workOrderId": "WO-5521", "status": "Started"}
Response: 200 OK (but intermediate events never arrive)

We’re missing the Paused and Resumed transitions completely, causing inventory sync failures in downstream systems. There’s no webhook retry logic when our endpoint is temporarily unavailable, no delivery tracking to confirm receipt, and event ordering isn’t guaranteed. We need proper queue management to buffer events during network issues. How should webhook delivery reliability be configured in FT 10.0?

2

Here’s a comprehensive solution for reliable webhook delivery with proper retry logic, delivery tracking, event ordering, and queue management:

WEBHOOK RETRY LOGIC IMPLEMENTATION: Since FT 10.0 lacks native retry capabilities, implement a webhook relay service:

  1. Relay Service Architecture:

    • Deploy a dedicated webhook relay between FT MES and your WMS
    • Configure FT MES to send webhooks to relay instead of directly to WMS
    • Relay persists events to durable message queue before acknowledging receipt
    • Separate worker processes consume from queue and deliver to WMS with retry logic

  2. Retry Configuration:


// Pseudocode - Webhook delivery with retry logic:
1. Receive webhook from FT MES, respond 200 OK immediately
2. Persist event to message queue (RabbitMQ/Kafka)
3. Worker process attempts delivery to WMS endpoint
4. On failure: retry with exponential backoff (1s, 2s, 4s, 8s, 16s)
5. Maximum 5 retry attempts over 31 seconds
6. On persistent failure: move to dead letter queue for manual review
7. Log all delivery attempts with status codes and error messages

DELIVERY TRACKING SYSTEM: Implement comprehensive tracking for webhook lifecycle:

  1. Tracking Database Schema:

    • Event ID (unique identifier)
    • Work Order ID
    • Event type (status change, transition)
    • Source timestamp (when FT MES generated event)
    • Receipt timestamp (when relay received webhook)
    • Delivery timestamp (when WMS acknowledged)
    • Delivery attempts count
    • Current status (pending/delivered/failed)
    • Response codes from delivery attempts

  2. Tracking API Endpoints:


GET /webhook-relay/events/{eventId}/status
Response: {
  "eventId": "evt-89234",
  "status": "delivered",
  "attempts": 2,
  "deliveryTime": "2025-06-01T09:15:23Z",
  "latency": 3420
}
  1. Monitoring Dashboard:
    • Real-time delivery success rate
    • Average delivery latency
    • Failed delivery alerts
    • Queue depth monitoring
    • Event throughput metrics

EVENT ORDERING GUARANTEES: Ensure events are processed in correct sequence:

  1. Sequence Number Implementation: Modify FT MES webhook payload to include sequence information:
{
  "eventId": "evt-89234",
  "workOrderId": "WO-5521",
  "status": "Paused",
  "timestamp": "2025-06-01T09:15:20.543Z",
  "sequenceNumber": 3,
  "previousStatus": "Started"
}

  1. WMS-Side Ordering Logic:

    • Buffer events per work order in memory
    • Process events in sequence number order
    • Hold out-of-order events for up to 60 seconds
    • If gap persists, query FT MES API for missing events
    • Reject events with sequence numbers older than last processed

  2. Ordering in Message Queue:

    • Use message queue partitioning by work order ID
    • Ensures all events for same work order processed by same consumer
    • Maintains FIFO ordering within each work order’s event stream
    • Configure Kafka topic with key-based partitioning or RabbitMQ with routing keys

QUEUE MANAGEMENT STRATEGY: Implement robust queue management for reliability:

  1. Queue Configuration:

Queue: ftmes-webhooks-primary
Durability: Persistent (survives broker restart)
Ack Mode: Manual (consumer explicitly acknowledges processing)
Prefetch: 10 messages per consumer
TTL: 7 days (events older than 7 days moved to archive)
DLQ: ftmes-webhooks-failed (for permanently failed deliveries)

  1. Consumer Pool Management:

    • Deploy 3-5 consumer workers for redundancy
    • Each consumer processes events independently
    • Automatic consumer failover on worker failure
    • Dynamic scaling based on queue depth (scale up if depth >1000)

  2. Backpressure Handling:

    • Monitor queue depth continuously
    • If depth exceeds 5000 messages: alert operations team
    • If depth exceeds 10000: temporarily pause FT MES webhook generation
    • Implement circuit breaker pattern to prevent queue overflow

CONFIGURATION CHANGES IN FT MES:

  1. Webhook Delivery Thread Pool: Increase thread pool size in ftmes-webhooks.properties:

webhook.delivery.threadPool.size=25
webhook.delivery.timeout=15000
webhook.delivery.queueCapacity=500
  1. Webhook Endpoint Configuration: Point webhooks to relay service:

Webhook URL: http://webhook-relay.internal:8080/ftmes-events
Headers: X-Source: factorytalk-mes, X-API-Key: [relay-api-key]

FAILURE RECOVERY PROCEDURES:

  1. WMS Downtime Handling:

    • Events accumulate in relay queue during WMS downtime
    • Upon WMS recovery, consumers process backlog at controlled rate
    • Implement rate limiting (max 50 events/second) to prevent overwhelming WMS
    • Monitor queue drain rate and adjust consumer count as needed

  2. Relay Service Failure:

    • Deploy relay service in high-availability configuration (3 instances)
    • Use load balancer with health checks
    • If all relay instances fail, FT MES webhooks fail but events remain in FT MES audit log
    • Recovery procedure: query FT MES audit API for missed events and replay to relay

  3. Message Queue Failure:

    • Use message broker clustering (RabbitMQ cluster or Kafka replication)
    • Ensure queue persistence to disk
    • Regular queue backups for disaster recovery

MONITORING AND ALERTING: Set up comprehensive monitoring:

  • Webhook delivery success rate (alert if <95%)
  • Average delivery latency (alert if >10 seconds)
  • Queue depth (alert if >5000 messages)
  • Dead letter queue depth (alert on any messages)
  • Out-of-order event rate (alert if >1%)
  • Consumer health checks (alert on consumer failures)

IMPLEMENTATION ROADMAP:

  1. Phase 1 (Week 1-2): Deploy webhook relay service with basic queue
  2. Phase 2 (Week 3): Implement retry logic and delivery tracking
  3. Phase 3 (Week 4): Add event ordering and sequence numbers
  4. Phase 4 (Week 5): Implement monitoring dashboard and alerts
  5. Phase 5 (Week 6): Load testing and failure scenario validation

ADDITIONAL RECOMMENDATIONS:

  1. Implement idempotency keys in webhook payloads so WMS can safely process duplicate deliveries
  2. Use webhook signatures (HMAC) to verify authenticity of events received by WMS
  3. Create reconciliation job that compares FT MES work order state with WMS state daily
  4. Document runbooks for common failure scenarios (relay down, queue full, WMS unavailable)
  5. Consider implementing webhook replay API for manual recovery of specific missed events

This architecture provides reliable webhook delivery with retry logic, comprehensive delivery tracking, guaranteed event ordering, and robust queue management. The relay service decouples FT MES webhook generation from WMS delivery, ensuring no events are lost even during downstream system failures or rapid work order state transitions.

3

Check your webhook timeout configuration. If set too low, legitimate deliveries fail during network latency spikes. Also verify your WMS endpoint can handle concurrent webhook deliveries - if FT MES fires multiple webhooks simultaneously and your endpoint has request rate limiting, some deliveries will fail. The lack of delivery tracking means you won’t even know which events were dropped unless you implement your own logging on both sides.

4

FT 10.0’s webhook system doesn’t include built-in retry logic or delivery guarantees. When your endpoint returns anything other than 200 OK within the timeout window (default 5 seconds), the webhook is silently dropped. For rapid state transitions, webhooks are fired synchronously during the transaction, so if multiple transitions happen faster than webhooks can be delivered, you’ll see delivery failures. You need to implement an external message queue between FT MES and your WMS.

5

We have webhook timeout set to 10 seconds, and our WMS endpoint can handle 100 requests/second, so that shouldn’t be the bottleneck. The bigger issue is the missing retry logic. When our WMS is briefly down for deployment, we lose all webhook notifications during that window. Is there any way to enable retry attempts in FT 10.0’s webhook configuration?

6

Another consideration - verify your work order state transitions aren’t happening so rapidly that FT MES is queueing multiple webhook deliveries before the first one completes. If transitions occur within milliseconds of each other, the webhook delivery thread pool might be exhausted, causing later webhooks to be dropped. Check the webhook delivery thread pool size in your FT MES server configuration. Default is usually 10 concurrent deliveries, which can be insufficient for high-volume operations.

7

FT 10.0 doesn’t have native webhook retry capabilities. You need to implement a webhook relay service that sits between FT MES and your WMS. The relay receives webhooks from FT MES, persists them to a durable queue (RabbitMQ or Kafka), then delivers to your WMS with retry logic and exponential backoff. This decouples webhook receipt from delivery, ensuring no events are lost even during WMS downtime. The relay also provides delivery tracking and event ordering guarantees.

8

For event ordering guarantees, make sure you’re including sequence numbers or timestamps in your webhook payload. FT MES doesn’t guarantee in-order delivery when multiple webhooks are fired rapidly. Your WMS needs to handle out-of-order events gracefully - either by buffering events and reordering based on timestamp, or by querying FT MES for the current state when an out-of-order event is detected. Without this, you’ll process events in the wrong sequence even when all webhooks do get delivered.