Material batch tracking loses sync when IoT devices disconnect

MES AVEVA MES
, , , , , , , ,
1

Our material management module tracks batches using MQTT-connected IoT devices. When devices lose network connectivity (which happens frequently in our plant), batch tracking falls out of sync and we lose critical traceability data.

MQTT configuration snippet:


broker: tcp://mqtt.plant.local:1883
qos: 0
cleanSession: true
keepalive: 60

The devices reconnect automatically, but messages sent during the disconnection period are lost. We’re using AM 2022.1 with default MQTT settings. This creates gaps in our batch genealogy and causes compliance issues during audits. Need advice on preventing message loss during network interruptions.

2

Your QoS level is the problem. QoS 0 means fire-and-forget with no delivery guarantees. Switch to QoS 1 (at least once delivery) or QoS 2 (exactly once) depending on your traceability requirements. Also set cleanSession to false so the broker persists messages during disconnections.

3

For testing, disconnect a device manually and publish test messages to its topic. When you reconnect the device, it should receive all queued messages. Monitor your MQTT broker logs for persistence events. Also check the broker’s admin console - most show pending message counts per client. In AM 2022.1, you can enable MQTT client debug logging to see exactly when messages are received and processed by the material management module.

4

Thanks for the QoS tip. I’ve updated to QoS 1 and cleanSession false. How do I verify that messages are actually being persisted during disconnections? Want to test this before relying on it in production.

5

Beyond QoS settings, you need message persistence configured on your MQTT broker. Most brokers support durable subscriptions that queue messages for offline clients. Check if your broker has persistence enabled and sufficient storage allocated. We use 10GB persistence storage for our material tracking IoT devices and it handles disconnections up to 48 hours without data loss.

6

I’ve implemented robust MQTT-based batch tracking systems and can guide you through the complete solution addressing all critical aspects.

MQTT QoS configuration is foundational. Update your settings:


broker: tcp://mqtt.plant.local:1883
qos: 1
cleanSession: false
keepalive: 60
clientId: mes_batch_tracker_001

QoS 1 ensures at-least-once delivery with acknowledgments. The cleanSession=false flag creates a persistent session - the broker maintains subscriptions and queues messages during disconnections. Use unique, persistent clientId values for each tracking device.

Message persistence requires broker-level configuration. Enable durable message storage with adequate capacity:

  • Configure persistence directory with sufficient disk space (minimum 5GB per 1000 devices)
  • Set message TTL based on your maximum expected disconnection time
  • Enable persistence backup for critical batch tracking data
  • Monitor broker memory usage as persistent queues consume resources

Device reconnection handling needs application-level logic in your material management module. Implement a reconnection protocol:

  1. Device publishes reconnection event with last known message sequence number
  2. Module queries for missed messages using sequence gaps
  3. Reconcile batch state by replaying queued messages in order
  4. Flag any irrecoverable gaps for manual review

Data reconciliation is critical for audit compliance. Build a reconciliation service that:

  • Compares device-reported batch states with MES records after reconnection
  • Identifies discrepancies and generates alerts
  • Maintains an audit log of all reconnection events and data gaps
  • Provides a UI for operators to review and approve reconciled data

Additional recommendations:

  • Implement Last Will and Testament (LWT) messages so the broker notifies the MES when devices disconnect unexpectedly
  • Use retained messages for critical batch state information
  • Add client-side message buffering on IoT devices as a backup to broker persistence
  • Set up monitoring alerts for devices that remain offline beyond threshold periods
  • Test failover scenarios regularly with simulated network interruptions

This comprehensive approach ensures batch traceability integrity even with unreliable network conditions. We’ve maintained 99.9% data completeness across 500+ IoT devices using this architecture.

7

Don’t forget about device reconnection handling on the application side. Even with proper MQTT persistence, your material management module needs to handle reconnection scenarios gracefully. Implement logic that reconciles batch state when devices come back online. We add sequence numbers to messages so we can detect and flag gaps in the data stream.