We recently replaced several edge sensors in our manufacturing facility and now facing critical issues with data ingestion to our billing engine. The sensors are online and transmitting, but the billing system shows gaps in usage data causing invoice generation delays.
Device registration in AWS IoT Core appears successful - all new sensors show as connected in the console. However, when I check the MQTT topic subscriptions, I’m seeing intermittent message delivery. Here’s what I’m observing:
aws iot describe-thing --thing-name sensor_edge_42
Status: CONNECTED
LastActivityTime: 2025-03-15T08:45:12Z
The IoT policy assignments look correct, but I suspect there might be an issue with certificate rotation or topic mapping. Device shadow updates are inconsistent - some sensors update every 30 seconds as expected, others haven’t updated in hours despite showing connected status.
This is impacting our monthly billing cycle. Any guidance on troubleshooting the data flow from device shadow to billing engine would be appreciated.
Update: I’ve attached the missing policies and verified the MQTT topics. Still seeing some inconsistency though. The shadow updates are working better now but wondering if there’s a systematic way to validate the entire pipeline from sensor to billing engine.
Also worth checking device shadow sync configuration. If your sensors are updating the shadow but the billing engine is polling instead of using shadow delta events, you might be missing updates during the polling interval. We switched to event-driven shadow updates using IoT Rules Engine to trigger our billing calculations immediately when sensor data changes. Reduced our billing data latency from 5 minutes to near real-time.
The shadow delta approach Marcus mentioned is solid. Another thing - are you using classic shadows or named shadows? If you’re using classic and have multiple data streams per device, you might be overwriting billing data with other telemetry. Named shadows let you separate billing metrics from operational telemetry. Create a dedicated ‘billing_metrics’ named shadow and have your billing engine subscribe only to that. Keeps the data streams isolated and prevents interference.