We’re experiencing shadow state synchronization failures after pushing firmware v2.8.1 to our fleet of 200+ industrial sensors. The devices connect successfully via MQTT but shadow updates aren’t reflecting in the cloud platform.
The MQTT session seems to maintain persistence across the firmware update, but shadow state schema appears misaligned. Our firmware MQTT client config uses QoS 1 for shadow topics:
mqtt.shadow.topic=$aws/things/{deviceId}/shadow/update
mqtt.qos=1
mqtt.cleanSession=false
Devices show as online in the registry but shadow reported state remains stale. Has anyone encountered similar issues with shadow sync post-firmware updates? We need to understand if this is a client configuration problem or a platform-side schema validation issue.
I’ve seen similar behavior when the shadow document format changed between firmware versions. Check if your new firmware is sending the same JSON structure. Even small differences in property names or nesting can cause silent failures in shadow updates.
This sounds like a schema validation issue on the platform side. After a firmware update, if your device shadow state schema changed even slightly, SAP IoT will reject the updates without sending an error back to the device. You need to check the Thing Modeler definition and compare it with what your new firmware is actually sending. Use the SAP IoT API to retrieve the current shadow schema and validate your payload against it. The cleanSession=false setting should maintain the subscription but won’t help if the payload format is wrong.
Are you seeing any errors in the device logs? When we had shadow sync issues, our MQTT client was publishing to the wrong topic after the firmware update. The client configuration file wasn’t being preserved during the OTA process. Verify that your MQTT client config is actually being applied post-update and not reverting to defaults. Also check the SAP IoT Thing Modeler to ensure the device type schema hasn’t been modified recently - that could explain why updates are being rejected.
I encountered this exact issue last month. The problem is that firmware updates often introduce subtle changes to the telemetry data structure, and if your MQTT client isn’t explicitly handling shadow schema versioning, you’ll get silent failures. Check your firmware MQTT client implementation - make sure it’s reading the device type definition from the platform before publishing shadow updates.
Here’s what you need to do systematically:
1. MQTT Session Persistence Verification:
Your cleanSession=false is correct, but verify the broker is actually maintaining the session. Check SAP IoT connection logs to confirm subscriptions persist across reconnection. If sessions are being cleared, your shadow delta messages won’t reach the device.
2. Device Shadow State Schema Alignment:
This is likely your main issue. Compare your firmware’s shadow payload with the Thing Modeler schema:
// Query current schema via API
GET /iot/core/api/v1/tenant/{tenantId}/things/types/{typeId}
// Compare with your device payload structure
Even minor differences (property type changes, missing required fields, additional properties not in schema) cause silent rejections. The platform validates against the schema strictly.
3. Firmware MQTT Client Config Validation:
Your QoS 1 setting is appropriate, but ensure your firmware is handling PUBACK correctly. Add this to your client config:
mqtt.shadow.update.timeout=5000
mqtt.keepalive=60
mqtt.reconnect.backoff=exponential
Diagnostic Steps:
- Enable debug logging on your MQTT client to capture full publish/subscribe flow
- Use SAP IoT Device Management API to check if shadow updates are being received but rejected
- Validate your JSON payload against the Thing Modeler schema using the schema validation endpoint
- Check if your firmware update process is correctly migrating shadow state from old to new schema format
Common Root Causes:
- Firmware introduced new telemetry properties not defined in Thing Modeler
- Property data types changed (e.g., integer to string)
- Required properties missing in new firmware payload
- Shadow document exceeds size limits (8KB for SAP IoT)
Update your Thing Modeler device type definition to match your new firmware schema, or modify your firmware to maintain backward compatibility with the existing schema. The silent failure behavior is by design - invalid shadow updates are dropped to prevent schema corruption.
Good point about the config persistence. I checked and the MQTT config file is being correctly applied. However, I did find something interesting in the device logs - the shadow update payloads are being sent but there’s no acknowledgment from the platform. The devices are publishing to the correct topic but it seems like the messages are being dropped somewhere in the pipeline.