Device shadow synchronization fails after firmware update on edge devices in device shadow module

IOT SAP IoT
, , , , , , , ,
1

After a firmware update, device shadow states are not synchronizing properly between our edge devices and SAP IoT. The devices continue to send telemetry data successfully, but shadow updates fail with connection errors.

MQTT shadow update topic we’re using:


$aws/things/{deviceId}/shadow/update
Payload: {"state":{"reported":{"firmwareVersion":"2.1.5"}}}

The firmware update completes successfully and the devices reboot, but when they attempt to publish shadow updates, we see authentication failures in the device logs. Monitoring is completely disrupted as we can’t see current device states. The strange part is that regular telemetry on standard topics works fine - only shadow topics fail. Has anyone dealt with shadow sync issues post-firmware update?

2

Your shadow synchronization failure is caused by a combination of topic format issues, authentication scope, and firmware publish logic. Let me address each area:

MQTT Shadow Update Topic: SAP IoT 2505 uses a specific shadow topic hierarchy that differs from AWS IoT. The correct format is:


iot/devices/{deviceId}/shadow/update/reported

Note the /reported suffix - this is mandatory for reported state updates. For desired state, use:


iot/devices/{deviceId}/shadow/update/desired

Your firmware is likely publishing to the generic /update endpoint which isn’t valid in SAP IoT’s shadow implementation. Update your firmware’s MQTT publish logic to use the correct topic path.

Also verify the payload structure matches SAP IoT’s schema:

{
  "state": {
    "firmwareVersion": "2.1.5",
    "lastUpdate": "2025-05-12T08:30:00Z"
  }
}

The outer state wrapper should be removed - SAP IoT expects the properties directly in the payload.

Firmware Publish Logic: Your firmware update likely changed how shadow updates are published. Common issues:

  1. QoS Level: Shadow updates require QoS 1 minimum. Verify firmware isn’t using QoS 0:

mqttClient.publish(topic, payload, 1, false); // QoS=1, retain=false
  1. Publish Timing: After firmware reboot, devices must wait for MQTT CONNACK before publishing shadows. Add connection verification:

if (mqttClient.isConnected() && mqttClient.getConnectionState() == CONNECTED) {
  publishShadowUpdate();
}
  1. Retained Messages: Firmware might be setting retain=true on shadow updates, causing shadow service confusion. Shadow updates must have retain=false.

Device Authentication: The authentication failure specifically on shadow topics indicates insufficient permissions. After firmware updates, devices need to re-establish their authentication context:

  1. Force device re-authentication: Disconnect and reconnect MQTT with clean session:

mqttClient.disconnect();
mqttClient.connect(cleanSession=true);

  1. Verify shadow permissions: In IoT Service Cockpit, check device permissions include:

    • `iot.Device.Read
    • `iot.Device.Write
    • iot.DeviceShadow.Write (critical for shadow updates)

  2. Certificate validation: If using X.509 certificates, firmware updates sometimes invalidate the certificate chain. Re-provision device certificates:


POST /iot/core/api/v1/devices/{deviceId}/credentials
{
  "type": "X509Certificate",
  "certificate": "{newCertPEM}"
}

Shadow Service Cache Issue: The shadow service caches device authentication state. After firmware updates, this cache can be stale. Force cache refresh by updating device metadata:


PATCH /iot/core/api/v1/devices/{deviceId}
{
  "customProperties": {
    "firmwareVersion": "2.1.5",
    "lastFirmwareUpdate": "2025-05-12T08:30:00Z"
  }
}

This triggers shadow service to reload device authentication context.

Recommended Firmware Update Procedure:

  1. Before firmware update: Publish shadow state with updating=true flag
  2. Perform firmware update and reboot
  3. On reboot: Wait 5 seconds for network stability
  4. Reconnect MQTT with cleanSession=true
  5. Wait for CONNACK confirmation
  6. Publish shadow update with new firmware version using correct topic format
  7. Verify shadow update success before resuming telemetry

Implementing these changes, especially correcting the shadow topic format and ensuring proper authentication scope, should restore shadow synchronization after firmware updates.

3

I’ve seen this before - firmware updates can change the device’s MQTT client ID generation logic. If the client ID changes, the shadow service doesn’t recognize it as the same device and blocks updates. Verify your firmware is using consistent client IDs across updates. The client ID should match the device ID registered in SAP IoT.

4

You’re right about the topic format - we were using AWS-compatible topics and the firmware update changed that. However, even after correcting the topic to SAP IoT format, we’re still seeing authentication errors specifically on shadow topics. Regular telemetry topics authenticate fine with the same credentials.