Device shadow fails to sync with actual device state in SAP IoT sapiot-23

IoT SAP IoT
, , , , , , , ,
1

Our device shadows are not reflecting the actual state of physical devices. We’re seeing a significant lag (sometimes 15-20 minutes) between actual device state changes and shadow updates. This is breaking our monitoring and control workflows.

The MQTT topic configuration looks correct (devices/+/shadow/update), and devices are publishing state updates every 30 seconds. But the shadow in SAP IoT shows stale data. Checking the shadow document shows last_updated timestamps that don’t match recent device activity.


// Shadow shows:
last_updated: "2024-12-08T10:45:00Z"
state: { "temperature": 22.5, "active": true }

// Actual device (from direct MQTT):
current_time: "2024-12-08T11:15:00Z"
state: { "temperature": 28.3, "active": false }

We have about 450 devices affected. Is there a shadow sync configuration issue or an MQTT topic mapping problem?

2

Check if your shadow update messages are using the correct JSON schema. sapiot-23 is very strict about the shadow update format - if the message doesn’t match the expected schema, it gets silently dropped instead of updating the shadow. Look for schema validation errors in the device shadow service logs.

3

Good point about schema validation. I checked the logs and found some “schema_mismatch” warnings, but not for all devices. About 200 devices show schema errors while the other 250 just aren’t updating. The MQTT messages look properly formatted to me - they include state, metadata, and timestamp fields. Could there be a topic subscription issue on the shadow service side?

4

I’ve resolved this exact scenario multiple times. Your issue has three interconnected components that all need addressing:

Device Shadow Configuration: The shadow service configuration needs tuning for your device count. Update shadow-service-config.json:

{
  "mqtt_subscription_mode": "per_device",
  "max_concurrent_updates": 500,
  "update_throttle_ms": 5000,
  "schema_validation": "strict"
}

Changing from wildcard subscription to per_device mode eliminates the message dropping issue with 450+ devices.

MQTT Topic Structure: Your shadow updates need to follow the exact schema. Here’s the correct format:

{
  "state": {
    "reported": {
      "temperature": 28.3,
      "active": false
    }
  },
  "metadata": {
    "timestamp": 1702037700000
  },
  "version": 1
}

The critical issue: you need “reported” nested under “state”. Many devices send state directly without the reported wrapper, causing schema validation failures.

State Update Synchronization: Enable the shadow sync monitor to track update lag:


POST /shadow/v1/monitoring/enable
{
  "device_group": "all",
  "alert_threshold_seconds": 60,
  "auto_reconcile": true
}

Implementation steps:

  1. Upgrade to sapiot-23.2 immediately - this is critical for your device count
  2. Update shadow service configuration to per_device subscription mode
  3. Fix device message format to include the “reported” wrapper in state updates
  4. Enable shadow sync monitoring with auto-reconciliation
  5. Use the schema validator API during device firmware updates to prevent future format issues

The 15-20 minute lag you’re experiencing is because failed shadow updates (due to schema errors) trigger a retry mechanism with exponential backoff. Once you fix the message format and upgrade to sapiot-23.2, shadow updates should occur within 2-3 seconds of device state changes.

For your 200 devices with schema errors: use the bulk message format correction API to republish their last known state in the correct format, then update device firmware to send properly formatted messages going forward.

5

We’re on sapiot-23.1 so maybe the patch would help. But our devices only send updates every 30 seconds, so we shouldn’t be hitting throttle limits. The schema errors are concerning though - is there a way to validate our message format against the expected schema before devices publish? We need to fix both the schema issues and the general sync lag.

6

The shadow service in sapiot-23 has a known issue with MQTT topic wildcards when you have more than 300 devices. The subscription pattern devices/+/shadow/update can become overwhelmed and start dropping messages. SAP released a patch (sapiot-23.2) that improves the topic subscription handling. Also verify that your shadow update rate isn’t exceeding the configured throttle limits - default is 1 update per 10 seconds per device.