Our application updates device shadow state through the Cloud IoT Device Manager API and we’re experiencing frequent ETag mismatch errors when multiple services try to update the same device simultaneously. This causes device state to become out of sync with actual hardware status.
Typical workflow:
- Mobile app updates desired state (target temperature)
- Analytics service updates reported state (current readings)
- Monitoring service updates metadata (last seen timestamp)
All three services fetch the current device state, modify their respective fields, and attempt to update. Error we see:
ETag mismatch: Expected 'abc123' but current ETag is 'def456'
The device state was modified by another request
We’re using standard REST API calls with the device state endpoint. Each service retrieves the full device object, modifies only its relevant fields, and sends the update with the ETag from the GET request. The ETag handling should prevent conflicts, but we’re getting these errors constantly during peak usage.
How should optimistic concurrency be implemented for device shadow updates? Should we be using a different approach than GET-modify-PUT with ETag validation?
This is classic optimistic locking failure. When you have multiple services updating the same resource, you need retry logic with exponential backoff. Catch the ETag mismatch error, re-fetch the latest state, apply your changes to the new state, and retry the update. Most systems implement 3-5 retry attempts before failing.
Don’t forget about the rate limits on the IoT Core API. If you’re doing multiple GET-PUT cycles with retries, you might be hitting quota limits which makes the problem worse. Check your API usage metrics.
Consider using field masks in your updates instead of full object replacement. The Cloud IoT API supports update masks that let you modify specific fields without touching others. This reduces conflicts because you’re not competing to update the entire device state, just your specific fields. Set the updateMask parameter to only include the fields you’re changing.
The field mask approach sounds promising but I’m not sure how that interacts with ETag validation. If I use an updateMask, does the API still check the ETag? And what if two services try to update different fields at the exact same time - will that still cause conflicts?
ETag validation happens regardless of update masks - it’s checking the overall resource version, not individual fields. For your use case with multiple services, you might want to redesign your architecture. Instead of having three services directly update device state, use a message queue where all updates are serialized through a single device state manager service. This eliminates concurrent writes entirely. Pub/Sub would work well for this pattern.