After bulk registering 800+ devices for a new facility deployment, we’re seeing significant delays in device provisioning events appearing in our data storage analytics. The devices are registered successfully and sending telemetry, but the provisioning event records that we use for compliance auditing are delayed by 4-6 hours in the data lake.
This impacts our analytics and processing pipelines that depend on provisioning timestamps. We’ve checked storage quota and have plenty of capacity remaining. The indexing queue monitoring shows a backlog building up after bulk registration operations. Is there a way to prioritize provisioning event indexing or optimize bulk registration to reduce the impact on storage performance?
The indexing priority setting sounds promising. Where exactly is that configured? I’ve looked through the data storage settings but don’t see an obvious priority configuration. Also, is there a way to monitor indexing queue depth in real-time so we know when it’s safe to do another bulk registration without causing backlogs?
There’s an indexing priority setting in the data storage configuration that you can adjust. By default, all events have equal priority, but you can boost provisioning events to process faster than standard telemetry. This helps during bulk registration scenarios but may slightly slow down real-time telemetry indexing.
Bulk registration generates a massive spike in provisioning events that all hit the indexing queue simultaneously. The storage service processes events in order but has throughput limits. Consider splitting your bulk registration into smaller batches of 50-100 devices with delays between batches to avoid overwhelming the indexing pipeline.
The indexing priority is in the advanced storage configuration, not the main settings. Navigate to Data Services > Storage Configuration > Advanced > Event Processing Priorities. You can set provisioning events to ‘High’ priority there. For monitoring, use the storage API endpoint /api/v0002/storage/indexing/metrics which returns queue depth and processing rates in real-time.