After deploying a new ML model version, our device shadows are updating with prediction results, but these fields aren’t appearing in our IoT Analytics dataset. The shadow document shows the predicted values correctly:
{
"state": {
"reported": {
"temperature": 72.5,
"mlPrediction": {
"failureProbability": 0.23,
"modelVersion": "v2.1"
}
}
}
}
Our IoT Analytics pipeline is configured to ingest shadow updates, but the mlPrediction field is missing from query results. The schema seems correct, and older prediction fields from the previous model version are still visible in historical data. We need these predictions for our maintenance dashboard that aggregates failure probabilities across our equipment fleet.
Is there a schema evolution issue with IoT Analytics when new nested fields are added to device shadows?
Thanks. I’m using SELECT * in the dataset query, so that shouldn’t be the issue. Where exactly do I update the field mapping? Is this in the channel configuration or in one of the pipeline activities?
Check your pipeline’s Transform activity if you have one. That’s where field mappings are defined. If you don’t have a Transform activity, the channel might be using an implicit schema that was created when the pipeline was first set up. You may need to recreate the channel or add a Transform activity to explicitly map the nested mlPrediction fields to top-level columns in your datastore.
The root cause is usually the datastore schema being immutable after creation. IoT Analytics datastores don’t support schema evolution for nested objects by default. You have two options: either flatten the structure before ingestion, or create a new datastore with the updated schema and migrate your data. I’d recommend the flattening approach for ongoing flexibility.
I’ve dealt with this exact scenario. The issue is that IoT Analytics uses a fixed schema at ingestion time. When you add new fields to your shadow document, the pipeline doesn’t automatically recognize them unless you’ve configured schema-on-read. You need to update the pipeline activity that processes the shadow updates to include the new mlPrediction field mapping. Also check if your dataset’s SQL query explicitly selects this field or uses SELECT *.
IoT Analytics doesn’t automatically detect new fields in nested JSON structures. You likely need to update your pipeline’s channel or datastore schema. Check if your pipeline activity is configured to handle dynamic schema changes, or if you need to explicitly add the new field to the schema definition.