IoT Hub metrics not updating in Azure Monitor after monitoring agent redeployment

IOT Microsoft Azure IoT
, , , , , , , ,
1

After redeploying the Azure Monitor agent to our monitoring infrastructure, IoT Hub metrics stopped updating in our dashboards. The agent shows as healthy in the portal, but metrics like ‘messages sent’, ‘connected devices’, and ‘routing latency’ are all flatlined since the redeployment three days ago. I suspect the agent resource configuration or IoT Hub metric namespaces changed, but I can’t pinpoint what’s different.


Agent status: Healthy
Data collection rules: Active
Metrics last updated: 3 days ago
Agent logs: No errors

The agent permissions look correct at first glance. Has anyone dealt with Azure Monitor agent breaking IoT Hub metric collection after updates?

2

Check your data collection rules (DCRs). The new agent version might have changed the default metric namespaces or the DCR might not be associated with your IoT Hub resource anymore after the redeployment.

3

Here’s the complete solution addressing all three focus areas that break after Azure Monitor agent redeployment:

Agent Resource Configuration: The redeployment created a new agent instance with a new managed identity, which lost its resource associations. You need to reconfigure the agent:

  1. First, verify the agent’s managed identity has the correct roles:
az role assignment create \
  --assignee <agent-managed-identity-id> \
  --role "Monitoring Reader" \
  --scope /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub}
  1. The agent also needs ‘Log Analytics Contributor’ if you’re sending metrics to a Log Analytics workspace:
az role assignment create \
  --assignee <agent-managed-identity-id> \
  --role "Log Analytics Contributor" \
  --scope /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace}
  1. Recreate the agent configuration file (/etc/opt/microsoft/azuremonitoragent/config.json) to include IoT Hub as a data source:
{
  "dataSources": [
    {
      "type": "platformMetrics",
      "resourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub}",
      "streams": ["Microsoft-InsightsMetrics"]
    }
  ]
}

IoT Hub Metric Namespaces: The metric namespace configuration likely changed during redeployment. IoT Hub uses specific namespaces that must be explicitly configured:

  1. Update your Data Collection Rule (DCR) to include the correct namespaces:
{
  "properties": {
    "dataSources": {
      "platformTelemetry": [
        {
          "streams": ["Microsoft.Insights/metrics"],
          "name": "iotHubPlatformMetrics",
          "resourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub}",
          "metricNamespaces": [
            "Microsoft.Devices/IotHubs"
          ]
        }
      ]
    },
    "destinations": {
      "azureMonitorMetrics": {
        "name": "azureMonitorMetrics-default"
      }
    },
    "dataFlows": [
      {
        "streams": ["Microsoft.Insights/metrics"],
        "destinations": ["azureMonitorMetrics-default"]
      }
    ]
  }
}

The key is specifying ‘Microsoft.Devices/IotHubs’ as the metric namespace. This is different from the generic ‘Microsoft.Insights’ namespace and is often missed during redeployment.

Agent Permissions: Beyond the Monitoring Reader role, the agent needs specific diagnostic settings permissions:

  1. Enable diagnostic settings on IoT Hub if not already enabled:
az monitor diagnostic-settings create \
  --name iot-hub-diagnostics \
  --resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub} \
  --metrics '[{"category":"AllMetrics","enabled":true}]' \
  --workspace /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace}
  1. Grant the agent’s managed identity ‘Monitoring Metrics Publisher’ role (this is often missed):
az role assignment create \
  --assignee <agent-managed-identity-id> \
  --role "Monitoring Metrics Publisher" \
  --scope /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub}

This role is required for the agent to WRITE metrics to Azure Monitor, not just read them.

Reassociate DCR with IoT Hub: The critical step that’s usually missed:

az monitor data-collection rule association create \
  --name iot-hub-dcr-association \
  --rule-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Insights/dataCollectionRules/{dcr-name} \
  --resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Devices/IotHubs/{hub}

Without this association, the DCR doesn’t know to collect metrics from your IoT Hub.

Verification Steps:

  1. Check DCR associations: `az monitor data-collection rule association list --resource
  2. Verify agent is collecting data: Check agent logs at /var/opt/microsoft/azuremonitoragent/log/mdsd.err
  3. Query metrics directly: `az monitor metrics list --resource --metric “d2c.telemetry.ingress.allProtocol” If metrics appear in the direct query but not in dashboards, the issue is with the dashboard configuration, not the agent. If metrics don’t appear in the direct query, the agent isn’t collecting them properly.

After these changes, metrics should start flowing within 5-10 minutes. The three-day gap you’re experiencing is typical when the DCR association is missing - the agent is healthy but has no instruction to collect IoT Hub metrics.

4

Also make sure the DCR is configured to collect platform metrics, not just guest metrics. IoT Hub metrics are platform-level metrics that require a different configuration than VM guest metrics. The DCR needs to specify the correct data sources.

5

Yes, that’s likely the problem. When you recreate a DCR, you need to reassociate it with the target resources. The old DCR-to-IoTHub association was lost during redeployment. You need to create a new data collection rule association linking the new DCR to your IoT Hub.

6

I’ve seen this before. The Azure Monitor agent for IoT Hub metrics requires a specific managed identity configuration. After redeployment, the managed identity might have lost its role assignment to the IoT Hub. Verify the agent’s managed identity has ‘Monitoring Reader’ role on the IoT Hub resource.