Greengrass edge monitoring metrics not updating in CloudWatch, missing device health data

IoT AWS IoT
, , , , , , , ,
1

Our Greengrass v2 edge devices stopped sending monitoring metrics to CloudWatch three days ago. The devices are operational and processing data normally, but CloudWatch dashboards show no new metrics since April 5th.

Greengrass CloudWatch integration was working fine for two months. Checking the Greengrass logs shows attempts to publish metrics but failures:


CloudWatch metrics publish failed
HTTP 403: Access Denied
Endpoint: monitoring.us-west-2.amazonaws.com

IAM role permissions haven’t changed recently according to CloudTrail. The metric publishing endpoint appears correct. We need visibility back for our edge fleet monitoring. Has anyone experienced CloudWatch integration suddenly breaking without configuration changes?

2

HTTP 403 from CloudWatch typically means IAM permissions issue even if the role itself hasn’t changed. Check if the IAM role’s trust relationship is still valid. Sometimes AWS service updates require trust policy updates. Also verify the Greengrass core device role has cloudwatch:PutMetricData permission.

3

I’d also check if your IAM role has resource-level restrictions. If the policy limits cloudwatch:PutMetricData to specific metric namespaces and Greengrass changed its namespace format in a recent update, that would cause 403 errors. Look at the Resource field in your IAM policy - it might be too restrictive. Greengrass v2 uses AWS/Greengrass namespace by default.

4

Good suggestions. Checked the IAM role trust relationship and it looks correct - it trusts credentials.iot.amazonaws.com as required. The policy does have cloudwatch:PutMetricData but I see it’s restricted to Resource: arn:aws:cloudwatch:::metric/CustomMetrics/*. That could be the issue if Greengrass uses AWS/Greengrass namespace. Would that restriction block Greengrass system metrics?

5

I’ll provide the complete solution addressing all three focus areas:

Greengrass CloudWatch Integration: Greengrass v2 publishes system metrics through the aws.greengrass.Telemetry component. Verify this component is deployed and running:

sudo /greengrass/v2/bin/greengrass-cli component list
# Check aws.greengrass.Telemetry status

The component configuration should specify CloudWatch as the telemetry destination. Check component config:

{
  "telemetryPublisher": "CloudWatch",
  "publishInterval": "300",
  "metricsNamespace": "AWS/Greengrass"
}

IAM Role Permissions: Your IAM role has an incorrect Resource restriction. CloudWatch PutMetricData doesn’t use namespace in the Resource ARN. Update your IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "cloudwatch:PutMetricData",
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "cloudwatch:namespace": [
          "AWS/Greengrass",
          "CustomMetrics"
        ]
      }
    }
  }]
}

The Token Exchange Service (TES) role must also have these permissions. Verify TES role:

aws iot describe-role-alias --role-alias GreengrassCoreTokenExchangeRoleAlias

Ensure the role referenced has the CloudWatch policy attached.

Metric Publishing Endpoint: Verify the Greengrass core device can reach the CloudWatch endpoint. Test connectivity:

curl -I https://monitoring.us-west-2.amazonaws.com

If using VPC endpoints, ensure the endpoint policy allows cloudwatch:PutMetricData:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "cloudwatch:PutMetricData",
    "Resource": "*"
  }]
}

Resolution Steps:

  1. Update IAM policy to remove incorrect Resource restriction
  2. Use Condition block with cloudwatch:namespace instead
  3. Verify TES role has same permissions
  4. Restart Greengrass core to pick up new credentials:
    sudo systemctl restart greengrass



5. Monitor Greengrass logs for successful metric publishing:
   ```bash
   sudo tail -f /greengrass/v2/logs/aws.greengrass.Telemetry.log
  1. Check CloudWatch console after 5-10 minutes for metrics in AWS/Greengrass namespace

The root cause is the Resource restriction in your IAM policy using an invalid ARN format for CloudWatch metrics. Metrics should appear in CloudWatch within 10 minutes of applying the corrected IAM policy and restarting Greengrass.

6

James is right about the ARN format issue. However, instead of using Resource: “*”, you can scope it properly using Condition blocks with cloudwatch:namespace. That maintains security while allowing Greengrass metrics. Check your Greengrass component configuration too - the telemetry component might need the correct CloudWatch endpoint configured for your region.

7

Another thing to verify - make sure the Greengrass Token Exchange Service role has the necessary permissions. That’s separate from the component-level IAM role and handles credential vending for AWS service calls including CloudWatch. The TES role needs both cloudwatch:PutMetricData and logs:CreateLogStream permissions.

8

Yes, that Resource restriction is definitely your problem. CloudWatch metric ARN format doesn’t include the namespace in the Resource field the way you have it configured. The correct format for allowing Greengrass metrics is to use Resource: “*” for cloudwatch:PutMetricData or remove the Resource restriction entirely. CloudWatch uses the namespace parameter in the API call, not in the ARN. Your current policy is likely blocking all metric submissions because the ARN format doesn’t match. This probably broke when someone tightened IAM policies thinking they were improving security.