Cloud Monitoring API not returning expected metrics for custom dashboards

Cloud Platforms Google Cloud Platform (GCP)
, , , , , , , ,
1

Our Cloud Monitoring API queries are returning incomplete metric data for custom dashboards. We’re trying to fetch specific custom metrics, but the API returns empty result sets even though the metrics exist in the console.

The issue seems related to metric type filtering and possibly IAM permissions. We’ve verified our service account has Monitoring Viewer role, but API resource scoping might be the problem.


GET /v3/projects/{project}/timeSeries
filter: metric.type="custom.googleapis.com/app/latency"
Response: { "timeSeries": [] }

The same metrics display correctly in the GCP Console monitoring dashboard. What could cause this discrepancy?

2

Your issue involves three interconnected problems that need systematic resolution. Let me address each area based on your symptoms.

Metric type filtering: Custom metrics require exact filter syntax with proper escaping. Your filter looks correct syntactically, but you need to verify the metric is actually registered. List all metric descriptors first:


GET /v3/projects/{project}/metricDescriptors
filter: metric.type=starts_with("custom.googleapis.com")

This confirms your custom metric exists and shows its exact type string. Common issues: extra spaces, wrong domain prefix, or the metric uses a different project. Once confirmed, your timeSeries query needs aggregation parameters:


filter: metric.type="custom.googleapis.com/app/latency"
interval.endTime: 2024-12-14T16:00:00Z
interval.startTime: 2024-12-14T15:00:00Z
aggregation.alignmentPeriod: 60s
aggregation.perSeriesAligner: ALIGN_MEAN

IAM permissions: Monitoring Viewer alone isn’t sufficient for programmatic access to custom metrics. Your service account needs these specific permissions:

  • monitoring.timeSeries.list (read metric data)
  • monitoring.metricDescriptors.list (discover available metrics)
  • monitoring.metricDescriptors.get (read metric metadata)

Create a custom role or use Monitoring Metric Reader role which includes all three. The console works because it uses your user credentials with broader permissions.

API resource scoping: This is likely your main issue. Custom metrics can have resource labels that must be included in your filter. If your metric has resource.type or resource.labels, you MUST filter on them:


filter: metric.type="custom.googleapis.com/app/latency" AND
        resource.type="gce_instance" AND
        resource.labels.instance_id="your-instance-id"

The console automatically scopes to visible resources, but the API requires explicit scoping. List your metric descriptor to see required resource labels, then include them in every query. Without proper resource scoping, the API returns empty results even when data exists.

Test with a simplified query first - remove all optional parameters and just query the last hour with required resource filters. Once that works, add aggregation and additional filters incrementally.

3

We added explicit time intervals but still getting empty results. The service account definitely has monitoring.timeSeries.list permission. Could this be related to metric descriptor registration?

4

The console uses different default aggregation than the API. When you query via API without specifying aggregation parameters, it returns raw data points. The console automatically aggregates. Try adding aggregation.alignmentPeriod and aggregation.perSeriesAligner to your API request. This caught me off guard multiple times.