Device offline status not updating in gateway management after Pub/Sub migration

IoT Google Cloud IoT
, , , , , , , ,
1

After migrating Pub/Sub topics, device offline/online status isn’t updating in our gateway management dashboard. This is breaking our monitoring and alerting.

We recently moved our IoT telemetry from legacy Pub/Sub topics to new topic structure with better naming conventions. The migration went smoothly for data flow, but now our gateway management system can’t detect when devices go offline.

The device state events should be published to a specific Pub/Sub topic that our monitoring service subscribes to. Here’s what we see in the subscription:


{
  "deviceId": "gateway-floor-3",
  "eventType": "disconnect",
  "timestamp": "2025-01-19T14:15:00Z"
}
// But dashboard still shows "online"

The events are reaching Pub/Sub correctly, but our monitoring service isn’t processing them. We suspect it’s either IAM permissions on the new topics or the subscription configuration. Device state event subscriptions worked perfectly before the migration. Anyone dealt with this after a topic migration?

2

I’ve seen this exact issue. The problem is usually that the new subscription has a different delivery method. Were your old subscriptions push-based and you created new ones as pull? Or vice versa? Your monitoring service might be configured to receive push notifications at a specific endpoint, but the new subscription is pull-based so nothing’s being delivered.

3

This is almost certainly an IAM permissions issue. IoT Core publishes device state events to Pub/Sub using a service account. After migrating topics, you need to grant that service account publish permissions on the new topic. Check if the IoT Core service account (usually cloud-iot@system.gserviceaccount.com) has pubsub.publisher role on your new topics.

4

Did you recreate the subscriptions or just update the topic references? When you migrate topics, you need to update the subscription’s topic binding. Also verify that your monitoring service’s service account has the pubsub.subscriber role on the new topic.

5

You’ve hit the classic post-migration configuration gap. All three focus areas need attention:

Pub/Sub Topic Migration Checklist: When migrating Pub/Sub topics, you need to update multiple configurations:

  1. IoT Core device registry must point to the new state topic:

gcloud iot registries update REGISTRY_NAME \
  --region=REGION \
  --state-pubsub-topic=projects/PROJECT/topics/NEW_TOPIC

This is the most commonly missed step. Even though events are appearing in your subscription, they might be coming from a different source than expected. Verify your registry configuration with gcloud iot registries describe.

Device State Event Subscriptions: Your new subscription needs these exact settings to match the old behavior:

  • Delivery type: must match your monitoring service (push vs pull)
  • Message retention: set to at least 7 days for reliability
  • Ack deadline: 60 seconds minimum for state processing
  • Dead letter topic: configure to catch processing failures
  • Message ordering: enable if you need sequential state updates

Critically, check if your old subscription had a message filter. Device state events include connects, disconnects, errors, and config updates. If your monitoring service only processes connect/disconnect events, add this filter:


attributes.subFolder="state" AND (attributes.deviceId!="")

IAM Permissions Verification: You need three separate IAM bindings:

  1. IoT Core service account → Publisher on state topic:

gcloud pubsub topics add-iam-policy-binding NEW_TOPIC \
  --member=serviceAccount:cloud-iot@system.gserviceaccount.com \
  --role=roles/pubsub.publisher
  1. Monitoring service account → Subscriber on subscription:

gcloud pubsub subscriptions add-iam-policy-binding SUBSCRIPTION \
  --member=serviceAccount:monitoring-service@PROJECT.iam.gserviceaccount.com \
  --role=roles/pubsub.subscriber
  1. Monitoring service account → Viewer on subscription (for monitoring):

gcloud pubsub subscriptions add-iam-policy-binding SUBSCRIPTION \
  --member=serviceAccount:monitoring-service@PROJECT.iam.gserviceaccount.com \
  --role=roles/pubsub.viewer

The third binding is often forgotten but necessary for the monitoring service to check subscription metrics.

Debugging steps:

  1. Enable Cloud Logging for your subscription to see delivery attempts
  2. Check if messages are being delivered but not acked (indicates processing failure)
  3. Verify your monitoring service logs for permission errors
  4. Test with a simple pull subscription to confirm messages are reachable

One gotcha we discovered: if you’re using push subscriptions with authentication, the endpoint URL must be updated in the subscription configuration. The old subscription might have had a different push endpoint than what your monitoring service is now listening on.

After fixing all three areas, you should see state updates within 30 seconds of device events. Set up monitoring on subscription/num_undelivered_messages to alert if state events aren’t being processed.

6

Rachel, I checked and the IoT Core service account does have publisher permissions. Kevin, we created new subscriptions pointing to the new topics and deleted the old ones. The subscription configuration looks identical to the old setup. Could there be something else in the subscription settings that matters?

7

Check your monitoring service’s subscription filter. If you had a message filter configured on the old subscription and forgot to add it to the new one, your service might be receiving all events instead of just state changes. This could overwhelm the service or cause it to ignore messages that don’t match expected format. Also verify the ack deadline - if it’s too short, messages might be redelivered before processing completes.