Asset tracking integration: Pub/Sub event stream versus direct API polling for real-time location updates

IoT Google Cloud IoT
, , , , , , , ,
1

Designing an asset tracking system that needs to process location updates from 500+ mobile IoT devices. Evaluating two integration patterns:

  1. Event-driven: Device telemetry → Cloud IoT Core → Pub/Sub → Cloud Functions → Processing
  2. Polling-based: Periodic API calls to fetch device states and telemetry

The event-driven approach seems more scalable, but I’m concerned about message ordering and delivery guarantees. Polling is simpler but might not scale well. Looking for insights on latency, scalability, and integration complexity from teams running similar systems in production.

2

At 5000 devices with minute-level polling, you’re making 5000 API calls per minute (83 QPS). That’s well within Cloud IoT Core API limits. However, event-driven scales effortlessly - Pub/Sub handles millions of messages per second. The integration complexity of event-driven is higher initially but pays off at scale. You also get better cost efficiency with events - you only process when devices send data, not on fixed polling intervals.

3

Good point about actual latency requirements. Our use case is tracking high-value equipment in warehouses where real-time visibility helps prevent loss. Sub-minute latency would be ideal. What about scalability - if we grow to 5000 devices, does polling become impractical?

4

Latency isn’t the only consideration. Event-driven systems add complexity - you need to handle message ordering, duplicate detection, and backpressure. Polling is much simpler to implement and reason about. For 500 devices, polling every minute is totally manageable. The question is: do you really need sub-second latency for asset tracking? Most logistics applications are fine with minute-level updates.

5

Event-driven architecture is the clear winner for real-time tracking. With Pub/Sub, you get sub-second latency from device to processing pipeline. Polling introduces artificial delays based on your polling interval - even 30-second polling means you’re always 0-30 seconds behind real-time. For asset tracking where location accuracy matters, event-driven is the only practical choice at scale.

6

The choice between event-driven Pub/Sub and direct API polling for asset tracking fundamentally depends on your latency requirements, scale trajectory, and operational complexity tolerance. Let me break down the three key areas you’ve identified based on production experience with both patterns.

Event-Driven vs Polling Architecture:

Event-driven (Pub/Sub) architecture provides true real-time processing with inherent scalability. When a device publishes location telemetry, it flows through Cloud IoT Core to Pub/Sub within milliseconds, triggering immediate processing. This push-based model means your system reacts to data as it arrives, not on arbitrary polling schedules.

Polling architecture introduces latency equal to your polling interval plus processing time. With 1-minute polling, your average latency is 30 seconds (assuming uniform distribution of device updates). For asset tracking in warehouses where equipment movement patterns matter for theft prevention or process optimization, this delay can be significant.

The complexity trade-off is real but often overstated. Event-driven systems require handling:

  • Message ordering (use Pub/Sub ordering keys if sequence matters)
  • Duplicate detection (implement idempotency in processing logic)
  • Backpressure management (use Pub/Sub flow control settings)

However, polling systems have their own complexity:

  • Pagination logic for large device fleets
  • Efficient change detection (tracking what’s new since last poll)
  • Rate limit management across multiple polling workers
  • Handling partial failures and retry logic

In practice, event-driven complexity is front-loaded during design, while polling complexity accumulates during scaling.

Latency and Scalability Characteristics:

Event-driven latency profile:

  • Device to Cloud IoT Core: 100-300ms (MQTT/HTTP publish)
  • Cloud IoT Core to Pub/Sub: 50-150ms (internal routing)
  • Pub/Sub to Cloud Functions: 50-200ms (trigger invocation)
  • Total end-to-end: 200-650ms typical, sub-second 95th percentile

Polling latency profile:

  • Minimum: Polling interval (e.g., 60 seconds)
  • Average: Half the polling interval (30 seconds)
  • Maximum: Full interval plus processing time (65+ seconds)
  • Cannot achieve sub-second updates regardless of optimization

Scalability comparison at different fleet sizes:

500 devices:

  • Polling: 500 API calls/interval (8.3 QPS for 60s interval) - trivial load
  • Event-driven: Variable based on device activity, typically 5-50 messages/second
  • Winner: Either works; polling is simpler

5,000 devices:

  • Polling: 5,000 API calls/interval (83 QPS) - manageable but requires careful orchestration
  • Event-driven: 50-500 messages/second - Pub/Sub handles effortlessly
  • Winner: Event-driven starts showing advantages

50,000 devices:

  • Polling: 50,000 API calls/interval (833 QPS) - approaching API quota limits, requires distributed polling workers
  • Event-driven: 500-5,000 messages/second - still well within Pub/Sub capacity
  • Winner: Event-driven is clearly superior

Cost efficiency also favors event-driven at scale. Polling incurs API call costs regardless of device activity (fixed cost), while event-driven only processes when devices send data (variable cost proportional to actual usage).

Integration Complexity:

Event-driven integration requires these components:

  • Pub/Sub subscription (push or pull)
  • Cloud Functions or Cloud Run for processing
  • Idempotency logic (Redis/Memorystore for deduplication)
  • Error handling and dead letter queues
  • Monitoring for message lag and delivery failures

Polling integration requires:

  • Scheduled job orchestration (Cloud Scheduler + Cloud Functions)
  • Stateful tracking of last poll timestamp per device
  • Efficient pagination handling for large device lists
  • Change detection logic to identify new/updated telemetry
  • Distributed locking if running multiple polling workers

Integration with downstream systems:

  • Event-driven: Natural fit for streaming analytics, real-time dashboards, immediate alerting
  • Polling: Better for batch processing, periodic reports, systems with eventual consistency tolerance

For your warehouse asset tracking use case specifically, event-driven integration enables:

  • Real-time geofencing alerts (device leaves designated area)
  • Movement pattern detection (unusual activity indicating theft)
  • Instant location updates on tracking dashboards
  • Immediate notifications to security personnel

These capabilities are difficult or impossible to achieve with polling-based integration.

Practical Recommendation:

For your 500-device asset tracking system with plans to scale, implement event-driven architecture using Pub/Sub. The initial complexity investment is justified by:

  • Sub-second latency enabling real-time theft prevention
  • Effortless scaling to 5,000+ devices without architectural changes
  • Cost efficiency (only pay for actual device activity)
  • Better integration with real-time analytics and alerting systems

Implementation roadmap:

  1. Configure Cloud IoT Core to publish device telemetry to Pub/Sub topic
  2. Create Pub/Sub subscription with appropriate ack deadline (30-60 seconds)
  3. Deploy Cloud Functions to process location updates (include idempotency check)
  4. Implement geofencing and alerting logic in the processing function
  5. Export processed data to BigQuery for historical analysis
  6. Build real-time dashboard using Looker or custom web app

Use polling only for supplementary use cases like periodic health checks or configuration synchronization, not for primary location tracking. This hybrid approach, as mentioned by another commenter, leverages the strengths of both patterns.

The event-driven complexity is manageable with proper design patterns, and the latency and scalability benefits are substantial for your real-time asset tracking requirements.