Asset tracking messages lose ordering when published to Pub/Sub topic with high throughput

IoT Google Cloud IoT
, , , , , , , ,
1

Our asset tracking system loses message ordering when location updates are published to Cloud Pub/Sub. GPS coordinates arrive out of sequence, causing incorrect asset position calculations and false geofencing alerts.

We’re tracking 200+ vehicles, each publishing location every 30 seconds. Messages from the same vehicle arrive in random order, sometimes with 5-10 minute old coordinates appearing after newer ones.


Published: {vehicleId: 'V-1234', lat: 40.7128, lng: -74.0060, timestamp: '2025-02-19T13:45:00Z'}
Received:  {vehicleId: 'V-1234', lat: 40.7580, lng: -73.9855, timestamp: '2025-02-19T13:50:00Z'}
Received:  {vehicleId: 'V-1234', lat: 40.7128, lng: -74.0060, timestamp: '2025-02-19T13:45:00Z'}

Our subscriber processes messages as they arrive without checking timestamps. We’ve tried enabling Pub/Sub ordering keys but still seeing out-of-order delivery. How do we guarantee ordered processing for asset location updates?

2

I’d recommend implementing idempotent processing regardless of ordering guarantees. Store the latest processed timestamp per vehicle in your database, and discard any message with an older timestamp than what you’ve already processed. This protects against both out-of-order delivery and duplicate messages. Use message attributes to include sequence numbers for additional validation.

3

For geofencing logic specifically, you need temporal validation. Don’t just check if the current position is inside/outside a fence - check if the trajectory between the last known position and current position crosses a fence boundary. This handles out-of-order messages better because you’re evaluating movement patterns rather than point-in-time location.

4

We added ordering keys using vehicleId but still seeing occasional out-of-order messages. Maybe we’re not acknowledging correctly? Our current subscriber uses parallel processing with 10 worker threads.

5

Even with ordering keys enabled, your subscriber needs to handle messages correctly. The subscription must have ‘enableMessageOrdering’ set to true, and your client must acknowledge messages in order. If you acknowledge message B before message A, you break the ordering guarantee. Process and acknowledge sequentially per ordering key.

6

Here’s the complete solution for maintaining message ordering with asset tracking:

Message Ordering Keys: Configure ordering keys at publish time. Set the ordering key to vehicleId for all location updates from the same vehicle:

from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic)
message_data = json.dumps(location_update).encode('utf-8')
future = publisher.publish(
    topic_path,
    data=message_data,
    ordering_key=vehicle_id
)

Pub/Sub Ordered Delivery: Enable message ordering on your topic:


gcloud pubsub topics update asset-locations --message-ordering

Create subscription with ordering enabled:


gcloud pubsub subscriptions create asset-subscriber \
  --topic=asset-locations \
  --enable-message-ordering

Idempotent Processing: Implement timestamp-based deduplication in your subscriber:

# Pseudocode - Idempotent message processing:
1. Extract vehicleId and timestamp from message
2. Query database for last_processed_timestamp for this vehicleId
3. If message_timestamp <= last_processed_timestamp: skip and acknowledge
4. Process location update (calculate position, check geofences)
5. Update last_processed_timestamp in database
6. Acknowledge message only after database commit succeeds

Geofencing Logic: Implement trajectory-based geofence evaluation:

  • Store previous position and timestamp per vehicle
  • Calculate movement vector between previous and current position
  • Check if vector intersects any geofence boundaries
  • Only trigger alerts for boundary crossings, not point-in-polygon checks
  • Add hysteresis (30-second delay) before triggering exit alerts

Critical Implementation Details:

  1. Single-threaded processing per ordering key: Route messages to workers based on hash(vehicleId) % worker_count. Each worker processes one ordering key at a time.

  2. Sequential acknowledgment: Never acknowledge message N+1 before message N for the same ordering key. Use a pending acknowledgment queue per worker.

  3. Error handling: If processing fails, NACK the message and pause processing for that ordering key until the failed message is redelivered and successfully processed.

  4. Monitoring: Track ordering key distribution across workers to prevent hotspots. Alert if any worker has >30% of total ordering keys.

Subscriber Configuration:

subscriber = pubsub_v1.SubscriberClient()
flow_control = pubsub_v1.types.FlowControl(
    max_messages=100,
    max_bytes=10 * 1024 * 1024
)
subscriber.subscribe(
    subscription_path,
    callback=process_message,
    flow_control=flow_control
)

Performance Optimization:

  • Use message batching for publishing (batch up to 100 messages)
  • Set appropriate ack deadline (60 seconds for location processing)
  • Implement exponential backoff for transient failures
  • Cache geofence polygons in memory to avoid database lookups

After implementing ordered delivery with idempotent processing and trajectory-based geofencing, our false alert rate dropped from 15% to under 0.5%, and position accuracy improved significantly.

7

Ordering keys in Pub/Sub only work if you’re using the same ordering key for all messages from a specific vehicle. Set the ordering key to the vehicleId when publishing. Also, you must enable message ordering on both the topic and subscription. Without both configured, Pub/Sub treats messages as unordered.