We’re running into a critical issue with our billing engine module when uploading large batches of billing records to Pub/Sub. The process worked fine with smaller datasets (under 5K records), but now we’re hitting ‘Quota Exceeded’ errors when processing monthly billing cycles with 50K+ records.
The batch upload optimization strategy we implemented doesn’t seem to handle Pub/Sub quota management effectively. We’re also not implementing proper exponential backoff handling, so failed batches just fail completely rather than retrying.
Error: 429 RESOURCE_EXHAUSTED: Quota exceeded
at pubsub.publish(batch_data)
Billing cycle delayed by 6+ hours
This is causing delayed billing cycles and impacting our revenue recognition timeline. Has anyone dealt with Pub/Sub quota limits for high-volume billing data ingestion?
Another thing to watch: are you publishing synchronously or asynchronously? Async publishing with futures will give you better throughput and make it easier to implement retry logic. When a publish fails with quota exceeded, catch the exception and use exponential backoff starting at 1 second, doubling up to a max of 32 seconds. This gives the quota time to reset.
We had a similar issue last year. Beyond rate limiting, make sure you’re enabling message batching at the Pub/Sub client level, not just application level. The client library can automatically batch multiple publish calls into single API requests, which is more efficient. Set batch settings like max messages (100-500) and max bytes (1MB) to optimize throughput while staying under quota limits.
Each billing record is about 2-3KB with nested line items. We’re currently batching 1000 records per publish call, which means roughly 2-3MB per batch. We send about 50 batches in rapid succession during the monthly cycle, which probably explains why we’re hitting the quota. The system doesn’t wait between batches at all.