We’re experiencing significant event processing delays in our IoT Production Monitoring deployment. Our SLA requires event processing within 2 seconds, but we’re seeing 8-12 second delays during peak hours (500K+ events/minute).
Current setup has 4 stream partitions with default buffer settings. Queue depth monitoring shows consistent backlog growth, and we suspect deduplication logic is adding overhead. Here’s our current stream config:
{
"partitions": 4,
"bufferSize": 1000,
"batchInterval": 500
}
Has anyone optimized event stream partitioning and batch tuning for high-volume scenarios? Need guidance on horizontal scaling approach and whether increasing partitions alone will help.
One thing people miss is partition key selection. If your keys aren’t well distributed, you get hot partitions that bottleneck everything. Check partition distribution metrics - you want roughly equal message counts across partitions. We use device region hash as partition key rather than device ID to ensure better distribution across geographic clusters.
Your partition count is definitely too low for 500K events/minute. We handle similar volume with 16-24 partitions. The buffer size also needs tuning - 1000 is causing frequent flushes. Start by doubling partitions to 8 and increasing buffer to 5000 events.
Beyond partitioning, your batch interval at 500ms might be creating too many small batches. We found optimal performance at 2000-3000ms intervals with larger batches. This reduces processing overhead significantly. Also check if your deduplication is running on every event - consider moving it to post-aggregation stage where you process fewer unique records. Queue depth monitoring should trigger auto-scaling before backlog grows. What’s your current scaling policy configuration?