CloudWatch Logs Insights API batch query limits and performance tuning for high-volume log analytics

Cloud Platforms Amazon Web Services (AWS)
, , , , , , , ,
1

We’re building an internal analytics dashboard that needs to query CloudWatch Logs Insights across 50+ log groups spanning multiple AWS accounts. The challenge is that we’re hitting API rate limits and experiencing significant delays when trying to batch multiple queries together. Each log group contains application logs from different microservices, and we need to aggregate metrics every 5 minutes.

Our current approach starts 50 separate queries using StartQuery API, then polls GetQueryResults for each. This consistently hits the 20 concurrent query limit per region, causing some queries to fail with throttling errors. The entire batch takes 8-12 minutes to complete, which is too slow for our near-real-time dashboard requirements. Has anyone successfully optimized large-scale Logs Insights API usage for analytics pipelines?

2

Consider using StepFunctions to orchestrate your queries with built-in retry and error handling. We created a state machine that chunks queries into batches of 15, waits for each batch to complete, then starts the next batch. The workflow handles throttling automatically and completes our 60 log group queries in about 6 minutes consistently. You can also implement parallel query execution across multiple regions if your logs are distributed.

3

Let me address the three key areas for optimizing your Logs Insights API usage at scale.

Logs Insights API Quotas: The fundamental constraints you’re working with are:

  • 20 concurrent queries per region per account (default)
  • 30 StartQuery API calls per second
  • Query timeout of 15 minutes for queries that scan large amounts of data
  • Results limited to 10,000 records per query (use pagination for more)

Request a quota increase to 50 concurrent queries through AWS Service Quotas console. This is typically approved within 24-48 hours for production accounts with valid use cases. For your 50 log groups, this means you can run all queries in parallel with some headroom.

However, there’s a more efficient approach: use the log group name filter in a single query. Instead of 50 separate queries, you can query multiple log groups in one Insights query by specifying them in the log group list parameter. Group your log groups into batches of 10-15 and run 4-5 parallel queries instead of 50. This reduces API calls and improves overall performance.

Batch Query Optimization: Implement a tiered query strategy based on data freshness requirements:

  1. Hot tier (last 1 hour): Query every 5 minutes with small time windows
  2. Warm tier (1-24 hours): Query every 15 minutes, cache results for 10 minutes
  3. Cold tier (older than 24 hours): Query on-demand only, cache aggressively

For your 5-minute refresh requirement, only query the hot tier in real-time. Pre-aggregate warm and cold tier data in DynamoDB or S3 to avoid repeated expensive queries. Use CloudWatch Logs Insights’ time-based filtering to minimize data scanned:

fields @timestamp, @message | filter @timestamp > ago(5m) | stats count() by service

This limits scan volume and improves query performance from 8-12 minutes to 2-3 minutes for your batch.

Pagination and Retry Logic: Implement robust error handling with these patterns:

For throttling (TooManyRequestsException):

  • Exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s (max)
  • Add jitter: random(0, min(cap, base * 2^attempt))
  • Queue failed queries for retry rather than failing fast

For pagination when results exceed 10,000 records:

  • Always check the ‘status’ field in GetQueryResults response
  • If status=‘Complete’ and ‘nextToken’ is present, make another GetQueryResults call with the token
  • Aggregate results client-side across all pages
  • Set a maximum page limit (e.g., 5 pages = 50,000 records) to prevent runaway queries

For concurrent query management:

  • Use a semaphore or queue to limit concurrent StartQuery calls to your quota (20 or 50)
  • Track query IDs in a priority queue, with newer time ranges prioritized
  • Poll GetQueryResults with 2-second intervals (don’t poll too aggressively)
  • Implement timeout handling - if a query exceeds 5 minutes, cancel it and retry with a smaller time range

Implementation pattern:

  1. Chunk your 50 log groups into 5 batches of 10
  2. For each batch, start a single Insights query targeting all 10 log groups
  3. Poll results with 2-second intervals, implementing pagination if needed
  4. Move to next batch only after previous batch completes or times out
  5. Cache results in ElastiCache with 3-minute TTL to serve dashboard requests

This approach should reduce your total execution time from 8-12 minutes to under 4 minutes while staying within API limits. The key is reducing the number of individual queries through log group batching and implementing intelligent caching to avoid redundant API calls for dashboard refreshes.

4

Have you looked at consolidating your log groups? Instead of querying 50 separate log groups, you could centralize logs into fewer groups using subscription filters that forward to a central location. This reduces the number of API calls needed. We went from 40 log groups to 5 by organizing logs by service tier rather than individual microservice, and it dramatically improved query performance.

5

We had similar issues. The 20 concurrent query limit is a hard quota, but you can request an increase through AWS Support. We got it raised to 50 for our production account. Also consider using the FilterLogEvents API for simple pattern matching instead of Insights queries - it has higher rate limits and is faster for basic filtering.

6

For pagination, make sure you’re using the nextToken properly when results exceed 10,000 records. We implemented exponential backoff with jitter for retries on throttling errors - start with 1 second delay, double it each retry up to 32 seconds max. This smooths out the API calls and prevents retry storms. Also batch your StartQuery calls with a delay between each batch to stay under the concurrency limit.