Athena queries fail on S3 logs generated by ECS FireLens due to JSON parsing errors

Cloud Platforms Amazon Web Services (AWS)
, , , , , , , ,
1

We configured ECS FireLens to send container logs to S3 for analysis with Athena, but queries are failing with “HIVE_BAD_DATA: Error parsing field value” errors. The logs appear to be in JSON format but Athena can’t parse them correctly.

FireLens configuration in task definition:

"logConfiguration": {
  "logDriver": "awsfirelens",
  "options": {
    "Name": "s3"
  }
}

Our Athena table is defined with JSON SerDe, but queries fail on about 40% of log entries. When we examine the S3 files directly, some log lines have nested JSON that seems to break the parser. We need to analyze application logs for error patterns and performance metrics. Is there a specific FireLens output format we should be using, or do we need to modify our Athena table schema to handle the nested JSON structure?

2

That makes sense - I can see the double-encoding in the S3 files now. How do we configure FireLens to output clean JSON that Athena can parse directly? Do we need to add Fluent Bit filters to the FireLens configuration?

3

Complete Solution for FireLens to Athena JSON Parsing

Your issue stems from FireLens wrapping container logs in metadata, creating nested JSON that Athena’s SerDe can’t parse by default. Here’s the comprehensive solution:

FireLens Log Format Issue:

FireLens outputs logs in this format:

{"log":"{level:ERROR,message:API timeout}","container_id":"abc123","container_name":"/ecs-app","source":"stdout","ecs_cluster":"prod"}

The actual application log is double-encoded inside the log field. Athena fails because it expects flat JSON or properly nested structures, not escaped JSON strings.

Solution 1: Custom Fluent Bit Configuration (Recommended)

Create a Fluent Bit config file that parses and flattens the logs. Upload this to S3:

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json_parser
    Reserve_Data On

[PARSER]
    Name json_parser
    Format json

Update your ECS task definition FireLens configuration:

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "config-file-type": "s3",
    "config-file-value": "arn:aws:s3:::my-bucket/fluent-bit.conf",
    "enable-ecs-log-metadata": "false"
  }
}

Setting enable-ecs-log-metadata to false reduces metadata overhead. The parser filter extracts JSON from the log field and promotes it to top-level fields.

Solution 2: Athena JSON Parsing with Nested Structure

If you can’t modify FireLens config, adjust your Athena table to handle nested JSON:

CREATE EXTERNAL TABLE ecs_logs (
  log STRING,
  container_id STRING,
  container_name STRING,
  source STRING,
  ecs_cluster STRING
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://my-logs-bucket/firelens/';

Then parse the nested JSON in queries:

SELECT
  json_extract_scalar(log, '$.level') as log_level,
  json_extract_scalar(log, '$.message') as message,
  container_name,
  ecs_cluster
FROM ecs_logs
WHERE json_extract_scalar(log, '$.level') = 'ERROR';

S3 Event Notifications and Partitioning:

Your 40% failure rate might also indicate partitioning issues. FireLens typically writes logs with date-based prefixes like year=2025/month=06/day=03/.

Create a partitioned table:

CREATE EXTERNAL TABLE ecs_logs (
  log STRING,
  container_id STRING
)
PARTITIONED BY (year STRING, month STRING, day STRING)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://my-logs-bucket/firelens/';

Use partition projection for automatic partition discovery:

ALTER TABLE ecs_logs SET TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.year.type' = 'integer',
  'projection.year.range' = '2024,2026',
  'projection.month.type' = 'integer',
  'projection.month.range' = '1,12',
  'projection.day.type' = 'integer',
  'projection.day.range' = '1,31',
  'storage.location.template' = 's3://my-logs-bucket/firelens/year=${year}/month=${month}/day=${day}'
);

This eliminates the need for MSCK REPAIR TABLE and makes new logs immediately queryable.

Validation and Troubleshooting:

  1. Test Fluent Bit config locally:
docker run --rm -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
  amazon/aws-for-fluent-bit:latest /fluent-bit/bin/fluent-bit \
  -c /fluent-bit/etc/fluent-bit.conf --dry-run

  1. Check FireLens container logs in CloudWatch Logs group /ecs/ecs-firelens-container for parsing errors

  2. Query S3 directly to verify log format:

aws s3 cp s3://my-logs-bucket/firelens/year=2025/month=06/day=03/logs.json - | head -n 5
  1. Test Athena queries on small date ranges first to isolate parsing issues

Recommended Approach:

Implement Solution 1 (custom Fluent Bit config) for clean, flat JSON that’s easier to query. Add partition projection for automatic partition handling. This combination provides the best query performance and eliminates parsing errors entirely. Monitor FireLens container logs during initial deployment to catch configuration issues early.

4

We tried adding a custom Fluent Bit config but our ECS tasks are failing to start now. The task keeps stopping with exit code 1. Is there a way to validate the Fluent Bit configuration before deploying it to ECS?