OEE dashboard shows data lag when using Kafka Streams for real-time metrics

MES GE Vernova
, , , , , , , ,
1

We’ve implemented Kafka Streams to feed real-time production data into our OEE dashboard in the performance analysis module. The setup works but we’re seeing significant data lag - sometimes 3-5 minutes between an event happening on the shop floor and it appearing in the dashboard.

Our Kafka consumer lag metrics show the consumers are falling behind during peak production hours. We’ve tuned the dashboard refresh interval to 10 seconds, but that doesn’t help if the data isn’t available yet. The thread pool for the Kafka consumer is set to default values and I suspect that might be the bottleneck.

Production managers are complaining they can’t make real-time decisions when the OEE metrics are stale. Has anyone optimized Kafka Streams integration for high-throughput scenarios in GPSF 2023?

2

Consumer lag is usually about throughput vs processing capacity. How many partitions do you have on your production events topic? If you only have a few partitions but high message volume, you’re limiting parallelism. Also check your consumer fetch settings - small fetch sizes mean more network round trips which adds latency.

3

I’ve tuned several high-throughput Kafka integrations for GPSF OEE dashboards. Here’s a comprehensive solution addressing all three key areas.

Kafka Consumer Lag Optimization:

First, increase your consumer thread pool in the GPSF Kafka integration configuration:

kafka.consumer.threads=12
kafka.streams.num.stream.threads=8
kafka.consumer.max.poll.records=500
kafka.consumer.fetch.min.bytes=1048576

With 6 partitions and 12 consumer threads, you’ll have 2 threads per partition which provides good parallelism without over-subscribing. The fetch settings reduce network overhead by batching more records per request.

Dashboard Refresh Interval Configuration:

Your 10-second refresh is actually too aggressive for a lagging system. Reduce dashboard query load while you fix the lag:

oee.dashboard.refresh.interval=30000
oee.dashboard.query.timeout=5000
oee.dashboard.cache.enabled=true
oee.dashboard.cache.ttl=15000

This enables caching with 15-second TTL, reducing database queries while keeping data reasonably fresh. Once lag is resolved, you can decrease the refresh interval.

Thread Pool Tuning:

The critical fix is optimizing your Kafka Streams topology. Modify your stream processor to use non-blocking operations:

// Pseudocode - Optimized stream processing:
1. Configure Kafka Streams with increased parallelism:
   - Set num.stream.threads to match partition count (6-8 threads)
   - Enable RocksDB tuning for state stores
2. Use async processing for OEE calculations:
   - Stream raw events directly to staging topic (no processing)
   - Separate consumer group calculates derived metrics asynchronously
3. Implement windowed aggregations instead of per-message calculations:
   - Use 5-second tumbling windows for availability metrics
   - Materialize window results to queryable state store
4. Configure proper commit intervals:
   - Set commit.interval.ms to 5000 (5 seconds)
   - Balance between data freshness and commit overhead
// See Kafka Streams Performance Guide for complete optimization patterns

The key insight is separating concerns: use Kafka Streams for fast data routing and simple windowed aggregations, then do complex OEE calculations in a separate async process that doesn’t block the main event stream.

Additional Optimizations:

Increase JVM heap for the Kafka Streams application:


-Xms4g -Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=20

Monitor with these metrics:

kafka.metrics.recording.level=INFO
kafka.streams.metrics.enabled=true

After implementing these changes, monitor your consumer lag metrics. You should see lag drop from 3-5 minutes to under 10 seconds within an hour. The combination of increased parallelism, non-blocking processing, and proper caching will handle your 500-800 msg/sec throughput easily.

For the dashboard, users will now see OEE metrics updated within 15-20 seconds of shop floor events, which is acceptable for real-time decision making. If you need sub-10-second latency, consider implementing a WebSocket push mechanism instead of polling refresh, but fix the consumer lag first before optimizing the presentation layer.

4

We have 6 partitions on the production events topic and about 500-800 messages per second during peak hours. The stream processor does calculate some derived metrics like availability percentage before writing to the OEE tables. Maybe that’s too much work in the stream? Should we just pass through raw events and let the dashboard do the calculations?

5

Three to five minute lag suggests you might have a consumer group rebalancing issue or the Kafka Streams application is doing too much processing per message. Are you doing any heavy calculations or database lookups in your stream processor? That would slow down the entire pipeline. Consider moving complex calculations to a separate async process and just use Kafka Streams for data routing and simple aggregations.

6

I’ve seen this pattern before - the dashboard refresh interval is a red herring. Even with 1-second refresh, if your consumer is lagging, you’re just refreshing stale data faster. Focus on the consumer lag first. Check if your Kafka Streams state stores are growing too large - that can cause processing slowdowns. Also verify your commit interval isn’t too aggressive, which can cause excessive overhead.