Real-time vs batch data integration for machine status updates - performance tradeoffs

MES Honeywell MES
, , , , , , , ,
1

I’m interested in hearing experiences with real-time versus batch integration approaches for machine status updates in shop floor control. We’re currently using 5-minute batch updates from our machine interfaces, but production wants real-time visibility.

The tradeoffs I’m weighing: real-time gives immediate status visibility but increases network traffic and database load. Batch processing is more efficient but introduces latency in status updates. Our network reliability varies across facilities - some plants have solid infrastructure, others experience occasional drops.

Has anyone implemented a hybrid approach? I’m thinking real-time for critical status changes (machine down, quality alert) and batch for routine updates (cycle counts, temperature readings). Curious about latency requirements others have successfully met and how you’ve handled network reliability issues in real-time scenarios.

2

We went through this exact evaluation last year. Started with pure real-time and quickly found our network couldn’t handle it during peak production. Switched to a hybrid model similar to what you’re describing. Critical events (machine stops, errors) trigger immediate updates, everything else batches every 2 minutes. Works well and production is happy with the visibility.

3

We use industrial PCs as edge collectors - they’re relatively cheap and reliable. For timestamp handling, the edge collector stamps events when they occur, not when they’re transmitted. MES accepts the original timestamp so historical data stays accurate. The only challenge is if the buffer fills up during extended outages - we have logic to prioritize critical events and aggregate routine data if storage gets tight.

4

I want to add the infrastructure perspective since network reliability was mentioned. We’ve found that hybrid integration strategies work best when you have clear event classification. Here’s what we’ve learned across multiple implementations:

Network Reliability Considerations:

Real-time integration requires consistent network availability - aim for 99.5% uptime minimum. If your plant networks can’t meet this, real-time will cause more problems than it solves. Key factors:

• Wireless networks are problematic for real-time - latency spikes during interference

• Wired networks are more predictable but still need quality of service (QoS) configuration

• Segment your network - don’t mix machine data traffic with office traffic

• Monitor packet loss - anything above 1% will cause issues with real-time protocols

Latency Requirements by Use Case:

Different scenarios have different tolerance:

• Machine downtime alerts: <5 seconds acceptable, real-time justified

• Quality failures: <30 seconds acceptable, near-real-time sufficient

• Production counts: <2 minutes acceptable, batch processing fine

• Temperature/pressure readings: <5 minutes acceptable, batch processing preferred (reduces noise)

• Tool wear indicators: <10 minutes acceptable, batch definitely sufficient

Hybrid Integration Strategy Framework:

The approach Susan and you mentioned works well. Here’s how to structure it:

  1. Event Classification: Define three tiers - Critical (real-time), Important (near-real-time), Routine (batch)

  2. Transport Layer: Use message queuing for real-time events (ensures delivery even if MES is briefly unavailable), REST APIs for batch uploads

  3. Database Impact: Real-time events write to a hot table with short retention (24 hours), batch processes write to main tables. Reduces database contention.

  4. Failover Logic: When real-time connection fails, queue events locally and switch to batch mode automatically. Resume real-time when connection restores.

Performance Optimization:

If you implement hybrid integration:

• Batch your routine updates but vary the batch timing across machines (don’t have 200 machines all sending updates at :00, :05, :10)

• Use delta updates for batch processing - only send values that changed

• Compress batch payloads if sending large datasets

• Implement backpressure handling - if MES is slow to respond, increase batch intervals temporarily

The sweet spot we’ve found is real-time for about 10-15% of events (the critical ones), near-real-time (30-60 second batches) for another 20%, and regular batching (2-5 minutes) for the remaining 65-70%. This balances visibility with system load effectively.

5

Great points all around. Marco, your edge computing approach is interesting. Are you using dedicated hardware for the edge collectors or just running services on existing plant servers? And how do you handle the synchronization when buffered data finally reaches MES - any timestamp conflicts?

6

From an operations perspective, the latency requirements really depend on your production model. In our high-mix low-volume environment, we need real-time because line changeovers happen frequently and we need immediate visibility. But for long-running processes, 5-minute batches would be perfectly adequate. Don’t over-engineer if your production doesn’t require it.