We recently completed a major performance optimization for our operational dashboard monitoring 50K industrial sensors. Previous implementation queried raw measurement data on-demand, resulting in 45-60 second dashboard load times. We redesigned using Cumulocity’s Streaming Analytics pipeline with pre-computed metric aggregation.
The solution combines Apama EPL streaming rules with Smart Rules Engine for real-time processing. Dashboard now loads in 2-3 seconds with incremental updates every 30 seconds. Our metric caching strategy maintains hourly/daily aggregates in dedicated managed objects, eliminating expensive time-series queries.
Key implementation: Streaming Analytics listens for measurement events, computes rolling aggregates (5min, 1hour, 24hour windows), and updates summary objects. Dashboard queries these pre-computed summaries instead of raw measurements. Happy to share technical details and code patterns.
Great question. We only update current and recent windows (last 24 hours). Historical data older than 24 hours goes through batch recalculation job that runs nightly. The Streaming Analytics handles real-time/near-real-time only. This keeps the streaming pipeline fast and prevents memory bloat from historical reprocessing.
We partition by device groups (500 devices per group). Each Apama monitor handles one group, maintaining in-memory state for rolling windows. This parallelizes processing and limits memory per monitor to ~200MB. Group assignments are static, stored in device managed objects. EPL code uses sliding window aggregates - much more memory-efficient than storing full history.
monitor DeviceGroupAggregator {
dictionary<string, MeasurementWindow> deviceWindows;
action onload() {
on all Measurement(source.id in deviceGroup) as m {
updateRollingAggregate(m);
}
}
}
This sounds exactly like what we need! What’s your Apama EPL rule structure? We’re struggling with memory usage when computing aggregates for 30K+ devices. Do you partition devices into groups or process everything in single stream?