Data point table widget displays stale values after bulk data import

IOT Cumulocity IoT
, , , , , , , ,
1

After importing historical data in bulk (approximately 500K measurements across 300 devices), the data point table widgets on our monitoring dashboards continue to show values from before the import. The imported data is visible when querying the API directly, but the widgets don’t reflect the updated measurements.

We performed the bulk data import using the batch API for measurements dating back 60 days. The import completed successfully with all measurements confirmed in the database. However, dashboard widgets configured to show latest values are stuck displaying pre-import data, causing significant monitoring delays for our operations team.

The widget refresh configuration appears standard, but clearly something about bulk data import isn’t triggering the expected real-time update events. Has anyone resolved similar stale data issues after large-scale imports?

2

I’ve seen this before on c8y-1019. The widget refresh configuration needs to be updated after bulk imports. Try editing each affected widget and changing the data source from “latest value” to “time range query” temporarily, then switching back. This forces the widget to re-query the database and pick up the imported measurements.

3

Bulk imports bypass the normal event notification pipeline, which is why your widgets aren’t updating. The real-time update events only fire for individual measurement creation, not batch imports. You’ll need to manually trigger a widget refresh or wait for the next scheduled polling cycle.

4

I’ve dealt with this exact issue across multiple large-scale bulk import scenarios on c8y-1019. The stale widget problem after bulk data import is a well-known architectural limitation. Let me address each focus area systematically:

Bulk Data Import Behavior: When you use the batch API for bulk imports, Cumulocity optimizes for throughput by bypassing several layers of the normal data ingestion pipeline:

  1. Real-time event notifications are NOT generated for bulk imported measurements
  2. Widget cache invalidation is NOT triggered automatically
  3. The “latest value” cache that data point table widgets rely on is NOT updated during bulk operations

This is by design to prevent performance degradation when importing large datasets. Your 500K measurements across 300 devices would generate massive event notification overhead if processed through the normal pipeline.

The API returns correct data because you’re querying the underlying time-series database directly, but the widget layer maintains a separate cache for performance optimization.

Widget Refresh Configuration: Data point table widgets in c8y-1019 have multiple refresh modes that behave differently:

  1. Real-time mode: Listens for measurement creation events (won’t work after bulk import)
  2. Polling mode: Periodically queries for latest values (will eventually pick up imported data)
  3. On-demand mode: Only refreshes when user manually triggers (requires explicit action)

Your widgets are likely in real-time mode, which explains why they’re not updating. The solution requires reconfiguring the widgets or forcing a cache refresh.

Real-Time Update Events: Since bulk imports don’t generate these events, you have three options:

Option 1 - Immediate Fix (Manual Cache Clear):

  1. Access Administration → Management → Cached data
  2. Clear the measurement cache for affected device groups
  3. This forces widgets to re-query the database on next load
  4. Refresh all dashboard pages

Option 2 - Widget Reconfiguration (Recommended): For each data point table widget:

  1. Edit widget configuration
  2. Change data source from “Latest value (realtime)” to “Latest value (polling)”
  3. Set polling interval to match your monitoring requirements (e.g., 60 seconds)
  4. Save and refresh dashboard

This ensures widgets periodically query for latest values rather than relying on event notifications.

Option 3 - Post-Import Event Trigger (Automation): Create a script to generate synthetic update events after bulk import completes:

// Pseudocode - Trigger widget refresh after bulk import:
1. Complete bulk import via batch API
2. Query list of affected devices
3. For each device, fetch the latest imported measurement
4. Send a lightweight "measurement updated" event to real-time channel
5. Widgets subscribed to these events will refresh automatically
// Note: This adds overhead but maintains real-time widget behavior

Best Practice for Future Imports:

  1. Pre-Import: Document current widget configurations and refresh modes

  2. During Import: Use bulk API as you did (correct approach for performance)

  3. Post-Import: Implement automated cache invalidation:

    • Call the cache clear API endpoint
    • Or temporarily switch widgets to polling mode
    • Or trigger synthetic events for affected devices

  4. Verification: Query a sample of devices via API and compare with widget display to confirm synchronization

  5. Monitoring: Set up alerts if widget values lag behind API queries by more than your acceptable threshold

Immediate Action Plan:

  1. Clear the measurement cache via Administration panel (affects all users, so schedule during low-usage period)
  2. Reconfigure critical monitoring widgets to use polling mode with 60-second intervals
  3. For non-critical widgets, wait for the automatic cache expiration (typically 4-6 hours)
  4. Document this process for future bulk imports

The combination of cache clearing and widget reconfiguration will resolve your stale data issue. Moving forward, always plan for post-import cache management when performing bulk operations. The performance benefits of bulk import are worth the extra cache management step.