Best practices for integrating multiple data sources into a single ThingWorx mashup dashboard

[poojacoder]

Looking to gather insights on best practices for building ThingWorx 9.6 dashups that pull data from multiple sources - IoT devices, SQL databases, REST APIs, and internal ThingWorx streams. Our current dashboards are slow and difficult to maintain.

We’re pulling data from 5-6 different sources in a single mashup, with each widget calling its own service. This creates dozens of concurrent requests when the dashboard loads. Looking for strategies around data aggregation, optimal mashup refresh intervals, and widget performance tuning.

What approaches have worked well for you when building complex multi-source dashboards? Particularly interested in techniques for reducing load times and managing widget refresh cycles efficiently.

[jim_534]

Good question. We implement timeout handling in the aggregator service. Each data source call has a 3-second timeout. If a source doesn’t respond, we return cached data for that source or null values, but don’t block other sources. The mashup displays available data and shows an indicator for unavailable sources. This partial-loading approach ensures the dashboard remains functional even when some data sources are down.

[sofia_831]

The aggregator service approach sounds promising. How do you handle scenarios where one data source is slow or unavailable? Does it block the entire dashboard or do you have fallback strategies?

[jasoncoder]

After extensive experience building multi-source dashboards, here’s a comprehensive approach addressing all three key areas:

Data Aggregation Strategies: The fundamental principle is to minimize client-side data fetching. Implement a server-side aggregator pattern where a single ThingWorx service orchestrates data retrieval from all sources. This service acts as a facade, abstracting the complexity of multiple data sources from the mashup.

Create a dedicated “DashboardDataAggregator” Thing with services for each dashboard or dashboard section. The service fetches data from all required sources in parallel (using async patterns where possible), performs any necessary data transformation or joining, and returns a unified dataset. Benefits include: reduced network round trips, centralized error handling, easier caching implementation, and simplified mashup logic.

For data joining and transformation, do it server-side in the aggregator service rather than in mashup expressions. ThingWorx services can efficiently join data from different InfoTables, perform calculations, and format data for visualization. This moves processing load from the browser to the server, which scales better.

Implement smart caching within the aggregator. Store aggregated results in a DataTable with timestamp and cache key. Before fetching fresh data, check if cached data exists and is within the acceptable staleness threshold. This is especially effective for data that doesn’t change frequently (configuration data, reference data, historical summaries).

For truly real-time data that must be current, use ThingWorx Streams or subscriptions rather than polling. Widgets can subscribe to property changes and update automatically when data changes, eliminating unnecessary refresh cycles.

Mashup Refresh Intervals: Differentiate between widget types and their refresh requirements. Not all data needs the same update frequency. Categorize widgets:

1. Real-time critical (device status, alarms): 10-15 second refresh or subscription-based
2. Near real-time (production metrics, active counters): 30-60 second refresh
3. Periodic (hourly/daily summaries, trends): 5-minute refresh
4. Static (configuration, reference data): Manual refresh only or on-demand

Implement staggered refresh timing. If you have 10 widgets that need 30-second refresh, don’t refresh all at 0, 30, 60 seconds. Offset them: widget1 at 0s, widget2 at 3s, widget3 at 6s, etc. This distributes server load evenly rather than creating periodic spikes.

Use mashup parameters to control refresh behavior. Add a “refreshMode” parameter with options: auto, manual, on-demand. Let users choose based on their needs. Power users might want manual refresh for better control, while operators prefer auto-refresh for monitoring.

Implement smart refresh logic that pauses when the mashup tab is not active (using browser visibility API). No point refreshing data when users aren’t looking at the dashboard. Resume refresh when tab becomes active again.

Widget Performance Tuning: Optimize individual widget configuration for better performance:

1. Grid widgets: Enable pagination, limit visible rows to 50-100, implement virtual scrolling for large datasets
2. Chart widgets: Limit data points displayed (max 500-1000 points), use aggregation for time-series data, avoid complex chart types (3D, pie charts with many slices)
3. Value displays: Use number formatting to limit decimal places, avoid complex expressions in widget properties
4. Lists: Implement lazy loading, show only top N items with “Show More” button

Minimize widget count. Each widget adds overhead. Instead of 20 small widgets, consider fewer composite widgets that display multiple related metrics. Use tabs or accordions to organize content and load tab content on-demand rather than all at once.

Optimize data binding expressions. Complex expressions in widget properties are evaluated on every refresh, consuming browser resources. Move complex logic to service-side data preparation. Keep mashup expressions simple: basic property access and formatting only.

Use widget visibility expressions wisely. Hidden widgets still execute their services and consume resources. Use mashup states or contained mashups to truly unload widgets when not needed.

Implement dashboard-level loading states. Show a loading overlay while initial data loads, then reveal the dashboard. This provides better UX than showing empty widgets that gradually populate.

Monitor dashboard performance using browser developer tools. Identify slow services, excessive network requests, and memory leaks. ThingWorx monitoring tools can track service execution times and help identify bottlenecks.

With these strategies implemented holistically, complex multi-source dashboards can load in 3-5 seconds and provide responsive user experience even with dozens of widgets and millions of data points. The key is moving intelligence to the server, implementing smart caching, and optimizing refresh patterns based on actual data currency requirements rather than defaulting to aggressive refresh intervals.

[jim_534]

We consolidated data sources using an aggregator service that runs server-side. Instead of each widget calling different services, one service fetches data from all sources, combines it, and returns a unified dataset. Widgets then bind to different parts of this dataset. Reduced our dashboard load time from 25 seconds to 6 seconds.

[javier_master]

Data aggregation is key, but also consider caching strategies. We implemented a caching layer using ThingWorx DataTables that stores aggregated data for 5-10 minutes. The aggregator service updates the cache, and widgets read from cache instead of hitting live data sources every time. For real-time critical data, we use shorter cache intervals (30-60 seconds), but most dashboard widgets don’t need second-by-second updates. This approach dramatically reduced load on our external databases and APIs while keeping dashboards responsive.

[juantechie]

Refresh intervals are often overlooked. Don’t set all widgets to auto-refresh every 5 seconds - it’s usually unnecessary and kills performance. We categorize widgets by update frequency requirements: critical real-time (10-15 sec), near real-time (30-60 sec), periodic (5 min), and static (manual refresh only). Stagger refresh intervals so not all widgets refresh simultaneously. This spreads server load and prevents the dashboard from freezing periodically when everything refreshes at once.

[kimberly246]

From a UX perspective, consider implementing progressive loading. Show critical widgets immediately with cached data, then update with fresh data as it loads. Add loading indicators so users know the dashboard is working. We also implemented a dashboard health indicator showing which data sources are current vs stale vs unavailable. This transparency helps users trust the dashboard data.