Insight Advisor ML models not updating after data reload in analytics app with custom script

BA/BI Qlik Sense
, , , , , , , ,
1

I’m experiencing an issue where Insight Advisor’s ML models aren’t refreshing after scheduled data reloads in our analytics app. We’re using Qlik Sense 2019 with custom data connectors pulling from our retail database. The reload task completes successfully in QMC, and I can see the updated data in sheets, but when users interact with Insight Advisor, they’re getting recommendations based on the old dataset from two weeks ago.

I’ve checked the reload task configuration in QMC and the Engine API logs, but I’m not seeing any obvious errors related to ML refresh triggers. The models seem to be cached somewhere and not regenerating. Has anyone encountered similar issues with Insight Advisor ML model updates? I’m wondering if there’s a specific API call or QMC setting needed to force ML model regeneration after data reloads.

2

I’ve seen this before. The ML models in Insight Advisor don’t automatically regenerate on every reload by default. You need to check if your reload task is triggering the cognitive engine refresh. In QMC, go to your reload task settings and look for the advanced options. There should be a checkbox for ‘Refresh cognitive engine after reload’ that might be disabled.

3

Thanks Sarah! I checked the QMC settings but I don’t see that specific checkbox in our 2019 version. Is this a feature that was added in later releases? Our reload tasks show all the standard options but nothing explicitly mentioning cognitive engine or ML model refresh.

4

Let me provide a comprehensive solution addressing all aspects of your Insight Advisor ML refresh issue:

1. Insight Advisor ML Refresh Triggers Configuration: The timeout errors you’re seeing indicate the cognitive engine can’t complete model generation within the default time window. For Qlik Sense 2019, you need to modify the Engine configuration file (typically located at C:\Program Files\Qlik\Sense\Engine\settings.ini). Add or update these settings:


[CognitiveEngine]
ModelGenerationTimeout=900
MaxConcurrentModelGenerations=2

This extends the timeout to 15 minutes and limits concurrent generations to prevent resource exhaustion. After changing these settings, restart the Qlik Sense Engine Service.

2. Qlik Engine API Usage for Forced Refresh: Since automatic triggers aren’t working reliably with your dataset size, implement a post-reload API call to force ML model regeneration. You can use the Engine JSON API with a script that runs after your reload task:

// Connect to Engine API
const enigma = require('enigma.js');
// Open app and trigger cognitive refresh
await app.clearAllCognitiveModels();
await app.generateCognitiveModels();

This ensures models regenerate after each successful data reload, giving you explicit control over the timing.

3. Reload Task Configuration in QMC: In QMC, modify your reload task to include a post-reload trigger. Create a custom batch script or PowerShell script that:

  • Verifies reload completion status
  • Calls the Engine API refresh endpoint
  • Logs the ML model regeneration status
  • Sends notification if regeneration fails

Set this as an external program task that runs immediately after your main reload task completes.

Additional Optimization Tips:

  • Consider implementing incremental loads instead of full reloads to reduce processing time
  • Use QVD files for intermediate data storage to speed up reload operations
  • Monitor the cognitive engine memory usage during model generation - 15 million rows may require increasing the Engine memory allocation
  • If possible, create a separate analytics app with aggregated data specifically for Insight Advisor to reduce model generation complexity

Verification Steps:

  1. After implementing these changes, check Engine logs for ‘Cognitive model generation completed successfully’ messages
  2. Test Insight Advisor recommendations immediately after a reload to confirm they reflect current data
  3. Monitor the ModelGenerationTimeout metric to ensure 900 seconds is sufficient (increase if needed)

This approach addresses all three focus areas: proper trigger configuration, API-based control, and QMC task orchestration. The combination ensures reliable ML model updates even with large datasets in Qlik Sense 2019.

5

Adding to Lisa’s point - you can also use the Engine API to programmatically trigger ML model regeneration. After your reload completes, you could add a step that calls the cognitive engine refresh endpoint. The API has methods to force model updates without waiting for the automatic triggers. This gives you more control over when models refresh, especially useful for large datasets where you want to schedule regeneration during off-peak hours.

6

Raj, I checked the Engine logs and found some warnings about timeout during model generation. The logs show ‘Cognitive model generation exceeded time limit’ messages around the time of our reloads. Our dataset is pretty large - about 15 million rows across multiple tables. Could this be causing the ML refresh to fail silently?

7

That timeout issue is definitely your problem. With 15 million rows, the default timeout for cognitive model generation is probably too short. You can adjust this in the settings.ini file for the Engine service. Look for the CognitiveEngine section and increase the ModelGenerationTimeout value. I’d recommend starting with 600 seconds (10 minutes) instead of the default 300. Also consider optimizing your data model - sometimes pre-aggregating data or using calculated dimensions can help the ML algorithms process faster.