Firmware update job stuck 'In Progress' in monitoring dashboard despite device confirmations

IoT PTC ThingWorx
, , , , , , , ,
1

We’re experiencing a monitoring issue with firmware update jobs in ThingWorx 9.5. When launching bulk firmware updates from the monitoring dashboard (targeting 80 gateway devices), the job status gets stuck at ‘In Progress’ indefinitely, even though the devices are successfully updating and sending completion callbacks. The device callback endpoint appears to be receiving POST requests (we see them in nginx access logs), but the dashboard polling interval seems to never pick up these status changes.

Checking individual device status directly shows ‘Update Complete’ with correct firmware versions, but the aggregate job status in the monitoring dashboard remains at ‘In Progress’ with 0/80 completed. We’ve verified the status event processing queue isn’t backed up. The dashboard auto-refresh is set to 30 seconds. Is there a known issue with job status aggregation in 9.5, or could this be related to how device callbacks are being processed by the platform?

2

Also worth checking the status event processing configuration. Even with correct callbacks, if the event stream processor that handles status updates is misconfigured or the subscription isn’t active, events won’t flow to the dashboard. Navigate to the FirmwareUpdateJobManager thing and verify there’s an active subscription to device status events. The subscription should be enabled and show a recent last processed timestamp.

3

That’s your problem right there. The custom endpoint isn’t connected to the job management system. You need to either modify your custom endpoint to call the job manager’s update service, or reconfigure your devices to use the standard ThingWorx callback endpoint. The jobId is absolutely required - without it, the platform has no way to associate the device callback with the specific bulk update job that initiated it.

4

Let me provide a comprehensive solution addressing all three problem areas:

Device Callback Endpoint: Your custom /api/firmware/callback endpoint needs to bridge to ThingWorx’s job management system. You have two options:

Option 1 - Modify Custom Endpoint (if you need custom processing): Update your custom endpoint service to call the job manager after processing:

// Your custom endpoint service
let jobId = me.GetActiveJobForDevice({deviceId: deviceId});
if (jobId) {
  Things["FirmwareUpdateJobManager"].UpdateDeviceStatus({
    jobId: jobId,
    deviceId: deviceId,
    status: status,
    firmwareVersion: version
  });
}

Option 2 - Use Standard Endpoint (recommended): Reconfigure devices to POST directly to ThingWorx’s built-in endpoint:

`https://your-server/Thingworx/Things/FirmwareUpdateJobManager/Services/UpdateDeviceStatus With this payload format:

{
  "jobId": "FW_UPDATE_20250708_001",
  "deviceId": "GW_001",
  "status": "completed",
  "firmwareVersion": "4.2.1",
  "timestamp": 1720444800000
}

The jobId is critical - it’s provided to devices when they receive the update command. Ensure your update initiation service includes this in the device payload.

Status Event Processing: Verify the event processing pipeline is functioning:

  1. Check subscription status:

    • Navigate to FirmwareUpdateJobManager thing in Composer
    • Open the Subscriptions tab
    • Verify “DeviceStatusUpdateSubscription” is Enabled
    • Check that LastProcessedTime is recent (within last few minutes)

  2. If subscription is disabled or missing:

// Re-enable subscription programmatically
Things["FirmwareUpdateJobManager"].EnableSubscription({
  subscriptionName: "DeviceStatusUpdateSubscription"
});
  1. Verify the event stream isn’t backed up:
    • Check System > Monitoring > Event Queue Metrics
    • If queue depth > 1000, you may have a processing bottleneck
    • Consider increasing event processing threads in platform settings

Dashboard Polling Interval: The 30-second dashboard refresh is fine, but it depends on the underlying data stream being updated. The issue isn’t the polling frequency - it’s that the data source (job manager status) isn’t being updated due to the callback problems above.

However, you can optimize dashboard performance:

  1. Verify the dashboard is bound to the correct data source:

    • Dashboard should subscribe to FirmwareUpdateJobManager.JobStatusStream
    • Refresh type should be “On Data Change” rather than “Polling”
    • This makes updates near-instantaneous when status changes

  2. Check for dashboard binding issues:

    • Open the dashboard in Composer
    • Verify all status widgets are bound to live data properties
    • Test by manually updating a device status and watching for dashboard refresh

Complete Implementation Steps:

  1. First, identify all active jobs stuck in ‘In Progress’:
let stuckJobs = Things["FirmwareUpdateJobManager"].GetJobsByStatus({
  status: "InProgress"
});
  1. For each stuck job, manually sync device statuses:
let devices = Things["FirmwareUpdateJobManager"].GetJobDevices({jobId: jobId});
for each (device in devices) {
  let actualStatus = Things[device.thingName].GetFirmwareVersion();
  // Update job with actual device state
  Things["FirmwareUpdateJobManager"].UpdateDeviceStatus({
    jobId: jobId,
    deviceId: device.deviceId,
    status: actualStatus.matches(targetVersion) ? "completed" : "failed",
    firmwareVersion: actualStatus
  });
}

  1. Implement proper callback handling going forward (use Option 1 or 2 above)

  2. Add monitoring alerts for stuck jobs:

    • Create a scheduled service that checks for jobs in ‘InProgress’ > 2 hours
    • Send alerts when detected
    • Auto-trigger status reconciliation if needed

Testing: After implementing these fixes, test with a small job (5-10 devices):

  1. Launch update job from dashboard
  2. Monitor callback logs to confirm proper endpoint calls
  3. Watch job status update in real-time as devices complete
  4. Verify dashboard reflects changes within 5-10 seconds

The core issue is the disconnect between your custom callback endpoint and ThingWorx’s job management system. Once callbacks properly update the job manager, both the status event processing and dashboard polling will work correctly.

5

Our devices are posting to /api/firmware/callback which we configured as a custom endpoint. Here’s the callback format:

{
  "deviceId": "GW_001",
  "status": "completed",
  "version": "4.2.1"
}

I don’t see jobId in our callback payload. Is that required for the job manager to track progress?

6

I suspect the callback endpoint routing is misconfigured. In ThingWorx 9.5, device firmware update callbacks should POST to /Thingworx/Things/FirmwareUpdateJobManager/Services/UpdateDeviceStatus with the job ID and device ID in the payload. If your nginx logs show requests to a different path, or if the payload format is incorrect, the job manager won’t process them. Can you share what endpoint path your devices are calling?