API token expiry causes access denied errors on secure endpoints despite auto-renewal logic

IOT PTC ThingWorx
, , , , , , , ,
1

We have scheduled jobs that call secure API endpoints using application keys for authentication. The jobs run every 4 hours to sync data with external systems, but intermittently fail with “Access Denied” errors even though the app key is valid.

The token renewal logic we implemented should refresh tokens before expiry, but it seems the job scheduling timing doesn’t align with token lifetimes. We also suspect server time sync issues between ThingWorx and the authentication server might be causing premature expiration.

var headers = {
  appKey: applicationKey,
  Accept: 'application/json'
};
var result = Resources['ContentLoaderFunctions'].GetJSON({
  url: secureEndpoint,
  headers: headers
});

Data sync failures are creating gaps in our reporting. How do others handle token expiry in long-running or scheduled API integrations?

2

Look at your NTP configuration on both the ThingWorx server and the authentication server. Clock skew of even a few seconds can cause token validation failures if your security implementation uses timestamp-based validation. Run ntpq -p to check time sync status.

3

I’ve seen this when scheduler jobs overlap. If your 4-hour job takes longer than expected and a new instance starts before the previous one completes, you can get authentication conflicts. Add mutex logic to your scheduler to prevent concurrent executions of the same job.

4

The app key is set to never expire and the user permissions are correct. The errors seem random - sometimes the job succeeds, sometimes it fails with Access Denied. Could this be related to session management or concurrent requests?

5

Another possibility: if you’re using a reverse proxy or API gateway in front of ThingWorx, it might have its own authentication layer that’s timing out independently. Check your proxy logs for authentication failures that correlate with your Access Denied errors.

6

After extensive debugging, here’s the complete solution covering all three focus areas:

Token Renewal Logic: The intermittent failures were caused by stale app key references. Implement proper token validation before each API call:

var appKeyEntity = Things['ApplicationKeyThing'];
var keyInfo = appKeyEntity.GetKeyInfo();

if (keyInfo.expirationDate && keyInfo.expirationDate < new Date()) {
  logger.warn('App key expired, regenerating');
  appKeyEntity.RegenerateKey();
  applicationKey = appKeyEntity.keyId;
}

var headers = {
  appKey: applicationKey,
  Accept: 'application/json'
};

Even if keys are set to never expire, validate them before critical operations. Also implement retry logic with exponential backoff for authentication failures.

Job Scheduling: The root cause was overlapping job executions. Implement a distributed lock mechanism:

var lockName = 'DataSyncJob_Lock';
var lockAcquired = Resources['PersistenceManager'].AcquireLock({
  lockName: lockName,
  timeout: 60
});

if (!lockAcquired) {
  logger.warn('Previous job still running, skipping');
  return;
}

try {
  // Execute sync job
} finally {
  Resources['PersistenceManager'].ReleaseLock({lockName: lockName});
}

This prevents concurrent executions that can cause authentication conflicts. Also adjust your scheduler configuration to use a longer execution timeout and add monitoring for job duration trends.

Server Time Sync: Clock skew was indeed a contributing factor. Here’s how we fixed it:

  1. Synchronized all servers using NTP with the same time source
  2. Verified time zones are configured correctly in platform-settings.json
  3. Added time drift monitoring to our health checks
  4. Implemented timestamp tolerance in our security validation:
var CLOCK_SKEW_TOLERANCE = 300000; // 5 minutes in ms
var serverTime = new Date().getTime();
var tokenTime = token.issuedAt;

if (Math.abs(serverTime - tokenTime) > CLOCK_SKEW_TOLERANCE) {
  logger.error('Clock skew detected: ' + (serverTime - tokenTime) + 'ms');
  // Trigger time sync alert
}

Additionally, we discovered that our load balancer was caching authentication headers incorrectly. Disable caching for endpoints that use app key authentication, and ensure your Load Balancer passes through authentication headers without modification.

Finally, enable detailed security logging in ThingWorx (set SecurityLogger to DEBUG level) to capture the exact reason for Access Denied errors. This helped us identify that some failures were due to permission changes on the user account, not token issues.