Web service token expiry interrupts simulation data upload in batch mode

PLM Oracle Agile PLM
, , , , , , , ,
1

Our simulation data upload process uses Agile web services to push large CAE result files. Recently started getting token expiration errors mid-upload for files over 500MB.

The REST API response:


HTTP 401 Unauthorized
{"error": "Token expired", "code": "AUTH_TOKEN_INVALID"}

We authenticate once at the start of the batch job and reuse the token for multiple file uploads. Token lifetime is set to 3600 seconds but uploads fail after about 45 minutes. Each simulation file takes 20-30 minutes to upload due to size.

Is there token refresh logic we should implement? The web service documentation doesn’t mention handling token expiration for large file uploads.

system 2

The core issue is understanding how Agile’s web service token lifetime and token refresh logic work, especially for large file upload handling in batch scenarios. Let me provide the complete solution.

Web Service Token Lifetime Architecture:

Agile web services use JWT-based tokens with two distinct timeout values:

  1. Absolute Lifetime (token.lifetime): Maximum token validity from creation (3600s default)
  2. Idle Timeout (token.idle.timeout): Maximum time between API calls (2700s default)
  3. Active Session Timeout: Separate from token timeouts, tracks actual authentication session

Your tokens expire at 45 minutes (2700s idle timeout) because file upload streams don’t count as API activity for token validation purposes.

Token Refresh Logic Implementation:

The solution requires proactive token management in your batch upload process:

// Pseudocode - Token refresh during batch upload:
1. Authenticate and store token with timestamp
2. Before each file upload, check: (currentTime - tokenTime) > 2400s
3. If near expiration, call /auth/refresh endpoint
4. Update stored token and timestamp
5. Proceed with file upload using fresh token
6. Repeat check for each file in batch

Key implementation details:

  • Refresh at 2400s (40 min) to stay under 2700s idle timeout
  • Store token creation timestamp, not just the token string
  • Handle refresh failures gracefully (re-authenticate if refresh fails)
  • Log all token operations for troubleshooting

Large File Upload Handling - Optimal Strategy:

For 500MB+ simulation files, implement a three-tier approach:

Tier 1 - Chunked Upload with Heartbeat:


Chunk size: 100MB (as you're doing)
Heartbeat: Send keep-alive API call every 20 minutes during chunk upload
Endpoint: /api/session/keepalive (lightweight, resets idle timer)

This keeps the token active even during long chunk transfers.

Tier 2 - Pre-Upload Token Refresh:


Before starting each file:
1. Calculate estimated upload time: fileSize / averageUploadSpeed
2. If (estimatedTime + currentTokenAge) > 2400s, refresh token
3. Start upload with fresh token
4. Track actual upload time for better estimates

Tier 3 - Chunk-Level Token Validation:


For each 100MB chunk:
1. Check token age before chunk upload
2. If > 2000s (33 min), refresh token
3. Upload chunk with current valid token
4. Continue to next chunk

Configuration Changes:

Adjust timeout values for simulation data workloads:

  1. Increase idle timeout (if permitted by security policy):

    
webservice.token.idle.timeout=5400
(90 minutes instead of 45)

  2. Enable automatic token extension for file operations:

    
webservice.file.upload.extend.token=true
webservice.file.upload.reset.idle=true

  3. Configure chunked upload timeout separately:

    
webservice.chunk.upload.timeout=7200
(2 hours for very large files)

These settings go in agile.properties and require app server restart.

Batch Processing Best Practices:

For your 8-10 file batch scenario:

  1. Authenticate once per batch (not per file)
  2. Implement token refresh every 30 minutes regardless of upload progress
  3. Track cumulative batch time and force re-authentication if approaching absolute lifetime
  4. Use parallel uploads with shared token if network bandwidth allows (max 3 concurrent)
  5. Implement exponential backoff for 401 errors (don’t immediately fail the batch)

Error Recovery Strategy:


On 401 Token Expired:
1. Attempt token refresh (may succeed if within absolute lifetime)
2. If refresh fails, re-authenticate with credentials
3. Resume upload from last completed chunk
4. Log token expiration for monitoring
5. Don't restart entire file upload - resume from failure point

Monitoring and Diagnostics:

Add these logging points to your integration:

  • Token creation timestamp
  • Token refresh events with remaining lifetime
  • Upload start/end times per file
  • Chunk upload duration
  • Token expiration events (even if recovered)

This data helps optimize refresh timing and identify network issues vs. token issues.

Alternative Approach - Multipart Upload API:

Agile 9.3.4 includes a multipart upload endpoint specifically for large files:


Endpoint: /api/v2/files/multipart
Features:
- Automatic token refresh during upload
- Built-in chunk management
- Resume capability for failed uploads
- Progress tracking

This API handles token lifecycle internally, eliminating the need for manual refresh logic. Consider migrating to this endpoint for simulation data uploads.

Testing the Solution:

  1. Implement token refresh at 40-minute intervals
  2. Test with a batch of 10 files, each 500MB
  3. Monitor token refresh events in logs
  4. Verify no 401 errors during 3-hour batch run
  5. Confirm all files upload successfully

With proper token refresh logic, your simulation data uploads should complete reliably regardless of batch size or cumulative upload time.

This draft is based on general Oracle Agile PLM knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

Web service tokens expire based on activity, not just time. If the upload takes longer than the idle timeout, the token becomes invalid. Check your web service configuration for ‘token.idle.timeout’ setting - it’s probably much shorter than the 3600 second lifetime.

kavyalead 4

“Tested this on Agile PLM 9.3.6 — adjusting token.lifetime beyond our batch upload window and implementing proactive JWT refresh calls every 2400 seconds eliminated mid-upload session failures.”

6

Found the idle timeout setting - it’s 2700 seconds (45 minutes). That explains why uploads fail at that point. But shouldn’t the upload activity itself count as keeping the token active? The connection is continuously sending data during the upload.

7

File upload streams don’t reset the idle timer because they’re processed by a different component than the authentication handler. The token validation happens at the API gateway level, while file uploads go through the content handler.

You need to implement token refresh in your upload logic. Before starting each file upload, check token age and refresh if it’s close to expiration.

8

For large file uploads, use chunked transfer with periodic token refresh. Split your 500MB files into 100MB chunks, refresh token between chunks. This keeps the token active and provides better error recovery if a chunk fails.

Also consider using the multipart upload API endpoint which handles token refresh automatically for large files.

9

We’re already using chunked uploads (100MB chunks as you suggested) but still hitting token expiry. The problem is the cumulative time for all chunks in a batch exceeds the idle timeout. We process 8-10 simulation files per batch, each taking multiple chunks.

10

Then you need to refresh the token proactively during the batch. Don’t wait for 401 errors. Check token expiration time before each file upload starts and call the refresh endpoint if less than 10 minutes remain. The refresh endpoint returns a new token without requiring re-authentication.