Procurement automation fails due to short-lived API tokens expiring mid-batch

SCM Manhattan Active Supply Chain
, , , , , , , ,
1

Our automated procurement scripts are failing with 401 unauthorized errors halfway through processing large purchase order batches. We’re using OAuth2 for API authentication with Manhattan Associates, but the access tokens expire after 30 minutes, which isn’t enough time for our batch jobs that process 500-1000 POs.

The OAuth2 token lifetime seems too short for batch processing resilience. We’re currently requesting a new token at the start of each job, but we’re not using refresh tokens - I wasn’t aware Manhattan’s API supported them until recently.


HTTP 401 Unauthorized
WWW-Authenticate: Bearer error="invalid_token", error_description="Token expired"
Processed: 287/1000 POs
Elapsed time: 32 minutes

This is causing incomplete purchase order creation and requires manual intervention to restart failed batches. How should we properly handle OAuth2 token refresh in long-running automation scenarios?

2

You absolutely need to implement refresh token logic for long-running batch processes. Manhattan’s OAuth2 implementation follows standard patterns - when you request an access token, you also receive a refresh token that’s valid for much longer (typically 24 hours or more). Your scripts should detect 401 errors and automatically request a new access token using the refresh token without restarting the entire batch.

3

That makes sense. So instead of just catching the 401 and failing, I should catch it, use the refresh token to get a new access token, and retry the failed request? Do I need to store the refresh token securely between job runs, or should I request both tokens fresh at the start of each batch?

4

For batch jobs, I recommend requesting fresh tokens at the start of each run rather than persisting refresh tokens between runs. This reduces security risks from token storage. However, during a single batch execution, definitely cache both access and refresh tokens in memory. When you hit a 401, use the refresh token to get a new access token, update your cache, and retry. This pattern provides both security and resilience without requiring secure persistent storage for automation credentials.

5

Thanks everyone. I’ve implemented the refresh token logic and added retry logic with backoff. Testing now with a 500 PO batch. The approach of requesting fresh tokens per batch run makes sense from a security perspective - we’re using client credentials grant anyway, so there’s no user context to maintain across runs.

6

We had the same issue last year. One thing to watch out for: if your batch processing is really slow (like 1000 POs taking over an hour), you might hit the refresh token expiration too. Consider breaking your batch into smaller chunks with separate OAuth sessions for each chunk. We process 200 POs per chunk with a fresh token for each chunk, which keeps us well within the 30-minute access token window.

7

Another consideration: implement exponential backoff when retrying after token refresh. Sometimes the new token takes a few seconds to propagate through Manhattan’s API gateway, so immediate retry might still fail with 401. Wait 2-5 seconds after refreshing before retrying the failed request. Also log all token refresh events so you can monitor how often you’re hitting token expiration and potentially optimize your batch processing speed.

8

Perfect timing to provide a complete solution now that you’ve started implementation. Here’s the comprehensive approach to handling OAuth2 token lifecycle in Manhattan Associates procurement automation:

1. OAuth2 Token Lifetime Understanding

Manhattan’s OAuth2 implementation typically provides:

  • Access Token: 30-60 minutes (yours is 30 minutes)
  • Refresh Token: 24 hours (much longer than access token)
  • Refresh Token Grant: Allows obtaining new access token without re-authenticating

For batch processing, you must leverage refresh tokens to handle expiration gracefully.

2. Recommended Token Management Pattern


// Initial token acquisition at batch start
OAuth2TokenResponse tokens = requestTokens(clientId, clientSecret);
String accessToken = tokens.getAccessToken();
String refreshToken = tokens.getRefreshToken();
long expiresIn = tokens.getExpiresIn(); // seconds until expiration

3. Proactive Token Refresh Strategy

Don’t wait for 401 errors. Refresh proactively:

  • Track token acquisition timestamp
  • Refresh when 80% of lifetime elapsed (24 minutes for 30-minute tokens)
  • This prevents mid-request expiration

if (currentTime - tokenAcquiredTime > (expiresIn * 0.8)) {
    tokens = refreshAccessToken(refreshToken);
    accessToken = tokens.getAccessToken();
    refreshToken = tokens.getRefreshToken(); // Update refresh token too
}

4. Reactive Token Refresh (401 Handling)

Always implement 401 fallback as safety net:


try {
    response = procurementAPI.createPO(poData, accessToken);
} catch (HTTP401Exception e) {
    // Token expired unexpectedly, refresh and retry
    tokens = refreshAccessToken(refreshToken);
    accessToken = tokens.getAccessToken();
    Thread.sleep(3000); // 3-second backoff for propagation
    response = procurementAPI.createPO(poData, accessToken); // Retry once
}

5. Batch Processing Resilience

For your 500-1000 PO batches:

Option A: Single Session with Refresh (Recommended for < 1 hour jobs)

  • Request initial tokens at batch start
  • Proactively refresh every 24 minutes
  • Implement 401 retry logic as backup
  • Store tokens in memory only (never persist to disk)

Option B: Chunked Processing (For > 1 hour jobs)

  • Break batch into 200-PO chunks
  • Request fresh tokens for each chunk
  • Provides natural isolation if one chunk fails
  • Easier to resume from failure point

6. Complete Implementation Checklist

Token Request Flow:

  • Use client credentials grant for automation (no user interaction)
  • Store client credentials in secure vault (not hardcoded)
  • Request both access_token and refresh_token in initial OAuth call
  • Cache tokens in memory during batch execution

Refresh Logic:

  • Implement proactive refresh at 80% of token lifetime
  • Handle refresh token expiration (rare but possible after 24 hours)
  • Update both access and refresh tokens after each refresh operation
  • Add logging for all token operations (acquisition, refresh, expiration)

Error Handling:

  • Catch HTTP 401 specifically (not generic exceptions)
  • Implement single retry after token refresh (not infinite loops)
  • Add 2-5 second exponential backoff before retry
  • Fail batch and alert if refresh token is also expired

7. Monitoring and Alerting

Track these metrics:

  • Token refresh frequency (should be every ~24 minutes for 30-min tokens)
  • 401 error rate (should be near zero with proactive refresh)
  • Batch completion time (optimize if approaching refresh token lifetime)
  • Refresh token expiration failures (indicates jobs running > 24 hours)

8. Security Best Practices

  • Never log full token values (log only last 4 characters for debugging)
  • Clear tokens from memory after batch completion
  • Rotate client credentials quarterly
  • Use separate client credentials for each automation service
  • Implement rate limiting to prevent token exhaustion

9. Performance Optimization

To reduce batch execution time and token pressure:

  • Process POs in parallel (5-10 threads) if API rate limits allow
  • Implement connection pooling for API calls
  • Cache static data (vendors, categories) to reduce API calls
  • Use bulk API endpoints if Manhattan provides them

10. Testing Strategy

Validate your implementation:

  • Test with batches that exceed 30 minutes (force token expiration)
  • Simulate 401 errors by using expired tokens intentionally
  • Test refresh token expiration scenario (mock 24-hour job)
  • Verify retry logic doesn’t create duplicate POs
  • Load test with maximum batch size (1000 POs)

Implementation Priority:

  1. Add proactive token refresh (80% lifetime threshold)
  2. Implement 401 retry logic with backoff
  3. Add comprehensive logging for token operations
  4. Set up monitoring alerts for token failures
  5. Consider chunked processing if batches exceed 45 minutes

This approach ensures your procurement automation is resilient to OAuth2 token expiration while maintaining security best practices. The combination of proactive refresh and reactive 401 handling provides defense in depth - you should rarely see 401 errors once proactive refresh is working correctly.