Batch upload of quality records via REST API fails with timeout on large datasets

PLM Windchill
, , , , , , , ,
1

We’re attempting to upload quality inspection records in batches using the Windchill 11.2 M030 REST API. Individual uploads work fine, but when we try to batch upload 500+ records, we get 413 Payload Too Large errors from the server. We’ve tried reducing batch size to 100 records, but then we hit timeout issues - the request takes over 5 minutes and eventually fails. Our quality data includes measurement values, inspection results, and associated documents.

Error response:


HTTP/1.1 413 Payload Too Large
Content-Length: 0
Connection: close

This is blocking our daily quality data import process, causing significant delays in quality reporting. We need to import 2000+ records daily. What’s the recommended approach for handling large batch uploads?

system 2

Here’s a comprehensive solution addressing all three key areas:

REST API Payload Limits: First, configure your server to handle larger payloads. Update these settings:

In site.xconf:

<Property name="wt.httpgw.maxPostSize" value="20971520"/>
<Property name="wt.method.server.maxRequestSize" value="20971520"/>

If using Apache/nginx proxy, also increase client_max_body_size to 20MB. However, increasing limits alone isn’t the solution - you need proper batch processing.

Batch Processing Strategy: Implement a chunked upload pattern with optimal batch sizing:


// Pseudocode - Chunked batch upload:
1. Split 2000 records into batches of 25 records each
2. Create upload queue with all batches
3. Initialize 3 parallel worker threads
4. Each worker: fetch batch, POST to /quality-records/batch
5. Handle response: success -> next batch, failure -> retry logic
6. Track progress and log results

For 15KB records, 25-record batches = ~375KB per request, well under limits and processes in 20-30 seconds. This gives you 80 batches total, processed by 3 workers in parallel = ~10-12 minutes total for 2000 records.

Server/Proxy Configuration: Beyond payload limits, optimize timeout settings:

  • Method server connection timeout: Increase to 120 seconds minimum
  • Database connection pool: Ensure maxActive >= number of worker threads + 10
  • Transaction timeout: Set to 180 seconds for batch operations

In site.xconf:

<Property name="wt.pom.dbcp.maxWait" value="120000"/>
<Property name="wt.method.server.connectionTimeout" value="120000"/>

Implement retry logic with exponential backoff for failed batches. Use HTTP status codes to determine retry strategy: 413/408 = reduce batch size, 500/503 = retry after delay, 4xx client errors = log and skip.

For production deployment, add monitoring to track batch processing metrics: success rate, average processing time per batch, and failure patterns. This helps tune batch size and worker count based on actual system performance.

Consider implementing a staging table approach where you bulk insert to a temporary table first, then use Windchill’s bulk loader utilities to process into final quality objects. This can be 5-10x faster for very large datasets.

This draft is based on general Windchill knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

The 413 error suggests you’re hitting either the Windchill server’s request size limit or an intermediate proxy limit. Check your Apache/web server configuration for client_max_body_size or similar settings. For Windchill specifically, look at the wt.httpgw.maxPostSize property in site.xconf. Default is usually 2MB which is too small for bulk quality data. You might need to increase it to 10-50MB depending on your record size.

4

Even if you fix the payload size limit, uploading 500 records in a single request isn’t optimal. The REST API has to process each record sequentially, validate relationships, and commit to the database. That’s why you’re seeing timeouts. Consider implementing chunked uploads - break your 2000 records into smaller batches of 20-50 records each and process them in parallel. This also gives you better error handling - if one batch fails, you don’t lose the entire import.

6

Good point about chunking. What’s the recommended batch size for quality records specifically? Our records average about 15KB each with all the measurement data and metadata. Also, if we process batches in parallel, won’t that cause database locking issues? We’re using SQL Server as the backend.

8

For SQL Server backend with quality records, I’d recommend batches of 25-30 records max. This keeps each transaction under 500KB and completes in under 30 seconds typically. For parallel processing, use a queue-based approach with 3-4 worker threads maximum to avoid lock contention. Quality records don’t usually have complex dependencies, so parallel inserts should be safe as long as you’re not trying to update the same parent objects simultaneously.

9

Also consider using async processing if your Windchill version supports it. Submit the batch upload as a background job through the REST API and poll for completion status. This prevents timeout issues entirely since the client connection isn’t held open during processing. You can submit multiple async jobs in parallel for different batches and monitor their progress. Check the /background-jobs endpoint in your API documentation.