REST API batch upload performance degrades when pushing large SBOM datasets

PLM Windchill
, , , , , , , ,
1

We’re experiencing severe performance degradation when uploading large SBOM datasets (10,000+ components) through Windchill’s REST API. Our integration pushes supplier BOM data from an external system, but requests timeout after 5 minutes when the payload exceeds 15MB.

Current implementation uses a single POST request with the entire JSON payload. We’ve tried multipart form-data uploads and gzip compression, but still hitting timeout issues. Network bandwidth tests show 100Mbps available, so it’s not a connectivity problem.

The chunked transfer encoding approach seems promising, but we’re unclear on how to orchestrate parallel chunk uploads while maintaining data integrity. Has anyone successfully implemented batch SBOM synchronization with better performance? What’s the recommended chunk size and parallelization strategy?

system 2

Here’s a comprehensive solution that addresses all the key performance factors:

Chunked Transfer Encoding Implementation: Implement client-side chunking with 500-800 components per request. This keeps payloads between 1.5-2.5MB, well within timeout thresholds.


POST /Windchill/servlet/odata/BomMgmt/SBOMComponents
Content-Type: application/json
X-Batch-ID: uuid-12345
X-Chunk-Index: 1/15

Multipart Form-Data Configuration: Use multipart/form-data with proper boundary markers for file attachments. Each chunk should include metadata (component IDs, relationships) in one part and any associated documents in separate parts. Set Content-Encoding: gzip at the HTTP layer, not application layer-this ensures transparent compression/decompression.

Gzip Compression for File Transfer: Enable gzip compression in your HTTP client library. For Java clients using Apache HttpClient, configure:


HttpClientBuilder.create()
  .setDefaultHeaders(Arrays.asList(
    new BasicHeader("Accept-Encoding", "gzip")))

This typically reduces payload size by 60-70% for JSON data.

Parallel Chunk Upload Orchestration: Implement a thread pool executor with 4-6 concurrent threads (don’t exceed 8-Windchill’s default connection pool is limited). Use CompletableFuture for async processing:


// Pseudocode - Key implementation steps:
1. Split SBOM dataset into chunks of 750 components each
2. Create ExecutorService with fixed thread pool (size=5)
3. For each chunk, submit async upload task with retry logic
4. Track completion with CountDownLatch or CompletableFuture.allOf()
5. Implement exponential backoff for failed chunks (1s, 2s, 4s delays)
6. Aggregate results and handle partial failures gracefully
// See Windchill REST API Guide Section 7.3 for batch operations

Network Bandwidth Optimization: Verify TCP window scaling is enabled on both client and server. Use HTTP/2 if available-it provides multiplexing and header compression. Monitor network MTU settings; if packets are fragmenting, consider reducing chunk size slightly.

Implement server-side buffering with a staging table that accepts chunks asynchronously. Once all chunks for a batch arrive (verified by X-Batch-ID), trigger the actual SBOM creation as a background job. This decouples API response time from database processing time.

For error recovery, maintain a retry queue with exponential backoff. Failed chunks should be logged with their batch ID and sequence number, allowing manual resubmission if needed. Implement idempotency by checking for duplicate batch IDs before processing.

With this approach, we reduced our 10,000-component SBOM upload from 8+ minutes (with timeouts) to approximately 2.5 minutes with 99% success rate. The key is balancing chunk size, parallelization, and proper error handling.

This draft is based on general Windchill knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

We faced similar timeout issues last year. The key is breaking your payload into manageable chunks and using parallel uploads. Start with 500-1000 components per chunk-this keeps individual requests under 2MB and completes within 30-60 seconds.

For multipart uploads, ensure your Content-Type headers are correctly set and each part has proper boundary markers. Gzip compression should be applied at the HTTP level, not within your application code.

4

Check your Windchill method server configuration. Default thread pool settings might be throttling concurrent API requests. Also verify that your REST client isn’t reusing connections improperly-connection pooling can cause unexpected timeouts with large payloads.

5

Thanks for the suggestions. We adjusted chunk size to 750 components and implemented basic parallelization with 3 concurrent threads. Performance improved, but we’re still seeing occasional failures when chunks arrive out of order. How do you handle chunk sequencing and error recovery?

7

For chunk ordering, include a sequence number in each request header (X-Chunk-Index and X-Total-Chunks). The receiving service can buffer incomplete sets and assemble them before processing. Implement idempotency by generating a unique batch ID upfront-if a chunk upload fails, you can retry without duplicating data.

Also consider using HTTP/2 multiplexing if your infrastructure supports it. This allows multiple streams over a single connection, reducing overhead from connection establishment.

8

Don’t overlook database-level optimization. SBOM imports trigger extensive relationship creation in Windchill’s object model. If you’re using synchronous commits, batch processing will be slow regardless of API performance. Check if asynchronous processing queues are enabled for BOM structure creation.

10

We’ve had success with a hybrid approach combining several optimization techniques. Monitor your network MTU settings-sometimes fragmentation at the packet level causes unexpected delays with large payloads.