Bulk part import fails in cloud deployment due to file size limits

PLM Windchill
, , , , , , , ,
1

We’re migrating 50,000 parts to Windchill 11.2 M030 cloud deployment and bulk import jobs are failing with ‘EntityTooLarge’ errors. The import tool processes the first few thousand parts successfully, then fails when the import file exceeds about 10MB.

Error from the import log:


ERROR: Import failed - Request entity too large
File: bulk_import_batch_05.xml (12.3 MB)
Endpoint: /Windchill/servlet/odata/BulkImport/UploadFile
HTTP Status: 413

I suspect the API gateway file size limit is blocking larger import files, or there are cloud storage upload restrictions we’re hitting. The bulk import tool configuration might also need adjustments for cloud environments. This is blocking our part onboarding schedule - we need to complete the migration within the next two weeks.

system 2

I’ll walk through the complete solution addressing API gateway limits, cloud storage restrictions, and bulk import tool configuration for cloud deployments.

API Gateway File Size Limit Workaround: AWS API Gateway has a hard 10MB payload limit that cannot be increased. For bulk imports, you have two architectural options:

Option 1 - Direct ALB Access: Create a separate DNS record pointing directly to your Application Load Balancer for bulk import operations, bypassing API Gateway. Configure this in Route53:


bulk-import.windchill.yourcompany.com -> ALB DNS name
api.windchill.yourcompany.com -> API Gateway (existing)

Update your bulk import tool configuration to use the direct ALB endpoint. This removes the 10MB limit entirely since ALB supports payloads up to 1GB.

Option 2 - Pre-signed S3 Upload: Implement a two-step import process where the tool first uploads large files directly to S3 using pre-signed URLs, then triggers import via a lightweight API call that references the S3 location. Modify your import workflow:


1. Request pre-signed URL from Windchill
2. Upload import file directly to S3
3. Call import API with S3 object key (small payload)

This is the recommended approach for very large imports (100MB+) as it’s more resilient and provides better progress tracking.

Cloud Storage Upload Restrictions: For S3 uploads over 100MB, you must use multipart upload to ensure reliability and enable resume capability. Configure your bulk import tool’s S3 client to automatically use multipart upload:

In bulk-import-config.properties:

cloud.storage.multipart.enabled=true

cloud.storage.multipart.chunkSize=10485760 (10MB chunks)

cloud.storage.multipart.threshold=52428800 (start multipart at 50MB)

This ensures large import files are uploaded in manageable chunks. If upload fails partway through, it can resume from the last completed chunk rather than restarting the entire file.

Also verify your S3 bucket CORS configuration allows the upload methods:


[
  {
    "AllowedOrigins": ["https://windchill.yourcompany.com"],
    "AllowedMethods": ["PUT", "POST"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }
]

Bulk Import Tool Configuration for Cloud: The bulk import tool needs several cloud-specific settings to handle large files efficiently. Update your bulk-import-config.xml:

Enable chunking to process large imports in smaller batches:

import.chunking.enabled=true

import.chunking.batchSize=1000 (parts per chunk)

import.chunking.maxFileSize=10485760 (10MB, under API Gateway limit)

Configure streaming mode to avoid loading entire file into memory:

import.processing.mode=streaming

import.processing.bufferSize=8192

Set appropriate timeouts for cloud latency:

import.http.connectionTimeout=60000

import.http.readTimeout=300000 (5 minutes for large batches)

Enable retry logic for transient cloud failures:

import.retry.enabled=true

import.retry.maxAttempts=3

import.retry.backoffMultiplier=2

Optimized Migration Strategy: For your 50,000 part migration, implement this approach:

  1. Split your parts into files of 5,000 parts each (approximately 8-9MB per file)
  2. Use the direct ALB endpoint or S3 pre-signed upload method
  3. Run 3-4 parallel import jobs to maximize throughput
  4. Configure import tool to commit every 500 parts (enables partial recovery if job fails)
  5. Enable detailed logging to track progress: import.logging.level=DEBUG

With proper configuration, you should achieve 2,000-3,000 parts per hour throughput, completing your 50,000 part migration in about 20-25 hours of actual processing time. Run imports during off-peak hours to minimize impact on other users.

Monitoring and Validation: Set up CloudWatch alarms for:

  • ALB 5xx errors exceeding 1% (indicates backend issues)
  • Import job duration exceeding 2 hours (indicates performance degradation)
  • S3 multipart upload failures (indicates network or permission issues)

After each import batch completes, run a validation query to confirm part count matches expected: SELECT COUNT(*) FROM WTPart WHERE createdDate > ‘batch_start_time’. This ensures no parts were silently dropped during import.

This draft is based on general Windchill knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

The 413 error confirms it’s an API gateway limit. Most cloud API gateways have a 10MB payload limit by default. You’ll need to either increase the limit or split your import files into smaller batches. What gateway are you using - AWS API Gateway, Azure APIM, or something else?

4

We’re using AWS API Gateway in front of our Windchill application load balancer. I can split the files smaller, but that would mean hundreds of import jobs instead of dozens. Is there a way to increase the API Gateway limit to handle 50-100MB files?

ryanallen 5

Tested this on AWS with Windchill 12.1, and routing bulk part imports directly through the ALB endpoint completely bypassed the 10MB API Gateway payload restriction.

6

AWS API Gateway has a hard limit of 10MB for payload size that can’t be increased. Your options are to use direct ALB access (bypassing API Gateway for bulk imports), implement multipart upload, or split files. For large data migrations, I’d recommend bypassing API Gateway entirely and using a direct endpoint.

7

Bulk import tool has configuration options for chunking large imports. Check your import configuration file for maxBatchSize and enableChunking parameters. You can also configure it to use streaming upload instead of single payload upload, which bypasses the size limit issue entirely.

8

Don’t forget about S3 upload limits if you’re using pre-signed URLs for import file staging. Standard PUT has a 5GB limit but you need to use multipart upload for anything over 100MB to get reliable performance. The bulk import tool should handle this automatically but verify your S3 configuration.

9

Found the combination of issues - API Gateway limit, missing chunking configuration, and S3 multipart not enabled. Working on the fixes.