Bulk part creation via XML import in part management is extremely slow and causes Java heap errors

PLM Oracle Agile PLM
, , , , , , , ,
1

We’re experiencing severe performance issues when importing large XML files for bulk part creation in Agile 9.3.4. Files containing 5000+ parts are taking over 6 hours to process, which is unacceptable for our onboarding schedule.

Our current approach uses the Agile SDK to parse the entire XML file into memory before processing:

Document doc = builder.parse(new File("parts.xml"));
NodeList parts = doc.getElementsByTagName("part");
for (int i = 0; i < parts.getLength(); i++) {
    createPart(parts.item(i));
}

We’re seeing Java heap space errors around the 3000-part mark, and WebLogic threads seem to be maxed out. I’ve tried adjusting batch sizes but haven’t found the right balance between memory usage and throughput. Has anyone successfully optimized large XML imports for part creation? We need to handle files up to 10,000 parts efficiently.

2

We had this exact issue last year. Two things helped immediately: First, we switched to processing in smaller batches of 500 parts with commits between batches. Second, we increased the JVM heap from 2GB to 6GB and tuned the garbage collection settings. The combination reduced our import time from 5 hours to about 90 minutes for 5000 parts. Are you committing after each part or batching your commits?

3

Here’s the complete solution we implemented that reduced our 10,000-part import from 8 hours to under 2 hours.

XML Streaming Parser: Replace DOM parsing with StAX for memory efficiency:

XMLStreamReader reader = factory.createXMLStreamReader(new FileInputStream("parts.xml"));
while (reader.hasNext()) {
    if (reader.isStartElement() && "part".equals(reader.getLocalName())) {
        processPart(reader);
    }
}

JVM Heap Tuning: Increase heap size and optimize GC:

  • Set -Xms4g -Xmx6g for initial and maximum heap
  • Use G1GC: `-XX:+UseG1GC -XX:MaxGCPauseMillis=200
  • Monitor with `-XX:+PrintGCDetails -XX:+PrintGCTimeStamps Batch Import Strategy: Commit every 100 parts to reduce database overhead:
int batchSize = 100;
for (int i = 0; i < parts.size(); i++) {
    createPart(parts.get(i));
    if ((i + 1) % batchSize == 0) {
        session.commit();
    }
}

WebLogic Thread Pool: Edit config.xml to increase capacity:

<execute-thread-count>50</execute-thread-count>
<max-threads>75</max-threads>

Agile PX Parallelization: Split XML into chunks and process concurrently:

  • Divide input file into 4-6 segments based on available CPU cores
  • Use ExecutorService with fixed thread pool
  • Each thread processes its segment with independent SDK session
  • Aggregate results after all threads complete

Additional Optimizations:

  • Disable unnecessary change order creation during import
  • Turn off workflow notifications temporarily
  • Schedule imports during off-peak hours
  • Add database indexes on part number and class columns
  • Increase database connection pool: `maxActive=100, maxIdle=20 Monitoring: Track these metrics during import:
  • JVM heap usage via JConsole
  • WebLogic thread utilization
  • Database connection pool stats
  • Import rate (parts per minute)

With these changes, we consistently process 10,000 parts in 1.5-2 hours with no heap errors. The key is combining streaming parsing, batch commits, and parallel processing while ensuring your infrastructure (JVM, WebLogic, database) can support the load.

4

One more consideration: Agile PX can parallelize part creation if configured correctly. You can split your XML file into chunks and process them concurrently using multiple threads, but be careful with transaction boundaries. We typically use 4-6 parallel threads for optimal throughput without overwhelming the database. Each thread handles its own batch with independent transactions.

5

Loading the entire XML into memory is your main bottleneck. For files with thousands of parts, you need a streaming parser approach. Consider using SAX or StAX instead of DOM parsing - they process XML incrementally without loading everything into heap. Also check your WebLogic thread pool configuration; default settings often can’t handle sustained batch operations.

6

The per-part commit pattern is killing your performance. Database round-trips add up quickly. We use batch commits every 100 parts and saw a 70% improvement. Also monitor your database connection pool - if you’re exhausting connections, consider increasing the pool size. For WebLogic, bump maxThreadPoolSize to at least 50 for import operations. Don’t forget to tune your database indexes on the parts table, especially if you’re doing lookups during import to check for duplicates.