Part import validation fails on duplicate attribute values during bulk upload

PLM Oracle Agile PLM
, , , , , , , ,
1

We’re running bulk part imports in our QA environment using Data Import tool and consistently hitting validation failures when duplicate attribute values exist across parts. The import succeeds for the first 50-60 parts, then fails with constraint violation errors.

Our attribute uniqueness constraints are configured correctly in the class structure, and we’ve verified the data pre-validation logic checks for duplicates. However, the import process doesn’t catch these until mid-batch:


ERROR: Duplicate value 'SUPP-12345' for attribute 'Supplier Code'
at ImportValidator.validate(line 234)
Batch halted at record 64 of 200

The error handling doesn’t roll back successfully imported parts, leaving us with partial datasets that complicate testing. We need better bulk import error handling that validates uniqueness before starting the batch and provides clear failure reporting. Has anyone implemented pre-validation best practices for large part imports that catch attribute conflicts early?

2

Are you using the standard Import template or a custom one? Standard templates don’t have robust duplicate checking across the batch file itself - only against existing database records. The partial import problem occurs because there’s no transaction boundary around the entire batch. Each successful row commits immediately, so failures mid-batch leave orphaned records. You might need to add explicit rollback handling in your import configuration.

3

I’ve seen this exact issue. The Data Import tool validates row-by-row during processing rather than pre-validating the entire batch. You need to implement a pre-validation script that checks uniqueness constraints before submitting to Agile. We built a simple Java validator that queries existing parts and flags duplicates in the import file itself.

4

Here’s a comprehensive solution addressing all three aspects - attribute uniqueness constraints, bulk import error handling, and data pre-validation best practices.

Pre-Validation Implementation: Build a validation layer before the import process. Use Agile SDK to query existing parts and check your import file for internal duplicates:

// Pre-validate unique attributes
IAgileClass partClass = session.getAdminInstance().getAgileClass("Part");
IAttribute supplierAttr = partClass.getAttribute("Supplier Code");
if (supplierAttr.isUnique()) {
    validateUniqueness(importData, supplierAttr);
}

Bulk Import Error Handling: Implement transaction-like behavior by maintaining a validation log. Before starting the import, run a complete validation pass that checks:

  1. Attribute uniqueness constraints against existing database records
  2. Internal duplicates within the import file itself
  3. Required field completeness
  4. Format and data type compliance

Create a validation report that lists all constraint violations with row numbers and specific attribute conflicts. This allows you to fix data issues before committing any records.

Best Practices for Large Imports:

  • Split large imports into logical batches (50-100 records) with validation checkpoints
  • Implement a staging table approach where you load data to a temporary table first, run all validations, then promote to Agile only if 100% clean
  • Use the Agile API’s batch processing capabilities with explicit error handling
  • Maintain detailed logs of validation failures with specific attribute values and conflict details
  • Create a rollback script that can remove partially imported parts based on import session ID

Validation Script Structure: Your pre-validator should:

  1. Load the import file into memory or staging database
  2. Query Agile for all existing values of unique attributes
  3. Build a hash map of existing values for O(1) lookup
  4. Scan import file for duplicates against both existing data and within the file itself
  5. Generate detailed validation report before any import execution

This approach has reduced our import failures from 40% to under 5%, and when failures do occur, we have clean datasets with no partial imports to clean up. The validation step adds about 2-3 minutes to a 200-part import but saves hours of troubleshooting and data cleanup.

5

Check your import batch size settings. We reduced batch size from 200 to 50 records and added checkpoint validation between batches. This doesn’t prevent duplicates but makes error recovery much cleaner since you only lose 50 records maximum on failure instead of the entire dataset. Also helps identify which specific records are problematic faster.