Supplier sync integration fails with duplicate vendor IDs when batch importing

QMS MasterControl Quality Excellence
, , , , , , , ,
1

Our supplier-mgmt integration is failing during bulk import operations in MC 2022.2. We’re syncing vendor data from our ERP system via REST API, and the batch import consistently fails with duplicate vendor ID errors, even though we’ve verified the source data has unique IDs.

Error we’re seeing:


BatchImportException: Duplicate key violation - vendor_id 'V-10234' already exists
Failed at record 47 of 200 in batch

The strange part is that vendor V-10234 doesn’t exist in our supplier database when we check manually. We’re handling batch import error scenarios, but the duplicate key validation seems to be triggering incorrectly. Our API payload structure follows the documentation. Has anyone dealt with phantom duplicate key errors in supplier batch imports?

2

Here’s a comprehensive solution addressing all three aspects of your batch import issue:

1. Batch Import Error Handling: Implement proper error handling with transaction rollback. When a duplicate is detected, the entire batch fails by default in MC 2022.2. Use the continueOnError parameter:

{
  "batchConfig": {
    "continueOnError": true,
    "errorThreshold": 10
  }
}

This allows the batch to continue processing and returns a detailed error report showing which specific records failed and why.

2. Duplicate Key Validation: Implement client-side deduplication before sending to MC API. Here’s the approach:

  • Extract unique vendor IDs from your merged ERP data using a Set or HashMap
  • For duplicates within source data, apply merge logic (latest timestamp wins, or merge fields)
  • Validate against MC’s existing vendor IDs using GET `/api/v1/suppliers?fields=vendorId
  • Only send deduplicated records in batch

Pseudocode for deduplication:


// Pseudocode - Key implementation steps:
1. Load all vendor records from ERP sources into collection
2. Create HashMap with vendorId as key, full record as value
3. For duplicate vendorIds, compare timestamps and keep latest
4. Query MC API for existing vendorIds in system
5. Mark records as 'insert' or 'update' based on existence
6. Build final batch payload with operation type per record
// This ensures no intra-batch duplicates

3. API Payload Structure: Structure your batch payload correctly for upsert operations:

{
  "suppliers": [
    {
      "vendorId": "V-10234",
      "operation": "upsert",
      "name": "Acme Supplies",
      "status": "active"
    }
  ],
  "batchConfig": {
    "validateOnly": false,
    "continueOnError": true
  }
}

Key recommendations:

  1. Always deduplicate source data before API calls - don’t rely on MC to handle it
  2. Use ‘upsert’ operation mode if your ERP is source of truth
  3. Implement pre-flight validation: call the API with validateOnly: true first to catch issues
  4. Enable detailed error logging to track which specific records fail
  5. Consider implementing batch size limits (100-200 records per batch) for better error isolation

If you’re still seeing phantom duplicates after deduplication, check for case sensitivity issues in vendor IDs. MC 2022.2 treats ‘V-10234’ and ‘v-10234’ as different IDs in validation but may normalize them during insert, causing conflicts.

3

Check if you have soft-deleted suppliers in the system. MC 2022.2 validates vendor IDs against both active and archived records by default. If a supplier was previously deactivated but not purged, the ID still exists in the validation cache. Try querying with includeArchived=true parameter to see all records.

4

That makes sense. We’re pulling data from multiple ERP modules and merging them, so duplicates could sneak in. What’s the best way to handle this? Should we deduplicate client-side before sending, or is there an API parameter to handle duplicate IDs within a batch gracefully?

5

There’s no built-in API parameter for graceful duplicate handling within a batch - you must deduplicate client-side. However, you can use the ‘upsert’ operation mode instead of ‘insert’. With upsert, if a vendor ID exists (either in DB or earlier in batch), it updates the record instead of failing. Add "operation": "upsert" to your batch request. This works well if your ERP is the source of truth and you want to ensure latest data wins.

6

Yes, that’s exactly what happens. The batch import API validates each record in the batch sequentially, and once a vendor ID is processed in the current batch, it’s added to the in-memory validation set. If the same ID appears later in the same batch, it triggers a duplicate key error. The API doesn’t distinguish between ‘already in database’ vs ‘already in current batch’. You need to deduplicate your payload before sending it. Also check if your ERP export is somehow duplicating records.