Bulk import in contact management bypasses duplicate detection

CRM Microsoft Dynamics 365 Sales
, , , , , , , ,
1

Running into a frustrating issue with D365 Sales 9.0. When importing contacts in bulk through the Data Import Wizard, our duplicate detection rules are being completely bypassed. We’re ending up with hundreds of duplicate contacts after each import run.

Our duplicate detection rule is configured to match on email address and company name, and it works perfectly when users create contacts manually through the UI. But bulk imports just ignore it. Here’s what the import log shows:

<ImportResult>
  <RecordsProcessed>1247</RecordsProcessed>
  <RecordsCreated>1247</RecordsCreated>
  <DuplicatesDetected>0</DuplicatesDetected>
  <DuplicateDetectionEnabled>false</DuplicateDetectionEnabled>
</ImportResult>

We need proper duplicate detection settings for bulk operations, better bulk import configuration, and a reliable post-import cleanup process. The data quality issues are getting worse with each import cycle. Anyone solved this in 9.0?

system 2

Let me walk you through a complete solution covering all three focus areas:

Duplicate Detection Settings: First, verify the system-wide setting at Settings > Data Management > Duplicate Detection Settings. Ensure “Enable duplicate detection” is checked for both “When a record is created or updated” AND “During data import”. This is the master switch.

Next, review your duplicate detection rule. Open it and check:

  1. Status must be “Published” (not just saved)
  2. Base Record Type and Matching Record Type both set to “Contact”
  3. The rule criteria matches your data - if imports use different field formatting (like email in uppercase), the match will fail
  4. Rule is not expired - check the Valid Until date

Test the rule manually: Create a contact, then try to create another with the same email. If it doesn’t trigger a duplicate warning, the rule itself is broken.

Bulk Import Configuration: The 5,000 record threshold mentioned earlier is real in 9.0. Here’s the proper configuration:

<ImportConfig>
  <BatchSize>2500</BatchSize>
  <EnableDuplicateDetection>true</EnableDuplicateDetection>
  <DuplicateDetectionMode>Strict</DuplicateDetectionMode>
</ImportConfig>

In the Data Import Wizard:

  1. Map your fields carefully - ensure email and company name are mapped to the correct D365 fields
  2. On the “Review Mapping and Import Data” page, click “Advanced Options”
  3. Check “Enable duplicate detection”
  4. Select your specific duplicate detection rule from the dropdown (don’t rely on default)
  5. Set “Duplicate Handling” to “Do not import duplicate records” rather than “Import and mark as duplicates”

For CSV imports, ensure email addresses are properly formatted and company names match your existing data casing (“Acme Corp” vs “ACME CORP” won’t match unless you use case-insensitive matching).

Post-Import Cleanup Process: Even with proper settings, you need a safety net. Here’s a three-tier cleanup approach:

  1. Immediate Post-Import Scan (within 1 hour):

    • Run the built-in “Detect Duplicates” job targeting contacts created today
    • Filter to only records from your import file (add a “Import Batch ID” field to track this)
    • Review and merge obvious duplicates immediately

  2. Weekly Duplicate Audit (automated):

    • Create an Advanced Find view: Contacts where Email contains data AND Created On = Last 7 Days
    • Export to Excel, use Remove Duplicates function on email column
    • Compare count - difference shows duplicate volume
    • Use D365’s Merge Records feature for cleanup

  3. Preventive Data Scrubbing (before import):

    • Export your import file to Excel first
    • Use Excel’s Remove Duplicates on email column before importing
    • Standardize company names (TRIM, PROPER functions)
    • Validate email format with a formula

For automated cleanup, create a workflow or Power Automate flow that runs daily:

  • Query contacts created in last 24 hours
  • Group by email address having count > 1
  • Send report to data stewards with merge recommendations
  • Optionally auto-merge if confidence is high (exact email + company match)

The key is defense in depth: fix the detection settings, optimize your import configuration, and maintain cleanup processes for anything that slips through.

This draft is based on general Microsoft Dynamics 365 Sales knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

The import log is showing the problem right there - DuplicateDetectionEnabled is false. In the Data Import Wizard, there’s a checkbox on the review mapping page that enables duplicate detection. It’s easy to miss because it’s near the bottom. Make sure that’s checked before running the import.

4

I’ve checked that box multiple times, but the import still shows it as disabled in the logs. Could there be a system setting that’s overriding the wizard option? Or maybe a permission issue?

erp_eta 5

Confirmed this resolves the issue — publishing the duplicate detection rule and enabling ‘During data import’ in Data Management Settings stopped duplicate contacts from slipping through our bulk imports.

6

Check Settings > Data Management > Duplicate Detection Settings. There’s a system-wide setting for “Enable duplicate detection during import”. If that’s unchecked, it overrides the per-import checkbox. Also verify your duplicate detection rules are published and active. An unpublished rule won’t run even if detection is enabled. Go to Settings > Data Management > Duplicate Detection Rules and check the status column.

7

In 9.0, there’s also a performance threshold that disables duplicate detection automatically for large imports. If you’re importing more than 5,000 records, the system may be silently disabling it to avoid timeout issues. Try breaking your import into smaller batches of 2,000-3,000 records each. Not ideal, but it’s a known limitation in this version that was improved in later releases.

8

For post-import cleanup, I built a Power Automate flow that runs after imports. It queries for contacts created in the last hour, groups them by email, and flags potential duplicates in a custom field. Then our data stewards review and merge them. Not perfect, but it catches what the import missed. You could also use the Detect Duplicates job in Advanced Find to identify them after the fact.