I’ll provide a comprehensive solution addressing data import deduplication, API query filters, and backend data consistency issues.
Understanding the Duplicate Records Issue:
The duplicate records appearing in API search results but not in the UI indicates a disconnect between three components:
- Primary database (where actual contact records are stored)
- Search index (used by the API for fast queries)
- UI query layer (which may apply additional deduplication logic)
Your bulk import likely created true duplicate records in the database that the UI is filtering out client-side, while the API returns raw search index results.
Data Import Deduplication:
Step 1: Verify Actual Duplicates in Database
First, confirm whether duplicates exist in the primary database or just the search index:
GET /api/admin/contacts/duplicates?field=email&threshold=exact
Response:
{
"duplicate_groups": [
{
"email": "john.doe@example.com",
"record_ids": [12345, 67890],
"created_dates": ["2025-09-25", "2025-09-25"]
}
],
"total_duplicates": 1247
}
This confirms true duplicates exist in the database.
Step 2: Analyze Import Job for Root Cause
Review your import job configuration and logs:
GET /api/admin/imports/job-12345/details
Response:
{
"job_id": "job-12345",
"records_processed": 50000,
"records_inserted": 51247, // More than source!
"duplicates_detected": 0,
"deduplication_enabled": false, // Root cause
"match_fields": []
}
The issue: deduplication wasn’t enabled during import, so all 50,000 records were inserted even if some matched existing contacts.
Step 3: Configure Proper Deduplication Rules
For future imports, enable deduplication with appropriate match fields:
POST /api/admin/imports/configure
{
"deduplication": {
"enabled": true,
"match_fields": ["email"],
"match_strategy": "exact",
"on_duplicate": "update_existing",
"preserve_master_record": "oldest"
}
}
This ensures future imports update existing contacts rather than creating duplicates.
API Query Filters for Deduplication:
While you fix the backend data, you can use API filters to work around the duplicate results:
Option 1: Use Distinct Query Parameter
GET /api/contacts/search?email=john.doe@example.com&distinct=true&distinct_field=email
This returns only one record per unique email address.
Option 2: Filter by Master Record Flag
GET /api/contacts/search?email=john.doe@example.com&master_only=true
This returns only records marked as master (not duplicates).
Option 3: Client-Side Deduplication
If API parameters aren’t available, deduplicate in your code:
response = GET("/api/contacts/search?email=john.doe@example.com")
contacts = response.json()["contacts"]
# Deduplicate by email, keep oldest record
unique_contacts = {}
for contact in contacts:
email = contact["email"]
if email not in unique_contacts or \
contact["created_at"] < unique_contacts[email]["created_at"]:
unique_contacts[email] = contact
result = list(unique_contacts.values())
Backend Data Consistency Resolution:
Step 1: Merge Duplicate Records
Use the bulk merge API to consolidate duplicates:
POST /api/admin/contacts/merge-duplicates
{
"match_field": "email",
"match_strategy": "exact",
"merge_strategy": {
"master_selection": "oldest",
"field_resolution": "prefer_master",
"preserve_relationships": true
},
"batch_size": 1000
}
Response:
{
"job_id": "merge_job_xyz",
"status": "processing",
"estimated_duration": "45 minutes"
}
This automatically merges all duplicate contacts, preserving the oldest record as master and transferring all relationships (opportunities, activities, etc.) to the master record.
Step 2: Monitor Merge Progress
GET /api/admin/contacts/merge-duplicates/merge_job_xyz
Response:
{
"status": "completed",
"duplicates_found": 1247,
"records_merged": 1247,
"master_records_retained": 1247,
"duplicate_records_archived": 1247
}
Step 3: Rebuild Search Index
After merging duplicates, refresh the search index:
POST /api/admin/contacts/search-index/rebuild
{
"full_rebuild": true,
"priority": "high"
}
Response:
{
"job_id": "reindex_abc",
"status": "queued",
"estimated_duration": "60 minutes",
"records_to_index": 248753
}
The full rebuild ensures the search index perfectly matches the database after the merge operation.
Step 4: Verify Resolution
After reindexing completes, verify the duplicates are gone:
GET /api/contacts/search?email=john.doe@example.com
Response:
{
"contacts": [
{
"id": 12345, // Only one record now
"email": "john.doe@example.com",
"is_master": true,
"merged_from": [67890]
}
],
"total": 1
}
Preventing Future Duplicates:
- Enable Real-Time Deduplication:
POST /api/admin/settings/contacts
{
"duplicate_prevention": {
"enabled": true,
"match_fields": ["email", "phone"],
"match_threshold": "exact",
"block_duplicate_creation": true
}
}
- Implement Pre-Import Validation:
Before importing, check for existing records:
def validate_import(contacts):
existing_emails = GET("/api/contacts/emails/bulk-check",
json={"emails": [c["email"] for c in contacts]})
new_contacts = [c for c in contacts
if c["email"] not in existing_emails]
updates = [c for c in contacts
if c["email"] in existing_emails]
return new_contacts, updates
- Schedule Regular Duplicate Scans:
POST /api/admin/schedules/create
{
"task": "duplicate_detection",
"frequency": "weekly",
"auto_merge": false,
"notify_on_duplicates": true
}
Complete Resolution Timeline:
- Immediate (5 min): Enable API query filters to work around duplicates
- Short-term (1 hour): Run duplicate merge job to consolidate records
- Medium-term (2 hours): Rebuild search index after merge completes
- Long-term (ongoing): Enable real-time deduplication and scheduled monitoring
After completing these steps, your contact search API will return consistent, deduplicated results matching the UI behavior, and your sales team’s lead assignment automation will function correctly with reliable contact IDs.
This draft is based on general Adobe Experience Cloud knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.