I’ll provide a comprehensive solution addressing lead deduplication logic, external ID mapping, and proper upsert API operation configuration to eliminate duplicate leads in your integration.
Lead Deduplication Logic:
SAP CX 2111 uses a multi-tier matching strategy for lead deduplication during upsert operations. You need to configure this properly:
-
Configure deduplication rules in SAP CX:
- Navigate to: Lead Management > Configuration > Deduplication Rules
- Enable ‘Use External ID for Matching’ (priority 1)
- Enable ‘Email Address Matching’ (priority 2, fallback)
- Enable ‘Phone Number Matching’ (priority 3, fallback)
- Set matching logic: “Match if ANY rule matches” (not ALL)
-
Implement validation in integration middleware:
function validateLeadBeforeUpsert(lead) {
if (!lead.externalId || lead.externalId.trim() === '') {
// Try to retrieve externalId from SAP CX using email
const existingLead = querySAPCX({email: lead.email});
if (existingLead) {
lead.externalId = existingLead.externalId;
} else {
// Generate stable external ID from marketing platform
lead.externalId = generateExternalId(lead);
}
}
return lead;
}
- Configure duplicate detection in SAP CX:
- Administration > Data Quality > Duplicate Detection
- Enable real-time duplicate checking for leads
- Set matching threshold: 85% (balance between strict and flexible)
- Configure matching fields: externalId (exact), email (fuzzy), firstName+lastName+company (fuzzy)
External ID Mapping:
Proper external ID mapping is critical for preventing duplicates. Here’s the complete configuration:
-
Configure external ID field in SAP CX Lead entity:
- Navigate to: Administration > Data Modeling > Lead
- Locate field: ‘externalId’ (create if doesn’t exist)
- Field properties:
- Type: String
- Max length: 100
- Required: Yes (make mandatory)
- Unique: Yes (enable unique constraint)
- Indexed: Yes (for performance)
-
Map external IDs between systems:
Marketing Platform → SAP CX:
- Marketing Platform Lead ID → externalId (primary matching key)
- Marketing Platform Email → email (secondary matching key)
- Marketing Platform Contact ID → externalContactId (related contact reference)
-
Implement external ID generation strategy:
- Use marketing platform’s unique lead identifier as externalId
- Format: “MKTG-{platform_lead_id}” (e.g., “MKTG-12345”)
- Ensure consistency: same lead always gets same externalId
- Store mapping in integration database for reference
-
Handle external ID synchronization:
function syncExternalId(marketingLead) {
// Generate or retrieve external ID
const externalId = `MKTG-${marketingLead.id}`;
// Check if lead exists in SAP CX
const sapLead = await querySAPCXLead({externalId: externalId});
if (sapLead) {
// Update existing lead
return {action: 'update', sapLeadId: sapLead.id, externalId: externalId};
} else {
// Check for potential duplicate by email
const duplicateCheck = await querySAPCXLead({email: marketingLead.email});
if (duplicateCheck) {
// Update existing lead with external ID
await updateSAPCXLead(duplicateCheck.id, {externalId: externalId});
return {action: 'update', sapLeadId: duplicateCheck.id, externalId: externalId};
}
// Create new lead
return {action: 'create', externalId: externalId};
}
}
Upsert API Operation:
Configure the upsert API operation correctly to leverage external ID matching:
- Proper upsert API payload structure:
{
"externalId": "MKTG-12345",
"email": "john.doe@example.com",
"firstName": "John",
"lastName": "Doe",
"company": "Acme Corp",
"leadSource": "Marketing Campaign",
"leadScore": 75,
"status": "Qualified"
}
- Use correct API endpoint and method:
POST /sap/cx/api/v1/leads/upsert
Content-Type: application/json
X-Matching-Strategy: externalId
- Configure upsert behavior in API request headers:
X-Matching-Strategy: externalId (use external ID as primary matching key)
X-Create-If-Not-Found: true (create new lead if no match)
X-Update-Mode: merge (merge new data with existing, don't overwrite all fields)
- Implement proper error handling for upsert operations:
async function upsertLead(lead) {
try {
const response = await axios.post('/sap/cx/api/v1/leads/upsert', lead, {
headers: {
'X-Matching-Strategy': 'externalId',
'X-Create-If-Not-Found': 'true',
'X-Update-Mode': 'merge'
}
});
if (response.data.action === 'created') {
logLeadCreation(lead.externalId, response.data.sapLeadId);
} else if (response.data.action === 'updated') {
logLeadUpdate(lead.externalId, response.data.sapLeadId);
}
return response.data;
} catch (error) {
if (error.response?.status === 409) {
// Duplicate detected - retrieve existing lead and update
const existingLead = await getSAPCXLead({externalId: lead.externalId});
return await updateSAPCXLead(existingLead.id, lead);
}
throw error;
}
}
- Implement pre-upsert duplicate check (recommended for high-volume integrations):
async function smartUpsert(marketingLead) {
// Step 1: Check if lead exists by external ID
let sapLead = await querySAPCX({externalId: marketingLead.externalId});
if (sapLead) {
// Lead exists - perform update
return await updateSAPCXLead(sapLead.id, marketingLead);
}
// Step 2: Check for potential duplicate by email
sapLead = await querySAPCX({email: marketingLead.email});
if (sapLead) {
// Duplicate found - update with external ID
marketingLead.id = sapLead.id;
return await updateSAPCXLead(sapLead.id, marketingLead);
}
// Step 3: No duplicate found - create new lead
return await createSAPCXLead(marketingLead);
}
Additional Recommendations:
-
Implement sync audit logging:
- Log every upsert operation with timestamp, external ID, SAP lead ID, and action taken
- Monitor for duplicate creation patterns
- Set up alerts for unexpected duplicate creation
-
Run duplicate cleanup:
- Execute one-time duplicate merge for existing duplicates
- Use SAP CX duplicate management tools: Lead Management > Data Quality > Find Duplicates
- Merge duplicates keeping the record with external ID populated
-
Configure sync frequency:
- Real-time sync for high-priority leads (immediate upsert)
- Batch sync for bulk updates (deduplicate before sending)
- Implement rate limiting to avoid API throttling
-
Monitor integration health:
- Track duplicate creation rate (should be near 0%)
- Monitor upsert success rate (should be >99%)
- Alert on missing external IDs in sync payload
After implementing these changes, test thoroughly with your marketing platform integration. The combination of proper external ID mapping, configured deduplication rules, and correct upsert API usage should eliminate duplicate lead creation. Monitor the integration for 48 hours after deployment to ensure duplicates are no longer being created.