Lead Management: Duplicate detection failing for leads with multiple custom identifiers

[victoria_powerbiar]

Our Lead Management system is creating duplicate lead records even though we have duplicate detection rules configured. The issue specifically occurs when leads come in with multiple custom identifiers that we use for lead tracking-things like partner_referral_code, campaign_tracking_id, and customer_internal_id.

We’ve configured the standard duplicate detection to match on email address and phone number, which works fine for simple cases. However, when a lead has been referred through multiple channels or has interacted with different campaigns, they might have different combinations of these custom identifiers. The duplicate detection doesn’t seem to recognize that these are the same person.

For example, a lead might come in from Partner A with partner_referral_code=“PRTNR-A-12345” and campaign_tracking_id=“CAMP-2025-Q2”. Then the same person fills out another form from Partner B with partner_referral_code=“PRTNR-B-67890” and a different campaign ID. Even though the email and phone match, the system creates two separate lead records because the custom identifiers are different.

How can we configure the matching rule to properly handle custom field deduplication while still maintaining data quality?

Your duplicate detection challenges stem from trying to use tracking identifiers as primary identity fields when they should be treated as relationship data. Let me provide a comprehensive solution addressing all three aspects:

Custom Field Deduplication Strategy: The fundamental issue is that your custom identifiers (partner_referral_code, campaign_tracking_id, customer_internal_id) are transactional tracking fields, not identity fields. They represent a lead’s interaction with your marketing channels, not the lead’s core identity. Mixing these two concepts in your duplicate detection logic creates the exact problem you’re experiencing.

Here’s how to restructure your matching rule configuration:

1. Primary Identity Fields: Configure your duplicate detection to use ONLY true identity fields:

• Email address (highest weight)
• Phone number (high weight)
• LinkedIn profile URL (medium weight, if you collect it)
• Company name + job title combination (low weight, as tie-breaker)

2. Exclude Tracking Fields from Duplicate Detection: Your custom identifiers should NOT be part of the duplicate matching rule. Navigate to Lead Management > Settings > Duplicate Detection Rules. Edit your existing rule or create a new one:

Rule Name: “Primary Lead Identity Matching”

Matching Logic:

IF (email_exact_match = true) OR
   (phone_normalized_match = true AND company_name_fuzzy_match = true)
THEN flag_as_duplicate = true

Notice that partner_referral_code, campaign_tracking_id, and customer_internal_id are completely absent from this rule. They should never prevent duplicate detection.

3. Store Tracking Data Separately: Instead of storing custom identifiers as fields on the Lead record, create a related “Lead Source Tracking” object or table. Each lead can have multiple tracking records:

Lead Record:

• Lead ID: L-00001
• Email: john.doe@example.com
• Phone: +1-555-0100

Related Lead Source Tracking Records:

• Tracking ID: T-00001, Lead ID: L-00001, Source: Partner A, partner_referral_code: PRTNR-A-12345, campaign_tracking_id: CAMP-2025-Q2, Date: 2025-06-15
• Tracking ID: T-00002, Lead ID: L-00001, Source: Partner B, partner_referral_code: PRTNR-B-67890, campaign_tracking_id: CAMP-2025-Q3, Date: 2025-06-18

This structure preserves ALL tracking information while preventing it from interfering with duplicate detection. You maintain complete attribution visibility without compromising data quality.

Matching Rule Configuration: Within AEC 2021 Lead Management, configure your duplicate detection with these specific settings:

1. Field-Level Matching Configuration: Navigate to Lead Management > Configuration > Matching Rules. For each identity field, set:

Email Address:

• Match Type: Exact (case-insensitive)
• Weight: 100
• Required: Yes
• Normalization: Lowercase, trim whitespace

Phone Number:

• Match Type: Normalized
• Weight: 80
• Required: No
• Normalization: Remove spaces, dashes, parentheses; add country code if missing

Company Name:

• Match Type: Fuzzy (85% similarity threshold)
• Weight: 30
• Required: No
• Normalization: Remove “Inc”, “LLC”, “Ltd” suffixes

2. Matching Threshold: Set your duplicate detection threshold to 100 points. This means:

• Email match alone = duplicate (100 points)
• Phone match + company match = duplicate (80 + 30 = 110 points)
• Phone match alone = not duplicate (80 points, below threshold)

This configuration ensures high confidence in duplicate detection while avoiding false positives.

3. Automated vs Manual Merge: For matches scoring exactly 100 points (email only), enable automatic merging. For matches scoring 100-150 points (phone + company), flag for manual review. This gives your sales ops team control over edge cases while automating clear duplicates.

Data Quality Implementation: To maintain data quality while handling multiple custom identifiers:

1. Lead Ingestion Workflow: Modify your lead capture process to separate identity data from tracking data at the point of entry. When a form is submitted:

Step 1: Extract identity fields (email, phone, company)

Step 2: Check for existing lead using matching rules

Step 3a: If duplicate found, update lead record’s “last_interaction” date

Step 3b: If duplicate found, CREATE NEW tracking record with current custom identifiers

Step 4a: If no duplicate, create new lead record

Step 4b: If no duplicate, create initial tracking record

This workflow ensures you never lose tracking attribution while maintaining clean lead deduplication.

2. Merge Policy for Custom Fields: Even with proper duplicate detection, you need a merge policy for when duplicates are identified. In Lead Management > Settings > Merge Policies, configure:

For Standard Fields (email, phone, company):

• Strategy: Keep most recent
• Conflict Resolution: Manual review if both records updated within 7 days

For Custom Identifier Fields (if you must keep them on Lead record):

• Strategy: Concatenate all unique values
• Format: “PRTNR-A-12345|PRTNR-B-67890” (pipe-separated)
• Maximum: Keep last 10 values

This preserves historical tracking while maintaining a single lead record.

3. Data Quality Monitoring: Set up automated data quality checks:

• Daily report: Leads created in last 24 hours with duplicate email/phone
• Weekly report: Lead records with >5 custom identifier values (suggests merge issues)
• Monthly audit: Sample 100 leads and verify tracking records properly linked

These reports help you catch duplicate detection failures quickly and refine your matching rules based on real patterns.

4. Lead Scoring Adjustment: If you use lead scoring, adjust your scoring model to account for multiple tracking records. A lead with multiple partner referral codes or campaign interactions should receive bonus points for multi-touch attribution, not be penalized as a potential duplicate.

Migration Plan for Existing Duplicates: You likely have existing duplicate leads in your system. Here’s how to clean them up:

1. Identify Existing Duplicates: Run a one-time analysis query to find leads with matching email/phone but different custom identifiers:

SELECT email, COUNT(*) as duplicate_count
FROM leads
GROUP BY email
HAVING COUNT(*) > 1

2. Extract Tracking Data: For each duplicate set, extract all custom identifier combinations into a temporary table before merging.

3. Merge Duplicates: Use Lead Management’s bulk merge functionality to consolidate duplicate leads, keeping the oldest lead record as primary (it likely has the most complete interaction history).

4. Create Tracking Records: For each merged lead, create individual Lead Source Tracking records for each unique custom identifier combination that was preserved.

5. Validation: After migration, verify that:

• Total lead count decreased by expected amount
• No tracking attribution was lost
• Lead scoring reflects combined interaction history

Long-term Recommendations: For sustainable data quality:

1. Standardize Lead Capture: Ensure all lead sources (web forms, partner portals, imports) follow the same identity + tracking separation pattern.

2. Partner Integration Guidelines: Provide partners with API documentation that clearly separates identity fields from tracking fields in the payload structure.

3. Regular Matching Rule Tuning: Quarterly, review your duplicate detection match rate and false positive rate. Adjust field weights and thresholds based on actual data patterns.

4. Consider Platform Upgrade: While you can solve this in AEC 2021, upgrading to AEC 2023 or later would give you access to the Identity Service, which handles complex identity graphs natively and would eliminate the need for custom matching rule configuration.

By separating identity management from attribution tracking and configuring matching rules to focus only on true identity fields, you’ll eliminate the duplicate lead creation issue while preserving complete multi-touch attribution data for your sales and marketing teams.

---

This draft is based on general Adobe Experience Cloud knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

[analyst_carmen]

The default duplicate detection in Lead Management uses exact matching on specified fields. If your custom identifier fields are different between the two submissions, the system won’t consider them duplicates even if email and phone match. You need to adjust your matching rule configuration to prioritize core identity fields (email, phone) over custom tracking fields. The custom identifiers should probably be stored as related records or in a separate tracking table rather than being part of the primary duplicate detection logic.

[adob_larsson]

I think your matching rules might be configured with AND logic when they should use OR logic for the custom fields. Check your duplicate detection settings-there’s usually an option for how multiple criteria are combined. If it’s set to “match ALL fields,” then different custom identifiers will prevent duplicate detection. Change it to “match ANY key identity field” and make sure email and phone are marked as key identity fields while the custom tracking fields are marked as supplementary.

[clark_ashley]

Another approach is to use identity resolution at the platform level rather than relying solely on Lead Management’s duplicate detection. Adobe Experience Platform has an Identity Service that can handle complex identity graphs with multiple identifiers. You’d map your custom identifiers to identity namespaces, and the Identity Service would recognize that different combinations of identifiers can belong to the same person. This requires some setup but provides more sophisticated duplicate detection than field-level matching rules.

“Tested this on Marketo with three custom identifier fields, and separating partner_referral_code into relationship data instead of identity fields eliminated our false duplicate alerts immediately.”

[deepak_singla]

The Identity Service approach sounds interesting, but we’re on AEC 2021 and I’m not sure if we have access to that level of identity resolution. Our current setup is fairly basic-just Lead Management with standard duplicate detection rules. Is there a way to configure the matching rules to handle this without requiring platform-level identity management? We need something that works within our current licensing and configuration.

[steve_wilson]

For AEC 2021 Lead Management, you can create custom matching rules that use weighted scoring instead of exact matching. Set up a rule where email address match = 100 points, phone match = 80 points, and each custom identifier match = 20 points. Then set your duplicate threshold to 100 points. This way, matching email OR phone will trigger duplicate detection regardless of whether the custom identifiers match. The custom identifiers add confidence but aren’t required for duplicate detection.

[patriciaad]

Don’t forget about the merge strategy for when duplicates are detected. Even if you fix the detection rules, you need to decide how to handle the custom identifier fields during merge. Should the system keep all custom identifiers from both records, or only from the primary record? In Lead Management settings, there’s a field merge policy where you can specify that certain fields (like your custom identifiers) should be aggregated rather than overwritten during duplicate merge operations.