Automated contact enrichment pipeline using Launch events and third-party APIs to eliminate manual data entry

CRM Adobe Experience Cloud
, , , , , , , ,
1

I wanted to share our implementation of an automated contact enrichment pipeline that reduced manual data entry by 80% and improved contact data quality scores from 62% to 94% over six months.

We built this on AEC 2022 using Launch event capture to trigger enrichment workflows whenever a new contact is created or updated. The system routes events to third-party APIs (Clearbit, ZoomInfo, LinkedIn) for data enrichment, processes responses asynchronously, and updates contact records with deduplication logic.

The architecture handles 2,000+ contact updates daily with minimal manual intervention. Key components include Launch event listeners, async job processing queues, error handling with automatic retries, and intelligent deduplication to prevent data conflicts. I’ll share the technical implementation details and lessons learned from running this in production for eight months.

2

The deduplication logic is what interests me most. When you get data back from multiple third-party APIs (Clearbit, ZoomInfo, LinkedIn), how do you decide which data source to trust when they provide conflicting information? Do you have a hierarchy of source reliability, or do you use some kind of confidence scoring?

3

Good question. We use Launch’s custom event rules with field-level change detection. The event only fires when specific ‘enrichment trigger’ fields are modified - things like email address, company name, or job title. We also implement a cooldown period: once a contact is enriched, we don’t re-enrich for 90 days unless a trigger field changes. This reduced our API calls by 75% compared to enriching on every update.

The Launch rule looks like:

if (contactUpdate.changedFields.includes('email') ||
    contactUpdate.changedFields.includes('company')) {
  _satellite.track('contactEnrichment', contactData);
}
4

This is a fantastic implementation. Can you provide more detail on the complete architecture? I’d love to understand the full data flow from Launch event capture through to final contact record updates, including how you handle the deduplication and error handling components you’ve mentioned.

5

Absolutely. Here’s the complete technical implementation that achieved our 80% reduction in manual data entry:

Launch Event Capture and Routing:

We configured Launch with custom event rules that monitor contact creation and updates. The key is field-level change detection to avoid unnecessary enrichment:

// Launch custom code - contact change listener
function shouldEnrich(contact, changedFields) {
  const triggers = ['email', 'company', 'jobTitle', 'phone'];
  const hasRelevantChange = changedFields.some(f => triggers.includes(f));
  const lastEnriched = contact.lastEnrichmentDate;
  const cooldownPeriod = 90 * 24 * 60 * 60 * 1000; // 90 days

  return hasRelevantChange &&
         (!lastEnriched || Date.now() - lastEnriched > cooldownPeriod);
}

When enrichment is triggered, Launch publishes an event to our enrichment API gateway, which validates the request and publishes to AWS SQS. This decouples Launch from the processing pipeline and provides resilience.

Third-Party API Integration:

We built a Node.js worker service that polls SQS and orchestrates API calls. The service runs 5 concurrent worker threads to handle our 2,000+ daily volume:

// Pseudocode - API orchestration flow:
1. Retrieve contact enrichment job from SQS queue
2. Load contact data from AEC API (current state)
3. Call enrichment APIs in parallel with 15s timeout:
   - Clearbit Company API (company data)
   - ZoomInfo Person API (contact details)
   - LinkedIn Profile API (employment history)
4. Collect responses and handle errors per retry policy
5. Apply deduplication logic to merge responses
6. Update contact record via AEC API
7. Delete message from SQS queue (success) or requeue (retry)
// See AWS Lambda documentation for async patterns

Each API call includes circuit breaker logic - if an API fails 10+ times in 5 minutes, we temporarily stop calling it and alert our ops team.

Async Job Processing:

The async architecture is critical for handling API latency and failures gracefully:

  1. Job Queue (AWS SQS): Holds enrichment requests with 24-hour visibility timeout
  2. Worker Pool: 5 EC2 instances running Node.js workers (auto-scales based on queue depth)
  3. State Management: DynamoDB table tracking enrichment status per contact
  4. Dead Letter Queue: Failed jobs after 3 retries move here for manual investigation
  5. Monitoring: CloudWatch dashboards showing queue depth, processing rate, error rate, API latency

The worker service updates contact status in real-time:

  • Job received: Status = ‘pending’
  • API calls started: Status = ‘in_progress’
  • All APIs responded: Status = ‘merging_data’
  • Contact updated: Status = ‘completed’
  • Any failure: Status = ‘failed’ with error details

Deduplication and Error Handling:

The deduplication engine is the most sophisticated component. When multiple APIs return data for the same field, we apply this decision logic:

// Confidence scoring pseudocode:
1. Load field reliability weights from config:
   reliabilityScores = {
     clearbit: {company: 0.95, jobTitle: 0.78},
     zoominfo: {jobTitle: 0.91, phone: 0.85},
     linkedin: {employment: 0.88, company: 0.72}
   }
2. For each conflicting field value:
   score = reliabilityScore * freshnessScore * completenessScore
3. Select value with highest confidence score
4. Log decision to audit table for review
5. If scores within 5% margin: flag for human review
// See deduplication algorithm documentation

Freshness score: Data updated within 30 days = 1.0, 30-90 days = 0.8, 90+ days = 0.5

Completeness score: All required subfields present = 1.0, partial = 0.7, minimal = 0.4

Error handling implements multiple fallback layers:

  1. Field-level fallback: If primary API fails, use secondary API for that field
  2. Partial enrichment: Update fields we successfully enriched, mark others as ‘pending_retry’
  3. Manual queue: Contacts with 3+ failed enrichment attempts go to manual review queue
  4. Data validation: Run schema validation before updating contact (prevent bad data from entering CRM)
  5. Rollback capability: Maintain pre-enrichment snapshot for 7 days in case of data quality issues

Contact Record Updates:

After merging and deduplicating API responses, we update the contact record using AEC’s bulk API to minimize API calls:

  1. Batch updates: Accumulate 50 contacts before calling bulk update API
  2. Field mapping: Transform API response format to AEC field schema
  3. Audit trail: Create enrichment history record showing which fields changed and data sources
  4. Trigger suppression: Temporarily disable workflow triggers during enrichment update to prevent loops
  5. Verification: Query updated contact to confirm changes persisted correctly

We also implemented a reconciliation job that runs nightly to catch any contacts stuck in ‘in_progress’ status for > 4 hours and retry them.

Results and Key Metrics:

After 8 months in production:

  • Manual data entry reduced from 400+ hours/month to 80 hours/month (80% reduction)
  • Contact data quality score improved from 62% to 94%
  • Average enrichment time: 45 seconds (from event trigger to contact updated)
  • Success rate: 96.5% (3.5% require manual intervention)
  • API cost: $0.12 per enriched contact (blend of all three APIs)
  • Processing capacity: 2,000+ contacts/day with current infrastructure

Lessons Learned:

  1. Start with cooldown periods: Our initial implementation enriched too frequently, wasting API credits on unchanged data
  2. Invest in monitoring: We added comprehensive dashboards after missing a 3-day API outage that created a huge backlog
  3. Manual review queue is essential: Some edge cases (executives, uncommon names, international contacts) need human judgment
  4. API reliability varies by field type: Don’t assume one API is universally better - test per field type
  5. Batch processing saves money: Accumulating updates before calling AEC API reduced our API usage by 60%

Implementation Timeline:

Week 1-2: Launch event capture setup and SQS integration

Week 3-4: Third-party API integration (one API per week)

Week 5-6: Deduplication logic and confidence scoring

Week 7-8: Error handling, monitoring, and manual review queue

Week 9-10: Load testing and production rollout (10% of contacts)

Week 11-12: Full production rollout and optimization

Total implementation: 12 weeks with 2 developers and 1 data analyst

The system has been running reliably for 8 months with minimal maintenance. We adjust API reliability scores quarterly based on data quality audits, and we’ve added two additional enrichment APIs (Hunter.io for email verification, Lusha for mobile numbers) using the same architecture pattern.

6

How do you handle the async job processing and error scenarios? Third-party APIs can be unreliable - timeouts, rate limits, invalid responses. What’s your retry strategy, and how do you ensure contact records don’t get stuck in a partially enriched state?

7

We use a multi-stage async processing pipeline with AWS SQS as the job queue. When Launch fires the enrichment event, it publishes a message to SQS with the contact ID and trigger fields. Worker processes poll the queue and call the third-party APIs with a 15-second timeout per API.

Retry strategy:

  • Timeout or 5xx error: Retry up to 3 times with exponential backoff (2s, 8s, 32s)
  • 429 rate limit: Exponential backoff starting at 60s
  • 4xx error (bad data): Move to dead letter queue for manual review
  • After 3 failed retries: Mark contact with ‘enrichment_failed’ flag and alert operations team

We use a state machine pattern where each contact has an enrichment status field (pending/in_progress/completed/failed) to prevent partial updates.

8

This sounds like exactly what we need to implement. Can you share more details about the Launch event capture setup? Specifically, how do you distinguish between contact updates that should trigger enrichment versus routine updates that shouldn’t? We don’t want to call expensive third-party APIs for every minor field change.

9

We use a weighted confidence scoring system based on six months of data quality analysis. Each API gets a reliability score per field type:

Clearbit: 95% accurate for company data, 78% for job titles

ZoomInfo: 91% accurate for job titles, 85% for phone numbers

LinkedIn: 88% accurate for current employment, 72% for company data

When APIs return conflicting data, we calculate a confidence score for each value based on source reliability and data freshness. The highest scoring value wins. We also maintain a change log showing which source provided each field value, so data quality teams can audit decisions and adjust weights over time.