Contract renewal batch jobs spike CPU and slow down other processes

CRM Adobe Experience Cloud
, , , , , , , ,
1

Our nightly contract renewal batch jobs in AEC 2022 are causing severe CPU spikes that impact other system processes. The job runs at 2 AM and processes about 15,000 contract records to check renewal dates, update statuses, and trigger notification workflows. During execution, CPU usage jumps to 85-95% and stays there for 90-120 minutes.

This causes major problems: scheduled reports time out, integration sync jobs fail, and morning users experience slow system response. The batch job efficiency seems poor - it’s processing contracts sequentially and making individual database calls for each record. Job concurrency settings appear to be default values.

We need to optimize this process to reduce CPU impact and execution time. What’s the best approach for tuning batch job performance and database query efficiency without compromising data accuracy?

system 2

I’ll provide a comprehensive optimization strategy addressing all three focus areas:

Batch Job Optimization Architecture: Redesign your batch job from sequential single-threaded processing to a parallel chunk-based architecture. Divide the 15,000 contracts into logical chunks of 1,000 records each, creating 15 processing units. Configure the batch framework to execute 4-6 chunks concurrently using separate worker threads.

Implement a coordinator pattern where a master thread queries and partitions the work, then delegates to worker threads. Each worker processes its chunk independently with its own database connection from a connection pool. Critical: ensure chunks are partitioned by contract_id ranges or hash distribution to prevent lock contention on the same records.

Add checkpoint/restart capability. Store progress after each chunk completes so if the job fails, it can resume from the last checkpoint rather than restarting entirely. This also enables you to set a maximum execution time (e.g., 60 minutes) and resume the next night if needed, preventing indefinite CPU monopolization.

Database Query Efficiency: Your current approach of individual queries per contract is creating 15,000+ database round trips. Implement bulk query patterns:

Initial data fetch: Single query with optimized WHERE clause to retrieve all contracts where renewal_date is within processing window. Use pagination (FETCH FIRST 1000 ROWS) to load chunks rather than the entire result set.

Create composite indexes specifically for the batch job:

  • Index on (renewal_date, status, contract_id) for the initial fetch query
  • Index on (contract_id, last_modified_date) for update conflict detection

Bulk updates: After processing each chunk in memory, use batch UPDATE statements that modify multiple records in a single database call. Prepare a statement like:

UPDATE contracts SET status = ?, renewal_date = ?, last_modified = ? WHERE contract_id IN (?, ?, …)

Execute with batches of 500 IDs at a time. This reduces 1,000 individual updates to just 2 bulk operations.

Implement read-only query optimization: For contracts that don’t need updates (already processed, not yet due), use read-only transactions with lower isolation levels to reduce locking overhead.

Connection pool tuning: Configure dedicated connection pool for batch jobs with minimum 8 connections, maximum 16. This ensures batch workers don’t starve other application processes of database connections.

Job Concurrency Settings: Configure proper resource limits and scheduling:

Thread pool configuration:

  • Core worker threads: 4 (for 4-core system) or 6 (for 8+ core system)
  • Maximum thread pool size: 8 (allows burst capacity)
  • Queue capacity: 50 (queues additional chunks if all workers busy)
  • Thread timeout: 5 minutes (releases idle threads)

CPU throttling: Implement adaptive throttling where the job monitors system CPU usage and scales back parallelism if total CPU exceeds 70%. If CPU > 70%, reduce active workers from 6 to 3. If CPU > 85%, pause processing for 30 seconds.

Memory management: Configure heap size appropriately for batch processing. Allocate 2-4GB heap for the batch job process, separate from main application heap. Implement aggressive garbage collection after each chunk completes to prevent memory buildup.

Scheduling optimization: Move batch execution to 11:30 PM and implement job chaining with dependencies. Schedule integration sync jobs to run AFTER contract batch completes, not concurrently. Use job scheduler’s dependency features to create execution order.

Implement async workflow triggering: Instead of triggering notification workflows synchronously during contract processing, write renewal events to a message queue. Configure separate workflow processor that consumes from the queue at a controlled rate (e.g., 100 notifications/minute) to spread the load.

Monitoring and Tuning: Add comprehensive metrics:

  • Records processed per minute (target: 200-250)
  • Average chunk processing time (target: < 3 minutes per 1000 records)
  • Database query execution time per chunk (target: < 500ms)
  • CPU utilization by worker thread
  • Database connection pool utilization

Set up alerts if processing rate drops below 150 records/minute or CPU exceeds 75% for more than 15 minutes.

With these optimizations, your 15,000 contract batch should complete in 35-45 minutes with CPU usage averaging 45-55%, leaving adequate headroom for other system processes. The parallel architecture with bulk operations will deliver 3-4x performance improvement while maintaining data integrity.

This draft is based on general Adobe Experience Cloud knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

Sequential processing of 15K records with individual database calls is definitely your problem. That’s 15,000+ separate queries hitting the database. You need to implement bulk operations. Fetch all contracts needing renewal in a single query with proper WHERE clause filtering, process them in memory in batches of 500-1000, then update in bulk. This alone will reduce your execution time by 70-80%.

4

Check your batch job thread configuration. Default single-threaded execution is killing you. AEC 2022 supports parallel batch processing - configure 4-6 worker threads to process contract batches concurrently. Make sure each thread works on a distinct subset of records to avoid lock contention. Also, your notification workflow triggers might be synchronous, causing the batch job to wait for each notification to complete. Switch to asynchronous notification queuing so the batch job can continue processing while notifications are handled separately.

5

The CPU spike pattern suggests you’re probably doing heavy computation or complex business logic for each contract. Are you recalculating renewal terms, pricing, or running validation rules during the batch? Consider pre-computing as much as possible. For example, renewal dates and terms should be calculated when contracts are created or modified, not during the batch run. The batch job should just be checking dates and updating statuses, not performing complex calculations.

ishaan_adob 6

“Tested this on AEM’s Sling Job framework — splitting 15,000 contracts into 1,000-record chunks with 4 concurrent Sling Job Queue threads cut our CPU spikes by 60%.”

7

Your timing is also problematic. Running at 2 AM might conflict with other scheduled jobs. We moved our contract batch to 11 PM and staggered other batch jobs with 30-minute offsets. This reduced resource contention significantly. Also implement CPU throttling for batch jobs - configure a maximum CPU utilization threshold so the job scales back when system load is high.

8

Look at your database indexes. If the query to fetch contracts needing renewal is doing table scans, that explains the CPU usage. Create composite indexes on renewal_date and status columns. Also check transaction isolation levels - if you’re using serializable isolation for the entire batch, you’re causing massive lock overhead. Use read committed isolation and implement optimistic locking for updates instead.

9

The workflow triggering is likely a major bottleneck. Each contract renewal probably triggers multiple workflows (notifications, approval routing, etc.). If these are executed synchronously within the batch transaction, you’re serializing everything. Implement an event queue pattern where the batch job publishes contract renewal events to a queue, and separate worker processes consume those events asynchronously to handle workflows. This decouples batch processing from workflow execution.