Asset management depreciation calculation API batch job fail

ERP SAP S/4HANA
, , , , , , , ,
1

We’re running SAP S/4HANA 2020 and our monthly asset depreciation batch job through the REST API is failing halfway through processing around 15,000 assets. The job processes fine for the first 8,000-9,000 records then crashes without meaningful error logging. This is blocking our month-end close.

We need better error logging to identify which assets are causing failures, and some kind of checkpoint mechanism so we don’t have to restart from scratch each time. I’m also concerned about retry strategy - should failed records go to a dead letter queue or should we implement automatic retries?

Current approach just logs generic timeout errors:


ERR: Batch job timeout after 3600s
Failed at record: unknown
Stack trace: <null>

Has anyone implemented robust error handling for large batch API operations? We need this resolved before next month-end close in 12 days.

system 2

Let me provide a comprehensive solution that addresses all your requirements - error logging, checkpointing, retry strategy, and dead letter queue handling.

1. Error Logging Implementation: Replace your generic logging with structured record-level tracking. Create a batch processing log table:

BatchLog log = new BatchLog();
log.setBatchRunId(UUID.randomUUID());
log.setAssetNumber(asset.getNumber());
log.setStatus("PROCESSING");
log.setTimestamp(Instant.now());

Log before and after each asset processing attempt with full exception details including stack traces and asset-specific context.

2. Checkpoint/Restart Mechanism: Implement micro-batching with state persistence. Break your 15,000 assets into batches of 500 records. After each successful batch, commit a checkpoint:


CHECKPOINT_TABLE:
batch_run_id | checkpoint_position | last_processed_asset | timestamp

On restart, query the checkpoint table to resume from the last successful position rather than starting over. This alone will save you hours during month-end close.

3. Retry Strategy Configuration: Implement a tiered retry approach based on error types:

  • Transient errors (network timeout, temporary unavailability): Exponential backoff retry with delays of 5s, 15s, 45s (max 3 attempts)
  • Data validation errors (invalid asset number, missing required fields): No retry, immediate DLQ routing
  • Business logic errors (depreciation calculation conflicts): Single retry after 60s, then DLQ

Add retry metadata to your processing state:


retry_count | last_retry_timestamp | error_category | next_retry_time

4. Dead Letter Queue Setup: Establish a separate DLQ table or message queue for failed records:


DLQ_TABLE:
record_id | asset_number | error_message | error_category |
original_payload | failed_timestamp | retry_count

Critical: Implement a DLQ processing workflow:

  • Daily review of DLQ contents by operations team
  • Root cause analysis for patterns (are specific asset classes failing?)
  • Manual correction capability for data issues
  • Bulk resubmission after fixes

5. Additional Recommendations:

Connection Pool Tuning: Increase your database connection pool settings:


maxActive=100
maxIdle=50
maxWait=30000

Parallel Processing: Consider processing multiple batches in parallel (4-6 threads) to improve throughput while maintaining checkpointing per thread.

Monitoring Dashboard: Create real-time visibility:

  • Total records processed vs remaining
  • Current processing rate (records/minute)
  • Error rate percentage
  • Estimated completion time
  • DLQ depth

Testing Strategy: Before production deployment:

  1. Test with 1,000 asset sample with injected failures
  2. Verify checkpoint recovery works correctly
  3. Confirm DLQ routing for different error types
  4. Load test with 20,000 assets to validate scalability

Month-End Close Impact: With this implementation, even if you encounter failures, you’ll have:

  • Clear visibility into which assets failed and why
  • Ability to resume processing without full restart
  • Automated retry handling for transient issues
  • Organized queue of records needing manual attention

This should reduce your month-end close risk significantly and provide operational confidence. The initial implementation will take 3-4 days but will pay dividends in reduced firefighting during critical close periods.

Start with the logging and checkpointing first - those provide immediate value. Then layer in the retry logic and DLQ handling. Let me know if you need help with specific implementation details.

This draft is based on general SAP S/4HANA knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.

3

I’ve dealt with similar batch processing issues. First thing - that generic error logging is your biggest problem. You need structured logging with record-level tracking. Implement a correlation ID for each batch run and log every asset number being processed. Without knowing which specific assets fail, you’re flying blind and will keep hitting the same wall.

sap_watson 4

Tested this on S/4HANA 2023 FPS01 — the BatchLog UUID tracking immediately pinpointed which asset numbers were causing our depreciation calculation API timeouts during month-end batch runs.

5

The 8,000-9,000 record threshold suggests memory accumulation or connection pool exhaustion rather than data issues. Are you processing all 15,000 assets in a single API call? That’s asking for trouble. Break it into smaller batches of 500-1,000 records with checkpointing between batches. Each batch should commit independently so failures don’t cascade. Also check your database connection pool settings - you might be hitting maxActive limits. What’s your current batch size and do you have connection pooling configured properly?

7

We’re sending everything in one massive batch - didn’t realize that was problematic. Connection pool is default settings, probably need to tune that. The checkpoint idea makes sense but how do we track processing state? Do we need a separate database table to store which assets have been processed successfully?

8

Yes, absolutely maintain a processing state table. Structure it with batch_run_id, asset_number, status (pending/processing/completed/failed), error_message, timestamp columns. Before processing each mini-batch, mark records as ‘processing’, then update to ‘completed’ or ‘failed’ based on results. This gives you both checkpoint capability and audit trail. For retry strategy, I’d recommend exponential backoff for transient errors (network, timeout) but immediate dead letter queue for data validation errors. You don’t want to waste cycles retrying malformed data. Implement a retry_count column with a max threshold of 3 attempts.

9

Don’t forget monitoring and alerting. Set up CloudWatch or equivalent to track batch progress metrics - records processed per minute, error rates, queue depths. You want visibility into whether jobs are slowing down before they fail completely. Also implement health check endpoints in your API that report processing status. For the dead letter queue, make sure you have a separate process to review and reprocess those failed records after root cause analysis.

10

I want to add context about the timeout you’re seeing. 3600 seconds for 15,000 records means you’re averaging about 4 records per second, which is actually quite slow for depreciation calculations. This suggests either network latency between your API client and S/4HANA, or inefficient queries on the backend. Before implementing all the retry logic, profile your API calls to identify bottlenecks. You might have N+1 query problems or missing database indexes.