Your batch job is stuck due to a combination of factors involving the scheduler, locked records, and insufficient debug logging. Let me address each focus area:
Batch Job Scheduler:
The ICS 2023-1 batch scheduler has a known issue where jobs can enter a zombie state if the scheduler service experiences a brief network interruption or memory pressure event. The job process continues running, but it loses its connection to the scheduler coordination service. This explains why you see ‘PROCESSING’ status with 0% progress and near-zero resource usage.
To diagnose scheduler connectivity:
- Check scheduler service health: Navigate to Admin → System Monitor → Scheduler Service Status
- Look for ‘Orphaned Jobs’ count - if non-zero, you have disconnected job processes
- Review scheduler service logs at
/logs/scheduler-service.log for connection reset events around your job start time
The safe recovery procedure:
- Mark the job as failed in the scheduler: `scheduler-admin --job-id --force-fail
- This releases the job from the scheduler’s active queue without killing the process
- The job process will detect the status change and terminate gracefully within 5 minutes
- Any acquired locks will be released through normal transaction rollback
Locked Records:
Your observation about 12 database sessions with locks belonging to the job itself indicates the job successfully acquired locks on the first batch of lease records but then stalled before processing them. This is characteristic of a resource deadlock or external dependency failure.
To identify the blocking resource:
SELECT s.sid, s.serial#, s.wait_class, s.event, s.seconds_in_wait,
l.object_id, o.object_name, l.locked_mode
FROM v$session s
JOIN v$lock l ON s.sid = l.sid
JOIN dba_objects o ON l.object_id = o.object_id
WHERE s.program LIKE '%LeaseRenewal%'
ORDER BY s.seconds_in_wait DESC;
If the wait_event shows ‘SQLNet message from client’ or 'SQLNet more data from client’, the job is waiting for application-layer processing, not database resources. This suggests the batch job code has a deadlock or infinite loop.
Safe lock release procedure:
- Identify the session IDs from the query above
- For each session, check if it has uncommitted work: `SELECT USED_UREC FROM V$TRANSACTION WHERE ADDR IN (SELECT TADDR FROM V$SESSION WHERE SID = )
- If USED_UREC > 0, the session has uncommitted changes that will be rolled back
- Kill the sessions: `ALTER SYSTEM KILL SESSION ‘,<serial#>’ IMMEDIATE;
- Monitor rollback progress: Check
V$FAST_START_TRANSACTIONS for recovery status
Data corruption risk is minimal because ICS batch jobs use ACID-compliant transactions. When you kill the sessions, Oracle will automatically roll back any uncommitted work, leaving your lease data in a consistent state.
Debug Logging:
The absence of error logs indicates your batch job logging configuration is insufficient for troubleshooting. ICS 2023-1 batch jobs have multiple logging levels that must be enabled separately.
Enable comprehensive batch job logging:
- Edit
batch-job-config.properties:
batch.job.logging.level=DEBUG
batch.job.progress.reporting=true
batch.job.checkpoint.logging=true
batch.job.transaction.trace=true
batch.job.exception.stacktrace=true
- Enable lease-specific debug logging:
lease.renewal.debug=true
lease.renewal.record.level.trace=true
- Configure separate log file for batch jobs:
batch.job.log.file=/logs/batch-jobs/lease-renewal.log
batch.job.log.rotation=daily
batch.job.log.retention=30
- Restart the batch job service to apply changes
With debug logging enabled, you’ll see detailed output including:
- Each lease record being processed
- Transaction commit/rollback points
- External API calls (if any)
- Lock acquisition and release events
- Progress percentage calculations
- Any exceptions or warnings, even if they’re caught and handled
For immediate diagnosis of your current stuck job, you can enable runtime logging without restart:
scheduler-admin --job-id <ID> --set-log-level DEBUG
This will start generating logs for the running job (if it’s actually processing), or confirm it’s completely stalled (if no new log entries appear).
Recommended Resolution:
- Enable debug logging configuration first (for future jobs)
- Use the scheduler force-fail command to safely terminate the stuck job
- Wait for database rollback to complete (monitor V$FAST_START_TRANSACTIONS)
- Verify all locks are released: Check v$lock for your lease tables
- Investigate scheduler service logs for the root cause of the disconnect
- Restart the lease renewal job with debug logging enabled
- Monitor the new job execution with detailed logs to ensure it progresses normally
The underlying issue is likely scheduler service instability. Check for memory pressure, network issues, or service restarts around the time your job started. Consider implementing job timeouts (recommended: 6 hours for your 340-lease workload) to prevent indefinite stuck states in the future.
This draft is based on general Infor CloudSuite knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.