We’re experiencing a persistent issue with our Aras 14.0 cloud deployment where validation jobs submitted through the validation management module get stuck in ‘Running’ state and never complete. This is blocking our entire validation workflow.
The problem started after we scaled up our validation workload last week. Jobs that previously completed in 10-15 minutes now just hang indefinitely. We’ve checked the job queue management interface and can see multiple jobs piling up, all showing ‘Running’ status. We suspect it might be related to cloud resource quotas being exceeded, but we’re not sure how to verify this or what the actual limits are for our subscription tier.
We’ve tried canceling and resubmitting jobs, but they immediately return to the stuck state. The validation service pods appear to be running normally in the cloud console, but something is clearly preventing job completion. Has anyone encountered similar issues with cloud-based validation processing? Any guidance on troubleshooting job queue bottlenecks or checking resource allocation would be greatly appreciated.
I ran into this exact scenario three months ago. The stuck jobs issue typically stems from a combination of factors that need systematic resolution.
Job Queue Management Investigation:
First, verify your job queue health by checking the cloud admin portal under Monitoring > Job Queues. Look for these specific metrics: queue depth (should be under 50 pending jobs), average processing time (baseline this against historical data), and job failure rate. If queue depth exceeds 100, you’re experiencing queue saturation. Access the job details API to identify if specific validation types are consuming disproportionate resources.
Cloud Resource Quotas Analysis:
Your 87% CPU and 92% memory utilization are critical thresholds. Cloud platforms typically throttle at 90% to prevent instability. Request a quota increase through your Aras cloud support portal - specify current limits and desired capacity (I’d recommend 50% increase initially). While waiting for approval, implement these immediate mitigations:
Reduce concurrent validation jobs to 4 (Settings > Validation > Concurrency Limit)
Enable job prioritization to process critical validations first
Set job timeout to 30 minutes to prevent indefinite hangs
Service Pod Restart Strategy:
The pod restarts you’re seeing indicate memory pressure. Execute a controlled restart of validation service pods during off-peak hours:
In cloud console: Services > Validation Management > Actions > Rolling Restart
This clears memory leaks and resets stuck job handlers
Schedule this weekly until root cause is resolved
Implement pod autoscaling rules: scale up when memory exceeds 75% for 5 minutes, scale down when below 50% for 15 minutes. Configure health checks to automatically restart pods that haven’t processed jobs in 20 minutes.
Long-term Resolution:
After implementing these fixes, monitor for 2-3 days. If issues persist, the validation logic itself may need optimization. Review validation methods for inefficient queries, excessive object loading, or missing result caching. We reduced our validation memory footprint by 65% by implementing result caching and lazy loading patterns.
Also verify your cloud subscription tier supports your workload volume. Enterprise tier provides better resource guarantees and burst capacity compared to Standard tier.
This draft is based on general Aras Innovator knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.
I’ve seen similar behavior with cloud validation jobs. First thing to check is your subscription’s compute quota limits. Navigate to the cloud admin console and look under Resource Management > Quotas. You should see current usage vs limits for CPU, memory, and concurrent jobs. If you’re hitting 90%+ on any of these, that’s likely your bottleneck causing jobs to queue but not execute.
Check the validation service logs specifically. In our cloud deployment, we discovered that stuck jobs were actually failing silently due to insufficient memory allocation per pod. The jobs showed ‘Running’ but were repeatedly restarting. You can access logs through the cloud console under Services > Validation Management > View Logs. Look for OutOfMemory errors or pod restart events around the time jobs get stuck.
Thanks for the suggestions. I checked the quota dashboard and we’re at 87% CPU utilization and 92% memory usage. The validation service logs show several pod restarts with memory pressure warnings. It looks like we need to either increase our resource allocation or optimize the validation jobs themselves. Are there configuration options to limit concurrent validation jobs or adjust memory per job?
For immediate relief, you can adjust concurrent job limits in the cloud configuration. Go to Settings > Validation Management > Resource Controls. Lower the max concurrent validations from default (usually 10) to something like 5-6 until you upgrade resources. This prevents resource exhaustion. Also check if your validation rules are loading large datasets into memory unnecessarily. We reduced memory usage by 40% by optimizing query filters in our validation methods.
Beyond the immediate fixes, consider implementing validation job batching. Instead of validating items individually, group them into batches of 50-100 items per job. This significantly reduces overhead and pod startup costs in cloud environments. We implemented this pattern and cut our validation processing time by 60% while using fewer resources. The trade-off is slightly less granular progress tracking, but the performance gains are worth it.