Our consolidation data sync jobs are consistently failing when they run during the scheduled maintenance window (Saturday 2-6 AM UTC). The jobs pull financial data from three subsidiary entities and consolidate into the parent company structure.
The error logs show connection timeout errors after approximately 90 seconds:
ERROR: Connection timeout at ConsolidationSync.executeDataPull()
Entity: SUBSIDIARY-APAC
Timeout: 90000ms exceeded
Status: FAILED_PARTIAL
During normal business hours, the same sync completes in 45-60 seconds without issues. We suspect the maintenance window involves backend processes that slow down API responses, but we need the sync to run during off-hours to avoid impacting users.
Is there a way to configure longer timeout values for sync jobs? Or should we be scheduling these jobs to run after the maintenance window completes? The problem is our reporting deadline requires consolidated data by Monday morning, and we need that Saturday processing time.
Where do we configure the timeout settings? I’ve looked in the consolidation module settings but don’t see a specific timeout parameter for sync jobs. Is this something that needs to be set at the API client level in our integration code?
You can request maintenance window changes, but Infor typically only accommodates this for enterprise-tier customers, and it takes 30-60 days to implement. A better approach is implementing retry logic with exponential backoff. If the sync fails during maintenance, automatically retry every 15 minutes for up to 2 hours. This way you catch the window as soon as maintenance completes without manual intervention.
The challenge is that our subsidiary in APAC needs to close their books by Friday evening their time (which is Saturday morning UTC), and we need consolidated results ready for Monday morning executive review. If we wait until after maintenance, we lose critical processing time. Can we request a different maintenance window for our tenant?
I’ll provide a comprehensive solution addressing all three critical aspects of your synchronization challenge:
Maintenance Window Awareness:
CloudSuite maintenance windows are published in advance but require proactive monitoring. Here’s how to work with them effectively:
-
Access Maintenance Schedule:
- Admin Portal > Tenant Information > Maintenance Calendar
- Subscribe to maintenance notifications (Settings > Notifications > System Maintenance)
- Maintenance typically occurs weekly: Saturdays 02:00-06:00 UTC for most regions
-
Maintenance Impact on APIs:
During maintenance, CloudSuite performs:
- Database index optimization (causes query slowdowns)
- Cache rebuilding (affects data retrieval performance)
- Security patches (may cause brief API unavailability)
- Log rotation (impacts audit trail queries)
API response times can increase 3-5x during these operations. Your 45-second normal sync extending to 90+ seconds aligns with this pattern.
-
Working Around Maintenance:
Instead of fighting the maintenance window, implement a maintenance-aware scheduling strategy:
if (currentTime in maintenanceWindow) {
schedule.delay(maintenanceEndTime + 30min);
} else {
execute.syncJob();
}
Sync Job Scheduling Optimization:
Redesign your consolidation workflow to accommodate the maintenance window:
-
Split Your Process:
- Phase 1 (Friday 23:00 UTC): Pre-pull subsidiary data to staging tables
- Phase 2 (Saturday 06:30 UTC): Execute consolidation calculations
- Phase 3 (Saturday 07:00 UTC): Finalize and publish consolidated reports
-
Update Job Configuration:
Navigate to: Consolidation > Administration > Sync Jobs
Modify your job schedule:
- Primary Schedule: Saturday 06:30 UTC (after maintenance)
- Backup Schedule: Saturday 08:00 UTC (if primary fails)
- Emergency Schedule: Sunday 00:00 UTC (final fallback)
-
Configure Job Dependencies:
Set up job chaining so Phase 2 only starts after Phase 1 completes successfully:
Job: ConsolidationCalc
Trigger: OnSuccess(DataPull_APAC, DataPull_EMEA, DataPull_AMER)
Schedule: Dependent (no fixed time)
Retry Logic for Timeouts:
Implement intelligent retry mechanisms to handle transient failures:
-
Extend Timeout Values:
In your sync job configuration (Consolidation > Sync Jobs > Advanced):
connection.timeout=180000
read.timeout=240000
socket.timeout=180000
This gives 3-4 minutes per operation, sufficient for maintenance-window slowdowns.
-
Implement Exponential Backoff:
Add retry logic to your sync job script:
int maxRetries = 5;
int baseDelay = 300000; // 5 minutes
for (int attempt = 1; attempt <= maxRetries; attempt++) {
try {
executeSyncJob();
break; // Success
} catch (TimeoutException e) {
if (attempt == maxRetries) throw e;
Thread.sleep(baseDelay * Math.pow(2, attempt-1));
}
}
This retries with delays of 5min, 10min, 20min, 40min, 80min - spanning the typical maintenance window.
3. **Partial Sync Recovery:**
Modify your sync to support checkpoint/resume:
if (syncStatus == FAILED_PARTIAL) {
resume(lastSuccessfulEntity);
}
This prevents re-syncing entities that completed successfully (APAC and EMEA) when only AMER failed.
**Immediate Action Plan:**
1. **This Week (Emergency Fix):**
- Increase timeout to 240000ms
- Add basic retry (3 attempts, 10min intervals)
- Monitor Saturday execution
2. **Next Week (Optimization):**
- Implement data pre-pull on Friday evening
- Reschedule consolidation to 06:30 UTC Saturday
- Add maintenance window detection logic
3. **Long-term (Robust Solution):**
- Implement full exponential backoff retry
- Add checkpoint/resume capability
- Set up maintenance calendar monitoring
- Configure automatic notifications on sync failures
**Monitoring and Alerting:**
Set up proactive monitoring:
- CloudWatch/Splunk integration for sync job status
- Email alerts on timeout failures
- Slack/Teams webhook for real-time notifications
- Dashboard showing sync duration trends (identify maintenance impact)
By implementing maintenance-aware scheduling combined with robust retry logic, you'll achieve reliable consolidation processing while accommodating CloudSuite's operational requirements. The Friday pre-pull strategy is particularly effective as it front-loads the data extraction before maintenance begins, leaving only the calculation phase for Saturday morning.
Maintenance windows definitely affect API performance. The system is doing index rebuilds, cache refreshes, and other housekeeping tasks. Your 90-second timeout is probably too aggressive for this scenario. Check if you can extend it to 300 seconds (5 minutes) in your sync job configuration.
Rather than fighting the maintenance window, work around it. Infor publishes the exact maintenance schedule for each tenant - check your tenant’s maintenance calendar in the admin portal. Typically it’s 2-4 AM UTC for 2-3 hours. Schedule your sync jobs to start at 6:30 AM UTC, giving 30 minutes buffer after maintenance completes. This ensures the system is fully operational and responsive when your jobs run.