Here’s a comprehensive approach to resolve your work order locking issues:
1. Optimize Database Lock Settings
Update your PostgreSQL configuration to handle concurrent manufacturing operations better. In postgresql.conf, adjust these settings:
- lock_timeout = 10s (fail fast instead of queuing)
- deadlock_timeout = 1s (detect deadlocks quickly)
- max_locks_per_transaction = 128 (increase from default 64)
Restart PostgreSQL after making these changes.
2. Refactor Automated Actions
Identify all automated actions triggered by work order status changes. Move non-critical operations to scheduled actions:
- Dashboard metric updates: Run every 2 minutes via scheduled action
- Notification emails: Queue them and send in batches
- Reporting calculations: Defer to nightly batch processing
Keep only essential operations (inventory moves, quality check creation) in the real-time transaction.
3. Implement Optimistic Locking Pattern
For work order status updates, reduce lock scope by updating only the specific work order record first, then handle related records in a separate transaction:
- Update work order status (quick, minimal locking)
- Commit transaction immediately
- Process related updates (stock moves, parent MO) in subsequent transaction
- Use try-except blocks with retry logic for conflicts
4. Add Database Indexes for Concurrent Access
Ensure you have indexes on frequently locked columns to speed up lock acquisition. Work orders are typically queried by production_id and state during updates, so verify these indexes exist.
5. Implement Client-Side Retry Logic
On your shop floor tablets, implement exponential backoff retry logic for status updates. If a lock timeout occurs, wait 1-2 seconds and retry. This prevents the lock convoy effect where multiple tablets keep hammering the database with failed requests.
6. Review Work Order Update Workflow
Consider breaking up complex status transitions. Instead of a single ‘Complete’ button that updates work order, stock moves, quality checks, and parent MO simultaneously, use a two-phase approach:
Phase 1: Mark work order as ‘Completed’ (fast, minimal locks)
Phase 2: Background job processes inventory and quality updates (runs within seconds, but doesn’t block UI)
7. Monitor Lock Contention
Create a monitoring view to track lock waits:
Query pg_stat_activity and pg_locks to identify which specific queries are causing the most lock contention. This helps you prioritize which operations to optimize first.
Expected Results:
After implementing these changes, work order status updates should complete in under 2 seconds even during shift changes. Lock timeouts should become rare (less than 1% of operations). The system will gracefully handle 15-20 concurrent updates without performance degradation.
The key is reducing transaction scope, deferring non-critical operations, and implementing proper retry logic. Your shop floor workers will experience immediate, responsive updates without the frustrating timeouts.