We deployed ML-based demand forecasting across three product lines about 18 months ago, and the models performed well initially—hitting around 93% accuracy in the first quarter. By month six, though, we started seeing drift. Accuracy dropped to around 85%, and we were accumulating excess inventory in some SKUs while running short on others.
The root cause was that our models were trained on pre-pandemic data, and customer behavior had shifted. Seasonal patterns changed, lead times from suppliers weren’t matching historical norms, and input costs were volatile. We hadn’t built in any kind of drift detection, so the degradation was silent until it showed up in our working capital numbers.
We implemented a continuous monitoring framework that tracks prediction error rates weekly, compares live data distributions against training baselines, and flags when feature distributions drift beyond thresholds. We also set up automated retraining pipelines triggered by performance drops or significant data changes. It took about four months to stabilize, but we’re now back to around 91% accuracy and the system adapts much faster when market conditions shift. The big lesson for us was that deploying the model was just the beginning—keeping it relevant requires ongoing work.