We went live with our ERP refresh two months ago and turned on the embedded Copilot feature for invoice GL coding. During UAT it was fantastic—95%+ accuracy, finance team loved it, leadership was excited about the automation. Now we’re six weeks past cutover and it’s become a mess. Real supplier invoices are flowing through at 5x the volume we tested with, and the AI is confidently miscoding things left and right. Vendor names spelled slightly differently than our master data, line items in unexpected formats, charges we never saw in test data—the model just guesses and posts with high confidence. We’ve had to pull people off other work to manually review everything, and they’re starting to ignore the suggestions entirely because trust is gone. Our trial balance is a disaster and we’re racing to clean it up before month-end close.
The bigger problem is we also activated the demand forecast feature in supply chain planning around the same time, and we’re starting to see similar drift there—forecasts that looked solid in testing are now missing by wide margins as real market patterns diverge from the training data. It feels like we turned these features on before the system was actually ready to handle production data at scale, and now we’re paying for it in rework and lost confidence.
I’m curious if others have hit this wall post-go-live with Copilot or AI features in ERP. Did you disable and stabilize first, or try to tune the models in production? How do you even know when to retrain, and who owns that decision? Would appreciate hearing what worked (or didn’t) for teams who’ve been through this.
AI Feature Drift Post-Go-Live: Stabilization and Recovery Path
What you’re describing is AI model drift compounded by training data distribution shift—your UAT corpus didn’t represent production volume, vendor variance, or real-world invoice format diversity. The 95% UAT accuracy was valid for that dataset; it was never a guarantee for production. Both your Copilot GL coding and demand forecast symptoms share the same root cause.
Pre-Upgrade / Pre-Reactivation Checks
Before touching model config or retraining anything, establish your baseline:
Audit confidence score thresholds: Most embedded Copilot implementations expose a minimum confidence parameter (verify in your version). If auto-post is triggered above, say, 70% confidence, lower that threshold or disable auto-post entirely until trust is rebuilt.
Pull your false-positive rate by vendor and cost element: Miscodes almost certainly cluster around specific vendor name variants or charge types. Identify the top 10 offending patterns—this scopes your remediation effort.
Freeze demand forecast auto-publish: Do not let drifted forecasts feed MRP or replenishment runs until you’ve quantified the miss rate. Separate that problem from the GL coding crisis operationally.
Check master data hygiene: Vendor name mismatches between invoice data and master data (vendor master, supplier master) are often a data quality problem, not a model problem. Confirm whether fuzzy matching or name normalization is configured.
Validate your retraining data pipeline: Confirm whether the system supports continuous learning from confirmed postings or requires a manual retraining trigger. Verify in your version—this varies significantly across ERP platforms.
Disable auto-posting for Copilot invoice coding immediately. Move to suggestion-only mode where a reviewer confirms before posting.
Tag all AI-suggested transactions from go-live forward with a custom field or posting key so you can isolate them for trial balance reconciliation.
Export the last 6 weeks of coding decisions with outcome labels (accepted/corrected). This becomes your retraining dataset—it’s more valuable than any UAT data you have.
Segment invoices by risk tier: high-volume/low-variance vendors (safe to re-enable with supervision), low-volume/high-variance (manual only until retrained).
Retrain or fine-tune the GL coding model using corrected production postings. Minimum viable dataset size depends on your platform (verify in your version), but generally 500–1,000 confirmed corrections per major charge category is a reasonable starting point.
Re-enable in shadow mode: Let the model suggest, compare suggestions against manual decisions without acting on them, and measure accuracy for two weeks before re-enabling any automation.
For demand forecast: run parallel forecasts (AI vs. statistical baseline) and don’t retire the statistical model until AI miss rates are consistently lower over a full demand cycle.
Rollback Procedure
If retraining doesn’t stabilize within your close window:
Revert GL coding to rule-based auto-coding using existing account determination configuration—this is your pre-AI fallback and should still be intact.
Suspend the Copilot feature license/module activation (verify deactivation path in your version) to prevent background learning from contaminated data.
For demand forecasting, revert planning runs to the last known-good statistical model snapshot before AI feature activation.
Who owns the retrain decision? Operationally it’s a joint call between Finance (GL coding accuracy SLA) and whoever owns your AI/data platform—not IT alone. Set a formal accuracy threshold (e.g., >90% acceptance rate over 500 transactions) as the documented gate for re-enabling automation. Without that gate, you’ll have the same trust collapse again.
This draft is based on general ERP knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.
We had almost the exact same experience. Went live last year, turned on invoice coding AI in week two, and within a month we were drowning in miscoded transactions. The root cause for us was data quality—our vendor master was a mess (duplicates, inconsistent naming, missing attributes) and the AI had no idea how to handle the variation. We ended up disabling the feature, spending eight weeks cleaning master data and mapping invoice patterns, and only then re-enabled it in a phased way. Now it actually works, but we learned the hard way that you can’t layer AI on top of bad foundational data.
The forecast drift problem is real. Our demand planning module was trained on three years of history, but when supply chain disruptions hit and customer buying patterns shifted, the model just kept predicting based on outdated patterns. We didn’t have any retraining process in place—no one even knew it needed retraining until forecasts were obviously wrong. Eventually we set up monthly model refresh cycles and added a feedback loop where planners flag bad forecasts so the system can learn. But yeah, it should have been part of the design from day one, not something we bolted on after the fact.
One thing that helped us was establishing clear governance before go-live about which decisions the AI could make autonomously versus which required human approval. For invoice coding, we set thresholds—anything under a certain amount and from known vendors could auto-post, but new vendors or high-value transactions required manual review. That way when the AI encountered edge cases it hadn’t seen in training, it didn’t just guess. We also built in audit trails so we could see exactly what the AI coded and why. The governance framework probably saved us from the chaos you’re describing.
Model drift monitoring is something most ERP implementations just don’t include, and it kills AI features post-go-live. You need automated tracking of accuracy metrics over time and clear triggers for when retraining should happen. We built dashboards that show coding accuracy week-over-week and alert when it drops below thresholds. When drift is detected, there’s a defined process for retraining on updated data. Without that infrastructure, the model just silently degrades until someone notices the damage. It’s not a one-time deployment; it’s ongoing system maintenance.
I’d argue part of the problem is activating these features too early in the stabilization window. The first 60-90 days post-go-live are already chaotic—users are learning new processes, dealing with cutover issues, and under a lot of stress. Throwing unproven AI features into that mix just adds to the chaos and damages trust in both the core system and the AI. We’ve had better success treating AI as a phase-two capability: stabilize the core ERP first, validate data quality, train users on standard workflows, and only then gradually introduce AI features in controlled pilots. It takes longer, but adoption and trust are much higher.
The volume spike post-go-live is no joke. We tested with maybe 500 clean invoices during UAT, then went live and suddenly had 3,000 invoices a week with all kinds of formats and edge cases the model had never seen. The false positive rate on exceptions was so high that my team couldn’t keep up, and they started ignoring the AI entirely. We ended up having to go back and retrain the model on a much larger, more diverse dataset that actually reflected real production patterns. Testing on sanitized data just doesn’t prepare the system for the real world.
This is why deep workflow integration matters more than the AI tech itself. If the Copilot feature is just a bolt-on suggestion tool with no context about your actual processes, vendor relationships, or GL structure, it’s going to fail when it hits real data. The systems that work well are the ones where the AI is tightly integrated into the workflow—it knows your approval hierarchies, has access to contract data, understands your cost center logic, and routes exceptions intelligently. Generic AI bolted onto ERP rarely survives production. You need domain-specific training and architecture, not just a chatbot in the corner of the screen.