Learned the Hard Way: Why Our AI Design Validator Worked in Pilot but Failed at Scale

PLM AI Adoption in PLM
, , , , , , , ,
1

We spent nine months piloting an AI tool to automate design reviews and catch BOM errors before release. In controlled tests with standardized drawings and a narrow set of manufacturing processes, the tool flagged about 85% of known issues—missing dimensions, tolerance conflicts, manufacturability risks. Leadership loved the demos, and we got budget to roll it out enterprise-wide.

Then reality hit. Production drawings came from multiple sites with different CAD standards, inconsistent metadata, and variable lighting on scanned markups. Accuracy dropped to maybe 50%, and half the flags were false positives that just annoyed engineers. Worse, the tool had no idea when supplier constraints or tooling availability had changed since the data refresh, so it approved BOMs that couldn’t actually be built. We ended up with rework cycles and engineers bypassing the system entirely, keeping their shadow spreadsheets.

The real lesson wasn’t that the AI was bad—it was that we skipped the prerequisites. Our constraint definitions were scattered across emails and tribal knowledge. Our BOM semantics were a mess: engineering, manufacturing, and procurement all used different structures with the same part numbers. We never clarified who owned validation decisions at each handoff. The pilot succeeded because we controlled the variables; production exposed that our data and processes weren’t AI-ready. We’ve since stepped back to fix data governance and semantic alignment before trying again.

2

The supplier constraint issue resonates. We had a similar problem where the AI validated a design based on outdated tooling data. By the time we caught it, procurement had already placed orders for components we couldn’t manufacture. The system didn’t have real-time visibility into plant capacity or supplier availability, so it was basically guessing. Now we’re building integrations to MES and ERP before we trust any automated validation.

3

This mirrors what we saw. Our drawings had annotations in six different formats—some hand-sketched on PDFs, some in structured tables, some just reference numbers. The AI extracted BOMs but they were incomplete or conflicting because it couldn’t interpret context the way a human would. We ended up spending more time validating AI outputs than we would have spent doing manual reviews.

4

Curious how you’re approaching the governance rebuild. Are you documenting constraint ownership explicitly, or is it more about standardizing data formats? We’re stuck in a similar place where everyone agrees the data is messy but no one wants to own the cleanup effort because it’s cross-functional and politically messy.