We tested the client's data before building the RAG system. 4 in 10 answers were right; here's what got it to 9.

Every RAG demo works. Then you point it at a company's actual documents. Client is a fifth-generation textiles and furnishings company with 10,000+ products, 1,000+ internal documents, and live inventory. They wanted AI across the business and weren't sure their data could support it. Fair doubt. So before scoping a build, our team at BotsCrew ran a pilot on their real data. Took the questions employees actually ask, ran them against what already existed. Four in ten answers came back correct. None of those failures were model failures: Conflicting documents, no way to tell which version was current Scanned and unreadable files are sitting in the same index as good content Retrieval returns the plausible document instead of the authoritative one Business rules that existed nowhere except in people's heads The first fix was to give up on one retrieval path and answer everything. Product questions went to the catalog search with exact match, stock questions to the live inventory API, policy and general knowledge to a cleaned-up RAG layer. A lot of what gets called a RAG accuracy problem is a routing problem wearing a costume. Then the least interesting work on the project: mapping their internal acronyms. Region codes every employee knows, and the model had never seen. Boring, and one of the biggest single jumps we got. What made any of it measurable was a fixed test set of the same questions, every iteration. That's how we could say under 60% became roughly 90%, instead of arguing about whether it felt better. The part I keep thinking about came later, on the analytics side. The same question asked a week apart returned different numbers, because the warehouse backfills. We had to agree on what "correct" even meant before we could measure it. And more than once, the honest answer to "where does this number come from" was that nobody was sure. That's what AI transformation mostly is. Not model selection. Whether your data is where you think it is, whether your processes exist outside people's heads, and who decides when two documents disagree. A pilot is the cheapest readiness assessment you'll ever run. It tells you what's broken before you commit to an architecture.