RAG vs. fine-tune vs. just better prompts: how we decide on client projects

I lead AI delivery at 247 Labs. Most of what we get asked for arrives pre-diagnosed. The client has already decided they need a fine-tuned model, usually because a competitor announced one. We run the same decision ladder every time, in this order, and we only move down a rung when the one above it demonstrably fails. Sharing it because the "RAG vs fine-tuning" question gets argued in the abstract a lot more than it gets tested. Rung 1. Is the prompt actually bad? It's slightly embarrassing how often this is the whole answer. Before anything else we build an eval set of real examples with known-correct outputs and run the existing prompt against it. More often than is comfortable, structured prompting plus a few worked examples closes enough of the gap that nothing further is needed. The tell that you're on this rung: failures look like inconsistent formatting, ignored instructions, or the model doing three things when it was asked for one. Those are prompt problems, not model problems. Rung 2. Does it need knowledge it doesn't have? If the failures are factual, where the model confidently invents a policy, a SKU, a contract clause, that's retrieval, not tuning. Fine-tuning teaches behaviour far more reliably than it teaches facts, and a fine-tune of your knowledge base goes stale the day someone edits a document. Where enterprise RAG projects actually die, in our experience, is never the vector store: The source corpus contradicts itself and nobody owns which version is authoritative Chunking splits tables and procedures mid-context, so the retrieved fragment is technically relevant and practically useless Nobody scoped permissions, so retrieval happily surfaces documents the asking user was never cleared to see Access control got treated as a phase-two problem For regulated clients that last one is not a detail, it's the project. Budget for it at the start or pay for it twice. Rung 3. Does it need a behaviour it can't be instructed into? This is the narrow band where fine-tuning earns its cost. The legitimate reasons we've actually used it: A rigid output format that has to hold across high call volumes without drift A domain register or house style that few-shot prompting can't hold consistently Latency or unit cost at volume, where a smaller tuned model beats a large prompted one on both Only the first of those is about capability. The other two are economics. That's the honest version of the fine-tuning pitch, and it is rarely how it gets sold. The part that decides all three You cannot tell which rung you're on without an eval set. Not vibes, not a demo that impressed a stakeholder. A fixed set of inputs with known-good outputs that you re-run on every change. Every engagement where we skipped this, we ended up rebuilding against the client's opinion of last Tuesday's output. Building the eval set is usually the least popular line in the proposal and the only one I'd never cut. Interested in where this ladder breaks for other people. Has anyone got a case where fine-tuning was clearly right and retrieval clearly wasn't? Genuinely asking. We may be over-indexed on retrieval because of our client mix.