Using LLMs to measure what LLMs cost and why smaller models aren't always cheaper

A cheaper model per call can make the whole workflow more expensive, and most AI business cases don't measure the places where that happens. The pattern goes like this. On straightforward cases, a small model does fine and the saving is real. On the harder cases, its suggestions are weak enough that the person reviewing reads the suggestion, checks it against the source documents, and then does the work themselves. That case now takes longer than it did before the AI existed. Meanwhile the errors that get through review cost far more to fix later than the inference ever cost to run. So the model bill goes down and the cost of the work goes up. Nobody notices, because the model bill is the only number anyone is tracking. Review time sits in a different system, and later corrections are rarely linked back to the task that caused them. Two things follow from this that I found more interesting than the headline: Routing by difficulty helps, but its value depends entirely on the price gap between models. A single price drop can remove the reason for the routing layer altogether, so it should be cheap to unwind. A router can be financially better and still fail on quality. Send enough hard cases to the weak model and the overall error rate goes above where the manual process was, even while the spreadsheet says you're saving money. The fix isn't complicated, just unglamorous: give every task one record that ties model usage, review time and later corrections together, so you can see the full cost of a completed task rather than the cost of a call. https://kairosbrief.substack.com/p/using-llms-to-measure-what-llms-cost