Choose what LLMs can and can’t do well

Another pattern from building a multi-agent system, following up on the typed contracts post from a while back. LLMs are excellent at one kind of task and mediocre at another, and most of the pain I've hit in multi-agent systems comes from not respecting that split. What they're genuinely good at: judgment calls with no single correct answer. Given these three signals, which one matters most here and why. Given this messy input, what's the plausible interpretation. This is reasoning under ambiguity, and it's the actual value an LLM adds. You couldn't write a deterministic function for it even if you wanted to, because there isn't one right answer to find. What they're mediocre at: consistent computation. Ask an LLM to turn a set of inputs into a score, a ranking, a number, and it'll give you something confident and plausible looking. It's not computing that number though, it's pattern-matching to what a score like that tends to look like given the surrounding text. Change the order you present the inputs, rephrase one sentence, and the same underlying data can quietly produce a different number. Nothing errors. It just looks exactly as trustworthy when it's wrong as when it's right, which makes it worse than a normal bug, there's no stack trace pointing at the problem. So the pattern I landed on: the LLM only ever does the first kind of task. Anything in the second category goes to plain code. ```python mediocre task, asked of the LLM score = llm("score this from 0-100 based on the signals") split by what each part is actually good at weights = llm.decide_which_signals_matter(inputs) # judgment, ambiguous score = composite_score_tool(inputs, weights) # computation, one right answer