Better Models, Worse Tool Calls

Flask author Armin Ronacher’s investigation of a Pi bug reveals a disturbing truth: tool calling in the new Claude models (Opus 4.8, Sonnet 5) is regressing — not getting better, but worse. And he’s found the root cause: RL post-training has over-fitted to Claude Code’s own tool schema, making alternative tool schemas increasingly off-distribution. This is a must-read for anyone building their own agent harness.

A strange Pi issue has sent me down a deep rabbit hole for the past two days. Short version: the newest Claude models sometimes add extra, hallucinated fields to the nested edits[] array when calling Pi’s edit tool. Not Haiku or some small model — Opus 4.8. The edit itself is usually correct, but the parameters don’t match the schema because the model invents keys that don’t exist, causing Pi to reject the tool call and demand a retry.

This is not entirely unexpected — models emit malformed tool calls occasionally, especially small ones. But what surprised me is that it’s getting worse with Anthropic’s new models. Opus 4.8 and Sonnet 5 both exhibit this problem, whereas previous old models did not. In other words, the SOTA model in this family is worse on a particular tool schema than its older siblings.

Tool Calls Are Just Text

If you haven’t spent much time looking under the hood of LLM tool calling, the key thing to understand is: tool calls are not magic. The model receives a transcript, a system prompt, and a list of available tools. The server stuffs these into one big prompt with special tokens. Because the model was trained and reinforced on examples in that format, at some point during generation it emits something that the API or client interprets as “call this tool with these arguments.”

The details: JSON inside nested arrays is serialized inside XML tags. Basic top-level string parameters appear inline, while arrays of objects are implemented via JSON serialization. This matters because when the model has to decide between } and , "..." after a several-hundred-token escaped string, that’s exactly the highest-entropy point.

The Failure

Pi’s edit tool supports multiple exact string replacements in one call, so the arguments contain an edits array. In the failing cases, the model produced entries like:

{ "oldText": "...", "newText": "...", "requireUnique": true }

Or:

{ "oldText": "...", "newText": "...", "oldText2": "", "newText2": "" }

Across repeated trials I saw a whole zoo of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, and even an event.0.additionalProperties inside the edit object.

The most annoying part is that the actual oldText and newText payloads in the invalid calls I inspected were byte-correct. The model had produced the correct invocation but then added nonsense at the end of the object.

The failure is also heavily context-dependent. A fresh single-turn “edit this file” prompt did not reproduce it at all. With agent history — the model read files, diagnosed a problem, and composed a multi-line edit — it did reproduce. And not all transcripts show it. Turning on strict tool invocation eliminated the problem in my runs.

Why It’s Getting Worse

My strongest hypothesis is that this is not random deterioration but a training artifact.

When older Anthropic models were trained, they were trained on some tools, but that training didn’t yet have a user-shipped harness like Claude Code as the obvious target. Modern Anthropic models are most likely different because their post-training includes Claude Code or a harness that looks very similar. The model learns what a successful tool call looks like in that environment. It also learns what mistakes are tolerated.

Claude Code’s own tools are comparatively flat. The ordinary edit tool is not Pi’s nested edits[] shape; it’s closer to file_path, old_string, new_string, and an optional flag (replace_all). Looking at Claude Code’s client is very instructive: it contains retry paths for malformed tool use, parameter aliases, type coercions, Unicode repairs, and filtering of unknown keys. In other words, Anthropic’s own client appears to expect and accept a fair amount of slop and repairs it, mostly silently.

If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field, or using a nearby parameter name.

Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape. A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.

This is not too surprising, but it is a change from how things were a few months ago. When Opus 4.5 launched, it adapted to other edit tools exceptionally well. I was pretty convinced then that we were on a good path — models were more likely to adapt to any tool shape as long as the instructions were good. Now I’m somewhat worried about the track we’re on. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented.

The Slop Harness

Claude Code is closed-source, but we can look at the minified code. Honestly, it’s very forgiving of incoming data.

For a start, Claude Code checks the model’s visible text for leaked <antml:functions> tags and attempts to verify tool use. It also has a resolve_tool_use function that normalizes the received tool call. In that function, there is:

1. A validation step that, if that fails, does another pass of trying to find function calls in the transcript.
2. An attempt to fix a specific Unicode encoding issue reported for Python.
3. A step that removes extra keys from the JSON object not needed for the function call.

The fact that the client removes extra keys is interesting. That means the client expects extra keys. That’s effectively training the model to be sloppy.

Nested Parameters Are Hard

It’s worth taking a step back and looking at how tool parameters with nested arrays are handled by different providers. The general approach is quite different.

OpenAI for instance uses a JSON-based representation with a tool execution boundary. In their function calling, everything is plain JSON. The schema can be defined with JSON Schema and the model is guided to produce valid JSON. Anthropic uses a slightly different mechanism: tool calls are serialized as part of the message content using their ANTML format. In that format, the tool call is embedded as content after a special function call tag. Parameter structure is typically flat or uses JSON for nested objects.

Consider a function call like:

<antml:functions>
<antml:invoke name="edit">
<antml:parameter name="file_path">some/file.py</antml:parameter>
<antml:parameter name="edits">[{"oldText": "text to replace","newText": "replacement text"}]</antml:parameter>
</antml:invoke>
</antml:functions>

The edits parameter here is JSON serialized as a string inside an XML element. If the edits array is long or contains large strings, the model has to produce that JSON escaped inside the XML. That means it is writing a string inside a tag, and inside that string is JSON with escaped quotes and nested content. It is doing this without any grammar guidance beyond what it learned in training.

An important thing to note here: while this looks like XML, it’s not really XML. It’s just a thing they found convenient to tokenize and train on. The other thing to note is that a basic top-level string parameter appears inline, whereas an array of objects is implemented via JSON serialization. While I’m not entirely sure this is exactly how it works, there are indications it’s not too far off.

There are two very different ways to make the model produce such a structure:

1. You can ask the model to produce valid JSON matching a schema and validate afterwards.
2. You can constrain the sampler so that invalid JSON or invalid schema shapes cannot be sampled.

The second approach is grammar-aware or constrained decoding. The sampler masks tokens that would violate the grammar. Grammar-aware decoding can enforce syntactically valid JSON, specific enum values, or keys.

Without constraints, the model is merely following a learned convention. Harmony, the protocol used by Github’s Copilot and OpenAI’s Codex in their agent mode, uses a protocol like this:

assistant<|channel|>commentary to=functions.get_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

The important bit is <|constrain|>json. The model can express in-band that the message body is JSON, and the inference stack can switch into JSON-constrained sampling. Some of this also happens in Anthropic’s models, at least in strict mode. The marker helps the sampler detect when to use a specific grammar and makes it easy to do.

For hosted GPT models, there is an option to provide a LARK grammar for custom tools. Anthropic seems different, though maybe not entirely. If arrays of objects are represented as JSON, then the model writes JSON inside the tool parameter. There may be basic grammar-constrained sampling, which could partly explain the extra keys. For a nested array parameter, that JSON includes escaped multi-line file content inside string literals. The unexpected keys appear exactly at the highest-entropy point: after closing a several-hundred-token escaped newText string, where the model must decide } vs , "...".

Opus 4.8 and Sonnet 5 seem to have much stronger priors about what an edit tool call should look like — and that prior appears to be Claude Code’s flat old/new string pair plus the optional replace_all flag. My guess is that Opus has learned that an edit operation may have one extra optional field, but under Pi’s nested oldText/newText shape it has no trained name for that field. So it samples a plausible name fresh each time, producing dozens of random keys rather than one stable alias.

Since strict mode in Anthropic appears to fix this, I presume the server side refuses to sample a key not permitted by the JSON schema. That would explain why they have limits on tool definition complexity when strict mode is enabled.

So far, the Codex models I tested did not show this regression. I tested all available ones except 5.6, which I don’t have access to yet.

What This Means For Harnesses

The uncomfortable lesson is that tool schemas are not neutral, at least on Anthropic models. We like to pretend a schema is an abstract contract and the model follows it as a general reasoner, but that may no longer be the case. Tool schemas lie somewhere in the distribution. Some shapes are close to post-training, some are far. Some are easy for the provider’s hidden encoding (e.g., top-level attributes in ANTML), while others require large escaped JSON objects inside nested arrays after long multiline strings. The model may understand the schema but still be bad at sampling the exact shape under pressure.

If this continues, what are the implications for harnesses? You could turn on strict sampling in Anthropic and the problem should go away. But that the model exhibits this behavior reveals the impact of reinforcement learning. Fighting that prior is probably futile if you want the best performance.

Right now, Claude Code is not open source, and we can’t know what they do in their RL environments. We can’t assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness must inherit its quirks.

I used to be more skeptical of strict grammar-constrained tool invocation because constrained decoding can have quality tradeoffs. That may still be true in general, but this bug significantly shifted my priors. If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.

If you want to find out more, or discuss this, consider reading the issue on the Pi tracker.

This entry was tagged ai and pi.

copy as markdown / view