Better Models, Worse Tools: Claude tool calls regress on Sonnet 5 and Opus 4.8

Open-source developer Armin Ronacher published a reproducible finding on 2026-07-04: newer Claude models — Opus 4.8 and Sonnet 5 — emit malformed tool calls against Pi’s replacements[] array schema roughly 20% of the time in long sessions, while older Claude models stay clean. The failure is specific. The argument’s oldText and newText are byte-correct; the model just adds a junk key after closing the object and the schema validator rejects the call.

Source: Armin Ronacher, “Better Models: Worse Tools” (pocoo.org, 2026-07-04) — Screenshot, no license stated, used editorially for news commentary.

What happened

The first signal came from Petr Baudis, who opened Pi issue #6278 on 2026-07-03 after Claude started returning arguments like { "oldText": "...", "newText": "...", "in_file": "..." }, or with requireUnique, oldText2 / newText2, or closeenough — none of them permitted by Pi’s edit-tool schema. Pi rejects the call, the model retries, and the loop eats turns.

Ronacher reproduced it in a controlled harness and collected the full set of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and event.0.additionalProperties. A reproducer gist by pasky replays a verbatim Anthropic Messages API payload without strict: true and finds the defect across transports: 4/11 invalid with OAuth + streaming, 4/20 OAuth without, 2/10 API key + streaming, 2/16 non-streaming with documented fields only.

The failure is context-dependent. A fresh single-turn prompt like “edit this file” reproduces it rarely or not at all. The failure rate climbs in long agentic sessions where the model has read files, diagnosed a problem, and composed a multi-line edit — exactly the workflow Claude Code is trained to do well.

Why it matters

This is a counterintuitive regression. The same family that shipped Sonnet 5’s effort-level API dial and a $2/$10 → $3/$15 per-MTok price cut on 2026-06-30 — see Sonnet 5 narrows gap to Opus — is worse at a basic contract every Claude developer relies on: emit a tool call that matches the declared schema.

Pi’s replacements[] shape is not exotic. Any tool that exposes a nested array of objects sits in the same risk band: batch updates, multi-file edits, multi-message dispatches, list-shaped APIs. If your Claude-facing schema has a “list of objects” anywhere, this finding is directly relevant to you.

What is going on (root-cause hypothesis)

Ronacher proposes a four-step mechanism. It is his hypothesis, not an Anthropic engineering statement, and he is explicit about that framing.

1. Tool calls are text the model emits. Anthropic uses in-band ANTML markers — they look like XML but are a tokenization convention, and Anthropic has not published a spec for them.
2. The sampling distribution over tool-call shapes was historically broad. Older Claude models spread probability mass across a wide range of plausible shapes; nested-array shapes were one valid path.
3. RL post-training inside Claude Code’s harness rewards the canonical shape and penalises alternatives. Claude Code’s edit tool is flat — file_path, old_string, new_string, optional replace_all — and its client silently filters unknown keys.
4. The model develops a strong prior that edits look a specific way. When a nested-array shape goes off-distribution and the model is sampling after a long escaped string, it generates a plausible-looking extra key rather than the closing brace.

The “slop harness” effect

Two factors are mutually reinforcing. The model is loose; the harness is forgiving. Claude Code filters unexpected keys, repairs Unicode escapes, and does not default to strict: true. Anthropic’s API does offer a strict: true tool mode that constrains the sampler to the declared schema — Ronacher’s runs show it eliminates the bug — but it applies complexity caps (per Pi PR #6266, 20 strict tools, 24 optional parameters, 16 union parameters per request). The fix is not a one-line toggle for anyone with a non-trivial tool surface.

Practical implications for builders

• Enable strict: true on Anthropic API tool definitions where your schema fits the complexity budget. Every object must declare additionalProperties: false and you must stay inside the caps.
• Keep tool JSON schemas shallow when possible. A single object with scalar fields travels closer to the Claude Code training distribution than a nested array of objects.
• Treat the post-training prior as Claude-Code-shaped. Tools that resemble Bash, Edit, Read, Write, Glob, or Grep work better than tools that resemble a domain API. If you can flatten to that shape, do.
• Regression-test new Claude model releases against your real tool schemas before swapping. Sonnet 5 and Opus 4.8 are not drop-in replacements for Sonnet 4.6 or Opus 4.5 on tool-calling workloads.
• For edit-style tools, prefer a single object with a replacement string field over a structured object inside an array. That is the shape the model has been trained on.

Risks and caveats

• This is a single-author analysis with reproducible experiments but no vendor response.
• The root-cause mechanism is a hypothesis, not an Anthropic-confirmed engineering explanation.
• The ~20% figure is the worst-case number in long sessions; a fresh single-turn prompt like “edit this file” reproduces it rarely or not at all.
• The Pi replacements[] schema is specific; results may not generalise to every nested-array tool.
• Anthropic has not, at the time of writing, published a statement, a system-card update, or a fix.

What to watch

• Anthropic’s official response: a system-card update, a blog post, or a default change to strict: true for the tool-use API.
• Community reproductions on other tool shapes, especially non-edit operations.
• Whether the same regression appears in tool-call evaluations published alongside the next Anthropic model release.
• Pi’s own fix path — already tracked in issue #6278.
• Comparable analyses for other vendors: OpenAI’s strict-mode behaviour, Google’s function-calling on Gemini.

Sources

• Armin Ronacher, Better Models: Worse Tools — 2026-07-04
• Pi issue #6278 — 2026-07-03
• pasky reproducer gist — 2026-07-04
• Anthropic — Tool use / function calling overview
• AI Newsroom — Sonnet 5 narrows gap to Opus (2026-07-01)