Token Reduction Is Not Cost Reduction

# Token Reduction Is Not Cost Reduction: An Empirical Study of End-to-End Efficiency in API-Based Coding Agents
Source: [https://arxiv.org/html/2607.12161](https://arxiv.org/html/2607.12161)
\(July 2026\)

###### Abstract

Context\-reduction layers for API\-based coding agents—command\-output compressors, retrieval rankers, and payload\-optimizing proxies—are usually evaluated by how much text they remove\. We ask instead: when does reducing retrieved context or tool output reduce the*actual billed cost*of a coding agent without decreasing task success or lengthening its trajectory? Our primary evidence is a pre\-specified, hash\-frozen, paired campaign of 2,908 provider\-billed Claude Code runs \(2,848 analyzed; 103 tasks, 7 repositories, three models\) comparing a baseline against two hook\-based compression generations and an API\-boundary proxy, within a broader measured program of roughly 5,500 billed executions\. Three findings emerge\. First, prompt\-cache traffic dominates cost composition: cache creation and reads accounted for approximately 87% of the reconstructed four\-component cost \(about 80% of the actual bill\), with an explicitly disclosed 8\.7% dollar\-weighted residual that the retained telemetry could not attribute—and which, on Haiku 4\.5, scales with the thinking\-effort setting\. Second, tool\-output reduction did not reliably predict billed\-cost reduction: an arm that removed 38% of estimated raw tool\-output tokens showed a*higher*paired cost \(\+6\.8%\+6\.8\\%, 95% CI\[\+2\.8,\+11\.3\]\[\+2\.8,\\,\+11\.3\]\), and per\-task reduction was a weak, unstable predictor of cost change \(Pearsonr=0\.15r=0\.15, CI crossing zero\)\. Third, compression can harm task completion by destroying action\-critical evidence: in a small single\-shot study on SWE\-bench\-derived Go tasks, compression reduced patch application from 27/40 to 15/40 by corrupting verbatim edit anchors, and the compressed grounded arm produced descriptively fewer solves at higher observed cost per solve\. We propose a layered evidence standard culminating in success\-adjusted billed cost\.

## 1Introduction

API\-based coding agents such as Claude Code\[[3](https://arxiv.org/html/2607.12161#bib.bib3)\], SWE\-agent\[[33](https://arxiv.org/html/2607.12161#bib.bib33)\], and OpenHands\[[29](https://arxiv.org/html/2607.12161#bib.bib29)\]operate by iteratively invoking tools—shell commands, file reads, searches, test runs—and feeding the resulting text back into a large language model\. Each turn re\-transmits an ever\-growing context: repository excerpts, prior tool outputs, test logs, diffs, and conversation history\. This growth has made*context economy*an engineering concern in its own right, and a market of mitigation layers has emerged: deterministic command\-output compressors, query\-aware retrieval rankers, semantic and structural code search, and full API\-boundary proxies that rewrite payloads in flight\.

Nearly all of these layers are evaluated by some form of*tokens removed*: compression ratios on captured command outputs, recall\-at\-budget on retrieval corpora, or preserved\-marker counts on synthetic fixtures\. This paper argues, and provides paired\-campaign evidence for the evaluated workloads, that tokens removed is an incomplete—and sometimes actively misleading—metric for the quantity practitioners care about: the provider\-billed cost of completing a task successfully\.

Three mechanisms break the naive equivalence\. First, modern serving stacks bill cached context at sharply discounted rates\[[2](https://arxiv.org/html/2607.12161#bib.bib2),[14](https://arxiv.org/html/2607.12161#bib.bib14),[20](https://arxiv.org/html/2607.12161#bib.bib20)\]; in the workloads we measure, the large majority of the reconstructed spend is prompt\-cache traffic whose price per token is 10–125% of the nominal input price, so removing text from it saves far less than a token count suggests\. Second, an agent is a closed loop: if compression removes context the model later needs, the model can search again, re\-read files, or take extra diagnostic turns, and those added turns re\-transmit the*entire*context prefix\. Third, compression can corrupt downstream consumers that are not the model at all—in one failure mode we analyze, ranked search summaries were piped into shell counting pipelines that expected rawgreplines, silently producing wrong answers\.

We therefore separate six measurement layers that are frequently conflated: \(1\) component\-level compression ratios; \(2\) retrieval\-quality metrics; \(3\) model\-visible context reduction; \(4\) billed API\-token reduction; \(5\) end\-to\-end task success; and \(6\) cost per successful execution\. Using a completed evidence audit over all retained campaign artifacts \(transcripts, ledgers, manifests, gate traces, and analysis code\), we place every system for which artifacts exist—baseline Claude Code, RTK, the RTK\-ML architecture, Headroom, a vendor CLI compressor, theml\_lexicalengine, Caveman, and lexical, embedding\-based, structural, and union retrieval—at the highest evidence layer its artifacts support, and we report end\-to\-end results only where paired, actually\-billed campaigns exist\.

#### Scale of the broader program\.

The results reported here emerged from a substantially larger research and development effort: across the broader program, exploratory, superseded, tuning, failure\-analysis, and infrastructure experiments consumed API resources well beyond the campaigns analyzed here\. This paper does not treat that broader spend as a statistical sample\. Instead, it reports in detail the controlled campaigns for which tasks, configurations, provider\-returned usage, success judgments, transcripts, and analysis artifacts were retained and could be audited; those auditable campaigns account for the approximately $643 of measured API spend reconciled in the campaign inventory \([table˜3](https://arxiv.org/html/2607.12161#S5.T3), Appendix[A\.2](https://arxiv.org/html/2607.12161#A1.SS2)\)\. We disclose the broader experimental investment to convey the scale of the search process, while restricting quantitative claims to the smaller evidence set that satisfies the study’s reproducibility and statistical requirements\.111The broader development expenditure is contextual and is excluded from all sample sizes, confidence intervals, cost decompositions, and system comparisons in this paper\. No repository ledger aggregates it, so we state no dollar figure for it here\.

#### Research questions\.

- •RQ1\.Which components dominate the billed cost of API\-based coding agents in realistic paired workloads?
- •RQ2\.How much of that cost is addressable by tool\-output compression and retrieval\-context reduction?
- •RQ3\.Do component\-level compression and retrieval gains predict end\-to\-end task efficiency?
- •RQ4\.How do model tier, effort level, task family, and trajectory structure mediate system effects?
- •RQ5\.What benchmark design is required for trustworthy token\-efficiency claims?

#### Contributions\.

1. 1\.A calibrated cost anatomy for API\-based coding agents: a four\-component decomposition that reproduces most individual bills with a median per\-run residual of≈\\approx1%, showing that cache creation and reads accounted for≈\\approx87% of the reconstructed cost \(≈\\approx80% of the actual bill\), alongside an explicitly quantified 8\.7% dollar\-weighted residual that the retained telemetry cannot attribute \([section˜6](https://arxiv.org/html/2607.12161#S6)\)\.
2. 2\.A large paired, provider\-billed comparison of context\-reduction layers for coding agents—the primary campaign alone spans 2,908 runs, 103 tasks, 7 repositories, 3 models, up to 5 effort levels, and 4 arms from identical fresh working copies with block\-randomized order, within a measurement program of 5,493 billed executions and approximately $643 total API spend \(5,123 of them in append\-only cost ledgers\), plus 8,263 zero\-inference\-cost component evaluations \([section˜5](https://arxiv.org/html/2607.12161#S5)\)\.
3. 3\.Direct evidence that token reduction and cost reduction decouple: a 38\.4% reduction in estimated raw tool\-output tokens coexisting with a\+6\.8%\+6\.8\\%paired cost increase, and a near\-zero per\-task correlation \(r=0\.15r=0\.15\) between tool\-output reduction and billed\-cost change \([section˜7](https://arxiv.org/html/2607.12161#S7)\)\.
4. 4\.A trajectory\- and phase\-level account of*where*savings are created and repaid, built from 13,620 deterministically classified assistant turns \([section˜9](https://arxiv.org/html/2607.12161#S9)\)\.
5. 5\.Two documented compression\-safety failure modes—shell\-pipeline corruption of ranked search output, and destruction of the verbatim edit anchors that patch application depends on—together with the mitigations that fixed them \([sections˜10](https://arxiv.org/html/2607.12161#S10)and[7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\)\.
6. 6\.A single\-shot grounded\-completion study on SWE\-bench\-derived Go tasks in which byte\-exact context grounding was associated with the largest observed solve\-rate improvement \(1/29 to 5/29\), while three tasks solved by the grounded raw arm were not solved by the grounded compressed arm and no task showed the reverse pattern \(exact pairedp=0\.25p=0\.25; descriptively higher cost per solve under compression;[section˜7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\)\.
7. 7\.A layered evidence taxonomy and benchmark\-design recommendations for future token\-efficiency claims \([sections˜11](https://arxiv.org/html/2607.12161#S11)and[14](https://arxiv.org/html/2607.12161#S14)\)\.

## 2Cost Anatomy of API\-Based Coding Agents

### 2\.1Terminology

We use the following terms throughout; they are deliberately not interchangeable\.

Uncached input tokensPrompt tokens processed at the full input price \(provider usage fieldinput\_tokens\)\.

Cache\-creation tokensPrompt tokens written into the provider’s prompt cache \(usage fieldcache\_creation\_input\_tokens\), billed at a multiplier of the input price\.

Cache\-read tokensPrompt tokens served from the prompt cache \(cache\_read\_input\_tokens\), billed at a discount\.

Generated output tokensTokens produced by the model \(output\_tokens\), including visible text and tool\-call arguments\.

Tool\-use blockA structured tool invocation emitted by the model within an assistant turn\.

Raw tool outputThe bytes a tool actually produced before any compression layer \(observable in these campaigns only for the RTK arms, via the hook\-side ledger\)\.

Delivered \(model\-visible\) tool outputThe text that reached the model after any compression or ranking layer\. Raw and delivered tool\-output token counts are estimates from RTK’s embedded local BPE tokenizer \(tiktokeno200k\_basetables\), not the provider tokenizer\.

TurnOne assistant message; a run’s*trajectory length*is its number of turns\.

Cost per run / per successful executionBilled dollars per run; and*cost per successful execution*: total billed cost across all executions in an arm divided by the number of successful executions; repeated executions of the same underlying task remain separate attempts\. We use*success\-adjusted cost*for the broader concept, and we do not report a per\-unique\-task estimator\.

Raw shell output, delivered tool output, and generated output tokens are three different quantities; conflating any two of them produces most of the inflated savings claims we discuss in[section˜11](https://arxiv.org/html/2607.12161#S11)\.

### 2\.2Formal cost model

Let a run consumeTuncT\_\{\\mathrm\{unc\}\}uncached input tokens,TcwT\_\{\\mathrm\{cw\}\}cache\-creation tokens,TcrT\_\{\\mathrm\{cr\}\}cache\-read tokens, andToutT\_\{\\mathrm\{out\}\}generated tokens, on a model with input pricepinp\_\{\\mathrm\{in\}\}and output pricepoutp\_\{\\mathrm\{out\}\}\(USD per10610^\{6\}tokens\)\. The billed cost recorded per run is the provider’stotal\_cost\_usd; our reconstruction is

C^=1106​\(Tunc​pin\+Tcw​μw​pin\+Tcr​μr​pin\+Tout​pout\),\\widehat\{C\}\\;=\\;\\frac\{1\}\{10^\{6\}\}\\Bigl\(T\_\{\\mathrm\{unc\}\}\\,p\_\{\\mathrm\{in\}\}\\;\+\\;T\_\{\\mathrm\{cw\}\}\\,\\mu\_\{\\mathrm\{w\}\}\\,p\_\{\\mathrm\{in\}\}\\;\+\\;T\_\{\\mathrm\{cr\}\}\\,\\mu\_\{\\mathrm\{r\}\}\\,p\_\{\\mathrm\{in\}\}\\;\+\\;T\_\{\\mathrm\{out\}\}\\,p\_\{\\mathrm\{out\}\}\\Bigr\),\(1\)with cache multipliersμw\\mu\_\{\\mathrm\{w\}\}\(write\) andμr\\mu\_\{\\mathrm\{r\}\}\(read\)\. Calibration against the billed amounts \([section˜6\.1](https://arxiv.org/html/2607.12161#S6.SS1)\) selects the provider’s published standard prices—Haiku 4\.5 at $1/$5, Sonnet 5 at $3/$15, Opus 4\.8 at $5/$25 per million input/output tokens—withμr=0\.1\\mu\_\{\\mathrm\{r\}\}=0\.1andμw=1\.25\\mu\_\{\\mathrm\{w\}\}=1\.25\(the five\-minute cache TTL\)\[[4](https://arxiv.org/html/2607.12161#bib.bib4),[2](https://arxiv.org/html/2607.12161#bib.bib2)\]\.

Cost per successful execution \(CPS\) for an armaaover runsRaR\_\{a\}with success setSaS\_\{a\}\(its successful executions\) is

CPSa=∑r∈RaCr\|Sa\|\.\\mathrm\{CPS\}\_\{a\}\\;=\\;\\frac\{\\sum\_\{r\\in R\_\{a\}\}C\_\{r\}\}\{\\lvert S\_\{a\}\\rvert\}\.\(2\)
For a reduction layerjj, we distinguish its*addressable share*AjA\_\{j\}—the fraction ofC^\\widehat\{C\}attributable to context that layerjjis able to modify \(for a tool\-output compressor, the delivered tool text and its downstream cache traffic\)—from its*realized saving*, the paired change in billed cost,

Δ​Cj=𝔼tasks​\[C¯t\(j\)−C¯t\(base\)\],\\Delta C\_\{j\}\\;=\\;\\mathbb\{E\}\_\{\\text\{tasks\}\}\\bigl\[\\,\\overline\{C\}^\{\\,\(j\)\}\_\{t\}\-\\overline\{C\}^\{\\,\(\\mathrm\{base\}\)\}\_\{t\}\\,\\bigr\],\(3\)estimated over paired blocks \([section˜5](https://arxiv.org/html/2607.12161#S5)\)\. The two differ by a*trajectory term*: added or removed turns change every component of[Equation˜1](https://arxiv.org/html/2607.12161#S2.E1)because each turn re\-transmits the context prefix\. WritingΔ​turnsj\\Delta\\mathrm\{turns\}\_\{j\}for the paired change in trajectory length, the*success\-adjusted net effect*of a layer is the paired change inCPS\\mathrm\{CPS\}, which can be negative \(a saving\) only if per\-turn savings exceed the combined trajectory and success penalties\. This is the quantity we treat as decision\-grade\.

#### Marginal cost of a delivered tool token\.

A simplified marginal model makes the addressable share concrete\. A delivered tool\-output token inserted into the prompt at turnttincurs \(per10610^\{6\}tokens\)

Ctool\-token,t=μw​pin\+Nfuture reads,t​μr​pin,C\_\{\\text\{tool\-token\},t\}\\;=\\;\\mu\_\{\\mathrm\{w\}\}\\,p\_\{\\mathrm\{in\}\}\\;\+\\;N\_\{\\text\{future reads\},t\}\\;\\mu\_\{\\mathrm\{r\}\}\\,p\_\{\\mathrm\{in\}\},\(4\)whereμw=1\.25\\mu\_\{\\mathrm\{w\}\}=1\.25is the observed five\-minute cache\-write tier,μr=0\.1\\mu\_\{\\mathrm\{r\}\}=0\.1, andNfuture reads,tN\_\{\\text\{future reads\},t\}is the number of later model calls that reuse the cached prefix containing the token\. Its first cache creation is priced*above*ordinary input; later reuse is discounted\. An early token is therefore more addressable than a late one: with the authoritative campaign’s mean trajectory of≈\\approx4\.5 assistant turns, a token delivered on the first turn is re\-read on≈\\approx3\.5 subsequent calls and costs≈\(1\.25\+0\.35\)​pin=1\.60​pin\\approx\(1\.25\+0\.35\)\\,p\_\{\\mathrm\{in\}\}=1\.60\\,p\_\{\\mathrm\{in\}\}, versus1\.25​pin1\.25\\,p\_\{\\mathrm\{in\}\}for a final\-turn token—and removing an early token also removes its future reads\. A removed token can additionally alter trajectory length itself, so addressability depends on turn position and future reuse, not on token count alone\. The retained telemetry does not record, per token, the turn of entry and the number of later calls that reused it, so we present[eq\.˜4](https://arxiv.org/html/2607.12161#S2.E4)as a bounded conceptual model; exact per\-token addressability measurement is future work\.

### 2\.3Empirical cost stack

[Figure˜1](https://arxiv.org/html/2607.12161#S2.F1)and[table˜6](https://arxiv.org/html/2607.12161#S6.T6)preview the paper’s central descriptive fact: in the evaluated workloads, generated output is a small minority of cost, and cache creation plus cache reads account for approximately 87% of the reconstructed four\-component cost—about 80% of the actual bill, with a further 8\.7% of the bill left unattributed by the reconstruction \(shown explicitly as a fifth category\)\. These are empirical averages from the coding\-agent workloads evaluated in this study, not industry averages; denominators, uncertainty, and caveats are given in[section˜6](https://arxiv.org/html/2607.12161#S6)\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x1.png)Figure 1:Cost decomposition as shares of*actual billed*cost \(n=2,848 analyzed runs; component prices as in[eq\.˜1](https://arxiv.org/html/2607.12161#S2.E1)\)\. The hatched fifth segment is the unattributed billed\-cost residual \(billed minus reconstructed\)\. Numbers inside segments are percentages; run\-bootstrap 95% CIs for every segment are in[table˜6](https://arxiv.org/html/2607.12161#S6.T6)\. Normalized four\-component shares appear in Appendix[A\.3](https://arxiv.org/html/2607.12161#A1.SS3)\.

## 3Optimization Surfaces and Architectures

Context\-reduction systems differ in*where*they act on the agent loop, what they can see, and whether their transformations are reversible\.[Table˜1](https://arxiv.org/html/2607.12161#S3.T1)summarizes the systems studied or audited here; evidence levels refer to the layered taxonomy of[section˜11](https://arxiv.org/html/2607.12161#S11)\(L1–L8\)\.

Table 1:System architecture and evidence matrix\. “Billed E2E” means paired end\-to\-end campaigns with provider\-billed cost and deterministic task scoring exist in the retained artifacts\. Evidence levels: L1 compression ratio, L2 retrieval/preservation quality, L3 model\-visible context change, L4 production\-path activation, L5 task success, L6 billed cost, L7 cost per successful execution, L8 trajectory/failure analysis\.#### Hook\-based compression \(RTK\)\.

RTK installs aPreToolUsehook that rewrites eligible shell commands to route through a CLI proxy; the proxy applies YAML\-configured, strategy\-based compression to the command’s output before it reaches the model\. RTK is the shipped deterministic layer\. The*RTK\-ML*architecture adds nine independently flag\-gated capabilities on top: \(1\) query propagation from the user prompt into the proxied command; \(2\) retrieval routing of search\-type commands \(grep,rg,find,ls, …\) into a grouped, ranked*search\-group*representation; \(3\) query\-aware deterministic re\-ranking; \(4\) a lexical scoring leg; \(5\) an embedding scoring leg backed by a loopback BGE sidecar\[[30](https://arxiv.org/html/2607.12161#bib.bib30)\]; \(6\) a task\-family router that classifies the context; \(7\) a preserve gate that bypasses compression entirely for high\-recall families; \(8\) structural retrieval using ast\-grep\[[5](https://arxiv.org/html/2607.12161#bib.bib5)\]to inject focus\-file hints; and \(9\) a session\-scoped focus cache\. All nine are off by default, and the disabled paths covered by the dedicated equivalence tests were byte\-equivalent; a safety router \(default\-on\) prevents semantic stages from touching sensitive tool output, and a stdout\-consumer guard \([section˜10](https://arxiv.org/html/2607.12161#S10)\) suppresses ranking when the command’s stdout feeds another program\.

#### API\-boundary proxying \(Headroom\)\.

Headroom\[[15](https://arxiv.org/html/2607.12161#bib.bib15)\]interposes on the provider API endpoint viaANTHROPIC\_BASE\_URLand optimizes payloads in flight\. Its observed configuration in these campaigns was token\-mode optimization with caching enabled and code\-aware mode disabled\. Because it is a black box to our harness, we can measure its billed effects and trajectories but cannot attribute them to internal mechanisms; we therefore avoid causal claims about*why*it behaves as it does\.

#### Deterministic CLI compression \(external vendor tool\)\.

One audited system is a commercially distributed deterministic command\-output compressor from an unaffiliated vendor\. The retained artifacts contain only that vendor’s own per\-category claimed savings \(75–80% forgit diff,git log,ls,grep\), embedded as reference literals in a benchmark harness\. No same\-harness, paired, actually\-billed end\-to\-end run of it exists in the artifacts, so it appears in this paper only as a related architectural approach\.

#### Query\-aware research engine \(ml\_lexical\)\.

Theml\_lexicalengine is an implemented research architecture combining chunking, lexical and cosine ranking, packing with protected regions, and a BGE embedding sidecar\. Its evidence is component\-level retrieval quality only \([section˜4](https://arxiv.org/html/2607.12161#S4)\); it has no end\-to\-end task\-success or billed cost evidence\.

#### Agent\-prose compression \(Caveman\)\.

Caveman compresses LLM\-generated prose \(never tool output\) under an explicit preservation contract \(paths, line numbers, test names, error types, metrics, and code blocks are kept verbatim\)\. It is not registered in any production strategy table; its evidence is a seven\-fixture component study \([section˜4](https://arxiv.org/html/2607.12161#S4)\)\.

## 4Benchmarks and Datasets

[Table˜2](https://arxiv.org/html/2607.12161#S4.T2)maps every benchmark used in the underlying research program to what it does and does not measure\. Two properties matter most: whether the benchmark observes*end\-to\-end task success*, and whether its costs are*actually billed*rather than reconstructed\.

Table 2:Benchmark capability matrix\. “Billing” distinguishes: provider\-returned usage with execution\-result cost cross\-checked against the pricing schedule \(billed\+usage\); cost reconstructed from provider\-returned token counts \(tokens\); billed cost recovered from frozen reports, ledgers, or traces \(billed, with the source named\); and length\-based estimates \(est\.\)\.#### Why gold context does not determine agent behavior\.

Component benchmarks score a compressor against human\- or test\-annotated*gold*context\. Four mechanisms break the link from gold\-context preservation to agent outcomes\. \(i\) Agents can*recover*: if a needed line is removed, the model can search again or re\-read the file, converting a preservation failure into a trajectory cost rather than a task failure\. \(ii\) Agents can*succeed without the gold*: prior knowledge or inference over remaining context is often sufficient\. \(iii\) Agents can*fail with the gold*: receiving the right lines does not guarantee correct edits\. \(iv\) Retrieval is one stage of a multi\-stage trajectory; a single extra diagnosis\-and\-retry loop re\-transmits the full context prefix and can eliminate the nominal savings of a much larger compression \([section˜9](https://arxiv.org/html/2607.12161#S9)\)\. Long\-context research documents related gaps between context presence and context use\[[23](https://arxiv.org/html/2607.12161#bib.bib23),[7](https://arxiv.org/html/2607.12161#bib.bib7)\]\.

#### Suite\-level benchmarks and created corpora\.

Beyond the benchmarks above, the program built and ran a family of suite\-level corpora: a 159\-case command\-output compression corpus \(cmdcompress\) with 34 paired live\-API cases, privacy\-redaction and intent\-classification suites \(privacybench,observer\_intent\), retrieval\-mode and output\-compression exercises, and a path\-coverage suite proving which changed code each arm actually executes\. It also*created*the synthetic task fixtures the end\-to\-end campaigns run on: seven scripted repository tasks for the pilot harness, fourteen fixture suites \(343 files, including 150/400\-test pass/fail suites and bulk\-output generators\) for the comprehensive campaign, three synthetic large repositories \(911 files\) for the long\-session marathons, seeded synthetic monorepos with committed ground truth for Stages 1–2 and the authoritative campaign, the 7 Caveman prose fixtures, and the 24\-observation captured pytest corpus\. Appendix[A\.12](https://arxiv.org/html/2607.12161#A1.SS12)inventories all of them with sizes and paths\.

#### SWE\-bench lineage\.

The grounded\-completion program \([section˜7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\) draws its tasks from SWE\-bench\-derived corpora: the ContextBench\-Go rows are Multi\-SWE\-bench Go instances\[[36](https://arxiv.org/html/2607.12161#bib.bib36)\]exposed through the ContextBench loader, the agent smoke ran on theContextbench/SWE\-bench\_Prodataset\[[11](https://arxiv.org/html/2607.12161#bib.bib11)\], and the IC\-SWE pilot uses SWE\-bench Python repositories at pinned base commits\[[19](https://arxiv.org/html/2607.12161#bib.bib19)\]\. Task success is scored by whether the row’s FAIL\_TO\_PASS tests flip to passing after the agent’s patch applies\.

#### Snapshots\.

The authoritative campaign is fully frozen \(task manifests, six pinned repository SHAs plus a seeded synthetic monorepo, sha256\-pinned binaries, gzipped transcripts\)\. The component\-benchmark side is weaker: the ContextBench dataset revision hash was not recorded \(row identifiers are retained\), and parts of the LoCoEval artifacts were written to volatile temporary directories\. Appendix[A\.9](https://arxiv.org/html/2607.12161#A1.SS9)lists the exact snapshot state per dataset\.

## 5Experimental Methodology

### 5\.1Campaign hierarchy

The underlying research program produced several campaign generations \([table˜3](https://arxiv.org/html/2607.12161#S5.T3)\)\. The program’s evidence classes are reconciled exactly in Appendix[A\.2](https://arxiv.org/html/2607.12161#A1.SS2): 5,123 executions recorded in append\-only runtime cost ledgers totaling $628\.80 \(guarded re\-executions such as the Stage\-1 post\-guard rerun and the two separately\-billed long\-session sibling campaigns counted at their true cost\); 8 costsmoke executions \($1\.01\) whose count derives from the suite results file rather than a per\-run ledger; and 362 billed single\-shot trials whose reported or trace\-reconstructed spend totals approximately $13\.29 \([table˜4](https://arxiv.org/html/2607.12161#S5.T4), analyzed in[section˜7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\)\. The combined measured program therefore comprises 5,493 billed executions and approximately $643\.10 in spend, in addition to 8,263 zero\-inference\-cost component evaluations and a free deterministic gate\-activation proof \(Stage 0\)\. Later campaigns supersede earlier ones where they overlap\.[Table˜3](https://arxiv.org/html/2607.12161#S5.T3)lists them with their role in this paper\. All quantitative end\-to\-end claims in[Sections˜6](https://arxiv.org/html/2607.12161#S6),[7](https://arxiv.org/html/2607.12161#S7),[8](https://arxiv.org/html/2607.12161#S8)and[9](https://arxiv.org/html/2607.12161#S9)come from the*authoritative campaign*unless explicitly labeled otherwise; the long\-session campaign is reported separately because its RTK arm was built from an earlier binary lineage \(v0\.7\.0\-line “observer” build\) and must not be pooled with the RTK\-ML build; earlier pilots are cited only for provenance and for the failure analysis of[section˜10](https://arxiv.org/html/2607.12161#S10)\.

Table 3:Campaign inventory and role, entire measurement program\. Spend and execution counts are totaled from each campaign’s append\-only runtime cost ledger \(every billed execution counted once, including guarded re\-executions\); analysis populations de\-duplicate rescored rows separately\. The authoritative campaign comprises a base phase and a pre\-specified expansion \(new hash\-frozen holdout\) analyzed jointly\. Stage 0 is a free deterministic proof and is excluded from the paid total\.Table 4:The single\-shot grounded\-completion and agent\-smoke program \(companion bench harness; claude\-sonnet\-4\-6; billedanthropic\_api\_usage\)\. Trials are \(row×\\timesarm\) agent calls;∼\\simmarks report\-approximate or trace\-derived spend\.
### 5\.2Design of the authoritative campaign

#### Arms\.

\(1\)*Baseline*: Claude Code 2\.1\.201 with no layers\. \(2\)*RTK*: the shipped deterministic hook layer, built from its release commit\. \(3\)*RTK\-ML*: the flag\-gated architecture at the guard commit, all nine gates enabled, safety router in enforce mode, BGE embedding sidecar health\-checked before every batch\. \(4\)*Headroom v0\.27\.0*viaANTHROPIC\_BASE\_URL\. Binary sha256 hashes and exact provenance are in Appendix[A\.8](https://arxiv.org/html/2607.12161#A1.SS8)\.

#### Tasks and repositories\.

103 analyzed tasks across 24 defined families \(23 with analyzed tasks; the one family without an analyzed task contained only the excluded defective task\) \([section˜8](https://arxiv.org/html/2607.12161#S8)\) over 7 repositories: six pinned open\-source repositories \(click, cobra, express, flask, gin, requests\) and a seeded synthetic monorepo\. Tasks are frozen in pre\-specified, hash\-frozen manifests with per\-task judges \(deterministic answer/regex/test checks\), allowed/forbidden file constraints, timeouts, and design attributes \(difficulty, retrieval mode, session band, output band\)\.

#### Pairing and randomization\.

The unit of execution is a*block*: one task×\\timesmodel×\\timeseffort×\\timesrepetition, in which all four arms run from identical fresh working copies in randomized order\. Runs execute in an isolated per\-run$HOMEwith a scrubbed environment \(allCLAUDE\*/ANTHROPIC\*and compression\-layer variables removed except the API key\), fixed harness settings \(project\-only setting sources so no user hooks load\), per\-run budget caps, and a global spend cap enforced under a lock\.

#### Models and effort\.

Claude Haiku 4\.5, Claude Sonnet 5, and Claude Opus 4\.8, at effort levels low/medium/high/xhigh/max for Haiku and low/high for Sonnet and Opus \(unmeasured cells are reported as such, never imputed;[section˜8](https://arxiv.org/html/2607.12161#S8)\)\.

#### Measurement\.

Cost is the provider\-billedtotal\_cost\_usdfrom the harness JSON output \(fieldcost\_sourceequal toactual\_billedon every analyzed run\)\. For every execution,total\_cost\_usdwas obtained from the Claude Code/API execution result and independently cross\-checked against the provider\-returned usage fields and the applicable Anthropic pricing schedule; it was not estimated from text length or local token heuristics\. Usage tokens come from the provider usage object; success is scored by the frozen deterministic judges; gate activation is read from per\-run gate traces; raw\-vs\-delivered tool tokens come from the hook\-side ledger \(RTK arms only\), counted with an embedded local BPE tokenizer \(estimates, not provider token counts\)\. Every transcript is retained gzipped, enabling the turn\-level ledgers and phase analysis of[section˜9](https://arxiv.org/html/2607.12161#S9)\.

#### Pre\-specification and split roles\.

The campaign design, arms, tasks, judges, and analysis plan were frozen in hash\-pinned manifests and pre\-specification documents before execution \(the expansion added a new frozen holdout before its spend\)\. There was no public, independently timestamped preregistration; we therefore use “pre\-specified” throughout rather than “pre\-registered\.” Findings are labeled by provenance: development/base\-split observations \(which also drove the guard fix and judge repairs\), frozen\-holdout confirmations \([table˜8](https://arxiv.org/html/2607.12161#S7.T8)\), pooled estimates, and exploratory strata\.

#### Exclusions and integrity\.

Analysis rows are de\-duplicated keep\-last per run id \(rescored rows are appended, never overwritten\), restricted to completed non\-infrastructure\-failure runs, and exclude one manifest\-flagged defective task and the calibration split: 2,848 analyzed runs \($175\.92 billed\) out of 2,908 executed\. Five judge\-defect classes found before the holdout were fixed and rescored symmetrically across all arms from stored outputs, with the full append\-only row history retained\.

### 5\.3Prompt\-cache carryover between arms \(threat to validity\)

Because the measured outcome is directly affected by provider\-side prompt caching, within\-block cache carryover is a first\-order threat: all four arms of a block run the same task from identical working copies, per\-run$HOMEisolation does not isolate the provider’s cache, and the median gap between consecutive runs in a block is 10\.8 s \(p90 28\.6 s; 100% of the 2,136 consecutive pairs fall inside the five\-minute cache TTL\)\. A TTL\-separated re\-analysis is therefore impossible on this data — carryover cannot be ruled out by temporal separation\. Three retained\-data checks bound the concern \([table˜5](https://arxiv.org/html/2607.12161#S5.T5); full numbers in the reproducibility package\)\. First, the design is exactly balanced at the margins: each arm ran 712 times in total and each block position contains exactly 712 runs \(arm\-by\-position cell counts vary 147–221 under randomization, which is not stratified by position\)\. Second, the headline paired deltas do not depend on relative order: for every arm, the delta measured when the arm ran*before*baseline is statistically indistinguishable from the delta when it ran*after*\(all order\-difference CIs cross zero\)\. Third, later block positions show no cache\-write deflation—mean cache\-creation tokens at positions 1–3 are 3\.7%*higher*than at position 0, the opposite of what shared\-prefix reuse would produce, and mean cost varies non\-monotonically by position \($0\.0597–$0\.0652\)\. Randomized order limits systematic bias by construction, and these checks detect no order signature in cost, cache traffic, or success; they do not prove carryover is absent, and we carry the point as a limitation\.

Table 5:Order\-sensitivity check: paired per\-task cost delta vs\. baseline, split by whether the arm executed before or after baseline within its randomized block\. All order\-difference CIs cross zero\.
### 5\.4Statistical procedure

The task is the unit of inference for comparative arm effects\. Descriptive cost\-composition and reconstruction intervals use run\-level resampling and are not interpreted as task\-level treatment\-effect inference\. For any comparative metric we form paired within\-block differences \(arm minus baseline\), average them within task, and bootstrap over tasks \(10,000 resamples, fixed seed\) for 95% confidence intervals; the expansion analysis additionally reports repository\-clustered bootstrap intervals and leave\-one\-repository\-out sensitivity for the same contrasts\. Resampling units by analysis: arm comparisons use the task\-clustered bootstrap; cost\-composition shares and the reconstruction residual use a run\-level bootstrap \(descriptive only; run\-level resampling does not account for repeated\-task dependence\); repository sensitivity uses a repository\-clustered bootstrap; grounded paired outcomes use an exact paired analysis; exploratory task\-family cells are descriptive or task\-bootstrap as flagged in each table\. Repeated runs of the same task are never treated as independent: intraclass correlation of cost across repetitions was 0\.37–0\.55 depending on arm, giving Kish effective sample sizes of roughly 38–45 tasks per arm despite 712 runs per arm\. Family\-level results are exploratory screening: the underlying campaign screened families by whether task\-clustered bootstrap intervals excluded zero, but no per\-contrastpp\-values were computed, so no false\-discovery\-rate control is claimed; we report family\-level effect sizes with confidence intervals and label them exploratory\. Cells with fewer than six tasks are flagged unstable and reported as exploratory\. We report effect sizes with confidence intervals and avoid binary significance language where power is insufficient\. Failed tasks are never dropped from cost aggregates, and lower cost caused by early failure is not treated as efficiency: cost per successful execution \([eq\.˜2](https://arxiv.org/html/2607.12161#S2.E2)\) keeps failed runs in the numerator\.

## 6Aggregate Cost Composition

### 6\.1Calibration of the decomposition

Applying[eq\.˜1](https://arxiv.org/html/2607.12161#S2.E1)with the provider’s published standard prices and the five\-minute cache\-write multiplier \(μw=1\.25\\mu\_\{\\mathrm\{w\}\}\{=\}1\.25,μr=0\.1\\mu\_\{\\mathrm\{r\}\}\{=\}0\.1\) reproduces the billed cost of individual runs with median per\-run residuals of\+0\.9%\+0\.9\\%\(Haiku,n=1,748n\{=\}1\{,\}748\),\+0\.0%\+0\.0\\%\(Sonnet,n=948n\{=\}948\), and\+0\.2%\+0\.2\\%\(Opus,n=152n\{=\}152\)\. Alternative configurations are cleanly rejected by the same calibration: the one\-hour cache multiplier leaves−23\-23% to−33\-33% median residuals, and Sonnet 5 introductory pricing leaves\+33%\+33\\%unexplained—so the campaign was billed at standard prices under the five\-minute cache tier\.

The median residual and the aggregate residual answer different questions, and the distinction matters for every headline in this paper\. The four\-component model is accurate for most*individual*runs \(the median residuals above\), but the*dollar\-weighted aggregate*residual is\+8\.7%\+8\.7\\%of total billed spend \(95% CI \[7\.2, 10\.1\]; by arm: baseline\+7\.1%\+7\.1\\%, RTK\+5\.5%\+5\.5\\%, RTK\-ML\+4\.9%\+4\.9\\%, Headroom\+14\.5%\+14\.5\\%\)—an economically material share that cannot be hidden behind the small median\. A post\-hoc per\-model analysis \(scriptscripts/extract\_thinking\_addressability\.py; retained run records only\) localizes it\. On Sonnet 5 the four components account for the bill at both measured efforts \(residual\+0\.2%\+0\.2\\%\), so its aggregateoutput\_tokensappear to include thinking charges\. On Haiku 4\.5 the residual rises monotonically with the thinking\-effort setting—from $0\.0038 to $0\.0064 per run \(11\.0% to 18\.3% of the cell’s bill; implied\+753\+753to\+1,283\+1\{,\}283tokens at the output price\)—while measured output \(≈\\approx780/run\), cache volumes, and turn counts stay flat\. Cache\-tier and turn\-count explanations are ruled out by those flat covariates, and secondary\-model auxiliary calls contribute only a smaller, flat component \(residual 5\.2% for one\-model vs\. 7\.2% for two\-model runs\): the effort\-scaling component is therefore*thinking\-consistent*, and on this model thinking charges sit inside the residual rather than the measured output component\. Opus 4\.8 cells are too small to conclude \(n=24/90\)\. Provider usage semantics are thus not uniform across models in this harness—the same aggregate field appears to include thinking on Sonnet 5 and exclude it on Haiku 4\.5\. The reconstruction therefore does*not*account for every billed dollar, and all headline shares are reported against the actual bill with the residual shown explicitly; the “generated output” component is*measured*output\. The larger Headroom residual is an observation, not evidence of any particular internal mechanism\.

### 6\.2Where the dollars went

Table 6:Cost decomposition as shares of*actual billed*cost with run\-bootstrap 95% CIs, by slice, including the unattributed residual \(billed minus reconstructed\)\. Population: 2,848 analyzed runs \($175\.92 billed\)\. Cache write plus cache read is≈\\approx80% of the bill \(≈\\approx87% of the reconstructed four\-component cost\)\. Normalized four\-component shares for finer slices are in Appendix[A\.3](https://arxiv.org/html/2607.12161#A1.SS3)\.Across all 2,848 analyzed runs \([table˜6](https://arxiv.org/html/2607.12161#S6.T6),[fig\.˜1](https://arxiv.org/html/2607.12161#S2.F1)\), as shares of the*actual bill*: cache creation 44\.3% \(CI \[43\.2, 45\.3\]\), cache reads 35\.4% \[34\.5, 36\.3\], generated output 10\.4% \[10\.0, 10\.9\], uncached input 1\.3% \[1\.1, 1\.5\], and unattributed residual 8\.7% \[7\.2, 10\.1\]\. Equivalently, cache creation and cache reads accounted for approximately 87% of the reconstructed four\-component cost \(48\.5% and 38\.7% ofC^\\widehat\{C\}\) and about 80% of the bill itself\. The mean run carried≈\\approx117k cache\-read tokens and≈\\approx12k cache\-creation tokens against only≈\\approx715 generated tokens\.

This composition, not any property of a specific compression layer, is the paper’s central explanatory fact \(RQ1\):

> *A system may remove a large fraction of a particular tool output while changing only a small fraction of total billed cost, because the removable output may represent a small part of the complete prompt\-cache and trajectory cost stack\.*

Slices sharpen the picture \(normalized four\-component shares; Appendix[A\.3](https://arxiv.org/html/2607.12161#A1.SS3)\)\. Failed runs shift spend toward cache reads \(43\.5% vs\. 38\.6% ofC^\\widehat\{C\}on successes\) and generated output \(14\.1% vs\. 11\.3%\)—a pattern consistent with additional re\-reading or recovery behavior\. Long sessions raise the output share from 7\.4% \(short\) to 18\.4% and nearly triple cache\-read tokens per run\. Headroom’s profile has the lowest output share \(8\.5%\) and the highest cache share of any arm; the tested configuration was associated with higher cache\-side usage; Opus is the only model with a material uncached\-input share \(11\.3%\)\. Because cache reads are billed at0\.1×0\.1\\timesinput price, the*addressable*saving from removing one delivered tool token is an order of magnitude smaller than nominal token counts suggest once that token has entered the cached prefix—and is repaid on every subsequent turn only at the discounted rate\. This is why the cost surface of a real agent differs qualitatively from simplistic input\-versus\-output diagrams \(RQ2\)\.

## 7End\-to\-End Results

### 7\.1Aggregate outcomes

Table 7:Aggregate end\-to\-end results, authoritative campaign \(per arm: 712 runs, 103 tasks\)\. CPS = cost per successful execution \([eq\.˜2](https://arxiv.org/html/2607.12161#S2.E2)\)\.Δ\\Deltacost is the paired per\-task billed\-cost change vs\. baseline as % of the baseline mean, task\-clustered bootstrap 95% CI;Δ\\Deltasucc\. is the paired success change in percentage points\.[Table˜7](https://arxiv.org/html/2607.12161#S7.T7)reports the headline end\-to\-end comparison\. No success\-rate difference was detected within the precision of the campaign \(all paired success CIs cross zero; RTK\-ML\+0\.9\+0\.9pp \[\-0\.6, \+2\.5\], Headroom\+1\.4\+1\.4pp \[\-0\.2, \+3\.1\]\); the study was not designed as a formal non\-inferiority trial, and the high baseline success rate \(96–98%\) limits sensitivity to small performance losses \([section˜13](https://arxiv.org/html/2607.12161#S13)\)\. Costs, by contrast, separate clearly: RTK is null\-to\-slightly cheaper than baseline \(−2\.7%\-2\.7\\%, CI \[\-5\.6, \-0\.1\]\); the RTK\-ML build is a small net cost \(\+6\.8%\+6\.8\\%, CI \[\+2\.8, \+11\.3\]\); and Headroom carries a large, consistent penalty \(\+48\.4%\+48\.4\\%, CI \[\+42\.3, \+55\.0\]\), while no success\-rate difference was detected at the campaign’s precision\. The expansion analysis confirmed the Headroom and RTK\-ML\-vs\-RTK orderings under repository\-clustered intervals and leave\-one\-repository\-out sensitivity\.[Table˜8](https://arxiv.org/html/2607.12161#S7.T8)separates the frozen\-holdout estimates from the pooled ones: the Headroom penalty \(\+47\.3%\+47\.3\\%\[\+35\.2, \+59\.2\]\) and the RTK\-ML cost \(\+9\.1%\+9\.1\\%\[\+2\.4, \+16\.9\]\) are confirmed on the holdout alone, whereas RTK’s small saving is a pooled estimate whose holdout\-only interval crosses zero \(−2\.3%\-2\.3\\%\[\-7\.4, \+2\.1\]\)—it should be read as suggestive, not holdout\-confirmed\.

Table 8:Frozen\-holdout\-only vs\. pooled paired billed\-cost change vs\. baseline\. Findings first observed during development and confirmed on the frozen holdout are distinguished from pooled estimates\.
### 7\.2Token reduction versus cost reduction

The hook\-side ledger makes raw\-vs\-delivered tool output observable for the the RTK arms\. Over the campaign, the RTK\-ML arm reduced estimated raw tool\-output tokens from 5\.62M to 3\.46M delivered \(−38\.4%\-38\.4\\%\), while RTK delivered essentially raw output \(−1\.3%\-1\.3\\%\)\. Raw and delivered counts are produced by RTK’s hook\-side ledger using an embedded local BPE tokenizer \(tiktokeno200k\_basemerge tables\), not the provider tokenizer, so we report them as estimates; the reduction percentage compares like\-for\-like counts under the same tokenizer\. Yet the RTK\-ML arm’s paired billed cost was*higher*than baseline and RTK’s was marginally lower\. At the task level \([fig\.˜2](https://arxiv.org/html/2607.12161#S7.F2)\), observed tool\-output reduction was a weak, unstable predictor of paired billed\-cost change: Pearsonr=0\.154r=0\.154\[\-0\.051, \+0\.356\], Spearmanρ=0\.013\\rho=0\.013\[\-0\.082, \+0\.339\] over 100 Haiku tasks; the Pearson interval crosses zero, the point estimate rises to0\.240\.24when the five highest\-reduction tasks are removed, and it is0\.130\.13\[\-0\.09, \+0\.36\] on the 68 tasks where the gate actually routed output\. Two caveats bound the interpretation\. First, reduction is concentrated: 87 of 100 tasks saw under 0\.1% reduction, and the four tasks in the 5–20% band show a*positive*mean cost change \(\+24\.7%\+24\.7\\%\[\+6\.3, \+46\.0\]; binned means in[fig\.˜2](https://arxiv.org/html/2607.12161#S7.F2)\)\. Second, observed reduction is endogenous—the optimized arm can issue different commands and different trajectories, so raw\-output exposure is partly determined by the model’s own actions, and the compression percentage is not an externally assigned treatment dose\. We therefore readrrdescriptively: in this evaluated Haiku task set, per\-task observed tool\-output reduction was a weak predictor of paired billed\-cost change; this does not establish that compression cannot affect cost\. Answering RQ3, component\-level reduction did not predict end\-to\-end cost movement in these workloads\. The phase analysis \([section˜9](https://arxiv.org/html/2607.12161#S9)\) identifies trajectory patterns consistent with a possible mechanism: the realized saving \([eq\.˜3](https://arxiv.org/html/2607.12161#S2.E3)\) is small once cache prices and added turns are accounted for—retrieval\-stage savings were real but small at cache prices, and were repaid downstream by added diagnosis, testing, and re\-retrieval turns\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x2.png)Figure 2:Per\-task estimated raw tool\-output reduction \(local BPE tokenizer\) vs\. paired billed\-cost change \(RTK\-ML arm, Haiku 4\.5; each point one task,n=100n=100; task means over paired blocks; red squares: binned means with bootstrap 95% CIs\)\. Pearsonr=0\.154r=0\.154\[\-0\.051, \+0\.356\]; Spearmanρ=0\.013\\rho=0\.013\[\-0\.082, \+0\.339\] \(task bootstrap, 10,000 resamples, seed 7\)\. Observed reduction is endogenous to the arm’s own trajectory, not an assigned dose\.
### 7\.3Cost per successful execution

Table 9:Cost per successful execution \(USD;[eq\.˜2](https://arxiv.org/html/2607.12161#S2.E2)\) with task\-bootstrap 95% CIs\. Top: per arm, with paired difference and ratio vs\. baseline \(pairing preserved by resampling the same task set for both arms\)\. Bottom: per arm and model;nTn\_\{T\}= unique tasks, s/r = successes/runs \(u= fewer than six tasks; Opus cells rest on 38 runs each and should be read as thin\)\. Failed runs remain in the cost numerator\.
[Table˜9](https://arxiv.org/html/2607.12161#S7.T9)and[fig\.˜3](https://arxiv.org/html/2607.12161#S7.F3)give the decision\-grade metric with its uncertainty\. Per\-arm CPS ratios vs\. baseline are: RTK 0\.968 \[0\.937, 1\.004\] \(the interval includes 1\), RTK\-ML 1\.051 \[1\.006, 1\.100\] \(a small but resolvable penalty\), and Headroom 1\.464 \[1\.397, 1\.531\]\. On every model, Headroom’s cost per successful execution is the highest \(e\.g\., Opus 4\.8: $0\.1989 vs\. $0\.1189 baseline\)\. The two RTK generations bracket the baseline: RTK slightly below on Sonnet/Opus, the RTK\-ML build slightly above, with the gap concentrated where its compression surface is smallest\.[Figure˜4](https://arxiv.org/html/2607.12161#S7.F4)shows the run\-level relationship between trajectory length and billed cost that drives these aggregates: cost grows with turns on every arm, so any layer that adds turns must overcome that growth before its per\-turn savings show up as net savings\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x3.png)Figure 3:Cost per successful execution by system and model \(whiskers: task\-bootstrap 95% CIs; 10,000 resamples, seed 7; success and run counts in[table˜9](https://arxiv.org/html/2607.12161#S7.T9)\)\.![Refer to caption](https://arxiv.org/html/2607.12161v1/x4.png)Figure 4:Billed cost vs\. trajectory length \(assistant turns\) per run, by model \(n=2,848 runs; log\-scaled cost\)\. Each added turn re\-transmits the cached prefix, so trajectory changes dominate per\-turn savings\.
### 7\.4Single\-shot grounded\-completion study \(separate harness\)

A companion program ran a*single\-shot*SEARCH/REPLACE coding agent \(claude\-sonnet\-4\-6, billedanthropic\_api\_usage\) over SWE\-bench\-derived Go tasks in the benchmark worktree harness \([table˜4](https://arxiv.org/html/2607.12161#S5.T4)\)\. It is not a Claude Code campaign—one model, one shot, smallnn—but it is the only place in the artifact record where*task success under compression*is measured jointly with a grounding intervention, and it illustrates a broader risk for edit formats that depend on verbatim anchors\.

#### Compression destroys verbatim edit anchors\.

On the full ContextBench\-Go population \(40 rows×\\times\{raw, compressed\}\), the compressed arm cut billable tokens by 73% and cost by 59%, yet lost patch\-*applicability*on a net 12 rows \(raw applied 27/40, compressed 15/40; failure mixedit\_apply\_failed20 vs\. 12\): SEARCH/REPLACE editing requires the agent to reproduce a code anchor byte\-for\-byte, and aggressive compression rewrote or dropped exactly those spans\. Resolved tasks were 2/40 \(raw\) vs\. 1/40 \(compressed\), and on the 10\-row subset where both arms applied, solve rate was equal \(1–1\)\. The observed deficit was concentrated in patch applicability; the experiment does not isolate whether compression also affected reasoning quality\. The causal chain for application failures is mechanical: the model receives context, emits an edit whose SEARCH block must match the on\-disk file byte\-for\-byte, and application precedes any test; compression that rewrites, omits, normalizes, or reorders the anchored span fails the patch even when the intended change is conceptually correct\. The failure taxonomy therefore separates four states—evidence loss before reasoning, reasoning failure, edit\-application failure, and test failure after successful application—and the observed failure mix moves between them: under grounding,edit\_apply\_failedcollapsed \(17→\\to5 on the compressed arm\) andtests\_failedbecame the dominant residual \(genuine reasoning limits\)\. A context\-only agent on SWE\-bench\_Pro Go rows applied 0/18 patches, a pattern consistent with the same anchor\-availability limitation\. The frozen report itself warns against the “successes\-per\-million\-tokens” framing under which the compressed arm would appear to win: with 1 vs\. 2 solves on non\-overlapping subsets, that ratio metric inverts the verdict while hiding the apply\-rate collapse\.

#### In this small single\-shot experiment, the largest observed improvement was associated with byte\-exact grounding rather than compression\.

Phase G2 re\-ran 29 oracle\-resolved rows across four arms, adding a*grounding*prelude—byte\-exact±\\pm30\-line windows read from the prepared repository at the gold\-context file paths—ahead of the \(raw or compressed\) broad context \([table˜10](https://arxiv.org/html/2607.12161#S7.T10)\)\. Grounding lifted apply rates from 76% to 90% \(raw\) and from 31% to 83% \(compressed\) and produced the program’s first real solve signal: 5/29 for grounded raw at $0\.248 per resolved row\. Under grounding, the compressed arm’s solves were a strict subset of the raw arm’s: three tasks solved by the grounded raw arm were not solved by the grounded compressed arm, while no task showed the reverse pattern \([table˜11](https://arxiv.org/html/2607.12161#S7.T11)\)\. With only three discordant pairs the paired contrast is not statistically resolvable \(exact two\-sidedp=0\.25p=0\.25\), and per\-row costs were not retained, so no confidence interval can be attached to the per\-resolved\-row cost ratio; descriptively, the compressed grounded arm produced fewer solves at a higher observed cost per resolved row \($0\.515 vs\. $0\.248\) even though its cost*per attempted row*was lower \($0\.0355 vs\. $0\.0427\)—compression reduced per\-attempt spend while reducing application probability, and the solve economics went the other way\. These counts are small; we report them as mechanism evidence, not as a stable general result\. A family\-routing variant of the same design showed the preserve policy \([section˜3](https://arxiv.org/html/2607.12161#S3), gate 7\) protecting exactly this surface: preserve vs\. aggressive compression applied 83% vs\. 48% context\-only, with evidence\-loss failures 5 vs\. 15, and grounded\-preserve solving most \(6\) at the lowest observed cost per resolved row \($0\.214\)\.

Table 10:Phase G2 grounded\-completion results \(29 oracle\-resolved ContextBench\-Go rows per arm, single\-shot, claude\-sonnet\-4\-6, billed usage; $4\.72 total\)\. In this small study, the largest observed improvement was associated with grounding; under grounding, the compressed arm produced descriptively fewer solves at higher observed cost per resolved row \([table˜11](https://arxiv.org/html/2607.12161#S7.T11)\)\.Table 11:Paired task outcomes and cost decomposition in the grounded study \(29 rows; grounded raw vs\. grounded compressed\)\. With three discordant pairs, the exact two\-sidedppis 0\.25; the comparison is descriptive\. Per\-row costs were not retained, so uncertainty for the per\-resolved\-row cost cannot be computed\.Paired outcome \(29 rows\)CountSolved by both arms2Solved only by grounded raw3Solved only by grounded compressed0Solved by neither24Exact two\-sidedpp\(discordant pairs\)0\.25Cost per attempted row \(raw / compressed\)$0\.043 / $0\.035Cost per applied patch$0\.048 / $0\.043Cost per resolved row$0\.248 / $0\.515
#### Scope limits\.

These are single\-shot trials: the agent cannot search again or recover, so the recovery\-loop dynamics of[section˜9](https://arxiv.org/html/2607.12161#S9)are structurally invisible here \(the IC\-SWE pilot records them explicitly asnot\_measured\)\. Sample sizes are small \(solve counts of five and two\), one model \(Sonnet 4\.6\) and one edit format \(SEARCH/REPLACE\) were used, tasks are SWE\-bench\-derived Go rows, and one compression configuration was evaluated\. Nothing here should be generalized beyond that envelope\. We read this program as mechanism evidence—what the evaluated compression configuration does to edit anchors, and what grounding does to apply rates—not as a system ranking; its direction is consistent with the paired multi\-turn campaigns\.

### 7\.5Long\-session evidence \(separate lineage\)

A separate long\-session program \(Opus 4\.8; three long\-horizon code\-change tasks on synthetic large repositories; paired design\) provides the long\-session view: a 529\-session main campaign \($288\.08 billed\), preceded by a superseded 133\-session two\-system generation \($66\.48\) and followed by a 60\-session attribution follow\-up \($39\.33\), all separately billed \(disjoint session identifiers\), with the caveat that its RTK “current” arm was an earlier binary lineage \(the pre\-RTK\-ML observer build\), so its results must not be merged with[table˜7](https://arxiv.org/html/2607.12161#S7.T7)\. There, RTK was cost\-neutral \(−1\.3%\-1\.3\\%, n\.s\.\), the observer\-lineage build was6\.5%6\.5\\%*more*expensive \(CI \[1\.5, 11\.7\]\), and Headroom was60\.9%60\.9\\%more expensive, with 131/132 task success\. The qualitative pattern—no free lunch from compression on long sessions, large proxy overhead—is consistent with the authoritative campaign\.

## 8Heterogeneity by Model, Effort, and Task Family

### 8\.1Model×\\timeseffort

Table 12:Model×\\timeseffort matrix: paired billed\-cost change vs\. baseline \(% of same\-cell baseline mean; task\-clustered bootstrap 95% CI;nn= paired tasks\)\. Cells never measured in any campaign are stated as such, not imputed\.- •uexploratory stratum \(n=20n\{=\}20tasks or fewer; the campaign’s power analysis flags such cells and the expansion analysis found arm×\\timeseffort interactions not significant\)\.

[Table˜12](https://arxiv.org/html/2607.12161#S8.T12)and[fig\.˜5](https://arxiv.org/html/2607.12161#S8.F5)present all nine observed model–effort cells\. Three observations matter\. First, Headroom’s penalty is positive in every observed cell \(\+9\.0%\+9\.0\\%to\+110\.2%\+110\.2\\%\)\. Second, RTK effects hover in single digits and flip sign between cells; the expansion campaign’s interaction tests found neither arm×\\timesmodel nor arm×\\timeseffort effects significant, and specifically retired the apparent effort flips—e\.g\., the RTK\-ML Haiku\-medium cell at\+32\.9%\+32\.9\\%\[\+6\.2, \+69\.5\] against the Haiku\-max cell at−21\.2%\-21\.2\\%—as small\-sample artifacts of 20\-task exploratory strata\. We therefore do not promote any effort\-conditioned deployment rule\. Third, coverage is incomplete by design and budget: Sonnet 5 and Opus 4\.8 were never run at medium, xhigh, or max effort in any campaign; those cells are absent from every figure and table rather than estimated\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x5.png)Figure 5:Paired billed\-cost change vs\. baseline by model–effort cell \(task\-clustered 95% CIs\)\. Unmeasured cells \(Sonnet 5 / Opus 4\.8 at medium, xhigh, max\) are annotated, not imputed\.
### 8\.2Task families

Table 13:Task\-family effects for families with≥\\geq3 analyzed tasks: paired billed\-cost change vs\. baseline \(% of family baseline mean; task\-clustered bootstrap 95% CI\)\. One\- and two\-task families are exploratory and reported in Appendix[A\.4](https://arxiv.org/html/2607.12161#A1.SS4)\. Families are descriptive decompositions, not pre\-specified contrasts\.Across the 23 analyzed families \(12 with≥\\geq3 tasks in[table˜13](https://arxiv.org/html/2607.12161#S8.T13); the remainder in Appendix[A\.4](https://arxiv.org/html/2607.12161#A1.SS4);[fig\.˜6](https://arxiv.org/html/2607.12161#S8.F6)shows all\), Headroom’s cost delta is positive in*all*of them \(range\+14\.7%\+14\.7\\%to\+82\.4%\+82\.4\\%\)\. RTK effects are family\-heterogeneous: RTK shows its strongest savings on dependency tracing and multi\-file debugging; RTK\-ML saves modestly on bulk lexical retrieval and shell pipelines while paying on semantic search and multi\-file debugging\. The double\-digit RTK\-ML cells are all one\-to\-four\-task families and are labeled exploratory\. Session length is the strongest task\-side moderator of cost composition \([table˜6](https://arxiv.org/html/2607.12161#S6.T6)\); the expansion analysis found retrieval\-mode and output\-band interactions not significant\. The overall answer to RQ4 is that system effects are*composition\-dependent and heterogeneous*: the earlier campaign’s “weak\-model win” for the RTK\-ML stack was real on its bulk\-retrieval\-heavy mix but did not generalize as a model\-monotonic gradient on the broader read\-only mix\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x6.png)Figure 6:Paired billed\-cost change by task family \(task counts in parentheses; CIs where≥\\geq3 tasks\)\. Headroom is right of zero in every family; RTK effects change sign across families\.

## 9Trajectory and Phase Efficiency

### 9\.1Phase taxonomy and validation

Every retained transcript of the base campaign was decomposed into assistant turns and each turn assigned one of eight phases \(orientation, retrieval, diagnosis, planning, implementation, testing/debug, verification, final response\) plus an ambiguous/mixed catch\-all, using a frozen deterministic priority rule \(edits win over tests, tools over text; no model calls\)\. The ledger covers 13,620 classified turns from 1,120 base campaign runs\. A manual implementation audit independently reproduced 40/40 sampled rule assignments, and 2\.0% of turns \(187/9,523 in the audited slice\) fell into the ambiguous class\. The audit validates that the deterministic rules were applied as specified; it does not establish that the labels match independent human judgments of cognitive phase\. Mixed tool\-using turns are forced into mutually exclusive categories, so the taxonomy characterizes observable turn behavior, not latent reasoning; a blind semantic validation by independent annotators was not performed and is listed as a limitation\. The near\-zero planning share is primarily a property of the frozen classifier, which assigns mixed tool\-using turns to other phases; it should not be read as evidence that planning is absent from coding agents\. Expansion\-split phase claims require regeneration from the retained transcripts, which has not been performed; all phase numbers here are base\-campaign numbers\.

### 9\.2Where generated tokens go

Table 14:Share of generated \(output\) tokens by phase and arm \(characterization split; 10,376 turns\)\. Retrieval plus diagnosis account for≈\\approx58–61% of generated tokens on every arm\.[Table˜14](https://arxiv.org/html/2607.12161#S9.T14)and[fig\.˜7](https://arxiv.org/html/2607.12161#S9.F7)show that coarse phase\-token shares were similar across arms under the frozen turn\-level taxonomy: retrieval≈\\approx35% \(33–36%\), diagnosis≈\\approx24%, implementation 26–29%\. Retrieval plus diagnosis account for approximately 58% of generated tokens \(58\.8%/59\.2%/58\.6%/60\.9% across the four arms\)\. Within this campaign and under the frozen turn\-level taxonomy, the arms differed less in coarse phase composition than in the measured cost and context volume associated with those phases \(delivered tool bytes and the cache traffic they induce\)\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x7.png)Figure 7:Generated\-token composition by phase \(characterization split; 10,376 turns\)\. Coarse phase\-token shares were similar across arms under the frozen turn\-level taxonomy\.
### 9\.3Where savings are created and repaid

Table 15:Phase\-resolved paired cost deltas vs\. baseline \(USD per run; billed cost allocated to turns by price\-weighted token usage; base\-campaign phase ledger\)\. The RTK arms save in retrieval \(P2\); the RTK\-ML arm repays the saving downstream\.The phase\-resolved cost waterfall \([table˜15](https://arxiv.org/html/2607.12161#S9.T15)\) shows both the RTK arms first diverging from baseline at the retrieval phase in the saving direction, while Headroom diverges at orientation already—its cache\-read excess is front\-loaded \(62% of it accumulates before the first edit\) and reaches roughly\+180\+180k cache\-read tokens per run by the final phase\. For the RTK\-ML arm the retrieval saving is repaid approximately fourfold downstream: added diagnosis, implementation, testing/debug, and ambiguous\-turn cost, a later first edit \(\+0\.94\+0\.94turns\), more post\-edit retrieval re\-entries \(\+0\.32\+0\.32/run\), and more repeated\-search tokens \(\+301\+301/run\)\. RTK is the only arm with baseline\-shaped downstream behavior and slightly*earlier*first edits \(−0\.60\-0\.60turns,−0\.26\-0\.26re\-entries\)\. These are observed associations from the frozen ledgers, not causal decompositions\.

Two milestone facts complete the picture \([fig\.˜8](https://arxiv.org/html/2607.12161#S9.F8)\)\. Median cache\-read context at the first code edit is essentially identical for baseline and both RTK arms \(≈\\approx25\.7–25\.9k tokens\) but 49% higher for Headroom \(38\.7k\)\. And on Haiku—the model where the RTK\-ML compression surface is largest—the median estimated delivered tool\-token count before the first edit was 442 for the RTK\-ML arm versus 579 for baseline, approximately 24% lower, with heavy upper tails \(a frozen campaign report characterized the reduction as roughly twofold, but its exact metric, population, and denominator could not be reconstructed from the retained artifacts, so we report only the auditable per\-run ledger median\), while on Sonnet the same arm reached milestones≈\\approx0\.4 turns later\. Reduced delivered retrieval, in other words, was achievable without milestone regression on the weak model, but did not translate into net billed savings once trajectory effects are included\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x8.png)Figure 8:Left: median cache\-read context at the first code edit \(runs with an implementation phase, base splits, all models\)\. Right: median estimated delivered tool tokens before the first edit \(Haiku runs; bytes/4 estimate\)\. Headroom arrives at the first edit≈\\approx49% heavier; the RTK arms match baseline context at the same milestone\.

## 10Failure Modes and Safety Constraints

#### The pipeline\-corruption failure\.

The clearest safety result in the artifact record is a real corruption mode discovered in the first paid gate campaign: when retrieval routing was enabled,*every*genuine gates\-on task failure \(4 of 4\) followed the same pattern—the agent piped proxied search output into a shell counting pipeline \(grep … \| cut \-d: \-f1 \| sort \| uniq \-c\), which then parsed the ranked*search\-group summary*instead of rawfile:line:contentmatches and produced garbage answers \(e\.g\.,src/8instead of a file name\)\. Tool tokens on that campaign fell 40\.3% while gates\-on success fell to 93\.3% against 100% for every other arm: a textbook case of a compression layer optimizing the metric while corrupting the task\. The fix marks a rewritten command segment as a*stdout consumer*whenever its stdout feeds a pipe or a stdout redirect; the proxy then skips ranking for that segment \(output stays byte\-preserving\) and records the skip in the gate trace\. Stderr and stdin redirects still rank; command substitution was already never rewritten\. In subsequent campaigns the guard suppressed 42% \(Haiku\) and 30% \(Sonnet\) of otherwise\-eligible routes, and in the authoritative campaign it recorded 339/150/28 skips against 521/208/26 routes on Haiku/Sonnet/Opus\. The corrected reruns superseded the earlier aggregate verdict; the failure finding stands, and we state it as a design lesson:

> *Dense, load\-bearing evidence such as tracebacks, test output, structured shell streams, and patch anchors requires a different policy from redundant retrieval context\.*

#### The anchor\-destruction failure\.

The grounded\-completion study \([section˜7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\) documents the second corruption class: aggressive compression of retrieved code strips the byte\-exact spans that SEARCH/REPLACE patch application depends on, reducing patch application by approximately 44% \(27/40→\\to15/40; 83% vs\. 48% in the family study\) while looking excellent on token metrics \(−73%\-73\\%billable\)\. The mitigation that worked was content\-aware: the task\-family preserve gate bypasses compression for high\-recall and grounded\-editing families, and byte\-exact grounding windows restore anchors outright \(apply 31%→\\to83% on the compressed arm\)\.

#### Other observed failure surfaces\.

\(i\)*Over\-compression as trajectory inflation*: removed context that the model still needed returned as repeated searches and diagnosis re\-reads \([section˜9](https://arxiv.org/html/2607.12161#S9)\); the cost of recovery is paid at full trajectory rates\. \(ii\)*Early failure looks cheap*: a run that fails quickly can undercut every successful run on cost; all aggregates here therefore report success alongside cost and use cost per successful execution for decisions\. \(iii\)*Inactive code paths create false comparisons*: a free pre\-campaign check proved the gate environment variables are inert in RTK \(byte\-identical output on the covered test paths, no gate trace\) and that the gate build changes bytes only when enabled \(11,050 B rawgrepvs\. 2,570 B search\-group on the fixture\)—without such activation proofs, an “optimized” arm can silently be the baseline\. \(iv\)*Component benchmarks can encourage unsafe optimization*: the pytest\-rule example in[section˜11](https://arxiv.org/html/2607.12161#S11)shows a preservation gate catching a compression rule that scored well on savings while dropping load\-bearing markers\.

#### Gate activation summary\.

[Table˜16](https://arxiv.org/html/2607.12161#S10.T16)condenses the per\-gate activation evidence: every gate is off by default, and the disabled paths covered by the dedicated equivalence tests were byte\-equivalent; the embedding, structural, and focus\-cache paths were enabled and healthy in the authoritative campaign \(zero sidecar outages\), but no task was identified in which any of them was uniquely decisive—activation is not effect\.

### 10\.1Performance degradation under context reduction

Collecting the harm evidence in one place, ordered by directness:

1. 1\.Binary task\-success degradation\.The pre\-guard Stage\-1 campaign: 93\.3% vs\. 100% success \(4/4 failures from pipeline corruption\)\. The grounded study: three tasks solved by grounded raw were not solved by grounded compressed, none reverse \(exact pairedp=0\.25p=0\.25; descriptive\)\. The authoritative campaign detected no aggregate success\-rate degradation, but it has a 96–98% ceiling and was not a non\-inferiority design; its intervals still permit small losses or gains\.
2. 2\.Patch\-application degradation\.27/40→\\to15/40 applies under compression \(Phase E\); 83% vs\. 48% preserve\-vs\-aggressive in the family study;edit\_apply\_failed20 vs\. 12\.
3. 3\.Evidence/marker degradation\.The pre\-fix pytest rule scored 31\.9% savings at only 56\.1% marker recall; ContextBench one\-size\-aggressive reached 39\.8% savings with catastrophic context loss on 45 of 50 conversations\.
4. 4\.Trajectory degradation\.RTK\-ML: later first edit \(\+0\.94\+0\.94turns\), more post\-edit retrieval re\-entries \(\+0\.32\+0\.32/run\), more repeated\-search tokens \(\+301\+301/run\), downstream phase\-cost repayment \([table˜15](https://arxiv.org/html/2607.12161#S9.T15)\)\.
5. 5\.Structured\-consumer corruption\.The stdout\-pipeline failure above: compression corrupted program\-consumed shell semantics, not model comprehension\.
6. 6\.Unmeasured quality dimensions\.Code maintainability, patch quality beyond test outcomes, partial correctness, latent bug introduction, reasoning quality, and human review effort were not measured in any campaign; success here is deterministic test/judge outcome only\.

Table 16:Gate\-activation matrix for RTK\-ML’s nine gates \(condensed\)\. “Fired” cites the authoritative campaign traces \(routes/query\-propagations/guard\-skips on Haiku/Sonnet/Opus: 521/208/26, 916/395/57, 339/150/28; all comparator arms 0/0/0\)\.

## 11Why Component Benchmarks Are Not Enough

The artifact record contains component\-level results for every retrieval and compression approach studied; none of them, alone, would have predicted the end\-to\-end outcomes of[section˜7](https://arxiv.org/html/2607.12161#S7)\.

#### ContextBench \(gold\-context survival\)\.

On the 50\-conversation policy study, an oracle per\-task\-type policy achieved 12\.3% savings at 95\.2% recall with 0/50 catastrophic drops; a one\-size conservative policy achieved 7\.1% at 95\.7% recall with 7/50 catastrophic failures; and a one\-size aggressive policy reached 39\.8% savings at the price of catastrophic context loss on 45 of 50 conversations\. The instructive spread is oracle\-vs\-fixed, not the absolute numbers: safe savings are policy\-dependent and modest\. These are retrieval\-quality proxies; the study records no task success\.

#### LoCoEval \(long\-history preservation\)\.

The retained runs are a preservation\-only proxy \(reference\-answer token survival after compressing long conversations\); the pipeline’s LLM\-judge stage was unavailable, and token savings above the counting API’s size cap fall back to length/4 estimates\. We cite it only as evidence that deterministic command\-level compression is a no\-op on conversational history—a surface it was never built for—and that long\-history compression requires its own track\.

#### InterCode\-proxy \(dense diagnostic evidence\)\.

On 24 pytest observations \(12 MBPP tasks×\\timespass/bug\), the original deterministic pytest rule scored 31\.9% savings at only 56\.1% marker recall—dropping tracebacks’ load\-bearing lines; after a preservation\-first fix, the rule scores 15\.3% savings at 99\.2% marker recall with zero critical drops\. Reducing nominal savings from 31\.9% to 15\.3%*doubled*the fidelity of the evidence the model needs; interactive task\-success evaluation remained blocked by the missing container runtime\.

#### Structural retrieval \(candidate quality\)\.

On 8 repository tasks \(38 query rows\) with test\-grounded gold, plain ripgrep\[[13](https://arxiv.org/html/2607.12161#bib.bib13)\]returned a mean 310\.9 matches / 2,905 tokens per query at 0\.66 precision, against 170\.9 / 2,090 at 0\.995 for the ast\-grep hybrid; on the 14 point\-definition queries the contrast is 75\.6 matches / 752\.8 tokens / 0\.44 precision vs\. 17\.2 / 335\.4 / 0\.99\. Under a 150\-token budget, gold survival was 28\.6% \(ripgrep\) vs\. 85\.7% \(structural\)\. This demonstrates candidate\-quality improvement, not end\-to\-end task\-success improvement: agent\-level evaluation was not performed\.

#### Caveman \(agent\-prose compression\)\.

On seven synthetic fixtures, Caveman achieved 22\.61% real\-token savings \(721→\\to558 tokens by provider count; 25\.84% by bytes—bytes overstate token savings\) with zero critical\-marker drops\. It is a constrained component study on synthetic prose, not an end\-to\-end coding\-agent result\.

#### Embedding retrieval\.

The BGE embedding leg was enabled, health\-checked, and outage\-free throughout the authoritative campaign, and its component\-level union configurations improved recall\-at\-savings on ContextBench slices; but no evaluated task shows a decisive embedding\-attributable effect in the gate traces\. Successful activation is not measured task\-success improvement\.

#### Task success at component scale\.

Even where the component harness could measure task success directly \([section˜7\.4](https://arxiv.org/html/2607.12161#S7.SS4)\), the evaluated compression configuration showed no positive solve\-level return in that small single\-shot study, and a plausible\-looking ratio metric \(successes per million tokens\) would have inverted the verdict; ratio metrics over tiny, non\-overlapping success sets are a reporting hazard, not evidence\.

#### The general lesson\.

Recall is not task success; marker preservation is not task success; candidate precision is not task success; synthetic critical\-drop tests are not task success; bytes removed are not billed savings; and shorter context can cause longer trajectories\. Each component metric is a necessary screen—the InterCode fix and the stdout\-consumer guard both came from component\-level failures—but end\-to\-end, success\-adjusted, actually\-billed evaluation is where claims must terminate\.[Figure˜9](https://arxiv.org/html/2607.12161#S11.F9)summarizes the evidence taxonomy we propose: L1 compression ratio, L2 preservation/retrieval quality, L3 model\-visible context change, L4 production\-path activation, L5 task success, L6 billed cost, L7 cost per successful execution, L8 trajectory\-and\-failure analysis\.

![Refer to caption](https://arxiv.org/html/2607.12161v1/x9.png)Figure 9:Layered evidence taxonomy for token\-efficiency claims\. Every system in[table˜1](https://arxiv.org/html/2607.12161#S3.T1)is placed at the highest layer its retained artifacts support\.

## 12Discussion

#### Addressable share is the budget line\.

The cost stack of[Section˜6](https://arxiv.org/html/2607.12161#S6)bounds what any tool\-output layer can achieve: generated output is≈\\approx11% of the reconstructed cost \(≈\\approx10% of the bill\), uncached input≈\\approx1%, and the remainder of the reconstruction \(≈\\approx87%; about 80% of the actual bill, with the 8\.7% dollar\-weighted residual unattributed\) is cache traffic priced at 10–125% of the input rate\. A layer that touches only delivered tool text competes for a slice of the cache\-read stream that is already discounted10×10\\times\. This does not make compression pointless—it makes*trajectory\-neutral*compression the only kind that pays, and it explains why the largest observed penalties \(Headroom’s\+48%\+48\\%; the RTK\-ML arm’s downstream repayment\) are trajectory\- and cache\-borne rather than output\-borne\.

#### Stage structure vs\. stage cost\.

In these campaigns and under the frozen turn\-level taxonomy, coarse phase\-token shares were similar across arms while phase*prices*moved\. This suggests evaluating interventions as price changes on a fixed pipeline \(retrieval, diagnosis, implementation, verification\), with special attention to re\-entry: the expensive failure is not a big payload but a repeated one\.

#### Addressable versus framework\-owned cache\.

The cache\-dominated bill does not imply a large compression opportunity\. In the baseline arm, the first API call already reads 16\.5k/23\.2k/14\.4k cached tokens \(Haiku/Sonnet/Opus\) before any run content exists—a pre\-existing shared framework prefix—giving a measured lower bound of 76%/83%/85% of all cache\-read volume that is framework\-owned rather than user\-addressable\. Delivered tool output averages only≈\\approx2,122/417/520 estimated tokens per run, of which≈\\approx36% \(Haiku\) flows through the hook\-rewritable shell route\. Bounded scenarios for hook\-addressable cache dollars: conservative≈\\approx4% of cache dollars \(shell\-only surface\), central≈\\approx12% on Haiku \(≈\\approx3% Sonnet,≈\\approx5% Opus\), upper≈\\approx25–30% \(all incremental non\-assistant content\)\. The realized contrast is sobering: the RTK\-ML arm removed 38% of estimated shell tool\-output tokens yet showed cache volumes no lower than baseline \(writes\+3%\+3\\%, reads\+4\.7%\+4\.7\\%per run\)—trajectory response consumed the content savings\. The addressable share also*falls*with trajectory length \(tool volume grows 1\.7×\\timesfrom short to 7\+\-turn runs while cache reads grow 5\.7×\\times\)\. Exact attribution would require per\-request source labeling \(system prompt, tool schemas, user prompt, assistant history, tool output, hook\-injected text, framework metadata\) for every cache write and read; that instrumentation does not exist in the retained campaign\.

#### Heterogeneity, not a winner\.

No system dominated\. RTK was near cost\-neutral in the aggregate campaign, with no detected success\-rate reduction at that study’s precision \(its guard rails—byte\-preserving passthrough when rules do not match—also cap its upside; and its small pooled saving is not holdout\-confirmed,[table˜8](https://arxiv.org/html/2607.12161#S7.T8)\)\. The RTK\-ML stack bought real model\-visible reduction and paid a net cost on the broad mix; its wins concentrated where bulk lexical retrieval dominates and the model is weak\. The tested Headroom v0\.27\.0 configuration showed higher billed cost on the measured workloads without a detected success\-rate difference; the study cannot generalize to other configurations or versions\. Model and effort mattered less than task composition; none of the arm×\\timesmodel or arm×\\timeseffort interactions survived the expansion analysis\.

#### Content\-type policies\.

The safest observed behaviors are content\-aware: preserve\-by\-default for dense evidence \(tracebacks, test output, exact\-byte tasks\), compress redundant retrieval, never transform streams consumed by other programs\. Semantic rankers earn their complexity only where lexical signals fail; the structural results suggest AST\-anchored retrieval is the most promising*precision*lever, but it awaits end\-to\-end validation\.

#### Association vs\. mechanism\.

Our black\-box observations of Headroom \(front\-loaded cache\-read growth, higher re\-entry, the largest billed\-vs\-usage residual\) are associations consistent with several mechanisms \(payload transformation, request handling, retry behavior\); we cannot distinguish them without instrumenting the proxy\. Similarly, the RTK\-ML downstream repayment is an observed pattern in the frozen ledgers, not a proven causal chain\.

## 13Limitations

#### Authoritative campaign\.

One agent harness \(Claude Code\) and one provider’s billing model, prices, and cache semantics at one point in time \(dollar results embed July\-2026 prices; token quantities are reported alongside\)\. Three model families, but Sonnet 5 and Opus 4\.8 were not measured at medium/xhigh/max effort, and Opus cells are thin \(8–30 paired tasks\)\. Success rates sit near a 96–98% ceiling, the study was not designed as a formal non\-inferiority trial, and no pre\-specified non\-inferiority margin exists: success intervals still permit small positive or negative differences, and small performance losses may be undetectable\. Raw \(pre\-layer\) tool\-output boundaries are unobservable for baseline and Headroom\. All within\-block runs fall inside the provider’s five\-minute cache TTL, so cross\-arm cache carryover cannot be excluded by temporal separation \([section˜5\.3](https://arxiv.org/html/2607.12161#S5.SS3)\); randomization balance and null order\-interactions are the available evidence\. The cost reconstruction leaves an 8\.7% dollar\-weighted aggregate residual, with per\-model usage for possible secondary calls not persisted\. Judges are deterministic scripts; graded quality is not measured\.

#### Thinking tokens\.

Thinking tokens were not exposed separately by the API and telemetry used in these campaigns and therefore cannot be reconstructed from the retained artifacts; reasoning\-token usage was not separately observable; generated output and observable diagnosis\-phase behavior provide only partial behavioral indicators and are not direct measurements of latent reasoning\. Post\-hoc evidence \([section˜6\.1](https://arxiv.org/html/2607.12161#S6.SS1)\) indicates the exposure differs by model: Sonnet 5’s aggregate usage reconciles with its bill \(\+0\.2%\+0\.2\\%\), while Haiku 4\.5 carries an effort\-scaling, thinking\-consistent charge inside the disclosed residual; cross\-model decompositions should not assume uniform usage\-field semantics\.

#### Headroom\.

One tested version \(0\.27\.0\) and one configuration \(token mode, caching enabled, code\-aware disabled\); black\-box internals; raw\-vs\-delivered output unobservable; the largest unexplained billing residual \(\+14\.5%\+14\.5\\%\)\. Results cannot be generalized to other configurations or versions\.

#### Vendor CLI compressor andml\_lexical\.

The vendor CLI compressor has no same\-harness end\-to\-end evidence \(vendor\-claimed figures only\)\. Theml\_lexicalengine has component\-level retrieval evidence but no billed task\-success study\.

#### Caveman\.

Seven synthetic fixtures; no production wiring; no real\-traffic evaluation\.

#### Grounded single\-shot study\.

One model \(Sonnet 4\.6\); smallnn\(solve counts of five and two; exact pairedp=0\.25p=0\.25\); no recovery loop; SEARCH/REPLACE\-specific edit application; SWE\-bench\-derived Go tasks; one compression configuration; per\-row costs not retained \(no uncertainty for the per\-resolved\-row cost\); the IC\-SWE pilot’s cost is trace\-derived\.

#### Phase analysis\.

Deterministic rules validated by an implementation audit, not by blind semantic annotation; mutually exclusive labels force mixed turns into single categories; planning is structurally undercounted; base\-campaign splits only\.

#### Binary lineage and composition\.

The long\-session campaign used an earlier RTK lineage and is reported separately throughout\. The task mix is weighted toward retrieval, navigation, and debugging; generalization beyond Claude Code and the tested workloads is not established\.

## 14Recommendations for Benchmark Design

A trustworthy token\-efficiency benchmark for coding agents should report, per paired task: task success; actual billed cost; cache\-creation, cache\-read, uncached\-input, and generated tokens; turns and tool calls; raw and delivered tool volume where observable; retries and failures with exclusion policy; production\-path activation proof; model and effort; task family; cost per successful execution; and confidence intervals with the clustering unit stated\. Aggregates that hide any of these invite the failure modes documented above \(early\-failure “savings”, inactive\-path comparisons, trajectory\-blind ratios\)\.

Because content types fail differently, we recommend separate tracks rather than one blended score: \(i\) redundant bulk retrieval; \(ii\) long\-history memory; \(iii\) dense diagnostic evidence \(tests, tracebacks, logs\) with preservation gates; \(iv\) structural code search; and \(v\) full end\-to\-end software tasks with success\-adjusted billing as the terminal metric\. Component tracks \(i\)–\(iv\) remain essential as screens and for failure analysis—they found both safety bugs reported here—but claims about*efficiency*should be licensed only by track \(v\)\.

## 15Related Work

#### Coding\-agent benchmarks\.

SWE\-bench and its verified subset\[[19](https://arxiv.org/html/2607.12161#bib.bib19),[25](https://arxiv.org/html/2607.12161#bib.bib25)\]established repository\-level issue resolution as the standard end\-to\-end coding task; InterCode\[[32](https://arxiv.org/html/2607.12161#bib.bib32)\]standardized interactive coding with execution feedback; HumanEval and MBPP\[[9](https://arxiv.org/html/2607.12161#bib.bib9),[6](https://arxiv.org/html/2607.12161#bib.bib6)\]anchor function\-level synthesis\. Agent harnesses around these benchmarks\[[33](https://arxiv.org/html/2607.12161#bib.bib33),[29](https://arxiv.org/html/2607.12161#bib.bib29),[28](https://arxiv.org/html/2607.12161#bib.bib28),[34](https://arxiv.org/html/2607.12161#bib.bib34)\]report success and sometimes token counts, but rarely billed cost with cache decomposition;τ\\tau\-bench\[[35](https://arxiv.org/html/2607.12161#bib.bib35)\]adds user\-interaction realism\. Our contribution is orthogonal: a cost\-anatomy and evidence\-standard study over paired arms of the same harness\. Many prior systems are primarily evaluated with component\-level metrics, while fewer report paired task success, provider\-returned usage, trajectory length, and success\-adjusted dollar cost together;[table˜17](https://arxiv.org/html/2607.12161#S15.T17)maps representative prior work onto the evidence layers of[section˜11](https://arxiv.org/html/2607.12161#S11)\.

Table 17:Evidence dimensions reported by representative prior work vs\. this study\. Many prior systems are primarily evaluated with component\-level metrics; entries reflect the cited papers’ primary evaluations, not all follow\-up work\. TS = task success\.
#### Long context and retrieval\.

Long\-context evaluation shows that models use long inputs unevenly\[[23](https://arxiv.org/html/2607.12161#bib.bib23),[7](https://arxiv.org/html/2607.12161#bib.bib7)\], and that retrieval remains competitive with long\-context reading\[[31](https://arxiv.org/html/2607.12161#bib.bib31)\]; retrieval\-augmented generation\[[21](https://arxiv.org/html/2607.12161#bib.bib21)\]and repository\-level retrieval for code\[[37](https://arxiv.org/html/2607.12161#bib.bib37),[16](https://arxiv.org/html/2607.12161#bib.bib16),[12](https://arxiv.org/html/2607.12161#bib.bib12)\]motivate the ranking legs studied here, with BGE embeddings\[[30](https://arxiv.org/html/2607.12161#bib.bib30)\]as the semantic backbone and ast\-grep\[[5](https://arxiv.org/html/2607.12161#bib.bib5)\]for structural search\.

#### Context compression\.

Prompt\- and context\-compression methods\[[17](https://arxiv.org/html/2607.12161#bib.bib17),[18](https://arxiv.org/html/2607.12161#bib.bib18),[22](https://arxiv.org/html/2607.12161#bib.bib22),[24](https://arxiv.org/html/2607.12161#bib.bib24)\]optimize token counts at fixed quality, typically measured by downstream QA accuracy rather than billed agent cost\. Our results argue that for agents, such component gains must additionally clear cache\-pricing and trajectory effects before they become savings\.

#### Serving economics and cost\-aware inference\.

Prompt caching and attention reuse\[[14](https://arxiv.org/html/2607.12161#bib.bib14),[20](https://arxiv.org/html/2607.12161#bib.bib20)\]changed the price structure this paper measures; provider documentation specifies the billed multipliers\[[2](https://arxiv.org/html/2607.12161#bib.bib2),[4](https://arxiv.org/html/2607.12161#bib.bib4)\]\. Cost\-aware cascades and routing\[[8](https://arxiv.org/html/2607.12161#bib.bib8)\]and the efficiency survey ofWan et al\. \[[27](https://arxiv.org/html/2607.12161#bib.bib27)\]treat model choice as the lever; we treat context policy as the lever and show its billed effect is dominated by cache and trajectory terms\. Tool\-use foundations\[[26](https://arxiv.org/html/2607.12161#bib.bib26)\]underlie the agent loop whose economics we measure\.

## 16Conclusion

Token compression and cost reduction are not interchangeable quantities\. In 2,848 paired, provider\-billed coding\-agent runs, cache creation and cache reads dominated the reconstructed cost composition \(≈\\approx87% of the four\-component reconstruction;≈\\approx80% of the actual bill, with an explicitly quantified 8\.7% unattributed residual\), generated output was a minority share, and removing an estimated 38% of raw tool output coexisted with a\+6\.8%\+6\.8\\%paired cost increase as trajectory changes—extra diagnosis, testing, and re\-retrieval turns—repaid the retrieval\-stage savings\. Component benchmarks were indispensable for finding safety failures \(traceback truncation, shell\-pipeline corruption\) but did not predict end\-to\-end outcomes; per\-task tool\-output reduction was a weak, unstable predictor of billed\-cost change\. Effective context policy in these workloads is task\-, content\-, model\-, and effort\-aware: preserve dense evidence, compress redundant retrieval, never transform program\-consumed streams, and evaluate every claim at the level that matters—end\-to\-end, success\-adjusted, actually\-billed cost\. No system we measured is universally best, and we make no such claim for any; the durable output of this study is the measurement standard itself\.

## Conflict of Interest

The authors are affiliated with PointFive, the organization developing RTK and RTK\-ML\. Headroom and the audited vendor CLI compressor are unaffiliated external systems\. To reduce evaluation bias, the study uses frozen task manifests, symmetric deterministic judges, randomized arm order, retained transcripts, append\-only ledgers, and released analysis artifacts; these controls reduce but do not eliminate bias\. All arms, including Headroom, were configured by the authors: Headroom ran its open\-source distribution with default proxy settings \(token mode, caching enabled, code\-aware mode disabled\) and no vendor guidance was used; no vendor was given an opportunity to verify the configuration; the authors had access to RTK internals but not to Headroom’s\.

## Artifact Availability

This paper is accompanied by a benchmark\-and\-data release \(a partial artifact release\): the benchmark harness and deterministic judges, frozen task manifests and pre\-specification documents, arm and run provenance \(including pinned binary hashes\), fixture builders with pinned repository SHAs, the campaign analysis scripts, and the derived run\-level datasets \(per\-run metadata with provider usage and billed cost, the append\-only cost ledger, per\-turn and per\-tool\-call ledgers, phase assignments, and machine\-readable statistical outputs\)\. Raw transcripts are withheld because they embed third\-party repository content and proxied system payloads; derived per\-turn ledgers are released in their place, so the statistical results are computationally reproducible from the release while raw\-trace re\-audits are not\. The release contains no paper text or result narratives; the paper’sLaTeXsources and its figure/table generation scripts accompany the arXiv submission\. The release is available at[https://github\.com/PointFiveLabs/ai\-efficiency\-benchmark](https://github.com/PointFiveLabs/ai-efficiency-benchmark)\(immutable release tagv1\.0\.0\), licensed under CC BY 4\.0\. We do not claim full reproducibility of the campaigns themselves, which would require re\-running paid API workloads\.

## References

- Anonymous \[2026\]Anonymous\.LoCoEval: Long\-conversation evaluation pipeline\.[https://anonymous\.4open\.science/r/LoCoEval](https://anonymous.4open.science/r/LoCoEval), 2026\.Anonymized research artifact; accessed 2026\-06\.
- Anthropic \[2026a\]Anthropic\.Prompt caching — claude developer platform documentation\.[https://platform\.claude\.com/docs/en/build\-with\-claude/prompt\-caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), 2026a\.Accessed 2026\-07\.
- Anthropic \[2026b\]Anthropic\.Claude code documentation\.[https://code\.claude\.com/docs](https://code.claude.com/docs), 2026b\.Accessed 2026\-07\.
- Anthropic \[2026c\]Anthropic\.Claude developer platform pricing\.[https://platform\.claude\.com/docs/en/pricing](https://platform.claude.com/docs/en/pricing), 2026c\.Accessed 2026\-07\.
- ast\-grep contributors \[2026\]ast\-grep contributors\.ast\-grep: A CLI tool for code structural search, lint, and rewriting\.[https://ast\-grep\.github\.io](https://ast-grep.github.io/), 2026\.Version 0\.43\.0; accessed 2026\-07\.
- Austin et al\. \[2021\]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, et al\.Program synthesis with large language models\.*arXiv preprint arXiv:2108\.07732*, 2021\.
- Bai et al\. \[2024\]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\.LongBench: A bilingual, multitask benchmark for long context understanding\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.arXiv:2308\.14508\.
- Chen et al\. \[2023\]Lingjiao Chen, Matei Zaharia, and James Zou\.FrugalGPT: How to use large language models while reducing cost and improving performance\.*arXiv preprint arXiv:2305\.05176*, 2023\.
- Chen et al\. \[2021\]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- ContextBench contributors \[2026a\]ContextBench contributors\.ContextBench dataset\.Hugging Face datasetContextbench/ContextBench, configurationcontextbench\_verified, 2026a\.Dataset revision hash not recorded in the retained artifacts; row identifiers listed in the reproducibility appendix\.
- ContextBench contributors \[2026b\]ContextBench contributors\.SWE\-bench\_pro dataset \(go split\)\.Hugging Face datasetContextbench/SWE\-bench\_Pro, test split, 2026b\.Accessed 2026\-05; Go rows selected by repository language\.
- Feng et al\. \[2020\]Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou\.CodeBERT: A pre\-trained model for programming and natural languages\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, 2020\.
- Gallant \[2026\]Andrew Gallant\.ripgrep: Recursively search directories for a regex pattern\.[https://github\.com/BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep), 2026\.Accessed 2026\-07\.
- Gim et al\. \[2024\]In Gim, Guojun Chen, Seung\-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong\.Prompt cache: Modular attention reuse for low\-latency inference\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.
- Headroom Labs \[2026\]Headroom Labs\.Headroom: An API\-boundary context optimization proxy\.[https://github\.com/headroomlabs\-ai/headroom](https://github.com/headroomlabs-ai/headroom), 2026\.Version 0\.27\.0; open\-source distribution; accessed 2026\-07\.
- Husain et al\. \[2019\]Hamel Husain, Ho\-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt\.CodeSearchNet challenge: Evaluating the state of semantic code search\.*arXiv preprint arXiv:1909\.09436*, 2019\.
- Jiang et al\. \[2023\]Huiqiang Jiang, Qianhui Wu, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\.LLMLingua: Compressing prompts for accelerated inference of large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.
- Jiang et al\. \[2024\]Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\.LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.
- Jimenez et al\. \[2024\]Carlos E\. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\.SWE\-bench: Can language models resolve real\-world GitHub issues?In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2310\.06770\.
- Kwon et al\. \[2023\]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles \(SOSP\)*, 2023\.
- Lewis et al\. \[2020\]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Li et al\. \[2023\]Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin\.Compressing context to enhance inference efficiency of large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.
- Liu et al\. \[2024\]Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024\.
- Mu et al\. \[2023\]Jesse Mu, Xiang Lisa Li, and Noah Goodman\.Learning to compress prompts with gist tokens\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- OpenAI \[2024\]OpenAI\.Introducing SWE\-bench Verified\.[https://openai\.com/index/introducing\-swe\-bench\-verified/](https://openai.com/index/introducing-swe-bench-verified/), 2024\.Accessed 2026\-07\.
- Schick et al\. \[2023\]Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Wan et al\. \[2024\]Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang\.Efficient large language models: A survey\.*Transactions on Machine Learning Research*, 2024\.arXiv:2312\.03863\.
- Wang et al\. \[2024a\]Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji\.Executable code actions elicit better LLM agents\.In*International Conference on Machine Learning \(ICML\)*, 2024a\.arXiv:2402\.01030\.
- Wang et al\. \[2024b\]Xingyao Wang, Boxuan Li, Yufan Song, Frank F\. Xu, Xiangru Tang, Mingchen Zhuge, et al\.OpenHands: An open platform for AI software developers as generalist agents\.*arXiv preprint arXiv:2407\.16741*, 2024b\.
- Xiao et al\. \[2023\]Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff\.C\-Pack: Packaged resources to advance general Chinese embedding\.*arXiv preprint arXiv:2309\.07597*, 2023\.BGE embedding model family\.
- Xu et al\. \[2024\]Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro\.Retrieval meets long context large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2310\.03025\.
- Yang et al\. \[2023\]John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao\.InterCode: Standardizing and benchmarking interactive coding with execution feedback\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.arXiv:2306\.14898\.
- Yang et al\. \[2024\]John Yang, Carlos E\. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press\.SWE\-agent: Agent\-computer interfaces enable automated software engineering\.*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.arXiv:2405\.15793\.
- Yao et al\. \[2023\]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.ReAct: Synergizing reasoning and acting in language models\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Yao et al\. \[2024\]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.*arXiv preprint arXiv:2406\.12045*, 2024\.
- Zan et al\. \[2025\]Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, et al\.Multi\-SWE\-bench: A multilingual benchmark for issue resolving\.*arXiv preprint arXiv:2504\.02605*, 2025\.
- Zhang et al\. \[2023\]Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian\-Guang Lou, and Weizhu Chen\.RepoCoder: Repository\-level code completion through iterative retrieval and generation\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.

## Appendix AReproducibility Appendix

### A\.1Artifact map and campaign hierarchy

The study is based on retained transcripts, ledgers, manifests, gate traces, source code, and generated analysis files; no paid runs were executed for the paper\-writing audit\. The evidence index is the audit directoryresearch/token\_efficiency\_paper\_gap\_audit/\(fifteen deliverables plusgap\_evidence\.json,composition\_data\.json,family\_matrix\.json,phase\_shares\.json, and four evidence\-miner reports\)\. The authoritative campaign lives inresearch/benchmark\_validation\_campaign\_20260707/withdeliverables/EXPANSION\_FINAL\_REPORT\.mdas its final report; its raw ledgers areresults\.jsonl\(2,977 rows\),ledger\_turns\.jsonl\(13,620\),ledger\_tool\_calls\.jsonl\(7,097\),phase\_assignments\.jsonl\(13,620\),COST\_LEDGER\.csv, and the frozen manifests\. Earlier campaigns:research/full\_benchmark\_campaign\_20260705\_150132\_full/\(Stage 0\),research/full\_benchmark\_campaign\_20260701\_133214/\(Stages 1–2\), and the early\-generation campaigns and long\-session campaign under the companion research tree’sbenchmark\_comparison/directory\. Component benchmarks and the single\-shot grounding program live in the benchmark worktree \(cli/harnesses;outputs/agent\_pilot/,outputs/ast\_retrieval/,outputs/intercode\_proxy/\) and the companion outputs tree \(outputs/contextbench/sweeps,outputs/locoeval/,outputs/caveman/, thecb\_e2e\_\*phase reports,benchmark\_sweep\_2026\-06\-07\.md,TEST\_RESULTS\_SUMMARY\.md, andcontextbench\_agent\_smoke\.md\); their extraction into this paper isscripts/extract\_bench\_program\.py→\\todata/bench\_program\.json\.

### A\.2Campaign reconciliation

Table 18:Exact construction of the global execution and spend totals, by evidence class\.The costsmoke comparison is an independent early campaign \(not a subset or duplicate of any other row\); its $1\.01 spend is recorded in its suite results file, and its execution count \(4 tools×\\times2 cases\) is derived from that file rather than from a runtime ledger\. The IC\-SWE pilot’s≈\\approx$0\.37 is derived from per\-turn token fields in its trajectory traces at Sonnet 4\.6 prices\. All other totals sum append\-only per\-run ledger rows\.

### A\.3Normalized four\-component shares

The table below restates the cost composition over the four\-component reconstruction denominator \(the one used by earlier drafts\), excluding the billed residual\.

Table 19:Normalized four\-component shares \(% of the reconstructedC^\\widehat\{C\}, excluding the billed residual\), by slice — the denominator used by earlier drafts and by finer slices \(models, efforts, sessions, outcomes, families\)\. Billed\-basis shares including the residual are in[table˜6](https://arxiv.org/html/2607.12161#S6.T6)\.
### A\.4One\- and two\-task families \(exploratory\)

The table below lists the exploratory family cells excluded from the main\-text family comparison\.

Table 20:Task families with fewer than three analyzed tasks \(descriptive only; no CIs; do not compare against the well\-powered families of[table˜13](https://arxiv.org/html/2607.12161#S8.T13)\)\.
### A\.5Result\-to\-claim traceability

The table below maps each headline claim to its artifact, script, population, and provenance class\.

Table 21:Headline claims mapped to artifacts, scripts, populations, and provenance class \(PS = pre\-specified analysis on the frozen campaign; HC = holdout\-confirmed; PL = pooled; EX = exploratory; DS = descriptive\)\.
### A\.6Coverage, formulas, and unavailable fields

Model×\\timeseffort coverage is exactly the nine cells of[Table˜12](https://arxiv.org/html/2607.12161#S8.T12); Sonnet 5 and Opus 4\.8 at medium/xhigh/max were never measured\. Billed cost is providertotal\_cost\_usd; reconstruction follows[eq\.˜1](https://arxiv.org/html/2607.12161#S2.E1)with Haiku $1/$5, Sonnet 5 $3/$15, Opus 4\.8 $5/$25 per MTok,μr=0\.1\\mu\_\{\\mathrm\{r\}\}\{=\}0\.1,μw=1\.25\\mu\_\{\\mathrm\{w\}\}\{=\}1\.25; the calibration table \(standard vs\. introductory prices, 5\-minute vs\. 1\-hour multipliers\) is in the audit’sCOST\_FIELDS\_AND\_FORMULAS\.md\. Fields unavailable in the retained artifacts: separately\-exposed thinking tokens; per\-tool\-call dollars; per\-model usage splits within a run; raw tool\-output boundaries for baseline/Headroom; time\-to\-first\-correct\-file; Stage\-1 transcripts \(deleted post\-harvest\); expansion\-split phase ledgers \(regenerable from retained gzipped transcripts\)\.

### A\.7Exclusion policy

Billing totals \([table˜3](https://arxiv.org/html/2607.12161#S5.T3)\) count every ledger row \(each row is one billed execution\); analysis populations apply, separately: keep\-last dedupe perrun\_id; requirestatus=completedand no infrastructure failure; drop the calibration split; drop the one manifest\-flagged defective task \(a holdout refactor whose file constraints contradicted its edit requirements\); judge\-defect fixes applied symmetrically by rescoring stored outputs, with append\-only history\. Result: 2,848 analyzed of 2,908 executed runs\.

### A\.8System provenance

Table 22:System versions and provenance \(authoritative campaign\)\.Gate activation: routes/query\-propagations/skips of 521/916/339 \(Haiku\), 208/395/150 \(Sonnet\), 26/57/28 \(Opus\) on the RTK\-ML arm and exactly zero on all comparator arms; free\-fixture byte proof 11,050 B→\\to2,570 B with gates on, byte\-identical with gates off, and byte\-identical with all gate variables set on RTK \(gate\-env inert\)\.

### A\.9Dataset snapshots

Frozen: task manifests \(45 base \+ expansion; 103 analyzed\), six repository pin files \(click16fc00e2…, cobraad460ea8…, expressba006766…, flask36e4a824…, gin34dac209…, requests4c800e9a…\), seeded synthetic monorepo builders, gzipped transcripts for every authoritative run\. Weaker: the ContextBench Hugging Face revision hash was not recorded \(n=10 row identifiers retained: 445, 4, 138, 210, 135, 324, 307, 492, 454, 198\); LoCoEval hop files were fetched on demand; the 50\-row preservation results are retained in the companion outputs tree while some intermediate artifacts resided in temporary directories; several component reports cross\-reference a sibling research tree by absolute path\.

### A\.10Statistical procedure and regeneration

Statistics follow[section˜5\.4](https://arxiv.org/html/2607.12161#S5.SS4)\(task\-clustered pairing; 10,000 bootstrap resamples, seed 7; repository\-clustered and leave\-one\-repository\-out variants; exploratory family screening by CI indicators \(nopp\-values, no FDR control claimed\); ICC/Kish effective sample sizes;n<6n<6cells flagged\)\. All tables and figures in this paper regenerate offline with:

> python3 scripts/extract\_paper\_data\.py python3 scripts/extract\_bench\_program\.py python3 scripts/extract\_thinking\_addressability\.py python3 scripts/make\_tables\.py python3 scripts/make\_figures\.py make paper

### A\.11Integrity statement

No tracked production file was modified during the audit\. The generated research outputs were written under the untracked research directory\. Pre\-existing untracked directories require separate session\-start evidence and are not themselves proof of audit modifications\. No paid API experiments were run for the audit or for this paper; every quantitative statement derives from retained artifacts of the completed campaigns\.

### A\.12Created benchmark corpora and synthetic fixtures

Table 23:Benchmark suites run and synthetic corpora created by the measurement program \(companion research trees; counts from the retained artifacts\)\.

## Appendix BThe Headroom 18/21 Record

An early 21\-task pilot produced an apparent contradiction between “18/21” and “21/21” for Headroom\. The record resolves cleanly: 18/21 is Headroom’s*task\-success*count in that pilot; 21/21 is its*runs\-completed*count in the same campaign—the same denominator with two different numerators\. No retained source claims 21/21 task success\. In that pilot \(single model, one repetition\), the Headroom arm completed 21/21 runs and passed 18/21 tasks against the baseline’s 10/21; the source report itself flags this as a candidate proxy confound requiring replication, since an API\-boundary proxy can also alter request handling\. The subsequent authoritative campaign did not reproduce a success advantage \(success deltas’ CIs cross zero;[Table˜7](https://arxiv.org/html/2607.12161#S7.T7)\) and established a per\-task cost penalty positive in all 23 analyzed task families \(\+14\.7%\+14\.7\\%to\+82\.4%\+82\.4\\%;[Table˜13](https://arxiv.org/html/2607.12161#S8.T13)\) and in every observed model–effort cell \(\+9\.0%\+9\.0\\%to\+110\.2%\+110\.2\\%;[table˜12](https://arxiv.org/html/2607.12161#S8.T12)\)\.

## Appendix CEvidence Boundaries by System

Table 24:Publishable vs\. unsupported claims by system \(condensed from the audit’s evidence\-boundary and unpublishable\-claims deliverables\)\.Table 25:Reproducibility matrix \(condensed\)\.