CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

# CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Source: [https://arxiv.org/html/2606.24083](https://arxiv.org/html/2606.24083)
Morayo Danielle Adeyemi Independent morayo\.danielle@gmail\.com &Ryan A\. Rossi Adobe Research ryrossi@adobe\.com &Franck Dernoncourt Adobe Research franck\.dernoncourt@adobe\.com

###### Abstract

“Talk short\. Drop grammar\. Save token\.”This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel \(the user’s prompt or the model’s response\) is being compressed\. We presentCavewoman, a two\-channel evaluation protocol that scores every generation on task accuracy, realized per\-item cost, and reference\-text agreement against the model’s unconstrained reference\. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items\. Output compression cuts realized cost on most API models \(1\.4–2\.4×\\timesper model, up to3×\\timesin the best case\) and on all four open\-weight models under public\-tier pricing\. Input compression has the opposite effect, a strict lose\-lose: it raises net cost rather than lowering it \(≈1\.15×\\approx 1\.15\\timeson the five\-benchmark mean, up to1\.8×\\timeson the worst dataset and2\.7×\\timesunder stronger compression\), because models compensate with longer responses even as accuracy collapses\. Under the same setting, surface text diverges from the unconstrained reference: on the non\-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model’s own unconstrained baseline generation\. The divergence survives length\-controlled re\-scoring, multiple\-comparisons correction, and replication under complementary semantic measures\. Code and data are available at[https://github\.com/danielle34/cavewoman](https://github.com/danielle34/cavewoman)\.

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Morayo Danielle AdeyemiIndependentmorayo\.danielle@gmail\.comRyan A\. RossiAdobe Researchryrossi@adobe\.comFranck DernoncourtAdobe Researchfranck\.dernoncourt@adobe\.com

![Refer to caption](https://arxiv.org/html/2606.24083v1/x1.png)Figure 1:Cavewomanframework\. Theinput\-compressionchannel applies a deterministic part\-of\-speech filter to the user prompt at five reduction levels, leaving the system prompt fixed\. Theoutput\-compressionchannel leaves the prompt verbatim and replaces the system prompt with a level\-specific instruction that requires the same reduction in the response\. Every generation is scored on task accuracy, reference\-text agreement against the model’s unconstrained reference under bidirectional NLI \(plus eleven complementary measures\), and per\-item input/output token cost\.## 1Introduction

Inference cost in large language models scales with both input and output token counts, and output tokens are typically priced 4–8×\\timeshigher than input tokens\(Ahiaet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib60); Naget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib59)\)\. Existing compression methods reduce one side or the other, either the prompt\(Jianget al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib1); Panet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib3); Brussee,[2026](https://arxiv.org/html/2606.24083#bib.bib57); Peltomäki,[2026](https://arxiv.org/html/2606.24083#bib.bib58)\)or the response\(Xiaet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib15); Songet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib14)\), but the two are studied separately and both are evaluated almost entirely through task accuracy at a reduced token count\. Accuracy at a token count is the wrong instrument for two reasons\. It does not separate realized cost from prompt\-token reduction \(a shorter prompt does not reduce realized cost when the model answers at greater length\), and it collapses each generation to a binary outcome that cannot distinguish a compressed answer that agrees with the model’s unconstrained reasoning from one that diverges from it\.

We address both gaps withCavewoman, a two\-channel evaluation protocol that scores every generation on three axes: task accuracy, realized per\-item cost on the priced channel, and reference\-text agreement against the model’s own unconstrained generation\. The protocol measures*input compression*\(the prompt is filtered before the model sees it\) and*output compression*\(the model is instructed to answer in a constrained register\) at five reduction levels, holding model and item fixed\. We evaluate eight models \(Qwen2\.5\-VL\-7B, Qwen3\.5\-9B, DeepSeek\-R1\-Distill\-Qwen\-7B, Gemma\-4\-E4B, GPT\-4o, GPT\-5\.4, Claude Haiku 4\.5, Claude Sonnet 4\.6\) on five benchmarks \(GSM8K, BoolQ, ARC\-Easy, CommonsenseQA, MMLU\-STEM\) with complete coverage of both channels at all five levels\.

##### Contributions\.

1. 1\.We propose a two\-channel evaluation protocol that scores compression on realized cost with audited answer\-extraction rates and a twelve\-metric semantic battery \(§[3](https://arxiv.org/html/2606.24083#S3)\)\.
2. 2\.We measure cost asymmetry between input and output compression on the same items \(§[4\.1](https://arxiv.org/html/2606.24083#S4.SS1)\)\.
3. 3\.We measure a surface\-text divergence between correct answers and the model’s unconstrained reference under output compression, replicated across complementary semantic measures \(§[4\.2](https://arxiv.org/html/2606.24083#S4.SS2)\)\.

## 2Related Work

##### Input compression\.

Input compression splits along a hard–soft axis\(Liet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib9)\)\. Hard methods prune tokens by self\-information\(Liet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib2)\), perplexity\(Jianget al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib1)\), question\-aware scoring\(Jianget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib4)\), or distilled token classification\(Panet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib3)\), with the same logic at the document level via retrieved\-context summarization\(Xuet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib5)\)\. Soft methods encode prompts as gist tokens\(Muet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib6)\), recursive summary vectors\(Chevalieret al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib7)\), or autoencoder slots\(Geet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib8)\), with rate–distortion bounds inNagleet al\.\([2024](https://arxiv.org/html/2606.24083#bib.bib10)\)\. None measures whether the response says the same thing it would have without compression\.

##### Output compression\.

Output\-side methods constrain generation via modified decoding \(length controls\(Kikuchiet al\.,[2016](https://arxiv.org/html/2606.24083#bib.bib12)\), budget\-signalling positional encodings\(Takase and Okazaki,[2019](https://arxiv.org/html/2606.24083#bib.bib13)\), countdown mechanisms\(Songet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib14)\)\) or via post\-training and prompting \(token\-skipping chains\(Xiaet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib15)\), cognitive\-inspired routing\(Ayteset al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib16)\), difficulty\-aware prompting\(Hanet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib17)\), RL\-driven demonstration compression\(Huanget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib18)\)\)\. The content/function\-word split the caveman register exploits is long\-standing in linguistics, but this line of work still reports only task accuracy at a token budget\.

##### Semantic fidelity, verbosity, and cost\.

Bidirectional NLI entailment separates propositional content from lexical form\(Kuhnet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib19)\)and has scored summarization faithfulness\(Maynezet al\.,[2020](https://arxiv.org/html/2606.24083#bib.bib20)\)and inter\-system consistency\(Labanet al\.,[2022](https://arxiv.org/html/2606.24083#bib.bib21)\); relatedly, chains of thought can be plausible yet causally disconnected from the prediction\(Turpinet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib22); Lanhamet al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib23)\)\. Length itself matters: verbose outputs score lower\(Zhanget al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib24)\), verbosity carries cost\(Borisovet al\.,[2026](https://arxiv.org/html/2606.24083#bib.bib25)\), irrelevant padding degrades reasoning\(Levyet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib26)\), and chain\-of\-thought length tracks accuracy independently of trace correctness\(Jinet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib27); Sunet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib28); Wanget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib29)\)\. The cost–quality frontier is itself a design surface, in selective routing\(Chenet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib30)\), quality thresholds\(Dinget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib31)\), and preference routers\(Onget al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib32)\); valid benchmarking calls for multi\-metric measurement\(Bowman and Dahl,[2021](https://arxiv.org/html/2606.24083#bib.bib33); Lianget al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib34); Gehrmannet al\.,[2021](https://arxiv.org/html/2606.24083#bib.bib35)\)\.

##### Positioning\.

The closest prior work compresses one side only: LLMLingua\(Jianget al\.,[2023](https://arxiv.org/html/2606.24083#bib.bib1); Panet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib3); Jianget al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib4)\)on the input; TokenSkip\(Xiaet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib15)\), Hansel\(Songet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib14)\), and Sketch\-of\-Thought\(Ayteset al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib16)\)on the output, all reporting task accuracy at compressed token budgets\.Cavewomanmeasures both channels on the same items, reports realized per\-item cost rather than token reduction, audits answer\-extraction rates before any accuracy claim, and scores reference\-text agreement under complementary semantic criteria\. The input\-channel divergence reproduces under LLMLingua\-2 \(Appendix[D](https://arxiv.org/html/2606.24083#A4)\)\.

## 3Methodology

### 3\.1Experimental Design

##### Setup\.

Letℳ\\mathcal\{M\}be a fixed language model and letx=\(w1,…,wn\)x=\(w\_\{1\},\\dots,w\_\{n\}\)be a question, a sequence ofnntokens from thespaCytokenizer\. Each tokenwwcarries a Penn Treebank part\-of\-speech tagg​\(w\)g\(w\)assigned byspaCywith theen\_core\_web\_smmodel;ggis deterministic given a fixedspaCyversion and model\.

For a setSSof part\-of\-speech tags, write\(wi\)i:g​\(wi\)∈S\\big\(w\_\{i\}\\big\)\_\{\\,i\\,:\\,g\(w\_\{i\}\)\\in S\}for the subsequence ofxxthat retains exactly the tokens whose tag lies inSS, with the indicesiitaken in increasing order\. Lettrunck​\(z\)\\mathrm\{trunc\}\_\{k\}\(z\)denote the prefix of a sequencezzof lengthmin⁡\(\|z\|,k\)\\min\(\|z\|,k\), and letNN∗\\texttt\{NN\}\{\*\}andVB∗\\texttt\{VB\}\{\*\}denote the Penn Treebank noun and verb tag families\.

We study a single reduction parameter, the levelℓ∈\{0,1,2,3,4\}\\ell\\in\\\{0,1,2,3,4\\\}, named L0 through L4\. The level selects one filter from the familyϕ0,…,ϕ4\\phi\_\{0\},\\dots,\\phi\_\{4\}defined by

ϕ0​\(x\)\\displaystyle\\phi\_\{0\}\(x\)=x,\\displaystyle=x,\(1\)ϕ1​\(x\)\\displaystyle\\phi\_\{1\}\(x\)=\(wi\)i:g​\(wi\)∉\{DT,IN,CC,RP,TO,MD\},\\displaystyle=\\big\(w\_\{i\}\\big\)\_\{\\,i\\,:\\,g\(w\_\{i\}\)\\notin\\\{\\texttt\{DT\},\\texttt\{IN\},\\texttt\{CC\},\\texttt\{RP\},\\texttt\{TO\},\\texttt\{MD\}\\\}\},ϕ2​\(x\)\\displaystyle\\phi\_\{2\}\(x\)=\(wi\)i:g​\(wi\)∈NN⁣∗∪VB⁣∗∪\{CD\},\\displaystyle=\\big\(w\_\{i\}\\big\)\_\{\\,i\\,:\\,g\(w\_\{i\}\)\\in\\,\\texttt\{NN\}\{\*\}\\,\\cup\\,\\texttt\{VB\}\{\*\}\\,\\cup\\,\\\{\\texttt\{CD\}\\\}\},ϕ3​\(x\)\\displaystyle\\phi\_\{3\}\(x\)=\(wi\)i:g​\(wi\)∈NN⁣∗∪\{CD\},\\displaystyle=\\big\(w\_\{i\}\\big\)\_\{\\,i\\,:\\,g\(w\_\{i\}\)\\in\\,\\texttt\{NN\}\{\*\}\\,\\cup\\,\\\{\\texttt\{CD\}\\\}\},ϕ4​\(x\)\\displaystyle\\phi\_\{4\}\(x\)=trunc15​\(ϕ3​\(x\)\)\.\\displaystyle=\\mathrm\{trunc\}\_\{15\}\\big\(\\phi\_\{3\}\(x\)\\big\)\.The family is nested: for every questionxxand everyℓ≥1\\ell\\geq 1,ϕℓ​\(x\)\\phi\_\{\\ell\}\(x\)is a subsequence ofϕℓ−1​\(x\)\\phi\_\{\\ell\-1\}\(x\); a largerℓ\\ellthus applies a stricter reduction\. The box below gives the linguistic interpretation of each level\.

##### Two conditions\.

Cavewomanholdsℳ\\mathcal\{M\}andxxfixed and applies the same reductionϕℓ\\phi\_\{\\ell\}at one of two points \(Figure[1](https://arxiv.org/html/2606.24083#S0.F1)\)\. The question and the system prompt are formatted with the model’s chat template, held fixed throughout\. The L0 baseline is condition\-specific: Condition A uses the neutral system prompt below, while Condition B uses the unconstrained step\-by\-step prompt of Appendix[H\.2](https://arxiv.org/html/2606.24083#A8.SS2); this difference is by design and later matters for the L0\-A/L0\-B noise\-floor limitation\.

Condition A \(input compression\)\.The model receives the filtered questionϕℓ​\(x\)\\phi\_\{\\ell\}\(x\)under a neutral system prompt\. Condition A tests whetherℳ\\mathcal\{M\}needs a full grammatical question\.

Condition B \(output constraint\)\.The model receives the unmodified questionxx, and the system prompt instructs it to answer in the reduced form thatϕℓ\\phi\_\{\\ell\}produces\. Prompts are task\-neutral:ℳ\\mathcal\{M\}infers the answer format fromxxand a final\-lineAnswer: <answer\>convention\. Condition B tests whetherℳ\\mathcal\{M\}needs expressive freedom in its response\.

The levelℓ\\ellthus indexes both conditions through one family of filters: Condition A reduces what the model reads, and Condition B reduces what the model may write\.

### 3\.2Levels of Linguistic Reduction

We define five linguistic reduction levels, with short names we use throughout the paper: the*unconstrained baseline*\(L0\), the*telegraphic*register \(L1\), the*keyword\-only*register \(L2\), the*noun\-phrase skeleton*\(L3\), and the*15\-token budget*\(L4\)\. Each level removes further word classes from the previous level, forming a monotone hierarchy in which a change in accuracy between adjacent levels is attributable to the words removed at that step\. The POS filter is a deterministic, transparent reduction chosen for measurement clarity; learned compressors with task\-aware token scoring \(Appendix[D](https://arxiv.org/html/2606.24083#A4)\) make different trade\-offs\.

Reduction LevelsL0: Unconstrained baseline\.ϕ0\\phi\_\{0\}is the identity and keeps every tag\. Condition A presentsxxverbatim, and Condition B asks for step\-by\-step reasoning in full sentences\.L1: Telegraphic\.ϕ1\\phi\_\{1\}removes the closed\-class tagsDT,IN,CC,RP,TO, andMD: determiners, prepositions, conjunctions, particles, infinitival*to*, and modals\.L2: Keyword only\.ϕ2\\phi\_\{2\}keeps only nouns, verbs, and cardinal numbers\.L3: Noun\-phrase skeleton\.ϕ3\\phi\_\{3\}keeps only nouns and cardinal numbers\.L4: 15\-token budget\.ϕ4\\phi\_\{4\}appliesϕ3\\phi\_\{3\}and then truncates the result to its first1515tokens\.

The same five levels apply to both conditions but enter the pipeline at different points\. In Condition A \(input compression\),ϕℓ\\phi\_\{\\ell\}rewrites the user message via the deterministic POS\-tag filter\. In Condition B \(output constraint\), a level\-specific system prompt instructs the model to produce its response in the matching register; the user message is left intact\. The per\-level decoder budget ismax\_new\_tokens∈\{400,300,200,150,20\}\\texttt\{max\\\_new\\\_tokens\}~\\in\\\{400,300,200,150,20\\\}across L0–L4, identical for both conditions\. Both the verbatim per\-level filter rules \(Condition A\) and the verbatim per\-level system prompts \(Condition B\) are listed in Appendix[H](https://arxiv.org/html/2606.24083#A8)\. L4 is excluded from reference\-text\-agreement scoring since its Condition B system prompt asks the model to emit only the answer; the 15\-token budget is conveyed as a prompt instruction rather than a hard decoder cap, and its soft\-enforcement details are in Appendix[G](https://arxiv.org/html/2606.24083#A7)\. A worked example of one question filtered through all five Condition A levels is in Appendix[H](https://arxiv.org/html/2606.24083#A8)\(Figure[8](https://arxiv.org/html/2606.24083#A8.F8)\)\.

### 3\.3Evaluation Metrics

We score every generation on three axes: task accuracy \(the regex\-extracted answer matches the item’s ground\-truth answer, with a 0\.01 tolerance on numeric answers\), per\-item realized token cost on the priced channel, and reference\-text agreement against the model’s unconstrained generation on the same item and channel\. Reference\-text agreement is operationalized by bidirectional NLI entailment, with a DeBERTa\-based NLI judge\(Heet al\.,[2021](https://arxiv.org/html/2606.24083#bib.bib44)\)as the conservative headline criterion, and replicated under eleven complementary semantic criteria \(Appendix[C](https://arxiv.org/html/2606.24083#A3)\)\.

### 3\.4Dissociation Table

For each levelLxL\_\{x\}we build a2×22\\times 2table that crosses task correctness with*reference\-text agreement against L0*\(Table[1](https://arxiv.org/html/2606.24083#S3.T1)\)\.C2C\_\{2\}andC3C\_\{3\}are distinct outcomes that accuracy\-only evaluation cannot separate; outcome shares are robust to metric choice \(Appendix[C](https://arxiv.org/html/2606.24083#A3)\)\.

Table 1:2×22\\times 2dissociation table, applied at L1–L3 \(L4 excluded from semantic evaluation; see §[3\.2](https://arxiv.org/html/2606.24083#S3.SS2)\)\. “Entails L0” means bidirectional NLI entailment against the*same\-channel*L0 reference\.C2C\_\{2\}is correct answers paired with surface\-text divergence from the same\-channel L0 reference;C3C\_\{3\}is reference\-text agreement despite an incorrect answer\. Accuracy\-only evaluation cannot separate these outcomes\. Operationalized by bidirectional NLI in the main text and replicated under the alternative criteria of Appendix[C](https://arxiv.org/html/2606.24083#A3)\.
### 3\.5Datasets

We use five datasets spanning four task types and three answer formats \(Table[2](https://arxiv.org/html/2606.24083#S3.T2)\): GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.24083#bib.bib36)\), BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2606.24083#bib.bib37)\), ARC\-Easy\(Clarket al\.,[2018](https://arxiv.org/html/2606.24083#bib.bib38)\), CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2606.24083#bib.bib39)\), and the STEM split of MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.24083#bib.bib40)\)\. BoolQ and CommonsenseQA use validation splits; GSM8K, ARC\-Easy, and MMLU\-STEM use test splits; MMLU is restricted to 20 STEM subjects\.

Table 2:Datasets used inCavewoman, spanning math word problems \(GSM8K\), passage yes/no \(BoolQ\), science multiple\-choice \(ARC\-Easy\), commonsense multiple\-choice \(CommonsenseQA\), and STEM multiple\-choice \(MMLU\-STEM\)\. A single model is evaluated on 11,465 items at five reduction levels under both conditions\.
### 3\.6Models

We evaluate Qwen2\.5\-VL\-7B\(Baiet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib46)\), Qwen3\.5\-9B\(Yanget al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib47)\), DeepSeek\-R1\-Distill\-Qwen\-7B\(Guoet al\.,[2025](https://arxiv.org/html/2606.24083#bib.bib49)\), Gemma\-4\-E4B\(Google,[2026](https://arxiv.org/html/2606.24083#bib.bib48)\), GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib50)\), GPT\-5\.4\(OpenAI,[2026](https://arxiv.org/html/2606.24083#bib.bib55)\), Claude Haiku 4\.5\(Anthropic,[2025](https://arxiv.org/html/2606.24083#bib.bib51)\), and Claude Sonnet 4\.6\(Anthropic,[2026](https://arxiv.org/html/2606.24083#bib.bib56)\)\. All eight models are evaluated on every benchmark under both conditions\. Qwen2\.5\-VL\-7B is included on these text\-only benchmarks because the evaluation uses its text backbone in ordinary chat mode rather than any vision input path\. Two reasoning\-protocol details affect interpretation: DeepSeek\-R1 emits hidden<think\>traces that count against the same token budget as its visible output, and Qwen3\.5\-9B has an optional thinking mode that we leave off \(the model default\)\. We nonetheless group Qwen3\.5\-9B with the reasoning models because, like the distilled reasoner, its unconstrained generations are already short and terse, the property that governs how much surface text can diverge under output compression\. Mechanics and the exclusion of a third reasoning model \(Kimi\-K2\.6Kimi Team \([2026](https://arxiv.org/html/2606.24083#bib.bib61)\)\) are in Appendix[E](https://arxiv.org/html/2606.24083#A5); full inference configuration is in Appendix[A\.3](https://arxiv.org/html/2606.24083#A1.SS3)\.

## 4Results

![Refer to caption](https://arxiv.org/html/2606.24083v1/x2.png)Figure 2:Answer accuracy across the five reduction levels for all models and benchmarks\. Solid bars denote input compression; hatched bars denote output compression\. Significance markers indicate Wilcoxon signed\-rank tests against the within\-model unconstrained baseline \(p∗<\.05\{\}^\{\*\}p\{<\}\.05,p∗∗<\.01\{\}^\{\*\*\}p\{<\}\.01,p∗⁣∗∗<\.001\{\}^\{\*\*\*\}p\{<\}\.001\)\. The dashed gold line marks the random\-guessing baseline for each benchmark\. Kimi\-K2\.6 is shown for completeness only and is excluded from all aggregates \(Appendix E\)\.### 4\.1Finding 1: Cost asymmetry between channels

Finding 1\.Output compression cuts realized cost on most API models \(1\.4–2\.4×\\timesper model, up to3×\\timeson the best cell; cheaper on 17 of 20 ‘\(model, dataset\)’ cells\) and all four open\-weight models\. Input compression instead raises net cost through compensatory output expansion: up to1\.15×\\timeson the five\-benchmark mean and1\.8×\\timeson the worst individual dataset, growing to2\.7×\\timesat deeper reductions as accuracy collapses\.

Per\-item cost for an API\-served model isC=nin​pin\+nout​poutC=n\_\{\\text\{in\}\}\\,p\_\{\\text\{in\}\}\+n\_\{\\text\{out\}\}\\,p\_\{\\text\{out\}\}, whereninn\_\{\\text\{in\}\}andnoutn\_\{\\text\{out\}\}are the input and output token counts andpinp\_\{\\text\{in\}\},poutp\_\{\\text\{out\}\}the corresponding per\-token prices \(May 2026\)\. Figure[3](https://arxiv.org/html/2606.24083#S4.F3)reports the relative change in realized cost against the same\-channel unconstrained baseline\. Output compression reduces realized per\-item cost on GPT\-4o, Claude Haiku 4\.5, and Claude Sonnet 4\.6 by the per\-model margins above, cheaper on every one of their fifteen benchmarks; the exception is GPT\-5\.4, whose billed output is dominated by hidden reasoning tokens \(Appendix[E](https://arxiv.org/html/2606.24083#A5)\), so it is cheaper on only two of its five benchmarks \(17 of 20 API cells in all\)\. Input compression at the same reduction levels raises net cost on the remaining models \(GPT\-4o aside\) before accuracy collapses at the strictest level \(Figure[2](https://arxiv.org/html/2606.24083#S4.F2)\)\. This is a strict lose\-lose: the telegraphic level already raises net cost \(up to 1\.8×\\timeson individual datasets\) while degrading accuracy, and at deeper reduction levels the worst\-case penalty grows to∼\\sim2\.7×\\timesas accuracy collapses to single digits; the input channel raises cost and lowers accuracy at the same time\.

![Refer to caption](https://arxiv.org/html/2606.24083v1/x3.png)Figure 3:Relative change in estimated per\-item inference cost against the unconstrained baseline, averaged across the four API models\. Left panel: input compression\. Right panel: output compression\. Rows are benchmarks; columns are the four non\-zero reduction levels\. Red denotes a cost increase, green a cost reduction\. The two channels move in opposite directions at the same reduction level\. Worst\-case input\-channel penalties reach1\.8×1\.8\\timesat L1 and∼2\.7×\\sim 2\.7\\timesat deeper reductions \(Finding 1\)\.##### The mechanism is compensatory output expansion\.

Stripping function words from the prompt saves a small number of input tokens, but the model answers at greater length and output tokens cost several times more than input tokens; the net change is positive\. Every API model saves input tokens but spends more on output, and the priced ratio between the two leaves net cost higher on every API model but GPT\-4o, where the two effects roughly cancel \(Table[3](https://arxiv.org/html/2606.24083#S4.T3)\)\. Output compression instead saves on the output side, which dominates cost: the most favorable case \(GPT\-4o on ARC\-Easy\) is roughly three times cheaper at the same accuracy, and on GSM8K the same model trades a small accuracy gain against a halving of cost\. The saving requires that the priced output actually shrink under the constraint, which holds for every model whose billed output matches its visible response\.

Table 3:Per\-model token economics under input compression at the telegraphic level, averaged across the five benchmarks\. Input tokens fall on every model; output tokens rise on every model; net cost is positive on all but one configuration\.
##### Apparent accuracy gains can be a parser artifact\.

Answer\-extraction rates can inflate apparent accuracy gains under compression\. MMLU\-STEM provides the clearest example, several API models have lower answer\-extraction rates under the unconstrained setting than under compressed output, so apparent improvements partly reflect easier parsing rather than better reasoning\. To avoid attributing parser recovery to the model itself, we only report “compressed exceeds unconstrained” gains when the unconstrained extraction rate is at least 0\.95, and flag the remaining cases as extractor\-suppressed \(Appendix[G](https://arxiv.org/html/2606.24083#A7)\)\.

##### Scope of the cost win\.

Output\-channel cost savings are largest on benchmarks whose ground\-truth answer is already short \(BoolQ yes/no, MCQ\-letter\): the L1 instruction collapses the response to roughly the answer plus minimal reasoning, and savings are partly mechanical\. The per\-model savings \(mean1\.41\.4–2\.4×2\.4\\timeson the three models that save\) should therefore be read against this format\-collapse component; on GSM8K, where the answer requires multi\-step arithmetic, savings are smaller but accuracy is preserved on the cells with≥0\.95\\geq 0\.95L0 parse rate\. Apparent single\-cell gains on MMLU\-STEM \(e\.g\. Gemma\-4\-E4B,\+35\.3\+35\.3pp\) are confounded by answer\-extraction recovery \(the L0\-B parse rate rises from0\.490\.49to0\.880\.88at L1\) and are reported as extractor\-suppressed rather than reasoning gains \(Appendix[G](https://arxiv.org/html/2606.24083#A7)\)\.

Under public\-tier pricing, all four open\-weight models also save cost at L1 under output compression, in the same direction as the API panel \(Appendix[A\.6](https://arxiv.org/html/2606.24083#A1.SS6)\)\.

#### 4\.1\.1Channel\-specific degradation

The two channels also degrade differently\. Under input compression, accuracy and reference\-text agreement fall together as the level increases; under output compression, classification accuracy holds through deep levels while reference\-text agreement falls sharply at the first level\. The input channel spends accuracy; the output channel spends agreement with the unconstrained reference\. The output\-channel pattern motivates Finding 2\.

### 4\.2Finding 2: Accuracy decouples from same\-channel reference text under output compression

Finding 2\.Across the six non\-reasoning models,51\.9%\\mathbf\{51\.9\\%\}of all L1 output\-compression generations are correct yet have surface text that no longer matches what the model would have written without the constraint on the six\-non\-reasoning panel; under length\-matched re\-scoring on the same panel, the rate rises to80\.4%\\mathbf\{80\.4\\%\}\.

The headline rate is the share of all generations at L1 output compression that are correct yet no longer entail the same\-channel unconstrained reference under a bidirectional NLI judge \(theC2C\_\{2\}cell of Table[1](https://arxiv.org/html/2606.24083#S3.T1)\)\. Pooled across the six non\-reasoning models in our panel, the rate is 51\.9% on the six\-non\-reasoning panel\. The two reasoning models in the panel \(DeepSeek\-R1\-Distill and Qwen3\.5\-9B\) show a smaller divergence since their unconstrained generations are already short \(per\-model values in Appendix[C\.3](https://arxiv.org/html/2606.24083#A3.SS3)\)\. On every non\-reasoning model, the dominant off\-diagonal outcome is correct\-but\-divergent rather than incorrect\-but\-faithful\.

##### Accuracy holds while the reasoning trace drifts\.

Figure[4](https://arxiv.org/html/2606.24083#S4.F4)resolves L1 output compression into its2×22\{\\times\}2outcome cells per model\. The amberC2C\_\{2\}segment is the dissociation: correct answers whose surface text no longer entails the model’s same\-channel L0 reference\. On every non\-reasoning model theC2C\_\{2\}band is the dominant off\-diagonal cell; DeepSeek\-R1 inverts the pattern \(Appendix[E](https://arxiv.org/html/2606.24083#A5)\)\.

![Refer to caption](https://arxiv.org/html/2606.24083v1/x4.png)Figure 4:Per\-model2×22\{\\times\}2dissociation at L1 output compression, summed across the five benchmarks\. Each stacked bar partitions L1 output\-compression generations into the four outcome cells:C1C\_\{1\}\(correct \+ entails L0\),C2C\_\{2\}\(correct \+ does not entail L0\),C3C\_\{3\}\(wrong \+ entails L0\), andC4C\_\{4\}\(wrong \+ does not entail L0\)\. Accuracy isC1\+C2C\_\{1\}\{\+\}C\_\{2\}and bidirectional NLI rate isC1\+C3C\_\{1\}\{\+\}C\_\{3\}; the amberC2C\_\{2\}share is the dissociation\. DeepSeek\-R1 inverts the pattern\. Kimi\-K2\.6 is shown for completeness only and is excluded from all aggregates \(Appendix E\)\.
##### Length is not the explanation\.

Length\-matched re\-scoring \(truncating L0 to the L1\-B wordpiece\-token length\)*increases*the divergence on every non\-reasoning model \(Table[16](https://arxiv.org/html/2606.24083#A3.T16), Appendix[C\.3](https://arxiv.org/html/2606.24083#A3.SS3)\); the headline is the conservative reading\.

##### Robustness across judges, metrics, and statistics\.

The bidirectional NLI judge has a calibrated false\-negative rate of2\.9%2\.9\\%at L1 on synthetic positive pairs \(Appendix[C\.1](https://arxiv.org/html/2606.24083#A3.SS1)\); all twelve semantic measures report a substantial divergence, with our headline sitting near the conservative end of the family \(Table[15](https://arxiv.org/html/2606.24083#A3.T15)\)\. Wilcoxon signed\-rank tests against the within\-model, within\-channel unconstrained baseline survive Benjamini–Hochberg correction atα=0\.05\\alpha=0\.05on every significant cell \(Appendix[F](https://arxiv.org/html/2606.24083#A6)\)\.

##### Generalization and task\-type dependence\.

The divergence also holds under the learned LLMLingua\-2 compressor\(Panet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib3)\):C2\>0C\_\{2\}\>0on every cell atτ=0\.5\\tau\{=\}0\.5, with rate\-driven recovery atτ=0\.8\\tau\{=\}0\.8\(Appendix[D](https://arxiv.org/html/2606.24083#A4)\)\. The divergence is smaller on arithmetic benchmarks than on classification \(Table[13](https://arxiv.org/html/2606.24083#A2.T13)\)\. Register traces across L1–L4 are in Appendix[I](https://arxiv.org/html/2606.24083#A9)\.

### 4\.3Robustness to compression varies across models

Finding 3\.Relative robustness \(telegraphic output\- vs\. input\-constraint accuracy\) spans a1\.5×\\mathbf\{1\.5\\times\}to3\.7×\\mathbf\{3\.7\\times\}range across models and is not predicted by parameter count or unconstrained accuracy\.

Let robustness be the ratio of accuracy under the telegraphic output constraint to accuracy under the telegraphic input constraint, at the same item and the same reduction level\. The ratio is reported per model in Table[4](https://arxiv.org/html/2606.24083#S4.T4)\. The most output\-robust model overall is the open\-weight Gemma\-4\-E4B, and the most output\-robust API model \(Claude Sonnet 4\.6\) retains nearly three times as much accuracy under the output constraint as under the input constraint; the highest\-accuracy API model on the unconstrained baseline \(GPT\-5\.4\) is least robust among the four API models \(second\-least across all eight\)\. Gemma\-4\-E4B tops the full ranking, and Qwen2\.5\-VL\-7B outranks two of the four API models\. DeepSeek\-R1’s value should be read with the reasoning\-token caveat of Appendix[E](https://arxiv.org/html/2606.24083#A5), since its hidden chain\-of\-thought tokens are charged against the same visible\-output budget; with two API vendors at two models each, we draw no vendor\-level conclusion\. The practical implication is that a deployment should rank candidates at the constraint level under which it will be deployed, not at the unconstrained baseline\.

Table 4:Output\-vs\-input accuracy ratio at the telegraphic level, per model\. Higher values indicate greater robustness to output compression\. Parameter count does not predict the ordering\.

## 5Discussion

Output is the channel to compress on when the model answers at length \(Finding 1\)\. It produces real cost savings on every API model whose billed output matches its visible response, while input compression does not\.

Robustness to output compression does not follow from unconstrained accuracy\. The highest\-accuracy API model in our panel is least robust among the four API models \(second\-least across all eight\), and a 7B open\-weight model outranks two of the four API models under the output constraint\.

The accuracy/reference\-text divergence is a surface\-text observation, not a propositional claim\. Findings 1 and 2 answer different questions and can both hold: correctness is graded against the ground\-truth answer, while reference\-text agreement is graded against the model’s own L0 generation\. For deployments that consume only the final answer, the divergence does not affect outcomes; for deployments that consume the generation as text \(transcripts, audit trails, reasoning displays\), it does\.

These findings suggest that single\-axis evaluation of compression is underdetermined\. Realised cost on the priced channel is not reducible to prompt\-token reduction; observed accuracy is conditional on extraction\-rate reliability; and reference\-text agreement is conditional on the choice of semantic axis\. The three quantities dissociate in our panel; any composite metric that aggregates them can therefore mis\-rank methods on the dimension that ultimately determines deployment cost\.

## 6Conclusion

Compressing language\-model inference is a two\-channel problem, and accuracy alone cannot tell the channels apart\. Output compression cuts realized cost on most API models \(1\.4–2\.4×\\timesper model, up to 3×\\timeson the best cell, at the first reduction level\) and all four open\-weight models under public\-tier pricing; input compression instead raises net cost through compensatory output expansion \(up to∼\\sim15% on the five\-benchmark mean and 1\.8×\\timeson individual datasets, growing to 2\.7×\\timesat deeper reductions as accuracy collapses\)\. On the same cost\-saving settings, 51\.9% of generations on the six\-non\-reasoning panel are correct yet their surface text no longer matches the model’s unconstrained reference, a divergence that strengthens under length\-controlled re\-scoring and replicates across complementary semantic measures\. Robustness to output compression varies widely across models and is not predicted by parameter count or unconstrained accuracy; candidates should therefore be ranked at the constraint level they will be deployed at, not at the unconstrained baseline\.

### 6\.1Limitations

Our bidirectional NLI judge measures surface\-text divergence rather than propositional drift; we bound the length\-and\-register confound through length\-controlled re\-scoring \(Appendix[C\.3](https://arxiv.org/html/2606.24083#A3.SS3)\) and replicate under eleven complementary measures plus the headline judge \(Appendix[C](https://arxiv.org/html/2606.24083#A3)\), but the rate is not a propositional\-content claim\. Part of the Condition B divergence may also reflect the system\-prompt register change between conditions; the L0\-A/L0\-B noise floor that would isolate this is not separately measured \(Appendix[C\.3](https://arxiv.org/html/2606.24083#A3.SS3)\)\. All five benchmarks have short, structured answers \(numeric, boolean, MCQ letter\); we make no claim about content preservation in long\-form generation tasks such as summarization or open\-ended QA\. The eight\-model panel uses greedy decoding only, with two API vendors at two models each; we therefore draw no vendor\- or family\-level conclusion; sampled decoding and a hard\-decoder L4 are out of scope\. Full per\-confound discussion is in Appendix[F](https://arxiv.org/html/2606.24083#A6)\.

## Acknowledgments

We thank MIT Engaging for providing the GPU compute used for the local\-model runs\.

## References

- Do all languages cost the same? tokenization in the era of commercial language models\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=OUmxBN45Gl)Cited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1)\.
- Anthropic \(2025\)System card: claude haiku 4\.5\.Technical reportAnthropic\.Note:System card documenting improvements and model safety testing for Claude Haiku 4\.5External Links:[Link](https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf)Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- Anthropic \(2026\)System card: claude sonnet 4\.6\.Technical reportAnthropic\.Note:Technical system card documenting model safeguards, model characteristics, and deployment of Claude Sonnet 4\.6External Links:[Link](https://www-cdn.anthropic.com/bbd8ef16d70b7a1665f14f306ee88b53f686aa75.pdf)Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- S\. A\. Aytes, J\. Baek, and S\. J\. Hwang \(2025\)Sketch\-of\-thought: efficient llm reasoning with adaptive cognitive\-inspired sketching\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 24307–24331\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025\)Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- V\. Borisov, M\. Gröger, M\. Mikhael, and R\. H\. Schreiber \(2026\)Do chatbot llms talk too much? the yapbench benchmark\.arXiv preprint arXiv:2601\.00624\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Bowman and G\. Dahl \(2021\)What will it take to fix benchmarking in natural language understanding?\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4843–4855\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Brussee \(2026\)Caveman\.Note:[https://github\.com/juliusbrussee/caveman](https://github.com/juliusbrussee/caveman)GitHub repositoryCited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1)\.
- L\. Chen, M\. Zaharia, and J\. Zou \(2024\)FrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=cSimKw5p6R)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Chevalier, A\. Wettig, A\. Ajith, and D\. Chen \(2023\)Adapting language models to compress contexts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 3829–3846\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)Boolq: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 \(long and short papers\),pp\. 2924–2936\.Cited by:[§3\.5](https://arxiv.org/html/2606.24083#S3.SS5.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§3\.5](https://arxiv.org/html/2606.24083#S3.SS5.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§3\.5](https://arxiv.org/html/2606.24083#S3.SS5.p1.1)\.
- D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Rühle, L\. Lakshmanan, and A\. H\. Awadallah \(2024\)Hybrid llm: cost\-efficient and quality\-aware query routing\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 41348–41366\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Ge, H\. Jing, L\. Wang, X\. Wang, S\. Chen, and F\. Wei \(2024\)In\-context autoencoder for context compression in a large language model\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uREj4ZuGJE)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Gehrmann, T\. Adewumi, K\. Aggarwal, P\. S\. Ammanamanchi, A\. Aremu, A\. Bosselut, K\. R\. Chandu, M\. Clinciu, D\. Das, K\. Dhole,et al\.\(2021\)The gem benchmark: natural language generation, its evaluation and metrics\.InProceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM 2021\),pp\. 96–120\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- Google \(2026\)Gemma 4: our most capable open models to date\.Note:[https://blog\.google/innovation\-and\-ai/technology/developers\-tools/gemma\-4/](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/)Google BlogCited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- T\. Han, Z\. Wang, C\. Fang, S\. Zhao, S\. Ma, and Z\. Chen \(2025\)Token\-budget\-aware llm reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 24842–24855\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1)\.
- P\. He, X\. Liu, J\. Gao, and W\. Chen \(2021\)\{deberta\}: \{decoding\}\-\{enhanced\} \{bert\} \{with\} \{disentangled\} \{attention\}\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=XPZIaotutsD)Cited by:[§3\.3](https://arxiv.org/html/2606.24083#S3.SS3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§3\.5](https://arxiv.org/html/2606.24083#S3.SS5.p1.1)\.
- X\. Huang, L\. L\. Zhang, K\. Cheng, F\. Yang, and M\. Yang \(2024\)Fewer is more: boosting math reasoning with reinforced context pruning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 13674–13695\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu \(2023\)LLMLingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13358–13376\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.825/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.825)Cited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1)\.
- H\. Jiang, Q\. Wu, X\. Luo, D\. Li, C\. Lin, Y\. Yang, and L\. Qiu \(2024\)Longllmlingua: accelerating and enhancing llms in long context scenarios via prompt compression\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1658–1677\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Jin, Q\. Yu, D\. Shu, H\. Zhao, W\. Hua, Y\. Meng, Y\. Zhang, and M\. Du \(2024\)The impact of reasoning step length on large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 1830–1842\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Kikuchi, G\. Neubig, R\. Sasano, H\. Takamura, and M\. Okumura \(2016\)Controlling output length in neural encoder\-decoders\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1328–1338\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1)\.
- Kimi Team \(2026\)Kimi k2\.6 technical report\.Moonshot AI\.External Links:[Link](https://platform.kimi.ai/docs/overview)Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VD-AYtP0dve)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. Hearst \(2022\)SummaC: re\-visiting nli\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics10,pp\. 163–177\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion,et al\.\(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Levy, A\. Jacoby, and Y\. Goldberg \(2024\)Same task, more tokens: the impact of input length on the reasoning performance of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15339–15353\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, B\. Dong, F\. Guerin, and C\. Lin \(2023\)Compressing context to enhance inference efficiency of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 6342–6353\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Li, Y\. Liu, Y\. Su, and N\. Collier \(2025\)Prompt compression for large language models: a survey\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7182–7195\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Re, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. WANG, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. S\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. A\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Note:Featured Certification, Expert Certification, Outstanding CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=iO4LZibEqW)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald \(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 1906–1919\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Mu, X\. Li, and N\. Goodman \(2023\)Learning to compress prompts with gist tokens\.Advances in Neural Information Processing Systems36,pp\. 19327–19352\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Nag, A\. Mukherjee, N\. Ganguly, and S\. Chakrabarti \(2024\)Cost\-performance optimization for processing low\-resource language tasks using commercial LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 15681–15701\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.920/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.920)Cited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1)\.
- A\. Nagle, A\. Girish, M\. Bondaschi, M\. Gastpar, A\. V\. Makkuva, and H\. Kim \(2024\)Fundamental limits of prompt compression: a rate\-distortion framework for black\-box language models\.Advances in Neural Information Processing Systems37,pp\. 94934–94970\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica \(2025\)RouteLLM: learning to route LLMs from preference data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8sSqNntaMr)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- OpenAI \(2026\)Introducing GPT\-5\.4: designed for professional work\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4/](https://openai.com/index/introducing-gpt-5-4/)OpenAI BlogCited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin,et al\.\(2024\)Llmlingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 963–981\.Cited by:[Appendix D](https://arxiv.org/html/2606.24083#A4.p1.3),[§1](https://arxiv.org/html/2606.24083#S1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2606.24083#S4.SS2.SSS0.Px4.p1.3)\.
- W\. Peltomäki \(2026\)Caveman compression\.Note:[https://github\.com/wilpel/caveman\-compression](https://github.com/wilpel/caveman-compression)GitHub repositoryCited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/),[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[Table 7](https://arxiv.org/html/2606.24083#A1.T7.4.8.3.2)\.
- S\. Song, J\. Lee, and H\. Ko \(2025\)Hansel: output length controlling framework for large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 25146–25154\.Cited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Sun, H\. Wang, J\. Li, J\. Liu, X\. Li, H\. Wen, Y\. Yuan, H\. Zheng, Y\. Liang, Y\. Li,et al\.\(2025\)An empirical study of llm reasoning ability under strict output length constraint\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 7663–7682\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Takase and N\. Okazaki \(2019\)Positional encoding to control output sequence length\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 3999–4004\.External Links:[Link](https://aclanthology.org/N19-1401/),[Document](https://dx.doi.org/10.18653/v1/N19-1401)Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)Commonsenseqa: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[§3\.5](https://arxiv.org/html/2606.24083#S3.SS5.p1.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.Advances in Neural Information Processing Systems36,pp\. 74952–74965\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wang, S\. Jain, D\. Zhang, B\. Ray, V\. Kumar, and B\. Athiwaratkun \(2024\)Reasoning in token economies: budget\-aware evaluation of llm reasoning strategies\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 19916–19939\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Xia, C\. T\. Leong, W\. Wang, Y\. Li, and W\. Li \(2025\)Tokenskip: controllable chain\-of\-thought compression in llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3351–3363\.Cited by:[§1](https://arxiv.org/html/2606.24083#S1.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px4.p1.1)\.
- F\. Xu, W\. Shi, and E\. Choi \(2024\)Recomp: improving retrieval\-augmented lms with context compression and selective augmentation\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 43478–43502\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.6](https://arxiv.org/html/2606.24083#S3.SS6.p1.1)\.
- Y\. Zhang, S\. S\. S\. Das, and R\. Zhang \(2025\)Demystify verbosity compensation behavior of large language models\.InProceedings of the 2nd Workshop on Uncertainty\-Aware NLP \(UncertaiNLP 2025\),pp\. 160–178\.Cited by:[§2](https://arxiv.org/html/2606.24083#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AImplementation Details

Pricing, dataset statistics and licenses, inference configuration, token accounting, cost estimates, and the L1 decoder\-truncation check\. Verbatim system prompts and POS\-tag rules are in Appendix[H](https://arxiv.org/html/2606.24083#A8); released artifacts in Appendix[J](https://arxiv.org/html/2606.24083#A10)\.

### A\.1Pricing Assumptions and Model Snapshots

May 2026 per\-token API prices for the four closed models \(Table[5](https://arxiv.org/html/2606.24083#A1.T5)\)\. Per\-item API cost isC=nin⋅pin\+nout⋅poutC=n\_\{\\text\{in\}\}\\cdot p\_\{\\text\{in\}\}\+n\_\{\\text\{out\}\}\\cdot p\_\{\\text\{out\}\}, with tokens counted by each model’s own tokenizer\. Open\-weight models run on local GPUs and are excluded from the dollar\-cost analysis; token accounting is in Appendix[B](https://arxiv.org/html/2606.24083#A2)\.

ModelInputOutputSnapshot / endpoint\($/M tokens\)GPT\-4o2\.5010\.00gpt\-4o\-2024\-11\-20GPT\-5\.41\.2510\.00gpt\-5\-2025\-08\-07Claude Haiku 4\.51\.005\.00claude\-haiku\-4\-5\-20251001Claude Sonnet 4\.63\.0015\.00claude\-sonnet\-4\-6\-20250929Qwen 2\.5\-VL\-7Blocal GPUQwen/Qwen2\.5\-VL\-7B\-InstructQwen 3\.5\-9Blocal GPUQwen/Qwen3\.5\-9BDeepSeek\-R1\-Distill 7Blocal GPUdeepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-7BGemma\-4\-E4Blocal GPUgoogle/gemma\-4\-e4bTable 5:Per\-token API pricing \(USD/M tokens, May 2026\) and model snapshots\. The four open\-weight models run on2×2\\timesNVIDIA L40S 48GB atbfloat16; the full open\-weight evaluation across all five benchmarks and both channels took approximately 425 GPU\-hours\.
### A\.2Dataset Statistics and Licenses

Per\-dataset sizes, splits, answer formats, mean L0 token lengths, and licenses are in Table[6](https://arxiv.org/html/2606.24083#A1.T6)\.

Table 6:Dataset statistics\. Mean tokens at L0 under the GPT\-4o tokenizer\. All datasets used under their published licenses\.
### A\.3Inference Configuration

All eight models use greedy decoding under identical settings for both conditions; per\-level decoder budgets are the only level\-dependent parameter \(Table[7](https://arxiv.org/html/2606.24083#A1.T7)\)\.

Table 7:Inference configuration\. Identical across Conditions A and B\.
### A\.4Token Accounting

Mean tokens per level for Claude Haiku 4\.5, Condition A, CommonsenseQA \(Table[8](https://arxiv.org/html/2606.24083#A1.T8)\)\. Input tokens fall at L1–L2 but output expansion produces a net increase at L1 and marginal savings at L2; reductions appear at L3–L4, where entailment has already declined\.

Table 8:Token accounting for Claude Haiku 4\.5, Condition A, CommonsenseQA \(n=1,221n=1\{,\}221\)\. Total tokens rise at L1 from output expansion\.
### A\.5Cost Estimates

Cost model applied to Claude Haiku 4\.5, Condition A, CommonsenseQA \(Table[9](https://arxiv.org/html/2606.24083#A1.T9)\)\. Cost rises at L1 from output expansion; reductions appear only at higher levels where entailment has fallen\.

Table 9:Estimated inference cost per million items for Claude Haiku 4\.5, Condition A, CommonsenseQA\. Lower cost does not imply higher preservation\.Per\-model, per\-benchmark breakdowns are in Appendix[B](https://arxiv.org/html/2606.24083#A2); the input/output token split reproduces from the per\-token rates of Table[5](https://arxiv.org/html/2606.24083#A1.T5)\.

### A\.6Open\-Weight Cost Projection

The four open\-weight models \(Qwen2\.5\-VL\-7B, Qwen3\.5\-9B, DeepSeek\-R1\-Distill\-Qwen\-7B, Gemma\-4\-E4B\) have no metered API price, so we project their measured input/output token counts onto a six\-tier panel of May 2026 public pricing \(DSv4 Flash, DSv4 Pro, Haiku 4\.5, GPT\-4o, GPT\-5\.4, Sonnet 4\.6\) and average across tiers\. At L1 Cond B the mean projected savings are2\.5×2\.5\\timeson Qwen2\.5\-VL\-7B \(the largest open\-weight saving in the panel\),1\.18×1\.18\\timeson Qwen3\.5\-9B,1\.17×1\.17\\timeson DeepSeek\-R1\-Distill, and2\.09×2\.09\\timeson Gemma\-4\-E4B, all in the same direction as the API panel; the two reasoning\-distilled models save less because their unconstrained generations are already short\. Full per\-tier, per\-cell numbers are released in our repository\.

### A\.7Decoder\-Truncation Check at L1 Condition B

Fraction of items whose output hit the L1max\_new\_tokens=300 ceiling: 0\.9% \(GPT\-4o\), 7\.3% \(Haiku 4\.5\), 0\.9% \(Qwen2\.5\-VL\-7B\)\. The L1\-B cost savings of Finding 1 reflect natural stopping, not truncation\.

## Appendix BPer\-Level Results Tables

Per\-level accuracy and bidirectional NLI for all eight models \(Tables[10](https://arxiv.org/html/2606.24083#A2.T10)–[11](https://arxiv.org/html/2606.24083#A2.T11)\)\. Colored deltas show the change against each model’s L0 baseline \(red↓\\downarrowdegradation, green↑\\uparrowimprovement\)\. Green Acc deltas with red NLI deltas on classification benchmarks under Condition B quantify Finding 2; DeepSeek\-R1 shows the inverse pattern \(§[4\.2](https://arxiv.org/html/2606.24083#S4.SS2)\)\. Figure[5](https://arxiv.org/html/2606.24083#A2.F5)visualizes the accuracy data as per\-model, per\-dataset curves\.

Table 10:Per\-level task accuracy \(%\) and bidirectional NLI entailment rate \(%\) under Condition A \(input compression\) for all eight evaluated models\. NLI % is the rate at which a generation entails its L0 counterpart bidirectionally; L0 is 100 % by construction\. Colored subscripts show the change relative to each model’s L0 baseline \(red↓\\downarrowfor degradation, green↑\\uparrowfor improvement\)\. L4 NLI not reported; see §[3\.3](https://arxiv.org/html/2606.24083#S3.SS3)\.
†L0 extraction rate<0\.95<0\.95; affected cells are audited in Appendix G\.Table 11:Per\-level task accuracy \(%\) and bidirectional NLI entailment rate \(%\) under Condition B \(output constraint\) for all eight evaluated models\. NLI % is the rate at which a generation entails its L0 counterpart bidirectionally; L0 is 100 % by construction\. Colored subscripts show the change relative to each model’s L0 baseline \(red↓\\downarrowfor degradation, green↑\\uparrowfor improvement\)\. The contrast between green Acc deltas and red NLI deltas on classification benchmarks under non\-reasoning models is the surface\-text divergence of Finding 2\. L4 NLI not reported; see §[3\.3](https://arxiv.org/html/2606.24083#S3.SS3)\.
†L0 extraction rate<0\.95<0\.95; affected cells are audited in Appendix G\.![Refer to caption](https://arxiv.org/html/2606.24083v1/x5.png)Figure 5:Per\-level accuracy across the eight evaluated models and five benchmarks under both conditions\. Solid: Condition A\. Dashed: Condition B\. The per\-level bidirectional NLI grid is released alongside the artifact bundle\.### B\.1Threshold and Task\-Type Comparisons

Smallest levelLcL\_\{c\}at which accuracy \(LcaccL\_\{c\}^\{\\text\{acc\}\}\) and NLI \(LcsemL\_\{c\}^\{\\text\{sem\}\}\) cross the degradation criterion \(Table[12](https://arxiv.org/html/2606.24083#A2.T12)\): we define degradation as accuracy falling at least55pp below the L0 baseline \(LcaccL\_\{c\}^\{\\text\{acc\}\}, scanned over L1–L4\) and bidirectional NLI falling at least1515pp below the L0 anchor \(LcsemL\_\{c\}^\{\\text\{sem\}\}, scanned over L1–L3 since L4 carries no NLI score\); “—” marks a cell where the threshold is never crossed in that range \(non\-evaluable for the ordering\)\. Of 80 ‘\(model, dataset, condition\)’ cells, 60 are evaluable \(both thresholds crossed in range\), and on the 8\-model panel under strict bidirectional NLI,Lcsem≤LcaccL\_\{c\}^\{\\text\{sem\}\}\\leq L\_\{c\}^\{\\text\{acc\}\}on all 60\. Forward\-only NLI violates the ordering on 2 of 60; cosine on 36 of 60, confirming cosine is non\-monotone\. The rows in Table[12](https://arxiv.org/html/2606.24083#A2.T12)are a representative three\-model excerpt; the 60\-of\-60 figure is computed over the full eight\-model panel \(released data\)\. Table[13](https://arxiv.org/html/2606.24083#A2.T13)aggregates L1 non\-preservation by benchmark; Figure[6](https://arxiv.org/html/2606.24083#A2.F6)breaks the2×22\\times 2dissociation down per\-benchmark\.

Table 12:Threshold levels at which accuracy \(LcaccL\_\{c\}^\{\\text\{acc\}\},≥5\\geq 5pp below L0, scanned L1–L4\) and NLI \(LcsemL\_\{c\}^\{\\text\{sem\}\},≥15\\geq 15pp below L0, scanned L1–L3 as L4 has no NLI\) first cross the degradation criterion; “—” marks a cell where the threshold is never crossed in that range\. Of 80 ‘\(model, dataset, condition\)’ cells, 60 are evaluable \(both thresholds crossed in range\), andLcsem≤LcaccL\_\{c\}^\{\\text\{sem\}\}\\leq L\_\{c\}^\{\\text\{acc\}\}on all 60 on the full eight\-model panel under the strict bidirectional NLI criterion; the rows shown are a representative three\-model excerpt\. The frequent “—” under Cond B reflects accuracy that never degrades by55pp under output compression \(Finding 2\)\.Table 13:Mean L1 semantic non\-preservation \(%\) by benchmark under the headline bidirectional NLI criterion, averaged across the eight evaluated models\. The same task\-type ordering holds under the alternative criteria of §[3\.3](https://arxiv.org/html/2606.24083#S3.SS3)\(Appendix[C](https://arxiv.org/html/2606.24083#A3)\)\.![Refer to caption](https://arxiv.org/html/2606.24083v1/x6.png)Figure 6:2×22\\times 2dissociation by dataset, aggregated across L1–L3 and the eight evaluated models\. Each panel shows Condition A and Condition B side by side; bar segments are theC1C\_\{1\}–C4C\_\{4\}outcome shares\.

## Appendix CJudge Reliability and Semantic Robustness

Calibration and robustness evidence for Finding 2: judge reliability by compression level \(§[C\.1](https://arxiv.org/html/2606.24083#A3.SS1), Table[14](https://arxiv.org/html/2606.24083#A3.T14)\) and cross\-metric replication under eleven additional measures \(Table[15](https://arxiv.org/html/2606.24083#A3.T15)\)\. The headline NLI rate sits near the conservative end of the family\.

### C\.1NLI Judge Reliability by Compression Level

Calibration uses 70 POS\-filtered synthetic positive pairs per level: pairs are semantically equivalent by construction, so non\-entailment is attributable to the judge\. The Disagree\-A rate \(NLI fail with cosine\>0\.85\>0\.85\) at L1 Cond B is 17\.1%, leaving judge\-failure cases a clear minority\. L1 is judge\-reliable \(FN2\.9%2\.9\\%\), L2 supplementary \(FN28\.6%28\.6\\%\), L3 descriptive only \(FN50\.0%50\.0\\%\); L4 is excluded\.

Table 14:NLI judge reliability by compression level \(8\-model grand aggregate, Cond B\)\.*C2C\_\{2\}strict*: bidirectional entailment\.*C2C\_\{2\}fwd\-only*: forward direction only\.*Disagree\-A*: NLI fails but cosine\>0\.85\>0\.85\(Cond B\)\.*Judge FN*: false\-negative rate on POS\-filtered synthetic positive pairs\. L1’s2\.9%2\.9\\%FN underpins the L1 headline\.
### C\.2Robustness Across Alternative Semantic Measures

We re\-scored every \(compressed, L0\) pair under the eleven complementaryC2C\_\{2\}measures that, with the headline, make up the twelve: forward\-only and soft NLI, three independent NLI judges \(BART\-large\-MNLI, multilingual XNLI, DeBERTa\-large\), a faithfulness checker \(MiniCheck\), learned and surface similarity \(BLEURT, ROUGE\-L, METEOR\), an STS cross\-encoder, and a QA\-based propositional check\. Continuous\-similarity scores \(BERTScore, two sentence\-embedding cosines\) and an answer\-anchored NLI variant are reported alongside for context\. Table[15](https://arxiv.org/html/2606.24083#A3.T15)reports L1 Cond B aggregates on the 6\-non\-reasoning panel\. All twelveC2C\_\{2\}measures report a substantial divergence on the 6\-non\-reasoning panel \(roughly 41–88%\), with the headline bidirectional NLI near the conservative end of the family\. Per\-metric details and the full per\-cell breakdown are released in our repository\.

MeasureL1 Cond B valueType*The twelveC2C\_\{2\}measures \(thresholded\)*NLI bidirectional51\.9 %C2C\_\{2\}Thresholded rateNLI forward\-only47\.4 %C2C\_\{2\}Thresholded rateNLI soft \(mean prob\>0\.5\>0\.5\)41\.2 %C2C\_\{2\}Thresholded rateBART\-large\-MNLI bidirectional56\.1 %C2C\_\{2\}Independent NLI judgemDeBERTa\-v3\-XNLI bidirectional50\.2 %C2C\_\{2\}Multilingual NLI judgeMiniCheck54\.0 %C2C\_\{2\}Faithfulness checkerDeBERTa\-large bidirectional49\.5 %C2C\_\{2\}Larger NLI variantBLEURT \(Elron/bleurt\-base\-128\)87\.6 %C2C\_\{2\}\(BLEURT<0<0\)Learned similarityMETEOR81\.9 %C2C\_\{2\}\(METEOR<0\.3<0\.3\)Paraphrase\-aware surfaceROUGE\-L55\.9 %C2C\_\{2\}\(ROUGE\-L<0\.3<0\.3\)Surface overlapSTS cross\-encoder87\.2 %C2C\_\{2\}\(STS\-pass<0\.5<0\.5\)STSQA\-based propositional score60\.5 %C2C\_\{2\}Content recovery*Reported for context \(not counted\)*BERTScore \(roberta\-large\)0\.858 mean F1Token\-levelCosine \(MiniLM, paper\)0\.725 mean cosineEmbeddingCosine \(intfloat/e5\-base\-v2\)0\.904 mean cosineEmbedding, 2nd arch\.Answer\-anchored NLI11\.4 %C2C\_\{2\}Ground truthJudge calibration \(FN on L1 positives\)2\.9 %Lower boundTable 15:Cross\-metric agreement at L1 Condition B \(6\-non\-reasoning panel\)\. The twelveC2C\_\{2\}measures all report a substantial divergence on the 6\-non\-reasoning panel \(41%41\\%–88%88\\%\); the headline bidirectional NLI \(51\.9%51\.9\\%on the 6\-non\-reasoning panel\) sits near the conservative end of the family, which ranges from DeBERTa\-large \(49\.5%49\.5\\%on the 6\-non\-reasoning panel\) to BLEURT \(87\.6%87\.6\\%on the 6\-non\-reasoning panel\)\. Continuous\-similarity scores \(BERTScore, cosine\) and the answer\-anchored check are shown for context\. Answer\-anchored NLI is lower because a correct answer usually entails the ground\-truth\-answer hypothesis even when the surrounding reasoning text diverges\.
### C\.3Length\-Controlled NLI Re\-Scoring

Bounds the length\-and\-register confound\. For each \(L0, L1\-B\) pair we truncate L0 to the L1\-B wordpiece\-token length and re\-score with the same judge under the same bidirectional criterion\.

##### Procedure\.

For each \(model, dataset, item\) tuple at L1\-B, tokenize L0, truncate tomin⁡\(\|L0\|,\|L1\-B\|\)\\min\(\|\\text\{L0\}\|,\|\\text\{L1\-B\}\|\), detokenize, re\-score in both directions\. Same denominator as the Finding 2 headline\.

##### Headline\.

TheC2C\_\{2\}rate*rises*by\+28\.4\+28\.4pp on the 6\-non\-reasoning panel and\+21\.6\+21\.6pp on the 8\-model aggregate under length\-matched scoring \(Table[16](https://arxiv.org/html/2606.24083#A3.T16)\)\. Reasoning models \(DeepSeek\-R1, Qwen3\.5\-9B\) move by≤7\\leq 7pp because their L0 outputs are already short\.

Table 16:Length\-controlled bidirectional NLIC2C\_\{2\}at L1 Condition B\. L0 truncated to L1\-B’s wordpiece\-token length; same judge, same denominator as the headline\.
##### Per\-cell pattern\.

Same direction on every non\-reasoning ‘\(model, dataset\)’ cell\. Largest shifts: GPT\-4o on GSM8K \(18\.3%→\\to92\.0%,\+73\.8\+73\.8pp\), Sonnet on GSM8K \(\+55\.8\+55\.8pp\), Qwen2\.5\-VL\-7B on GSM8K \(\+55\.0\+55\.0pp\)\.

##### System\-prompt noise floor\.

We do not separately score an L0\-A vs\. L0\-B paired\-NLI baseline that would isolate the system\-prompt register shift between conditions; the headline should be read as a floor above any such \(unmeasured\) noise\.

## Appendix DComparison with LLMLingua\-2

LLMLingua\-2 was designed for compressing few\-shot demonstrations and long\-context inputs, not individual questions in short benchmarks\. Our comparison applies it outside its intended regime; we report it because it is the standard learned input\-side baseline\. We re\-ran the input channel under LLMLingua\-2\(Panet al\.,[2024](https://arxiv.org/html/2606.24083#bib.bib3)\)on 3 models \(GPT\-4o, Sonnet 4\.6, Qwen2\.5\-VL\-7B\)×\\times3 datasets \(GSM8K, BoolQ, ARC\-Easy\) at the paper\-defaultτ=0\.5\\tau\{=\}0\.5rate and atτ=0\.8\\tau\{=\}0\.8on Qwen2\.5\-VL\-7B \(matched toCavewoman’s telegraphic retention\)\. Table[17](https://arxiv.org/html/2606.24083#A4.T17)and Figure[7](https://arxiv.org/html/2606.24083#A4.F7)report cross\-cell aggregates\.

Table 17:Input compression, 3 models×\\times3 datasets atτ=0\.5\\tau\{=\}0\.5;τ=0\.8\\tau\{=\}0\.8was run on Qwen2\.5\-VL\-7B only as a rate\-sensitivity check at a retention rate matched toCavewoman’s telegraphic level\.C2\>0C\_\{2\}\>0on every cell\.![Refer to caption](https://arxiv.org/html/2606.24083v1/x7.png)Figure 7:Mean accuracy, bidirectional NLI entailment against L0, and dissociation rate \(C2C\_\{2\}\) for four input\-compression configurations, averaged over the 3 models×\\times3 datasets the comparison was run on \(τ=0\.8\\tau\{=\}0\.8is Qwen2\.5\-VL\-7B only; see Table[17](https://arxiv.org/html/2606.24083#A4.T17)\)\.C2\>0C\_\{2\}\>0under every method; the divergence reproduces beyond the POS filter\.Three results\. \(i\) The divergence generalizes beyond our POS filter:C2\>0C\_\{2\}\>0on every cell under both methods\. \(ii\) LLMLingua\-2’s published GSM8K robustness does not transfer to question\-text compression: GSM8K accuracy at the default rate is 20–35% across the three models, well below the∼\\sim79% they report for few\-shot demonstration compression\. \(iii\) The collapse is rate\-driven, not method\-driven: atτ=0\.8\\tau\{=\}0\.8, LLMLingua\-2 on Qwen2\.5\-VL\-7B*GSM8K*reaches0\.660\.66accuracy and0\.730\.73NLI rate, comparable toCavewoman’s telegraphic level \(the lowerτ=0\.8\\tau\{=\}0\.8values in Table[17](https://arxiv.org/html/2606.24083#A4.T17)are the three\-dataset Qwen2\.5\-VL\-7B mean\)\.

##### Compressed prompt atτ=0\.5\\tau\{=\}0\.5\.

Same GSM8K question \(Janet’s ducks, ground\-truth answer1818\) before and after LLMLingua\-2 compression\. Articles, prepositions, and “how much” phrasing are removed; the model often fails\. Theτ=0\.8\\tau\{=\}0\.8check preserves 51 of 64 tokens and recovers most of the accuracy gap; per\-cell numbers are released in our repository\.

Original prompt\(64 tokens\)Janet’s ducks lay 16 eggs per day\. She eats three for breakfast every morning and bakes muffins for her friends every day with four\. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg\. How much in dollars does she make every day at the farmers’ market?

LLMLingua\-2 atτ=0\.5\\tau\{=\}0\.5\(30 tokens;≈47%\\approx 47\\%kept\)Janet ducks lay 16 eggs\. eats three breakfast bakes muffins\. sells remainder farmers ’ market $ 2 fresh duck egg\. dollars?

##### Structural failure mode at the default rate\.

LLMLingua\-2 prunes single\-letter MCQ labels \(“A:”, “B:”, …\) as low\-information tokens, collapsing ARC\-Easy accuracy to 6–7% across all three models atτ=0\.5\\tau\{=\}0\.5\.Cavewoman’s POS filter retains these tokens by construction\.

## Appendix EReasoning\-Token Accounting \(DeepSeek\-R1 and GPT\-5\.4\)

DeepSeek\-R1\-Distill\-Qwen\-7B emits hidden<think\>\.\.\.</think\>traces before its visible answer; HuggingFacegeneratecounts these againstmax\_new\_tokens, so the per\-level decoder budget covers the trace and the visible response together\. The trace counts asoutput\_tokensand enters the NLI judge when it survives\. At L4 \(20\-subword cap\) the budget is exhausted before the visible answer starts, which is why DeepSeek\-R1 collapses to near\-zero accuracy at L4 on every dataset \(Table[11](https://arxiv.org/html/2606.24083#A2.T11)\); at L1–L3, partial traces bias bidirectional NLI against DeepSeek\-R1 relative to non\-reasoning models\.

The pattern also inverts DeepSeek\-R1’s accuracy\-vs\-reference\-text cells:C2C\_\{2\}is small \(∼\\sim19%\) on the DeepSeek\-R1 L1 panel andC3C\_\{3\}\(agreement despite incorrect\) is comparatively large \(∼\\sim21%\) there\. The reported output/input accuracy ratio of2\.42\.4is therefore not apples\-to\-apples with the non\-reasoning panel, since the visible budget is effectively shorter\.

GPT\-5\.4 exhibits the same accounting on the API side\. Its billedoutput\_tokensinclude server\-side reasoning tokens that never appear in the returned text: at L0 Condition B the visible response averages∼\\sim56 tokens on BoolQ while the billed output implies∼\\sim198, a3\.5×3\.5\\timesgap on that benchmark \(2\.9×2\.9\\timesaveraged across the five benchmarks\); GPT\-4o, Haiku, and Sonnet bill exactly their visible output\. Because the visible\-output constraint shortens only the returned text and not the hidden reasoning, output compression leaves GPT\-5\.4’s priced token count essentially unchanged, while the L1 telegraphic system prompt adds∼\\sim105 input tokens; its realized cost is therefore flat to slightly higher under output compression \(Finding 1\)\. Unlike DeepSeek\-R1, these reasoning tokens are not part of the visible text and so do not enter the bidirectional NLI judge, so GPT\-5\.4’s surface\-text divergence \(Finding 2\) is measured on visible text alone and is unaffected\.

A third reasoning\-protocol model, Kimi\-K2\.6, was instrumented in the evaluation but excluded from the eight\-model panel for a related reason: its constrained\-output responses \(L1 Condition B and stricter\) returned empty visible text on9999–100%100\\%of items, with token consumption equal to the per\-levelmax\_new\_tokenscap\. The reasoning\-block protocol consumes the entire budget under output constraint, the dual of the DeepSeek\-R1 case: where DeepSeek\-R1’s trace partially survives and biases entailment, Kimi’s consumes everything and leaves nothing visible\. We note it because the failure mode marks a boundary of the output\-constraint protocol for reasoning\-block models\.

## Appendix FLimitations in Detail

##### Length and register confound\.

L1–L3 output\-compression generations are forced into a shorter, function\-word\-light register than L0, and the DeBERTa\-NLI training distribution does not cover \(telegraphic, prose\) pairs\. The length\-controlled re\-scoring of Appendix[C\.3](https://arxiv.org/html/2606.24083#A3.SS3)bounds this; BART\-MNLI \(60\.3% inter\-judge agreement, 45\.7% both\-fail at L1\-B\) and answer\-anchored NLI corroborate\.

##### Judge reliability above L1\.

See Appendix[C\.1](https://arxiv.org/html/2606.24083#A3.SS1)\. L1 is judge\-reliable; L2 is supplementary; L3 is descriptive only\. L4 is excluded\.

##### Reference noise and multiple\-comparisons correction\.

The divergence is measured against one greedy L0 draw; sampled\-decoding stochasticity is not measured\. The L0\-A vs\. L0\-B noise floor is not separately measured\. Across the 320\-cell Wilcoxon family atα=0\.05\\alpha=0\.05, 278 of 320 cells are significant uncorrected and 278 remain after Benjamini–Hochberg correction \(zero lost\)\. Per\-cell bootstrap CIs \(nboot=1,000n\_\{\\text\{boot\}\}=1\{,\}000\) span 1–7 pp\.

## Appendix GExtraction\-Rate Audit and L4 Length Distribution

### G\.1Extraction\-Rate Audit

Accuracy is the fraction of items whose extracted answer matches the ground\-truth answer; un\-extracted items count as incorrect\. When L0 parse rate is materially below 1\.0, “L1 exceeds L0” partially reflects the extractor working better on shorter outputs\.

##### Headline gap on MMLU\-STEM \(Condition B\)\.

Among the models showing apparent L1 gains, L0\-B MMLU\-STEM parse rate ranges from0\.4920\.492\(Gemma\-4\-E4B\) to0\.8540\.854\(Sonnet\) and recovers to0\.8070\.807–0\.9690\.969at L1\-B\. Largest parse\-rate gaps: Gemma\-4\-E4B \(\+38\.7\+38\.7pp parse rate\), GPT\-4o \(\+16\.8\+16\.8pp parse rate\), Qwen2\.5\-VL\-7B \(\+14\.1\+14\.1pp parse rate\)\. We do not report L1\-vs\-L0 accuracy gains where L0 parse<0\.95<0\.95\.

##### Other affected cells\.

45 of 80 ‘\(model, dataset, condition\)’ cells have L0 parse<0\.95<0\.95\. Sonnet L0\-A CSQA is the most extreme \(0\.0370\.037\) and drives the apparent “Sonnet L1\-A exceeds L0\-A by55pp on CSQA” artifact in Table[10](https://arxiv.org/html/2606.24083#A2.T10)\. Qwen3\.5\-9B’s below\-random L0\-B accuracies on ARC\-Easy, CommonsenseQA, and MMLU\-STEM are not explained by the extraction\-rate evidence reported here\. The full per\-cell audit is released in our repository\.

### G\.2L4 Output\-Length Distribution

L4’s 15\-token target is conveyed through the prompt with amax\_new\_tokens=20\\texttt\{max\\\_new\\\_tokens\}=20ceiling \(Table[18](https://arxiv.org/html/2606.24083#A7.T18)\)\.

Table 18:L4 output\-length distribution \(40 model×\\timesdataset cells per condition\)\. Cond A overshoots the 15\-token target; the same ceiling under Cond B binds tightly\.L4\-A is retained in per\-level tables with the soft\-constraint caveat: “asked to compress to 15 tokens but allowed up to 20” rather than a strict 15\-token budget\.

## Appendix HConstraint\-Level Specifications

Verbatim system prompts and POS\-tag filter rules for both conditions; the implementing code is released in our repository\.

### H\.1Condition A: Input\-Compression Filter

System prompt is fixed across all five levels; only the user message changes via a deterministic spaCy POS\-tag filter\. Surviving tokens are rejoined with single whitespace; an empty filter output falls back to the original text\.

##### Neutral system prompt \(identical at all levels\)\.

Condition A: System Prompt \(L0–L4\)[⬇](data:text/plain;base64,WW91IGFyZSBhIGhlbHBmdWwgYXNzaXN0YW50LiBBbnN3ZXIgdGhlIGZvbGxvd2luZyBxdWVzdGlvbiBhY2N1cmF0ZWx5IGFuZCBjb21wbGV0ZWx5Lg==)Youareahelpfulassistant\.Answerthefollowingquestionaccuratelyandcompletely\.

##### POS\-tag filter rules\.

Table[19](https://arxiv.org/html/2606.24083#A8.T19)gives the per\-level filter rule applied to the user message\. The L4 row truncates the*token stream*\(not the character sequence\), preserving the monotone ladderL0⊇L1⊇L2⊇L3⊇L4\\text\{L0\}\\supseteq\\text\{L1\}\\supseteq\\text\{L2\}\\supseteq\\text\{L3\}\\supseteq\\text\{L4\}\.

Table 19:Condition A per\-level POS\-tag filter rules\.![Refer to caption](https://arxiv.org/html/2606.24083v1/x8.png)Figure 8:Worked example of the input\-compression filter applied to a single question at each of the five reduction levels\. L0 is the unmodified prompt; L1 removes closed\-class function words; L2 retains nouns, verbs, and cardinal numerals; L3 strips the verbs to leave a nominal skeleton; L4 truncates the L3 form to its first fifteen tokens\. The same filter family defines both channels\.

### H\.2Condition B: Per\-Level System Prompts

Each prompt has a named constraint type, a rule list, a task\-neutral example, and the answer\-format convention\. Decoder budgets:\{L0:400,L1:300,L2:200,L3:150,L4:20\}\\\{L\_\{0\}\\\!:\\\!400,\\ L\_\{1\}\\\!:\\\!300,\\ L\_\{2\}\\\!:\\\!200,\\ L\_\{3\}\\\!:\\\!150,\\ L\_\{4\}\\\!:\\\!20\\\}max\_new\_tokens\.

Condition B: L0 \(Unconstrained\)[⬇](data:text/plain;base64,QW5zd2VyIHRoZSBmb2xsb3dpbmcgcXVlc3Rpb24gYWNjdXJhdGVseS4KUmVhc29uIHN0ZXAgYnkgc3RlcCBpbiBmdWxsLCBncmFtbWF0aWNhbCBFbmdsaXNoIHNlbnRlbmNlcy4gQ29uY2x1ZGUgd2l0aCB0aGUgZmluYWwgYW5zd2VyIG9uIGl0cyBvd24gbGluZSBpbiB0aGUgZm9ybSAnQW5zd2VyOiA8YW5zd2VyPicuCgpUaGUgZmluYWwtbGluZSBhbnN3ZXIgbWF0Y2hlcyB3aGF0IHRoZSBxdWVzdGlvbiBhc2tzIGZvcjogYSBudW1iZXIgZm9yIG51bWVyaWMgcXVlc3Rpb25zLCAneWVzJyBvciAnbm8nIGZvciB5ZXMvbm8gcXVlc3Rpb25zLCBvciBhIHNpbmdsZSBsZXR0ZXIgKEEsIEIsIEMsIC4uLikgZm9yIG11bHRpcGxlLWNob2ljZSBxdWVzdGlvbnMu)Answerthefollowingquestionaccurately\.Reasonstepbystepinfull,grammaticalEnglishsentences\.Concludewiththefinalansweronitsownlineintheform’Answer:<answer\>’\.Thefinal\-lineanswermatcheswhatthequestionasksfor:anumberfornumericquestions,’yes’or’no’foryes/noquestions,orasingleletter\(A,B,C,\.\.\.\)formultiple\-choicequestions\.

Condition B: L1 \(Telegraphic\)[⬇](data:text/plain;base64,QW5zd2VyIHRoZSBxdWVzdGlvbiB1bmRlciBhIFRFTEVHUkFQSElDIGNvbnN0cmFpbnQuCgpSdWxlczoKICAtIERPIE5PVCB1c2UgYW55IGZ1bmN0aW9uIHdvcmRzLiBObyBhcnRpY2xlcyAodGhlLCBhLCBhbikuIE5vIGNvbmp1bmN0aW9ucyAoYW5kLCBidXQsIG9yLCBzbykuIE5vIHByZXBvc2l0aW9ucyAob2YsIGluLCB0bywgZm9yLCBhdCwgd2l0aCwgZnJvbSwgYnksIG9uLCBwZXIpLgogIC0gRE8gdXNlIG5vdW5zLCBtYWluIHZlcmJzLCBudW1iZXJzLCBhbmQgc3RhbmRhcmQgc3ltYm9scyAoKywgLSwgKiwgLywgPSkuCiAgLSBTaG93IGVhY2ggcmVhc29uaW5nIHN0ZXAuCiAgLSBFbmQgd2l0aCBhIGxpbmU6ICdBbnN3ZXI6IDxhbnN3ZXI+Jy4KCkV4YW1wbGUgKHRhc2stbmV1dHJhbCk6CiAgUHJlbWlzZSBtZW50aW9ucyBpdGVtIFguIFByb3BlcnR5IFkgaG9sZHMgWC4gTWF0Y2g6IHllcy4KICBBbnN3ZXI6IDxhbnN3ZXI+CgpUaGUgZmluYWwtbGluZSBhbnN3ZXIgbWF0Y2hlcyB3aGF0IHRoZSBxdWVzdGlvbiBhc2tzIGZvcjogYSBudW1iZXIgZm9yIG51bWVyaWMgcXVlc3Rpb25zLCAneWVzJyBvciAnbm8nIGZvciB5ZXMvbm8gcXVlc3Rpb25zLCBvciBhIHNpbmdsZSBsZXR0ZXIgKEEsIEIsIEMsIC4uLikgZm9yIG11bHRpcGxlLWNob2ljZSBxdWVzdGlvbnMu)AnswerthequestionunderaTELEGRAPHICconstraint\.Rules:\-DONOTuseanyfunctionwords\.Noarticles\(the,a,an\)\.Noconjunctions\(and,but,or,so\)\.Noprepositions\(of,in,to,for,at,with,from,by,on,per\)\.\-DOusenouns,mainverbs,numbers,andstandardsymbols\(\+,\-,\*,/,=\)\.\-Showeachreasoningstep\.\-Endwithaline:’Answer:<answer\>’\.Example\(task\-neutral\):PremisementionsitemX\.PropertyYholdsX\.Match:yes\.Answer:<answer\>Thefinal\-lineanswermatcheswhatthequestionasksfor:anumberfornumericquestions,’yes’or’no’foryes/noquestions,orasingleletter\(A,B,C,\.\.\.\)formultiple\-choicequestions\.

Condition B: L2 \(Keyword\-Only\)[⬇](data:text/plain;base64,QW5zd2VyIHRoZSBxdWVzdGlvbiB1bmRlciBhIEtFWVdPUkQtT05MWSBjb25zdHJhaW50LgoKUnVsZXM6CiAgLSBVc2UgT05MWSBub3VucyBhbmQgbWFpbiB2ZXJicy4gTm8gZ3JhbW1hciwgbm8gZnVsbCBzZW50ZW5jZXMuCiAgLSBPdXRwdXQgYXMgZnJhZ21lbnRzLCBzaG9ydCBsYWJlbHMsIG9yIGxpc3QgaXRlbXMuCiAgLSBOdW1iZXJzIGFuZCBzdGFuZGFyZCBzeW1ib2xzICgrLCAtLCAqLCAvLCA9KSBhcmUgYWxsb3dlZC4KICAtIEVhY2ggcmVhc29uaW5nIHN0ZXAgYXBwZWFycyBhcyBhIGZyYWdtZW50LgogIC0gRW5kIHdpdGggYSBsaW5lOiAnQW5zd2VyOiA8YW5zd2VyPicuCgpFeGFtcGxlOgogIEl0ZW06IFgKICBQcm9wZXJ0eSBZOiBob2xkcwogIE1hdGNoOiB5ZXMKICBBbnN3ZXI6IDxhbnN3ZXI+CgpUaGUgZmluYWwtbGluZSBhbnN3ZXIgbWF0Y2hlcyB3aGF0IHRoZSBxdWVzdGlvbiBhc2tzIGZvcjogYSBudW1iZXIgZm9yIG51bWVyaWMgcXVlc3Rpb25zLCAneWVzJyBvciAnbm8nIGZvciB5ZXMvbm8gcXVlc3Rpb25zLCBvciBhIHNpbmdsZSBsZXR0ZXIgKEEsIEIsIEMsIC4uLikgZm9yIG11bHRpcGxlLWNob2ljZSBxdWVzdGlvbnMu)AnswerthequestionunderaKEYWORD\-ONLYconstraint\.Rules:\-UseONLYnounsandmainverbs\.Nogrammar,nofullsentences\.\-Outputasfragments,shortlabels,orlistitems\.\-Numbersandstandardsymbols\(\+,\-,\*,/,=\)areallowed\.\-Eachreasoningstepappearsasafragment\.\-Endwithaline:’Answer:<answer\>’\.Example:Item:XPropertyY:holdsMatch:yesAnswer:<answer\>Thefinal\-lineanswermatcheswhatthequestionasksfor:anumberfornumericquestions,’yes’or’no’foryes/noquestions,orasingleletter\(A,B,C,\.\.\.\)formultiple\-choicequestions\.

Condition B: L3 \(Noun\-Phrase Skeleton\)[⬇](data:text/plain;base64,QW5zd2VyIHRoZSBxdWVzdGlvbiB1bmRlciBhIE5PVU4tUEhSQVNFIFNLRUxFVE9OIGNvbnN0cmFpbnQuCgpSdWxlczoKICAtIE5PIHZlcmJzIG9mIGFueSBraW5kLiBOb25lLgogIC0gVXNlIG9ubHkgbm9taW5hbCBmcmFnbWVudHM6IG5vdW5zLCBub3VuIGNvbXBvdW5kcywgbnVtYmVycywgYW5kIHN0YW5kYXJkIHN5bWJvbHMgKCssIC0sICosIC8sID0pLgogIC0gRWFjaCBzdGVwIGlzIGEgbm91biBwaHJhc2UgbGFiZWxpbmcgYSBxdWFudGl0eSwgY2xhaW0sIG9yIHByb3BlcnR5LgogIC0gRW5kIHdpdGggYSBsaW5lOiAnQW5zd2VyOiA8YW5zd2VyPicuCgpFeGFtcGxlOgogIEl0ZW06IFgKICBQcm9wZXJ0eSBpbiBxdWVzdGlvbjogWQogIE1hdGNoIHN0YXR1czogcG9zaXRpdmUKICBBbnN3ZXI6IDxhbnN3ZXI+CgpUaGUgZmluYWwtbGluZSBhbnN3ZXIgbWF0Y2hlcyB3aGF0IHRoZSBxdWVzdGlvbiBhc2tzIGZvcjogYSBudW1iZXIgZm9yIG51bWVyaWMgcXVlc3Rpb25zLCAneWVzJyBvciAnbm8nIGZvciB5ZXMvbm8gcXVlc3Rpb25zLCBvciBhIHNpbmdsZSBsZXR0ZXIgKEEsIEIsIEMsIC4uLikgZm9yIG11bHRpcGxlLWNob2ljZSBxdWVzdGlvbnMu)AnswerthequestionunderaNOUN\-PHRASESKELETONconstraint\.Rules:\-NOverbsofanykind\.None\.\-Useonlynominalfragments:nouns,nouncompounds,numbers,andstandardsymbols\(\+,\-,\*,/,=\)\.\-Eachstepisanounphraselabelingaquantity,claim,orproperty\.\-Endwithaline:’Answer:<answer\>’\.Example:Item:XPropertyinquestion:YMatchstatus:positiveAnswer:<answer\>Thefinal\-lineanswermatcheswhatthequestionasksfor:anumberfornumericquestions,’yes’or’no’foryes/noquestions,orasingleletter\(A,B,C,\.\.\.\)formultiple\-choicequestions\.

Condition B: L4 \(Hard 15\-Token Budget\)[⬇](data:text/plain;base64,QW5zd2VyIHRoZSBxdWVzdGlvbiB1bmRlciBhIEhBUkQgVE9LRU4gQlVER0VULgoKUnVsZXM6CiAgLSBZb3VyIEVOVElSRSByZXNwb25zZSBtdXN0IGJlIDE1IHRva2VucyBvciBmZXdlci4KICAtIFRoZSByZXNwb25zZSBNVVNUIGluY2x1ZGUgdGhlIGZpbmFsIGFuc3dlci4KICAtIFByZWZlciB0aGUgcmF3IGFuc3dlciBvdmVyIHByb3NlLgoKRXhhbXBsZTogJ0Fuc3dlcjogPGFuc3dlcj4nCgpUaGUgZmluYWwtbGluZSBhbnN3ZXIgbWF0Y2hlcyB3aGF0IHRoZSBxdWVzdGlvbiBhc2tzIGZvcjogYSBudW1iZXIgZm9yIG51bWVyaWMgcXVlc3Rpb25zLCAneWVzJyBvciAnbm8nIGZvciB5ZXMvbm8gcXVlc3Rpb25zLCBvciBhIHNpbmdsZSBsZXR0ZXIgKEEsIEIsIEMsIC4uLikgZm9yIG11bHRpcGxlLWNob2ljZSBxdWVzdGlvbnMu)AnswerthequestionunderaHARDTOKENBUDGET\.Rules:\-YourENTIREresponsemustbe15tokensorfewer\.\-TheresponseMUSTincludethefinalanswer\.\-Prefertherawansweroverprose\.Example:’Answer:<answer\>’Thefinal\-lineanswermatcheswhatthequestionasksfor:anumberfornumericquestions,’yes’or’no’foryes/noquestions,orasingleletter\(A,B,C,\.\.\.\)formultiple\-choicequestions\.

## Appendix IQualitative Examples

One BoolQ item \(Qwen2\.5\-VL\-7B, Condition B\) traced across all five levels \(L4 is shown for illustration only; it is excluded from the aggregate semantic scoring of §[3\.3](https://arxiv.org/html/2606.24083#S3.SS3)\)\. Box color:green =C1C\_\{1\}\(correct, entailment PASS\),orange =C2C\_\{2\}\(correct, entailment FAIL\), gray = L0\. The correct answer survives compression after the reasoning chain has collapsed \(theC2C\_\{2\}pattern of §[4\.2](https://arxiv.org/html/2606.24083#S4.SS2)\)\. A GSM8K arithmetic trace is in the released artifact bundle\.

### BoolQ: Passage Yes/No \(Qwen2\.5\-VL\-7B, Condition B\)

Question:is harry potter and the escape from gringotts a roller coaster ride Ground truth answer:yes \(BoolQ provides a supporting passage with each question; passage omitted here for space\.\)

L0 \(Unconstrained baseline\)Pred: yes GT: yesCorrectTo determine if this attraction is a roller coaster ride, I analyzed the passage\. The passage states it is an ‘‘indoor steel roller coaster’’ at Universal Studios Florida, which directly confirms it is a roller coaster\. Based on this evidence…Answer: yes

L1 \(Telegraphic\)Pred: yes GT: yesCorrectNLI:FAIL\[C2C\_\{2\}\]Yes\.Answer: yes

L2 \(Keyword\-Only\)Pred: yes GT: yesCorrectNLI:FAIL\[C2C\_\{2\}\]Yes

L3 \(Noun\-Phrase Skeleton\)Pred: yes GT: yesCorrectNLI:PASS\[C1C\_\{1\}\]Item: Harry Potter and the Escape from Gringotts Property in question: Roller coaster ride Match status: positiveAnswer: yes

L4 \(Hard 15\-Token Budget\)Pred: yes GT: yesCorrectNLI:FAIL\[C2C\_\{2\}\]Answer: Yes

Takeaway\.The correct answer survives at every level, but the reasoning chain collapses at L1, L2, and L4 \(NLI FAIL;C2C\_\{2\}cell\)\. The L0 output makes several inferential steps citing the passage; “Yes\.” and “Yes” make no such claims, so the L0 output does not entail them\. L3 unexpectedly recovers bidirectional entailment: the noun\-phrase skeleton labels the entity and property correctly, replicating enough of the L0 propositional structure for entailment to pass\. This item exemplifies why accuracy alone cannot distinguishC1C\_\{1\}fromC2C\_\{2\}\.

## Appendix JReleased Artefacts and Reproducibility

Each \(model, dataset, condition, level\) configuration releases three artifacts in our repository: a base inference record \(token counts, realized cost, extracted answer, ground\-truth answer\), a paired entailment record \(bidirectional NLI scores at L1–L4\), and a paired embedding record \(sentence\-embedding cosine at L0–L4\)\. Each configuration is accompanied by a run\-configuration manifest recording the git SHA, conda environment, package versions, and GPU\. Aggregate per\-cell summaries and a verification script that reproduces the paper’s reported numbers from those summaries are released alongside\.