Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

# How the Error Budget Masks Damage in Quantized LLM Agents
Source: [https://arxiv.org/html/2607.27275](https://arxiv.org/html/2607.27275)
## Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Jiwon Jang1,2\\equalcontrib, Kisu Yang1,3\\equalcontrib, Heuiseok Lim3\\corresponding, Hyunwoo Park2\\corresponding

###### Abstract

Post\-training quantization to 4\-bit weights is widely reported to be nearly lossless\. We test this claim for multi\-turn, tool\-calling agents, where it now matters most\. Onτ2\\tau^\{2\}\-bench, across two open\-weight model families in dense and MoE variants and two domains \(eight cells, 456 episodes each, at 16\-, 8\-, and 4\-bit weights\), quantization indeed looks free on the standard metric\. No cell shows a score change that survives multiple\-comparison correction, and in the cell that carries the largest process damage, equivalence testing bounds the change within±7\.5\\pm 7\.5points\. The process tells a different story\. Quantization amplifies the failure the model already exhibits at full precision \(tool\-name hallucination in telecom, with the same directional trend in retail entity errors\) by up to2\.5×2\.5\\timesin volume \(\+17\.6\+17\.6points per task\), while creating essentially no new failures\. The failure set is the same at every precision \(rank correlation≥0\.94\\geq 0\.94,0\.18%0\.18\\%novel events\)\. The score stays flat because the benchmark’s ten\-error budget absorbs the extra failures\. Shrinking the budget to two errors re\-exposes a score gap of1717points, and it does so only in the one cell where quantization added error volume, exactly as the masking account predicts\. A targeted error\-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives\. Both diagnostics, the per\-channel error rate and success under a shrinking budget, come from logs benchmarks already collect; we suggest reporting them alongside task reward\.

## Introduction

![Refer to caption](https://arxiv.org/html/2607.27275v1/x1.png)Figure 1:Flat score, amplified failures\.We run the same agent benchmark,τ2\\tau^\{2\}\-bench telecom with 456 episodes, using a BF16 agent and its AWQ\-INT4\-quantized version\. The cell shown is Gemma\-4\-31B\.\(a\)On the benchmark’s headline metric, the final task score, the two agents are statistically equivalent\. Quantization looks free\.\(b\)Underneath, the share of the agent’s tool calls that invoke a tool outside its tool list doubles, from19\.5%19\.5\\%to38\.3%38\.3\\%\.\(c\)These are not new failures\. Each dot is one hallucinated tool name\. Every tool the BF16 agent already invented is called about2\.5×2\.5\\timesmore often after quantization, and only0\.18%0\.18\\%of the INT4 agent’s hallucinated calls name a tool its BF16 original never used\.\(d\)The final score stays flat because the benchmark tolerates ten failed calls per episode and the agent recovers between failures\. Recomputing success under a stricter toleranceKKre\-exposes the hidden gap, which is 13×\\timeswider atK=2K\{=\}2than at the defaultK=10K\{=\}10\. Models without a pre\-existing failure tendency are not amplified at all, as Table[1](https://arxiv.org/html/2607.27275#Sx3.T1)shows\.Weight quantization is the default mechanism for serving large language models \(LLMs\) at scale\. Reducing weights from 16 to 4 bits cuts memory and latency several\-fold, and systematic evaluations report 4\-bit quantization as nearly lossless, degrading standard benchmark accuracy by only a point or two\(Frantaret al\.[2023](https://arxiv.org/html/2607.27275#bib.bib1); Linet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib2); Liet al\.[2024b](https://arxiv.org/html/2607.27275#bib.bib5); Jinet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib6)\)\. That verdict, however, rests almost entirely on*single\-turn*evaluations, while deployed LLMs increasingly act as*agents*, multi\-turn tool\-calling controllers that interact with an environment, observe feedback, and recover from their own mistakes\.

RQ1:*Does “nearly lossless” survive the shift to multi\-turn tool\-calling agents, and can the benchmark’s final score tell?*The natural way to ask turns out to be misleading\. The obvious experiment is to run quantized and full\-precision agents on an agentic benchmark and compare task success\. We do exactly this onτ2\\tau^\{2\}\-bench\(Barreset al\.[2026](https://arxiv.org/html/2607.27275#bib.bib17)\), a tool\-calling benchmark with a simulated user, across two open\-weight model families×\\times\{dense, MoE\}×\\timestwo domains \(the full eight\-cell grid, 456 simulations per arm\), plus two earlier Qwen\-3\.5 checkpoints in telecom\. The result is a flat line \(Figure[1](https://arxiv.org/html/2607.27275#Sx1.F1)a\)\. No score change survives multiple\-comparison correction, and the largest paired decline,2\.02\.0points, sits within1\.3×1\.3\\timesthe run\-to\-run noise floor\. Taken at face value, quantization is free in the agentic setting too\. Compression studies of agents have so far stopped at this aggregate verdict\(Donget al\.[2025](https://arxiv.org/html/2607.27275#bib.bib19)\); the rest of the paper asks what it conceals\.

We argue this conclusion is an artifact of*what task reward measures*\. Task reward is a binary, end\-of\-trajectory outcome\. It is swamped by task\-difficulty variance and, more importantly, the agentic loop actively*repairs*the errors quantization introduces before they reach the outcome\. Decomposed at the level of individual tool calls, trajectories instead reveal a large effect with two parts\. First, universally,*quantization does not change what the model gets wrong*\. In telecom, where the channel is tool\-name hallucination \(calls outside the agent’s tool list, nearly always real tools on the*user’s*side of the dual\-control interface\), the BF16 and INT4 distributions over hallucinated names are nearly identical, and the invariance holds at 8 bits\. The failure set is a property of the*model*, not of any precision\. Second, in a cell\-dependent way,*quantization multiplies how often that set fires*\. In the highest\-amplification cell \(Gemma\-4 dense, telecom\), 4\-bit weights produce2\.5×2\.5\\timesmore out\-of\-list calls \(649→1,646649\\\!\\to\\\!1\{,\}646\); across cells the multiplier ranges from≈1×\{\\approx\}1\\timesto4\.6×4\.6\\times, and in retail the channel that moves is entity/argument errors instead \(1\.45×1\.45\\times, Gemma\-4 MoE, a trend short of significance\)\. Quantization rescales the model’s existing error set; the scale factor is a property of the cell\.

RQ2:*Why does the final score stay flat while the agent’s per\-step failures multiply?*Because the loop hides the damage\.τ2\\tau^\{2\}\-bench, like other agentic benchmarks, allows ten failed tool calls per episode, and per\-error recovery is equally reliable at both precisions, so the budget absorbs the extra failures quantization produces\. Tightening the tolerance from ten errors to two re\-exposes the deficit, growing the INT4 score gap from1\.31\.3to16\.716\.7points, and it re\-exposes damage only in the one cell where quantization added error volume \(15\.415\.4points there, at most0\.20\.2elsewhere\)\. The score is flat because the environment is forgiving, not because the model is unharmed\. The same localization predicts the remedy, and a reflexive error\-repair prompt indeed collapses the channel wherever it runs but moves score only in the damaged cell\.

RQ3:*Why are some models damaged by quantization while others are unaffected?*Not size or dense\-versus\-MoE architecture but the model’s*full\-precision failure propensity*\. A model that rarely hallucinates tools at 16 bits \(Qwen\-3\.6\) has nothing to amplify, a model with a standing tendency \(Gemma\-4\) has that tendency magnified, and an older Qwen\-3\.5 checkpoint whose gate is slightly open is slightly, but significantly, amplified\. The open gate is necessary rather than sufficient, since a cell with an equally open gate \(Gemma\-4 MoE telecom,19\.3%19\.3\\%\) multiplies by only1\.05×1\.05\\times\. A logit\-margin account is consistent with this pattern\. Quantization noise flips only thin\-margin argmax decisions, precisely those the model already got wrong occasionally, making baseline propensity the observable shadow of the thin\-margin mass\.

Our contributions are:

- •The final score cannot detect 4\-bit damage \(RQ1\)\.End\-of\-episode task reward is not merely low\-powered but*actively masked*by the environment’s error budget; stopping at task success yields a false “all\-clear\.”
- •Quantization amplifies the existing failure set \(RQ1\)\.\(i\) The failure*set*is precision\-invariant across 16, 8, and 4 bits \(0\.970\.97correlation,0\.18%0\.18\\%novel events\), ruling out the “some models are just worse” confound; \(ii\) the failure*volume*multiplier is cell\-dependent,1×1\\timesto4\.6×4\.6\\times, so damage claims must be made per cell\.
- •The error budget masks the damage, falsifiably \(RQ2\)\.A counterfactual budget analysis shows the recovery slack hides the damage; its advance prediction, that tightening re\-exposes damage only in cells with added error volume, holds across all cells \(one adds volume, nine check the converse\), widening15\.415\.4points there versus at most0\.20\.2elsewhere; and a reflexive error\-repair system, run for five telecom models at all three precisions, recovers score only in the damaged cell\.
- •Full\-precision propensity gates the risk \(RQ3\)\.Neither architecture nor size does\. A standing full\-precision failure channel is the*precondition*, and the added error volume the observable*signature*, of maskable damage\. Both come from logs benchmarks already collect; we suggest reporting them alongside task reward, and low\-slack deployments face more 4\-bit risk than single\-turn benchmarks show\.

## Related Work

#### Post\-training quantization and its evaluation\.

Post\-training weight quantization\(Frantaret al\.[2023](https://arxiv.org/html/2607.27275#bib.bib1); Linet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib2); Xiaoet al\.[2023](https://arxiv.org/html/2607.27275#bib.bib3); Dettmerset al\.[2022](https://arxiv.org/html/2607.27275#bib.bib4)\)has become the default mechanism for serving large language models, and systematic evaluations consistently report that 4\-bit quantization leaves aggregate benchmark accuracy largely intact\(Liet al\.[2024b](https://arxiv.org/html/2607.27275#bib.bib5); Jinet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib6)\)\. A separate line of work, however, shows that this aggregate view conceals non\-uniform damage\. Compression disproportionately affects long\-tail and underrepresented inputs\(Hookeret al\.[2019](https://arxiv.org/html/2607.27275#bib.bib7); Ahiaet al\.[2021](https://arxiv.org/html/2607.27275#bib.bib8)\), increases hallucination rates in generation\(Liet al\.[2024a](https://arxiv.org/html/2607.27275#bib.bib9); Haqueet al\.[2025](https://arxiv.org/html/2607.27275#bib.bib10)\), and can erode alignment\- and safety\-related behaviors\(Rath and Maliakkal[2026](https://arxiv.org/html/2607.27275#bib.bib11)\)\. These studies establish*that*quantization damage hides below aggregate metrics, but in single\-turn, static generation; none characterize*where*it goes in a closed\-loop agentic trajectory, nor why end\-to\-end metrics stay flat while per\-step reliability degrades\.

#### Evaluating tool\-calling agents\.

Benchmarks for tool use have progressed from single\-turn function calling\(Patilet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib12)\), function\-calling leaderboards\(Patilet al\.[2025](https://arxiv.org/html/2607.27275#bib.bib13)\), and stepwise capability probes\(Chenet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib14); Qinet al\.[2024](https://arxiv.org/html/2607.27275#bib.bib15)\)to long\-horizon interactive environments with simulated users, most notablyτ\\tau\-bench\(Yaoet al\.[2025](https://arxiv.org/html/2607.27275#bib.bib16)\)and its dual\-control successorτ2\\tau^\{2\}\-bench\(Barreset al\.[2026](https://arxiv.org/html/2607.27275#bib.bib17)\)\. These environments score agents by end\-of\-episode task reward \(pass^kk\), and prior analyses note that tool use is fragile\(Cuadronet al\.[2026](https://arxiv.org/html/2607.27275#bib.bib18)\), yet the benchmarks ship with generous per\-episode error tolerances allowing retries\. All of this work evaluates full\-precision models; the interaction of error tolerance with per\-step error rate has not been examined as a measurement problem\.

#### Compression under agentic workloads\.

Closest to our setting,Donget al\.\([2025](https://arxiv.org/html/2607.27275#bib.bib19)\)evaluate quantized and pruned models on agentic capability suites and report capability\-level degradation patterns, andParamanayakamet al\.\([2025](https://arxiv.org/html/2607.27275#bib.bib20)\)document performance drops when quantized models serve as on\-device function\-calling agents\. Both efforts score isolated steps or aggregate accuracy on a fixed pipeline\. We instead evaluate quantized agents closed\-loop across single\- and dual\-control domains, decompose failures into domain\-specific channels, and trace individual failure events across precisions within a single model, yielding two findings unavailable to step\-level or aggregate evaluation\. Quantization amplifies the model’s pre\-existing failure set rather than creating new failures, and the final score is blind to the amplification because the environment’s error budget absorbs the extra failures until that budget is tightened\.

## Experimental Setup

#### Benchmark\.

We evaluate onτ2\\tau^\{2\}\-bench\(Barreset al\.[2026](https://arxiv.org/html/2607.27275#bib.bib17)\), a tool\-calling benchmark in which an agent resolves a customer request over a multi\-turn conversation with a simulated user, issuing tool calls against a stateful environment\. We use its two domains\.Retailis a single\-control setting in which only the agent acts on the environment;telecomis a dual\-control setting in which the user also executes actions on their own device and the agent must reason about state it cannot manipulate directly\. The domains differ in horizon and tool inventory \(telecom trajectories are longer, median∼\\sim16 versus∼\\sim11 agent turns, with a larger, less standard tool set\) and, as we show, expose different failure channels\. Each domain contains 114 tasks; an episode ends when the user closes the conversation, when the agent exhaustsmax\_errors=10\{=\}10failed tool calls, or atmax\_steps=300\{=\}300\. Task reward is the standardτ2\\tau^\{2\}\-bench outcome \(a final database\-state check plus per\-task assertions\)\. In practice it is binary in both domains, so we report its mean as the final score in percent\.

#### Models and quantization\.

We study two open\-weight LLM families, each in a dense and a mixture\-of\-experts \(MoE\) variant, namelyGemma\-4\-31B\(dense\) andGemma\-4\-26B\-A4B\(MoE, 4B active parameters\), andQwen\-3\.6\-27B\(dense\) andQwen\-3\.6\-35B\-A3B\(MoE, 3B active parameters\), plusQwen\-3\.5\-27BandQwen\-3\.5\-9B\(both dense\) as family\-version and size extensions\. Each model runs at three precision arms:BF16, the original 16\-bit weights and full\-precision reference;FP8; andINT4, quantized with AWQ\. All agents are served with vLLM on NVIDIA A100 GPUs; the A100 has no native FP8 compute, so the FP8 weights run*weight\-only*via the Marlin kernel \(W8A16\)\. Activations and the KV cache are 16\-bit in every arm; all three arms are weight\-only\. Recipes differ by family \(AWQ\-style withg=32g\{=\}32for Gemma INT4, zero\-point AWQ withg=128g\{=\}128for Qwen INT4\)\. The full cross\-product \(two families×\\times\{dense, MoE\}×\\timestwo domains, eight cells\) is complete at all three precisions, with both Qwen\-3\.5 telecom extensions on top\. Episodes terminated byinfrastructure\_error\(≤3\\leq 3per arm\) carry no reward record and are excluded \(one retail cell mixedmax\_stepsof 200 and 300; no episode reaches either cap\)\.

Final score \(%\)Active\-channel error rate \(%\)DomainModelBF16INT4ChangeBF16INT4Δ\\Deltarate4\-bit verdictretailGemma\-4\-31B47\.552\.1n\.s\.1\.782\.19\+0\.41\+0\.41–Gemma\-4\-26B\-A4B51\.847\.6n\.s\.4\.185\.65\+1\.48\+1\.48–Qwen\-3\.6\-27B53\.354\.2n\.s\.1\.701\.58−0\.13\-0\.13–Qwen\-3\.6\-35B\-A3B59\.960\.4n\.s\.1\.751\.69−0\.06\-0\.06–telecomGemma\-4\-31B66\.765\.4n\.s\.19\.5138\.26\+18\.75\\mathbf\{\+18\.75\}Damaged but maskedGemma\-4\-26B\-A4B54\.250\.7n\.s\.19\.2623\.05\+3\.79\+3\.79–Qwen\-3\.6\-27B96\.396\.9n\.s\.0\.380\.47\+0\.09\+0\.09–Qwen\-3\.6\-35B\-A3B98\.296\.3n\.s\.0\.040\.26\+0\.23\+0\.23–Qwen\-3\.5\-27B98\.296\.3n\.s\.0\.261\.21\+0\.95\\mathbf\{\+0\.95\}Damaged but maskedQwen\-3\.5\-9B91\.695\.2n\.s\.0\.610\.43−0\.18\-0\.18–Table 1:Quantization damages two of the ten cells, and the final score detects neither\.The score change is significant nowhere \(n\.s\.; seven cells are moreover TOST\-equivalent at±7\.5\\pm 7\.5points\), while the error channel rises significantly in exactly two cells, whoseΔ\\Deltarates areboldand whose verdicts read*damaged but masked*; dashes mark cells whose channel does not move significantly\. Runs are 456 episodes per arm \(a few unscored episodes leaven≥453n\{\\geq\}453in three arms\)\. The active channel is entity/argument errors in retail and tool\-name hallucination in telecom; rates are percentages of the agent’s own tool calls\. The rate delta is in points, judged by 95% task\-clustered bootstrap intervals \(quoted in the text\)\. The identical Qwen\-3\.6\-35B\-A3B and Qwen\-3\.5\-27B score rows are a counting coincidence\.
#### Protocol\.

To isolate the effect of*agent*quantization, the simulated user is held fixed at full precision \(Qwen\-3\.6\-27B, BF16\) across all conditions; only the agent’s precision varies\. In the Qwen\-3\.6 dense cells the agent thus shares a base model with the user simulator; the Qwen\-3\.5 and Qwen\-3\.6 MoE cells, which do not, behave identically, so this does not drive the family contrast\. We runnum\_trials=4\{=\}4trials per task, giving114×4=456114\\times 4=456simulations per cell\. vLLM serving is not bit\-deterministic \(continuous batching perturbs reduction order\); a repeated 456\-episode run shifted the mean score by about1\.51\.5points, our coarse single\-repeat estimate of the run\-to\-run noise floor\. Unless noted, comparisons are BF16 versus INT4 within a fixed model and domain, holding family, architecture, task distribution, and user simulator constant\.

#### Process metrics and failure channels\.

Beyond task reward, we log every tool call and its outcome\. Our primary process metric is the*channel rate*, the fraction of the agent’s own tool calls \(user\-simulator calls excluded\) that trigger a given failure channel, pooled over a cell; paired tests use the same rate computed per task\. We decompose failures into two channels by matching the environment’s error string\. Atool\-name hallucinationis a call to a tool that does not exist in the domain’s inventory \(errorTool ’XX’ not found\); this is the dominant channel in telecom\. Anentity/argument erroris a well\-formed call whose arguments reference a record that cannot be found \(e\.g\. a wrong customer or order identifier\); this is the channel in retail\. A raw “not found” count would conflate inventing a tool with passing a wrong identifier, and the two behave differently\. We additionally track trajectory length \(agent turns, tool calls\), termination reasons, and, for the mechanism analysis, the per\-task and per\-tool\-name distributions of hallucinations\.

#### Statistics\.

For each metric we report the change from BF16 to INT4 with a 95% confidence interval from5,0005\{,\}000bootstrap resamples\. Because the four trials of a task share its difficulty, we resample*tasks*with replacement, keeping all trials of each sampled task \(a cluster bootstrap over the 114 tasks\); an interval excluding zero denotes significance\. Episode\-level resampling gives narrower intervals; we report clustered intervals throughout\. Paired\-by\-task comparisons \(removing task\-difficulty variance\) and equivalence tests \(two one\-sided tests, TOST\) at±7\.5\\pm 7\.5points supplement these where noted\. For the mechanism analysis we compute, within one model×\\timesdomain, the BF16/INT4 correlation of hallucinated\-name distributions, the fraction of INT4 events unseen at BF16, and the per\-task BF16/INT4 rate relationship\.

#### Repair arm\.

For the mitigation study, five telecom models \(the grid’s four plus Qwen\-3\.5\-27B\) are additionally run at every precision with a*reflexive repair*loop: on a failed tool call, the agent is prompted to diagnose the error message and re\-issue a corrected call, up to three attempts\. Agent decoding is greedy in*all*arms \(verified from the logs: identical prefixes always yield identical outputs\); repair arms differ from base only in user\-simulator temperature,0\.60\.6versus0\.70\.7, so we read contrasts against the run\-to\-run noise floor\.

#### Counterfactual error budget\.

To ask whether the environment’s error tolerance hides the damage, we treatmax\_errorsas a knob\. For a budgetKK, the*surviving success rate*S​\(K\)S\(K\)is the fraction of episodes that succeeded using at mostKKfailed tool calls\. The projection is conservative \(an episode exceedingKKcounts as a failure even if it finished earlier\) but applied identically to both precisions, so the gapSBF16​\(K\)−SINT4​\(K\)S\_\{\\text\{BF16\}\}\(K\)\-S\_\{\\text\{INT4\}\}\(K\)is a fair comparison;too\_many\_errorsterminations occur at 10–11 counted errors, confirmingKKmatchesmax\_errors\.

## Results

### Final Task Reward Is Blind to Quantization

Table[1](https://arxiv.org/html/2607.27275#Sx3.T1)\(left\) and Figure[1](https://arxiv.org/html/2607.27275#Sx1.F1)a report the change in the final score from 16\- to 4\-bit weights\. In all ten cells the 95% interval includes zero; the largest point estimate is4\.64\.6points and five cells nominally*improve*\. The absence of a difference is not merely low power\. In seven cells, including the highest\-signal one below, the change is statistically*equivalent*to zero by TOST at±7\.5\\pm 7\.5points, paired by task\. In the three remaining Gemma\-4 cells the intervals are wider than the margin, so flatness there means no detectable change, not equivalence\. The largest paired decline,2\.02\.0points in Qwen\-3\.6 MoE telecom atp=\.049p\{=\}\.049, survives neither multiple\-comparison correction nor the1\.51\.5\-point noise floor and lies well inside the equivalence margin\. Paired by task in the highest\-signal cell, 4\-bit is worse on 33 tasks, better on 26, tied on 55\. Aggregate or paired,task success cannot distinguish a 4\-bit agent from a full\-precision one; the rest of this section shows why that all\-clear is wrong\.

### The Damage Is Real, in the Process, and Channel\-Specific

When we decompose trajectories into the two failure channels \(Table[1](https://arxiv.org/html/2607.27275#Sx3.T1), right, and Figure[1](https://arxiv.org/html/2607.27275#Sx1.F1)b\), a large and systematic effect emerges\. Intelecom, whose active channel is tool\-name hallucination, 4\-bit quantization raises the Gemma\-4 dense hallucination rate from19\.5%19\.5\\%to38\.3%38\.3\\%of the agent’s tool calls \(\+18\.8\{\+\}18\.8points, CI\[14\.0,23\.6\]\[14\.0,23\.6\]\), a2\.0×2\.0\\timesincrease, and drives a9×9\\timesrise in error\-budget\-exhausting terminations \(5→465\\\!\\to\\\!46of 456 episodes\)\. Because INT4 trajectories also contain more calls per episode \(7\.3→9\.47\.3\\\!\\to\\\!9\.4\), the2\.0×2\.0\\timesrate ratio is a2\.5×2\.5\\timesgrowth in event volume \(649→1,646649\\\!\\to\\\!1\{,\}646\), the multiplier we quote\. Inretail, whose active channel is entity/argument errors, the same family moves in*that*channel instead \(Gemma\-4 MoE,4\.2%→5\.7%4\.2\\%\\\!\\rightarrow\\\!5\.7\\%,\+1\.5\{\+\}1\.5, CI\[−0\.1,\+3\.1\]\[\-0\.1,\+3\.1\], a trend short of significance under clustered intervals\)\. Whichever failure the domain makes available, quantization pushes on it in Gemma\-4\. The channel structure is sharp\. Retail produces*zero*tool\-name events and telecom entity rates stay≤0\.1%\{\\leq\}0\.1\\%\. Each domain opens exactly one failure mode, and quantization can only amplify an open channel\.

Three observations sharpen the picture\. First, Qwen\-3\.6 is*not*amplified in any of its four cells\. Telecom tool\-name stays at0\.38%→0\.47%0\.38\\%\\\!\\to\\\!0\.47\\%, not significant, with 9 events at BF16, and retail entity rates are flat\. Second,architecture is not a stand\-alone predictor\. The one strongly amplified cell is dense, yet the dense Qwen\-3\.6 is flat everywhere, and the Gemma\-4 MoE shows only non\-significant trends \(\+3\.8\{\+\}3\.8points in telecom,\+1\.5\{\+\}1\.5in retail\), so dense\-versus\-MoE by itself does not sort damaged from undamaged cells\. Third, susceptibility ischeckpoint\-specific\. The previous\-generation Qwen\-3\.5, whose gate is slightly open \(0\.26%0\.26\\%\),*is*significantly amplified \(→1\.21%\\rightarrow 1\.21\\%,\+0\.95\{\+\}0\.95\[0\.5,1\.5\]\[0\.5,1\.5\]\), Gemma\-4’s pattern at a twentieth of the magnitude, while its successor and its smaller 9B sibling \(gate0\.61%0\.61\\%\) are both flat\. The discriminating variable is the model’s own baseline behavior, not size or architecture \(§ Analysis\)\.

### The Cliff Is \(Mostly\) at Four Bits

Sweeping the intermediate precision shows that 8 bits, weight\-only in our serving setup, is close to free\. No cell degrades significantly in score \(one nominally*improves*by9\.79\.7points; with ten cells tested, one excursion is unsurprising\) and no telecom channel moves at FP8; the jump to38\.3%38\.3\\%happens entirely between 8 and 4 bits\. One trend is instructive\. Gemma\-4 MoE retail already shows its full INT4\-sized entity/argument movement at FP8 \(\+1\.40\{\+\}1\.40, CI\[−0\.2,3\.0\]\[\-0\.2,3\.0\], not significant under clustered intervals\)\. If that trend is real, the safe\-precision boundary, too, is a property of the model×\\timeschannel pair, not a universal constant\.

![Refer to caption](https://arxiv.org/html/2607.27275v1/x2.png)Figure 2:Quantization amplifies the existing failure set within a single model and domain \(Gemma\-4\-31B, telecom\)\.\(a\)For each tool name the model invents at BF16, its amplification factor, INT4 count over BF16 count, versus how often it was already invented\. High\-volume tools sit tightly around the aggregate2\.5×2\.5\\times; the left\-edge scatter is 1–2\-event noise\. Only 3 of the1,6461\{,\}646INT4 events \(0\.18%0\.18\\%\) name a tool never invented at BF16 \(not shown\); the BF16/INT4 frequency distributions correlate at0\.970\.97\.\(b\)Per\-task increase in hallucination rate versus the task’s BF16 baseline \(114 tasks\), a flat cloud around\+17\.6\{\+\}17\.6points with least\-squares slope−0\.06≈0\{\-\}0\.06\\approx 0, so amplification is additive across tasks rather than proportional to baseline\.
### Quantization Amplifies the Existing Failure Set, Not New Ones

Is the damage a new pathology or a scaling\-up of what the model already did? We answer*within a single model and domain*, Gemma\-4 dense telecom, removing any family or architecture confound\.

At the level of individual tool names, quantization is almost purely multiplicative \(Figure[2](https://arxiv.org/html/2607.27275#Sx4.F2)a\)\. The set of hallucinated tool names is essentially unchanged\. The per\-tool frequency distributions at BF16 and INT4 correlate at0\.970\.97, only 3 of the1,6461\{,\}646INT4 hallucinations \(0\.18%0\.18\\%\) invoke a tool that was never invented at BF16, and the top\-15 fabricated tools at BF16 are all present at INT4, invoked2\.5×2\.5\\timesmore often \(649→1,646649\\\!\\rightarrow\\\!1\{,\}646total\)\.Quantization does not make the model imagine*new*tools; it makes it reach more often for the ones it was already tempted by\.

The invented names are not fabrications\. Of the 24 distinct hallucinated names, 23 are real tools*on the user’s side*of the dual\-control interface \(toggle\_airplane\_mode,run\_speed\_test, …; the 24th is a near\-miss of the user\-sidecheck\_sim\_status\), covering99\.8%99\.8\\%of BF16 and100%100\\%of INT4 events\. The channel is*role\-boundary confusion*, the agent reaching for actions only the user can perform, and quantization amplifies that pre\-existing confusion; retail’s structural zero follows, having no user side to reach across\.

At the level of tasks, the amplification takes the form of a near\-uniform shift rather than a proportional scaling \(Figure[2](https://arxiv.org/html/2607.27275#Sx4.F2)b\)\. Regressing the per\-task*increase*on the task’s full\-precision rate gives a slope of−0\.06\{\-\}0\.06, indistinguishable from zero\. The increase is a constant\+17\.6\{\+\}17\.6points regardless of where the task started \(tercile means\+18\.1\+18\.1,\+17\.2\+17\.2,\+17\.4\+17\.4\); the pooled rise is the slightly larger\+18\.8\{\+\}18\.8points because tasks contribute unequal call counts\. The propensity signal is therefore strongest as a*gate*, whether the model opens this failure mode at all \(649 baseline events in Gemma\-4 telecom, 9 in Qwen\-3\.6 telecom, 0 in the retail tool\-name channel\), rather than as fine per\-task proportionality\. Either way, quantization scales an existing failure distribution rather than creating a new one\.

![Refer to caption](https://arxiv.org/html/2607.27275v1/x3.png)Figure 3:The error budget masks the damage\. Projected final scoreS​\(K\)S\(K\)as the error budgetKKis tightened, for Gemma\-4\-31B telecom and the Qwen\-3\.6\-27B negative control\. The BF16–INT4 gap is1\.31\.3points at the defaultK=10K\{=\}10but16\.716\.7points atK=2K\{=\}2, roughly13×13\\times\. Qwen\-3\.6, with almost no failures, shows no gap at anyKK\.![Refer to caption](https://arxiv.org/html/2607.27275v1/x4.png)Figure 4:Where the extra failures go \(Gemma\-4\-31B, telecom\)\. Episodes by number of failed tool calls, split into episodes that still succeeded \(solid\) and failed \(lighter top\)\. Quantization drains the zero\-error bin and swells the≥5\{\\geq\}5tail from 51 to 173 episodes, the region where the 10\-error budget starts to bind \(all 46 INT4 budget exhaustions live here\)\. Across the low\-error bins most episodes still succeed\. Recovery, not correctness, keeps the score flat\.
### Why the Score Stays Flat: the Error Budget Masks It

Sections above leave a puzzle\. If per\-step failures rise2\.5×2\.5\\times, why is the final score unchanged? Because the agentic loop recovers the extra failures, and the benchmark’s error budget pays for the recovery \(Table[2](https://arxiv.org/html/2607.27275#Sx4.T2)\)\. Two facts establish this\. First, quantization raises failed tool calls per episode from1\.451\.45to3\.643\.64\(2\.5×2\.5\\times\), swelling the≥5\{\\geq\}5\-failure tail from 51 to 173 of 456 episodes \(Figure[4](https://arxiv.org/html/2607.27275#Sx4.F4)\) and stretching trajectories \(16\.4→17\.916\.4\\\!\\to\\\!17\.9turns,7\.3→9\.47\.3\\\!\\to\\\!9\.4calls; errors–turnsr=0\.48r\{=\}0\.48\), error\-driven recovery that quietly refunds part of the compute quantization saved\. Second, the model recovers from each individual failure about*equally well*at both precisions\. The recovery rate after at least one failed call is55%55\\%at both BF16 and INT4\.Quantization does not weaken recovery; it simply produces more failures to recover from, and the1010\-error budget absorbs them\.

Treating that budget as a knob makes the masking explicit and falsifiable \(Figure[3](https://arxiv.org/html/2607.27275#Sx4.F3)\)\. Recomputing the surviving success rateS​\(K\)S\(K\)under tighter budgets, the score gap between BF16 and INT4 grows asKKshrinks\. It is1\.31\.3points at the defaultK=10K\{=\}10, then7\.57\.5atK=5K\{=\}5and16\.716\.7atK=2K\{=\}2, roughly a13×13\\timeswidening; the same tightening costs BF161010points but INT42525\. The Qwen\-3\.6\-27B cell is a negative control\. With almost no baseline failures, masking predicts no hidden gap there, and indeed it is flat at everyKK, nothing for a tighter budget to expose\.

The budget response doubles as a falsification test\. If masking explains the flat score, tightening should re\-expose damage only where quantization*added*error volume\. Across all cells this is exactly what happens\. The gap widens by15\.415\.4points in Gemma\-4 dense telecom, the one cell with a large added volume, while no other cell widens by more than0\.20\.2points\. The largest movement elsewhere is a2\.22\.2\-point*narrowing*in Gemma\-4 MoE telecom, whose event volume grows only about5%5\\%\. Where nothing was hidden, nothing is revealed\. The score is flat because the environment is forgiving, and the damage re\-emerges precisely where, and only where, it exists\. The test is not vacuous, since extra failures landing in episodes already lost would re\-expose nothing, and this partly happens\. All 46 INT4 exhaustions score zero yet the default\-budget gap is only1\.31\.3points, so part of the damage lands in episodes lost anyway\. With one volume\-adding cell the positive arm has a single member; the eight flat cells check the converse, and a rerun at a tightened budget is the decisive version of the test\.

BaseWith repairGemma\-4\-31B, telecomBF16INT4BF16INT4Tool\-name hallucination \(%\)19\.5138\.260\.422\.90Failed tool calls per episode1\.453\.640\.070\.21Agent tool calls per episode7\.299\.436\.316\.20Agent turns per episode16\.3617\.9516\.0415\.91Budget exhaustions of 45654600Recovery rate after a failure \(%\)55555252Final score \(%\)66\.6765\.3568\.8671\.27Table 2:Recovery accounting in the highest\-damage cell\. Quantization creates2\.5×2\.5\\timesmore failures and29%29\\%more tool calls per episode, yet the recovery rate after a failure is unchanged at55%55\\%\. The error budget absorbs the damage until it exhausts, growing from 5 to 46 episodes\. Reflexive repair re\-injects the tool inventory after each failed call, and the channel collapses at both precisions, exhaustion disappears, and trajectories*shorten*while the score is preserved\.

## Why Quantization Amplifies Existing Failures

#### A logit\-margin account\.

The amplify\-but\-do\-not\-invent pattern of Figure[2](https://arxiv.org/html/2607.27275#Sx4.F2)follows from a simple property of quantized decoding\. At any generation step, let the model choose among tokens by their logits, and consider the correct tokenccand its strongest competitorww, with marginm=zc−zwm=z\_\{c\}\-z\_\{w\}\. Weight quantization perturbs the logits by approximately zero\-mean noiseη\\eta, so the post\-quantization margin ism′=m\+ηm^\{\\prime\}=m\+\\eta\. The probability that the argmax flips \(that the model now errs\) changes byΔ​p​\(m\)=𝔼η​\[σ​\(−\(m\+η\)/τ\)\]−σ​\(−m/τ\)\\Delta p\(m\)=\\mathbb\{E\}\_\{\\eta\}\[\\sigma\(\-\(m\+\\eta\)/\\tau\)\]\-\\sigma\(\-m/\\tau\)for a temperature\-like scaleτ\\tau\. It is near zero for large positivemm\(confident decisions\) and large negativemm\(already lost\), and*maximized nearm≈0m\\approx 0*\. Quantization damage therefore concentrates on thin\-margin decisions\.

Two predictions follow, both borne out\. First, a thin\-margin flip promotes the*already*\-closest competitor, the failure the model occasionally made anyway, so quantization reuses the existing failure set \(0\.970\.97correlation,0\.18%0\.18\\%novel\)\. Second, baseline error rate and quantization\-induced increase are governed by the same latent quantity, the mass of thin\-margin decisions, so a model’s full\-precision failure rate is the*observable shadow*of its quantization sensitivity; the prediction is not circular, being tested at a finer grain, the per\-tool\-name distribution, than the aggregate rate it predicts\. A positional test separates this account from a natural alternative in which quantization erodes instruction\-following as the context grows\. That account predicts damage concentrated late, but the INT4\-to\-BF16 ratio stays within1\.3×1\.3\\times–2\.0×2\.0\\timesin every turn bin and the hallucination rate*falls*in the longest\-context quartile at both precisions\.

#### What governs susceptibility\.

The account explains the family split\. Qwen\-3\.6 adheres almost perfectly to the tool schema at full precision \(0\.38%0\.38\\%, 9 events\)\. It has essentially no thin\-margin mass here and nothing to amplify, genuinely robust rather than merely saturated, as the process metric confirms\. Gemma\-4 carries a standing tendency to invent plausible tools \(649 events\), and quantization magnifies exactly that tendency\. The claim is not “Gemma is worse” but “quantization scales up whatever a model is already prone to\.” Architecture does not set this propensity; the checkpoint does\. The Qwen releases have near\-zero baselines \(0\.26%0\.26\\%versus0\.38%0\.38\\%\) yet land on opposite sides of significance \(\+0\.95\{\+\}0\.95versus\+0\.09\{\+\}0\.09points\), so near the closed end the gate pins the*ceiling*, damage near1%1\\%versus38%38\\%where it is wide open, rather than the exact magnitude\. The converse is sharper than raw counts suggest\. Gemma\-4 MoE telecom opens the gate as wide per call as the dense variant \(19\.3%19\.3\\%versus19\.5%19\.5\\%at BF16, 841 events\) yet multiplies by only1\.05×1\.05\\times, per\-tool factors scattering0\.5×0\.5\\times–1\.5×1\.5\\timesinstead of the dense2\.5×2\.5\\times\. The gate is necessary for large damage, not sufficient; the aggregate rate cannot see the thin\-margin mass, which varies by cell\. The domain contrast follows the same logic\. Telecom’s dual\-control split places plausible device actions just across the agent/user role boundary, creating thin\-margin choices between near\-synonymous agent\- and user\-side names; single\-control retail has no boundary and a structurally zero tool\-name channel\.

## Discussion

#### Agentic benchmarks need process\-level reporting\.

End\-of\-episode reward is blind to quantization for a structural reason\. It is a saturating outcome buffered by the environment’s error tolerance, so a benchmark reporting only task reward certifies a compressed agent whose per\-step reliability has halved\. Benchmarks should report*per\-channel error rates*alongside task reward and, where an error budget exists, success as a function of that budget; both come from existing logs and are far more sensitive\.

#### Full\-precision propensity is a cheap risk screen\.

Susceptibility is governed by full\-precision propensity, so risk can be screened*before*quantizing by measuring the per\-channel error rate on the target domain\. A near\-zero rate, as for Qwen\-3\.6, signals 4\-bit is likely safe there\. The screen is one\-directional, an elevated rate flagging risk without pinning the multiplier, since an equally open gate can stay flat \(Gemma\-4 MoE telecom\); it still beats size and architecture\.

#### Deployment slack is not always benchmark slack\.

The masking depends on a forgiving environment\. Many real deployments are not forgiving, with single\-shot interactions, strict error limits, irreversible actions, and no silent retries\. With the slack gone, amplified per\-step failures become outcome failures, as the budget\-tightening analysis projects\. “Lossless on the benchmark” need not be lossless where deployed\.

#### A mechanism\-targeted mitigation\.

The damage is amplification of a small,*enumerable*set the model already produces at full precision, with99\.8%99\.8\\%of INT4 hallucinations repeating BF16 tool names, so the remedy is forbidding that set at the source rather than general robustness\. Constrained decoding, native in vLLM, eliminates the dominant channel by restricting the tool\-name field to the inventory\. We run the lighter reflexive repair system of Setup for five telecom models at all precisions; the one significant gain is the damaged cell, Gemma\-4 dense at INT4,\+5\.9\{\+\}5\.9points \(CI\[\+1\.1,\+10\.8\]\[\+1\.1,\+10\.8\],p=\.017p\{=\}\.017\) to71\.3%71\.3\\%, above unmitigated BF16 \(66\.7%66\.7\\%\), while every other interval includes zero\. The mechanism is prevention rather than cure\. The in\-context inventory collapses the out\-of\-list volume from1,6461\{,\}646events to8282and*shortens*trajectories \(17\.9→15\.917\.9\\\!\\to\\\!15\.9turns\)\. Both negative controls behave\. Qwen cells, with nothing to repair, move within or near the±2\.0\\pm 2\.0\-point repeat\-run band, and repair trends negative only where nothing needs fixing \(Gemma\-4 MoE, whose score dip is*not*channel\-mediated,−2\.6\-2\.6at BF16 and−4\.2\-4\.2at INT4; Qwen\-3\.6−2\.4\-2\.4at FP8; none significant\)\. The same prompt suppresses its errors only4×4\\times\(882→217882\\\!\\to\\\!217\) versus20×20\\timesdense, and its trajectories lengthen\. Help only where the channel damage lives, nothing where nothing needs fixing, mild harm elsewhere, a third confirmation of the localization\. The fix is per\-channel\. Retail’s entity/argument errors instead call for validate\-and\-retry or argument grounding, the propensity screen deciding when either is needed\.

#### Beyond quantization\.

Nothing in the logit\-margin account is specific to quantization\. Any weight perturbation adding near\-zero\-mean logit noise \(pruning, low\-rank compression, distillation drift\) should likewise amplify existing thin\-margin failures, not create new ones\. The agentic loop accumulates and hides such damage, making multi\-turn tool use a probe for perturbations single\-turn accuracy misses\.

## Conclusion

Standard evaluations call 4\-bit quantization nearly lossless, and our ten agentic cells agree, with no final\-score change surviving correction\. The agreement is an artifact of what the score can see\. Underneath, quantization amplifies the failures the model already exhibits, the same tools invented more often, up to2\.5×2\.5\\timeswhile creating almost nothing new\. The benchmark’s error budget absorbs the extra failures, and tightening it re\-exposes the damage exactly where, and only where, it was added, and a channel\-targeted repair recovers the score there\. Susceptibility follows the model’s own full\-precision propensity, not its size or architecture\. Measure process, not just outcome\.

## References

- The low\-resource double bind: an empirical study of pruning for low\-resource machine translation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,Punta Cana, Dominican Republic,pp\. 3316–3333\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.282),[Link](https://aclanthology.org/2021.findings-emnlp.282/)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan \(2026\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306\.Note:OralExternal Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p2.5),[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3),[Benchmark\.](https://arxiv.org/html/2607.27275#Sx3.SS0.SSS0.Px1.p1.6)\.
- Z\. Chen, W\. Du, W\. Zhang, K\. Liu, J\. Liu, M\. Zheng, J\. Zhuo, S\. Zhang, D\. Lin, K\. Chen, and F\. Zhao \(2024\)T\-eval: evaluating the tool utilization capability of large language models step by step\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9510–9529\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.515),[Link](https://aclanthology.org/2024.acl-long.515/)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.
- A\. Cuadron, P\. Yu, Y\. Liu, and A\. Gupta \(2026\)SABER: small actions, big errors – safeguarding mutating steps in LLM agents\.InICLR 2026 Workshop on Memory for LLM\-Based Agentic Systems \(MemAgents\),External Links:2512\.07850,[Link](https://arxiv.org/abs/2512.07850)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30318–30332\.External Links:[Document](https://dx.doi.org/10.52202/068431-2198),[Link](https://papers.nips.cc/paper_files/paper/2022/hash/c3ba4962c05c49636d4c6206a97e9c8a-Abstract-Conference.html)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- P\. Dong, Z\. Tang, X\. Liu, L\. Li, X\. Chu, and B\. Li \(2025\)Can compressed LLMs truly act? an empirical evaluation of agentic capabilities in LLM compression\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 14169–14202\.External Links:[Link](https://proceedings.mlr.press/v267/dong25k.html)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p2.5),[Compression under agentic workloads\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px3.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2023\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.InInternational Conference on Learning Representations,External Links:2210\.17323,[Link](https://openreview.net/forum?id=tcbBPnfwxS)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p1.1),[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- M\. N\. Haque, E\. Lin, L\. Arkoh, B\. Tadesse, and B\. Xu \(2025\)Secure or suspect? investigating package hallucinations of shell command in original and quantized LLMs\.External Links:2512\.08213,[Document](https://dx.doi.org/10.48550/arXiv.2512.08213),[Link](https://arxiv.org/abs/2512.08213)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Hooker, A\. Courville, G\. Clark, Y\. Dauphin, and A\. Frome \(2019\)What do compressed deep neural networks forget?\.External Links:1911\.05248,[Document](https://dx.doi.org/10.48550/arXiv.1911.05248),[Link](https://arxiv.org/abs/1911.05248)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- R\. Jin, J\. Du, W\. Huang, W\. Liu, J\. Luan, B\. Wang, and D\. Xiong \(2024\)A comprehensive evaluation of quantization strategies for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 12186–12215\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.726),[Link](https://aclanthology.org/2024.findings-acl.726/)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p1.1),[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Li, J\. Chen, R\. Ren, X\. Cheng, X\. Zhao, J\. Nie, and J\. Wen \(2024a\)The dawn after the dark: an empirical study on factuality hallucination in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 10879–10899\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.586),[Link](https://aclanthology.org/2024.acl-long.586/)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Li, X\. Ning, L\. Wang, T\. Liu, X\. Shi, S\. Yan, G\. Dai, H\. Yang, and Y\. Wang \(2024b\)Evaluating quantized large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 28480–28524\.External Links:[Link](https://proceedings.mlr.press/v235/li24bb.html)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p1.1),[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)AWQ: activation\-aware weight quantization for on\-device LLM compression and acceleration\.InProceedings of Machine Learning and Systems,Vol\.6,pp\. 87–100\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html)Cited by:[Introduction](https://arxiv.org/html/2607.27275#Sx1.p1.1),[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- V\. Paramanayakam, A\. Karatzas, I\. Anagnostopoulos, and D\. Stamoulis \(2025\)Less is more: optimizing function calling for LLM execution on edge devices\.In2025 Design, Automation & Test in Europe Conference \(DATE\),pp\. 1–7\.External Links:2411\.15399,[Link](https://ieeexplore.ieee.org/document/10992798)Cited by:[Compression under agentic workloads\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px3.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 48371–48392\.External Links:[Link](https://proceedings.mlr.press/v267/patil25a.html)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.
- S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez \(2024\)Gorilla: large language model connected with massive APIs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 126544–126565\.External Links:[Document](https://dx.doi.org/10.52202/079017-4020),[Link](https://papers.nips.cc/paper_files/paper/2024/hash/e4c61f578ff07830f5c37378dd3ecb0d-Abstract-Conference.html)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InInternational Conference on Learning Representations,External Links:2307\.16789,[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/28e50ee5b72e90b50e7196fde8ea260e-Abstract-Conference.html)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.
- P\. K\. Rath and R\. Maliakkal \(2026\)Quantization undoes alignment: bias emergence in compressed LLMs across models and precision levels\.Note:Accepted at IEEE Cloud Summit 2026; author’s accepted version, with the version of record to appear in IEEE XploreExternal Links:2605\.15208,[Document](https://dx.doi.org/10.48550/arXiv.2605.15208),[Link](https://arxiv.org/abs/2605.15208)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 38087–38099\.External Links:[Link](https://proceedings.mlr.press/v202/xiao23c.html)Cited by:[Post\-training quantization and its evaluation\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2025\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InInternational Conference on Learning Representations,External Links:2406\.12045,[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html)Cited by:[Evaluating tool\-calling agents\.](https://arxiv.org/html/2607.27275#Sx2.SS0.SSS0.Px2.p1.3)\.

## Appendix AAdditional Results

Figure[5](https://arxiv.org/html/2607.27275#A1.F5)visualizes the precision sweep BF16→\\toFP8→\\toINT4 for every cell of the paper’s grid; Table[4](https://arxiv.org/html/2607.27275#A1.T4)reports the BF16→\\rightarrowFP8 comparison under the main protocol\. Figure[6](https://arxiv.org/html/2607.27275#A1.F6)tests whether the amplification depends on position in the trajectory\. Table[3](https://arxiv.org/html/2607.27275#A1.T3)reports the full repair grid behind the mitigation paragraph of the Discussion, and a closing paragraph disentangles three conflated senses of “horizon\.”

Table 3:Effect of the reflexive repair system on the final score for five telecom models \(456 episodes per arm; deltas in points\)\. Exactly one arm is individually significant under the 95% task\-paired intervals:Gemma\-4\-31B at INT4\(\+5\.9\+5\.9points, CI\[\+1\.1,\+10\.8\]\[\+1\.1,\+10\.8\],p=\.017p\{=\}\.017\), the one cell in which quantization adds error volume; every other interval includes zero, the widest being Qwen\-3\.6\-27B at FP8 \(−2\.4\-2\.4, CI\[−5\.0,0\.0\]\[\-5\.0,0\.0\]\) and roughly±5\\pm 5points for Gemma arms\. The localization evidence is this cross\-arm pattern rather than a significant precision×\\timesrepair interaction, which the sample does not resolve on its own \(\+3\.7\+3\.7points, CI\[−3\.2,\+10\.7\]\[\-3\.2,\+10\.7\]\)\. Agent decoding is greedy in every arm; the repair arms differ from base only in the user\-simulator temperature \(0\.60\.6versus0\.70\.7\), a nuisance shared by all fifteen arms, so it cannot produce the one\-cell pattern\.aTwo runs of this configuration under identical settings; their task\-level comparison yields the repeat\-run band of±2\.0\\pm 2\.0points\.Table 4:8\-bit quantization, from BF16 to FP8, under the same protocol and intervals as Table[1](https://arxiv.org/html/2607.27275#Sx3.T1); deltas are in points and bold marks an interval excluding zero\. No cell degrades significantly in score and one nominally improves\. No channel moves significantly; the one visible trend is Gemma\-4\-26B\-A4B retail, whose\+1\.40\+1\.40\-point entity movement at 8 bits already matches its INT4\-sized trend\.#### Horizon is three things, pulling in opposite directions\.

“Longer horizons hurt more” conflates three quantities that act oppositely\. The*task horizon*, the decisions a task requires,*generates*damage, since more decisions expose more thin\-margin steps \(telecom, median∼\\sim16 turns, shows the effect; retail,∼\\sim11, does not\)\. The*budget horizon*\(max\_errors\)*hides*it, as the loop recovers the amplified failures until the tolerance is tightened\.*Realized trajectory length*is neither, being a downstream symptom of recovery, driven by errors \(r=0\.48r\{=\}0\.48\)\.

![Refer to caption](https://arxiv.org/html/2607.27275v1/x5.png)Figure 5:The cliff is \(mostly\) at four bits: the active\-channel rate across the precision sweep BF16→\\toFP8→\\toINT4 for every cell \(log scale; solid dense, dashed MoE, dotted 9B;∗\*marks a significant change vs\. BF16\)\. In telecom, rates are flat through FP8 and jump only at INT4 \(Gemma\-4\-31B, Qwen\-3\.5\-27B\)\. The one FP8 movement is Gemma\-4\-26B\-A4B retail, whose entity/argument trend is already fully present at 8 bits, though not significant under clustered intervals\.![Refer to caption](https://arxiv.org/html/2607.27275v1/x6.png)Figure 6:Hallucination rate by agent turn index \(Gemma\-4\-31B, telecom\)\. Both precisions follow the same hump, near zero in the opening turns, then a mid\-trajectory peak and a late decline\. The rate also falls in the longest\-context quartile at both precisions \(BF1634\.5%→27\.2%34\.5\\%\\to 27\.2\\%, INT461\.8%→41\.9%61\.8\\%\\to 41\.9\\%\), and the INT4\-to\-BF16 ratio stays within1\.3×1\.3\\times–2\.0×2\.0\\timesin every bin\. An account in which quantization erodes instruction\-following as context grows predicts the ratio to rise with position; the observed uniformity is what a per\-decision account predicts\.

## Appendix BLimitations

The paper’s strongest claims are within\-model, robust to the family confound; four limitations bound the generalization\. \(i\) Two families plus two further Qwen checkpoints support the propensity claim as a gate, open versus closed, not a fitted curve; a curve needs more checkpoints\. \(ii\) The budget\-masking result is a projection from existing trajectories, conservative by construction; a rerun at tightened budgets would confirm it directly, and the margin account is likewise indirect, since token log\-probabilities are not in our logs, though Figure[6](https://arxiv.org/html/2607.27275#A1.F6)rules out the leading alternative\. \(iii\) The INT4 recipes differ between families \(Table[6](https://arxiv.org/html/2607.27275#A4.T6)\), partly conflating checkpoint with recipe; the confound is bounded, since the finer\-grouped, normally more accurate recipe belongs to the*amplified*family and the recipe\-matched Qwen\-3\.5/3\.6 pair is separated by baseline propensity alone, but a same\-recipe requantization is the missing control\. \(iv\) Our evidence spans one benchmark, two domains, and weight\-only recipes on A100 GPUs with a 16\-bit KV cache, a fixed full\-precision user simulator, and a telecom\-only repair arm; low\-bit KV caches, activation\-quantized FP8, other benchmarks, and compressed user simulators are open\.

## Appendix CReproducibility Details

#### Benchmark and protocol\.

All runs use a singleτ2\\tau^\{2\}\-bench commit with its standardretailandtelecomdomains \(114 tasks each\),num\_trials=4\{=\}4, seed300300,max\_steps=300\{=\}300,max\_errors=10\{=\}10, agent sampling temperature0\.00\.0, and user\-simulator temperature0\.70\.7\(0\.60\.6in the repair arms, which also set an explicit agent seed\)\. One nonuniformity remains\. The Gemma\-4\-31B retail arms ran atmax\_steps=200\{=\}200\(BF16, INT4\) and300300\(FP8\); no episode in any arm reaches either cap, so the difference is immaterial\. The user simulator is Qwen\-3\.6\-27B at BF16 in every arm, served as a separate vLLM instance; only the agent model and its precision vary across cells\.

#### Serving\.

Agents are served with vLLM on NVIDIA A100 GPUs with a 16\-bit KV cache\. Because the A100 has no native FP8 compute, vLLM executes the FP8 checkpoints as weight\-only 8\-bit via the Marlin kernel \(W8A16\); the dynamic per\-token activation quantization specified in the FP8 configurations is therefore inactive, and activations are 16\-bit in all three arms\.

#### Quantization configurations\.

Table[6](https://arxiv.org/html/2607.27275#A4.T6)lists each arm’s weight\-quantization recipe as read from the served checkpoints’quantization\_config\.

#### Coincidence note\.

The identical score triples of the Qwen\-3\.6\-35B\-A3B and Qwen\-3\.5\-27B telecom rows in Table[1](https://arxiv.org/html/2607.27275#Sx3.T1)are a counting coincidence\. Both cells succeed in 448 of 456 episodes at BF16 and 439 at INT4, while the underlying episode outcomes differ in 16 and 32 episodes respectively\.

#### Analysis\.

Every number in the paper is computed from the raw simulation logs by a single analysis script \(channel regexes, per\-cell bootstrap and paired tests,S​\(K\)S\(K\)projection, and figure/table generation\); the script and per\-episode logs will be released upon publication\.

## Appendix DSample Trajectories

The channel rates in the paper are computed by pattern\-matching the per\-episode logs\. This section grounds that measurement in raw material\. Figure[7](https://arxiv.org/html/2607.27275#A4.F7)shows one matched pair of episodes, BF16 against INT4 on the same task; Table[D](https://arxiv.org/html/2607.27275#A4)decomposes the two active channels into the individual tool names and error strings that produce the aggregate rates; and Figure[8](https://arxiv.org/html/2607.27275#A4.F8)shows what the reflexive repair loop does to the same failure mode, in the one cell where it is individually significant\.

Table 5:Composition of the two active channels, over all 456 episodes per arm, as counts and as a percentage of agent tool calls\. The totals are the corresponding points of the precision\-sweep figure: panel \(a\) is the Gemma\-4\-31Btelecomcurve, flat from BF16 to FP8 and jumping at four bits; panel \(b\) is the Gemma\-4\-26B\-A4Bretailcurve, whose movement is already complete at eight bits\. Two properties of the tool\-name channel are visible only in the decomposition\. First, the damage is distributed over the tool inventory rather than driven by one pathological name: every entry at least doubles from FP8 to INT4\. Second, and more informative, all 22 distinct names the agent hallucinates across the INT4 arm are*real tools on the user’s side of the interface*, and not one is an invented string\. What quantization degrades here is therefore not name recall but the agent/user boundary: the model tries to actuate the user’s device directly instead of instructing the user to do so\. Panel \(b\) counts only unresolved entities and schema\-invalid arguments; refusals that correctly enforce a domain rule \(Non\-delivered order cannot be exchangedand similar, 23/35/31 calls\) are excluded, as they are policy behaviour rather than channel errors\.Table 6:Weight\-quantization recipes per arm\. Hereggis the quantization group size,128×128128\{\\times\}128denotes block\-wise weight scales, and sym\. marks symmetric quantization\. BF16 is the original 16\-bit release in all cases\. The Gemma INT4 checkpoints store weights in the compressed\-tensors format with an MSE observer and exclude the vision tower from quantization, while the Qwen INT4 checkpoints use standard zero\-point AWQ with GEMM kernels\. The Qwen\-3\.5\-9B arms use community releases whose published configurations match these recipes; their serving\-host configurations were not re\-read\. All FP8 checkpoints are executed weight\-only on our hardware, as described under Serving\.
\(a\) INT4reward 0\.0termination: too\_many\_errors43 messages

```
1 U   Hi, I’m John Smith. I’ve been having trouble sending MMS messages from my
       phone [...] My phone number is 555-123-2002.
 2 A   call get_customer_by_phone({"phone_number": "555-123-2002"})
 3 T   {"customer_id": "C1001", "line_ids": ["L1001","L1002","L1003"], [...]}
 4 A   call get_details_by_id({"id": "L1001"})                            [ok]
 6 A   call get_details_by_id({"id": "L1002"})                            [ok]
 8 A   call can_send_mms({})
 9 T   Error: Tool ’can_send_mms’ not found.                        <-- error 1
10 A   call can_send_mms({})                             (verbatim repetition)
11 T   Error: Tool ’can_send_mms’ not found.                        <-- error 2
       [... two further identical call/error pairs: errors 3, 4 ...]
16 A   "[...] could you please tell me if you are able to make phone calls and
       if your mobile data is working for other things [...]"
17 U   call check_status_bar({}) ; call check_network_status({})   [user side]
19 T   Airplane Mode: OFF [...] Cellular Network Type: 2G [...] Roaming: Yes
23 A   "Since you are on a 2G network, that is likely the cause of the problem.
       [...] please use the ‘set_network_mode_preference‘ tool on your phone
       and set the mode to ‘"4g_5g_preferred"‘."           <-- correct diagnosis
24 U   call set_network_mode_preference({"mode": "4g_5g_preferred"})  [applied]
25 T   Preferred Network Mode set to: 4g_5g_preferred | 5G | Excellent
30 U   "[...] the messaging app still says it cannot send MMS messages [...]"
31 A   call check_apn_settings({})
32 T   Error: Tool ’check_apn_settings’ not found.                  <-- error 5
       [... five further identical call/error pairs: errors 6-10 ...]
       >>> error budget exhausted (10/10); episode terminated, reward 0.0
```

\(b\) BF16, same task and trialreward 1\.0termination: user\_stop32 messages

```
1 U   Hi, my name is John Smith. [...] unable to send MMS messages [...] I’m
       currently traveling in France [...]
 2 A   "[...] Could you please provide me with your phone number or your
       customer ID? [...]"
 4 A   call get_customer_by_phone({"phone_number": "555-123-2002"})       [ok]
 6 A   call get_details_by_id({"id": "L1001"})                           [ok]
 8 A   call get_details_by_id({"id": "L1002"})                           [ok]
10 A   call enable_roaming({"customer_id": "C1001", "line_id": "L1002"})
11 T   Roaming enabled successfully
12 A   "[...] I noticed that data roaming was disabled for your line [...]
       Could you please try sending an MMS message now [...]"
13 U   call can_send_mms({})                                      [user side]
16 A   "Please use the following tools on your device: 1. ‘can_send_mms()‘ to
       confirm [...] 2. ‘check_network_status()‘ to see your signal strength
       [...]"                                    <-- delegates, does not call
19 U   call check_network_status({})                              [user side]
20 T   Airplane Mode: OFF [...] Cellular Network Type: 2G [...] Roaming: Yes
22 A   "[...] your phone is currently connected to a 2G network. [...] please
       use the ‘set_network_mode_preference(mode="4g_5g_preferred")‘ tool on
       your device."                                      <-- same diagnosis
23 U   call set_network_mode_preference({"mode": "4g_5g_preferred"})  [applied]
28 T   Your messaging app can send MMS messages.
31 U   "[...] Have a wonderful day as well, and safe travels! ###STOP###"
       >>> 0 tool-name errors in the episode; task solved, reward 1.0
```

Figure 7:One matched pair of episodes, Gemma\-4\-31B ontelecom, task\[mms\_issue\]bad\_network\_preference\|user\_abroad\_roaming\_disabled\_on\[PERSONA:None\], trial 3; the\[PERSONA:\*\]suffix is part of the benchmark’s task identifiers and persona conditioning is disabled \(persona\_config=null\) in every arm\. Columns are message index, speaker \(A = agent, U = user, T = tool result\), and content; long content is elided with\[\.\.\.\]and tool results are abbreviated to the fields under discussion\.*The two agents reach the same diagnosis*\(2G network, switch to4g\_5g\_preferred\) and the INT4 agent even gets the fix applied at message 24\. Its episode nonetheless scores 0\.0, because ten of its tool calls name device tools that belong to the*user’s*side of the interface \(can\_send\_mmsandcheck\_apn\_settings, both of which the user invokes successfully in the same episode\) and exhaust themax\_errors=10 budget\. The failure is located in the tool\-name channel, not in task reasoning, and it is the budget rather than the reasoning that converts it into a lost point\. Note also that the INT4 agent reissues the identical invalid call rather than revising it, which is the behaviour the reflexive repair arm targets\. Because the user simulator is sampled stochastically, the two episodes share the task, the trial index and the fixed BF16 user simulator, but not token\-identical user turns; the comparison is task\-level, as everywhere in the paper\.\(a\) INT4, base armreward 0\.0termination: too\_many\_errors60 messages

```
[38] A  call get_customer_by_phone({"phone_number": "555-123-2002"})     [ok]
[40] A  call check_sim_status({})
[41] T  Error: Tool ’check_sim_status’ not found.                  <-- error 1
[42] A  call check_sim_status({})                       (verbatim repetition)
[43] T  Error: Tool ’check_sim_status’ not found.                  <-- error 2
        [... eight further identical call/error pairs: errors 3-10 ...]
        >>> error budget exhausted (10/10); episode terminated, reward 0.0
```

\(b\) INT4, reflexive repair arm, same task and trialreward 1\.0termination: user\_stop52 messages

```
[34] A  call get_details_by_id({"id": "D1002"})                          [ok]
[36] A  ** repair fired: fail_types=[schema,schema] attempts=2 repaired=yes
        "[...] I’d like to run some diagnostics on your phone to figure out
        why you’re still seeing ’No Signal’ [...] I will check your device’s
        network status and SIM card [...]"
[38] A  ** repair fired: fail_types=[schema] attempts=1 repaired=yes
        "[...] Actually, I’ll start by checking your status bar to see
        exactly what icons are visible."
[39] U  call check_status_bar({})                                 [user side]
[40] T  Status Bar: No Signal | Data Disabled | 80%
[42] A  ** repair fired: fail_types=[schema x3] attempts=3 repaired=yes
        "[...] Could you please try removing your SIM card and reinserting
        it (reseating the SIM)? [...]"
[43] U  call reseat_sim_card({})                                  [user side]
[46] A  ** repair fired: fail_types=[schema,schema] attempts=2 repaired=yes
        "[...] I am trying to run some diagnostics on my end [...] but I’m
        encountering some technical difficulties with my tools. [...] I am
        going to transfer you to a human technical support agent [...]"
        call transfer_to_human_agents({"summary": "[...]"})
        >>> 0 tool-name errors in the episode; task solved, reward 1.0
```

\(c\) What the loop checks, what it injects, and how the arm was configured

```
# Every agent tool call is validated before it reaches the environment:
tool = self._tools_by_name.get(tc.name)
if tool is None:
    errors[i] = f"Unknown tool ’{tc.name}’."        # <- tool-name channel
else:
    tool.params.model_validate(tc.arguments)        # <- argument channel

# On failure the loop appends the rejected attempt plus one tool-role
# message per call, then regenerates; up to repair_max = 3 attempts:
"ERROR: {err} Reissue this tool call with corrected, schema-valid
 arguments."

# Arm configuration, as recorded in the run artifact (endpoints masked):
agent llm_args: {"temperature": 0.0, "seed": 300, "mitigation": "repair",
                 "repair_mode": "reflexive", "repair_max": 3,
                 "api_base": "http://<host>:<port>/v1"}
user  llm_args: {"temperature": 0.6,
                 "api_base": "http://<host>:<port>/v1"}
```

Figure 8:How the reflexive repair loop intercepts the tool\-name channel\. Gemma\-4\-31B INT4 ontelecom, the one individually significant cell of the repair grid, on task\[service\_issue\]airplane\_mode\_on\|break\_apn\_settings\|lock\_sim\_card\_pin\|overdue\_bill\_suspension\|unseat\_sim\_card\[PERSONA:None\], trial 1\. In the base arm \(a\) the agent names the user\-side toolcheck\_sim\_statusand then reissues the identical call ten consecutive times, consuming the entire error budget in ten messages\. In the repair arm \(b\) the validator in \(c\) rejects the same class of call before it reaches the environment; at each of the four marked turns the regenerated message is policy\-conformant, instructing the user to run the device tool or escalating, and the episode reaches a reward of 1\.0 with no tool\-name errors at all\.*Two limitations of this excerpt should be read with it\.*First, the logs retain only the accepted message and its repair record \(fail\_types,attempts,repaired\), not the text of the rejected attempts, so panel \(b\) shows where the loop fired and what it accepted rather than what it discarded;fail\_types=schemais the validator category that includesUnknown tool, i\.e\. the tool\-name channel\. Second, a single episode pair cannot be causal evidence: the base and repair arms differ in their recorded sampling configuration \(see Reproducibility Details\), and the two episodes are not turn\-aligned\. The excerpt illustrates the mechanism; the evidence that the mechanism is localized to this cell is the cross\-arm pattern across the repair grid\.