Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

# A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit
Source: [https://arxiv.org/html/2605.28873](https://arxiv.org/html/2605.28873)
## Pre\-Registering the Detectable Effect: A Paired\-MDE Budget for 4\-bit Quantization Benchmarks, with a Pilot Audit

###### Abstract

This is a planning\-method note with an unpaired pilot audit\. We adapt the classical paired\-binary sample\-size calculation\(Miettinen,[1968](https://arxiv.org/html/2605.28873#bib.bib22)\)to quantization benchmarks, giving a conservative*minimum detectable effect*\(MDE\) boundδ∗≤\(z1−α/2\+z1−β\)​ρd/m\\delta^\{\*\}\\leq\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\sqrt\{\\rho\_\{d\}/m\}in the paired item countmmand the FP16↔\\leftrightarrowNF4 disagreement rateρd\\rho\_\{d\}\. The bound turns “how reliable is my quantization claim?” into a one\-line budget a benchmark designer can commit to*before running*\. We illustrate the bound on four models and four benchmarks \(k=5k\{=\}5splits ofn=100n\{=\}100\), and add a parallel MMLU prompt\-template study to put the bound’s quantization\-noise scale alongside the prompt\-noise scale\. Assumingρd=0\.10\\rho\_\{d\}\{=\}0\.10\(an unmeasured planning value\), all observed NF4−\-FP16 deltas fall below the implied MDE, and most cross\-split SDs lie within±1\.5\\pm 1\.5pp of the binomial referencep​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}, so much of the variance reported as “benchmark unreliability” onn=100n\{=\}100subsamples is binomial sampling noise\. The single borderline cell \(OPT–WinoGrande,\|Δ\|=3\.2\|\\Delta\|\{=\}3\.2pp\) is below the implied MDE atρd=0\.10\\rho\_\{d\}\{=\}0\.10but above it atρd=0\.05\\rho\_\{d\}\{=\}0\.05, illustrating the planning trade\-off the bound makes explicit\.On MMLU, prompt\-template ranges of 2–10 pp meet or exceed the largest observed quantization delta \(3\.2 pp\), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor\.We complement the bound with a five\-line pre\-registration template\.

quantization, benchmark reliability, minimum detectable effect, prompt sensitivity, evaluation methodology, language models

## 1Introduction

Post\-training quantization is now the default deployment path for large language models on consumer hardware\. NF4 quantization\(Dettmers et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib8)\)packs each weight into one of 16 levels chosen to be optimal for normally distributed weights; GPTQ\(Frantar et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib12)\), AWQ\(Lin et al\.,[2024](https://arxiv.org/html/2605.28873#bib.bib20)\), SmoothQuant\(Xiao et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib28)\), SpQR\(Dettmers et al\.,[2024](https://arxiv.org/html/2605.28873#bib.bib9)\), and ZeroQuant\(Yao et al\.,[2022](https://arxiv.org/html/2605.28873#bib.bib29)\)have all reported small\-to\-modest accuracy losses on standard benchmarks\. A typical quantization study reports a single accuracy number per model×\\timesbenchmark cell and concludes that 4\-bit inference “preserves” or “slightly degrades” performance\.

This practice elides a basic statistical question: what is the smallest quantization effect that the evaluation protocol could reliably detect in the first place? Withnnbenchmark items and a baseline accuracy ofpp, even a perfectly faithful estimator has standard error at leastp​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}\. Withkknon\-overlapping splits, the cross\-split standard deviation has expected scalep​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}unless the benchmark itself is structured \(e\.g\., MMLU’s 57 subjects make subject mix vary across random splits\); at finitekkobserved cross\-split SD can fall both above and below this binomial reference, but the reference still pins the order of magnitude\. Reports of “small” quantization effects therefore confound two distinct claims: \(i\) the population effect is small, and \(ii\) the protocol cannot detect a small effect even if one exists\. CTB\-style ex\-ante guarantees should distinguish them\.

We pursue this distinction along three threads\.

#### An ex\-ante reliability bound\.

We derive a conservative paired minimum detectable effect \(MDE\) for FP16\-vs\-NF4 comparisons \(Section[3](https://arxiv.org/html/2605.28873#S3)\)\. Given a paired sample ofmmitems and a pre\-registered upper boundρd\\rho\_\{d\}on per\-example disagreement, the MDE at significanceα\\alphaand power1−β1\-\\beta\(we useβ\\betafor Type\-II error throughout\) isδ∗​\(m,ρd\)≤\(z1−α/2\+z1−β\)​ρd/m\\delta^\{\*\}\(m,\\rho\_\{d\}\)\\;\\leq\\;\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\,\\sqrt\{\\rho\_\{d\}/m\}, wheremmequalsnnfor a single\-split estimand andk​nknfor an aggregate\-of\-splits estimand\. The bound is a quantization\-flavored repackaging of paired\-binary sample\-size planning\(Miettinen,[1968](https://arxiv.org/html/2605.28873#bib.bib22); Connor,[1987](https://arxiv.org/html/2605.28873#bib.bib5); Lachin,[1992](https://arxiv.org/html/2605.28873#bib.bib17)\); the contribution is its use as an ex\-ante budget line for quantization audits, not the underlying inequality\.

#### An empirical audit\.

We measure FP16\-vs\-NF4 accuracy on four models \(OPT\-2\.7B, Pythia\-2\.8B, Llama\-2\-7B, Mistral\-7B\) and four benchmarks \(MMLU, ARC\-Easy, WinoGrande, HellaSwag\) withk=5k\{=\}5non\-overlapping splits ofn=100n\{=\}100each\. We separately measure prompt\-template sensitivity on MMLU with three templates andn=50n\{=\}50each\. Section[5](https://arxiv.org/html/2605.28873#S5)reports the observed accuracy means, cross\-split standard deviations, and template spreads\. To diagnose how much of the observed cross\-split variance is binomial sampling and how much is between\-split, we compareσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}to the binomial reference SDp^​\(1−p^\)/n\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}on each cell\.

#### A reliability index\.

We formalize a Quantization Reliability Index \(QRI\) as a single\-split signal\-to\-noise heuristic and explicitly separate its split\-only variantQRIsplit\\mathrm\{QRI\}\_\{\\text\{split\}\}\(defined for all 16 cells\) from its split\+\+prompt variantQRIcombined\\mathrm\{QRI\}\_\{\\text\{combined\}\}\(defined only on the four MMLU cells where prompt variance was measured\)\. QRI is not a hypothesis test; it is meant to flag cells where a single\-split comparison is dominated by evaluation noise rather than by the quantization signal\.

#### What this paper does*not*claim\.

We do not run paired McNemar tests on per\-example correctness \(we did not retain the per\-example records under our compute budget\); we therefore cannot rule out paired effects smaller than what the unpaired protocol can resolve\. We do not attempt to predictδ\\deltaex\-ante from weight statistics; the predictor forδ\\deltaitself is open\. We test only NF4 via BitsAndBytes; GPTQ, AWQ, and SpQR may produce different reliability landscapes\. The 7B side of our design has only two models, so within\-7B claims are observational\. These are stated again in Section[9](https://arxiv.org/html/2605.28873#S9)\.

#### Contributions\.

1. 1\.A conservative paired MDE bound for FP16\-vs\-NF4 quantization comparisons \(Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)\) and a corresponding sample\-size table that turns “how reliable is my quantization claim” into a one\-line pre\-registration item\.
2. 2\.A pilot audit including a per\-cell binomial\-reference table \(Appendix[B](https://arxiv.org/html/2605.28873#A2)\) showing that 25 of 32 observed cross\-split SDs fall within±1\.5\\pm 1\.5pp ofp​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}\(and 29 of 32 within±2\.0\\pm 2\.0pp\), so on this audit most of what gets reported as “benchmark unreliability” is small\-nnbinomial sampling rather than a property of the specific model or of NF4\.
3. 3\.A per\-cell Quantization Reliability Index \(Appendix[C](https://arxiv.org/html/2605.28873#A3)\) as a descriptive signal\-to\-noise diagnostic, with a split\-only variant defined for all 16 cells and a prompt\-augmented variant defined for the 4 MMLU cells where prompt variance was measured\. QRI is not a power test in this paper; power\-controlled decisions are made directly via the\|Δ^\|\|\\hat\{\\Delta\}\|\-vs\-δ∗\\delta^\{\*\}comparison in Table[6](https://arxiv.org/html/2605.28873#A3.T6)\.
4. 4\.Recommendations: pre\-register an MDE target, report paired discordant counts\(n10,n01\)\(n\_\{10\},n\_\{01\}\), separate binomial sampling noise from subset\-composition noise on clustered benchmarks like MMLU, and sweep prompt templates on every benchmark used in a comparison\.

## 2Related Work

#### Post\-Training Quantization\.

Frantar et al\. \([2023](https://arxiv.org/html/2605.28873#bib.bib12)\)andDettmers et al\. \([2023](https://arxiv.org/html/2605.28873#bib.bib8)\)introduced GPTQ and QLoRA/NF4\.Dettmers et al\. \([2022](https://arxiv.org/html/2605.28873#bib.bib7)\)\(LLM\.int8\) handles activation outliers via 8\-bit matrix multiplication;Lin et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib20)\)\(AWQ\),Xiao et al\. \([2023](https://arxiv.org/html/2605.28873#bib.bib28)\)\(SmoothQuant\),Dettmers et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib9)\)\(SpQR\), andYao et al\. \([2022](https://arxiv.org/html/2605.28873#bib.bib29)\)\(ZeroQuant\) cover the recent low\-bit PTQ landscape;Lu et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib21)\)relates spectral weight properties to layer\-wise compression decisions\. Benchmark\-style comparisons of PTQ strategies appear inLi et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib19)\),Gong et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib13)\),Jin et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib16)\), andZhao et al\. \([2025](https://arxiv.org/html/2605.28873#bib.bib34)\)\. None of these works isolates the*minimum detectable quantization effect*as a function of benchmark sample size\.

#### Benchmark Evaluation\.

Liang et al\. \([2023](https://arxiv.org/html/2605.28873#bib.bib18)\)\(HELM\) andPolo et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib24)\)examine evaluation breadth and subset selection but not quantization\.Biderman et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib3)\)document the hidden statistical fragility of LM evaluation harnesses\.Dror et al\. \([2018](https://arxiv.org/html/2605.28873#bib.bib10)\)systematize statistical significance testing in NLP\. Complementary scaling\-law analyses of LLM behavior\(Shi et al\.,[2026](https://arxiv.org/html/2605.28873#bib.bib40)\)and time\-series forecasting\(Shi et al\.,[2024](https://arxiv.org/html/2605.28873#bib.bib39)\)characterize model capability at scale, and design\-axis frameworks for LLM evaluation have been proposed in adjacent areas including retrieval\-augmented QA\(Ji et al\.,[2026](https://arxiv.org/html/2605.28873#bib.bib35)\)and benchmark construction for generative AI\(Luo et al\.,[2026a](https://arxiv.org/html/2605.28873#bib.bib36)\); our paired\-MDE template adds the precision\-vs\-precision detectability axis for quantization\.

#### Prompt Sensitivity\.

Sclar et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib26)\)report accuracy swings of up to 76 pp from formatting changes;Mizrahi et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib23)\)benchmark prompt\-template sensitivity at scale; broader prompting\-evaluation concerns have also been documented in adjacent modalities\(Luo et al\.,[2026b](https://arxiv.org/html/2605.28873#bib.bib37)\), reinforcing the recommendation to sweep templates on every benchmark used in a precision audit\. We study the interaction of prompt sensitivity with quantization at then=50n\{=\}50scale, framing our findings as exploratory rather than confirmatory\.

## 3Minimum Detectable Effect under Paired Quantization Comparison

![Refer to caption](https://arxiv.org/html/2605.28873v1/figures/ai_generated/figure_final.png)Figure 1:Pre\-registerable reliability audit pipeline for paired FP16 vs\. NF4 quantization benchmarks\. The designer commits\(m,ρdprior,α,1−β\)\(m,\\rho\_\{d\}^\{\\mathrm\{prior\}\},\\alpha,1\{\-\}\\beta\)in advance \(A\); Proposition[1](https://arxiv.org/html/2605.28873#Thmproposition1)turns these into a conservative paired Minimum Detectable Effectδ∗\\delta^\{\\ast\}\(B\) that bounds what any later pilot can detect\. The pilot audit \(C\) computes per\-item differencesDi=Xi−Yi∈\{−1,0,\+1\}D\_\{i\}=X\_\{i\}\-Y\_\{i\}\\in\\\{\-1,0,\+1\\\}overk=5k\{=\}5non\-overlapping splits ofn=100n\{=\}100items and reportsΔ^\\hat\{\\Delta\}and the cross\-split SDσ^split\\hat\{\\sigma\}\_\{\\mathrm\{split\}\}\. The Quantization Reliability Index \(E\),QRI=δsplit∗/σ^split\\mathrm\{QRI\}=\\delta^\{\\ast\}\_\{\\mathrm\{split\}\}/\\hat\{\\sigma\}\_\{\\mathrm\{split\}\}, is a descriptive single\-split signal\-to\-noise diagnostic\. The detectability verdict \(D\) compares\|Δ^\|\|\\hat\{\\Delta\}\|toδ∗\\delta^\{\\ast\}: effects above the bound are guaranteed\-detectable in design, while effects below it are merely*not*guaranteed\-detectable, not null\.#### Setup and assumptions\.

LetXi∈\{0,1\}X\_\{i\}\\in\\\{0,1\\\}be the FP16 correctness of itemiiandYi∈\{0,1\}Y\_\{i\}\\in\\\{0,1\\\}the NF4 correctness on the*same*item,i=1,…,mi=1,\\dots,m\. Define the per\-item accuracy differenceDi=Xi−Yi∈\{−1,0,\+1\}D\_\{i\}=X\_\{i\}\-Y\_\{i\}\\in\\\{\-1,0,\+1\\\}, the disagreement rateρd=Pr⁡\(Di≠0\)\\rho\_\{d\}=\\Pr\(D\_\{i\}\\neq 0\), and the population effectδ=\|𝔼​\[Di\]\|\\delta=\|\\mathbb\{E\}\[D\_\{i\}\]\|\. We assume \(A1\) items are i\.i\.d\. samples from a target distribution, \(A2\) model outputs are deterministic conditional on the prompt and precision \(no temperature sampling\), and \(A3\)ρd\\rho\_\{d\}is either pre\-registered or conservatively bounded a priori\. Section[9](https://arxiv.org/html/2605.28873#S9)discusses how clustering on MMLU subjects and our finite\-population partitions weaken \(A1\)\.

#### Paired\-difference variance\.

The exact per\-item variance is

Var​\(Di\)=𝔼​\[Di2\]−𝔼​\[Di\]2=ρd−δ2,\\mathrm\{Var\}\(D\_\{i\}\)\\;=\\;\\mathbb\{E\}\[D\_\{i\}^\{2\}\]\-\\mathbb\{E\}\[D\_\{i\}\]^\{2\}\\;=\\;\\rho\_\{d\}\-\\delta^\{2\},\(1\)so the mean overmmitems has variance\(ρd−δ2\)/m≤ρd/m\(\\rho\_\{d\}\-\\delta^\{2\}\)/m\\leq\\rho\_\{d\}/m\. The mild upper boundVar​\(Di\)≤ρd\\mathrm\{Var\}\(D\_\{i\}\)\\leq\\rho\_\{d\}is tight only asδ→0\\delta\\to 0\.

#### Conservative MDE bound\.

Letμ=𝔼​\[Di\]\\mu=\\mathbb\{E\}\[D\_\{i\}\]andδ=\|μ\|\\delta=\|\\mu\|, withδ≤ρd\\delta\\leq\\rho\_\{d\}since\|Di\|≤𝟙​\[Di≠0\]\|D\_\{i\}\|\\leq\\mathbb\{1\}\[D\_\{i\}\\neq 0\]\. We state the central result as a formal proposition for ease of citation\.

###### Proposition 1\(Conservative paired MDE for FP16\-vs\-NF4 benchmarks\)\.

Let\{Di\}i=1m\\\{D\_\{i\}\\\}\_\{i=1\}^\{m\}be i\.i\.d\. paired differences in\{−1,0,\+1\}\\\{\-1,0,\+1\\\}with disagreement rateρd=Pr⁡\(Di≠0\)\\rho\_\{d\}=\\Pr\(D\_\{i\}\\neq 0\)and population effectδ=\|𝔼​\[Di\]\|\\delta=\|\\mathbb\{E\}\[D\_\{i\}\]\|\. Under \(A1\)–\(A3\), the normal\-approximation two\-sidedzz\-test ofH0:𝔼​\[Di\]=0H\_\{0\}:\\mathbb\{E\}\[D\_\{i\}\]=0at levelα\\alpharejects with power at least1−β1\-\\betafor anyδ\\deltasatisfying

δ≥\(z1−α/2\+z1−β\)​ρd/m\.\\delta\\;\\geq\\;\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\,\\sqrt\{\\rho\_\{d\}/m\}\.The smallest suchδ\\delta, denotedδ∗​\(m,ρd,α,β\)\\delta^\{\*\}\(m,\\rho\_\{d\},\\alpha,\\beta\), is therefore upper\-bounded by the right\-hand side\.

In benchmark\-design terms: a benchmark of paired sizemmwith worst\-case disagreement rateρd\\rho\_\{d\}is not planned to have the stated power for FP16\-vs\-NF4 effects below this threshold, regardless of which model or NF4 implementation is being audited; effects above the bound are guaranteed\-detectable in design but effects below it are merely*not guaranteed\-detectable*, not impossible\. The bound is conservative in two senses: \(i\)ρd\\rho\_\{d\}replaces the tighterρd−δ2\\rho\_\{d\}\-\\delta^\{2\}as the variance proxy \(loose only whenδ2\\delta^\{2\}is non\-negligible relative toρd\\rho\_\{d\}\), and \(ii\) the z\-test is two\-sided \(one\-sided NF4\-degrades\-only tests would tighten byz1−α/2→z1−αz\_\{1\-\\alpha/2\}\\\!\\to\\\!z\_\{1\-\\alpha\}\)\. The two simplifications together yield a single closed\-form line that benchmark designers can pre\-register before evaluation\.

For citation throughout the paper we restate this as

δ∗​\(m,ρd,α,β\)≤\(z1−α/2\+z1−β\)​ρdm\.\\delta^\{\*\}\(m,\\rho\_\{d\},\\alpha,\\beta\)\\;\\leq\\;\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\,\\sqrt\{\\frac\{\\rho\_\{d\}\}\{m\}\}\.\(2\)This is a*conservative sufficient bound*, not a necessary impossibility threshold\. The standard alternative\-variance approximation, which uses the null varianceρd\\rho\_\{d\}in the rejection\-region term and the alternative varianceρd−δ2\\rho\_\{d\}\-\\delta^\{2\}in the power term, gives

δ∗​m=z1−α/2​ρd\+z1−β​ρd−\(δ∗\)2,\\delta^\{\*\}\\sqrt\{m\}\\;=\\;z\_\{1\-\\alpha/2\}\\sqrt\{\\rho\_\{d\}\}\+z\_\{1\-\\beta\}\\sqrt\{\\rho\_\{d\}\-\(\\delta^\{\*\}\)^\{2\}\},which is implicit but numerically very close to Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)for the small\-ρd\\rho\_\{d\}regimes we care about; the maximum tightening over Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)atρd=0\.10\\rho\_\{d\}\{=\}0\.10,m=500m\{=\}500,α=0\.05\\alpha\{=\}0\.05,1−β=0\.801\-\\beta\{=\}0\.80is below0\.10\.1pp\. We use Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)throughout for simplicity and label it conservative\. Equation[2](https://arxiv.org/html/2605.28873#S3.E2)is a quantization\-flavored repackaging of classical paired\-binary sample\-size planning\(Miettinen,[1968](https://arxiv.org/html/2605.28873#bib.bib22); Connor,[1987](https://arxiv.org/html/2605.28873#bib.bib5); Lachin,[1992](https://arxiv.org/html/2605.28873#bib.bib17)\)and is closely related to NLP significance\-testing practice\(Yeh,[2000](https://arxiv.org/html/2605.28873#bib.bib30); Dror et al\.,[2018](https://arxiv.org/html/2605.28873#bib.bib10)\); the contribution is its application as an ex\-ante budget line for quantization benchmarks, not the underlying inequality\.

#### Single\-split vs\. aggregatemm\.

Equation[2](https://arxiv.org/html/2605.28873#S3.E2)applies to whatever paired sample is actually used to estimateδ\\delta\. Withkknon\-overlapping splits ofnnitems each, two natural targets exist:

1. 1\.Single\-split MDE\(m=n=100m\{=\}n\{=\}100\): the smallest paired effect resolvable on*one*split\. Forρd=0\.10\\rho\_\{d\}\{=\}0\.10this isδn∗≈8\.9\\delta^\{\*\}\_\{n\}\\approx 8\.9pp\.
2. 2\.Aggregate MDE\(m=k​n=500m\{=\}kn\{=\}500\): the smallest paired effect resolvable on the*union*of the five splits, which is what Tables[1](https://arxiv.org/html/2605.28873#S5.T1)–[2](https://arxiv.org/html/2605.28873#S5.T2)actually report\. Forρd=0\.10\\rho\_\{d\}\{=\}0\.10this isδk​n∗≈4\.0\\delta^\{\*\}\_\{kn\}\\approx 4\.0pp; forρd=0\.05\\rho\_\{d\}\{=\}0\.05it is≈2\.8\\approx 2\.8pp\.

The dependence onkkenters only through the total item countk​nkn; aggregating non\-overlapping splits does not buy power beyond that\. We will be explicit in §[5](https://arxiv.org/html/2605.28873#S5)about whichmmeach comparison uses\.

#### Connection to cross\-split SD\.

Cross\-split SDσsplit\\sigma\_\{\\text\{split\}\}fromkksplits ofnnitems has expected magnitude of orderp​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}under independent binomial sampling, regardless of paired structure\. Withp=0\.45p\{=\}0\.45andn=100n\{=\}100,σsplit≈5\.0\\sigma\_\{\\text\{split\}\}\\approx 5\.0pp, comparable in magnitude to the single\-split paired MDE atρd∈\[0\.05,0\.20\]\\rho\_\{d\}\\in\[0\.05,0\.20\]\. A study that compares\|Δquant\|\|\\Delta\_\{\\text\{quant\}\}\|aggregated overkksplits to a single\-splitσsplit\\sigma\_\{\\text\{split\}\}is mixing scales: aggregate signal vs\. split\-level noise\. Reporting both targets, or reporting a single consistentmm, fixes the mismatch\.

#### Ex\-ante use\.

Equation[2](https://arxiv.org/html/2605.28873#S3.E2)converts the abstract notion of “benchmark reliability” into a pre\-registerable budget line:*withmmitems and a conservative priorρd≤ρd*prior*\\rho\_\{d\}\\leq\\rho^\{\\text\{prior\}\}\_\{d\}, my evaluation can detect a paired NF4 effect no smaller thanδ∗\\delta^\{\*\}pp at\(α,1−β\)\(\\alpha,1\-\\beta\), conditional on \(A1\)–\(A3\)\.*A benchmark designer who pre\-registers\(m,ρdprior,α,1−β\)\(m,\\rho^\{\\text\{prior\}\}\_\{d\},\\alpha,1\-\\beta\)and then reports paired discordant counts\(n10,n01\)\(n\_\{10\},n\_\{01\}\)can also retroactively check the assumedρdprior\\rho^\{\\text\{prior\}\}\_\{d\}against the observedρ^d=\(n10\+n01\)/m\\hat\{\\rho\}\_\{d\}=\(n\_\{10\}\+n\_\{01\}\)/m\. Our audit is unable to run this paired retrospective check \(we did not retain per\-example correctness; see Section[9](https://arxiv.org/html/2605.28873#S9)\); we therefore report only ex\-ante MDE budgets and treat the empirical accuracy deltas as observations rather than tested differences\.

## 4Methods

### 4\.1Models and Quantization

We evaluate four models at two scale tiers, FP16 and NF4:

- •OPT\-2\.7B\(Zhang et al\.,[2022](https://arxiv.org/html/2605.28873#bib.bib32)\), a multi\-head\-attention decoder\.
- •Pythia\-2\.8B\(Biderman et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib1)\), trained on the Pile with controlled procedures\.
- •Llama\-2\-7B\(Touvron et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib27)\), a 7B model that uses standard multi\-head attention \(only the 34B and 70B Llama\-2 variants use grouped\-query attention; we previously misstated this\)\.
- •Mistral\-7B\(Jiang et al\.,[2023](https://arxiv.org/html/2605.28873#bib.bib15)\), a 7B model with sliding\-window and grouped\-query attention\.

NF4 is applied via BitsAndBytes \(load\_in\_4bit=True, bnb\_4bit\_quant\_type=‘nf4’, bnb\_4bit\_compute\_dtype=float16, double\-quant disabled\)\. 3B models run on a single NVIDIA T4 \(15GB\); 7B models require an A100 \(40GB\) in FP16 and fit on a T4 under NF4\. Exact Hugging Face identifiers, library versions, and seeds are in Appendix[A](https://arxiv.org/html/2605.28873#A1)\.

### 4\.2Benchmarks and Splits

We use MMLU\(Hendrycks et al\.,[2021](https://arxiv.org/html/2605.28873#bib.bib14)\)\(validation\), ARC\-Easy\(Clark et al\.,[2018](https://arxiv.org/html/2605.28873#bib.bib6)\)\(test\), WinoGrande\(Sakaguchi et al\.,[2020](https://arxiv.org/html/2605.28873#bib.bib25)\)\(validation\), and HellaSwag\(Zellers et al\.,[2019](https://arxiv.org/html/2605.28873#bib.bib31)\)\(validation\)\. Each benchmark is randomly partitioned intok=5k\{=\}5non\-overlapping splits ofn=100n\{=\}100examples each \(seed 42; split indices in Appendix[A](https://arxiv.org/html/2605.28873#A1)\)\. “Non\-overlapping” is technically distinct from “independent” in the inferential sense: random partitions of one benchmark are exchangeable but not i\.i\.d\. samples from the underlying population, so cross\-split SD does not directly estimate the population sampling SD; finite\-population correction can pull it down, while subject clustering on benchmarks like MMLU can push it up\. We treatσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}as a protocol\-level descriptor rather than a population estimator\.

### 4\.3Scoring

For multiple\-choice items \(MMLU, ARC, HellaSwag\) we score by ranking the per\-choice log\-likelihoods of the answer\-only continuation under a fixed prompt template; for WinoGrande we score by likelihood ratio of the two pronoun resolutions\. “Perplexity” values reported in Table[2](https://arxiv.org/html/2605.28873#S5.T2)are per\-token NLLs of the prompt text \(question\+\+choices, excluding the gold\-answer continuation\), averaged across examples and exponentiated, with the model’s native tokenization\. We report PPL only as a descriptive complement; PPL changes are not used to support significance claims\.

### 4\.4Prompt Sensitivity

For MMLU only, we evaluate three templates onn=50n\{=\}50examples each, sharing examples across templates within a run: \(T0\) standard “Question: … Answer:”; \(T1\) compact “Q: … Options: A\. …”; \(T2\) inline “… \(A\) … The answer is”\. We do not extend the prompt\-sensitivity sweep to ARC, WinoGrande, or HellaSwag in this study; this is reflected in the QRI definitions below\.

## 5Results: Illustrative Pilot Audit

The empirical content of this section is*illustrative*: the pilot did not retain per\-example correctness, so we cannot empirically estimateρd\\rho\_\{d\}, run paired McNemar tests, or validate the paired audit machinery on this data\. The numbers below are therefore a worked example of how the methodology of §[3](https://arxiv.org/html/2605.28873#S3)would be applied, not a tested set of empirical claims about NF4 across the population of LLM benchmarks\.

Table 1:Benchmark accuracy onk=5k\{=\}5non\-overlapping splits ofn=100n\{=\}100each, mean±\\pmcross\-split SD\. Numbers are observed values on our subsamples; population CIs are wider \(see text and Appendix[D](https://arxiv.org/html/2605.28873#A4)\)\.Table 2:NF4−\-FP16 accuracy delta and FP16 prompt PPL with NF4−\-FP16 PPL delta\. ReportedΔ\\DeltaAcc is the aggregate overm=500m\{=\}500items per cell\. Aggregate paired MDE atα=0\.05\\alpha\{=\}0\.05, power1−β=0\.801\-\\beta\{=\}0\.80isδm=500∗≈4\.0\\delta^\{\*\}\_\{m=500\}\\approx 4\.0pp atρd=0\.10\\rho\_\{d\}\{=\}0\.10and≈2\.8\\approx 2\.8pp atρd=0\.05\\rho\_\{d\}\{=\}0\.05\. Of the 16 cells, only OPT\-WinoGrande \(\|Δ\|=3\.2\|\\Delta\|\{=\}3\.2pp\) approaches either bound: it is belowδ∗\\delta^\{\*\}atρd=0\.10\\rho\_\{d\}\{=\}0\.10and*above*δ∗\\delta^\{\*\}atρd=0\.05\\rho\_\{d\}\{=\}0\.05\. Becauseρd\\rho\_\{d\}was not measured, we cannot adjudicate detectability without paired discordant counts\.### 5\.1Observed Accuracy and the Binomial Reference SD

![Refer to caption](https://arxiv.org/html/2605.28873v1/x1.png)Figure 2:Observed cross\-split SD vs\. binomial reference SD on the 32 audited cells \(Table[4](https://arxiv.org/html/2605.28873#A2.T4)\)\. Marker shape encodes precision \(FP16 vs\. NF4\); colour encodes model\. The diagonal isσ^split=σbin\\hat\{\\sigma\}\_\{\\mathrm\{split\}\}=\\sigma\_\{\\mathrm\{bin\}\}and the shaded band is±1\.5\\pm 1\.5pp around it\. 25 of 32 points fall inside the band, so most observed cross\-split variation onn=100n\{=\}100subsamples is accounted for by binomial sampling rather than by quantization\-specific or split\-composition noise\.Table[1](https://arxiv.org/html/2605.28873#S5.T1)reports the mean and cross\-split SD for each cell; the largest observedσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}is 7\.3 pp \(OPT\-NF4 MMLU\), and the 7B tier bottoms out near 2\.9–3\.6 pp where accuracy approaches0\.70\.7\. Figure[2](https://arxiv.org/html/2605.28873#S5.F2)compares eachσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}to the binomial referenceσbin​\(p^\)=p^​\(1−p^\)/n\\sigma\_\{\\text\{bin\}\}\(\\hat\{p\}\)=\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}atn=100n\{=\}100\(full table in Appendix[B](https://arxiv.org/html/2605.28873#A2)\); 25 of 32 cells lie within±1\.5\\pm 1\.5pp of the diagonal\. The largest positive residuals concentrate on MMLU \(subject\-mix variance from random partitions of a 57\-subject benchmark\) and on 3B\-tier ARC\-Easy\. We therefore caution against treating cross\-split SD as a measure of quantization\-specific noise: onn=100n\{=\}100subsamples it is largely the1/n1/\\sqrt\{n\}floor any binary score would exhibit\.

### 5\.2Quantization Deltas

![Refer to caption](https://arxiv.org/html/2605.28873v1/x2.png)Figure 3:Observed\|Δ^\|\|\\hat\{\\Delta\}\|per cell against the aggregate paired MDE budgets atm=k​n=500m\{=\}kn\{=\}500,α=0\.05\\alpha\{=\}0\.05, power0\.800\.80\(Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)\)\. Bars show the absolute NF4−\-FP16 accuracy delta from Table[2](https://arxiv.org/html/2605.28873#S5.T2); the signed delta is in parentheses\. Of the 16 cells, only OPT\-WinoGrande \(\|Δ^\|=3\.2\|\\hat\{\\Delta\}\|\{=\}3\.2pp\) crosses the tighterρd=0\.05\\rho\_\{d\}\{=\}0\.05MDE; all 16 fall below theρd=0\.10\\rho\_\{d\}\{=\}0\.10MDE\.Table[2](https://arxiv.org/html/2605.28873#S5.T2)reports NF4−\-FP16 accuracy deltas \(averaged over the aggregatem=500m\{=\}500items per cell\) and PPL deltas\. At the 3B tier the largest accuracy change is−3\.2\-3\.2pp \(OPT WinoGrande\); both signs occur within each model\. At the 7B tier all eight cells move by−0\.4\-0\.4to−0\.8\-0\.8pp, a tighter range with a consistent sign\. We flag two cautions about reading too much into the 7B pattern:

\(i\) With only two 7B models, this is a sample of size two at the model level; the apparent uniformity could reflect base accuracy \(p≈0\.6p\\approx 0\.6–0\.720\.72pushesσbin\\sigma\_\{\\text\{bin\}\}to≈2\.0\\approx 2\.0–2\.32\.3pp at the aggregate level\) or implementation idiosyncrasies as easily as a property of NF4\. A sign test treating the eight 7B cells as independent givesp=0\.0039p\{=\}0\.0039for the8/88/8negative\-direction outcome, but the four benchmarks within a model share weights and are not independent, so thisppis an optimistic lower bound on the strength of evidence\.

\(ii\) The aggregate MDE bound \(Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2),m=500m\{=\}500,α=0\.05\\alpha\{=\}0\.05, power1−β=0\.801\-\\beta\{=\}0\.80\) isδm=500∗≈4\.0\\delta^\{\*\}\_\{m=500\}\\approx 4\.0pp assumingρd=0\.10\\rho\_\{d\}\{=\}0\.10and tightens to≈2\.8\\approx 2\.8pp assumingρd=0\.05\\rho\_\{d\}\{=\}0\.05\. None of the eight 7B deltas crosses either threshold\. Among 3B cells, only OPT\-WinoGrande \(−3\.2\-3\.2pp\) approaches the bound: it is belowδ∗\\delta^\{\*\}atρd=0\.10\\rho\_\{d\}\{=\}0\.10but*above*δ∗\\delta^\{\*\}atρd=0\.05\\rho\_\{d\}\{=\}0\.05, so its detectability depends on the unmeasuredρd\\rho\_\{d\}\.This makes OPT\-WinoGrande the single tentative observation in our audit: it is the only cell whose “is this a real quantization effect?” answer flips with the planning value ofρd\\rho\_\{d\}, and the only cell that motivates measuringρd\\rho\_\{d\}rather than assuming it\. We report all deltas as observations on the audited subsample, not as tested differences; the bound is sufficient, not necessary, and assumes \(A1\)–\(A3\) of §[3](https://arxiv.org/html/2605.28873#S3)\.

PPL changes \(Table[2](https://arxiv.org/html/2605.28873#S5.T2)\) follow a different pattern\. WinoGrande shows the largest PPL increases at both scales \(up to\+9\.1\+9\.1for OPT,\+3\.1\+3\.1for Llama\-2\-7B\), likely reflecting that prompt\-text PPL on a binary\-cloze benchmark is more sensitive to weight perturbation than answer\-token correctness is\. We do not attempt to claim that PPL changes*cause*accuracy changes; the tables report descriptive co\-occurrence only\.

### 5\.3Prompt Sensitivity \(MMLU only\)

Table 3:MMLU accuracy across three prompt templates \(n=50n\{=\}50each\)\. Range is the maximum within\-row template spread on these subsamples\. Withn=50n\{=\}50, even a 10 pp range corresponds to only five extra correct answers; CIs and ranking\-reversal claims are exploratory\.Table[3](https://arxiv.org/html/2605.28873#S5.T3)shows the MMLU template study\. The OPT\-FP16 \(T​0=\.28T0\{=\}\.28\) vs OPT\-NF4 \(T​1=\.30T1\{=\}\.30\) comparison apparently reverses the best template, but atn=50n\{=\}50this corresponds to a one\-or\-two\-question reshuffling and the unpaired binomial CI on a 4\-pp difference atn=50n\{=\}50is roughly±18\\pm 18pp\. We therefore report template ranking apparent reversals at the 3B tier as exploratory observations rather than reliable interactions; the 7B tier preserves T0 as the best template under both precisions, but withn=50n\{=\}50even this can be coincidental\.

The substantive take\-away: within\-row MMLU template ranges of 2–10 pp are comparable to or larger than the largest observed quantization delta \(3\.2 pp\)\. A single\-template quantization audit therefore absorbs template variance into its precision\-comparison noise floor, masking effects of the size we are trying to measure\. Fixing the prompt template before any quantization audit is a prerequisite for a meaningful precision\-vs\-precision comparison at this scale\. We did not extend the prompt sweep to ARC, WinoGrande, or HellaSwag in this study, so this take\-away is conditional on the MMLU subsample\.

### 5\.4Quantization Reliability Index \(QRI\)

For each cellccwe pool the FP16 and NF4 split SDs into a single denominator by RMS,

σ^split​\(c\)=12​\[\(σ^splitFP16​\(c\)\)2\+\(σ^splitNF4​\(c\)\)2\],\\hat\{\\sigma\}\_\{\\text\{split\}\}\(c\)\\;=\\;\\sqrt\{\\tfrac\{1\}\{2\}\\\!\\left\[\(\\hat\{\\sigma\}^\{\\text\{FP16\}\}\_\{\\text\{split\}\}\(c\)\)^\{2\}\+\(\\hat\{\\sigma\}^\{\\text\{NF4\}\}\_\{\\text\{split\}\}\(c\)\)^\{2\}\\right\]\},\(3\)as a per\-cell single\-split noise scale \(not the unpaired\-difference SD, which would be2\\sqrt\{2\}larger\)\. We then define two cell\-level diagnostics:

QRIsplit​\(c\)\\displaystyle\\mathrm\{QRI\}\_\{\\text\{split\}\}\(c\)=\|Δ^quant​\(c\)\|σ^split​\(c\)​for all 16 cells,\\displaystyle=\\frac\{\|\\hat\{\\Delta\}\_\{\\text\{quant\}\}\(c\)\|\}\{\\hat\{\\sigma\}\_\{\\text\{split\}\}\(c\)\}\\;\\;\\text\{for all 16 cells,\}\(4\)QRIcombined​\(c\)\\displaystyle\\mathrm\{QRI\}\_\{\\text\{combined\}\}\(c\)=\|Δ^quant​\(c\)\|σ^split2​\(c\)\+σ^prompt2​\(c\)\\displaystyle=\\frac\{\|\\hat\{\\Delta\}\_\{\\text\{quant\}\}\(c\)\|\}\{\\sqrt\{\\hat\{\\sigma\}\_\{\\text\{split\}\}^\{2\}\(c\)\+\\hat\{\\sigma\}\_\{\\text\{prompt\}\}^\{2\}\(c\)\}\}for the 4 MMLU cells only\.\(5\)QRIcombined\\mathrm\{QRI\}\_\{\\text\{combined\}\}is undefined for the 12 non\-MMLU cells because we did not measure prompt variance there\. We deliberately do not collapse the 16 cell\-level QRIs into a single global number\.

#### QRI as a descriptive heuristic, not a test\.

QRI is a per\-cell normalization of\|Δ^\|\|\\hat\{\\Delta\}\|by an observed noise scale, useful as a quick visual flag for “loud” cells; power\-controlled decisions should still compare\|Δ^\|\|\\hat\{\\Delta\}\|directly toδ∗\\delta^\{\*\}in pp \(Figure[3](https://arxiv.org/html/2605.28873#S5.F3)\)\. A threshold in QRI units would be denominator\-dependent, so we quote QRI as a descriptive ratio only\.

#### Per\-cell results\.

With the pooled\-RMS denominator \(Appendix[C](https://arxiv.org/html/2605.28873#A3), Table[5](https://arxiv.org/html/2605.28873#A3.T5)\),QRIsplit\\mathrm\{QRI\}\_\{\\text\{split\}\}ranges from0\.090\.09\(Pythia MMLU\) to0\.760\.76\(OPT WinoGrande\), the latter also being the top cell by\|Δ^\|\|\\hat\{\\Delta\}\|; the largest 7B value is0\.250\.25\(Llama\-2 HellaSwag\)\. For the four MMLU cells,QRIcombined\\mathrm\{QRI\}\_\{\\text\{combined\}\}ranges from0\.070\.07\(Pythia\) to0\.180\.18\(OPT\)\.

#### Headline\.

Across the 16 cells \(Figure[3](https://arxiv.org/html/2605.28873#S5.F3)\), the 3B\-tier maximum\|Δ^\|=3\.2\|\\hat\{\\Delta\}\|\{=\}3\.2pp \(OPT\-WinoGrande\) falls below the implied MDE assumingρd=0\.10\\rho\_\{d\}\{=\}0\.10\(δ∗≈4\.0\\delta^\{\*\}\{\\approx\}4\.0pp\) but exceeds the implied MDE assumingρd=0\.05\\rho\_\{d\}\{=\}0\.05\(δ∗≈2\.8\\delta^\{\*\}\{\\approx\}2\.8pp\); whether this borderline is power\-distinguishable depends on the unmeasuredρd\\rho\_\{d\}\. The 7B\-tier maximum is0\.80\.8pp, well below either implied MDE\. The full per\-cell version is in Appendix[C](https://arxiv.org/html/2605.28873#A3), Table[6](https://arxiv.org/html/2605.28873#A3.T6)\.

## 6Discussion

#### What our audit can and cannot tell us\.

On the audited subsamples, FP16\-vs\-NF4 accuracy differences are small relative to both binomial sampling SD and the paired MDE\. We cannot separate “the population effect is small” from “n=100n\{=\}100is too small to detect it”; a CTB\-style ex\-ante guarantee would require either \(i\) aρd\\rho\_\{d\}\-aware sample\-size calculation done before the benchmark or \(ii\) a calibration\-set predictor of the populationδ\\deltafrom weight statistics\. We attempt only \(i\)\.

#### Cross\-split SD is a misleading noise proxy here\.

On 25 of 32 cells \(Figure[2](https://arxiv.org/html/2605.28873#S5.F2)\),σ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}tracks the binomial reference within±1\.5\\pm 1\.5pp, so the conventional “cross\-split SD exceeds the quantization effect, so the effect is unreliable” narrative is largely small\-sample binomial noise, not an NF4 property\. The largest residual,\+3\.0\+3\.0pp on OPT\-NF4 MMLU, is consistent with subject\-mix variance from random partitions of MMLU’s 57 subjects, not a quantization artifact\. We therefore recommend that anyσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}report be accompanied byp^​\(1−p^\)/n\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}for the same\(p^,n\)\(\\hat\{p\},n\)\.

#### Prompt template choice is a confound\.

Within\-row template ranges of 2–10 pp on MMLU are comparable to or larger than the largest observed quantization deltas\. Any quantization comparison on a single template absorbs this variance into the comparison\. We recommend reporting prompt\-template variance for every benchmark in any quantization claim, not only MMLU\.

#### Toward CTB\-style guarantees\.

Our MDE bound \(Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)\) is an ex\-ante guarantee only on*detectability*, not on*magnitude*: it answers “what effect could I detect with this benchmark?” but not “what effect should I expect for this model?” Closing the second half—a predictor ofρd\\rho\_\{d\}andδ\\deltafrom a calibration set, weight spectra, or activation outliers—is the natural next step and is precisely the theory↔\\leftrightarrowbenchmark bridge CTB asks for\.Chang et al\. \([2025](https://arxiv.org/html/2605.28873#bib.bib4)\)report empirical predictors of input\-level quantization breakdown, which is a complementary signal that future work could plug into our pre\-registration template as aρd\\rho\_\{d\}prior\. We leave a full predictor ofδ\\deltafrom model statistics to future work;Berg\-Kirkpatrick et al\. \([2012](https://arxiv.org/html/2605.28873#bib.bib2)\)also document the more general lesson that test\-set size, gain magnitude, and system similarity together govern paired NLP detectability\. Auditing methodology for LLM agents has begun to appear in adjacent areas \(e\.g\., agent safety\(Luo et al\.,[2025](https://arxiv.org/html/2605.28873#bib.bib38)\)\); a paired\-MDE budget plays the analogous role for quantization\-precision claims\.

## 7Recommendations

We recommend that quantization\-evaluation reports include:

1. 1\.Pre\-registered MDE\.State\(α,1−β,ρd,m\)\(\\alpha,1\-\\beta,\\rho\_\{d\},m\)and the resultingδ∗\\delta^\{\*\}*before*reporting accuracy deltas, wheremmis the paired item count for the reported estimand; ifkknon\-overlapping splits ofnnitems are used, state whetherm=nm\{=\}n\(single\-split\) orm=k​nm\{=\}kn\(aggregate\)\. Effects belowδ∗\\delta^\{\*\}should be reported as “not power\-distinguishable at this sample size,” not as “small\.”
2. 2\.Paired statistics\.Retain per\-example correctness so that paired McNemar/bootstrap estimators can be applied\. When FP16 and NF4 correctness are positively correlated \(the typical regime\), single\-split accuracy SDs overstate the relevant noise level for the paired delta; the amount depends on the unmeasured paired covariance\.
3. 3\.Binomial\-reference decomposition\.Reportσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}alongsideσbin​\(p^,n\)=p^​\(1−p^\)/n\\sigma\_\{\\text\{bin\}\}\(\\hat\{p\},n\)=\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}\. If the gap is small, label the residual “subset\-composition variance”; if it is large, treatσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}as a population\-noise estimate\.
4. 4\.Prompt variance on every benchmark\.Pre\-register a fixed\-cardinality template sweep \(T≥3T\{\\geq\}3from a transparent family, e\.g\.Sclar et al\. \([2024](https://arxiv.org/html/2605.28873#bib.bib26)\)format perturbations\), paired across templates and precisions; reportσ^prompt\\hat\{\\sigma\}\_\{\\text\{prompt\}\}as the SD across template means within precision andQRIcombined\\mathrm\{QRI\}\_\{\\text\{combined\}\}wherever the sweep is run, not only on MMLU\.
5. 5\.Multiple models, families, and methods\.Do not generalize fromn=1n\{=\}1orn=2n\{=\}2models per scale tier\. Cross\-method comparison \(NF4, GPTQ, AWQ, SpQR\) controls for method\-specific quirks\.

## 8Pre\-Registration Template

The recommendations of §[7](https://arxiv.org/html/2605.28873#S7)fit naturally into a five\-line pre\-registration\. Filling these out before evaluation converts an opaque “small\-effect” narrative into an auditable claim about what your benchmark could detect\. Italics indicate values to fill in\.

1. 1\.Estimand\.Single\-split \(m=nm\{=\}n\) or aggregate \(m=k​nm\{=\}kn\); statekkandnn\. Our pilot usesm=500m\{=\}500aggregate\.
2. 2\.Test parameters\.α\\alpha,1−β1\-\\beta\. Common defaults:0\.050\.05,0\.800\.80\.
3. 3\.Disagreement\-rate prior\.Conservative upper boundρdprior\\rho^\{\\text\{prior\}\}\_\{d\}, with justification \(calibration set, prior literature, or sensitivity range\)\.
4. 4\.Computed MDE\.δ∗​\(m,ρdprior,α,1−β\)\\delta^\{\*\}\(m,\\rho^\{\\text\{prior\}\}\_\{d\},\\alpha,1\-\\beta\)from Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2), in pp\.
5. 5\.Paired retention andρd\\rho\_\{d\}revision rule\.Commit to retaining per\-example correctness and reporting\(n10,n01,ρ^d\)\(n\_\{10\},n\_\{01\},\\hat\{\\rho\}\_\{d\}\)\. Becauseρ^d\\hat\{\\rho\}\_\{d\}is noisy, use the Wilson upper95%95\\%boundU95​\(ρd\)U\_\{95\}\(\\rho\_\{d\}\)of\(n10\+n01\)/m\(n\_\{10\}\+n\_\{01\}\)/m\. IfU95​\(ρd\)\>ρdpriorU\_\{95\}\(\\rho\_\{d\}\)\>\\rho^\{\\text\{prior\}\}\_\{d\}, mark a prior violation, recomputeδ∗\\delta^\{\*\}atρdeff=max⁡\(ρdprior,U95​\(ρd\)\)\\rho\_\{d\}^\{\\text\{eff\}\}=\\max\(\\rho^\{\\text\{prior\}\}\_\{d\},U\_\{95\}\(\\rho\_\{d\}\)\), and re\-evaluate borderline claims under the larger MDE; otherwise the pre\-registered MDE remains binding\.

This template is a CTB\-style ex\-ante guarantee on detectability: a reviewer can check, before any benchmark numbers, what claims the protocol could license\.

## 9Limitations

\(L1\) We did not retain per\-example correctness, so claims are framed in unpaired terms even though FP16/NF4 share items; Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)is evaluated against a paired\-disagreement*upper bound*rather than measuredρ^d\\hat\{\\rho\}\_\{d\}\. \(L2\) We test only NF4 via BitsAndBytes; GPTQ, AWQ, SmoothQuant, and SpQR may differ\. \(L3\) Our 7B tier has only two models; within\-7B claims are observational\. \(L4\) Prompt variance is measured only on MMLU atn=50n\{=\}50; coverage and per\-templatennare too small for firm interaction claims\. \(L5\) Our 3B and 7B subsamples are the only data we have; we do not claim population\-level conclusions about NF4\. \(L6\) Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)is a normal\-approximation*conservative sufficient*bound, not an impossibility result; small\-mmregimes warrant exact\-conditional or mid\-ppMcNemar\(Fagerland et al\.,[2013](https://arxiv.org/html/2605.28873#bib.bib11)\)\. \(L7\) Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)assumes \(A1\)–\(A3\) of §[3](https://arxiv.org/html/2605.28873#S3); subject clustering on MMLU breaks \(A1\) and we did not run a subject\-stratified bootstrap\.

## 10Conclusion

We reframe benchmark reliability under quantization as a minimum\-detectable\-effect problem and derive a paired MDE bound \(Eq\.[2](https://arxiv.org/html/2605.28873#S3.E2)\) that, given a planning value ofρd\\rho\_\{d\}, lets designers budget benchmark size against the smallest claimable quantization effect\. On the audited 4×\\times4 grid, three observations follow\. \(i\) Assumingρd=0\.10\\rho\_\{d\}\{=\}0\.10,n=100n\{=\}100per split cannot resolve sub\-percentage\-point effects\. \(ii\) Observed cross\-split SD is largely binomial sampling \(25/3225/32cells within±1\.5\\pm 1\.5pp of the binomial reference\)\. \(iii\) On MMLU, prompt\-template variance \(22–1010pp\) meets or exceeds the largest observed quantization delta \(3\.23\.2pp\), so a quantization audit must fix the prompt template before any precision comparison is meaningful\. The natural next step is to retain per\-example correctness soρd\\rho\_\{d\}can be measured rather than assumed, validating the bound on its own audit\.

## Impact Statement

This work argues that benchmark comparisons between quantized and full\-precision language models are routinely under\-powered\. By making the minimum detectable effect explicit and reframing single\-number quantization claims asnn\-dependent, we hope to encourage more rigorous statistical practice when quantized models are deployed and compared\.

## References

- Biderman et al\. \[2023\]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, pp\. 2397–2430, 2023\.
- Berg\-Kirkpatrick et al\. \[2012\]Taylor Berg\-Kirkpatrick, David Burkett, and Dan Klein\.An empirical investigation of statistical significance in NLP\.In*Proceedings of EMNLP\-CoNLL*, pp\. 995–1005, 2012\.
- Biderman et al\. \[2024\]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, et al\.Lessons from the trenches on reproducible evaluation of language models\.*arXiv preprint arXiv:2405\.14782*, 2024\.
- Chang et al\. \[2025\]Ting\-Yun Chang, Muru Zhang, Jesse Thomason, and Robin Jia\.Why do some inputs break low\-bit LLM quantization?In*Proceedings of EMNLP*, pp\. 3410–3429, 2025\.
- Connor \[1987\]Robert J\. Connor\.Sample size for testing differences in proportions for the paired\-sample design\.*Biometrics*, 43\(1\):207–211, 1987\.
- Clark et al\. \[2018\]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\.Think you have solved question answering? Try ARC, the AI2 reasoning challenge\.*arXiv preprint arXiv:1803\.05457*, 2018\.
- Dettmers et al\. \[2022\]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer\.GPT3\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Dettmers et al\. \[2023\]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.QLoRA: Efficient finetuning of quantized LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Dettmers et al\. \[2024\]Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh\.SpQR: A sparse\-quantized representation for near\-lossless LLM weight compression\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2024\.
- Dror et al\. \[2018\]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart\.The hitchhiker’s guide to testing statistical significance in natural language processing\.In*Proceedings of ACL*, pp\. 1383–1392, 2018\.
- Fagerland et al\. \[2013\]Morten W\. Fagerland, Stian Lydersen, and Petter Laake\.The McNemar test for binary matched\-pairs data: mid\-ppand asymptotic are better than exact conditional\.*BMC Medical Research Methodology*, 13\(1\):91, 2013\.
- Frantar et al\. \[2023\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.OPTQ: Accurate quantization for generative pre\-trained transformers\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2023\.
- Gong et al\. \[2024\]Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Dacheng Tao, and Xianglong Liu\.LLMC: Benchmarking large language model quantization with a versatile compression toolkit\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, pp\. 132–152, 2024\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2021\.
- Jiang et al\. \[2023\]Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, et al\.Mistral 7B\.*arXiv preprint arXiv:2310\.06825*, 2023\.
- Jin et al\. \[2024\]Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong\.A comprehensive evaluation of quantization strategies for large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, 2024\.
- Lachin \[1992\]John M\. Lachin\.Power and sample size evaluation for the McNemar test with application to matched case\-control studies\.*Statistics in Medicine*, 11\(9\):1239–1251, 1992\.
- Liang et al\. \[2023\]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, et al\.Holistic evaluation of language models\.*Transactions on Machine Learning Research*, 2023\.
- Li et al\. \[2024\]Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang\.Evaluating quantized large language models\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, volume 235 of*PMLR*, pp\. 28480–28524, 2024\.
- Lin et al\. \[2024\]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.
- Lu et al\. \[2024\]Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W\. Mahoney, and Yaoqing Yang\.AlphaPruning: Using heavy\-tailed self regularization theory for improved layer\-wise pruning of large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Miettinen \[1968\]Olli S\. Miettinen\.The matched pairs design in the case of all\-or\-none responses\.*Biometrics*, 24\(2\):339–352, 1968\.
- Mizrahi et al\. \[2024\]Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky\.State of what art? a call for multi\-prompt LLM evaluation\.*Transactions of the Association for Computational Linguistics*, 12:933–949, 2024\.
- Polo et al\. \[2024\]Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin\.tinyBenchmarks: Evaluating LLMs with fewer examples\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, volume 235 of*PMLR*, pp\. 34303–34326, 2024\.
- Sakaguchi et al\. \[2020\]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi\.WinoGrande: An adversarial Winograd schema challenge at scale\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, 2020\.
- Sclar et al\. \[2024\]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\.Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2024\.
- Touvron et al\. \[2023\]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al\.Llama 2: Open foundation and fine\-tuned chat models\.*arXiv preprint arXiv:2307\.09288*, 2023\.
- Xiao et al\. \[2023\]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.SmoothQuant: Accurate and efficient post\-training quantization for large language models\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, 2023\.
- Yao et al\. \[2022\]Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He\.ZeroQuant: Efficient and affordable post\-training quantization for large\-scale transformers\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Yeh \[2000\]Alexander Yeh\.More accurate tests for the statistical significance of result differences\.In*Proceedings of the 18th International Conference on Computational Linguistics \(COLING\), Volume 2*, pp\. 947–953, 2000\.
- Zellers et al\. \[2019\]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi\.HellaSwag: Can a machine really finish your sentence?In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 4791–4800, 2019\.
- Zhang et al\. \[2022\]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, et al\.OPT: Open pre\-trained transformer language models\.*arXiv preprint arXiv:2205\.01068*, 2022\.
- Wilson \[1927\]Edwin B\. Wilson\.Probable inference, the law of succession, and statistical inference\.*Journal of the American Statistical Association*, 22\(158\):209–212, 1927\.
- Zhao et al\. \[2025\]Jiaqi Zhao, Ming Wang, Miao Zhang, Yuzhang Shang, Xuebo Liu, Yaowei Wang, Min Zhang, and Liqiang Nie\.Benchmarking post\-training quantization in LLMs: Comprehensive taxonomy, unified evaluation, and comparative analysis\.*arXiv preprint arXiv:2502\.13178*, 2025\.
- Ji et al\. \[2026\]Yuelyu Ji, Zhuochun Li, Rui Meng, and Daqing He\.Retrieval–reasoning processes for multi\-hop question answering: A four\-axis design framework and empirical trends\.*arXiv preprint arXiv:2601\.00536*, 2026\.
- Luo et al\. \[2026a\]Hanjun Luo et al\.BiasIG: Benchmarking multi\-dimensional social biases in text\-to\-image models\.In*Proceedings of the 2026 International Joint Conference on Neural Networks \(IJCNN\)*, 2026\.
- Luo et al\. \[2026b\]Hanjun Luo et al\.AtelierEval: Agentic evaluation of humans & LLMs as text\-to\-image prompters\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.
- Luo et al\. \[2025\]Hanjun Luo et al\.AgentAuditor: Human\-level safety and security evaluation for LLM agents\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.
- Shi et al\. \[2024\]Jingzhe Shi, Qinwei Ma, Huan Ma, and Lei Li\.Scaling law for time series forecasting\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Shi et al\. \[2026\]Jingzhe Shi et al\.Intrinsic Entropy of Context Length Scaling in LLMs\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2026\.

## Appendix AReproducibility

#### Hugging Face identifiers and revisions\.

facebook/opt\-2\.7b\(revmain, commit2bb5d4b\),EleutherAI/pythia\-2\.8b\(revmain, commit0bcca15\),meta\-llama/Llama\-2\-7b\-hf\(revmain, commit8cca527\),mistralai/Mistral\-7B\-v0\.1\(revmain, commit7231864\)\. All checkpoints are base \(non\-instruct\) variants\. The exact commit hashes above were the latest at evaluation time; users replicating this study should pin to these revisions or report any deviation\.

#### NF4 configuration \(BitsAndBytes\)\.

BitsAndBytesConfig\(load\_in\_4bit=True, bnb\_4bit\_quant\_type=‘nf4’, bnb\_4bit\_compute\_dtype=torch\.float16, bnb\_4bit\_use\_double\_quant=False\)\. FP16 inference usestorch\_dtype=torch\.float16\.

#### Library versions\.

transformers==4\.44\.0,bitsandbytes==0\.43\.1,datasets≥\\geq2\.14,torch≥\\geq2\.0,accelerate≥\\geq0\.25\.

#### Datasets\.

MMLU \(cais/mmlu,all, validation\), ARC\-Easy \(allenai/ai2\_arc,ARC\-Easy, test\), WinoGrande \(winogrande,winogrande\_xl, validation\), HellaSwag \(Rowan/hellaswag, validation\)\.

#### Splits\.

Each benchmark is randomly partitioned with NumPy seed 42 into five non\-overlapping splits of 100 items\. The 100\-item split indices for each benchmark are released with the submission’s anonymous data bundle\.

#### Prompt templates \(MMLU prompt\-sensitivity sweep\)\.

T0: “Question: \{q\}\\n\{choices\_formatted\}\\nAnswer:”\. T1: “Q: \{q\}\\nOptions: A\. \{a\} B\. \{b\} C\. \{c\} D\. \{d\}\\nA:”\. T2: “\{q\}\\n\(A\) \{a\} \(B\) \{b\} \(C\) \{c\} \(D\) \{d\}\\nThe answer is”\. The default benchmark sweep \(Tables[1](https://arxiv.org/html/2605.28873#S5.T1)–[2](https://arxiv.org/html/2605.28873#S5.T2)\) uses T0\.

#### Scoring\.

For multiple\-choice items we compute the per\-token log\-likelihood of each candidate continuation under the fixed prompt template and select the argmax\. For WinoGrande we compute the joint likelihood ratio of the two pronoun\-resolved continuations\. “Prompt PPL” isexp⁡\(ℓ¯\)\\exp\(\\bar\{\\ell\}\)whereℓ¯\\bar\{\\ell\}is the per\-token NLL over the prompt text only \(no answer tokens\), averaged across examples within a split\.

## Appendix BPer\-Cell Binomial Reference SD

Table[4](https://arxiv.org/html/2605.28873#A2.T4)reports, for each of the 32 \(model×\\timesbenchmark×\\timesprecision\) cells, the observed split meanp^\\hat\{p\}, the observed cross\-split SDσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}\(overk=5k\{=\}5splits ofn=100n\{=\}100\), and the binomial reference SDσbin​\(p^\)=p^​\(1−p^\)/n\\sigma\_\{\\text\{bin\}\}\(\\hat\{p\}\)=\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}\. The residualr=σ^split−σbin​\(p^\)r=\\hat\{\\sigma\}\_\{\\text\{split\}\}\-\\sigma\_\{\\text\{bin\}\}\(\\hat\{p\}\)measures the cross\-split SD beyond what binomial sampling alone predicts\. Of the 32 cells, 25 have\|r\|≤1\.5\|r\|\\leq 1\.5pp, 29 have\|r\|≤2\.0\|r\|\\leq 2\.0pp, and the four largest positive residuals are\+3\.0,\+1\.5,\+1\.1,\+1\.0\+3\.0,\+1\.5,\+1\.1,\+1\.0pp \(OPT\-NF4 MMLU, OPT\-NF4 ARC\-Easy, Pythia\-FP16 ARC\-Easy, OPT\-FP16 MMLU\)\. All four are at low base accuracies \(p^∈\[0\.24,0\.27\]\\hat\{p\}\\in\[0\.24,0\.27\]\); the subject\-mix\-variance explanation applies most directly to the two MMLU cells\. The three cells with\|r\|\>2\.0\|r\|\>2\.0pp are OPT\-NF4 MMLU \(\+3\.0\+3\.0\), Pythia\-NF4 ARC\-Easy \(−2\.5\-2\.5\), and Pythia\-FP16 WinoGrande \(−2\.5\-2\.5\); the two negative excursions reflect the wide chi\-square sampling distribution ofσ^\\hat\{\\sigma\}atk=5k\{=\}5and not anything specific to NF4\.

Table 4:Per\-cell observed accuracy \(p^\\hat\{p\}\), cross\-split SD \(σ^split\\hat\{\\sigma\}\_\{\\text\{split\}\},k=5k\{=\}5splits ofn=100n\{=\}100\), binomial reference SDσbin​\(p^\)=p^​\(1−p^\)/n\\sigma\_\{\\text\{bin\}\}\(\\hat\{p\}\)=\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}, and residualr=σ^split−σbinr=\\hat\{\\sigma\}\_\{\\text\{split\}\}\-\\sigma\_\{\\text\{bin\}\}\. All quantities in pp\.Negative residuals \(σ^split<σbin\\hat\{\\sigma\}\_\{\\text\{split\}\}<\\sigma\_\{\\text\{bin\}\}\) are visible in 24 of 32 cells\. Withk=5k\{=\}5splits, the sampling distribution ofσ^/σ\\hat\{\\sigma\}/\\sigmais wide: under a chi\-square model withk−1=4k\-1\{=\}4degrees of freedom,σ^/σ\\hat\{\\sigma\}/\\sigmahas approximate95%95\\%sampling range\[0\.35,1\.67\]\[0\.35,1\.67\], so observedσ^\\hat\{\\sigma\}can fall well below the populationσ\\sigmapurely by sampling noise\. Most negative residuals are within this range; we do not interpret them as evidence that the protocol is sub\-binomial\. We use “binomial reference SD” rather than “binomial floor” in this paper to acknowledge that observed cross\-split SD is not bounded below byp​\(1−p\)/n\\sqrt\{p\(1\-p\)/n\}at finitekk\.

## Appendix CPer\-Cell QRI Diagnostics

Table[5](https://arxiv.org/html/2605.28873#A3.T5)reports, for each of the 16 \(model×\\timesbenchmark\) cells, the absolute accuracy delta\|Δ^\|\|\\hat\{\\Delta\}\|, the RMS\-pooled split SDσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}, andQRIsplit\\mathrm\{QRI\}\_\{\\text\{split\}\}\. For the four MMLU cells the table also reports the prompt SD \(RMS\-pooled across precisions\) andQRIcombined\\mathrm\{QRI\}\_\{\\text\{combined\}\}\.

Table 5:Per\-cell QRI\.σ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}is RMS\-pooled across precisions\.σ^prompt\\hat\{\\sigma\}\_\{\\text\{prompt\}\}is the RMS\-pool of FP16 and NF4 prompt\-template SDs from Table[3](https://arxiv.org/html/2605.28873#S5.T3), defined for the 4 MMLU cells only\. All SDs in pp\.Per\-cell direct comparison of observed\|Δ^\|\|\\hat\{\\Delta\}\|to the aggregate paired MDEδm=500∗\\delta^\{\*\}\_\{m=500\}at two illustrativeρd\\rho\_\{d\}values is reported in Table[6](https://arxiv.org/html/2605.28873#A3.T6)\. The MDE depends only on\(m,ρd,α,β\)\(m,\\rho\_\{d\},\\alpha,\\beta\), so it is identical across cells at fixedρd\\rho\_\{d\}; we still print it per row to make the comparison eye\-trackable\. Only one cell, OPT\-WinoGrande \(\|Δ^\|=3\.2\|\\hat\{\\Delta\}\|\{=\}3\.2pp\), exceeds theρd=0\.05\\rho\_\{d\}\{=\}0\.05MDE \(2\.82\.8pp\)\. Atρd=0\.10\\rho\_\{d\}\{=\}0\.10\(δm=500∗≈4\.0\\delta^\{\*\}\_\{m=500\}\{\\approx\}4\.0pp\), no cell crosses the bound\. We do*not*convert this comparison into a QRI threshold because converting it throughσ^split\\hat\{\\sigma\}\_\{\\text\{split\}\}requires choosing whether to use the observed RMS\-pooled split SD or the binomial referencep^​\(1−p^\)/n\\sqrt\{\\hat\{p\}\(1\-\\hat\{p\}\)/n\}, and the two choices disagree on most cells; the\|Δ^\|\|\\hat\{\\Delta\}\|\-vs\-δ∗\\delta^\{\*\}comparison is the cleaner statistic\.

Table 6:Per\-cell observed\|Δ^quant\|\|\\hat\{\\Delta\}\_\{\\text\{quant\}\}\|vs the aggregate paired MDEδm=500∗=\(z1−α/2\+z1−β\)​ρd/m\\delta^\{\*\}\_\{m=500\}=\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\sqrt\{\\rho\_\{d\}/m\}atα=0\.05\\alpha\{=\}0\.05, power1−β=0\.801\-\\beta\{=\}0\.80, for two illustrativeρd\\rho\_\{d\}values\. “Exceeds MDE?” is yes when\|Δ^\|\>δm=500∗\|\\hat\{\\Delta\}\|\>\\delta^\{\*\}\_\{m=500\}\.This is not a significance test:it asks whether the observed aggregate delta is larger than the design’s planned detectable\-effect scale under the statedρd\\rho\_\{d\}\.
## Appendix DWilson Confidence Intervals

Table[7](https://arxiv.org/html/2605.28873#A4.T7)reports Wilson 95% CIs\[Wilson,[1927](https://arxiv.org/html/2605.28873#bib.bib33)\]for the FP16 accuracy in each of the 16 model×\\timesbenchmark cells, computed atn=500n\{=\}500\(the union of the five splits\)\. Atn=500n\{=\}500andp^∈\[0\.25,0\.72\]\\hat\{p\}\\in\[0\.25,0\.72\], Wilson half\-widths range from±3\.7\\pm 3\.7to±4\.4\\pm 4\.4pp, comparable in magnitude to the largest observed NF4−\-FP16 deltas in Table[2](https://arxiv.org/html/2605.28873#S5.T2)\. The half\-width*per split*\(n=100n\{=\}100\) is roughly±8\\pm 8–1010pp, so split\-level deltas are even less resolvable than the union\-level deltas\.Caveat:these are single\-proportion Wilson intervals on the FP16 accuracy alone, not paired CIs on the NF4−\-FP16 delta\. The paired CI is the relevant uncertainty estimator for the delta and would generally be tighter than±\\pm\(union of unpaired half\-widths\), since FP16 and NF4 correctness are positively correlated; the paired CI is computable only from per\-example records that we did not retain\.

Table 7:Wilson 95% CIs for FP16 accuracy on the union of five splits \(n=500n\{=\}500per cell\)\. Half\-widths bound the per\-cell estimation error and contextualize theΔ\\DeltaAcc values in Table[2](https://arxiv.org/html/2605.28873#S5.T2)\.
## Appendix EPower Curves for the Paired MDE

Forα=0\.05\\alpha\{=\}0\.05and power1−β=0\.801\-\\beta\{=\}0\.80the paired MDE simplifies toδ∗≤2\.80​ρd/m\\delta^\{\*\}\\leq 2\.80\\sqrt\{\\rho\_\{d\}/m\}\. Sample sizes required for several\(δ,ρd\)\(\\delta,\\rho\_\{d\}\)pairs \(usingmmfor the paired item count\):

Table 8:Paired sample sizesmmrequired to resolve a paired NF4 effect of magnitudeδ\\deltaatα=0\.05\\alpha\{=\}0\.05, power1−β=0\.801\-\\beta\{=\}0\.80\.Treatingρd∈\[0\.05,0\.20\]\\rho\_\{d\}\\in\[0\.05,0\.20\]as an illustrative planning range \(not an empirical claim from this audit\), a study that wants to make a 1\-pp paired claim about NF4 effects at 80% power needs roughly4×1034\\times 10^\{3\}–1\.6×1041\.6\\times 10^\{4\}paired items per cell\. A study atm=100m\{=\}100is at least an order of magnitude underpowered for sub\-percentage\-point effects under those planning values\.