Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring

# Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring
Source: [https://arxiv.org/html/2607.01240](https://arxiv.org/html/2607.01240)
###### Abstract

Count\-based F1 is widely used as a proxy for LLM error\-detection quality, but we show that it can rise dramatically without any matching improvement in span localization—a gap we call*F1 Inflation*\. We present ErrorBench as a controlled stress\-test protocol for prompt\-induced count distortion\. ErrorBench covers six contemporary LLMs, five prompt conditions, and 4,290 responses over 143 CoNLL\-2014 passages\. Under CoNLL\-2014 M2\-style scoring \(overlap matching on hypothesis edits extracted from model descriptions\), anchored prompts drive up to0\.790\.79points of F1 Inflation, and up to0\.960\.96under strict matching\. A100100\-passage replication with the official ERRANT 3\.0\.0 pipeline and multi\-reference scoring reproduces the pattern: averaged over six models, the Blind→\\toAnchored move lifts Count\-F1 by\+0\.21\+0\.21while lifting multi\-reference ERRANTF0\.5F\_\{0\.5\}by only\+0\.04\+0\.04\. At the model\-family level, the count response is larger in highly instruction\-compliant GPT/Claude systems and smaller in the Gemini family, a descriptive pattern under the stress\-test protocol\. These findings suggest that practical LLM proofreading and document\-review systems should avoid pre\-populated error counts and should report span\-aware metrics alongside count\-based metrics\. We therefore treat count\-only evaluation as unreliable whenever prompts carry numeric expectations, and position ErrorBench as a stress\-test protocol for prompt\-induced evaluation distortion rather than an overall model\-quality comparison\.

Prompt Framing Distorts Count\-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring

Yangde KunZhejiang Universitypauliyangwork@gmail\.com

## 1Introduction

Large language models \(LLMs\) are increasingly deployed as automated proofreaders, code reviewers, and factual checkers, and their evaluation in these roles is often reduced to a single scalar: does the count of errors the model reports roughly match the count in the reference? When that count matches, the system is easily credited with an F1 near1\.01\.0even if its description of*which*errors occur is poor\. This paper is about a concrete consequence of that shortcut, which we callF1 Inflation: the wedge between count\-level agreement and span\-level agreement on the same outputs\. We show that this wedge can be driven wide by a single sentence in the prompt, and that it is large enough to change apparent conclusions drawn from count\-only evaluation\. The intended takeaway is not that one model family is better than another, but that a protocol which exposes expected counts can make count\-based metrics unreliable\.

To surface the wedge in a controlled way we need a mechanism that moves counts without changing the underlying text\. Numeric anchoring supplies exactly that mechanism\. A substantial body of cognitive psychology shows that humans adjust insufficiently away from a salient initial reference value\(Tversky and Kahneman,[1974](https://arxiv.org/html/2607.01240#bib.bib16); Epley and Gilovich,[2006](https://arxiv.org/html/2607.01240#bib.bib21)\), and recent work suggests LLMs exhibit similar behavior on general reasoning tasks\(Macmillan\-Scott and Musolesi,[2024](https://arxiv.org/html/2607.01240#bib.bib22)\)\. In realistic deployments the anchor is rarely arbitrary: an AI document\-review system may preface its query with “this contract has been flagged as containing approximately55errors,” a sycophancy\-adjacent signal\(Perezet al\.,[2022](https://arxiv.org/html/2607.01240#bib.bib14); Sharmaet al\.,[2024](https://arxiv.org/html/2607.01240#bib.bib15)\)that we can inject in a fully controlled way\. By pairing each passage with prompts that supply the true count, an inflated count, and a deflated count, we get a clean stress test for evaluation reliability: anything that moves Count\-F1 but leaves Span\-F1 unchanged is a direct observation of F1 Inflation\.

Our research questions are therefore inseparable from the evaluation critique they enable\. Does an anchored count inflate Count\-F1? Does that inflation survive a span\-aware re\-scoring? And which model\-family behaviors are most sensitive under this protocol? We make four contributions\.\(1\)We formalize and quantify F1 Inflation on a public benchmark, reporting gaps up to0\.790\.79under M2 overlap and0\.960\.96under strict M2 across 4,290 responses\.\(2\)We validate that the inflation is not an artifact of description\-based edit extraction: a full ERRANT 3\.0\.0 replication on a100100\-passage corrected\-text subset spanning*all six models*and multi\-reference \(Annotators 0\+\+1\) scoring reproduces the pattern \(mean Blind→\\toAnchored: Count\-F1\+0\.21\+0\.21, multi\-reference ERRANTF0\.5F\_\{0\.5\}\+0\.04\+0\.04; paired bootstrap showsΔ​Count−Δ​ERRANT\>0\\Delta\\mathrm\{Count\}\-\\Delta\\mathrm\{ERRANT\}\>0at95%95\\%CI for5/65/6models\)\.\(3\)We introduce ErrorBench as a stress\-test protocol: a 143\-passage CoNLL\-2014 testbed with five standardized prompt conditions, plus Count Bias \(CB\) and Anchoring Sensitivity Index \(ASI\) diagnostics for prompt\-induced count distortion\. Its role is analogous to a robustness audit for evaluation design, not a model\-quality comparison\.\(4\)A multi\-model, multi\-generational comparison provides model\-family descriptive evidence that strong instruction following can coincide with larger anchoring responses and therefore with larger F1 Inflation in this setup\. ErrorBench is not intended to compare contemporary models by overall quality; it is intended to diagnose whether an evaluation protocol is vulnerable to prompt\-induced count distortion\.

## 2Related Work

### 2\.1LLM Evaluation and Error Detection

A growing body of work evaluates LLMs on structured proofreading and fact\-checking tasks\.Fanget al\.\([2023](https://arxiv.org/html/2607.01240#bib.bib8)\)find that ChatGPT achieves competitive performance on grammatical error correction benchmarks while exhibiting systematic biases toward over\-correction\. Building on BEA\-2019\(Bryantet al\.,[2019](https://arxiv.org/html/2607.01240#bib.bib2)\)and CoNLL\-2014\(Nget al\.,[2014](https://arxiv.org/html/2607.01240#bib.bib1)\), recent studies demonstrate that LLMs can match or exceed dedicated GEC systems in low\-resource settings but remain sensitive to prompt phrasing\(Bryantet al\.,[2023](https://arxiv.org/html/2607.01240#bib.bib3)\)\. In the scientific domain, LLM\-assisted reviewing has attracted attention as a mechanism for targeted review support and peer\-review assistance\(Liu and Shah,[2023](https://arxiv.org/html/2607.01240#bib.bib7); Checcoet al\.,[2021](https://arxiv.org/html/2607.01240#bib.bib9)\)\.Liu and Shah \([2023](https://arxiv.org/html/2607.01240#bib.bib7)\)report pilot experiments on short scientific papers with deliberately inserted errors and find that LLMs perform worse when asked to review holistically but improve when explicitly instructed to find errors\. Complementarily, resources such as NLPeer provide broader infrastructure for the computational study of peer review and reviewing assistance\(Dyckeet al\.,[2023](https://arxiv.org/html/2607.01240#bib.bib10)\)\.

This finding directly motivates our work: if the binary error/no\-error framing already modulates behavior, then numerical anchors—which encode a specific expected count—may further distort model outputs in ways not yet studied\.

### 2\.2Prompt Sensitivity and Sycophancy

It is well established that LLM outputs are sensitive to prompt phrasing, instruction ordering, and demonstration selection\(Zhaoet al\.,[2021](https://arxiv.org/html/2607.01240#bib.bib11); Luet al\.,[2022](https://arxiv.org/html/2607.01240#bib.bib12); Minet al\.,[2022](https://arxiv.org/html/2607.01240#bib.bib13)\)\. A particularly relevant phenomenon is sycophancy: the tendency of LLMs to agree with user\-stated beliefs even when those beliefs are factually incorrect\(Perezet al\.,[2022](https://arxiv.org/html/2607.01240#bib.bib14); Sharmaet al\.,[2024](https://arxiv.org/html/2607.01240#bib.bib15)\)\.Perezet al\.\([2022](https://arxiv.org/html/2607.01240#bib.bib14)\)show that models frequently validate false user claims across a range of knowledge domains, suggesting that stated information in a prompt can override model knowledge\. Similarly, studies of LLM\-as\-judge evaluation reveal that positional and social biases systematically distort model\-generated assessments\(Yeet al\.,[2024](https://arxiv.org/html/2607.01240#bib.bib23)\), further motivating the need for anchor\-free evaluation protocols\. Our work differs in that we study numerical rather than factual anchoring, and focus specifically on count estimation in the error detection task\.

### 2\.3Anchoring Effects in LLMs

The anchoring effect—first described byTversky and Kahneman \([1974](https://arxiv.org/html/2607.01240#bib.bib16)\)—is one of the most robust findings in cognitive psychology\. The underlying mechanism, anchoring\-and\-adjustment\(Epley and Gilovich,[2006](https://arxiv.org/html/2607.01240#bib.bib21)\), posits that people start from a salient reference value and adjust toward their true estimate, but typically stop adjusting too early\. When people make numerical estimates, they are disproportionately influenced by this initial reference value\. Recent work has begun to investigate whether LLMs exhibit similar phenomena\.Macmillan\-Scott and Musolesi \([2024](https://arxiv.org/html/2607.01240#bib.bib22)\)provide a broad survey of cognitive biases in LLMs and find that anchoring\-like behavior arises across diverse reasoning tasks\. The Einstellung effect\(Luchins,[1942](https://arxiv.org/html/2607.01240#bib.bib17)\)—where prior experience with a solution strategy prevents consideration of alternatives—also has analogs in LLM reasoning\. We are not aware of prior work that directly examines numerical anchoring in error detection tasks, where the anchor is the stated count of errors in a passage\.

## 3Methodology

### 3\.1Dataset Construction

We build ErrorBench from the CoNLL\-2014 Shared Task dataset\(Nget al\.,[2014](https://arxiv.org/html/2607.01240#bib.bib1)\), which contains learner English essays annotated with grammatical errors in M2 format\. We group consecutive sentences into passages of four sentences each, taking care not to cross document boundaries, then compute passage\-level error counts from the M2 annotations\. For the main analysis, passage counts and gold edits are derived from Annotator\-0 only, because passage construction requires a single consistent reference across all four\-sentence windows\. We retain passages with 3–7 errors \(inclusive\) to ensure non\-trivial detection tasks while maintaining manageable complexity, sampling 30 passages per error\-count bucket \(3, 4, 5, 6, 7\) for a total of 143 passages \(the error\-count 7 bucket yielded 23 qualifying passages rather than 30\)\.

Each passage stores raw text, the Annotator\-0 error count, M2 error categories, and per\-error gold annotations \(token span, source text, correction, and sentence context\) for downstream matching\. These M2 labels are used descriptively on the full benchmark; a100100\-passage subset is additionally re\-scored with the official ERRANT 3\.0\.0 pipeline and with Annotator\-0\+\+Annotator\-1 multi\-reference max\-match scoring in Appendix[H](https://arxiv.org/html/2607.01240#A8)\.

### 3\.2Prompt Conditions

We design five prompt conditions as our primary independent variable, shown in Table[1](https://arxiv.org/html/2607.01240#S3.T1)\. All conditions share a fixed system prompt instructing the model to respond using a structured format \(“ERROR N: \[description\]” followed by “TOTAL ERRORS FOUND: N”\) to facilitate automated parsing\. Temperature is fixed at 0 across all experiments for determinism\.

Table 1:Five prompt conditions used in ErrorBench, varying the prior information provided about error count\.For the Misleading conditions, we setM=N\+2M=N\+2\(Mislead\-Over\) andM=max⁡\(1,N−2\)M=\\max\(1,N\-2\)\(Mislead\-Under\), whereNNis the true error count\. This ensures the anchor deviates from truth by exactly 2 in both directions, creating a symmetric experimental design\.

### 3\.3Models

We evaluate six contemporary LLMs spanning three organizations and two generational tiers\. This selection is designed to provide descriptive coverage of model families and instruction\-following regimes, not to establish an overall ordering of proprietary systems: GPT\-4o and GPT\-5\.4\(OpenAI,[2024](https://arxiv.org/html/2607.01240#bib.bib18),[2026](https://arxiv.org/html/2607.01240#bib.bib19)\), Claude Haiku 4\.5 and Claude Sonnet 4\.6\(Anthropic,[2025](https://arxiv.org/html/2607.01240#bib.bib5),[2026](https://arxiv.org/html/2607.01240#bib.bib6)\)\(abbreviated as Claude H\.4\.5 and Claude S\.4\.6 in tables and figures\), and Gemini 2\.5 Flash and Gemini 3\.1 Pro Preview\(Google DeepMind,[2025](https://arxiv.org/html/2607.01240#bib.bib20)\)\. All models use temperature=0=0andmax\_tokens=800=800\. We perform a single inference per sample\-condition\-model combination via an OpenAI\-compatible API proxy, yielding143×6×5=4,290143\\times 6\\times 5=4\{,\}290total API calls \(4,290 valid\)\.

### 3\.4Evaluation Metrics

ErrorBench is a controlled stress\-test protocol, so metrics are interpreted as diagnostics of prompt\-induced distortion under controlled prompt perturbations\. We report four metrics\. Count Bias \(CB\) is the signed difference between the model’s reported error count and the true error count:CB=reported​\_​N−true​\_​N\\mathrm\{CB\}=\\mathrm\{reported\\\_N\}\-\\mathrm\{true\\\_N\}\. Anchoring Sensitivity Index \(ASI\) is\|CBcondition−CBBlind\|/true​\_​N\|\\mathrm\{CB\}\_\{\\mathrm\{condition\}\}\-\\mathrm\{CB\}\_\{\\mathrm\{Blind\}\}\|/\\mathrm\{true\\\_N\}\. Count\-based Approximate F1 is derived from count overlap without span matching, usingTP=min⁡\(n^,n\)\\mathrm\{TP\}=\\min\(\\hat\{n\},n\),FP=max⁡\(0,n^−n\)\\mathrm\{FP\}=\\max\(0,\\hat\{n\}\-n\), andFN=max⁡\(0,n−n^\)\\mathrm\{FN\}=\\max\(0,n\-\\hat\{n\}\)\.

For span\-aware scoring we adopt the CoNLL\-2014 M2 scorer protocol, with one adaptation: because our prompts elicit natural\-language error descriptions rather than fully corrected sentences, we extract hypothesis edits\(sent​\_​idx,start,end,corr\)\(\\mathrm\{sent\\\_idx\},\\mathrm\{start\},\\mathrm\{end\},\\mathrm\{corr\}\)deterministically from each description by parsing quoted"src"/"src"→\\to"corr"fragments and locating them in the passage via token\-level substring match\. Descriptions that cannot be localized are counted as false positives\. We compute three corpus\-level microF0\.5F\_\{0\.5\}variants:strictmatches on\(sent​\_​idx,start,end,corr\)\(\\mathrm\{sent\\\_idx\},\\mathrm\{start\},\\mathrm\{end\},\\mathrm\{corr\}\),detectionmatches on\(sent​\_​idx,start,end\)\(\\mathrm\{sent\\\_idx\},\\mathrm\{start\},\\mathrm\{end\}\)only, andoverlapmatches on samesent​\_​idx\\mathrm\{sent\\\_idx\}with non\-empty token overlap \(standard lenience for boundary disagreement in GEC\)\. OverlapF0\.5F\_\{0\.5\}is our primary span\-aware metric; strict and detection are reported in Appendix[F](https://arxiv.org/html/2607.01240#A6)\. For continuity with earlier drafts we also retain a simpler heuristic text\-match Span\-F1 as a diagnostic secondary proxy\. F1 Inflation is defined asΔ​F=Count​\-​F1−M2​F0\.5overlap\\Delta F=\\mathrm\{Count\\mbox\{\-\}F1\}\-\\mathrm\{M2\}~F\_\{0\.5\}^\{\\mathrm\{overlap\}\}\.

For reproducibility, the accompanying source package includes data\-preparation, experiment, and analysis scripts together with raw JSONL outputs in theerrorbench/directory\. All figures were generated from theerrorbench/scripts/make\_figures\.pyscript using the passage\-level records inerrorbench/outputs/results\.jsonl\.

## 4Experiments and Results

### 4\.1Main Results

Table[2](https://arxiv.org/html/2607.01240#S4.T2)presents Count Bias and Count\-F1 across all six models and five conditions\. We treat these as descriptive model\-family patterns under a controlled stress test, not as evidence about general model quality\. Three findings stand out\. First, GPT\-5\.4 follows the Mislead\-Over anchor with zero variance \(CB=\+2\.000=\+2\.000, SD=0=0\) in every passage; Claude H\.4\.5 achieves the tightest Anchored compliance \(CB≈0\\approx 0, SD=0\.17=0\.17\) among all models\. Second, GPT\-5\.4 shows large unconstrained over\-reporting \(Blind CB=\+7\.3=\+7\.3\), Claude S\.4\.6 shows a smaller but still positive Blind bias \(\+2\.8\+2\.8\), and Claude H\.4\.5 is nearly unbiased under Blind \(\+0\.2\+0\.2\)\. Third, both Gemini models exhibit a stable undercount bias: Gemini 2\.5 \(CB≈−2\.1\\approx\-2\.1to−2\.6\-2\.6\) and Gemini 3\.1 \(CB≈−2\.7\\approx\-2\.7to−3\.0\-3\.0\) show minimal movement across conditions, suggesting that the Gemini family is comparatively resistant to numeric anchors in this protocol—with even lower ASI for the newer generation\.

Table 2:Count Bias \(mean±\\pmSD\) and Count\-F1 \(mean\) by model and prompt condition\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x1.png)Figure 1:Count Bias \(CB\) distributions across prompt conditions for all six models\. Boxes show median and IQR; whiskers extend to1\.5×1\.5\\timesIQR\. GPT\-5\.4 and Claude S\.4\.6 show strong anchoring with near\-zero variance in the Anchored and Mislead\-Over conditions; Claude H\.4\.5 also complies closely; Gemini 2\.5 maintains a stable undercount bias \(CB≈−2\.4\\approx\-2\.4\) across all conditions\. These panels should be read as stress\-test behavior profiles rather than overall model\-quality positions\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x2.png)Figure 2:Count\-based approximate F1 by condition and model\. The Anchored condition achieves near\-perfect Count\-F1 for GPT\-5\.4, Claude H\.4\.5, and Claude S\.4\.6, while Gemini 2\.5 shows minimal response to any anchor\.
### 4\.2Anchoring Sensitivity Index

Table[3](https://arxiv.org/html/2607.01240#S4.T3)shows ASI values across conditions and models\. GPT\-5\.4 records the largest ASI observed in this stress test: Mislead\-Under ASI=1\.922=1\.922, meaning the model shifts by nearly twice the true error count from its blind baseline when given a deflated anchor\. Claude S\.4\.6 shows the next\-largest observed value \(1\.066\)\. Both substantially exceed their within\-family predecessors \(GPT\-4o: 0\.816; Claude H\.4\.5: 0\.463\)\. By contrast, both Gemini models show very low ASI throughout: Gemini 2\.5 remains≤0\.178\\leq 0\.178and Gemini 3\.1 remains≤0\.132\\leq 0\.132, with the newer generation showing even greater resistance\.

Table 3:Anchoring Sensitivity Index \(ASI, mean±\\pmSD\) by model and prompt condition\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x3.png)Figure 3:Anchoring Sensitivity Index \(ASI\) heatmap across all six models and non\-Blind conditions\. Values are mean ASI across 143 passages\. GPT\-5\.4 and Claude S\.4\.6 show the largest anchoring \(ASI up to 1\.92\); Gemini 2\.5 remains≤0\.178\\leq 0\.178in all conditions\.
### 4\.3Statistical Significance

Because all prompt conditions are evaluated on the same passages, we re\-estimate condition effects with paired passage\-level t\-tests and Benjamini\-Hochberg FDR correction rather than independent\-samples tests\. For CB, Anchored and Mislead\-Under remain strongly different from Blind for GPT\-4o, GPT\-5\.4, Claude H\.4\.5, and Claude S\.4\.6 \(allq<0\.001q<0\.001\), and Mislead\-Over is also significant for all non\-Gemini models\. The Informed condition is not uniformly null: it produces small but reliable positive CB shifts for GPT\-4o \(Δ\\DeltaCB=\+0\.672=\+0\.672,q=0\.0024q=0\.0024\) and GPT\-5\.4 \(Δ\\DeltaCB=\+0\.483=\+0\.483,q=0\.0071q=0\.0071\), but remains non\-significant for both Claude models and Gemini\.

Paired effect sizes \(Cohen’sdz=mean​\(Δ​CB\)/sd​\(Δ​CB\)d\_\{z\}=\\mathrm\{mean\}\(\\Delta\\mathrm\{CB\}\)/\\mathrm\{sd\}\(\\Delta\\mathrm\{CB\}\)\) reinforce this picture\. GPT\-5\.4 shows large anchoring effects relative to its Blind baseline: Mislead\-Underdz=−2\.55d\_\{z\}=\-2\.55\(Δ\\DeltaCB=−8\.57=\-8\.57\), Anchoreddz=−2\.37d\_\{z\}=\-2\.37, and Mislead\-Overdz=−1\.73d\_\{z\}=\-1\.73—all well beyond conventional “very large” thresholds\. Claude S\.4\.6 and Claude H\.4\.5 are moderate\-to\-large on the Misleading conditions \(\|dz\|\|d\_\{z\}\|from0\.940\.94to1\.221\.22\), GPT\-4o is small\-to\-medium \(\|dz\|≤0\.53\|d\_\{z\}\|\\leq 0\.53\), and both Gemini models remain trivially small throughout \(\|dz\|≤0\.34\|d\_\{z\}\|\\leq 0\.34\)\. The fullΔ\\DeltaCB anddzd\_\{z\}table by model and condition is reported in Appendix[D](https://arxiv.org/html/2607.01240#A4)\.

### 4\.4Span\-level F1 and F1 Inflation

Table[4](https://arxiv.org/html/2607.01240#S4.T4)presents Count\-F1, M2 overlapF0\.5F\_\{0\.5\}, and F1 Inflation for all six models under key conditions\. Count\-F1 is inflated by0\.260\.26–0\.790\.79across models in the Anchored condition; the effect is substantially larger than a heuristic substring score would suggest\. GPT\-5\.4 shows the extreme case: Count\-F1 improves from0\.580\.58\(Blind\) to0\.990\.99\(Anchored\) while M2F0\.5F\_\{0\.5\}moves only from0\.200\.20to0\.200\.20, yielding an inflation of0\.790\.79\. Claude H\.4\.5 goes from Count\-F10\.860\.86to1\.001\.00while M2F0\.5F\_\{0\.5\}moves0\.43→0\.460\.43\\to 0\.46—nearly all the apparent gain is count compliance\. The Gemini family, which is largely anchor\-resistant in count terms, exhibits flat M2F0\.5F\_\{0\.5\}\(≈0\.24\\approx 0\.24–0\.340\.34\) across conditions and a correspondingly flat inflation, consistent with its stable undercount prior\. Under strict M2 matching \(Appendix[F](https://arxiv.org/html/2607.01240#A6)\) inflation reaches up to0\.960\.96, because anchor\-induced count agreement carries essentially no genuine localization benefit\. These results reinforce the main methodological point: count\-based gains under anchoring should not be interpreted as evidence of better error localization\.

Table 4:Count\-F1, M2 overlapF0\.5F\_\{0\.5\}, and F1 Inflation \(==Count\-F1−\-M2F0\.5overlapF\_\{0\.5\}^\{\\mathrm\{overlap\}\}\) by model and condition\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x4.png)Figure 4:Count\-based F1 \(solid bars\) vs\. heuristic text\-match Span\-F1 \(hatched bars\) by condition and model\. This figure is retained for diagnostic continuity; the authoritative span\-aware numbers are M2 overlapF0\.5F\_\{0\.5\}in Table[4](https://arxiv.org/html/2607.01240#S4.T4)\(primary\) and Appendix[F](https://arxiv.org/html/2607.01240#A6)\(full strict/detection/overlap breakdown\)\. The qualitative pattern—largest inflation in the Anchored condition for non\-Gemini models—is unchanged between the heuristic proxy and M2\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x5.png)Figure 5:F1 Inflation using the heuristic text\-match Span\-F1 proxy \(Count\-F1 minus heuristic Span\-F1\)\. Under M2 overlapF0\.5F\_\{0\.5\}the corresponding inflation values are substantially larger \(Table[4](https://arxiv.org/html/2607.01240#S4.T4)\); under strict M2 they reach up to0\.960\.96\(Appendix[F](https://arxiv.org/html/2607.01240#A6)\)\. Higher values indicate that count matching is more misleading relative to actual span localization quality\.![Refer to caption](https://arxiv.org/html/2607.01240v1/x6.png)Figure 6:Heuristic span\-level Precision and Recall across conditions for all six models\. A clear anchor\-driven Precision–Recall tradeoff is visible for all non\-Gemini models under Mislead\-Under \(high Precision, low Recall\) and Mislead\-Over \(low Precision, high Recall\), while Gemini remains comparatively flat\. Appendix[F](https://arxiv.org/html/2607.01240#A6)reports the corresponding M2 Precision and Recall under overlap matching\.
### 4\.5Case Studies

We present three qualitative cases drawn from our experimental data; full passage text, gold annotations, and verbatim model outputs are reported in Appendix[E](https://arxiv.org/html/2607.01240#A5)\.Case A: fabrication under inflated anchor \(GPT\-4o\)\.Passageconll\_0238contains 4 ground\-truth errors\. Under Blind prompting GPT\-4o identifies only 1 error; under Mislead\-Over \(anchor=6=6\), it reports exactly 6 errors, fabricating 5 additional ones to match the stated count\.Case B: suppression under deflated anchor \(Claude H\.4\.5\)\.Passageconll\_0210contains 3 annotated errors\. Under Blind prompting Claude H\.4\.5 reports 11 errors; under Mislead\-Under \(anchor=1=1\), it returns only 1 error, suppressing 10 previously identified findings\.Case C: resistance to anchoring \(Gemini 2\.5 Flash\)\.Passageconll\_0201contains 3 errors; the Mislead\-Over anchor is 5, yet Gemini reports only 2 errors, consistent with its stable undercount prior and general resistance to numerical anchoring cues\.

## 5Discussion

Our results provide evidence consistent with anchoring\-like behavior in LLM error detection tasks\. The pattern of findings is broadly compatible with[Tversky and Kahneman](https://arxiv.org/html/2607.01240#bib.bib16)’s anchoring\-and\-adjustment account\(Epley and Gilovich,[2006](https://arxiv.org/html/2607.01240#bib.bib21)\): models often appear to use the stated error count as a starting point and adjust insufficiently in the face of contradictory textual evidence\. A notable pattern in this dataset is that stronger instruction following does not uniformly reduce anchoring sensitivity\. GPT\-5\.4, which shows large unconstrained over\-reporting \(Blind CB=\+7\.3=\+7\.3\), is also the model most precisely controlled by the Mislead\-Over anchor \(SD=0=0\), yielding a large Anchored ASI \(=1\.62=1\.62\)\. Claude S\.4\.6 similarly records higher ASI values than the smaller Claude H\.4\.5, despite comparable Blind CB\. We interpret this as a capability\-alignment tradeoff in the present setup rather than as a universal scaling law\.

The six\-model comparison reveals both quantitative and qualitative differences as descriptive model\-family evidence\. Within the GPT and Claude families, newer generations show markedly larger anchoring responses in this protocol: GPT\-5\.4 Mislead\-Under ASI=1\.922=1\.922vs\. GPT\-4o=0\.816=0\.816; Claude S\.4\.6 ASI=1\.066=1\.066vs\. Claude H\.4\.5=0\.463=0\.463\. The Gemini family displays the opposite pattern: both Gemini 2\.5 and Gemini 3\.1 show very low ASI \(≤0\.178\\leq 0\.178and≤0\.132\\leq 0\.132respectively\), and the newer Gemini 3\.1 is even more resistant than its predecessor\. Gemini’s low ASI does not indicate superior detection accuracy; instead, it reflects a strong under\-reporting prior \(Gemini 2\.5 CB≈−2\.4\\approx\-2\.4; Gemini 3\.1 CB≈−2\.8\\approx\-2\.8\) that is difficult to move with numeric cues\.

Our findings have direct implications for LLM deployment in high\-stakes settings\. Document review systems, code auditing tools, and medical report checkers that pre\-populate error counts may systematically bias downstream LLM analysis\. A full ERRANT\-based replication on a100100\-passage subset across all six models with multi\-reference \(Annotators 0\+\+1\) scoring \(Appendix[H](https://arxiv.org/html/2607.01240#A8)\) reproduces the core F1\-Inflation pattern under the standard edit\-based GEC metric, confirming that it is not an artifact of our description\-anchored M2 extraction or of single\-annotator evaluation\.

### 5\.1Practical Recommendations for LLM Error\-Detection Evaluation

For journal readers interested in deploying or auditing LLM\-based proofreading and document\-review systems, the results imply four practical safeguards\.First, use blind prompts by default:evaluation prompts should not pre\-populate the expected number of errors unless the purpose is explicitly to test anchoring vulnerability\.Second, treat count metrics as auxiliary:Count\-F1 or exact\-count accuracy can diagnose coarse calibration, but they should not be reported as the sole evidence of error\-detection quality\.Third, report span\-aware metrics:evaluations should include M2, ERRANT, localization F\-score, or an equivalent metric that requires the system to identify where the error occurs, not merely how many errors are expected\.Fourth, monitor exact anchor matching in deployed systems:production pipelines should track how often an LLM returns precisely the user\- or system\-provided count, especially when upstream tools, templates, or human reviewers expose an expected number of issues\.

This study has several limitations\. Our primary span\-aware metric operates on hypothesis edits extracted deterministically from natural\-language error descriptions rather than from model\-produced corrected text, which is a lower bound on localization quality relative to full ERRANT pipelines; the main passage\-level analysis uses only Annotator\-0; cross\-model contrasts are treated descriptively; and all experiments use temperature=0=0\. The anchor range in our Anchored and Misleading conditions is fixed at the true count and±2\\pm 2respectively; sweeping multiple anchor magnitudes \(e\.g\.,±1,±3,±5\\pm 1,\\pm 3,\\pm 5\) is the natural next step\. Future work should also replicate findings across different error\-detection domains beyond grammatical error correction\.

## 6Conclusion

We present a controlled study of how prompt framing, especially numeric anchors, can distort LLM error\-detection evaluation\. Across six contemporary models and five prompt conditions, anchored prompts consistently shift reported counts for most models, with the largest effects appearing in highly instruction\-compliant GPT/Claude systems in our sample and weaker movement in the Gemini family; this is model\-family descriptive evidence, not a claim about overall model quality\. The central contribution of the paper is methodological: under CoNLL\-2014 M2\-style scoring on description\-anchored hypothesis edits, count\-based F1 gains under anchoring do not translate into span\-level gains, and F1 Inflation reaches up to0\.790\.79under overlap matching and up to0\.960\.96under strict matching\. We therefore position ErrorBench as a stress\-test protocol for prompt\-induced evaluation distortion rather than an overall model\-quality comparison\.

## Code and Data Availability

Code, prompts, and analysis scripts are available atTODO\-public\-repository\-url\. The public repository should include theerrorbench/directory, raw JSONL outputs, derived summary files, and the scripts used to regenerate all figures\.

## Acknowledgments

TODO Add acknowledgments and funding information for the public preprint version\.

## Limitations

We highlight several limitations that scope our claims\.

#### Description\-anchored M2 across all models; ERRANT on a 100\-passage subset\.

Our primary span\-aware metric across the full143143\-passage benchmark is CoNLL\-2014 M2\-styleF0\.5F\_\{0\.5\}\(strict / detection / overlap; Appendix[F](https://arxiv.org/html/2607.01240#A6)\), computed on hypothesis edits extracted deterministically from the*descriptions*produced by each model\. As a validation step, we re\-ran all six models on a stratified100100\-passage subset with an extended prompt that also elicits a corrected\-text block, and scored the resulting\(source,correction\)\(\\mathrm\{source\},\\mathrm\{correction\}\)pairs with the official ERRANT 3\.0\.0 pipeline under both single\-annotator and multi\-reference \(Annotators 0\+\+1\) scoring \(Appendix[H](https://arxiv.org/html/2607.01240#A8)\)\. The qualitative pattern—large Count\-F1 gains under Anchored that translate into at most small gains in span\-levelF0\.5F\_\{0\.5\}—is reproduced on every model\. Scaling this corrected\-text protocol to the full143143\-passage benchmark remains future work\. We also retain a legacy heuristic substring Span\-F1 \(Appendix[G](https://arxiv.org/html/2607.01240#A7)\) purely for continuity with the F1\-comparison figures\.

#### Instruction\-compliance floor for the Gemini family\.

In the ERRANT replication, Gemini 2\.5 and Gemini 3\.1 failed to produce a parseableCORRECTED TEXT:block on2525–80%80\\%of trials \(Appendix[H](https://arxiv.org/html/2607.01240#A8)\)\. This depresses their ERRANT rows and means their absolute ERRANTF0\.5F\_\{0\.5\}values should be read as a compliance\-bounded lower bound, not as an accuracy comparison against the GPT and Claude families\. Gemini Count\-F1 is unaffected \(the count is reported reliably in both the description and correction formats\)\.

#### Annotator reference\.

Passage construction and Count\-F1 ground truth still use Annotator\-0 from CoNLL\-2014 only; Annotator\-1 enters only in the multi\-reference ERRANT Appendix[H](https://arxiv.org/html/2607.01240#A8)\. Our Anchored and Misleading anchors are derived from Annotator\-0 counts, so a fraction of apparent over\- or under\-counts may be edits accepted by Annotator\-1; the multi\-reference ERRANT scores in Appendix[H](https://arxiv.org/html/2607.01240#A8)show that this does not change the Blind→\\toAnchored differential\.

#### Single inference per cell and fixed decoding\.

Each \(passage, model, condition\) cell corresponds to one API call at temperature0withmax\_tokens=800800\. We do not estimate sampling variance, do not run self\-consistency, and do not vary decoding temperature\. The headline effects are large relative to any plausible single\-sample noise, but the absolute numbers should not be treated as population estimates\.

#### Limited anchor range\.

The Mislead conditions fix\|M−N\|=2\|M\-N\|=2in both directions\. This is sufficient for the evaluation\-reliability claim—we need only one off\-by\-kkanchor to demonstrate F1 Inflation—but it does not let us recover the full dose–response curve that anchoring\-and\-adjustment\(Epley and Gilovich,[2006](https://arxiv.org/html/2607.01240#bib.bib21)\)predicts: whether compliance grows linearly, saturates, or reverses at large\|M−N\|\|M\-N\|\. Varying the anchor magnitude \(±1,±3,±5\\pm 1,\\pm 3,\\pm 5, and progressively more extreme distractors\) and measuring the resulting F1\-Inflation curve is the natural psychophysical follow\-up, and it would multiply the API budget by roughly3×3\\timesin our setup; we flag it explicitly as out of scope for this submission\.

#### Dataset and domain scope\.

CoNLL\-2014 is learner English from a specific population, and we filter to passages with33–77Annotator\-0 errors\. Generalization to other error\-detection settings \(factual checking, code review, scientific proofreading\) is not established here and is left for future work\.

#### Model\-specific confounds\.

For the Gemini family, low ASI coincides with a persistent undercount prior \(CB≈−2\.4\\approx\-2\.4to−2\.8\-2\.8\)\. Our data cannot cleanly separate anchor\-resistance from a systematic bias toward fewer reported edits\. We therefore treat cross\-family contrasts as descriptive stress\-test evidence rather than as overall ordering claims or causal claims about alignment techniques\.

#### API\-based contemporary models\.

All models were queried through an OpenAI\-compatible proxy endpoint between 2025 and 2026\. Provider\-side changes \(system prompts, safety filters, routing\) can alter behavior over time, and exact reproducibility depends on the model snapshots listed in Appendix[B](https://arxiv.org/html/2607.01240#A2)\.

## Ethics Statement

We use CoNLL\-2014, a publicly released shared\-task dataset of learner English essays, in accordance with its original terms\. It contains no personally identifying information as distributed\. Our study does not collect human subject data and does not label individual writers\. Model outputs in the Case Studies are produced on publicly released passages and contain no private content\. The study reports a failure mode—prompt\-induced count distortion—that, if ignored, could inflate reported LLM performance in high\-stakes proofreading deployments\. We therefore recommend that practitioners report span\-aware metrics alongside count\-based metrics and avoid prepopulating anchor counts in user\-facing pipelines\. No deployment of the framework itself is proposed in this paper\.

## References

- Anthropic \(2025\)Claude haiku 4\.5 system card\.Note:[https://www\.anthropic\.com/system\-cards](https://www.anthropic.com/system-cards)Model system cards page; Claude Haiku 4\.5 listed as October 2025Cited by:[§3\.3](https://arxiv.org/html/2607.01240#S3.SS3.p1.3)\.
- Anthropic \(2026\)Claude sonnet 4\.6 system card\.Note:[https://www\.anthropic\.com/system\-cards](https://www.anthropic.com/system-cards)Model system cards page; Claude Sonnet 4\.6 listed as February 2026Cited by:[§3\.3](https://arxiv.org/html/2607.01240#S3.SS3.p1.3)\.
- C\. Bryant, M\. Felice, Ø\. E\. Andersen, and T\. Briscoe \(2019\)The BEA\-2019 shared task on grammatical error correction\.InProceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications,pp\. 52–75\.External Links:[Link](https://aclanthology.org/W19-4406/),[Document](https://dx.doi.org/10.18653/v1/W19-4406)Cited by:[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- C\. Bryant, Z\. Yuan, M\. R\. Qorib, H\. Cao, H\. T\. Ng, and T\. Briscoe \(2023\)Grammatical error correction: a survey of the state of the art\.Computational Linguistics49\(3\),pp\. 643–701\.External Links:[Link](https://aclanthology.org/2023.cl-3.4/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00478)Cited by:[Appendix H](https://arxiv.org/html/2607.01240#A8.SS0.SSS0.Px4.p1.9),[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- A\. Checco, L\. Bracciale, P\. Loreti, S\. Pinfield, and G\. Bianchi \(2021\)AI\-assisted peer review\.Humanities and Social Sciences Communications8\(1\),pp\. 25\.External Links:[Link](https://www.nature.com/articles/s41599-020-00703-8),[Document](https://dx.doi.org/10.1057/s41599-020-00703-8)Cited by:[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- N\. Dycke, I\. Kuznetsov, and I\. Gurevych \(2023\)NLPeer: a unified resource for the computational study of peer review\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5049–5073\.External Links:[Link](https://aclanthology.org/2023.acl-long.277/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.277)Cited by:[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- N\. Epley and T\. Gilovich \(2006\)The anchoring\-and\-adjustment heuristic: why the adjustments are insufficient\.Psychological Science17\(4\),pp\. 311–318\.External Links:[Document](https://dx.doi.org/10.1111/j.1467-9280.2006.01704.x)Cited by:[§1](https://arxiv.org/html/2607.01240#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.01240#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.01240#S5.p1.3),[Limited anchor range\.](https://arxiv.org/html/2607.01240#Sx3.SS0.SSS0.Px5.p1.5)\.
- T\. Fang, S\. Yang, K\. Lan, D\. F\. Wong, J\. Hu, L\. S\. Chao, and Y\. Zhang \(2023\)Is ChatGPT a highly fluent grammatical error correction system? a comprehensive evaluation\.External Links:2304\.01746,[Link](https://arxiv.org/abs/2304.01746)Cited by:[Appendix H](https://arxiv.org/html/2607.01240#A8.SS0.SSS0.Px4.p1.9),[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- Google DeepMind \(2025\)Gemini 2\.5 flash\.Note:[https://deepmind\.google/en/models/gemini/flash/](https://deepmind.google/en/models/gemini/flash/)Official model pageCited by:[§3\.3](https://arxiv.org/html/2607.01240#S3.SS3.p1.3)\.
- R\. Liu and N\. B\. Shah \(2023\)ReviewerGPT? an exploratory study on using large language models for paper reviewing\.External Links:2306\.00622,[Link](https://arxiv.org/abs/2306.00622)Cited by:[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1)\.
- Y\. Lu, M\. Bartolo, A\. Moore, S\. Riedel, and P\. Stenetorp \(2022\)Fantastically ordered prompts and where to find them: overcoming few\-shot prompt order sensitivity\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8086–8098\.External Links:[Link](https://aclanthology.org/2022.acl-long.556/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.556)Cited by:[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.
- A\. S\. Luchins \(1942\)Mechanization in problem solving: the effect of einstellung\.Psychological Monographs54\(6\),pp\. 1–95\.External Links:[Document](https://dx.doi.org/10.1037/h0093502)Cited by:[§2\.3](https://arxiv.org/html/2607.01240#S2.SS3.p1.1)\.
- O\. Macmillan\-Scott and M\. Musolesi \(2024\)\(Ir\)rationality and cognitive biases in large language models\.Royal Society Open Science11\(6\),pp\. 240255\.External Links:[Link](https://royalsocietypublishing.org/doi/10.1098/rsos.240255),[Document](https://dx.doi.org/10.1098/rsos.240255)Cited by:[§1](https://arxiv.org/html/2607.01240#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.01240#S2.SS3.p1.1)\.
- S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer \(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 11048–11064\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.759/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.759)Cited by:[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.
- H\. T\. Ng, S\. M\. Wu, T\. Briscoe, C\. Hadiwinoto, R\. H\. Susanto, and C\. Bryant \(2014\)The CoNLL\-2014 shared task on grammatical error correction\.InProceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task,pp\. 1–14\.External Links:[Link](https://aclanthology.org/W14-1701/),[Document](https://dx.doi.org/10.3115/v1/W14-1701)Cited by:[Appendix H](https://arxiv.org/html/2607.01240#A8.p1.7),[§2\.1](https://arxiv.org/html/2607.01240#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.01240#S3.SS1.p1.1)\.
- OpenAI \(2024\)GPT\-4o system card\.Note:[https://openai\.com/index/gpt\-4o\-system\-card/](https://openai.com/index/gpt-4o-system-card/)Cited by:[§3\.3](https://arxiv.org/html/2607.01240#S3.SS3.p1.3)\.
- OpenAI \(2026\)GPT\-5\.4 thinking system card\.Note:[https://openai\.com/index/gpt\-5\-4\-thinking\-system\-card/](https://openai.com/index/gpt-5-4-thinking-system-card/)Cited by:[§3\.3](https://arxiv.org/html/2607.01240#S3.SS3.p1.3)\.
- E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving \(2022\)Red teaming language models with language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3419–3448\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.225/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.225)Cited by:[§1](https://arxiv.org/html/2607.01240#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.
- M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, E\. Durmus, Z\. Hatfield\-Dodds, S\. R\. Johnston, S\. M\. Kravec, T\. Maxwell, S\. McCandlish, K\. Ndousse, O\. Rausch, N\. Schiefer, D\. Yan, M\. Zhang, and E\. Perez \(2024\)Towards understanding sycophancy in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tvhaxkMKAn)Cited by:[§1](https://arxiv.org/html/2607.01240#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.
- A\. Tversky and D\. Kahneman \(1974\)Judgment under uncertainty: heuristics and biases\.Science185\(4157\),pp\. 1124–1131\.External Links:[Document](https://dx.doi.org/10.1126/science.185.4157.1124)Cited by:[§1](https://arxiv.org/html/2607.01240#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.01240#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.01240#S5.p1.3)\.
- J\. Ye, Y\. Wang, Y\. Huang, D\. Chen, Q\. Zhang, N\. Moniz, T\. Gao, W\. Geyer, C\. Huang, P\. Chen, N\. V\. Chawla, and X\. Zhang \(2024\)Justice or prejudice? quantifying biases in LLM\-as\-a\-judge\.External Links:2410\.02736,[Link](https://arxiv.org/abs/2410.02736)Cited by:[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.
- Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InProceedings of the 38th International Conference on Machine Learning,pp\. 12697–12706\.External Links:[Link](https://proceedings.mlr.press/v139/zhao21c.html)Cited by:[§2\.2](https://arxiv.org/html/2607.01240#S2.SS2.p1.1)\.

## Appendix APrompt Templates

All experiments share the system prompt in Figure[7](https://arxiv.org/html/2607.01240#A1.F7)and vary only the user prompt by condition \(Figure[8](https://arxiv.org/html/2607.01240#A1.F8)\)\. The placeholder\{text\}is the four\-sentence passage andNNis the Annotator\-0 error count\. Prompts are applied verbatim; no additional post\-processing is performed before the API call\.

System prompt \(all conditions\)You are a grammar error detection assistant\. Examine the provided English text and list every grammatical error you find\. For each error, write exactly one line:ERROR N: \[brief description, 10 words max\]\. After listing all errors, write on its own line:TOTAL ERRORS FOUND: N\. Do not include any other text\.Figure 7:Shared system prompt\.Figure 8:User prompts for the five conditions\.NNis the Annotator\-0 error count of the passage\.
## Appendix BModel Endpoints and Decoding Settings

All queries go through an OpenAI\-compatible proxy\. Endpoints and model identifiers used in the experiment are listed in Table[5](https://arxiv.org/html/2607.01240#A2.T5)\. Decoding settings are identical across conditions: temperature=0=0,max\_tokens=800=800\(max\_completion\_tokensfor GPT\-5\.4\), single sample per cell, with exponential backoff on HTTP 429\. The total number of API calls is143×6×5=4,290143\\times 6\\times 5=4\{,\}290\.

Table 5:Model labels and API identifiers used in the experiment\.
## Appendix CReproducibility

Data preparation \(prepare\_data\.py\), experiment runner \(run\_experiment\.py\), analysis \(analyze\_results\.py\), and figure generation \(make\_figures\.py\) scripts are provided in theerrorbench/directory alongside the raw per\-call JSONL \(outputs/results\.jsonl\) and derived summary CSVs\. Passage sampling uses a fixed random seed of4242\. Four\-sentence windows are restricted to within\-document boundaries so that no passage spans two CoNLL\-2014 source documents\.

## Appendix DPaired Effect Sizes for Count Bias

Table[6](https://arxiv.org/html/2607.01240#A4.T6)reports paired per\-passage shifts relative to the Blind baseline\. For each \(model, condition\), we computeΔ​CBi=CBicond−CBiBlind\\Delta\\mathrm\{CB\}\_\{i\}=\\mathrm\{CB\}\_\{i\}^\{\\mathrm\{cond\}\}\-\\mathrm\{CB\}\_\{i\}^\{\\mathrm\{Blind\}\}across all passagesiiwith both observations, and summarise with mean, standard deviation, and Cohen’sdz=mean​\(Δ​CB\)/sd​\(Δ​CB\)d\_\{z\}=\\mathrm\{mean\}\(\\Delta\\mathrm\{CB\}\)/\\mathrm\{sd\}\(\\Delta\\mathrm\{CB\}\)\. Sample sizes aren=143n=143for all non\-Gemini cells;n∈\[132,135\]n\\in\[132,135\]for Gemini 3\.1 \(a handful of API responses did not parse to a valid count\)\.

Table 6:PairedΔ\\DeltaCB relative to Blind and Cohen’sdzd\_\{z\}by model and condition\.dzd\_\{z\}is computed on the passage\-level paired differences\.
## Appendix ECase Studies: Full Passages and Model Outputs

We report the full source passages and verbatim model outputs for the three cases discussed in §[4\.5](https://arxiv.org/html/2607.01240#S4.SS5)\. Gold annotations are from CoNLL\-2014 Annotator\-0\.

#### Case A: fabrication under inflated anchor\.

Passageconll\_0238,N=4N=4\. GPT\-4o reports11error under Blind and exactly66errors under Mislead\-Over \(anchor=6=6\), inventing five low\-salience or non\-errors \(spacing around punctuation, an article choice that is standard in the original\) to fill the quota\.

Passage\.Because of this, other people start to think that our lives are full of happiness with no problem at all\. Other people then are influenced by us and tend to do the same\. As a result, people think of theirselves lowly and may lead to depression and mental disorder\. It also reduces our sympathy\.Gold \(4\)\.Nn: problem→\\toproblems; Pform: theirselves→\\tothemselves; Wci: lowly→\\tobadly; Pref:∅→\\emptyset\\tothis\.GPT\-4o, Blind \(reported=1\)\.ERROR 1: Incorrect use of "theirselves," should be "themselves\." TOTAL ERRORS FOUND: 1GPT\-4o, Mislead\-Over, anchor=6 \(reported=6\)\.ERROR 1: Incorrect spacing before comma, "this ,"\. ERROR 2: Incorrect spacing before period, "at all \."\. ERROR 3: Incorrect reflexive pronoun, "theirselves" should be "themselves"\. ERROR 4: Incorrect article usage, "a" before "depression"\. ERROR 5: Incorrect spacing before period, "disorder \."\. ERROR 6: Incorrect spacing before period, "sympathy \."\. TOTAL ERRORS FOUND: 6\.Table 7:Case A—GPT\-4o onconll\_0238\.
#### Case B: suppression under deflated anchor\.

Passageconll\_0210,N=3N=3\. Claude H\.4\.5 reports1111errors under Blind—mostly punctuation concerns outside the Annotator\-0 gold set—and collapses to a single report under Mislead\-Under \(anchor=1=1\), retaining only the most salient “the their” error\.

Passage\.The users, soon, will realise that they do not need such social media sites to keep in contact with the their friends because, if they really need somebody, they will reach the friends no matter what\. We think we are all connected through all sorts of social media, but do we really get connected or falling apart? Social media sites like Facebook and Twitter are born with the intention of connecting people, from around the world\. The idea seems great, and nowadays most people even the older generation will have at least one social media platform like Facebook\.Gold \(3\)\.ArtOrDet: the→∅\\to\\emptyset\(in “the their”\); ArtOrDet: the→∅\\to\\emptyset; Vform: falling→\\tofall\.Claude H\.4\.5, Blind \(reported=11\)\.11ERRORlines, largely covering comma placement and a subject\-verb disagreement on “get connected or falling apart”\.Claude H\.4\.5, Mislead\-Under, anchor=1 \(reported=1\)\.ERROR 1: "the their" \- redundant article, should be "their"\. TOTAL ERRORS FOUND: 1Table 8:Case B—Claude H\.4\.5 onconll\_0210\.
#### Case C: resistance to anchoring\.

Passageconll\_0201,N=3N=3\. Under Mislead\-Over \(anchor=5=5\), Gemini 2\.5 Flash reports only22errors, below both the anchor and the true count—consistent with its persistent undercount prior rather than compliance with the prompted count\.

Passage\.The next two paragraphs will be discussing about the advantages and disadvantages of using social media in our society\. …Social media sites such as Facebook has allow us to share our pictures or even chat online with our parents while we are overseas\. This approaches help the parents to communicate with their children …Gold \(3\)\.Prep: about→∅\\to\\emptyset; Vt: has→∅\\to\\emptyset; Pform: This→\\toThese\.Gemini 2\.5, Mislead\-Over, anchor=5 \(reported=2\)\.ERROR 1: Redundant "about" after "discussing"\. ERROR 2: Incorrect verb form and subject\-verb agreement \("has allow"\)\.Table 9:Case C—Gemini 2\.5 Flash onconll\_0201\.

## Appendix FFull M2 Scoring Tables

Table[10](https://arxiv.org/html/2607.01240#A6.T10)reports the three M2F0\.5F\_\{0\.5\}variants introduced in §3\.4 for all 30 \(model, condition\) cells\. Strict matches on\(sent​\_​idx,start,end,corr\)\(\\mathrm\{sent\\\_idx\},\\mathrm\{start\},\\mathrm\{end\},\\mathrm\{corr\}\); detection on\(sent​\_​idx,start,end\)\(\\mathrm\{sent\\\_idx\},\\mathrm\{start\},\\mathrm\{end\}\); overlap on same sentence with non\-empty token overlap\. Of the21,17221\{,\}172total model error descriptions that parse into our pipeline,16,55216\{,\}552\(78\.2%78\.2\\%\) are deterministically localized to a passage span; the remaining21\.8%21\.8\\%are conservatively counted as false positives\. Localization failures are dominated by descriptions without quoted fragments \(e\.g\., “Comma misuse, space before comma”\), which cannot be grounded to a specific token span\.

Table 10:M2F0\.5F\_\{0\.5\}under Strict \(S: span \+ correction\), Detection \(D: span only\), and Overlap \(O: same sentence, non\-empty token overlap\) matching, by model and condition \(Bl\.=Blind, In\.=Informed, An\.=Anchored, MO\.=Mislead\-Over, MU\.=Mislead\-Under\)\. Corpus\-level micro\-averages over143143passages\.
## Appendix GLegacy Heuristic Span\-F1

Table[11](https://arxiv.org/html/2607.01240#A7.T11)reports the heuristic case\-insensitive substring\-match Span\-F1 used in earlier drafts \(and still plotted in Figures[4](https://arxiv.org/html/2607.01240#S4.F4)–[6](https://arxiv.org/html/2607.01240#S4.F6)for continuity\)\. This metric counts a gold edit as matched whenever any word from itssrc/corrfields appears as a substring of any model description, with no span localization\. It is substantially more permissive than M2 overlap and is retained only for backward compatibility\.

Table 11:Legacy heuristic text\-match Span\-F1 by model and condition\. Provided for continuity with Figures[4](https://arxiv.org/html/2607.01240#S4.F4)–[6](https://arxiv.org/html/2607.01240#S4.F6); M2 overlapF0\.5F\_\{0\.5\}\(Table[4](https://arxiv.org/html/2607.01240#S4.T4)\) is the primary span\-aware metric\.
## Appendix HERRANT Full Replication \(Six Models, Multi\-Reference\)

To validate that the description\-anchored M2 metric used in the main results is not an artifact of hypothesis\-edit extraction, we re\-ran*all six*contemporary models on a stratified100100\-passage subset \(2020passages per Annotator\-0 error\-count bucket in\{3,…,7\}\\\{3,\\dots,7\\\}, seed=42=42\) with an extended prompt that requests both the originalERROR N:description list and aCORRECTED TEXT:block containing the full passage with all errors fixed\. This produces2,999/3,0002\{,\}999/3\{,\}000usable outputs \(6 models×\\times5 conditions×\\times100 passages; one Gemini 3\.1 call returned no content and is dropped\)\. Each corrected passage is tokenized and sentence\-aligned to the source, and edits are extracted and scored with the standard ERRANT 3\.0\.0 pipeline against gold edits produced through the same ERRANT tokenizer\. For multi\-reference scoring we use the official CoNLL\-2014 Annotator\-0 and Annotator\-1 gold edits from the shared\-task M2 release\(Nget al\.,[2014](https://arxiv.org/html/2607.01240#bib.bib1)\)and take the sentence\-level max\-match over the two references\. All metrics are corpus\-level micro\-averages\.

#### Compliance caveat\.

GPT\-4o, GPT\-5\.4, Claude H\.4\.5, and Claude S\.4\.6 produce a parseableCORRECTED TEXT:block on≥499/500\\geq 499/500trials\. Gemini 2\.5 and Gemini 3\.1 fail on2525–80%80\\%of trials depending on condition \(column noC in Table[12](https://arxiv.org/html/2607.01240#A8.T12)\)\. Empty outputs contribute zero predicted edits and therefore depress ERRANTF0\.5F\_\{0\.5\}of those rows; the Gemini rows should be read with this compliance floor in mind\. No Gemini row is used to support the core F1\-Inflation claim\.

#### Result\.

The qualitative pattern from Table[4](https://arxiv.org/html/2607.01240#S4.T4)is reproduced across every model\. The Anchored condition drives Count\-F1 to≥0\.93\\geq 0\.93for all six models \(mean0\.9750\.975, vs\.0\.7630\.763in Blind,Δ=\+0\.212\\Delta=\+0\.212\), while Annotator\-0 ERRANTF0\.5F\_\{0\.5\}under Anchored is within±0\.10\\pm 0\.10of the corresponding Blind baseline in five of six comparisons \(Gemini 3\.1 is the exception, and its Anchored gain is driven by a2424\-point reduction in the noC compliance gap rather than by improved span quality\)\. Averaged over the six models, Blind→\\toAnchored lifts Count\-F1 by\+0\.212\+0\.212while lifting ERRANTF0\.5F\_\{0\.5\}by only\+0\.047\+0\.047\(Annotator\-0\) or\+0\.038\+0\.038\(multi\-reference\)—a44–5×5\\timesgap that mirrors the description\-anchored M2 result on the full143143\-passage benchmark\. Adding Annotator\-1 raises absolute ERRANTF0\.5F\_\{0\.5\}by0\.120\.12–0\.160\.16on average but leaves the Blind→\\toAnchored differential essentially unchanged, confirming that the F1\-Inflation signal is not driven by single\-annotator idiosyncrasies\.

#### Statistical significance \(paired bootstrap\)\.

For each model we resample the 100 passages with replacement \(1,0001\{,\}000bootstrap iterations, seed=42=42, paired across Blind and Anchored\) and recompute Count\-F1 and multi\-reference ERRANTF0\.5F\_\{0\.5\}\. Figure[9](https://arxiv.org/html/2607.01240#A8.F9)plots the mean and95%95\\%CIs forΔ\\DeltaCount\-F1 andΔ\\DeltaERRANTF0\.5F\_\{0\.5\}\(Anchored−\-Blind\) per model\. EveryΔ\\DeltaCount\-F1 CI is strictly positive; four of sixΔ\\DeltaERRANT CIs cover or cross zero \(Claude H\.4\.5, Claude S\.4\.6, GPT\-4o, Gemini 2\.5\)\. The F1\-Inflation contrastΔ​Count​\-F1−Δ​ERRANTM\\Delta\\mathrm\{Count\}\\text\{\-F1\}\-\\Delta\\mathrm\{ERRANT\}\_\{\\mathrm\{M\}\}is strictly positive at95%95\\%for five of six models: GPT\-5\.4\+0\.359​\[\+0\.304,\+0\.416\]\+0\.359\\ \[\+0\.304,\+0\.416\], GPT\-4o\+0\.236​\[\+0\.113,\+0\.404\]\+0\.236\\ \[\+0\.113,\+0\.404\], Claude S\.4\.6\+0\.138​\[\+0\.083,\+0\.187\]\+0\.138\\ \[\+0\.083,\+0\.187\], Gemini 2\.5\+0\.136​\[\+0\.069,\+0\.210\]\+0\.136\\ \[\+0\.069,\+0\.210\], Claude H\.4\.5\+0\.118​\[\+0\.071,\+0\.163\]\+0\.118\\ \[\+0\.071,\+0\.163\]\. Gemini 3\.1 is the only borderline case \(\+0\.074​\[−0\.015,\+0\.154\]\+0\.074\\ \[\-0\.015,\+0\.154\]\), and as noted above its ERRANT row is compliance\-confounded\.

![Refer to caption](https://arxiv.org/html/2607.01240v1/x7.png)Figure 9:Paired bootstrap on the 100\-passage subset \(1,0001\{,\}000iterations, seed=42=42\)\. Bars show meanΔ\\DeltaCount\-F1 andΔ\\DeltaERRANTF0\.5F\_\{0\.5\}\(multi\-reference\) for Blind→\\toAnchored; error bars are95%95\\%percentile CIs\. For every model,Δ\\DeltaCount\-F1 is significantly positive and strictly larger thanΔ\\DeltaERRANT, and the contrast is significant \(95%95\\%CI\>0\>0\) for five of six models\.Table 12:ERRANT full replication on the 100\-passage Path\-B subset, all six models\. Count==Count\-F1; M2==description\-anchored M2 overlapF0\.5F\_\{0\.5\}from the main results, recomputed on this subset; noC==number of passages \(out of 100\) with no parseableCORRECTED TEXT:block; ERR0==ERRANTF0\.5F\_\{0\.5\}against Annotator\-0; ERRM==ERRANTF0\.5F\_\{0\.5\}under Annotator\-0\+\+Annotator\-1 max\-match multi\-reference scoring;Δ0E=\\Delta^\{\\mathrm\{E\}\}\_\{0\}=Count−\-ERR0\. Gemini 2\.5 and Gemini 3\.1 rows withnoC≫0\\mathrm\{noC\}\\gg 0depress ERR due to compliance rather than span quality\.
#### Context for absolute magnitudes\.

Our passage\-level multi\-reference ERRANTF0\.5F\_\{0\.5\}values of0\.400\.40–0\.570\.57are broadly consistent with previously reported LLM\-based GEC performance on CoNLL\-2014\(Fanget al\.,[2023](https://arxiv.org/html/2607.01240#bib.bib8)\), while supervised GEC systems on the sentence\-level CoNLL\-2014 test set report ERRANTF0\.5F\_\{0\.5\}in the0\.550\.55–0\.690\.69range\(Bryantet al\.,[2023](https://arxiv.org/html/2607.01240#bib.bib3)\)\. The absolute numbers are not the point: the central observation of this paper is that Count\-F1 is driven to≥0\.93\\geq 0\.93for every model under Anchored prompting while multi\-reference ERRANTF0\.5F\_\{0\.5\}stays within±0\.06\\pm 0\.06of the model’s own Blind baseline in five of six cases, so Count\-F1 cannot be trusted as a proxy for span\-level capability under anchor\-bearing prompts,*regardless*of where that model sits on any absolute GEC evaluation table\. This is why ErrorBench should be used as a stress\-test protocol for evaluation design rather than as an overall model\-quality comparison\.