Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

# Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers
Source: [https://arxiv.org/html/2607.16212](https://arxiv.org/html/2607.16212)
###### Abstract

Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim\. We recast the detection of such errors as*typed*verification: we introduce a five\-class typed\-quantity error taxonomy and a 1500\-item benchmark, rewritten from PMC and arXiv sources and labeled by two independent LLM annotators with adjudication \(Krippendorff’sα=0\.882\\alpha=0\.882\)\. A ModernBERT encoder fine\-tuned on this benchmark reachesm​a​c​r​o−F​1=0\.899macro\-F\\textsubscript\{1\}=0\.899, far above any off\-the\-shelf neural fact\-checker, yet four probes expose a sharp structural blind spot: on canonical\-equivalent rewrites of physically equivalent quantities \(e\.g\.,95°C95\\text\{\\,\}\\mathrm\{\\SIUnitSymbolCelsius\}↔\\leftrightarrow368\.15K368\.15\\text\{\\,\}\\mathrm\{K\}\) its accuracy collapses to36\.5%36\.5\\%\. We proposeSymbolic Augmentation, a training\-time framework that runs the modules of a symbolic verifier in reverse to generate label\-preserving augmented training data\. The augmentation lifts canonical\-equivalence robustness to98\.2%98\.2\\%while slightly improving in\-distribution accuracy \(m​a​c​r​o−F​1macro\-F\\textsubscript\{1\}:0\.899→0\.9020\.899\\to 0\.902\); the augmented encoder matches a closed\-frontier LLM at no inference cost and transfers to an external benchmark \(SciFact\-Open binarym​a​c​r​o−F​1macro\-F\\textsubscript\{1\}:0\.791→0\.8280\.791\\to 0\.828\)\. Two negative results sharpen the claim: symbolic features as auxiliary encoder inputs add nothing, and symbolic silver labels scale negatively under teacher noise\. Together these results identify training\-time augmentation as the right integration point between symbolic and learned components\.

Symbolic Augmentation Closes a Canonical\-Equivalence Blind Spot in Neural Fact\-Checkers

Genpei ZhangUniversity of Wisconsin\-Madison

## 1Introduction

LLM\-based scientific writing assistants are now widely deployed: search interfaces produce one\-sentence summaries of papers\(Singhet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib1); Asaiet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib2)\), agentic pipelines compress experimental protocols\(Boikoet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib3); M\. Branet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib4)\), and authoring tools draft method sections from raw notes\(Lianget al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib6)\)\. In all of these settings, numeric and unit hallucinations are a uniquely dangerous failure mode: a magnitude, unit, or comparator in the generated text disagrees with the source in a way that changes the scientific claim, while remaining grammatically fluent\. Unlike paraphrase errors, which a careful reader can often catch from context, a single mis\-stated unit can flip the safety profile of a drug candidate without any local linguistic cue\(Jiet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib7); Zhanget al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib8)\)\.

Consider an evidence sentence “Compound A inhibited EGFR with an IC50of12nM12\\text\{\\,\}\\mathrm\{nM\}” summarized as “…with an IC50of12mgml−112\\text\{\\,\}\\mathrm\{mg\}\\text\{\\,\}\{\\mathrm\{ml\}\}^\{\-1\}\.” A practicing biologist sees the problem at once: nM is a molar concentration while mg/mL is a mass\-volume concentration, and converting between them requires the molar mass of compound A—the two summaries support entirely different conclusions\. A generic fact\-check verifier, however, will at best answer “not entailed,” giving no signal about*which kind*of error occurred,*which span*to repair, or whether the defect is a typo, an order\-of\-magnitude slip, or a category\-level confusion\. We argue this task should be modeled as*typed*classification, with the verifier producing not a binary verdict but one of five labels \(Correct,Unit Error,Scale Error,Relation Error,Unsupported\) that each corresponds to a distinct symbolic axis of the quantitative claim\.

To enable systematic study of this task, we construct a frozen15001500\-sentence evaluation set, LLM\-rewritten from real PMC \(biomedical and materials science\) and arXiv \(computer science\) sources, balanced across the five classes and dual\-annotated by two independent LLMs with third\-party adjudication on disagreements; Krippendorff’sα=0\.882\\alpha=0\.882exceeds the standard0\.800\.80publishability threshold\(Ford,[2004](https://arxiv.org/html/2607.16212#bib.bib28)\)\. A small probe set with a disjoint perturbation generator and a 50\-item human spot\-check \(94%94\\%agreement\) address the self\-labeling circularity concern\.

On this benchmark, a fine\-tuned ModernBERT encoder\(Warneret al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib14); Sileo,[2024](https://arxiv.org/html/2607.16212#bib.bib15)\)reachesm​a​c​r​o−F​1=0\.899macro\-F\\textsubscript\{1\}=0\.899\(5\-fold CV\), far above any off\-the\-shelf neural fact\-checker we test and within0\.010\.01–0\.070\.07of closed\-frontier LLMs at a fraction of the cost\. The encoder is, by this measure, remarkably capable—yet it exhibits a sharp*structural*blind spot\. When we apply canonical\-equivalent rewrites to goldCorrectitems—rewriting 95∘C as 368\.15 K, or 0\.5 mol/L as 500 mmol/L—its accuracy on these physically equivalent pairs drops from96\.7%96\.7\\%to36\.5%36\.5\\%\. Three further probes \(lexical synonyms, unit\-name paraphrase, scientific notation\) localize the failure: the encoder is robust to lexical and syntactic variation but cannot canonicalize physically equivalent numerical surfaces\.

This blind spot suggests an integration question: a symbolic verifier canonicalizes units trivially via a registry, so can it supplement the encoder where the encoder is blind, and at which point in the pipeline? We answer by introducingSymbolic Augmentation, a training\-time framework that runs the modules of a symbolic verifier in reverse to generate label\-preserving training data\. Four augmentation families—covering unit canonical\-equivalence, comparator synonyms, unit\-name paraphrase, and scientific notation—are each derived from one verifier module \(§[4](https://arxiv.org/html/2607.16212#S4)\) and tested with a per\-family probe\. Training with the canonical\-equivalence family raises its probe accuracy from36\.5%36\.5\\%to98\.2%98\.2\\%while slightly improving in\-distribution accuracy; the augmented encoder matches a closed\-frontier LLM at no inference cost and transfers to an external benchmark\. We complement these positive results with two principled negatives—symbolic features as auxiliary inputs, and the verifier as a silver\-label teacher—that locate training\-time augmentation as the right integration point\.

#### Contributions\.

\(C1\) Task and benchmark\.We define a five\-class typed\-quantity error taxonomy with priority rules and a templated repair specification \(§[3](https://arxiv.org/html/2607.16212#S3)\), and release a 1500\-item benchmark rewritten from PMC and arXiv sources, dual\-annotated by two independent LLMs with third\-party adjudication \(α=0\.882\\alpha=0\.882\) and validated by a 50\-item human spot\-check \(§[5\.1](https://arxiv.org/html/2607.16212#S5.SS1)\)\.

\(C2\) Symbolic Augmentation\.We localize the fine\-tuned encoder’s residual failure to canonical\-equivalence rewrites through a four\-family probe suite, and propose a training\-time framework that derives label\-preserving augmentation rules from the modules of a symbolic verifier, closing the blind spot and transferring to an external benchmark \(§[4](https://arxiv.org/html/2607.16212#S4), §[5\.3](https://arxiv.org/html/2607.16212#S5.SS3.SSS0.Px2)\)\.

\(C3\) Where symbolic information should enter\.We compare five integration points and show that only training\-time augmentation delivers a measurable, transferable benefit: feature\-level concatenation, silver\-label teaching, and inference\-time ensembling all fail \(§[5\.4](https://arxiv.org/html/2607.16212#S5.SS4)\)\.

## 2Related Work

#### Faithfulness verification\.

The dominant paradigm casts faithfulness as binary or ternary entailment between a claim and its evidence\. MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib9)\)distills a calibrated fact\-checker matching GPT\-4 at lower cost, and a line of recent systems extends this paradigm—Bespoke\-MiniCheck\-7B\(Bespoke Labs,[2024](https://arxiv.org/html/2607.16212#bib.bib10)\), FactCG\(Leiet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib13)\), VeriFastScore\(Rajendhranet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib16)\), and Granite Guardian\(Padhiet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib17)\)—alongside sampling\- and tool\-based checkers\(Manakulet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib11); Chernet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib12)\)\. A parallel line moves beyond a single faithfulness scalar: ACUEval\(Wanet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib32)\)types atomic units as supported, missing, or hallucinated; FineSurE\(Songet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib33)\)predicts sentence\-level error types; and the HalluLens benchmark\(Banget al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib34)\)and Mu\-SHROOM shared task\(Vázquezet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib35)\)target fine\-grained hallucination\. These typed systems share our premise that a binary judgment is information\-poor, but organize errors around*content\-source*relations \(extraneous, missing, unsupported\) rather than the*symbolic structure*of quantitative claims; the two taxonomies are complementary\. A separate line characterizes*where*fine\-tuned encoders fail: probing methodology\(Belinkov,[2022](https://arxiv.org/html/2607.16212#bib.bib50); Ribeiroet al\.,[2020](https://arxiv.org/html/2607.16212#bib.bib51); Rogerset al\.,[2020](https://arxiv.org/html/2607.16212#bib.bib52)\)and input\-level blind\-spot prediction\(Miet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib54)\)diagnose narrow failure modes that aggregate metrics hide\. We follow this tradition in §[4](https://arxiv.org/html/2607.16212#S4)with a four\-family probe suite; concurrent work on OOD generalization in NLI\(Delbari and Pilehvar,[2025](https://arxiv.org/html/2607.16212#bib.bib55)\)similarly finds that in\-distribution accuracy can mask consequential robustness gaps\.

#### Numeric and scientific claim verification\.

Verification targeted specifically at numerical and scientific claims is more recent\. Number representation in NLP has been studied broadly\(Thawaniet al\.,[2021](https://arxiv.org/html/2607.16212#bib.bib19); Mishraet al\.,[2022](https://arxiv.org/html/2607.16212#bib.bib20)\), but benchmarks dedicated to numerical verification have only emerged in the past two years: QuanTemp\(Anandet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib36)\)and QuanTemp\+\+\(Venkteshet al\.,[2025a](https://arxiv.org/html/2607.16212#bib.bib37)\)taxonomize claims into statistical, temporal, comparison, and interval categories, and the CheckThat\! 2025 Task 3\(Venkteshet al\.,[2025b](https://arxiv.org/html/2607.16212#bib.bib38)\)adopts QuanTemp as a shared task with most submissions casting veracity as three\-class NLI\(Heil and Pramov,[2025](https://arxiv.org/html/2607.16212#bib.bib39); Aniket al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib40)\)reaching macro\-F1≤0\.57\\leq 0\.57; VerifierFC\(Chungkhamet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib41)\)explores test\-time scaling for numerical claims\. SciFact\(Waddenet al\.,[2020](https://arxiv.org/html/2607.16212#bib.bib21)\), SciFact\-Open\(Wrightet al\.,[2022](https://arxiv.org/html/2607.16212#bib.bib22)\), and SciVer\(Wadden and Lo,[2021](https://arxiv.org/html/2607.16212#bib.bib42)\)remain the canonical scientific fact\-check benchmarks, but their errors are paraphrased claims rather than unit\- or magnitude\-typed perturbations\. Domain\-adapted scientific LMs continue to appear—LLaMat\(Mishraet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib23)\), HalluMat\(Vangalaet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib43)\), MedRAGChecker\(Jiet al\.,[2026](https://arxiv.org/html/2607.16212#bib.bib44)\)—but as we show in §[5](https://arxiv.org/html/2607.16212#S5), surface domain familiarity does not translate into structural reasoning over quantities\. Two distinctions separate this entire line from our setting: existing taxonomies partition claims by*topic*\(temporal vs\. statistical\), while ours partitions them by*error type*\(dimension vs\. scale vs\. relation\); and existing benchmarks do not recover*which kind*of numerical defect is present, the specific capability our taxonomy operationalizes\.

#### Symbolic verification and data augmentation\.

UnitGraph Verifiersits in the lineage of symbolic and tool\-augmented verification\. Rule\-based pipelines using quantulum\-style extractors\(The quantulum3 contributors,[2024](https://arxiv.org/html/2607.16212#bib.bib25)\)and Pint unit registries\(Grecco and others,[2024](https://arxiv.org/html/2607.16212#bib.bib26)\)have long been used for unit canonicalization in scientific text mining but are seldom benchmarked as standalone verifiers\. More recent work integrates symbolic reasoning into verification: TabVer\(Aly and Vlachos,[2024](https://arxiv.org/html/2607.16212#bib.bib45)\)extends natural logic with set\-theoretic arithmetic for tabular claims, and CoSineVerifier\(Fenget al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib46)\)adds symbolic execution for computation\-heavy STEM answers\. Concurrent work\(Seoet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib56)\)similarly identifies pitfalls in fact verifiers and explores synthetic data generation as a remedy\.

A parallel line uses data augmentation to inject knowledge into encoder training rather than at inference time\. Counterfactual data augmentation\(Zhuet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib47); Zhanget al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib53)\)generates label\-flipping rewrites of training items, typically via learned paraphrasers or LLM prompting\(Whitehouseet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib48); Chenet al\.,[2023](https://arxiv.org/html/2607.16212#bib.bib49)\)\. Our Symbolic Augmentation \(§[4](https://arxiv.org/html/2607.16212#S4)\) is in this spirit but differs in that its rewrite rules are derived deterministically from the modules of a symbolic verifier—yielding stronger label guarantees—and are*label\-preserving*rather than label\-flipping, the appropriate shape for closing canonical\-equivalence blind spots\. The framework extends to any symbolic verifier whose modules expose deterministic, label\-preserving rewrites\.

## 3Typed\-Quantity Verification

![Refer to caption](https://arxiv.org/html/2607.16212v1/x1.png)Figure 1:Overview of the typed\-quantity verification task and the Symbolic Augmentation framework\.\(left\)The task: each evidence–summary pair is assigned one of five typed labels \(Correct,Unit Error,Scale Error,Relation Error,Unsupported\), with one example per label\.\(center\)Symbolic Augmentation, our contribution: the symbolic verifierUGV—which extracts quantities, checks dimension and scale, and detects comparator and superlative patterns—is run in reverse\. Each detection rule is inverted into a label\-preserving rewrite family \(F1–F4\), which is instantiated to produce augmented training pairs that target the encoder’s canonical\-equivalence blind spot\.\(right\)At inference time the fine\-tuned encoder runs alone, with no symbolic modules; a templated repair is optional\.### 3\.1Task formulation

Given an evidence sentenceeeand a summarysspurporting to restateee’s quantitative content, the verifier assigns a labelyyfrom one of five classes \(Correct,Unit Error,Scale Error,Relation Error,Unsupported\) and optionally emits a templated repairrrthat, applied toss, yields a sentence consistent withee\. Each non\-Correctclass corresponds to a distinct symbolic axis along whichsscan deviate fromee—unit dimension, magnitude scale, relational structure, and unsupported\-claim addition—whileCorrectcovers everything else, including*surface variations that preserve the underlying physical claim*, a property central to our augmentation framework \(§[4](https://arxiv.org/html/2607.16212#S4)\)\. The taxonomy is dataset\-agnostic; we verify transfer to an external benchmark in §[5\.3](https://arxiv.org/html/2607.16212#S5.SS3.SSS0.Px2)\.

### 3\.2Class definitions

#### Correct\.

ssis a paraphrase or faithful condensation ofee\. Crucially, this includes*canonical\-equivalent rewrites*—surface forms that canonicalize to the same physical quantity through a unit registry or notational equivalence\. For example, 95∘C and 368\.15 K denote the same temperature, 0\.5 mol/L and 500 mmol/L the same concentration, and 1\.2e\-9 and 0\.0000000012 the same value\. Such rewrites are correct restatements, not errors, even though their surface forms differ from the evidence\.

#### Unit Error\.

sscontains a quantity whose unit has a*different physical dimension*from the evidence’s\. Example: “12nM12\\text\{\\,\}\\mathrm\{nM\}”→\\to“12mgml−112\\text\{\\,\}\\mathrm\{mg\}\\text\{\\,\}\{\\mathrm\{ml\}\}^\{\-1\}\.”

#### Scale Error\.

sscontains a quantity with the*same physical dimension*as the evidence’s but a numerical value differing by at least one order of magnitude \(\|log10⁡\(qs/qe\)\|≥1\|\\log\_\{10\}\(q\_\{s\}/q\_\{e\}\)\|\\geq 1\)\. One exception applies: if two surface forms canonicalize to the same physical quantity within1%1\\%, the pair is a canonical\-equivalentCorrectrestatement, not a magnitude error\. Example: “92% conversion”→\\to“9\.2% conversion\.”

#### Relation Error\.

sscontains a*relational*discrepancy: \(a\) a comparator flip \(“higher”→\\to“lower”\), \(b\) a value bound to the wrong entity, or \(c\) a within\-order numerical change altering the claim’s scope \(“after24h24\\text\{\\,\}\\mathrm\{h\}”→\\to“after48h48\\text\{\\,\}\\mathrm\{h\}”\)\.

#### Unsupported\.

ssadds a claim or framing absent fromee\. This covers three sub\-types: superlative claims \(“the highest ever reported”\), causal claims \(“the dominant factor”\), and significance claims \(“statistically significant”\)\. Numeric content may still be correct; the defect is the unsupported addition\.

### 3\.3Priority and evaluation

#### Priority\.

A summary may exhibit multiple error types simultaneously\. We resolve conflicts in descending order:Unsupported,Unit Error,Scale Error,Relation Error,Correct\.Unsupporteddominates typed errors because it is an epistemic shift independent of the numeric content; the remaining ordering reflects axis precedence \(dimension before magnitude before relation\)\. The ordering is empirically active on116116of15001500items \(7\.7%7\.7\\%\) and is detailed further in Appendix[A](https://arxiv.org/html/2607.16212#A1)\.

#### Evaluation\.

We evaluate verifiers along three axes: \(i\) classification accuracy on a frozen 1500\-item test set \(§[5](https://arxiv.org/html/2607.16212#S5)\); \(ii\) robustness on per\-family probes that apply physical\-claim\-preserving surface variations \(§[4\.3](https://arxiv.org/html/2607.16212#S4.SS3)\); and \(iii\) out\-of\-distribution transfer to an external benchmark \(§[5\.3](https://arxiv.org/html/2607.16212#S5.SS3.SSS0.Px2)\)\. For systems that emit a templated repair, we report repair accuracy as a secondary metric \(Appendix[A](https://arxiv.org/html/2607.16212#A1.SS0.SSS0.Px2)\)\.

## 4Symbolic Augmentation

### 4\.1TheUnitGraph Verifier

UnitGraph Verifier\(UGV\) is a deterministic, four\-stage pipeline that mirrors the task’s symbolic structure \(§[3\.1](https://arxiv.org/html/2607.16212#S3.SS1)\): each stage targets one error axis from §[3\.2](https://arxiv.org/html/2607.16212#S3.SS2), and a fifth implicit branch returnsCorrectwhen no stage fires\. The pipeline is CPU\-only with no language model at inference time; its only non\-standard dependencies are quantulum\(The quantulum3 contributors,[2024](https://arxiv.org/html/2607.16212#bib.bib25)\)for quantity extraction and Pint\(Grecco and others,[2024](https://arxiv.org/html/2607.16212#bib.bib26)\)for the unit registry\. The full pseudocode is in Appendix[B](https://arxiv.org/html/2607.16212#A2), and Figure[1](https://arxiv.org/html/2607.16212#S3.F1)gives an overview\.

#### Stages 1–2: extraction and dimensional comparison\.

Quantulum maps each surface span to a tuple\(v,u,𝑠𝑝𝑎𝑛\)\(v,u,\\mathit\{span\}\)of value, unit, and character range\. A hand\-curated normalizer canonicalizes unit aliases \(M→\\tomol/L;wt%→\\topercent\), scientific notation, and context\-implicit units\. Aligned quantity pairs\(qe,qs\)\(q\_\{e\},q\_\{s\}\)are then checked for dimensional inequality \(dim\(qs\)≠dim\(qe\)\\dim\(q\_\{s\}\)\\neq\\dim\(q\_\{e\}\)\), yieldingUnit Error, or for order\-of\-magnitude inequality, yieldingScale Error—subject to the1%1\\%canonical\-equivalence override of §[3\.2](https://arxiv.org/html/2607.16212#S3.SS2)\.

#### Stages 3–4: relations and unsupported triggers\.

Stage 3 curates a small inventory of comparator clusters \(e\.g\., the “high” family:higher,elevated,greater\) and tags occurrences with polarity and arguments; a matched pair is inconsistent if polarities differ \(flip\), identities are swapped, or scope shifts\. Stage 4 firesUnsupportedwhen one of a finite set of trigger phrases \(superlative, causal, significance markers; full inventory in Appendix[B](https://arxiv.org/html/2607.16212#A2.SS0.SSS0.Px4)\) appears inssbut is absent fromeeand its surrounding context\.UGVreachesm​a​c​r​o−F​1=0\.537macro\-F\\textsubscript\{1\}=0\.537on the 1500\-item benchmark \(§[5\.2](https://arxiv.org/html/2607.16212#S5.SS2)\), exceeding every off\-the\-shelf neural fact\-checker we test but falling well below a fine\-tuned encoder\.

### 4\.2The augmentation framework

The fourUGVstages share a structural property we exploit: each is a*deterministic, label\-preserving rewrite rule*over surface forms\. Pint’s unit registry, for instance, defines when two surface forms \(95∘C and 368\.15 K\) denote the same physical quantity\. Run in reverse, this rule generates training data: given a goldCorrectitem, rewriting its quantity to a canonical\-equivalent form yields a newCorrectitem whose label is guaranteed correct, because the rule is label\-preserving by construction\.

We call the resulting frameworkSymbolic Augmentation\. It applies four augmentation families, each derived from oneUGVmodule:

F1: Canonical equivalence\.Convert quantities to physically\-equivalent units across systems \(95°C95\\text\{\\,\}\\mathrm\{\\SIUnitSymbolCelsius\}↔\\leftrightarrow368\.15 K; mg↔\\leftrightarrowg; mM↔\\leftrightarrowM;%\\%↔\\leftrightarrowfraction\)\.

F2: Comparator synonyms\.Swap comparator words on botheeandsssymmetrically \(“higher”↔\\leftrightarrow“elevated”; “lower”↔\\leftrightarrow“reduced”\)\.

F3: Unit\-name paraphrase\.Substitute unit symbols with their word forms in botheeandss\(M↔\\leftrightarrow“molar”;K↔\\leftrightarrow“kelvin”\)\.

F4: Scientific notation\.Convert numbers between exponential and decimal forms on the summary side \(1\.2e\-9↔\\leftrightarrow0\.0000000012\)\.

Augmentation is applied to goldCorrectitems only, so the augmented training set retains the original gold\-label distribution; F1 and F4 rewrite only the summary side, while F2 and F3 rewrite botheeandsssymmetrically to preserve the gold relation\. The framework is extensible: any symbolic verifier whose modules expose deterministic, label\-preserving rewrites can be added as a family\.

### 4\.3Per\-family probes

To diagnose where the encoder fails and to measure augmentation effect, we construct a probe set for each family\. A probe applies the family’s rewrite to the test set’sCorrectitems and asks whether the model still labels themCorrect; probe accuracy is the fraction it classifies correctly\. The four probes contain 181, 22, 30, and 25 variants for F1–F4 respectively, derived from the 333Correctitems in the test set\. Rule definitions and probe generation share code \(Appendix[C](https://arxiv.org/html/2607.16212#A3)\), so a probe and its corresponding augmentation test exactly the same transformation\. Probes thus serve a dual role: a diagnostic for where fine\-tuned encoders fail \(§[5\.2](https://arxiv.org/html/2607.16212#S5.SS2)\), and a benchmark for whether augmentation localizes its effect to the targeted failure \(§[5\.3](https://arxiv.org/html/2607.16212#S5.SS3)\)\.

Table 1:Main results on the 1500\-item benchmark: macro\-F1F\_\{1\}and per\-classF1F\_\{1\}over the five error types\.

## 5Experiments

### 5\.1Setup

#### Dataset\.

We evaluate on the15001500\-item typed\-quantity benchmark of §[3](https://arxiv.org/html/2607.16212#S3)\. Items are LLM\-rewritten from real PMC \(biomedical, materials science\) and arXiv \(computer science\) source sentences, balanced across the five classes, and dual\-annotated by two independent LLM annotators with third\-party adjudication \(Krippendorff’sα=0\.882\\alpha=0\.882\); a5050\-item human spot\-check agrees with the gold labels on94%94\\%of items\. Appendix[D](https://arxiv.org/html/2607.16212#A4)details the construction pipeline, including source sampling, rewrite prompting, and circularity controls\.

#### Baselines\.

We compare five groups of systems\.*\(1\) Trivial baselines:*three systems that establish the metric floor—a constantCorrectpredictor, a string\-overlap exact\-match check, and*unit\-substring*, which flags aUnit Errorwhenever a value\-and\-unit token in the summary is absent from the evidence\.*\(2\) Rule\-based:*quantulum\(The quantulum3 contributors,[2024](https://arxiv.org/html/2607.16212#bib.bib25)\)paired with the Pint unit registry\(Grecco and others,[2024](https://arxiv.org/html/2607.16212#bib.bib26)\)\.*\(3\) Our symbolic verifier:*UGV\(§[4\.1](https://arxiv.org/html/2607.16212#S4.SS1)\)\.*\(4\) Off\-the\-shelf neural fact\-checkers:*MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib9)\), Bespoke\-MiniCheck\-7B\(Bespoke Labs,[2024](https://arxiv.org/html/2607.16212#bib.bib10)\), FactCG\(Leiet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib13)\), a zero\-shot ModernBERT\-NLI\(Sileo,[2024](https://arxiv.org/html/2607.16212#bib.bib15)\), Granite Guardian\(Padhiet al\.,[2024](https://arxiv.org/html/2607.16212#bib.bib17)\), and VeriFastScore\(Rajendhranet al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib16)\)\.*\(5\) Closed\-frontier LLMs:*Claude Opus 4\.7, GPT\-5\.5, and DeepSeek V4, evaluated zero\-shot\. Versions, checkpoints, and invocation settings for every system are in Appendix[E](https://arxiv.org/html/2607.16212#A5)\.

#### Metrics\.

Our primary metric is the macro\-averagedF1F\_\{1\}score over the five classes\. For the augmentation experiments we additionally report*probe accuracy*: the fraction of a family’s probe items \(§[4\.3](https://arxiv.org/html/2607.16212#S4.SS3)\) that the model still classifies correctly\. For the out\-of\-distribution experiment we report binary macro\-F1F\_\{1\}on SciFact\-Open\. Exact metric definitions are given in Appendix[F](https://arxiv.org/html/2607.16212#A6)\.

#### Fine\-tuning and augmentation\.

Our fine\-tuned model is ModernBERT\-NLI\(Warneret al\.,[2025](https://arxiv.org/html/2607.16212#bib.bib14); Sileo,[2024](https://arxiv.org/html/2607.16212#bib.bib15)\)fine\-tuned on the typed\-quantity task and evaluated with 5\-fold cross\-validation\. Symbolic Augmentation is applied to the goldCorrecttraining items of each fold as described in §[4\.2](https://arxiv.org/html/2607.16212#S4.SS2)\. Training hyperparameters, hardware, and augmentation volumes are reported in Appendix[G](https://arxiv.org/html/2607.16212#A7)\.

### 5\.2The encoder’s blind spot

Table[1](https://arxiv.org/html/2607.16212#S4.T1)reports main results, and the picture is sharp\. No off\-the\-shelf neural system exceedsm​a​c​r​o−F​1=0\.295macro\-F\\textsubscript\{1\}=0\.295, and the per\-class columns show why:*every off\-the\-shelf neural system scores exactly0\.0000\.000on*Scale Error\. TheirCorrectscores vary widely \(0\.000\.00–0\.710\.71\)—some detect that a quantity is being restated, but none detect that a magnitude is wrong\. The rule\-based quantulum \+ Pint baseline \(0\.3850\.385\) andUGV\(0\.5370\.537\) both outperform every neural system, because explicit dimensional arithmetic is exactly whatScale ErrorandUnit Errorrequire\.

Fine\-tuning changes the picture entirely: ModernBERT fine\-tuned on the typed\-quantity task reachesm​a​c​r​o−F​1=0\.899macro\-F\\textsubscript\{1\}=0\.899\(5\-fold CV\), within0\.070\.07of the strongest closed\-frontier LLM \(Claude Opus 4\.7,0\.9730\.973\) and ahead of DeepSeek V4\-Flash\. A modest encoder, given a few hundred labeled examples per class, is competitive with frontier systems costing orders of magnitude more per query\. Yet one per\-class score stands out:Correctis the encoder’s*weakest*class atF1=0\.855F\_\{1\}=0\.855, well below itsUnit Error\(0\.9230\.923\) andUnsupported\(0\.9300\.930\) scores\. A model this strong should not findCorrecthardest—unless someCorrectitems share a structure it cannot handle\.

The four probes of §[4\.3](https://arxiv.org/html/2607.16212#S4.SS3)locate that structure\. The*none*row of Table[3](https://arxiv.org/html/2607.16212#S5.T3)reports the canonical encoder’s probe accuracies: on three of four—comparator synonyms \(100%100\\%\), unit\-name paraphrase \(96\.7%96\.7\\%\), scientific notation \(68%68\\%\)—it is robust or nearly so\. On canonical\-equivalence rewrites it scores only36\.5%36\.5\\%: shown a goldCorrectsummary that states a quantity in a physically equivalent but textually different form \(95°C95\\text\{\\,\}\\mathrm\{\\SIUnitSymbolCelsius\}as368\.15K368\.15\\text\{\\,\}\\mathrm\{K\}\), the encoder calls it an error nearly two\-thirds of the time\. This is the structure hidden in the0\.8550\.855Correctscore: canonical\-equivalent items are a minority ofCorrect—large enough to depress itsF1F\_\{1\}, too rare for the encoder to learn in\-distribution\. The failure is not noise; it concentrates on the one transformation that requires canonicalizing a numerical surface rather than matching lexical or syntactic patterns\.

### 5\.3Symbolic Augmentation closes the gap

Table 2:Headline results, 5\-fold pooled\. The F1\-probe column is canonical\-equivalence accuracy \(%\);UGVand a closed\-frontier LLM are shown for reference\.Table 3:Per\-family probe accuracy \(%\), single fold\. Rows are training configurations, columns the four probes; training with a family raises its own probe \(diagonal, bold\)\.Training with the F1 \(canonical\-equivalence\) family raises F1\-probe accuracy from36\.5%36\.5\\%to98\.2%98\.2\\%\(5\-fold pooled; Table[2](https://arxiv.org/html/2607.16212#S5.T2)\)—a 62\-point gain that closes essentially all of the blind spot—while macro\-F1F\_\{1\}rises slightly,0\.899→0\.9020\.899\\to 0\.902\. The robustness gain comes free, not bought with in\-distribution accuracy\. As Table[3](https://arxiv.org/html/2607.16212#S5.T3)shows, the F4 \(scientific\-notation\) family similarly lifts its own probe from68%68\\%to96%96\\%, while F2 and F3 yield no gain applied alone—their probes already start near saturation—so we use them only within the combined set\. Figure[2](https://arxiv.org/html/2607.16212#S5.F2)contrasts the canonical and augmented encoders across all four probes; training with all four families at once raises every probe to at least96%96\\%while holding macro\-F1F\_\{1\}at0\.8930\.893, statistically indistinguishable from baseline\. We therefore carry two variants forward: F1\-augmentation as the strongest single intervention, all\-four as the broadest\. We also observe cross\-family transfer—F1 alone lifts the F4 probe from68%68\\%to96%96\\%, so exposure to one numerical\-surface rewrite generalizes to another\. The blind spot is not unique to small encoders: DeepSeek V4\-Flash, a closed\-frontier model, scores83\.4%83\.4\\%on the same probe—far above the canonical encoder but short of robust\. F1\-augmented fine\-tuning thus*matches*V4\-Flash on overall macro\-F1F\_\{1\}\(0\.9020\.902vs\.0\.9080\.908\) and*exceeds*it on canonical\-equivalence robustness by nearly 15 points, while running locally at no per\-query cost\.

![Refer to caption](https://arxiv.org/html/2607.16212v1/x2.png)Figure 2:Per\-family probe accuracy before and after Symbolic Augmentation\. The canonical encoder collapses to36\.5%36\.5\\%on canonical\-equivalence rewrites \(F1\); augmentation closes this gap while leaving the other probes near\-saturated\.#### Mechanism\.

Figure[3](https://arxiv.org/html/2607.16212#S5.F3)traces the F1 probe at the prediction level\. All181181probe variants are goldCorrect, yet the canonical encoder labels only6666of themCorrect\. The115115misclassifications are not spread evenly:9999of them—86%86\\%—land inScale Error\. The encoder, shown a canonical\-equivalent rewrite such as95°C95\\text\{\\,\}\\mathrm\{\\SIUnitSymbolCelsius\}→\\to368\.15K368\.15\\text\{\\,\}\\mathrm\{K\}, most often reads it as a magnitude that has been altered\. A further1515land inUnit Error; almost none reachRelation Error, confirming that the failure is a*numerical\-surface*confusion rather than a relational one\. Symbolic Augmentation repairs114114of the115115\(99\.1%99\.1\\%\): the rewrittenCorrectexamples teach the encoder to read the altered surface as a restatement\. A single error persists—a milligram\-to\-gram rewrite the augmented model still scoresRelation Error—and one previously correct item is newly misread, for a net movement of\+113\+113items intoCorrect\.

![Refer to caption](https://arxiv.org/html/2607.16212v1/x3.png)Figure 3:Prediction flow on the F1 probe, all181181variants goldCorrect\. The canonical encoder \(left\) sends9999of its115115errors toScale Error; Symbolic Augmentation \(right\) recovers114114of the115115\.
#### Out\-of\-distribution transfer\.

To test whether the effect is specific to our benchmark, we evaluate on SciFact\-Open\(Wrightet al\.,[2022](https://arxiv.org/html/2607.16212#bib.bib22)\), an external scientific claim\-verification benchmark, cast as binary verification\. Symbolic Augmentation raises binary macro\-F1F\_\{1\}from0\.7910\.791to0\.8280\.828\(\+0\.037\+0\.037\) without any SciFact\-Open training data, extending a monotone ladder that runs from trivial baselines through rule\-based systems andUGVto the fine\-tuned encoders\. The improvement confirms that augmentation instills a transferable capability—canonicalizing numerical surfaces—rather than a benchmark\-specific pattern\. The full ladder and per\-class results are in Appendix[I](https://arxiv.org/html/2607.16212#A9)\.

### 5\.4Where symbolic information helps

Integration strategymacro\-F1F\_\{1\}Δ\\DeltaCanonical fine\-tuning0\.899—Inference\-time integrationLogit ensemble0\.895−0\.004\-0\.004Agreement filter0\.888−0\.011\-0\.011Feature\-level integrationUGVoutput as features0\.879−0\.020\-0\.020Silver\-label training withUGV12001200silver items0\.622−0\.277\-0\.27732983298silver items0\.518−0\.381\-0\.381Training\-data integration \(ours\)Symbolic Augmentation0\.902\+0\.003\\mathbf\{\+0\.003\}Table 4:Five integration strategies, grouped by where the verifier enters the pipeline, against canonical fine\-tuning\. Only training\-data integration \(Symbolic Augmentation\) improves macro\-F1F\_\{1\}; see Appendix[H](https://arxiv.org/html/2607.16212#A8)\.Symbolic Augmentation integrates the verifier at the training\-data layer\. Four other integration points are equally plausible a priori, yet Table[4](https://arxiv.org/html/2607.16212#S5.T4)shows none works\. At inference time, a logit ensemble withUGV\(−0\.004\-0\.004\) is within noise, and an agreement filter—take the joint prediction whenUGVand the encoder agree, else the encoder—reduces mathematically to the encoder alone\. At the feature level, concatenating the verifier’s output as a 12\-dimensional auxiliary vector—a one\-hotUGVprediction plus continuous module\-trace signals—costs0\.0200\.020macro\-F1F\_\{1\}: the encoder already extracts what the features encode\. Using the verifier as a silver\-label teacher is actively harmful and worsens with scale \(0\.6220\.622at12001200items,0\.5180\.518at32983298\), because the verifier’s labels are too noisy—it agrees with the gold label on only53\.1%53\.1\\%of items—and inject error faster than volume compensates\. Only training\-time augmentation both improves macro\-F1F\_\{1\}and delivers the 62\-point probe gain, because it is the one integration point that inherits the verifier’s deterministic label guarantee without inheriting its accuracy ceiling\. A full per\-method analysis is in Appendix[H](https://arxiv.org/html/2607.16212#A8)\.

## 6Conclusion

We framed the verification of numbers and units in LLM\-generated scientific text as*typed*classification and released a five\-class taxonomy and a 1500\-item benchmark to study it\. A fine\-tuned ModernBERT encoder reachesm​a​c​r​o−F​1=0\.899macro\-F\\textsubscript\{1\}=0\.899on this benchmark, far beyond any off\-the\-shelf neural fact\-checker, yet four probes expose a sharp blind spot: it scores only36\.5%36\.5\\%on canonical\-equivalent rewrites of physically equivalent quantities\. Our main contribution,Symbolic Augmentation, closes this gap by transferring the modules of a symbolic verifier into the encoder as training\-time data augmentation, raising canonical\-equivalence robustness to98\.2%98\.2\\%while slightly improving in\-distribution accuracy, matching a closed\-frontier LLM at no inference cost, and transferring to an external benchmark\. Of five ways to combine a symbolic verifier with a learned model, only training\-time augmentation delivers a transferable benefit\. We hope the benchmark and framework support further work on bringing symbolic guarantees into learned verifiers\.

## Limitations

Three scope conditions bound our results\. First, the benchmark is built from English\-language PMC and arXiv sources; the taxonomy is defined over language\-independent symbolic properties and is deliberately dataset\-agnostic, and we verify cross\-dataset transfer on SciFact\-Open \(§[5\.3](https://arxiv.org/html/2607.16212#S5.SS3)\), but other languages remain future work\. Second, Symbolic Augmentation requires a symbolic verifier whose modules implement deterministic, label\-preserving rewrites; the framework is general with respect to such verifiers, though constructing them for new domains needs additional engineering\. Third, gold labels are LLM\-produced; we mitigate the self\-labeling concern with a disjoint probe generator and a 50\-item human spot\-check that agrees with the gold labels on94%94\\%of items\.

## References

- R\. Aly and A\. Vlachos \(2024\)Tabver: tabular fact verification with natural logic\.Transactions of the Association for Computational Linguistics12,pp\. 1648–1671\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Anand, A\. Anand, V\. Setty,et al\.\(2024\)Quantemp: a real\-world open\-domain benchmark for fact\-checking numerical claims\.arXiv preprint arXiv:2403\.17169\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- A\. S\. Anik, M\. F\. K\. Chowdhury, A\. Wyckoff, and S\. R\. Choudhury \(2025\)ClaimIQ at checkthat\! 2025: comparing prompted and fine\-tuned language models for verifying numerical claims\.arXiv preprint arXiv:2509\.11492\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Asai, J\. He, R\. Shao, W\. Shi, A\. Singh, J\. C\. Chang, K\. Lo, L\. Soldaini, S\. Feldman, M\. D’arcy,et al\.\(2024\)Openscholar: synthesizing scientific literature with retrieval\-augmented lms\.arXiv preprint arXiv:2411\.14199\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- Y\. Bang, Z\. Ji, A\. Schelten, A\. Hartshorn, T\. Fowler, C\. Zhang, N\. Cancedda, and P\. Fung \(2025\)Hallulens: llm hallucination benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24128–24156\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Belinkov \(2022\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- Bespoke Labs \(2024\)Bespoke\-MiniCheck\-7B\.Note:[https://huggingface\.co/bespokelabs/Bespoke\-MiniCheck\-7B](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B)Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. Gomes \(2023\)Autonomous chemical research with large language models\.Nature624\(7992\),pp\. 570–578\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- J\. Chen, D\. Tam, C\. Raffel, M\. Bansal, and D\. Yang \(2023\)An empirical survey of data augmentation for limited data learning in nlp\.Transactions of the Association for Computational Linguistics11,pp\. 191–211\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p2.1)\.
- I\. Chern, S\. Chern, S\. Chen, W\. Yuan, K\. Feng, C\. Zhou, J\. He, G\. Neubig, P\. Liu,et al\.\(2023\)FacTool: factuality detection in generative ai–a tool augmented framework for multi\-task and multi\-domain scenarios\.arXiv preprint arXiv:2307\.13528\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Chungkham, V\. Viswanathan, V\. Setty, and A\. Anand \(2025\)Think right, not more: test\-time scaling for numerical claim verification\.InEmpirical Methods in Natural Language Processing \(EMNLP\) 2025,pp\. 24345–24363\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Delbari and M\. T\. Pilehvar \(2025\)Beyond accuracy: revisiting out\-of\-distribution generalization in nli models\.InProceedings of the 29th Conference on Computational Natural Language Learning,pp\. 557–570\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Feng, Z\. An, Y\. Wen, R\. Le, Y\. Jia, C\. Yang, Z\. Chen, L\. Chen, S\. Gao, S\. Shang,et al\.\(2025\)CoSineVerifier: tool\-augmented answer verification for computation\-oriented scientific questions\.arXiv preprint arXiv:2512\.01224\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p1.1)\.
- J\. M\. Ford \(2004\)Content analysis: an introduction to its methodology\.Personnel psychology57\(4\),pp\. 1110\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p3.4)\.
- H\. E\. Greccoet al\.\(2024\)Pint: a Python units library\.Note:[https://github\.com/hgrecco/pint](https://github.com/hgrecco/pint)Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2607.16212#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- M\. Heil and A\. Pramov \(2025\)DS@ gt at checkthat\! 2025: evaluating context and tokenization strategies for numerical fact verification\.arXiv preprint arXiv:2507\.06195\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Ji, M\. G\. Kwak, H\. Zhang, X\. Wu, C\. Li, and Y\. Wang \(2026\)MedRAGChecker: claim\-level verification for biomedical retrieval\-augmented generation\.arXiv preprint arXiv:2601\.06519\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- D\. Lei, Y\. Li, S\. Li, M\. Hu, R\. Xu, K\. Archer, M\. Wang, E\. Ching, and A\. Deng \(2025\)FactCG: enhancing fact checkers with graph\-based multi\-hop data\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5002–5020\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- W\. Liang, Z\. Izzo, Y\. Zhang, H\. Lepp, H\. Cao, X\. Zhao, L\. Chen, H\. Ye, S\. Liu, Z\. Huang,et al\.\(2024\)Monitoring ai\-modified content at scale: a case study on the impact of chatgpt on ai conference peer reviews\.arXiv preprint arXiv:2403\.07183\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. Schwaller \(2024\)Augmenting large language models with chemistry tools\.Nature machine intelligence6\(5\),pp\. 525–535\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)Selfcheckgpt: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 9004–9017\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Mi, A\. Villavicencio, and N\. S\. Moosavi \(2025\)From input perception to predictive insight: modeling model blind spots before they become errors\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 34316–34329\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Mishra, A\. Mitra, N\. Varshney, B\. Sachdeva, P\. Clark, C\. Baral, and A\. Kalyan \(2022\)NumGLUE: a suite of fundamental yet challenging mathematical reasoning tasks\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3505–3523\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- V\. Mishra, S\. Singh, D\. Ahlawat, M\. Zaki, V\. Bihani, H\. S\. Grover, B\. Mishra, S\. Miret, N\. Krishnan,et al\.\(2024\)Foundational large language models for materials research\.arXiv preprint arXiv:2412\.09560\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Padhi, M\. Nagireddy, G\. Cornacchia, S\. Chaudhury, T\. Pedapati, P\. Dognin, K\. Murugesan, E\. Miehling, M\. S\. Cooper, K\. Fraser,et al\.\(2024\)Granite guardian\.arXiv preprint arXiv:2412\.07724\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- R\. Rajendhran, A\. Zadeh, M\. Sarte, C\. Li, and M\. Iyyer \(2025\)VERIFASTSCORE: speeding up long\-form factuality evaluation\.arXiv preprint arXiv:2505\.16973\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh \(2020\)Beyond accuracy: behavioral testing of nlp models with checklist\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 4902–4912\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Rogers, O\. Kovaleva, and A\. Rumshisky \(2020\)A primer in bertology: what we know about how bert works\.Transactions of the association for computational linguistics8,pp\. 842–866\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Seo, S\. Han, J\. Jung, B\. Newman, S\. Lim, S\. Lee, X\. Lu, Y\. Choi, and Y\. Yu \(2025\)Verifying the verifiers: unveiling pitfalls and potentials in fact verifiers\.arXiv preprint arXiv:2506\.13342\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Sileo \(2024\)Tasksource: a large collection of NLP tasks with a structured dataset preprocessing framework\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 15655–15684\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1361)Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p4.6),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px4.p1.1)\.
- A\. Singh, J\. C\. Chang, D\. Haddad, A\. Naik, J\. D\. Hwang, R\. Kinney, D\. S\. Weld, D\. Downey, and S\. Feldman \(2025\)Ai2 scholar qa: organized literature synthesis with attribution\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),pp\. 513–523\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- H\. Song, H\. Su, I\. Shalyminov, J\. Cai, and S\. Mansour \(2024\)FineSurE: fine\-grained summarization evaluation using llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 906–922\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Tang, P\. Laban, and G\. Durrett \(2024\)Minicheck: efficient fact\-checking of llms on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8818–8847\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- A\. Thawani, J\. Pujara, F\. Ilievski, and P\. Szekely \(2021\)Representing numbers in nlp: a survey and a vision\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 644–656\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- The quantulum3 contributors \(2024\)Quantulum3: a Python library for quantity extraction from unstructured text\.Note:[https://github\.com/nielstron/quantulum3](https://github.com/nielstron/quantulum3)Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2607.16212#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px2.p1.1)\.
- B\. P\. Vangala, S\. Mahmud, P\. Neupane, J\. Selvaraj, and J\. Cheng \(2025\)HalluMat: detecting hallucinations in llm\-generated materials science content through multi\-stage verification\.arXiv preprint arXiv:2512\.22396\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Vázquez, T\. Mickus, E\. Zosa, T\. Vahtola, J\. Tiedemann, A\. Sinha, V\. Segonne, F\. Sánchez\-Vega, A\. Raganato, J\. Libovickỳ,et al\.\(2025\)SemEval\-2025 task 3: mu\-shroom, the multilingual shared\-task on hallucinations and related observable overgeneration mistakes\.InProceedings of the 19th International Workshop on Semantic Evaluation \(SemEval\-2025\),pp\. 2472–2497\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- V\. Venktesh, D\. Prabhu, and A\. Anand \(2025a\)A benchmark for open\-domain numerical fact\-checking enhanced by claim decomposition\.arXiv preprint arXiv:2510\.22055\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- V\. Venktesh, V\. Setty, A\. Anand, M\. Hasanain, B\. Bendou, H\. Bouamor, F\. Alam, G\. Iturra\-Bocaz, and P\. Galuščáková \(2025b\)Overview of the clef\-2025 checkthat\! lab task 3 on fact\-checking numerical claims\.Working Notes of CLEF\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. Hajishirzi \(2020\)Fact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7534–7550\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Wadden and K\. Lo \(2021\)Overview and insights from the sciver shared task on scientific claim verification\.InProceedings of the Second Workshop on Scholarly Document Processing,pp\. 124–129\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Wan, K\. Sinha, S\. Iyer, A\. Celikyilmaz, M\. Bansal, and R\. Pasunuru \(2024\)Acueval: fine\-grained hallucination evaluation and correction for abstractive summarization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 10036–10056\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen,et al\.\(2025\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2526–2547\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p4.6),[§5\.1](https://arxiv.org/html/2607.16212#S5.SS1.SSS0.Px4.p1.1)\.
- C\. Whitehouse, M\. Choudhury, and A\. F\. Aji \(2023\)LLM\-powered data augmentation for enhanced cross\-lingual performance\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 671–686\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p2.1)\.
- D\. Wright, D\. Wadden, K\. Lo, B\. Kuehl, A\. Cohan, I\. Augenstein, and L\. L\. Wang \(2022\)Generating scientific claims for zero\-shot scientific fact checking\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2448–2460\.Cited by:[Appendix I](https://arxiv.org/html/2607.16212#A9.SS0.SSS0.Px1.p1.4),[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2607.16212#S5.SS3.SSS0.Px2.p1.4)\.
- L\. Zhang, X\. Zhang, L\. Hu, D\. Song, and L\. Nie \(2025\)Dually self\-improved counterfactual data augmentation using large language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5216–5227\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p2.1)\.
- Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen,et al\.\(2023\)Siren’s song in the ai ocean: a survey on hallucination in large language models\. arxiv 2023\.arXiv preprint arXiv:2309\.01219\.Cited by:[§1](https://arxiv.org/html/2607.16212#S1.p1.1)\.
- Y\. Zhu, J\. Si, Y\. Zhao, H\. Zhu, D\. Zhou, and Y\. He \(2023\)Explain, edit, generate: rationale\-sensitive counterfactual data augmentation for multi\-hop fact verification\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 13377–13392\.Cited by:[§2](https://arxiv.org/html/2607.16212#S2.SS0.SSS0.Px3.p2.1)\.

## Appendix ATask Details

#### Priority resolution\.

A single rewritten summary can exhibit more than one error type at once—for instance, it may both mis\-state a magnitude and add an unsupported superlative\. In such cases the gold label is the single highest\-priority error present, resolved by the strict orderingUnsupported≻\\succUnit Error≻\\succScale Error≻\\succRelation Error≻\\succCorrect\. The ordering reflects two principles\. First,Unsupportedranks above every typed numeric error because it marks an*epistemic*shift independent of the numeric content: an unsupported causal or superlative claim misleads the reader whether or not its quantities are correct\. Second, among the three typed numeric errors the ordering follows*axis precedence*: a dimensional mismatch \(Unit Error\) is more fundamental than a magnitude mismatch \(Scale Error\), which is in turn more fundamental than a relational mismatch \(Relation Error\)\. The ordering is empirically active on116116of the15001500benchmark items \(7\.7%7\.7\\%\): in each, the rewriter was prompted for a typed numeric error but additionally introduced a superlative, causal, or significance phrase, and the annotators relabeled the itemUnsupported\.

#### Repair\.

Beyond assigning a label, a verifier may optionally emit a*templated repair*rr: a minimal edit that, applied to the summary, would render itCorrectwith respect to the evidence\. The repair template depends on the error type:

- •ForUnit ErrorandScale Error, the repair substitutes the evidence’s canonical quantity for the summary’s incorrect one\.
- •ForRelation Error, the repair restores the evidence’s original comparator direction or entity–value binding\.
- •ForUnsupported, the repair deletes the unsupported span\.

Repair is a secondary capability, not a headline metric: the systems we focus on—the fine\-tuned encoder and its augmented variants—are classifiers and emit no repair\.UGVproduces a repair as a byproduct of its symbolic trace, since each firing stage already localizes the offending span; we reportUGV’s repair accuracy only for completeness\.

## Appendix BTheUnitGraph VerifierPipeline

UnitGraph Verifier\(UGV\) is a deterministic, four\-stage verification pipeline\. Algorithm[1](https://arxiv.org/html/2607.16212#alg1)states it in full; this section describes each stage and the symbolic checks it performs\.

Algorithm 1UGVverification pipeline1:evidence

ee, summary

ss
2:label

y∈\{y\\in\\\{Correct,Unit Error,Scale Error,Relation Error,Unsupported

\}\\\}
3:

T←Triggers​\(s\)∖Triggers​\(e\)T\\leftarrow\\textsc\{Triggers\}\(s\)\\setminus\\textsc\{Triggers\}\(e\)
4:if

T≠∅T\\neq\\emptysetthenreturnUnsupported⊳\\trianglerightStage 4

5:endif

6:

Qe,Qs←Extract​\(e\),Extract​\(s\)Q\_\{e\},Q\_\{s\}\\leftarrow\\textsc\{Extract\}\(e\),\\textsc\{Extract\}\(s\)
7:

Qe,Qs←Normalize​\(Qe\),Normalize​\(Qs\)Q\_\{e\},Q\_\{s\}\\leftarrow\\textsc\{Normalize\}\(Q\_\{e\}\),\\textsc\{Normalize\}\(Q\_\{s\}\)
8:for

\(qe,qs\)\(q\_\{e\},q\_\{s\}\)in

Align​\(Qe,Qs\)\\textsc\{Align\}\(Q\_\{e\},Q\_\{s\}\)do

9:if

dim\(qs\)≠dim\(qe\)\\dim\(q\_\{s\}\)\\neq\\dim\(q\_\{e\}\)thenreturnUnit Error⊳\\trianglerightStage 2

10:endif

11:if

\|log10⁡\(qs/qe\)\|≥1\|\\log\_\{10\}\(q\_\{s\}/q\_\{e\}\)\|\\geq 1thenreturnScale Error⊳\\trianglerightStage 2

12:endif

13:endfor

14:if

ComparatorFlip​\(e,s\)\\textsc\{ComparatorFlip\}\(e,s\)or

EntitySwap​\(e,s\)\\textsc\{EntitySwap\}\(e,s\)thenreturnRelation Error⊳\\trianglerightStage 3

15:endif

16:returnCorrect

#### Stage 1: extraction and normalization\.

The pipeline first extracts every quantity mention from both the evidence and the summary using quantulum, which returns, for each mention, a tuple of numeric value, unit, and character span\. A hand\-curated normalizer then canonicalizes these tuples: it maps unit aliases to a single surface form \(for exampleMtomol/Landwt%to percent\), expands scientific notation to a plain decimal, and resolves context\-implicit units\. Normalization ensures that two physically equivalent quantities written differently are compared on equal footing in the later stages\.

#### Stage 2: dimensional and magnitude checks\.

After normalization, each summary quantityqsq\_\{s\}is aligned with its corresponding evidence quantityqeq\_\{e\}by matching entity head and modified property\. For each aligned pair,UGVapplies two checks\. The*dimensional check*compares physical dimensions: if

dim\(qs\)≠dim\(qe\),\\dim\(q\_\{s\}\)\\neq\\dim\(q\_\{e\}\),the pair is a dimensional mismatch and the pipeline returnsUnit Error\. Otherwise the*magnitude check*compares canonicalized values on a logarithmic scale: if

\|log10⁡\(qs/qe\)\|≥1,\\bigl\|\\log\_\{10\}\(q\_\{s\}/q\_\{e\}\)\\bigr\|\\geq 1,the two values differ by at least one order of magnitude and the pipeline returnsScale Error\. The magnitude check is suppressed when the two surface forms canonicalize to the same physical quantity within1%1\\%, which prevents a canonical\-equivalent restatement from being misreported as a magnitude error\.

#### Stage 3: relational checks\.

If no quantity pair triggers a dimensional or magnitude error,UGVchecks relational structure\. TheComparatorFliptest detects a reversed comparison: it tags comparator words in both sentences with a polarity drawn from curated clusters \(the “high” cluster contains*higher*,*elevated*,*greater*, and so on; the “low” cluster its antonyms\) and fires when matched comparators carry opposite polarity\. TheEntitySwaptest detects a quantity bound to the wrong entity\. If either test fires, the pipeline returnsRelation Error\.

#### Stage 4: unsupported\-claim triggers\.

Although highest in priority and therefore executed first, this stage checks whether the summary introduces a claim absent from the evidence\.UGVmaintains an inventory of trigger phrases in three groups:

Superlativehighest, lowest, largest, smallest, best, worst, unprecedented, record\.

Causalcauses, leads to, results in, due to, responsible for, drives, dominant factor\.

Significancestatistically significant, significant improvement, significantly,p<0\.05p<0\.05, demonstrates\.

The pipeline computes the set of trigger phrases occurring in the summary but not in the evidence; if this set is non\-empty, it returnsUnsupported\. Matching is case\-insensitive and operates over lemmatized tokens, so that inflectional variants \(*cause*,*causes*,*caused*\) are treated alike\. The five significance\-group phrases are reused, as the only unsupported triggers, by the unit\-substring and quantulum baselines described in Appendix[E](https://arxiv.org/html/2607.16212#A5)\.

## Appendix CProbe Construction

Each augmentation familyFiF\_\{i\}is defined by a set of deterministic, label\-preserving rewrite rules over quantity surfaces\. The per\-family*probe*is built directly from these rules, so that a probe and its corresponding training augmentation exercise exactly the same transformation—they call the same rewrite code\.

Probe construction proceeds as follows\. We take the333333test\-set items whose gold label isCorrect\. For each item, we apply the family’s rewrite rules to the summary, producing one or more*variants*in which a quantity has been re\-expressed in a physically equivalent but textually different form\. Because every rewrite rule is label\-preserving by construction, each variant still has gold labelCorrect\. The probe for familyFiF\_\{i\}is the resulting set of variantsViV\_\{i\}, and the model is scored on whether it continues to assign themCorrect\.

The four probes contain

\|V1\|=181,\|V2\|=22,\|V3\|=30,\|V4\|=25\|V\_\{1\}\|=181,\\quad\|V\_\{2\}\|=22,\\quad\|V\_\{3\}\|=30,\\quad\|V\_\{4\}\|=25variants respectively\. The counts differ because a family can only rewrite an item that contains an applicable quantity: F4 \(scientific\-notation rewriting\), for example, applies only to items whose summary contains a number that can be re\-expressed in scientific notation, and comparatively fewCorrectitems qualify\.

## Appendix DDataset Construction

This section details the four\-step construction pipeline: source\-sentence sampling, LLM rewriting, two\-annotator labeling with adjudication, and a human spot\-check\. Figure[4](https://arxiv.org/html/2607.16212#A4.F4)summarizes the labeling flow\.

#### Source sampling\.

Evidence sentences are drawn from three scientific domains, each contributing roughly one third of the benchmark:

- •PMC biomedical\(484484sentences\)\. We retrieve approximately120120open\-access articles through the NCBI E\-utilities API, using a query targeting concentration and pharmacokinetic terminology \(for examplemg/mL,µM,IC50,*pharmacokinetic*\), then extract sentences at the sentence level\.
- •PMC materials science\(507507sentences\)\. A parallel retrieval of roughly120120open\-access articles uses a query targeting battery and photovoltaic terminology \(for example*perovskite*,*cathode*,mAh/g,*photocurrent*\)\.
- •arXiv computer science\(509509sentences\)\. Sentences are retrieved through the Semantic Scholar and arXiv APIs with queries targeting evaluation and benchmarking terminology\.

A candidate sentence is retained only if quantulum can extract at least one quantity from it, ensuring every evidence sentence has quantitative content\. Evidence sentences are further restricted to at most three quantity tuples, which keeps quantity alignment unambiguous in later processing\. The15001500benchmark items derive from741741unique evidence sentences, so each evidence yields roughly two rewritten summaries on average\.

#### Rewriting\.

Each evidence sentence is rewritten into candidate summaries by claude\-haiku\-4\-5 \(maximum500500output tokens\)\. The system prompt supplies the complete five\-class definitions together with per\-class strategy guidelines describing how to construct each error type; the user prompt supplies one evidence sentence and one target label, and requests a single fluent summary that, compared against the evidence, exhibits the targeted class\. The model returns the summary wrapped in explicit delimiters\. We rewrite each evidence against multiple target labels, cycling through classes until each class has accumulated300300candidates, for15001500candidates in total\.

#### Annotation and adjudication\.

Every candidate summary is then independently labeled by two annotators drawn from*different model families*, which is central to avoiding a self\-labeling artifact:

- •Annotator A: claude\-haiku\-4\-5 \(Anthropic\)\.
- •Annotator B: gpt\-5\.4\-mini \(OpenAI\)\.
- •Adjudicator: claude\-opus\-4\-7 \(Anthropic\), invoked only on items where A and B disagree\.

Each annotator receives the evidence, the candidate summary, and the five\-class definitions, and returns a single label\. The two annotators agree on13601360of15001500items; on the remaining140140\(9\.3%9\.3\\%\) the adjudicator assigns the final label\. Inter\-annotator agreement is measured by Krippendorff’sα\\alphacomputed over annotators A and B only:

α=1−DoDe=0\.882,\\alpha=1\-\\frac\{D\_\{o\}\}\{D\_\{e\}\}=0\.882,whereDoD\_\{o\}andDeD\_\{e\}are the observed and expected disagreement\. The adjudicator does not enter this computation\. Because annotator B belongs to a different model family than the rewriter \(claude\-haiku\-4\-5\), thisα\\alphadoes not reflect a model scoring its own output, and the value comfortably exceeds the conventional0\.800\.80threshold for reliable annotation\.

evidence sentenceRewriter\(Haiku 4\.5\)evidence\+\+target label→\\tosummaryTwo\-annotator labelingA: Haiku 4\.5 B: GPT\-5\.4\-miniagree on1360/15001360/1500items→\\togoldAdjudicator\(Opus 4\.7\)resolves the140140disagreements \(9\.3%9\.3\\%\)final gold labelFigure 4:The annotation pipeline\. Each rewritten summary is labeled independently by two annotators from different model families; the9\.3%9\.3\\%of items on which they disagree are resolved by an adjudicator\. Krippendorff’sα\\alphais computed over the two annotators only\.
#### Gold label distribution\.

The rewriter targets a balanced300300items per class, but adjudication shifts the final distribution, because the priority rule of Appendix[A](https://arxiv.org/html/2607.16212#A1)relabels some items\. The gold distribution is:

The final distribution departs from the balanced300300\-per\-class target for two reasons: the priority rule of Appendix[A](https://arxiv.org/html/2607.16212#A1)relabels116116items toUnsupported, and annotators sometimes judged a rewrite to realize a different class than the one targeted\. Both effects move mass towardUnsupportedandCorrectand away fromUnit ErrorandScale Error\. The three source domains remain near\-balanced, contributing484484,507507, and509509items\.

#### Human spot\-check\.

To validate the LLM gold labels against a human standard, one of the authors—blind to all model predictions and to the paper’s analysis—independently labeled a random5050\-item sample\. The human label matches the adjudicated gold label on4747of5050items \(94%94\\%\)\. All three disagreements areScale Error\-versus\-Relation Errorboundary cases involving sub\-ten\-fold numerical differences, a region in which the LLM annotators also reported low confidence; they are genuine borderline cases rather than annotation errors\.

## Appendix EBaseline Configurations

We evaluate five groups of baseline systems\. This section specifies each one: its checkpoint or definition, its native output, and—for systems whose native output is not already in our label space—how that output is routed to a five\-class label\.

#### Trivial baselines\.

These establish the floor of the metric:

predict\-CorrectreturnsCorrectfor every item, unconditionally\. It measures the score obtainable with no modeling at all, and coincides numerically with a majority\-class predictor, sinceCorrectis the majority class of the development split\.

string\-overlapreturnsCorrectif the evidence and summary are identical after whitespace trimming, andUnsupportedotherwise\. It uses exact matching, with no similarity threshold\.

unit\-substringextracts each value\-and\-unit token from the summary with a regular expression\. It first checks for the five significance\-group trigger phrases \(Appendix[B](https://arxiv.org/html/2607.16212#A2.SS0.SSS0.Px4)\) and returnsUnsupportedif any is present; otherwise it returnsUnit Errorif any extracted token is absent from the evidence, andCorrectif all tokens are present\.

#### Rule\-based baseline\.

quantulum\+\+Pintextracts quantity tuples from both sentences with quantulum, position\-aligns them, and applies the Stage 2 checks of Appendix[B](https://arxiv.org/html/2607.16212#A2): a dimensional mismatch yieldsUnit Errorand an order\-of\-magnitude difference yieldsScale Error\. It performs no comparator or entity\-binding analysis, which is the capabilityUGVadds in Stage 3\.

#### Our symbolic verifier\.

UGVis the full four\-stage pipeline of Appendix[B](https://arxiv.org/html/2607.16212#A2)\. It uses neither task supervision nor a language model at inference time\.

#### Off\-the\-shelf neural fact\-checkers\.

Each of the following emits a binary or ternary entailment decision natively\. We list each system with its checkpoint and native output:

MiniCheck\(lytang/MiniCheck\-Flan\-T5\-Large\): a Flan\-T5 model emitting a “Yes”/“No” supported decision\.

FactCG\(yaxili96/FactCG\-DeBERTa\-v3\-Large\): a DeBERTa\-v3 classifier whose positive label denotes “supported\.”

ModernBERT\-NLI, zero\-shot\(tasksource/ModernBERT\-large\-nli\): a three\-way NLI model; we read the “entailment” class as supported\.

Bespoke\-MiniCheck\-7B\(bespokelabs/Bespoke\-MiniCheck\-7B\): a 7B instruction\-tuned model emitting a “Yes”/“No” decision\.

Granite Guardian\(ibm\-granite/granite\-guardian\-3\.3\-8b\): an 8B groundedness judge emitting a “Yes”/“No” decision via its chat template\.

VeriFastScore\(lytang/VeriFastScore\-8B\): an 8B high\-throughput verifier that emits a binary supported decision\.

Label routing\.Because these systems emit only a support decision, we route their output into the five\-class space with a fixed rule\. A “supported” or “entailment” output maps directly toCorrect\. For any non\-supported output, we apply the following cascade:

1. 1\.if the summary carries an unsupported trigger phrase absent from the evidence, returnUnsupported;
2. 2\.else if a comparator flip is detected between summary and evidence, returnRelation Error;
3. 3\.else returnUnit Erroras the default\.

This cascade has a structural consequence: it can never outputScale Error\. This is why every off\-the\-shelf neural baseline scores exactly0\.0000\.000onScale Errorin Table[1](https://arxiv.org/html/2607.16212#S4.T1)—a limitation of the binary\-entailment paradigm, not a tuning artifact\.

#### Closed\-frontier LLMs\.

We evaluate three closed\-weight models—Claude Opus 4\.7, GPT\-5\.5, and DeepSeek V4\-Flash—accessed through their respective APIs\. All three are run zero\-shot with chain\-of\-thought prompting and greedy decoding \(temperature0\)\. The system prompt contains the full five\-class definitions and four in\-prompt example triples, each a \(evidence, summary, label\) example for a distinct class; the user prompt supplies the evidence and summary and requests a label\. Output token budgets are512512for Claude and GPT and10241024for DeepSeek V4\-Flash, which requires extra space for reasoning tokens\. All three are evaluated on the full15001500\-item benchmark\.

## Appendix FMetrics

This section gives the precise definition of every metric used in the paper\.

#### Macro\-F1F\_\{1\}\.

For each classcc, letTPc\\mathrm\{TP\}\_\{c\},FPc\\mathrm\{FP\}\_\{c\}, andFNc\\mathrm\{FN\}\_\{c\}denote true positives, false positives, and false negatives\. The per\-class precision and recall are

Pc=TPcTPc\+FPc,Rc=TPcTPc\+FNc,P\_\{c\}=\\frac\{\\mathrm\{TP\}\_\{c\}\}\{\\mathrm\{TP\}\_\{c\}\+\\mathrm\{FP\}\_\{c\}\},\\qquad R\_\{c\}=\\frac\{\\mathrm\{TP\}\_\{c\}\}\{\\mathrm\{TP\}\_\{c\}\+\\mathrm\{FN\}\_\{c\}\},and the per\-classF1F\_\{1\}is their harmonic mean,

F1,c=2​Pc​RcPc\+Rc\.F\_\{1,c\}=\\frac\{2\\,P\_\{c\}\\,R\_\{c\}\}\{P\_\{c\}\+R\_\{c\}\}\.The primary metric, macro\-F1F\_\{1\}, is the unweighted mean over the five classes,

F1macro=15​∑c∈𝒞F1,c,F\_\{1\}^\{\\mathrm\{macro\}\}=\\frac\{1\}\{5\}\\sum\_\{c\\in\\mathcal\{C\}\}F\_\{1,c\},where𝒞\\mathcal\{C\}ranges over the five classesCorrect,Unit Error,Scale Error,Relation Error, andUnsupported\. The unweighted mean gives the rare and common classes equal weight, so a system cannot score well merely by serving the majority class\.

#### Probe accuracy\.

For augmentation familyFiF\_\{i\}with probe variant setViV\_\{i\}\(Appendix[C](https://arxiv.org/html/2607.16212#A3)\), every variantv∈Viv\\in V\_\{i\}has gold labelCorrect\. Probe accuracy is the fraction the modely^\\hat\{y\}still classifiesCorrect,

accFi=1\|Vi\|​∑v∈Vi𝟙​\[y^​\(v\)=Correct\],\\mathrm\{acc\}\_\{F\_\{i\}\}=\\frac\{1\}\{\|V\_\{i\}\|\}\\sum\_\{v\\in V\_\{i\}\}\\mathbb\{1\}\\\!\\left\[\\,\\hat\{y\}\(v\)=\\textsc\{Correct\}\\,\\right\],where𝟙​\[⋅\]\\mathbb\{1\}\[\\cdot\]is the indicator function\. Since every probe variant is a label\-preserving rewrite of a genuinelyCorrectitem, probe accuracy is exactly the model’s recall on that family’s surface transformation\.

#### Out\-of\-distribution metric\.

SciFact\-Open provides binary SUPPORT/CONTRADICT labels\. To apply our five\-class models to it, we fold each predictionyywith the mapping

ϕ​\(y\)=\{SUPPORTif​y=Correct,CONTRADICTotherwise,\\phi\(y\)=\\begin\{cases\}\\text\{SUPPORT\}&\\text\{if \}y=\\textsc\{Correct\},\\\\\[3\.0pt\] \\text\{CONTRADICT\}&\\text\{otherwise\},\\end\{cases\}that is, the four error classes all map to CONTRADICT\. We then report binary macro\-F1F\_\{1\}, the unweighted mean of the SUPPORT and CONTRADICTF1F\_\{1\}scores\. The out\-of\-distribution evaluation is detailed in Appendix[I](https://arxiv.org/html/2607.16212#A9)\.

## Appendix GTraining Details

#### Model and optimization\.

The fine\-tuned model uses tasksource/ModernBERT\-large\-nli \(400400M parameters\) as its backbone\. The pretrained three\-class NLI head is discarded and replaced with a randomly initialized five\-class linear classifier; the size mismatch between the old and new heads is handled by allowing mismatched checkpoint sizes at load time\. The full optimization configuration is:

- •Optimizer: AdamW, learning rate2×10−52\\times 10^\{\-5\}, weight decay0\.010\.01\.
- •Schedule: linear warmup over the first10%10\\%of steps, followed by linear decay\.
- •Batch size:88, limited by GPU memory at sequence length512512\.
- •Epochs:33, with evaluation at the end of training and no early stopping\.
- •Sequence length:512512tokens, with dynamic padding within each batch\.
- •Loss: cross\-entropy\.
- •Gradient clipping: maximum norm1\.01\.0\.
- •Precision: float32, with no mixed precision\.

#### Hardware\.

Each fold is trained on a single NVIDIA RTX 3090 GPU \(2424GB\)\. A single fold completes in two to three minutes, and the full five\-fold run completes in twelve to fifteen minutes of wall\-clock time\.

#### Cross\-validation\.

We use stratified five\-fold cross\-validation, stratified by gold label, with12001200training and300300test items per fold\. We report two evaluation modes\. The*5\-fold pooled*mode—used for all headline numbers—pools the out\-of\-fold predictions across all five folds and computes macro\-F1F\_\{1\}on the full15001500items\. The*single\-fold*mode uses one fixed80/2080/20stratified split\. Pooled numbers are more stable, since per\-fold single\-split numbers fluctuate with the particular split\.

#### Augmentation volume\.

Symbolic Augmentation is applied only to the goldCorrectitems of each training fold, so the label distribution of the augmented training set matches the original\. F1 augmentation adds approximately155155canonical\-equivalent items per fold \(its rewrite rules applied to the fold’sCorrectitems, capped at three new items per source item\), giving a combined training set of roughly13501350items\. All\-four augmentation adds approximately206206items per fold \(capped at four per source item across the four families\), giving roughly14061406items\.

## Appendix HNegative\-Result Setups

This section specifies the configuration of each integration point compared in Table[4](https://arxiv.org/html/2607.16212#S5.T4)\.

#### Auxiliary symbolic features\.

This setup concatenates the symbolic verifier’s output to the encoder as an auxiliary input\. TheUGVoutput is encoded as a1212\-dimensional vector: a55\-dimensional one\-hot of theUGVpredicted label, plus a77\-dimensional continuous trace of intermediate signals—the log\-scaled evidence and summary quantity counts, a dimensional\-mismatch flag, a comparator\-flip flag, a superlative\-trigger flag, a scale\-gap log\-ratio clamped to\[−3,3\]\[\-3,3\], and an entity\-swap flag\. Architecturally, the ModernBERT encoder output is mean\-pooled over non\-padding tokens, concatenated with a6464\-dimensional learned projection of the auxiliary vector, and passed through a two\-layer classification head\. Training otherwise matches the canonical fine\-tuning configuration of Appendix[G](https://arxiv.org/html/2607.16212#A7)\. To isolate the contribution of the auxiliary features, we train two variants that are identical except that one receives the true feature vector and the other receives a zeroed vector\.

#### Silver\-label teachers\.

This setup usesUGVas a teacher that labels training data, which is then used to train the encoder\. We evaluate it at two scales:

- •n=1200n=1200: the gold labels of the canonical training split are replaced withUGV’s predicted labels\.UGVagrees with the gold label on53\.1%53\.1\\%of these items, so the silver set is substantially noisier than gold\.
- •n=3298n=3298: in addition, five new perturbations per evidence are generated by gpt\-5\.4\-mini with the same five\-class taxonomy prompt as the main rewriter, and each is labeled byUGV\. After deduplication against existing test summaries,32983298silver items remain\.

In both cases the silver set replaces the gold training set entirely; the encoder is otherwise trained as in Appendix[G](https://arxiv.org/html/2607.16212#A7)\.

#### Inference\-time logit ensemble\.

This setup keeps the canonical encoder unchanged and combines its output withUGV’s at inference time\. SinceUGVis rule\-based and emits a hard label rather than a distribution, its prediction is represented as a one\-hot vector\. The ensemble distribution is the convex combination

p=\(1−α\)​pFT\+α⋅onehot​\(y^UGV\),p=\(1\-\\alpha\)\\,p\_\{\\mathrm\{FT\}\}\+\\alpha\\cdot\\mathrm\{onehot\}\(\\hat\{y\}\_\{\\textsc\{UGV\}\}\),wherepFTp\_\{\\mathrm\{FT\}\}is the encoder’s softmax distribution andα=0\.25\\alpha=0\.25\. The final prediction isarg⁡maxc⁡pc\\arg\\max\_\{c\}p\_\{c\}\.

#### Agreement filter\.

This setup usesUGV’s prediction only as a gate\. For each item, ifUGVand the encoder predict the same label, that label is returned; if they disagree, the encoder’s prediction is returned\. Formally the output is

y^=\{y^UGVif​y^UGV=y^FT,y^FTotherwise\.\\hat\{y\}=\\begin\{cases\}\\hat\{y\}\_\{\\textsc\{UGV\}\}&\\text\{if \}\\hat\{y\}\_\{\\textsc\{UGV\}\}=\\hat\{y\}\_\{\\mathrm\{FT\}\},\\\\\[3\.0pt\] \\hat\{y\}\_\{\\mathrm\{FT\}\}&\\text\{otherwise\.\}\\end\{cases\}In the first branchy^UGV=y^FT\\hat\{y\}\_\{\\textsc\{UGV\}\}=\\hat\{y\}\_\{\\mathrm\{FT\}\}, so the output equalsy^FT\\hat\{y\}\_\{\\mathrm\{FT\}\}; in the second branch it equalsy^FT\\hat\{y\}\_\{\\mathrm\{FT\}\}by definition\. The filter therefore returnsy^FT\\hat\{y\}\_\{\\mathrm\{FT\}\}on every item and is mathematically identical to the encoder alone, which is why its macro\-F1F\_\{1\}exactly matches the canonical baseline\.

## Appendix IOut\-of\-Distribution Evaluation Details

#### Evaluation set\.

We evaluate out\-of\-distribution transfer on SciFact\-Open\(Wrightet al\.,[2022](https://arxiv.org/html/2607.16212#bib.bib22)\)\. We load the test split and keep the206206items carrying a SUPPORT or CONTRADICT label—105105and101101respectively—discarding items labeled not\-enough\-information\. No further filtering by domain, quantity content, or sentence length is applied\. Both fine\-tuned models are evaluated zero\-shot: they are trained only on the typed\-quantity benchmark and never observe SciFact\-Open data or its label space, so their five\-class predictions are folded to binary using the mappingϕ\\phiof Appendix[F](https://arxiv.org/html/2607.16212#A6)\.

#### Ladder of systems\.

Table[5](https://arxiv.org/html/2607.16212#A9.T5)places the two fine\-tuned models on a ladder with the same rule\-based and symbolic systems used in the main experiments\. Performance increases monotonically along the ladder: from the trivial baselines, through the rule\-based systems andUGV, to the canonical fine\-tuned encoder, and finally to the F1\-augmented encoder\. Symbolic Augmentation adds\+0\.037\+0\.037binary macro\-F1F\_\{1\}on top of canonical fine\-tuning, entirely through zero\-shot transfer\.

Table 5:Out\-of\-distribution results on SciFact\-Open \(n=206n=206\), for the same rule\-based and fine\-tuned systems as the main experiments\.
#### Per\-class breakdown\.

Table[6](https://arxiv.org/html/2607.16212#A9.T6)breaks the two fine\-tuned models down by class\. The improvement is concentrated in SUPPORT recall, which rises from0\.6290\.629to0\.7140\.714, a gain of8\.58\.5points\. The augmented encoder recognizes more SciFact\-Open claims that paraphrase the evidence with canonical\-equivalent units as genuinely supporting the evidence, rather than mistaking them for relational errors or unsupported assertions\. Concretely, of the1111items the augmented model newly labelsCorrect—hence SUPPORT—seven had been assignedRelation Errorand fourUnsupportedby the canonical model\. This is the same capability the F1 probe measures in distribution, now visible on an external benchmark\.

Table 6:Per\-class out\-of\-distribution results on SciFact\-Open\. Symbolic Augmentation lifts SUPPORT recall by8\.58\.5points, the main source of the macro\-F1F\_\{1\}gain\.