相同数量,不同答案:语言模型中的数值表示不变性
摘要
本文评估了语言模型中的数值表示不变性,发现评估者界面问题可以模拟推理失败,并识别了如 Mistral Small 4 中单位转换问题等模型特定错误。
arXiv:2609.25009v1 Announce Type: new
Abstract: Numerically equivalent word problems should yield the same canonical answer whether a quantity is written as a decimal, fraction, percentage, number word, scientific notation, or an exactly converted unit. We generate 3,600 exact-rational problems and 8,600 prompts spanning five identity-preserving transformation families, and evaluate five open-weight systems. After a fixed syntax audit that normalizes common answer forms without an LLM judge, canonical accuracy is 0.969-0.996, but orbit correctness falls to 0.848-0.981 and orbit invariance to 0.851-0.981; invariant-but-wrong orbits account for at most 0.003. Most of the broad strict-parser collapse arises because multiplication-form scientific notation lies outside the implemented number grammar, illustrating how evaluator interfaces can masquerade as reasoning failures. A distinct semantic pathology remains: Mistral Small 4 scores 0.699 on unit-converted inputs and produces 265 errors differing from the label by exact powers of ten. In a separate 9,000-call experiment that allocates equal calls to the compared arms, representation consensus does not outperform paraphrase consensus on a low-error subset and produces substantially more false alarms. The accompanying ancillary archive contains the frozen benchmark, evaluation and audit records, consensus raw responses, manifests, analysis code, and a one-command paper build.
查看缓存全文
缓存时间: 2026/09/23 09:05
# Numerical Representation Invariance in Language Models
Source: [https://arxiv.org/html/2609.25009](https://arxiv.org/html/2609.25009)
## Same Quantity, Different Answer: Numerical Representation Invariance in Language Models
###### Abstract
Numerically equivalent word problems should yield the same canonical answer whether a quantity is written as a decimal, fraction, percentage, number word, scientific notation, or an exactly converted unit\. We generate 3,600 exact\-rational problems and 8,600 prompts spanning five identity\-preserving transformation families, and evaluate five open\-weight systems\. After a fixed syntax audit that normalizes common answer forms without an LLM judge, canonical accuracy is \.969–\.996, but orbit correctness falls to \.848–\.981 and orbit invariance to \.851–\.981; invariant\-but\-wrong orbits account for at most \.003\. Most of the broad strict\-parser collapse arises because multiplication\-form scientific notation lies outside the implemented number grammar, illustrating how evaluator interfaces can masquerade as reasoning failures\. A distinct semantic pathology remains: Mistral Small 4 scores \.699 on unit\-converted inputs and produces 265 errors differing from the label by exact powers of ten\. In a separate 9,000\-call experiment that allocates equal calls to the compared arms, representation consensus does not outperform paraphrase consensus on a low\-error subset and produces substantially more false alarms\. The supplementary archive contains the frozen benchmark, evaluation and audit records, consensus raw responses, manifests, analysis code, and a one\-command paper build\.
## 1Introduction
The correctness of a quantitative answer should not depend on whether a volume is written as3/4 Lor0\.75 L\. Numerical*representation invariance*requires the same normalized answer under an identity\-preserving rewrite of the input, including an exact rewrite into equivalent units\. ParaRel provides a methodological precedent for checking consistency under paraphrase\(Elazar et al\.,[2021](https://arxiv.org/html/2609.25009#bib.bib6)\); our transformations instead have algebraically known identity relations\. Aggregate benchmarks obscure violations because they expose only one surface form per item\.
Prior work establishes several neighboring effects\. Language models encode numbers imperfectly\(Wallace et al\.,[2019](https://arxiv.org/html/2609.25009#bib.bib23); Thawani et al\.,[2021](https://arxiv.org/html/2609.25009#bib.bib22)\); digit\-segmentation choices can affect arithmetic accuracy\(Singh & Strouse,[2024](https://arxiv.org/html/2609.25009#bib.bib21); McLeish et al\.,[2024](https://arxiv.org/html/2609.25009#bib.bib12)\); semantically irrelevant prompt formatting can move task accuracy substantially\(Sclar et al\.,[2024](https://arxiv.org/html/2609.25009#bib.bib20)\); and numeral, unit, or script changes can alter quantitative performance\(Xu et al\.,[2024](https://arxiv.org/html/2609.25009#bib.bib25); Bui et al\.,[2025](https://arxiv.org/html/2609.25009#bib.bib2); Reddy et al\.,[2026](https://arxiv.org/html/2609.25009#bib.bib18)\)\. These results motivate an exact, per\-item test that places heterogeneous representation changes in one framework without assuming a tokenization or decoding mechanism\.
Our framework has three parts\. First, each item is a typed semantic program with an exact rational answer\. A renderer expands it into an*orbit*of equivalent prompts under constrained decimal/fraction, scientific\-notation, percent/decimal, digits/words, and input\-unit transformations\. Equivalence follows from the program and registered exact conversion factors, rather than from post\-hoc judgment\.
Second, we separate a narrow parser interface from an audited value layer\. This distinction is necessary because the natural\-language instruction uses the placeholder<number\>but does not enumerate the parser’s accepted syntaxes\. The strict layer therefore measures compatibility with the implemented parser\. The audit strips markdown emphasis, case\-folds the answer tag, and normalizes multiplication\-form scientific notation to e\-notation before re\-running that same parser\. It is deliberately narrower than a general answer extractor\.
Third, the orbit supplies a ground\-truth\-free disagreement signal at inference time, in the spirit of metamorphic testing and selective prediction\. We compare representation consensus with equal\-call self\-consistency and fixed\-paraphrase controls under matched sampling settings\.
Running five pinned systems—Qwen3\.5 at 4B and 9B, the same 9B checkpoint at Q4\_K\_M and Q8\_0, GPT\-OSS\-20B, and Mistral Small 4—produces four findings\. First, strict\-parser compatibility is highly surface\-sensitive, chiefly because standard multiplication\-form scientific notation lies outside the implemented grammar\. Second, audited semantic outcomes remain representation\-sensitive: audited orbit correctness is \.848–\.981 and audited invariance is \.851–\.981, despite canonical correctness of \.969–\.996\. Third, Mistral’s unit\-conversion weakness is not repaired by the audit and is concentrated in two eligible templates\. Fourth, H3 is unsupported on the evaluated 120\-base subset: representation and paraphrase AURC are indistinguishable within paired\-bootstrap uncertainty, while representation views generate many more false alarms\.
#### Contributions\.
1. 1\.An exact\-rational orbit framework that reports correctness, invariance, and consistent\-wrong outcomes per base problem\.
2. 2\.A frozen benchmark of 3,600 typed base problems and 8,600 prompts, with explicit eligibility masks and 1,000 pairs per transformation family\.
3. 3\.A transparent separation between strict\-parser compatibility and a fixed audited value layer, including paired confidence intervals, directional discordance, multiplicity\-adjusted tests, and audited orbit metrics\.
4. 4\.A five\-system descriptive study with a same\-checkpoint Q4/Q8 comparison and template\-stratified analysis of the largest value failure\.
5. 5\.A secondary equal\-call consensus analysis with paired uncertainty and fallback\-order sensitivity, reported as an unsupported H3 rather than evidence of equivalence\.
## 2Related Work
#### Numeracy and number encoding\.
Wallace et al\. \([2019](https://arxiv.org/html/2609.25009#bib.bib23)\)showed that static and contextual embeddings encode numeracy imperfectly, andThawani et al\. \([2021](https://arxiv.org/html/2609.25009#bib.bib22)\)survey consequences of number surface forms in NLP\.Singh & Strouse \([2024](https://arxiv.org/html/2609.25009#bib.bib21)\)find that digit\-segmentation conventions affect arithmetic accuracy, whileMcLeish et al\. \([2024](https://arxiv.org/html/2609.25009#bib.bib12)\)improve arithmetic with digit\-position embeddings\. These works establish plausible mechanisms for other settings\. Our experiment does not manipulate tokenizers, so its cross\-system patterns cannot identify tokenization as the cause\.
#### Perturbing math word problems\.
GSM\-Symbolic\(Mirzadeh et al\.,[2025](https://arxiv.org/html/2609.25009#bib.bib13)\)templates GSM8K items and varies names and numeric values, demonstrating instability across instantiations; the values change but their representations do not, and the analysis is predominantly aggregate over 50\-instantiation sets\. Numeric\-remapping attacks\(Barker et al\.,[2026](https://arxiv.org/html/2609.25009#bib.bib1)\)remap values automatically and recompute labels via LLM\-inferred programs, with unspecified numeric precision\. Both lines vary the instantiated numerical quantities rather than their equivalent surface representations\. We hold the underlying quantities and requested answer fixed and vary only how the input quantities are written, with labels that are exact by construction rather than recomputed\.
#### Representation and format sensitivity\.
NUMCoT\(Xu et al\.,[2024](https://arxiv.org/html/2609.25009#bib.bib25)\)studies numeral and measurement\-unit choices in chain\-of\-thought reasoning and evaluates them with task\-specific answer extraction\.Bui et al\. \([2025](https://arxiv.org/html/2609.25009#bib.bib2)\)vary measurement systems over compiled real\-world facts and report increased test\-time compute for underrepresented systems\.Sclar et al\. \([2024](https://arxiv.org/html/2609.25009#bib.bib20)\)document large accuracy changes under prompt\-format variations that preserve content\.Reddy et al\. \([2026](https://arxiv.org/html/2609.25009#bib.bib18)\)provide the nearest numeral\-format comparison: they hold arithmetic fixed while varying scripts and formatting conventions on expressions, and distinguish correct\-value/wrong\-format responses\. Our narrower contribution is exact\-rational generation and unit canonicalization across five jointly analyzed families, with both orbit correctness and invariance measured per item\.
#### Behavioral and metamorphic testing\.
CheckList organizes behavioral tests as minimum\-functionality, invariance, and directional\-expectation tests\(Ribeiro et al\.,[2020](https://arxiv.org/html/2609.25009#bib.bib19)\)\. METAL applies metamorphic relations to analyze LLM qualities\(Hyun et al\.,[2024](https://arxiv.org/html/2609.25009#bib.bib9)\), and LLMorph develops natural\-language metamorphic transformations for NLP tasks\(Cho et al\.,[2025](https://arxiv.org/html/2609.25009#bib.bib4)\)\. Our orbits instantiate the same test\-oracle idea with typed quantitative programs: the input relation and exact output relation are generated together, and all answers are canonicalized to rational values\.
#### Consistency, selective prediction, and consensus\.
ParaRel\(Elazar et al\.,[2021](https://arxiv.org/html/2609.25009#bib.bib6)\)measures factual consistency under paraphrase and documents substantial inconsistency\. SelfCheckGPT uses cross\-sample agreement as a hallucination signal\(Manakul et al\.,[2023](https://arxiv.org/html/2609.25009#bib.bib11)\); selective prediction\(El\-Yaniv & Wiener,[2010](https://arxiv.org/html/2609.25009#bib.bib5); Geifman & El\-Yaniv,[2017](https://arxiv.org/html/2609.25009#bib.bib7)\)and selective question answering\(Kamath et al\.,[2020](https://arxiv.org/html/2609.25009#bib.bib10)\)formalize the risk–coverage tradeoff\. Self\-consistency prompting aggregates resampled chains\(Wang et al\.,[2023](https://arxiv.org/html/2609.25009#bib.bib24)\)\. Our H3 experiment isolates a different source of diversity—equivalent numerical representations—against equal\-call resampling and paraphrase controls\. The logic follows metamorphic testing\(Chen et al\.,[1998](https://arxiv.org/html/2609.25009#bib.bib3)\), but the empirical question is whether dissent predicts errors rather than merely surface sensitivity\.
## 3Framework and Benchmark
### 3\.1Semantic programs, orbits, and the two properties
The statistical unit is abase problemb=\(P,θ,d,u∗,y\)b=\(P,\\theta,d,u^\{\\ast\},y\): a typed programPPwith rational parametersθ\\theta, a physical dimensiondd, a canonical unitu∗u^\{\\ast\}, and an exact answery∈ℚy\\in\\mathbb\{Q\}\. A rendererρg\\rho\_\{g\}mapsbbto a prompt under each applicable viewg∈Gbg\\in G\_\{b\}\. Theorbitis\{ρg\(b\):g∈Gb\}\\\{\\rho\_\{g\}\(b\):g\\in G\_\{b\}\\\}\.
LetCCbe a parser and exact unit canonicalizer, and letzb,g=C\(f\(ρg\(b\)\)\)z\_\{b,g\}=C\(f\(\\rho\_\{g\}\(b\)\)\), withzb,gz\_\{b,g\}undefined when parsing fails\.*Orbit correctness*requires
zb,g=yfor everyg∈Gb,z\_\{b,g\}=y\\qquad\\text\{for every \}g\\in G\_\{b\},whereas*orbit invariance*requires everyzb,gz\_\{b,g\}to be defined and
zb,g=zb,hfor everyg,h∈Gb\.z\_\{b,g\}=z\_\{b,h\}\\qquad\\text\{for every \}g,h\\in G\_\{b\}\.An orbit may therefore be invariant but consistently wrong\. We report that event separately rather than treating agreement as correctness\.
We instantiateCCtwice: the strict parserCsC\_\{s\}and the audited canonicalizerCaC\_\{a\}, which applies the fixed rewrite in Section[3\.4](https://arxiv.org/html/2609.25009#S3.SS4)before the same parser\. The unit family is an input\-side invariance test: statement quantities are restated in an exact sub\-unit while the question still requests the canonical unit\. Output values in any whitelisted unit are converted by a rational factor\. Label construction, registered unit conversion, and answer comparison use rational arithmetic; model generation itself is not claimed to avoid floating\-point computation\.
### 3\.2Transformation families
Table 1:Transformation families\. Each transformed prompt preserves the expected canonical answer on its eligible bases\. Unit conversion changes only quantities in the statement; the requested answer unit remains canonical\.The constraints in Table[1](https://arxiv.org/html/2609.25009#S3.T1)make equivalence exact\. No decimal/fraction pair is rounded, and every unit factor is registered as a rational\. Scientific notation is restricted so its displayed mantissa and exponent equal the source integer exactly\. Applicability is part of the estimand: per\-view and paired results include only bases for which that family was generated\.
### 3\.3Dataset construction
Table 2:Template allocation\. Counts in parentheses are transformed prompts; every base also has one canonical prompt\. The allocations sum to 1,000 prompts for each transformation family\.The six templates yield 3,600 bases and 8,600 prompts \(Table[2](https://arxiv.org/html/2609.25009#S3.T2)\)\. Each template uses a private pseudorandom stream under seed 20260725; duplicate argument tuples are rejected\. Eligibility masks use fixed, template\-stratified quotas\. Consequently, a pooled family gap is descriptive over that family’s eligible template mixture, not a causal estimate separable from operation, template, or exponent range\. Scientific notation, percentage, and number words each occur in only one template\. We therefore scope claims to the generated support and stratify the largest unit result below\.
Generator version 0\.2\.0 and prompt version p1 producebase\_benchmark\.jsonlandviews\_benchmark\.jsonl\. SHA\-256 identifiers and generation parameters are included in the supplement; regeneration is byte\-identical, and expected answers round\-trip through the public parser\.
### 3\.4Prompt, strict parser compatibility, and audited values
Every prompt says:*“Return exactly one final line in this form:FINAL: <number\> <unit\>”*\. The placeholder<number\>does not enumerate the implemented grammar\. The strict parser accepts signed integers, finite decimals with optional comma grouping, integer fractions, and e\-notation, but not the conventional multiplication formm×10km\\times 10^\{k\}\. It also accepts the last of multipleFINALlines while flagging the response\. Strict scores are therefore named*strict\-parser compatibility*; calling every rejection a model contract violation would overstate what the prompt specified\.
The parser canonicalizes whitelisted units with rational factors\. Failures have fixed precedence:truncation\_error\(finish reason other than stop\),format\_error\(no uppercaseFINAL:line\),parse\_error,unit\_error, thenvalue\_error\. There is no LLM judge or free\-form repair\. The operational grammar summary appears in Appendix[B](https://arxiv.org/html/2609.25009#A2); the included parser source is authoritative\.
Audit versionaudit\_v2is applied to strict\-failed responses: remove markdown emphasis, case\-foldFINAL:, and normalizem×10km\\times 10^\{k\},x10ˆk, and Unicode\-superscript scientific forms to e\-notation\. The unchanged parser then runs again\. A recovered item shows that the transcript contains the expected value under this narrow normalization; it does not prove a particular internal calculation\. Residual parseable wrong values are tagged*magnitude\-drop*whenparsed×10k=expected\\mathrm\{parsed\}\\times 10^\{k\}=\\mathrm\{expected\}for an integer0<\|k\|≤150<\|k\|\\leq 15\. All comparisons are exact\.
### 3\.5Metrics, hypotheses, and statistics
For both strict and audited canonicalizers we report per\-view correctness, orbit correctness, orbit invariance, and consistent\-wrong rate\. Per\-view denominators are the eligible bases, not all 3,600 bases\. The orbit gap is canonical correctness minus orbit correctness\. For each scoring layer, family tables report canonical\-minus\-transformed correctness on the 1,000 paired bases, directional discordance, and exact McNemar tests\.
The protocol fixed a 3\-percentage\-point practical\-effect threshold: 30 outcomes per 1,000\-pair family\. This is a pragmatic materiality choice, not a power\-derived boundary\. Holm adjustment is performed separately within each scoring layer and system over exactly five family tests; it does not cover the five systems, orbit summaries, or exploratory strata\. Family\-gap and orbit\-gap intervals use 10,000 base\-clustered bootstrap draws\. Audited orbit rates use two\-sided 95% Wilson intervals\. The project plan fixed these choices before cross\-model runs but was not deposited in an external registry, so we use*protocol\-fixed*, not*preregistered*\.
Operational H1 asks whether any strict\-parser family gap is at least three points and Holm\-significant; operational H2 asks whether the strict\-parser orbit gap is at least three points\. Because of the prompt/parser mismatch, these hypotheses test the implemented interface, not arithmetic invariance alone\. We report the audited paired analysis as the primary semantic result without relabeling the original hypotheses\. H3 asks whether representation consensus improves risk–coverage over both equal\-call controls\. AURC averages uniformly over orderings within equal\-agreement ties\.
## 4Experimental Setup
### 4\.1Systems
Table 3:Evaluated systems\. Each checkpoint, quantization, and decoding combination is a distinct system\. Qwen3\.5\-9B Q4/Q8 shares a source checkpoint; GPT\-OSS necessarily uses its reasoning mode and a larger generation cap\.We use the official Qwen3\.5\-4B and Qwen3\.5\-9B releases\(Qwen Team,[2026a](https://arxiv.org/html/2609.25009#bib.bib16);[b](https://arxiv.org/html/2609.25009#bib.bib17)\), GPT\-OSS\(OpenAI et al\.,[2025](https://arxiv.org/html/2609.25009#bib.bib15)\), and Mistral Small 4\(Mistral AI,[2026](https://arxiv.org/html/2609.25009#bib.bib14)\)\. Repository revisions, source revisions when recorded, quantized filenames, and SHA\-256 hashes are in the stable provenance manifest\. Qwen and Mistral use their documented chat\-template controls to disable reasoning; GPT\-OSS runs at low reasoning effort\.
### 4\.2Inference protocol
All primary runs use one pinnedllama\.cppbuild\(Gerganov & contributors,[2023](https://arxiv.org/html/2609.25009#bib.bib8)\)\(b10107, commit c0bc8591e; CUDA backend\) on NVIDIA H100 80 GB HBM3 GPUs, with one request in flight\. Decoding is greedy \(temperature 0, top\-kk1, top\-pp1, seed 20260725\), but we do not infer mathematical determinism from those settings\. A ten\-prompt repeat check was 10/10 byte\- and answer\-identical for Qwen3\.5\-4B, both Qwen3\.5\-9B quantizations, and GPT\-OSS\. Mistral was answer\-identical on 9/10; one answer changed from 33\.79 to 33\.7925\. These observations characterize only the repeated prompts\. Every run stores request order, sampling settings, model and runtime identifiers, and raw responses\.
## 5Results: Representation Sensitivity Across Systems
Figure 1:Outcome composition on eligible prompts:n=3,600n\{=\}3\{,\}600for canonical andn=1,000n\{=\}1\{,\}000for each transformed view\. Solid teal is strict\-parser correct; diagonally hatched amber is correct after the fixed audit; cross\-hatched red is wrong or unparseable after audit\. Percent labels use one decimal place, and machine\-readable counts are provided with the figure\. Scientific notation is dominated by audit recoveries, whereas Mistral unit conversion contains a large residual\-failure segment\.Figure[1](https://arxiv.org/html/2609.25009#S5.F1)separates parser\-interface mismatch from residual value failure\. Scientific notation produces broad audit recovery across all systems\. Fraction inputs show smaller residual losses for Qwen and Mistral, and Mistral unit conversion forms a distinct value\-failure pattern\. A canonical\-only evaluation would hide all three structures\.
### 5\.1Audited semantic paired results
Under the implemented strict parser, operational H1 and H2 hold for all five systems\. We retain those protocol\-fixed interface results in Appendix[A](https://arxiv.org/html/2609.25009#A1); the main analysis instead foregrounds audited values because multiplication\-form scientific notation is outside the strict grammar\.
Table 4:Audited paired results,n=1,000n\{=\}1\{,\}000per family\. C/T is canonical/transformed correctness; the gap is C minus T with a 95% base\-bootstrap interval; c/t is canonical\-only/transformed\-only discordance\. Holm adjustment is within each system’s five audited tests\. Bold gaps exceed three points and have adjustedp<\.05p<\.05; this descriptive audit is not a relabeling of H1\.SystemFamilyC/TGap \[95% CI\]pHolmp\_\{\\text\{Holm\}\}c/tQwen3\.5\-4B Q4Sci\. notation1\.000/\.985\+\.015\+\.015\[\.008,\.023\]\.0002415/0Unit conversion\.995/\.998−\.003\-\.003\[−\-\.008,\.002\]1\.0002/5Fraction\.987/\.915\+\.072\\mathbf\{\+\.072\}\[\.054,\.091\]1\.7×10−141\.7\\times 10^\{\-14\}82/10Percentage\.984/\.988−\.004\-\.004\[−\-\.013,\.005\]1\.0009/13Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Qwen3\.5\-9B Q4Sci\. notation\.998/\.902\+\.096\\mathbf\{\+\.096\}\[\.078,\.115\]3\.1×10−273\.1\\times 10^\{\-27\}97/1Unit conversion\.993/\.997−\.004\-\.004\[−\-\.010,\.001\]\.5782/6Fraction\.989/\.928\+\.061\\mathbf\{\+\.061\}\[\.044,\.078\]3\.1×10−123\.1\\times 10^\{\-12\}70/9Percentage\.991/\.977\+\.014\+\.014\[\.004,\.025\]\.03821/7Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Qwen3\.5\-9B Q8Sci\. notation\.999/\.886\+\.113\\mathbf\{\+\.113\}\[\.093,\.133\]2\.8×10−322\.8\\times 10^\{\-32\}114/1Unit conversion\.997/\.999−\.002\-\.002\[−\-\.006,\.002\]\.9091/3Fraction\.990/\.910\+\.080\\mathbf\{\+\.080\}\[\.062,\.098\]4\.5×10−184\.5\\times 10^\{\-18\}87/7Percentage\.992/\.988\+\.004\+\.004\[−\-\.004,\.012\]\.90910/6Number words1\.000/\.997\+\.003\+\.003\[\.000,\.007\]\.7503/0GPT\-OSS\-20BSci\. notation\.997/\.988\+\.009\+\.009\[\.002,\.017\]\.14112/3Unit conversion\.997/\.971\+\.026\+\.026\[\.016,\.037\]4\.3×10−64\.3\\times 10^\{\-6\}28/2Fraction\.993/\.995−\.002\-\.002\[−\-\.009,\.005\]1\.0005/7Percentage\.993/\.987\+\.006\+\.006\[−\-\.002,\.014\]\.63011/5Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Mistral Small 4Sci\. notation\.996/\.996\.000\.000\[−\-\.005,\.006\]1\.0004/4Unit conversion\.980/\.699\+\.281\\mathbf\{\+\.281\}\[\.252,\.309\]5\.8×10−755\.8\\times 10^\{\-75\}288/7Fraction\.967/\.874\+\.093\\mathbf\{\+\.093\}\[\.070,\.116\]8\.2×10−158\.2\\times 10^\{\-15\}119/26Percentage\.916/\.914\+\.002\+\.002\[−\-\.016,\.020\]1\.00044/42Number words\.994/\.997−\.003\-\.003\[−\-\.007,\.000\]\.7500/3Table 4:Audited paired results \(continued\)\.Seven system–family cells exceed both the descriptive three\-point threshold and the adjusted significance criterion\. The largest audited gap is Mistral unit conversion at 28\.1 points\. Qwen fraction gaps are 6\.1–8\.0 points, and the two 9B scientific gaps are 9\.6 and 11\.3 points; Section[5\.2](https://arxiv.org/html/2609.25009#S5.SS2)separates cap\-sensitive records from parsed wrong values\.
Table 5:Audited orbit metrics on 3,600 bases\. C = correctness, I = invariance, CW = invariant but consistently wrong, and gap = canonical correctness minus orbit correctness\. Rate intervals are two\-sided 95% Wilson intervals; gap intervals use the paired base bootstrap\.Canonical\-minus\-audited\-orbit gaps are 3\.0, 5\.3, 6\.0, 1\.5, and 12\.1 points in table order\. Thus the same three\-point descriptive criterion is met by four systems, not GPT\-OSS\. Audited invariance closely tracks audited correctness because consistent wrong orbits are rare, but none of these estimates is perfect invariance\.
### 5\.2Two layers dissociate: shared audit recovery and one value pathology
Table 6:Scientific\-notation responses under strict and audited scoring \(n=1,000n\{=\}1\{,\}000\)\. Audit recovered counts expected values exposed by the fixed rewrite; residual magnitude errors differ from the label by an exact power of ten\.#### Scientific notation: values mostly survive a parser mismatch\.
Strict scientific correctness ranges from \.005 to \.347, while audited correctness ranges from \.886 to \.996 \(Table[6](https://arxiv.org/html/2609.25009#S5.T6)\)\. The rewrite recovers 638–991 expected values per system from forms such as1\.494 x 10ˆ4 liters\. Qwen scientific responses include 13 length finishes for 4B Q4, 85 for 9B Q4, and 110 for 9B Q8\. Excluding those records in a descriptive sensitivity analysis leaves audited accuracy of \.998 \(985/987\), \.986 \(902/915\), and \.996 \(886/890\), respectively\. This conditioning does not estimate what a higher\-cap rerun would produce\. The broad strict collapse establishes incompatibility with the implemented parser, not inability to perform the arithmetic\.
#### Unit conversion: a silent power\-of\-ten pattern\.
Mistral’s unit view remains \.699 after audit\. Of 301 residual errors, 265 have an exact power\-of\-ten relation to the label: 96 by1010, 70 by10210^\{2\}, and 99 by10310^\{3\}\. This pattern is consistent with carrying a sub\-unit count into a canonical\-unit answer, but output traces do not identify the model’s internal mechanism\. A syntax\-only constraint would not catch a parseable wrong value\.
Table 7:Mistral unit\-view result by eligible template\. The pooled \.699 score combines materially different strata; no unit\-conversion claim is made outside these four template/operation combinations\.PROMPT \(unit view; Repeated Length item 383\)Solve the quantitative problem\. Return exactly one final line in this form:FINAL: <number\> <unit\>A carpenter cuts 11 boards\. Each board is 330 centimeters long\.What is the total length of all the boards in meters?RESPONSE \(Mistral Small 4, greedy\)CALC: 11 \* 330 = 3630 centimetersFINAL: 3630 metersEXPECTEDFINAL: 36\.3 meters
Figure 2:Illustrative transcript excerpt\. Indentation is added for layout; prompt and response text are otherwise reproduced from the raw record, which is authoritative\. The parseable answer relabels a centimeter count as meters, so exact comparison to the semantic label is required\.
### 5\.3Audited value correctness by view
Table 8:Audited value correctness on each eligible view\. Scientific notation largely recovers after the fixed normalization; Mistral unit conversion does not\.The audit changes the scientific interpretation but not every family\. Audited fraction correctness is \.915 for Qwen3\.5\-4B Q4, \.928 for Qwen3\.5\-9B Q4, \.910 for Qwen3\.5\-9B Q8, \.995 for GPT\-OSS, and \.874 for Mistral\. Words remain \.997–1\.000\. These are within\-eligibility descriptions, not family effects averaged over a common base set\.
### 5\.4Observed scale and quantization comparisons
Within the evaluated Qwen releases, moving from 4B to 9B under Q4\_K\_M changes strict scientific correctness from \.347 to \.145 while canonical correctness rises\. For the same 9B source checkpoint, Q8\_0 changes strict scientific correctness to \.226 and audited scientific correctness from \.902 to \.886; audited fraction correctness changes from \.928 to \.910\. These two controlled comparisons show that the observed sensitivity is present at both tested precisions\. They do not establish a general relationship between scale, quantization, and invariance: there is one checkpoint pair, no unquantized baseline, and no replication across model families\.
### 5\.5Completion length and truncation
Table 9:Completion\-token median \[Q1, Q3\] and total length\-finished responses\. Quartiles use the inclusive empirical definition\. Values are comparable across views within a system; tokenizer and chat\-template differences preclude a controlled cross\-system token\-cost comparison\.Within each Qwen run, scientific prompts have longer median completions than canonical prompts, and length finishes concentrate in the scientific view \(13, 85, and 110 respectively\)\. Fraction prompts also lengthen completions for all five systems\. These are descriptive associations under greedy decoding; repeat checks do not establish that the additional tokens are deterministic, nor that length alone causes the accuracy pattern\.
## 6Secondary Analysis: Representation Consensus \(H3\)
#### Design and scope\.
The subset contains 120 bases: the first 30 by identifier from Combined Volume, Difference Distance, Repeated Length, and Fraction of Capacity\. It excludes Rate Speed \(too few applicable views\) and Large\-Volume Addition \(only canonical and scientific views\)\. Thus H3 does not test the largest parser\-interface failure and operates near the accuracy ceiling\.
Each arm uses three calls\. Self\-consistency resamples the canonical prompt\. Paraphrase consensus uses the canonical wording as position 1 and two fixed, manually authored wordings as positions 2–3; all numeric literals and units remain unchanged\. Representation consensus uses canonical, fraction, and unit views, with percent replacing unit for Fraction of Capacity\. The literal paraphrase templates are included in the supplement\. Calls use matched seed positions and caps\. The stochastic tier evaluates all arms; the artifact keydeterministicdenotes greedy decoding and contains the two prompt\-diverse arms\.
At least two equal canonicalized values produce a prediction; otherwise the method abstains\. Appendix[B](https://arxiv.org/html/2609.25009#A2)specifies the no\-abstention fallback and call\-priority sensitivity analysis\.
Figure 3:Non\-unanimous outcomes pooled over five systems\. Solid teal circles mark useful error flags; hatched amber squares mark false alarms\. Denominators are 600 for stochastic self\-consistency and 1,200 for each prompt\-diverse arm across both tiers\. Representation produces 122 flags: one useful flag \(0\.08%\) and 121 false alarms \(10\.1%\) under the default fallback\.
#### H3 is unsupported and the comparison is ceiling\-limited\.
Paraphrase AURC is zero in all ten system\-by\-tier cells\. Representation\-minus\-paraphrase AURC is exactly zero with paired bootstrap interval\[0,0\]\[0,0\]in nine cells\. In the remaining Mistral stochastic cell, the point difference is \.000069 with 95% paired base\-bootstrap interval\[0,\.000419\]\[0,\.000419\]\. The interval includes zero, so no cell supports a representation advantage or a resolved disadvantage\. Selective risk is zero in every paraphrase and representation cell\. These degenerate outcomes provide little power to rank the two methods; they are evidence that H3 is unsupported on this subset, not evidence that the methods are equivalent\.
At the 3\-of\-3 operating point, every unanimous decision is correct\. Representation creates 122 non\-unanimous decisions versus 15 for paraphrase\. A fallback\-order sensitivity analysis \(Appendix[B](https://arxiv.org/html/2609.25009#A2)\) changes at most one useful representation flag and does not alter the false\-alarm pattern\. The result is operationally cautionary but strictly scoped to these four templates and observed error rates\.
## 7Discussion
#### The prompt/parser boundary is a measurement choice\.
The prompt’s generic<number\>placeholder does not specify that only e\-notation is accepted\. Multiplication\-form scientific answers are therefore better described as incompatible with the implemented parser than as unambiguous instruction violations\. Reporting the strict and audited layers together exposes this boundary instead of attributing the entire strict collapse to numerical reasoning\.
#### Parser compatibility does not imply numerical safety\.
Mistral’s unit pattern is the converse: the final line is syntactically accepted but numerically wrong\. A grammar or JSON schema that constrains only the output shape could acceptFINAL: 3630 meters; semantic comparison is still required\. We do not claim every schema would accept it, because a schema could include problem\-specific numeric validation\.
#### Single\-surface correctness is not orbit invariance\.
After audit, orbit correctness still ranges from \.848 to \.981 and invariance from \.851 to \.981\. The strict parser exaggerates the gaps, but a canonical\-only score still misses residual representation sensitivity\. The Qwen comparisons show non\-monotonic outcomes in these specific checkpoints; they do not identify scale or quantization as a general cause\.
#### Disagreement is not automatically confidence\.
Representation views disagree more often than fixed paraphrases on the H3 subset, but almost every additional flag is a false alarm\. A deployment should estimate uncertainty on its own error distribution and document tie\-breaking; diversity alone does not validate an abstention policy\.
## 8Limitations
The benchmark uses synthetic templates and controlled language\. Exact equivalence and labels are strengths, but transfer to naturalistic quantitative text is unmeasured\. Family eligibility is not crossed with every template: scientific notation, percentage, and words each occur in one template, while fraction and unit allocations differ by template\. Pooled family results therefore mix representation with the eligible operation and parameter distribution\.
The study covers five quantized open\-weight systems from three model families on onellama\.cpp/H100 runtime\. The Qwen Q4/Q8 comparison has no unquantized baseline\. Primary runs use one greedy request per prompt\. The ten\-prompt repeat check is small, and Mistral changed one answer; whole\-run repeatability is unknown\. Qwen length\-finished responses also make the generation cap part of the observed scientific result\. GPT\-OSS has a larger cap and an unavoidable reasoning mode, so systems are not matched on effective compute\.
The audit recognizes only a fixed set of surface rewrites\. It can miss other correct values and should not be read as a complete semantic parser\. Its Wilson intervals quantify item\-sampling uncertainty within the frozen benchmark, not uncertainty over templates or model checkpoints\. H3 excludes scientific notation and has near\-zero error in the compared arms; the resulting intervals and sensitivity analysis do not support general claims about representation consensus\.
## 9Reproducibility
The supplementary archive is part of the submission\. It contains the two frozen benchmark JSONL files, source and tests, per\-response evaluation, audit, metrics, and analysis records for the five full runs, the five consensus\-run artifacts with raw responses, stable system aliases, sanitized manifests, figure source data, and a per\-member SHA\-256 manifest\. The full\-run raw response transcripts and tokenizer maps exceed the ancillary size limit and are omitted; they are available from the author on request\. Full model repository revisions and weight hashes are recorded indata/manifests/model\_hashes\.json; for the sharded Mistral artifact, the recorded file hash covers the configured entry\-point shard and is labeled accordingly rather than implying a whole\-model hash\. Raw artifacts remain immutable; regenerated metrics are derived files\.
From the archive root,make paperregenerates consensus analyses and per\-family reports from the frozen records and rebuilds the paper PDF; when the omitted full\-run transcripts are present, it also regenerates evaluations, audits, and figures\.make supplementrebuilds the deterministic supplement ZIP and checksum\. Stable aliases replace timestamped run\-directory names in the archive, and absolute workstation paths are removed\.
## 10Conclusion
Exact orbits make numerical representation invariance measurable, but the measurement boundary matters\. The implemented strict parser is substantially less tolerant than the prompt makes explicit, so its \.574–\.785 orbit correctness is an interface diagnostic rather than a pure arithmetic score\. Under the fixed audit, orbit correctness is \.848–\.981 and invariance is \.851–\.981, with few consistently wrong orbits\. The remaining failures are still consequential: Mistral produces 265 exact power\-of\-ten unit errors concentrated in two templates\. Representation diversity also fails to improve selective prediction on the ceiling\-limited H3 subset and mostly generates false alarms\. Robust evaluation should therefore report correctness and invariance, distinguish parser rejection from wrong values, stratify by eligible task support, and attach uncertainty to any disagreement\-based safeguard\.
## References
- Barker et al\. \(2026\)Malia Barker, Bishal Lakha, Edoardo Serra, and Francesco Gullo\.Testing LLM arithmetic reasoning generalization with automatic numeric\-remapping attacks\.*arXiv preprint arXiv:2606\.03606*, 2026\.
- Bui et al\. \(2025\)Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, and Katharina von der Wense\.On generalization across measurement systems: LLMs entail more test\-time compute for underrepresented cultures\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 21262–21276, Vienna, Austria, 2025\. Association for Computational Linguistics\.doi:10\.18653/v1/2025\.acl\-long\.1032\.
- Chen et al\. \(1998\)Tsong Yueh Chen, Shing Chi Cheung, and Siu Ming Yiu\.Metamorphic testing: A new approach for generating next test cases\.Technical Report HKUST\-CS98\-01, Hong Kong University of Science and Technology, 1998\.
- Cho et al\. \(2025\)Steven Cho, Stefano Ruberto, and Valerio Terragni\.LLMorph: Metamorphic testing of large language models for natural language processing\.*arXiv preprint arXiv:2511\.02108*, 2025\.URL[https://arxiv\.org/abs/2511\.02108](https://arxiv.org/abs/2511.02108)\.
- El\-Yaniv & Wiener \(2010\)Ran El\-Yaniv and Yair Wiener\.On the foundations of noise\-free selective classification\.*Journal of Machine Learning Research*, 11:1605–1641, 2010\.
- Elazar et al\. \(2021\)Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg\.Measuring and improving consistency in pretrained language models\.*Transactions of the Association for Computational Linguistics*, 9:1012–1031, 2021\.doi:10\.1162/tacl\_a\_00410\.
- Geifman & El\-Yaniv \(2017\)Yonatan Geifman and Ran El\-Yaniv\.Selective classification for deep neural networks\.In*Advances in Neural Information Processing Systems 30*, 2017\.
- Gerganov & contributors \(2023\)Georgi Gerganov and contributors\.llama\.cpp: LLM inference in C/C\+\+\.GitHub repository, commit c0bc8591e, 2023\.URL[https://github\.com/ggml\-org/llama\.cpp/tree/c0bc8591e](https://github.com/ggml-org/llama.cpp/tree/c0bc8591e)\.
- Hyun et al\. \(2024\)Sangwon Hyun, Mingyu Guo, and M\. Ali Babar\.METAL: Metamorphic testing framework for analyzing large\-language model qualities\.In*2024 IEEE Conference on Software Testing, Verification and Validation Workshops*, 2024\.doi:10\.1109/ICSTW60714\.2024\.10509967\.
- Kamath et al\. \(2020\)Amita Kamath, Robin Jia, and Percy Liang\.Selective question answering under domain shift\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp\. 5684–5696\. Association for Computational Linguistics, 2020\.doi:10\.18653/v1/2020\.acl\-main\.503\.
- Manakul et al\. \(2023\)Potsawee Manakul, Adian Liusie, and Mark J\. F\. Gales\.SelfCheckGPT: Zero\-resource black\-box hallucination detection for generative large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp\. 9004–9017, Singapore, 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.557\.
- McLeish et al\. \(2024\)Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R\. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein\.Transformers can do arithmetic with the right embeddings\.In*Advances in Neural Information Processing Systems 37*, 2024\.
- Mirzadeh et al\. \(2025\)Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar\.GSM\-symbolic: Understanding the limitations of mathematical reasoning in large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Mistral AI \(2026\)Mistral AI\.Mistral Small 4 119B 2603 model card\.Hugging Face model card, 2026\.URL[https://huggingface\.co/mistralai/Mistral\-Small\-4\-119B\-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)\.
- OpenAI et al\. \(2025\)OpenAI et al\.gpt\-oss\-120b & gpt\-oss\-20b model card\.*arXiv preprint arXiv:2508\.10925*, 2025\.URL[https://arxiv\.org/abs/2508\.10925](https://arxiv.org/abs/2508.10925)\.
- Qwen Team \(2026a\)Qwen Team\.Qwen3\.5\-4B model card\.Hugging Face model card, 2026a\.URL[https://huggingface\.co/Qwen/Qwen3\.5\-4B](https://huggingface.co/Qwen/Qwen3.5-4B)\.
- Qwen Team \(2026b\)Qwen Team\.Qwen3\.5\-9B model card\.Hugging Face model card, 2026b\.URL[https://huggingface\.co/Qwen/Qwen3\.5\-9B](https://huggingface.co/Qwen/Qwen3.5-9B)\.
- Reddy et al\. \(2026\)Varshini Reddy, Craig W\. Schmidt, Seth Ebner, Adam Wiemerslage, Yuval Pinter, and Chris Tanner\.1,729 vs\. 1729: The effect of scripts and formats on LLM numeracy\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pp\. 10679–10696, San Diego, California, United States, 2026\. Association for Computational Linguistics\.doi:10\.18653/v1/2026\.findings\-acl\.518\.
- Ribeiro et al\. \(2020\)Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh\.Beyond accuracy: Behavioral testing of NLP models with CheckList\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp\. 4902–4912\. Association for Computational Linguistics, 2020\.doi:10\.18653/v1/2020\.acl\-main\.442\.
- Sclar et al\. \(2024\)Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\.Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Singh & Strouse \(2024\)Aaditya K\. Singh and DJ Strouse\.Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs\.*arXiv preprint arXiv:2402\.14903*, 2024\.
- Thawani et al\. \(2021\)Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely\.Representing numbers in NLP: a survey and a vision\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp\. 644–656\. Association for Computational Linguistics, 2021\.doi:10\.18653/v1/2021\.naacl\-main\.53\.
- Wallace et al\. \(2019\)Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner\.Do NLP models know numbers? probing numeracy in embeddings\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pp\. 5307–5315, Hong Kong, China, 2019\. Association for Computational Linguistics\.doi:10\.18653/v1/D19\-1534\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V\. Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\.Self\-consistency improves chain of thought reasoning in language models\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Xu et al\. \(2024\)Ancheng Xu, Minghuan Tan, Lei Wang, Min Yang, and Ruifeng Xu\.NUMCoT: Numerals and units of measurement in chain\-of\-thought reasoning using large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pp\. 14268–14290, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-acl\.848\.
## Appendix AFull per\-family results
TableLABEL:tab:appendixreports strict\-parser compatibility for every system and family: canonical and transformed correctness on 1,000 paired bases, the paired gap with 95% CI, Holm\-adjusted exact McNemarpp\-value, and the canonical\-only/transformed\-only discordance\. The scientific rows contain 4,250 canonical\-only discordances in total; Section[5\.2](https://arxiv.org/html/2609.25009#S5.SS2)shows that most are audit\-recoverable\.
Table 10:Strict\-parser paired results\. C/T is canonical/transformed correctness; the gap is C minus T with a 95% base\-bootstrap interval; c/t is canonical\-only/transformed\-only discordance\. Holm adjustment is within each system’s five strict\-parser tests\.SystemFamilyC/TGap \[95% CI\]pHolmp\_\{\\text\{Holm\}\}c/tQwen3\.5\-4B Q4Fraction\.987/\.915\+\.072\+\.072\[\.054,\.091\]1\.3×10−141\.3\\times 10^\{\-14\}82/10Sci\. notation1\.000/\.347\+\.653\+\.653\[\.624,\.682\]2\.7×10−1962\.7\\times 10^\{\-196\}653/0Percentage\.984/\.988−\.004\-\.004\[−\-\.013,\.005\]1\.0009/13Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Unit conversion\.995/\.998−\.003\-\.003\[−\-\.008,\.002\]1\.0002/5Qwen3\.5\-9B Q4Fraction\.989/\.928\+\.061\+\.061\[\.044,\.078\]3\.1×10−123\.1\\times 10^\{\-12\}70/9Sci\. notation\.998/\.145\+\.853\+\.853\[\.830,\.874\]1\.7×10−2561\.7\\times 10^\{\-256\}853/0Percentage\.991/\.977\+\.014\+\.014\[\.004,\.025\]\.03821/7Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Unit conversion\.993/\.997−\.004\-\.004\[−\-\.010,\.001\]\.5782/6Qwen3\.5\-9B Q8Fraction\.990/\.910\+\.080\+\.080\[\.062,\.098\]4\.5×10−184\.5\\times 10^\{\-18\}87/7Sci\. notation\.999/\.226\+\.773\+\.773\[\.746,\.799\]2\.0×10−2322\.0\\times 10^\{\-232\}773/0Percentage\.992/\.988\+\.004\+\.004\[−\-\.004,\.012\]\.90910/6Number words1\.000/\.997\+\.003\+\.003\[\.000,\.007\]\.7503/0Unit conversion\.997/\.999−\.002\-\.002\[−\-\.006,\.002\]\.9091/3GPT\-OSS\-20BFraction\.993/\.995−\.002\-\.002\[−\-\.009,\.005\]1\.0005/7Sci\. notation\.997/\.017\+\.980\+\.980\[\.971,\.988\]9\.8×10−2959\.8\\times 10^\{\-295\}980/0Percentage\.993/\.987\+\.006\+\.006\[−\-\.002,\.014\]\.63011/5Number words1\.000/1\.000\.000\.000\[−\-\.003,\.003\]1\.0000/0Unit conversion\.997/\.971\+\.026\+\.026\[\.016,\.037\]3\.5×10−63\.5\\times 10^\{\-6\}28/2Mistral Small 4Fraction\.967/\.874\+\.093\+\.093\[\.070,\.116\]6\.1×10−156\.1\\times 10^\{\-15\}119/26Mistral Small 4 \(cont\.\)Sci\. notation\.996/\.005\+\.991\+\.991\[\.985,\.996\]4\.8×10−2984\.8\\times 10^\{\-298\}991/0Percentage\.916/\.914\+\.002\+\.002\[−\-\.016,\.020\]\.91444/42Number words\.994/\.997−\.003\-\.003\[−\-\.007,\.000\]\.5000/3Unit conversion\.980/\.699\+\.281\+\.281\[\.252,\.310\]4\.6×10−754\.6\\times 10^\{\-75\}288/7Table 10:Strict\-parser paired results \(continued\)\.
## Appendix BParser, unit registry, and audit specification
#### Strict parser grammar\.
The parser scans for uppercaseFINAL:lines and parses the last, flagging multiplicity\. Accepted numbers match signed integer, decimal with valid comma grouping, integer fractiona/b, or e\-notation forms\. Multiplication\-form scientific notation and mixed numbers are rejected\. One trailing period is tolerated\. An attached percent sign is split and applied as1/1001/100\. Failure precedence is truncation, missing tag, number parse, unit lookup, then value comparison\. This is an operational summary; the included regexes, tests, and unit registry are the complete executable specification\.
#### Unit registry\.
Canonical units and rational factors are liters/milliliters \(1/10001/1000\), kilometers/meters \(1/10001/1000\), meters/centimeters \(1/1001/100\), km/h, and dimensionless decimal/fraction/percent values\. Unit views rewrite only statement quantities; the question requests the canonical unit\.
#### Audit rewrite \(version 2\)\.
For strict\-failed responses, remove\*\*,\*, and backticks; case\-fold theFINALtag; and rewritem×10km\\times 10^\{k\}forms using U\+00D7,x, caret exponents, or Unicode superscripts to e\-notation\. Re\-run the unchanged parser\. Residual parseable values are magnitude drops exactly whenparsed×10k=expected\\mathrm\{parsed\}\\times 10^\{k\}=\\mathrm\{expected\}for an integer0<\|k\|≤150<\|k\|\\leq 15\. The audit changes no raw response and is not a general natural\-language extractor\.
#### Consensus protocol\.
The executable specification inconsensus\.pyfixes the subset, seeds, sampling, and call order\. Paraphrase position 1 is canonical; positions 2–3 use the two fixed template skeletons inrender\.pywhile preserving numbers, units, and operations\. Representation uses canonical, fraction, and unit \(percent for Fraction of Capacity\)\. At least two equal canonicalized values produce a prediction; otherwise the method abstains\. For no\-abstention evaluation, the fallback is the modal value, then the first parsed call when all parsed values differ; if no call parses, the fallback is wrong\. Cyclic call\-priority orders produce 1–2 useful representation flags and 120–121 false alarms, versus 0 useful paraphrase flags and 15 false alarms under every order\. Source fixes acceptance, fallback, and bootstrap details\.相似文章
Vernier: 探究因果推理中词汇缺口背后的表征错位
本文探究了为何指令调优的语言模型在将变量名替换为占位符后,对因果推理问题给出不同答案,发现问题源于表征错位而非信息丢失。作者引入了Vernier方法,通过配对视图权重更新和机制检查,揭示出答案相关内容在占位符视图中仍然存在但错位。
趋同演化:不同语言模型如何学会相似的数字表征
研究发现,尽管架构各异,语言模型在表示数字时会独立演化出相似的周期傅里叶特征,其中只有部分模型在模运算中实现了几何可分性。
当文本与数字不一致时:大型语言模型中的证据仲裁
本文介绍了一个基准测试,用于研究大型语言模型如何仲裁来自文本和数字来源的冲突证据,发现模型使用启发式策略,并偏向于近期性和外部工具。
SPACENUM: 重新审视VLMs中的空间数值理解
本文提出SpaceNum,一个统一的框架,用于评估视觉语言模型(VLMs)在空间上下文中理解数值的能力,发现当前模型在很大程度上未能将数字与空间对应起来,且常常表现出接近随机猜测的性能。
无理解的趋同:语言模型表征一致但推理分歧
本文通过考察来自8个家族的16个语言模型在800个推理问题上的表现,探究了Platonic Representation Hypothesis。研究发现,虽然模型在内部表征上趋于一致,但在推理过程中,尤其是决策后阶段,它们出现分歧,而且共享的表征对预测的因果影响极小。