Competence-Based Evaluation of Logical Reasoning in Large Language Models

# Competence-Based Evaluation of Logical Reasoning in Large Language Models
Source: [https://arxiv.org/html/2610.22894](https://arxiv.org/html/2610.22894)
Chengxiang ZhaiAffiliation:University of Illinois Urbana\-ChampaignEmail:[jizej2@illinois\.edu](mailto:)czhai@illinois\.edu

###### Abstract

Recent advances in large language models \(LLMs\) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation\. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity\. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks\. Such a performance\-based evaluation methodology says little about a model’s generalization capacity in reasoning\. To address this limitation, we argue for a different evaluation paradigm, called*competence\-based evaluation*\(CBE\), which tests whether models reliably apply the same competence across a family of problems\. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer\-preserving presentations test correctness beyond one formulation\. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule\. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family\. Chain\-of\-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held\-out transfer\. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models\.

## 1Introduction

Recent large language models \(LLMs\) achieve strong performance on many reasoning benchmarks, yet growing evidence suggests that high benchmark accuracy does not necessarily imply robust reasoning ability\([Song et al\., 2026](https://arxiv.org/html/2610.22894#bib.bib34)\)\. A single accuracy number on a static test set conflates many things: language understanding, memorization, shortcut exploitation, and genuine reasoning\. It tells us little about*what a model can actually do*\. Recent work has made this conflation increasingly hard to ignore\. Benchmarks designed to test logical reasoning can reward other strategies:[Chung et al\. \(2025\)](https://arxiv.org/html/2610.22894#bib.bib6)report higher GPT\-3\.5 scores on ReClor and LogiQA2 when the prompt asks it to avoid logical reasoning, illustrating that accuracy alone need not diagnose the intended competence\. Models that appear to reason well on standard test sets show systematic drops under controlled perturbations\([Mirzadeh et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib20);[Li et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib16);[Sanyal et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib27)\), and reasoning models can collapse as complexity grows\([Lin et al\., 2025](https://arxiv.org/html/2610.22894#bib.bib24);[Shojaee et al\., 2025](https://arxiv.org/html/2610.22894#bib.bib32)\)\. The natural reading is that reasoning is brittle, but this conflates two questions: whether the model fails to derive a conclusion from a fixed problem representation, or constructs a different representation in the first place\. Measuring performance on a fixed test set is insufficient to tell these apart, and therefore insufficient to characterize a model’s reasoning capability\.

We argue for a different evaluation paradigm:competence\-based evaluation\. Drawing on Chomsky’s distinction between knowledge of a language and its use\([Chomsky, 1965](https://arxiv.org/html/2610.22894#bib.bib5)\), we adapt the competence/performance distinction to reasoning: underlying task knowledge and observed behavior on specific instances\. CBE tests reliable application of that knowledge through an explicit behavioral condition\. We also draw on the cognitive\-science account given by[Stenning and van Lambalgen \(2008\)](https://arxiv.org/html/2610.22894#bib.bib37)for human reasoning, which distinguishes*reasoning to an interpretation*, establishing a problem’s representation and the logic governing it, from*reasoning from an interpretation*, deriving conclusions within that form\. This motivates separating the task’s latent structure from its realization in language\.

A competence claim concerns the family of problems governed by the same rule, including larger instances and equivalent formulations\. Exhaustively enumerating that family is infeasible\. Concretely, instead of measuring accuracy on a fixed test set, we define a*parameterized problem generator*for a target reasoning competence and test a necessary behavioral condition: the model must answer correctly across the generated instances and their answer\-preserving presentations\.The evaluation target is a problem family whose complexity can grow while the required competence remains the same\.The generator exposes the assumptions of that family and permits evaluation at new scales under the same criterion\. A finite experiment can expose violations of the condition; success supports only the evaluated coverage\.

CBE characterizes the model’s*competence boundary*, the region in the generator’s parameter space where correct, invariant behavior deteriorates and exposes biases that aggregate accuracy can conceal\. By holding logical form fixed and varying surface form across matched presentations, the evaluation produces a*competence profile*rather than a single score\. Paired outputs reveal whether correct answers survive equivalent formulations\. They do not by themselves locate a failure in either internal reasoning stage\.

This approach yields several advantages at once\. Fresh generation reduces dependence on particular published test items, although familiarity with the generator or learned shortcuts remains possible\. It produces richer diagnostics than a single accuracy number: the competence profile reveals*where*a model fails and*under which conditions*, including problem size, distractors, and equivalent reformulations\. Additionally, as models improve, the evaluation naturally extends by pushing the parameter range further, rather than requiring an entirely new benchmark\. Existing generators and consistency tests supply important precedents \(§[2](https://arxiv.org/html/2610.22894#S2)\); CBE makes the target competence, its family, and its correctness condition explicit within one evaluation design\.

We instantiate the competence\-based evaluation framework on*ordering tasks*, a fundamental reasoning primitive involving transitivity and comparison\. Ordering is well suited as a first instantiation: the ordering rule is unchanged for any finite number of entities, so an extremely simple task supports a scalable problem family\. The generator controls the latent order and renders matched presentations that vary premise order, converse phrasing, names order, and relation vocabulary \(Figure[1](https://arxiv.org/html/2610.22894#S4.F1)\)\. Failure on this elementary family violates the specified competence condition\. Chain length scales input and presentation load; endpoint queries admit a shortcut and do not isolate multistep inference \(§[4\.1](https://arxiv.org/html/2610.22894#S4.SS1)\)\. Appendix[C](https://arxiv.org/html/2610.22894#A3)illustrates adoption of the framework by Horn\-clause inference\.

Our results show why the distinction between performance and competence matters\. Even strong models fail the CBE condition across the tested ordering family, while performing reasonably well over bounded complexity ranges\. On front/behind atn=1024n=1024, GPT\-4\.1 retains 77\.5% original\-prompt accuracy but answers both matched presentations correctly on only 56\.8% of valid pairs\. Vocabulary changes alter model rankings\. Chain\-of\-thought extends strong performance, and GRPO improves held\-out transfer, but the tested interventions leave substantial failures\. Improved scores can therefore reflect an expanded range of successful performance without demonstrating competence across the family\.

#### Contributions\.

\(1\)Competence\-based evaluation as a general framework\.We formalize the shift from performance on selected instances to a competence condition on a specified problem family, through a parameterized task generator, presentation set, model, correctness criterion, and competence condition \(§[3](https://arxiv.org/html/2610.22894#S3)\)\. \(2\)The CBE Ordering Benchmark\.We instantiate the framework on an elementary ordering task with precisely controlled latent structure and matched presentations\. Its size parameter extends the evaluation while retaining the same ordering rule \(§[4](https://arxiv.org/html/2610.22894#S4)\)\. \(3\)A multidimensional competence analysis\.We evaluate six base models and reasoning/training interventions, showing where correctness fails under complexity and presentation changes \(§[5](https://arxiv.org/html/2610.22894#S5)–§[6](https://arxiv.org/html/2610.22894#S6)\)\. The analysis distinguishes gains in aggregate performance from gains in the range of correct, invariant behavior, and provides diagnostic evidence that can guide future model development\.

## 2Related Work

We position CBE relative to fixed test sets, generated complexity evaluations, and invariance tests\. The distinction is the evaluation target: a defined competence over a scalable problem family, with correctness required across equivalent presentations\. Appendix[D](https://arxiv.org/html/2610.22894#A4)compares the designs by axis\.

### 2\.1Natural\-Language Reasoning Benchmarks

ReClor, LogiQA, AR\-LSAT, and FOLIO assess reasoning on curated or annotated problems\([Yu et al\., 2020](https://arxiv.org/html/2610.22894#bib.bib41);[Liu et al\., 2020](https://arxiv.org/html/2610.22894#bib.bib17);[Zhong et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib42);[Han et al\., 2024a](https://arxiv.org/html/2610.22894#bib.bib11)\)\. DivLogicEval uses counterintuitive content to reduce shortcuts\([Chung et al\., 2025](https://arxiv.org/html/2610.22894#bib.bib6)\), while BIG\-benchlogical\_deductiongenerates linear\-ordering problems at several sizes\([Srivastava et al\., 2023](https://arxiv.org/html/2610.22894#bib.bib35)\)\. These datasets provide valuable samples of task performance\. CBE organizes correctness across equivalent presentations and sizes around an explicit competence condition on a specified problem family\.

### 2\.2Generated and Complexity\-Scaled Evaluation

RuleTaker, ProofWriter, CLUTRR, and PrOntoQA generate deductive or relational instances with controlled structure or depth\([Clark et al\., 2020](https://arxiv.org/html/2610.22894#bib.bib7);[Tafjord et al\., 2021](https://arxiv.org/html/2610.22894#bib.bib38);[Sinha et al\., 2019](https://arxiv.org/html/2610.22894#bib.bib33);[Saparov and He, 2023](https://arxiv.org/html/2610.22894#bib.bib28)\)\. DyVal, functional benchmarks, NPHardEval, and ZebraLogic likewise generate instances or vary complexity\([Zhu et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib45);[Srivastava et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib36);[Fan et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib9);[Lin et al\., 2025](https://arxiv.org/html/2610.22894#bib.bib24)\)\. ReCogLab is especially close: it generates relational reasoning, including transitive comparison, with entity count, reasoning length, and textual features as parameters\([Liu et al\., 2025](https://arxiv.org/html/2610.22894#bib.bib18)\)\. These works establish generation and complexity scaling as useful evaluation tools\. CBE couples that control to an explicit competence condition on matched presentations and evaluates the resulting profile across the family\.

### 2\.3Invariance and Diagnostic Evaluation

CheckList and metamorphic testing prescribe relations between outputs on transformed inputs\([Ribeiro et al\., 2020](https://arxiv.org/html/2610.22894#bib.bib26);[Chen et al\., 1998](https://arxiv.org/html/2610.22894#bib.bib3);[Segura et al\., 2016](https://arxiv.org/html/2610.22894#bib.bib30);[Zhou et al\., 2026](https://arxiv.org/html/2610.22894#bib.bib44)\)\. ParaRel reports consistency and*Consistent\-Acc*, correctness across all paraphrases\([Elazar et al\., 2021](https://arxiv.org/html/2610.22894#bib.bib8)\); BECEL shows that equal accuracy can hide pairwise inconsistency\([Jang et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib15)\)\. RobustLR, GSM\-Symbolic, GSM\-Plus, and premise\-order studies expose sensitivity to controlled changes\([Sanyal et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib27);[Mirzadeh et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib20);[Li et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib16);[Chen et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib4)\)\. These are direct precedents for the paired measurements\. CBE’s distinction is to make such requirements part of the competence definition and trace their satisfaction as complexity grows within one fixed task family\. A robustness score on a chosen set of examples, or an accuracy curve alone, leaves that combined question unanswered\.

## 3Competence\-Based Evaluation

### 3\.1A Competence and Its Problem Family

CBE has five components: a parameterized generatorGθG\_\{\\theta\}, a presentation setΠ\\Pi, a modelℳ\\mathcal\{M\}, a correctness criterion, and a competence condition\. The target competence fixes the problem rule\. Parametersθ\\thetavary instance size or presentation load without changing that rule\. The generator draws a latent instancezz; an exact oracle suppliesy∗​\(z\)y^\{\*\}\(z\)\. Eachπk∈Π⁡\(z\)\\pi\_\{k\}\\in\\Pi\(z\)preserves the answer,y∗​\(πk​\(z\)\)=y∗​\(z\)y^\{\*\}\(\\pi\_\{k\}\(z\)\)=y^\{\*\}\(z\)\.

WriteCk=𝟏\[ℳ\(πk\(z\)\)=y∗\(z\)\]C\_\{k\}=\\mathbf\{1\}\[\\mathcal\{M\}\(\\pi\_\{k\}\(z\)\)=y^\{\*\}\(z\)\]\. The necessary behavioral condition for competence on an instance is

C0=C1=⋯=CK=1\.C\_\{0\}=C\_\{1\}=\\cdots=C\_\{K\}=1\.\(1\)The family\-level condition requires this behavior throughout the specified instance and complexity space\. A finite evaluation can expose failures of this operational condition; it cannot certify competence exhaustively or identify their internal cause\. Equivalent presentations prevent success on one formulation from serving as the entire competence test\.

### 3\.2Profiles of Correctness and Invariance

For two presentations with valid binary answers, letRRbe the probability of both correct,DDboth wrong, andUUexactly one correct\. These robust\-correct, shared\-wrong, and presentation\-dependent masses satisfy

R\+D\+U=1,A¯=R\+U/2,ι=R\+D,R\+D\+U=1,\\quad\\bar\{A\}=R\+U/2,\\quad\\iota=R\+D,\(2\)whereA¯\\bar\{A\}is mean accuracy andι\\iotais invariance\. The*competence profile*isθ↦\(R,D,U\)​\(θ\)\\theta\\mapsto\(R,D,U\)\(\\theta\)\. Accuracy can conceal offsetting changes, and agreement can include shared errors\. Appendix[A](https://arxiv.org/html/2610.22894#A1)gives the elementary identities and marginal bounds\.

We also reportκn=\(ιn−ιe\)/\(1−ιe\)\\kappa\_\{n\}=\(\\iota\_\{n\}\-\\iota\_\{e\}\)/\(1\-\\iota\_\{e\}\), whereιe=q2\+\(1−q\)2\\iota\_\{e\}=q^\{2\}\+\(1\-q\)^\{2\}andqqis the pooled yes\-rate withinnn\. This discounts a common\-bias independent\-answer reference\. It is undefined for constant outputs and does not by itself identify independence or a failure mechanism\. The summaryκ¯\\bar\{\\kappa\}averagesκn\\kappa\_\{n\}over the grid\.

### 3\.3Readout and Inference

Profiles locate regions of deterioration, rather than a single threshold crossing\. Accuracy counts invalid responses wrong on all prompts; pair metrics use jointly parsed answers\. All profile identities use common support\. McNemar tests marginal accuracy differences, including format failures, with Holm correction\. Paired bootstrap intervals resample matched instances\. Decoding settings and statistical details appear in Appendix[B](https://arxiv.org/html/2610.22894#A2)\.

## 4An Instantiation of CBE: Ordering

### 4\.1The Ordering Family

An instance is a total order overnnnamed entities, described by itsn−1n\-1adjacent comparisons\. A yes/no query concerns the two endpoints\. Query direction and gold answer form four balanced cells\. Names and premises are shuffled independently of the latent order\. The ordering rule is unchanged asnnincreases\. We evaluaten∈\{4,8,…,1024\}n\\in\\\{4,8,\\ldots,1024\\\}in the base study\. Figure[1](https://arxiv.org/html/2610.22894#S4.F1)shows generation and paired scoring\.

1\. Sample latent order2\. Render matched prompts3\. Score paired answersIllustration:n=4n=4AmyBobCamDanfront⟶\\longrightarrowbackOriginalπ0\\pi\_\{0\}Names: Cam, Amy, Dan, BobCam is in front of Dan\.Amy is in front of Bob\.Bob is in front of Cam\.Equivalentπ1\\pi\_\{1\}Names: Bob, Dan, Amy, CamBob is behind Amy\.Bob is in front of Cam\.Dan is behind Cam\.Model outputs\(y^0,y^1\)\(\\hat\{y\}\_\{0\},\\hat\{y\}\_\{1\}\)RR: yes / yesBoth correctDD: no / noBoth wrongUU: yes / no or no / yesOne correctShared query:Is Amy in front of Dan?Gold:yesSwap arguments \+ use the converse phrase; shuffle premises and names independently\.Repeat over balanced query cells and chain lengthsnn⟶\\longrightarrowcompetence profile\(R,D,U\)n\(R,D,U\)\_\{n\}\.

Figure 1:Ordering generation and paired evaluation \(illustrative instance\)\. A latent order supplies adjacent premises and a gold endpoint answer\. Equivalent prompts swap arguments with converse phrasing and reshuffle names and premises, preserving the context and query\. Paired answers determine both\-correct \(RR\), both\-wrong \(DD\), or exactly\-one\-correct \(UU\) outcomes; repetition across instances and chain lengths yields the competence profile\.The original presentation uses the forward phrase, such as “Amy is in front of Bob”\. Its equivalent rewrites selected links as “Bob is behind Amy” and reshuffles the names list and premises\. Controls separate reshuffling from rewriting\. A catalog renders the same latent instances in 31 vocabularies across six relation families\. Negation preserves answers under the stated strict total order; distractors over disjoint entities add irrelevant input\.

Ordering supplies a scalable test of reliable application of the same rule\. Herennmeasures input size and presentation load: endpoint queries admit a source/sink shortcut using an incident premise\. Success need not reconstruct the chain\. Failures violate the specified competence condition, but do not isolate a multistep inference limit or distinguish inference failures from retrieval and presentation handling\. Appendix[C](https://arxiv.org/html/2610.22894#A3)illustrates adoption of CBE for Horn\-clause inference\.

### 4\.2Evaluation Setup and Models

The base evaluation uses seed\-1 instances, 200 pairs pernn, 50 per query cell, randomly selected rewrite positions, and identical latent instances across models, vocabularies, and presentation controls\. The six base models are GPT\-4\.1, GPT\-4\.1\-mini, GPT\-5, GPT\-5\-mini, Qwen3\.5\-27B, and Qwen3\.5\-9B; GPT\-5 uses minimal reasoning effort and Qwen thinking is disabled\. Qwen models cover the full catalog and GPT models six spatial\-line vocabularies\. Prompts request yes/no answers under provider\-default sampling\. Caps and parsing rules appear in Appendix[B](https://arxiv.org/html/2610.22894#A2), and decoding\-variability controls in Appendix[H](https://arxiv.org/html/2610.22894#A8)\.

Reasoning\-effort, distractor, and negation studies use 200 matched pairs per complexity level\. All effort levels receive identical prompts within each condition\. The separaten=2048n=2048block changes the presentation distribution, so its failures cannot be attributed solely to length\. Additional models and earlier evaluations are in Appendices[E](https://arxiv.org/html/2610.22894#A5)and[F](https://arxiv.org/html/2610.22894#A6)\.

## 5Competence on Ordering

We read performance through accuracy, invariance, and robust correctness\. Matched comparisons vary input size, vocabulary, presentation factors, and reasoning effort\.

### 5\.1Bounded Performance on Front/Behind

Table 1:Base ordering results pooled overn∈\{4,8,…,1024\}n\\in\\\{4,8,\\ldots,1024\\\}\(200 pairs per level\)\. Acc\.Org/Eqv score all prompts, counting invalid answers wrong; invariance \(Inv\.\) and robust correctness \(RR\) condition on jointly parsed answers\.κ¯\\bar\{\\kappa\}averages chance\-corrected invariance over the complexity grid\. Percentages exceptκ¯\\bar\{\\kappa\}\. Left/right renders the same latent instances with a different vocabulary\.Figure 2:Ordering competence profiles for the six base models on front/behind \(200 pairs per chain length\)\. Mean accuracy and both paired measures use the same valid\-pair support\. Robust correctnessRRrequires both presentations to be correct; invarianceR\+DR\+Dalso includes both\-wrong pairs\. Shading gives pointwise 95% Wilson intervals forRR\. Accuracy can decline less than robust correctness, while shared errors can keep invariance high\. Chance\-corrected profiles for both vocabularies appear in Appendix[E](https://arxiv.org/html/2610.22894#A5)\.GPT\-4\.1 and GPT\-5 perform well on shorter front/behind chains\. Throughn=128n=128, original accuracy remains at least 94\.0% and invariance at least 87\.9%\. Atn=1024n=1024, original accuracy is 77\.5% and 76\.0%, but invariance is only 67\.3% and 67\.0%, withκn=0\.30\\kappa\_\{n\}=0\.30and0\.160\.16\. The smaller GPT models and Qwen3\.5\-27B show earlier deterioration: invariance drops from 87\.0 to 72\.5, 85\.5 to 69\.0, and 95\.0 to 79\.0 betweenn=4n=4and88, respectively\. These profiles support reasonably good performance over bounded ranges, with clear failures of the competence condition across the full tested range\.

#### Accuracy understates paired deterioration\.

Figure[2](https://arxiv.org/html/2610.22894#S5.F2)makes the difference explicit\. On the common valid\-pair support, GPT\-4\.1’s mean accuracy falls from 98\.3% atn=4n=4to 73\.1% atn=1024n=1024, while invariance falls from 97\.5% to 67\.3% and robust correctness from 97\.0% to 56\.8%\. At the latter level, nearly a third of pairs are presentation\-dependent \(U=32\.7%U=32\.7\\%\)\. The pattern is not a universal ordering of decline rates: weak models can acquire shared wrong answers that increase agreement\. Qwen3\.5\-9B, for example, reaches 75\.3% raw invariance atn=8n=8but hasκ¯=0\.00\\bar\{\\kappa\}=0\.00over the grid\. Correctness and invariance must therefore be interpreted jointly\.

Marginal gaps also hide symmetric instability\. GPT\-5\-mini has pooled original/equivalent accuracy of 70\.0/72\.0% and no significant McNemar difference, yetR/D/U=54\.6/12\.6/32\.9%R/D/U=54\.6/12\.6/32\.9\\%\. Its discordant counts are278/314278/314: substantial correct\-to\-wrong changes are offset by changes in the reverse direction\. A small accuracy gap therefore does not establish stable task performance\.

### 5\.2Vocabulary Changes the Profile

Switching only front/behind to left/right changes the rankings \(Table[1](https://arxiv.org/html/2610.22894#S5.T1)\)\. GPT\-4\.1’s original accuracy falls from 93\.1% to 70\.9% andκ¯\\bar\{\\kappa\}from 0\.75 to 0\.33\. Qwen3\.5\-27B instead gains, fromκ¯=0\.37\\bar\{\\kappa\}=0\.37to0\.540\.54\. For GPT\-4\.1 on left/right, the original presentation elicits “yes” on 78% of prompts despite balanced gold answers; equivalent\-presentation accuracy exceeds original accuracy at everyn≥8n\\geq 8\. This is behavioral evidence of a vocabulary\-dependent answer bias\.

The full catalog shows that the base pair alone is insufficient to characterize the family\. Qwen3\.5\-27B’s family\-mean original accuracy andκ¯\\bar\{\\kappa\}are 79\.6%/0\.38 for spatial\-line relations and 87\.2%/0\.58 for comparatives; the corresponding 9B values are 61\.0%/0\.09 and 75\.7%/0\.30\. Across relations, original accuracy spans 27 points and invariance 23 points for a single model\. Nonce comparative terms can be handled well \(27B accuracy/invariance 87\.1/87\.6%\), while nonce direction terms yield only 54\.6% equivalent accuracy andκ¯≤0\.03\\bar\{\\kappa\}\\leq 0\.03for every tested model, even with the converse rule supplied\. Stating the rule is insufficient in this condition; these results do not establish that lexical familiarity is necessary for every way of learning the task\.

### 5\.3Separating Presentation Factors

Reshuffling alone changes GPT accuracy by at most 1\.4 points\. Fully rewriting premises costs GPT\-4\.1 and GPT\-5 a further 9\.1/11\.2 accuracy points and 12\.4/9\.7 invariance points relative to reshuffling alone \(Table[9](https://arxiv.org/html/2610.22894#A5.T9)\)\. The effects of reordering and converse phrasing differ, although both preserve the answer\.

Additional matched controls add distractors or use negation \(Table[10](https://arxiv.org/html/2610.22894#A5.T10)\)\. For GPT\-5\-mini, distractors produce 14\.8% correct\-to\-wrong changes and 14\.6% changes in the reverse direction, leaving aggregate performance almost unchanged\. Negation reduces agreement in all four tested models; for Qwen3\.5\-27B, syntactic and negated\-presentation agreement are 75\.2 and 60\.2%\. Both controls reinforce the need to follow matched instances\.

### 5\.4Reasoning Extends the Strong\-Performance Range

Table 2:GPT\-5\.4 by reasoning effort on front/behind \(200 pairs pernn\)\. Accuracy \(%\) on both presentations, paired invariance, chance correction, and mean reasoning tokens \(original/equivalent\)\. Every effort level uses the same prompts within each condition\. Then=2048n=2048block uses a distinct presentation distribution \(Appendix[B](https://arxiv.org/html/2610.22894#A2)\)\. No shown answer was truncated\.416642561k2k50507575100100Chain lengthnnAccuracy \(%\)Mean accuracynonelowmediumhigh416642561k2k−0\.2\-0\.2000\.50\.511Chain lengthnnκn\\kappa\_\{n\}Chance\-corrected invariance

Figure 3:GPT\-5\.4 reasoning\-effort curves on front/behind \(200 matched pairs per level\)\. Mean accuracy andκn\\kappa\_\{n\}share valid\-pair support\. Curves end atn=1024n=1024; isolatedn=2048n=2048markers, beyond the dashed divider, represent a separate presentation condition \(Appendix[B](https://arxiv.org/html/2610.22894#A2)\)\. More reasoning extends strong performance, while the separate longer\-chain condition retains failures\.Figure[3](https://arxiv.org/html/2610.22894#S5.F3)shows how elicited chain\-of\-thought changes the complexity profile\. On matched front/behind prompts, moving GPT\-5\.4 fromnonetolowincreases original accuracy by 20\.6 points and invariance by 16\.8 points \(95% paired interval 14\.7–18\.8\)\. Medium and high effort maintain near\-perfect aggregate performance throughn=1024n=1024, while low\-effort invariance falls to 78\.0% at that level\. Thus added reasoning extends the range supported by the evaluation\.

It does not establish competence across all tested conditions\. In the separaten=2048n=2048block, high effort attains 94\.0% original accuracy but 65\.0% equivalent accuracy and 67\.0% invariance, without truncation\. This condition also changes the presentation distribution \(Appendix[B](https://arxiv.org/html/2610.22894#A2)\), so the failure cannot be attributed solely to input length\. Effort levels remain matched within the condition\. Equivalent prompts use fewer reasoning tokens than originals at low and medium effort but more at high effort\. Neither additional effort nor available completion budget ensures correct, invariant answers under the tested settings\.

Qwen3\.5\-27B thinking results show a different limitation\. With a 24k\-token budget, accuracy among finished answers is 100\.0/99\.6% and invariance among jointly finished pairs is 99\.9%\. All\-prompt accuracy is 87\.0/79\.1%, and strict invariance is 75\.4% when unfinished pairs count as disagreements\. Conditional success and budget\-limited coverage must both be reported\.

## 6What Training Changes

We compare supervised fine\-tuning \(SFT\) and Group Relative Policy Optimization \(GRPO\) as LoRA interventions on Qwen3\.5\-9B\([Hu et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib13);[Shao et al\., 2024](https://arxiv.org/html/2610.22894#bib.bib31)\)\. Front/behind is held out\. SFT uses auxiliary relation data, paired presentations, or paired presentations with a consistency loss\. GRPO training excludes spatial\-line vocabularies to evaluate transfer to held\-out relation words\.

GRPO samplesG=4G=4answers per prompt, with four presentations of one instance in each batch\. A sampled answer receives reward

ri=ci\+12​bi,r\_\{i\}=c\_\{i\}\+\\tfrac\{1\}\{2\}b\_\{i\},\(3\)wherecic\_\{i\}is one for a correct answer and zero otherwise, andbib\_\{i\}is a nonnegative bonus for cross\-presentation agreement above a reference derived from the instance’s sampled yes/no balance\. Unparseable answers receive zero reward\. Rewards are centered and scaled within each prompt’s sample group to form policy advantages\. Appendix[B\.3](https://arxiv.org/html/2610.22894#A2.SS3)gives the full reward definition and training hyperparameters\.

Table 3:Training comparisons \(n=4n=4to10241024, 200 pairs per level\)\. All adapters use LoRA on Qwen3\.5\-9B\. GRPO reports seed 42, selected for higher mean invariance on the twelve clean held\-out relations; the other seed is in Appendix[G](https://arxiv.org/html/2610.22894#A7)\. JS adds a consistency loss\. Accuracy uses all prompts; pair measures use jointly parsed answers\. Percentages exceptκ¯\\bar\{\\kappa\}\.41664256102450507575100100Chain lengthnnAccuracy \(%\)Mean accuracyBase 9BSFT \+ JSGRPO416642561024−0\.2\-0\.2000\.50\.511Chain lengthnnκn\\kappa\_\{n\}Chance\-corrected invariance

Figure 4:Training changes the ordering complexity profile \(held\-out front/behind, 200 pairs per level\)\. Mean accuracy andκn\\kappa\_\{n\}share valid\-pair support\. The selected GRPO run is compared with the base 9B and SFT with a JS consistency loss\. Gains at smaller sizes coexist with weak long\-chain invariance\.#### SFT learns the training relations but transfers little\.

The supervised variants reach 100% on their held\-in relations\. On front/behind, invariance remains 56\.6–58\.3%, compared with 57\.3% for the base model; the cross\-relation paired change is−0\.4\-0\.4points with interval\[−3\.5,\+2\.5\]\[\-3\.5,\+2\.5\]\. The consistency term produces little additional held\-out improvement\. The larger\-model SFT replicate is in Appendix[F](https://arxiv.org/html/2610.22894#A6)\.

#### GRPO improves transfer under the tested recipe\.

Across twelve clean held\-out relations, GRPO raises invariance from 60\.0% to 83\.5% andκ¯\\bar\{\\kappa\}from 0\.10 to 0\.60\. On the nine relations with no partial phrase overlap, invariance increases from 59\.9% to 82\.1%\. This supports transfer beyond the training vocabulary\. The SFT and GRPO data, objectives, and budgets differ, so the comparison does not establish a general ranking of training methods\.

#### The competence profile still degrades\.

Figure[4](https://arxiv.org/html/2610.22894#S6.F4)resolves the gains by chain length\. On front/behind, GRPO increases original/equivalent accuracy to 81\.2/75\.1% and invariance to 73\.3%\. Atn=256,512,1024n=256,512,1024, its chance\-corrected invariance is−0\.12,−0\.11,−0\.20\-0\.12,\-0\.11,\-0\.20, respectively\. Thus improved pooled performance and cross\-vocabulary transfer coexist with substantial long\-chain presentation failures\. Training extends measured performance without meeting the competence condition across the tested range\.

## 7Discussion and Diagnostic Evidence

The competence profile distinguishes correct answers across presentations from agreement on an incorrect default\. A higher aggregate score can reflect a wider successful range while substantial failures persist elsewhere in the family\. The chain\-of\-thought and training studies illustrate this distinction\.

Open\-weight interventions provide supporting evidence\. In Qwen3\-8B, ablating selected heads raises invariance from 81\.0% to 93\.0% while original/equivalent accuracy falls from 64\.5/59\.5% to 59\.0/52\.0%\. Stable answers can therefore accompany reduced task signal\. In Qwen3\.5\-9B, short\-chain answers remain linearly decodable despite biased outputs; longer\-chain equivalents show reduced decodability at a measured layer and position\. GRPO strengthens the measured signal relative to the answer default\. These results characterize possible contributors without establishing a complete mechanism\. Appendix[I](https://arxiv.org/html/2610.22894#A9)reports the probes and interventions\.

CBE turns such observations into a reusable testing strategy: state the competence, specify a family with controlled complexity, certify answer\-preserving presentations, and report where correctness survives those changes\. This makes model comparisons and intervention gains interpretable in terms of the behavior they actually support\.

## 8Conclusion

We proposed*competence\-based evaluation*as a methodology for evaluating LLM reasoning through a problem family\. Its five components make the target competence and its necessary behavioral condition explicit\. The ordering instantiation tests reliable application of the same rule across growing inputs and equivalent presentations; it does not isolate an inference\-depth limit\.

Even strong models fail that condition across the tested ordering family, despite reasonably good performance over bounded complexity ranges\. Accuracy alone conceals additional deterioration and symmetric changes across equivalent presentations\. Chain\-of\-thought extends successful performance, and GRPO improves held\-out transfer under the tested recipes, but documented failures remain\. Better performance therefore does not establish competence across the family\.

These profiles can guide model development by locating gains across sizes and presentations, while benchmark designers extend coverage under the same competence condition\.

The intended impact is an evaluation practice that grows with models: generate fresh instances, extend the complexity range, and test the same competence under matched presentations\. CBE provides a reusable way to distinguish an expanded range of success from a demonstrated ability to handle the task throughout the evaluated family\.

## Limitations

CBE evaluates competence within a specified problem family\. Finite samples can reveal failures but cannot certify correctness throughout an unbounded family\. Our ordering study and Horn\-clause adoption example leave broader reasoning competences untested\. Extending CBE to richer tasks and wider presentation coverage would clarify how its diagnostic profiles generalize\.

Chain length measures input size and presentation load; endpoint queries admit a shortcut\. Our ordering results therefore do not isolate multistep inference capacity\. Length and presentation changes are also not always separated\. Future generators could independently control derivation depth, distracting information, and linguistic realization\.

Behavioral profiles and linear probes do not uniquely identify internal reasoning processes\. Paired metrics also condition on valid answers, so agreement must be read alongside completion coverage\. Broader sampling and causal interventions could connect systematic behavioral failures to their mechanisms\.

The reasoning and training interventions studied here leave many routes to improved competence unexplored\. Their gains do not establish a general ranking of training methods, particularly when data and budgets differ\. Matched comparisons across training objectives, model families, and compute budgets would help identify which approaches extend correct, invariant behavior beyond the trained range\.

## Ethical Considerations

CBE is intended to support research on reasoning competence\. A potential societal risk is treating success on a generated family as evidence of general reasoning ability or readiness for deployment in consequential settings\. Conversely, failures on this family should not be taken as evidence that a model lacks all useful reasoning abilities\. We mitigate these interpretation risks by specifying the competence condition and evaluation coverage, reporting correctness alongside invariance, and keeping conclusions scoped to the tested family\. Repeated optimization against a known generator may also encourage benchmark\-specific strategies; fresh instances alone do not establish transfer to other reasoning tasks\.

The reported ordering instances assign fictional relations to generic first\-name labels or random strings, and gold answers are computed from the generated structure\. These relations do not describe identifiable people’s actual attributes\. No human participants or crowdworker annotations are used to establish the task labels\. The English templates and name vocabulary do not constitute demographic or multilingual coverage\. Although some templates concern attributes such as wealth or seniority, they should not be interpreted as claims about real people or population groups\.

Model inference, extended reasoning traces, and adapter training consume computational resources and energy\. We reuse pretrained models and train LoRA adapters, but do not claim that these choices establish a low environmental footprint\. We report evaluation and training settings in Appendix[B](https://arxiv.org/html/2610.22894#A2); aggregate energy consumption and carbon emissions were not measured\. Reporting competence gains alongside their computational cost is an important direction for future evaluations\.

## References

- Belinkov \(2022\)Y\. BelinkovProbing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00422),[Link](https://aclanthology.org/2022.cl-1.7/)Cited by:[Appendix I](https://arxiv.org/html/2610.22894#A9.SS0.SSS0.Px1.p1.1)\.
- Berglundet al\.\(2024\)L\. Berglund, M\. Tong, M\. Kaufmann, M\. Balesni, A\. C\. Stickland, T\. Korbak, and O\. EvansThe reversal curse: LLMs trained on “a is b” fail to learn “b is a”\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=GPKTIktA0k)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.4.1.1.1)\.
- Chenet al\.\(1998\)T\. Y\. Chen, S\. C\. Cheung, and S\. M\. YiuMetamorphic testing: a new approach for generating next test cases\.Technical reportTechnical ReportHKUST\-CS98\-01,Department of Computer Science, The Hong Kong University of Science and Technology\.External Links:[Link](https://arxiv.org/abs/2002.12543)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.4.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Chenet al\.\(2024\)X\. Chen, R\. A\. Chi, X\. Wang, and D\. ZhouPremise order matters in reasoning with large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 6596–6620\.External Links:[Link](https://proceedings.mlr.press/v235/chen24i.html)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.3.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Chomsky \(1965\)N\. ChomskyAspects of the theory of syntax\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2610.22894#S1.p2.1)\.
- Chunget al\.\(2025\)T\. T\. Chung, L\. Liu, M\. Yu, and D\. YeungDivLogicEval: a framework for benchmarking logical reasoning evaluation in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 901–915\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.47/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.47),ISBN 979\-8\-89176\-335\-7Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.11.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Clarket al\.\(2020\)P\. Clark, O\. Tafjord, and K\. RichardsonTransformers as soft reasoners over language\.InProceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence, IJCAI\-20,pp\. 3882–3890\.External Links:[Link](https://doi.org/10.24963/ijcai.2020/537)Cited by:[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Elazaret al\.\(2021\)Y\. Elazar, N\. Kassner, S\. Ravfogel, A\. Ravichander, E\. Hovy, H\. Schütze, and Y\. GoldbergMeasuring and improving consistency in pretrained language models\.Transactions of the Association for Computational Linguistics9,pp\. 1012–1031\.External Links:[Link](https://aclanthology.org/2021.tacl-1.60/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00410)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.5.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Fanet al\.\(2024\)L\. Fan, W\. Hua, L\. Li, H\. Ling, and Y\. ZhangNPHardEval: dynamic benchmark on reasoning ability of large language models via complexity classes\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4092–4114\.External Links:[Link](https://aclanthology.org/2024.acl-long.225/)Cited by:[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Gardneret al\.\(2020\)M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala, N\. Gupta, H\. Hajishirzi, G\. Ilharco, D\. Khashabi, K\. Lin, J\. Liu, N\. F\. Liu, P\. Mulcaire, Q\. Ning, S\. Singh, N\. A\. Smith, S\. Subramanian, R\. Tsarfaty, E\. Wallace, A\. Zhang, and B\. ZhouEvaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Online,pp\. 1307–1323\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.117/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.117)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.3.1.1.1)\.
- Hanet al\.\(2024a\)S\. Han, H\. Schoelkopf, Y\. Zhao, Z\. Qi, M\. Riddell, W\. Zhou, J\. Coady, D\. Peng, Y\. Qiao, L\. Benson, L\. Sun, A\. Wardle\-Solano, H\. Szabó, E\. Zubova, M\. Burtell, J\. Fan, Y\. Liu, B\. Wong, M\. Sailor, A\. Ni, L\. Nan, J\. Kasai, T\. Yu, R\. Zhang, A\. Fabbri, W\. M\. Kryscinski, S\. Yavuz, Y\. Liu, X\. V\. Lin, S\. Joty, Y\. Zhou, C\. Xiong, R\. Ying, A\. Cohan, and D\. RadevFOLIO: natural language reasoning with first\-order logic\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 22017–22031\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1229/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1229)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.7.1.1.1),[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Hanet al\.\(2024b\)S\. Han, A\. Yu, R\. Shen, Z\. Qi, M\. Riddell, W\. Zhou, Y\. Qiao, Y\. Zhao, S\. Yavuz, Y\. Liu, S\. Joty, Y\. Zhou, C\. Xiong, D\. Radev, R\. Ying, and A\. CohanP\-FOLIO: evaluating and improving logical reasoning with abundant human\-written reasoning chains\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16553–16565\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.966/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.966)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.7.1.1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§6](https://arxiv.org/html/2610.22894#S6.p1.1)\.
- Huanget al\.\(2025\)K\. Huang, J\. Guo, Z\. Li, X\. Ji, J\. Ge, W\. Li, Y\. Guo, T\. Cai, H\. Yuan, R\. Wang, Y\. Wu, M\. Yin, S\. Tang, Y\. Huang, C\. Jin, X\. Chen, C\. Zhang, and M\. WangMATH\-Perturb: benchmarking LLMs’ math reasoning abilities against hard perturbations\.arXiv preprint arXiv:2502\.06453\.Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.12.1.1.1)\.
- Janget al\.\(2022\)M\. Jang, D\. S\. Kwon, and T\. LukasiewiczBECEL: benchmark for consistency evaluation of language models\.InProceedings of the 29th International Conference on Computational Linguistics,Gyeongju, Republic of Korea,pp\. 3680–3696\.External Links:[Link](https://aclanthology.org/2022.coling-1.324/)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.5.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Liet al\.\(2024\)Q\. Li, L\. Cui, X\. Zhao, L\. Kong, and W\. BiGSM\-plus: a comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2961–2984\.External Links:[Link](https://aclanthology.org/2024.acl-long.163/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.163)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.9.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Linet al\.\(2025\)B\. Y\. Lin, R\. Le Bras, K\. Richardson, A\. Sabharwal, R\. Poovendran, P\. Clark, and Y\. ChoiZebraLogic: on the scaling limits of LLMs for logical reasoning\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 37889–37905\.External Links:[Link](https://proceedings.mlr.press/v267/lin25i.html)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.10.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Liuet al\.\(2025\)A\. Liu, H\. Prior, G\. Balasubramaniam, R\. Moroshko, A\. Zait, I\. Labzovsky, D\. Karmon, I\. Dasgupta, K\. Stachenfeld, and K\. MarinoReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yORSk4Ycsa)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.13.1.1.1),[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Liuet al\.\(2020\)J\. Liu, L\. Cui, H\. Liu, D\. Huang, Y\. Wang, and Y\. ZhangLogiQA: a challenge dataset for machine reading comprehension with logical reasoning\.InProceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence,pp\. 3622–3628\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2020/501),[Link](https://www.ijcai.org/proceedings/2020/501)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.6.1.1.1),[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html)Cited by:[Appendix I](https://arxiv.org/html/2610.22894#A9.SS0.SSS0.Px1.p1.1)\.
- Mirzadehet al\.\(2024\)I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. FarajtabarGSM\-symbolic: understanding the limitations of mathematical reasoning in large language models\.External Links:[Link](https://arxiv.org/abs/2410.05229)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.11.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- nostalgebraist \(2020\)nostalgebraistInterpreting GPT: the logit lens\.Note:LessWrong blog postExternal Links:[Link](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)Cited by:[Appendix I](https://arxiv.org/html/2610.22894#A9.SS0.SSS0.Px1.p1.1)\.
- Parmaret al\.\(2024\)M\. Parmar, N\. Patel, N\. Varshney, M\. Nakamura, M\. Luo, S\. Mashetty, A\. Mitra, and C\. BaralLogicBench: towards systematic evaluation of logical reasoning ability of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13679–13707\.External Links:[Link](https://aclanthology.org/2024.acl-long.739/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.739)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.8.1.1.1)\.
- Patelet al\.\(2024\)N\. Patel, M\. Kulkarni, M\. Parmar, A\. Budhiraja, M\. Nakamura, N\. Varshney, and C\. BaralMulti\-LogiEval: towards evaluating multi\-step logical reasoning ability of large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 20856–20879\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1160/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1160)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.8.1.1.1)\.
- Rabernet al\.\(2026\)B\. Rabern, P\. Mondorf, and B\. PlankLogicSkills: a structured benchmark for formal reasoning in large language models\.External Links:2602\.06533,[Link](https://arxiv.org/abs/2602.06533)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.12.1.1.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 4902–4912\.External Links:[Link](https://aclanthology.org/2020.acl-main.442/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.2.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Sanyalet al\.\(2022\)S\. Sanyal, Z\. Liao, and X\. RenRobustLR: a diagnostic benchmark for evaluating logical robustness of deductive reasoners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 9614–9631\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.653/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.653)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.2.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Saparov and He \(2023\)A\. Saparov and H\. HeLanguage models are greedy reasoners: a systematic formal analysis of chain\-of\-thought\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qFVVBzXxR2V)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.9.1.1.1),[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Schaefferet al\.\(2023\)R\. Schaeffer, B\. Miranda, and S\. KoyejoAre emergent abilities of large language models a mirage?\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 55565–55581\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/adc98a266f45005c403b8311ca7e8bd7-Paper-Conference.pdf)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.7.1.1.1)\.
- Seguraet al\.\(2016\)S\. Segura, G\. Fraser, A\. B\. Sánchez, and A\. Ruiz\-CortésA survey on metamorphic testing\.IEEE Transactions on Software Engineering42\(9\),pp\. 805–824\.External Links:[Document](https://dx.doi.org/10.1109/TSE.2016.2532875)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.4.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§6](https://arxiv.org/html/2610.22894#S6.p1.1)\.
- Shojaeeet al\.\(2025\)P\. Shojaee, S\. I\. Mirzadeh, K\. Alizadeh, M\. Horton, S\. Bengio, and M\. FarajtabarThe illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=YghiOusmvw)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.5.1.1.1),[§1](https://arxiv.org/html/2610.22894#S1.p1.1)\.
- Sinhaet al\.\(2019\)K\. Sinha, S\. Sodhani, J\. Dong, J\. Pineau, and W\. L\. HamiltonCLUTRR: a diagnostic benchmark for inductive reasoning from text\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4506–4515\.External Links:[Link](https://aclanthology.org/D19-1458/)Cited by:[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Songet al\.\(2026\)P\. Song, P\. Han, and N\. GoodmanLarge language model reasoning failures\.External Links:2602\.06176,[Link](https://arxiv.org/abs/2602.06176)Cited by:[§1](https://arxiv.org/html/2610.22894#S1.p1.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastava, A\. Rastogi, A\. Rao,et al\.Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=uyTL5Bvosj)Cited by:[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Srivastavaet al\.\(2024\)S\. Srivastava, A\. M\. B, A\. P\. V, S\. Menon, A\. Sukumar, A\. S\. T, A\. Philipose, S\. Prince, and S\. ThomasFunctional benchmarks for robust evaluation of reasoning performance, and the reasoning gap\.arXiv preprint arXiv:2402\.19450\.External Links:[Link](https://arxiv.org/abs/2402.19450)Cited by:[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Stenning and van Lambalgen \(2008\)K\. Stenning and M\. van LambalgenHuman reasoning and cognitive science\.A Bradford Book,The MIT Press,Cambridge, MA\.External Links:ISBN 978\-0\-262\-19583\-6,[Link](https://mitpress.mit.edu/9780262195836/human-reasoning-and-cognitive-science/)Cited by:[§1](https://arxiv.org/html/2610.22894#S1.p2.1)\.
- Tafjordet al\.\(2021\)O\. Tafjord, B\. Dalvi Mishra, and P\. ClarkProofWriter: generating implications, proofs, and abductive statements over natural language\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,Online,pp\. 3621–3634\.External Links:[Link](https://aclanthology.org/2021.findings-acl.317/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.317)Cited by:[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.
- Viget al\.\(2020\)J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, Y\. Singer, and S\. ShieberInvestigating gender bias in language models using causal mediation analysis\.InAdvances in Neural Information Processing Systems,Vol\.33\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html)Cited by:[Appendix I](https://arxiv.org/html/2610.22894#A9.SS0.SSS0.Px1.p1.1)\.
- Whiteet al\.\(2025\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey, Shubh\-Agrawal, S\. S\. Sandha, S\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. GoldblumLiveBench: a challenging, contamination\-limited llm benchmark\.External Links:2406\.19314,[Link](https://arxiv.org/abs/2406.19314)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.6.1.1.1)\.
- Yuet al\.\(2020\)W\. Yu, Z\. Jiang, Y\. Dong, and J\. FengReClor: a reading comprehension dataset requiring logical reasoning\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2002.04326)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.6.1.1.1),[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Zhonget al\.\(2022\)W\. Zhong, S\. Wang, D\. Tang, Z\. Xu, D\. Guo, Y\. Chen, J\. Wang, J\. Yin, M\. Zhou, and N\. DuanAnalytical reasoning of text\.InFindings of the Association for Computational Linguistics: NAACL 2022,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 2306–2319\.External Links:[Link](https://aclanthology.org/2022.findings-naacl.177/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-naacl.177)Cited by:[§2\.1](https://arxiv.org/html/2610.22894#S2.SS1.p1.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, Y\. Zhu, D\. Antognini, Y\. Kim, and Y\. ZhangParaphrase and solve: exploring and exploiting the impact of surface form on mathematical reasoning in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2793–2804\.External Links:[Link](https://aclanthology.org/2024.naacl-long.153/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.153)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.10.1.1.1)\.
- Zhouet al\.\(2026\)Z\. Zhou, M\. Li, X\. Fang, X\. Zhou, W\. Lin, and Z\. ZhengLGMT: logic\-grounded metamorphic testing for evaluating the reasoning reliability of LLMs\.Knowledge\-Based Systems348,pp\. 116324\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2026.116324),[Link](https://arxiv.org/abs/2605.23965)Cited by:[Table 6](https://arxiv.org/html/2610.22894#A4.T6.2.4.1.1.1),[§2\.3](https://arxiv.org/html/2610.22894#S2.SS3.p1.1)\.
- Zhuet al\.\(2024\)K\. Zhu, J\. Chen, J\. Wang, N\. Z\. Gong, D\. Yang, and X\. XieDyVal: dynamic evaluation of large language models for reasoning tasks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gjfOL9z5Xr)Cited by:[Table 7](https://arxiv.org/html/2610.22894#A4.T7.2.8.1.1.1),[§2\.2](https://arxiv.org/html/2610.22894#S2.SS2.p1.1)\.

## Appendix AProofs and Statistical Notes

### A\.1Paired\-Outcome Identities

For the common valid\-pair support, writeAkA\_\{k\}for accuracy on presentationkkandA¯=\(A0\+A1\)/2\\bar\{A\}=\(A\_\{0\}\+A\_\{1\}\)/2\. The following elementary results clarify what paired observations add to marginal scores\.

###### Result 1\(Exact decomposition\)\.

A¯=R\+U/2=12\+\(R−D\)/2\\bar\{A\}=R\+U/2=\\tfrac\{1\}\{2\}\+\(R\-D\)/2, and feasible\(ι,A¯\)\(\\iota,\\bar\{A\}\)pairs satisfy\|A¯−12\|≤ι/2\|\\bar\{A\}\-\\tfrac\{1\}\{2\}\|\\leq\\iota/2\.

###### Result 2\(Marginals do not identify paired competence\)\.

Given only\(A0,A1\)\(A\_\{0\},A\_\{1\}\), robust correctness ranges over\[max⁡\(0,A0\+A1−1\),min⁡\(A0,A1\)\]\[\\max\(0,A\_\{0\}\+A\_\{1\}\-1\),\\min\(A\_\{0\},A\_\{1\}\)\]\. Every value in this interval is attainable\. A paired flip rate identifiesU=1−ιU=1\-\\iota, but alone does not split agreement intoRRandDD\. Labelled matched pairs identify all three masses\.

#### Proof of Result[1](https://arxiv.org/html/2610.22894#Thmresult1)\.

Answers are binary and both presentations share the gold label, so each matched pair realizes exactly one of the eventsRR\(both correct\),DD\(both wrong\) orUU\(exactly one correct\), andR\+D\+U=1R\+D\+U=1\. The random variable\(C0\+C1\)/2\(C\_\{0\}\+C\_\{1\}\)/2equals11onRR,12\\tfrac\{1\}\{2\}onUUand00onDD; taking expectations givesA¯=R\+U/2\\bar\{A\}=R\+U/2\. SubstitutingU=1−R−DU=1\-R\-DgivesA¯=12\+\(R−D\)/2\\bar\{A\}=\\tfrac\{1\}\{2\}\+\(R\-D\)/2\. Sinceι=R\+D\\iota=R\+DandR,D≥0R,D\\geq 0,\|R−D\|≤ι\|R\-D\|\\leq\\iota, hence\|A¯−12\|≤ι/2\|\\bar\{A\}\-\\tfrac\{1\}\{2\}\|\\leq\\iota/2; the three vertices of the triangle are the pure regimes \(all robust\-correct, all shared\-wrong, all presentation\-dependent\)\. Beyond binary answers, withUUstill meaning “exactly one correct,”A¯=R\+U/2\\bar\{A\}=R\+U/2still holds and agreement is at mostR\+DR\+D, soR≥A¯−\(1−ι\)/2R\\geq\\bar\{A\}\-\(1\-\\iota\)/2remains a conservative bound\. ∎

#### Proof of Result[2](https://arxiv.org/html/2610.22894#Thmresult2)\.

R=Pr⁡\(C0=1,C1=1\)R=\\Pr\(C\_\{0\}=1,C\_\{1\}=1\)is a functional of the joint law of\(C0,C1\)\(C\_\{0\},C\_\{1\}\), whereasA0=Pr⁡\(C0=1\)A\_\{0\}=\\Pr\(C\_\{0\}=1\)andA1=Pr⁡\(C1=1\)A\_\{1\}=\\Pr\(C\_\{1\}=1\)are its marginals\. For fixed marginals, the joint probability of\(1,1\)\(1,1\)ranges over the Fréchet–Hoeffding interval\[max⁡\(0,A0\+A1−1\),min⁡\(A0,A1\)\]\[\\max\(0,A\_\{0\}\+A\_\{1\}\-1\),\\min\(A\_\{0\},A\_\{1\}\)\], and each endpoint and every value between them is attained by a2×22\\times 2table with those marginals \(vary the\(1,1\)\(1,1\)cell and adjust the other three\)\. Any score that is a function of\(A0,A1\)\(A\_\{0\},A\_\{1\}\)therefore constrainsRRto that interval and no further\. A paired flip rate observes only whether the two answers agree, i\.e\.,ι=R\+D\\iota=R\+D, and the two tables\(R,D\)=\(ι,0\)\(R,D\)=\(\\iota,0\)and\(0,ι\)\(0,\\iota\)with the sameUUare indistinguishable to it\. Labelled pairs observe\(C0,C1\)\(C\_\{0\},C\_\{1\}\)directly and so identify all three masses\. ∎

#### Two different chance references\.

The tables use the pooled output\-bias referenceιe=q2\+\(1−q\)2\\iota\_\{e\}=q^\{2\}\+\(1\-q\)^\{2\}, whereqqis the pooled yes\-rate over both presentations within a complexity level\. This corresponds to independent answers with a common marginal rateqq\. If the two presentations have different yes\-ratesq0q\_\{0\}andq1q\_\{1\}, the independence reference with their separate output marginals is insteadq0​q1\+\(1−q0\)​\(1−q1\)q\_\{0\}q\_\{1\}\+\(1\-q\_\{0\}\)\(1\-q\_\{1\}\)\. The difference between the pooled and separate references is\(q0−q1\)2/2\(q\_\{0\}\-q\_\{1\}\)^\{2\}/2\. Thus the pooled convention additionally penalizes differing output biases\. Neither convention supplies a correctness\-based covariance interpretation\.

For correctness indicators, define the separate referenceeC=A0​A1\+\(1−A0\)​\(1−A1\)e\_\{C\}=A\_\{0\}A\_\{1\}\+\(1\-A\_\{0\}\)\(1\-A\_\{1\}\)\. SinceD=1−A0−A1\+RD=1\-A\_\{0\}\-A\_\{1\}\+R,

ι−eC=2​\(R−A0​A1\)=2​Cov​\(C0,C1\)\.\\iota\-e\_\{C\}=2\(R\-A\_\{0\}A\_\{1\}\)=2\\,\\mathrm\{Cov\}\(C\_\{0\},C\_\{1\}\)\.This identity concernseCe\_\{C\}, not the output\-bias reference used in the tables\. Balanced gold labels do not make the two references equal: a perfect predictor on a balanced task haseC=1e\_\{C\}=1, whileq=1/2q=1/2givesιe=1/2\\iota\_\{e\}=1/2\. We therefore do not read the sign of the reportedκ\\kappaas the sign of correctness covariance\. Ifqqis zero or one, the reported normalization is undefined\. All profile identities apply on the same valid\-pair support\.

#### Why the boundary is not reported as a threshold crossing\.

Letn^∗​\(τ\)\\hat\{n\}^\{\*\}\(\\tau\)denote the smallestnnat whichι^n\\hat\{\\iota\}\_\{n\}falls below a thresholdτ\\tau\. A parametric bootstrap \(binomial resampling of eachι^n\\hat\{\\iota\}\_\{n\}at the observed sample size,B=4,000B=4\{,\}000\) on the Protocol L base pair shows that the probability of reproducing the observed crossing atτ=0\.85\\tau=0\.85in a fresh dataset of 200 pairs pernnis0\.500\.50for GPT\-5 and0\.390\.39for GPT\-4o on front/behind, and remains0\.540\.54–0\.640\.64for GPT\-5 even at 400–3,200 pairs pernnbecause its curve sits close toτ\\tauat the crossing, whereas it is0\.870\.87–1\.001\.00for models whose curves cross steeply \(GPT\-4\.1, GPT\-4\.1\-mini, Qwen3\.5\-9B\)\. A curve that happens to sit on the threshold makes the crossing a coin flip at any sample size\. The shape of the profile, by contrast, is stable under resampling, which is why the main text describes where profiles drop steeply or diverge and summarizes withκ¯\\bar\{\\kappa\}\.

#### Sample sizes\.

At 200 pairs pernnthe 95% interval onιn\\iota\_\{n\}has half\-width±6\.9\\pm 6\.9points atι=0\.5\\iota=0\.5and±4\.9\\pm 4\.9atι=0\.85\\iota=0\.85; the overallι\\iotaover the ninennlevels \(1,800 pairs\) has half\-width±2\.3\\pm 2\.3points, and a paired difference inι\\iotabetween two arms on the same instances of about3\.73\.7points is detectable at 80% power\. Per\-cell accuracy within onenn\(50 instances\) has half\-width up to±14\\pm 14points, which is why per\-cell polarity claims are pooled overnn\.

## Appendix BGenerators, Protocols, and Training Details

For reproducibility, this appendix names the archived presentation conditions: Protocol P denotes the position\-uniform datasets used in the base evaluation, while Protocol L denotes the release datasets, including the matched GPT\-5\.4 reasoning\-effort comparison\. These labels identify dataset construction, not different competence criteria\. All effort levels in a condition use the same paired prompts\.

### B\.1Ordering Generator

#### Entity pool and premises\.

For theposrelation, names are sampled without replacement from a pool of about2,5002\{,\}500first names; a random\-name control replaces them with random five\-letter strings\. For each instance the generator samplesnnnamesa0,…,an−1a\_\{0\},\\ldots,a\_\{n\-1\}, emits then−1n\-1adjacent comparisonsai≺ai\+1a\_\{i\}\\prec a\_\{i\+1\}rendered in the relation’s forward phrase, and shuffles both the premise order and the names list independently of the chain\. The query is the endpoint pair\(a0,an−1\)\(a\_\{0\},a\_\{n\-1\}\)with direction and gold answer balanced over the four cells\.

#### Prompt format\.

A representative Protocol P prompt atn=8n=8, verbatim:

> There are 8 people standing in some order\. Their names are \[’Heather’, ’Charles’, ’Joseph’, ’James’, ’Andrew’, ’Michelle’, ’Mark’, ’Donna’\]\. Their standing order can be described by the following set of rules: James is in front of Andrew\. Joseph is in front of James\. Mark is behind Donna\. Heather is in front of Charles\. Donna is in front of Joseph\. Andrew is in front of Michelle\. Charles is in front of Mark\. Is Heather in front of Michelle? Provide your answer only as yes or no\. Answer:

The system prompt, identical for every model and for training, is “You are a helpful assistant\. Answer logical reasoning questions concisely\.”

#### Equivalent presentation under the two protocols\.

Under both protocols the equivalent presentation rewritesmmof then−1n\-1premises with the inverse phrase and swapped arguments and re\-shuffles the premise order and the names list\. Under Protocol L \(the release generator\),mmis drawn uniformly from\{max⁡\(1,⌊0\.2​\(n−1\)⌋\),…,n−1\}\\\{\\max\(1,\\lfloor 0\.2\(n\-1\)\\rfloor\),\\ldots,n\-1\\\}and the rewritten premises are the lastmmcomparisons in chain order forn≤1024n\\leq 1024, while atn=2048n=2048each premise is rewritten independently with probability1/2\\nicefrac\{\{1\}\}\{\{2\}\}\. Under Protocol P the same instances, names lines, premise orders and rewrite counts are kept and the rewritten positions are redrawn uniformly over the chain, so that relation phrasing is uncorrelated with chain position\. On the same instances, equivalent\-presentation accuracy is lower under Protocol P than under Protocol L in 43 of 50 \(model, vocabulary\) cells, by up to7\.77\.7points \(GPT\-5 on before/after,90\.5→82\.890\.5\\to 82\.8\); invariance is lower by22–44points for GPT\-5 on seven of eight arms and for GPT\-4\.1 on four of eight, and within noise for the mini models, both Qwen models and the LoRA adapter on all but two arms; original\-presentation accuracy is shared by construction\. The per\-nntrends in Appendix[F](https://arxiv.org/html/2610.22894#A6)that crossn=1024→2048n=1024\\to 2048therefore cross a change in the presentation distribution and are not interpreted\. The release code ships both samplers\.

#### Catalog\.

The 31 relations are rendered from the seed\-1 instances with the same orderings, name\-to\-position assignment, names line, premise order, rewrite count and rewrite positions aspos\(871 consistency checks; theposfiles are byte\-identical\)\. Families and phrases: spatial line \(in front of/behind, ahead of/behind, left/right, right/left, behind/in front of, before/after, north/south, east/west, closer to the front/back than, further forward/back than\); vertical \(above/below, higher up/lower down than\); temporal \(earlier/later to arrive, born before/after, earlier/later to finish, scheduled before/after\); comparative \(older, taller, shorter, heavier, faster, stronger, richer, each with its antonym\); rank and social \(more senior/junior, ranked higher/lower, more/less experienced, more/less popular, paid more/less, closer to/farther from the exit\); nonce \(more/less zorbic than; vorpward/morpward of, with the converse rule stated in the context\)\.

### B\.2Evaluation Protocol

#### Decoding\.

Provider\-default sampling was used throughout \(OpenAI chat completions at temperature1\.01\.0; Qwen models served by vLLM at temperature0\.60\.6, top\-pp0\.950\.95, top\-kk2020\)\. Completion caps:2020tokens for GPT\-4o, GPT\-4o\-mini, GPT\-4\.1, GPT\-4\.1\-mini and the Qwen models \(thinking disabled\); the GPT\-5 family atreasoning\_effort=minimalwithmax\_completion\_tokens=1024=1024\(observed reasoning tokens zero\); GPT\-5\.4 atnonewith the same cap and atlow/medium/highwith a32,76832\{,\}768\-token cap \(65,53665\{,\}536formediumandhighin then=2048n=2048block\); Qwen3\.5\-27B with thinking enabled at a2424k budget\. Identical\-prompt repeat agreement and its interpretation are reported in Appendix[H](https://arxiv.org/html/2610.22894#A8)\. Unparseable rates are below1%1\\%for every Protocol P model except Gemma\-4\-31B \(Appendix[E\.1](https://arxiv.org/html/2610.22894#A5.SS1)\)\.

The matched GPT\-5\.4 effort comparison uses Protocol L equivalents\. With Protocol P equivalents, the poolednonerow is 77\.1% original accuracy, 74\.9% equivalent accuracy, 78\.1% invariance, andκ¯=0\.40\\bar\{\\kappa\}=0\.40\.

#### Statistics\.

McNemar uses all\-prompt correctness, counting invalid answers wrong, with the continuity\-correctedχ2\\chi^\{2\}forb\+c\>25b\+c\>25and the exact binomial test otherwise\. Holm correction is across the compared models\. Bootstrap intervals carry all presentations of a matched instance together; the reported catalog and GRPO contrasts use2,0002\{,\}000resamples stratified by relation\. The new robust\-correct curves use pointwise95%95\\%Wilson intervals on the same jointly parsed support as all three plotted quantities\. Figure[2](https://arxiv.org/html/2610.22894#S5.F2)is recomputed from archived answers, with no new model calls\. Its source table records valid\-pair counts and input\-file hashes\.

### B\.3Training Details

#### Supervised adapters\.

The cross\-relation LoRA fine\-tune of Qwen3\.5\-9B uses45,60045\{,\}600chat\-formatted examples from5,7005\{,\}700orderings \(950950per relation,n≤128n\\leq 128\) over six auxiliary relations \(arrival order, priority, proximity, seniority, left/right, up/down\), each ordering contributing its four query cells under both presentations;posis held out\. LoRA rank1616,α=32\\alpha=32, dropout0\.050\.05, all attention and MLP projections, AdamW at2×10−42\\times 10^\{\-4\}with cosine schedule and0\.030\.03warm\-up, one epoch, effective batch1616, bf16,max\_length15361536, seed4242, one GPU\. The presentation\-pair SFT and the consistency\-loss variant \(a Jensen–Shannon term between the answer distributions of the presentations of an instance, with a chance\-corrected guard\) use the same pool\. The identical recipe was applied to Qwen3\.5\-27B\.

#### GRPO\.

GRPO adapters train on batches holding every presentation of an instance \(original, rewritten, reshuffled, fully rewritten\) withG=4G=4samples per prompt, learning rate10−510^\{\-5\},β=0\\beta=0,600600instances atn∈\[16,128\]n\\in\[16,128\],600600steps, and seeds4242and4343\. Training uses1919relations without spatial\-line vocabulary, of whichabove/below,taller/shorterandhigher/lowershare words with three evaluated relations that are therefore excluded from the clean held\-out set\.

For a parsed answerii, the implemented reward is

ri\\displaystyle r\_\{i\}=ci\+12​bi,\\displaystyle=c\_\{i\}\+\\tfrac\{1\}\{2\}b\_\{i\},\(4\)bi\\displaystyle b\_\{i\}=max⁡\{0,ai−\[qz2\+\(1−qz\)2\]\},\\displaystyle=\\max\\\{0,a\_\{i\}\-\[q\_\{z\}^\{2\}\+\(1\-q\_\{z\}\)^\{2\}\]\\\},whereci=𝟏\[parse\(oi\)=y∗\(z\)\]c\_\{i\}=\\mathbf\{1\}\[\\operatorname\{parse\}\(o\_\{i\}\)=y^\{\*\}\(z\)\],aia\_\{i\}is the fraction of parsed answers on the instance’s other presentations matching answerii, andqzq\_\{z\}is the yes\-rate among all parsed samples of the instance\. Unparseable answers receive zero reward\. For each promptpp, advantages are

Ai=ri−μpσp\+10−4,A\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{p\}\}\{\\sigma\_\{p\}\+10^\{\-4\}\},\(5\)whereμp\\mu\_\{p\}andσp\\sigma\_\{p\}are the mean and standard deviation of its four sampled rewards\. Appendix[G](https://arxiv.org/html/2610.22894#A7)analyzes the reward components under this normalization and reports the additional training seed\.

## Appendix CAdopting CBE for Horn\-Clause Inference

### C\.1Mapping the Framework to a New Task

This appendix illustrates adoption of CBE beyond the ordering benchmark\. The latent generator is a Horn program over pronounceable pseudoword atoms\. An exact forward\-chaining oracle supplies the gold answer to a goal\-derivability query\. The complexity parameter is the number of rules,n∈\{4,8,16,32,64,128\}n\\in\\\{4,8,16,32,64,128\\\}\. Positive programs derive the goal; negative near misses retain a rule headed by the goal but sever its derivation at one point\. Gold answers are balanced\. Rule count indexes program size\. Positive proof depth and distractor content co\-vary with rule count, so these profiles do not isolate proof\-depth effects\.

The two renderers express each rule either as “Ifaaandbb, thencc” or as “ccholds wheneveraaandbbhold”\. Rule order stays fixed\. The model and competence criterion are unchanged: both answers must be correct\. The same\(R,D,U\)\(R,D,U\)profile, invariance, and chance correction apply\. Thus adoption requires task\-specific latent instances, an oracle, and semantics\-preserving renderers, without changing the outcome decomposition or analysis\. This demonstrates a second instantiation; it does not measure implementation effort or imply that arbitrary presentation equivalence can be certified automatically\.

### C\.2Illustrative Results

Table 4:Horn\-clause inference pooled overn∈\{4,…,128\}n\\in\\\{4,\\dots,128\\\}rules \(Protocol L, 200 pairs pernn, pairs whose two answers both parse\)\. Accuracy \(%\) on both presentations, per\-pair invariance,κ¯\\bar\{\\kappa\}, and the joint readoutRR/DD/UU\(both correct / both wrong / exactly one correct, %\)\. No re\-sampling floor was measured in this run\.Eight models receive 200 pairs per rule count under Protocol L; no identical\-prompt retest was collected\. The paired profile again reveals information beyond marginal accuracy\. Pooled original/equivalent accuracy differs by less than two points for every model, whileUUranges from 18\.0% to 31\.6%\. Atn=4n=4, six models haveκn\\kappa\_\{n\}between 0\.67 and 0\.94\. Byn=32n=32, all eight models have original accuracy within four points of chance\. Qwen3\.5\-27B has the largest pooled robust\-correct mass,R=55\.4%R=55\.4\\%, alongsideD=26\.6%D=26\.6\\%andU=18\.0%U=18\.0\\%\. Its 82\.0% invariance therefore includes substantial shared error\. Table[5](https://arxiv.org/html/2610.22894#A3.T5)resolves the three masses by rule count\. These results are an adoption example with narrower protocol coverage than the main ordering study\.

Table 5:Horn\-clause inference by number of rulesnn\(Protocol L, 200 pairs pernn\)\.RR/DD/UU: percentage of jointly parseable pairs with both presentations correct / both wrong / exactly one correct\. Chance correction usesκn=\(ιn−ιe\)/\(1−ιe\)\\kappa\_\{n\}=\(\\iota\_\{n\}\-\\iota\_\{e\}\)/\(1\-\\iota\_\{e\}\)withιn=R\+D\\iota\_\{n\}=R\+Dandιe=q2\+\(1−q\)2\\iota\_\{e\}=q^\{2\}\+\(1\-q\)^\{2\}, whereqqis the pooled yes\-rate on the same support\. GPT\-4o and Qwen3\.5\-9B lose a few pairs to unparseable answers\. No retest control was collected\.

## Appendix DPer\-Axis Comparison with Prior Work

Tables[6](https://arxiv.org/html/2610.22894#A4.T6)and[7](https://arxiv.org/html/2610.22894#A4.T7)compare the designs behind Section[2](https://arxiv.org/html/2610.22894#S2)along generation, complexity control, presentation changes, and diagnostic readouts\. The descriptions distinguish tests of equivalent presentations of a fixed instance from changes to the answer, query, or reasoning problem\. Several predecessors provide parameterized generation or paired consistency and correctness measures, including ParaRel Consistent\-Acc\. CBE builds on these contributions by connecting matched complexity profiles to an explicit behavioral condition on a problem family\.

Table 6:Comparison of generation, complexity control, presentation changes and diagnostic readouts in the cited evaluation designs\. Descriptions identify the kind of change being tested, including whether the answer or query changes\. Further comparisons appear in Table[7](https://arxiv.org/html/2610.22894#A4.T7)\.Table 7:Further comparison of the evaluation designs described in the cited sources\. Generating fresh inputs, varying problem complexity, testing presentation changes and reporting diagnostic measurements have multiple precedents\. CBE connects these elements through its specified competence condition and matched profiles\.
## Appendix EAdditional Models and Per\-nnTables

### E\.1Additional Open\-Weight Models on a Nine\-Relation Core

Three further open\-weight models were run under Protocol P on a nine\-relation core \(front/behind, left/right, north/south, above/below, born before/after, taller, shorter, ranked higher/lower, nonce direction\): Gemma\-4\-31B, Qwen3\.6\-35B\-A3B \(3B active parameters\) and Qwen2\.5\-32B\-Instruct\. Table[8](https://arxiv.org/html/2610.22894#A5.T8)gives family means on the matched core\. Two observations\. First, the local ladder is not monotone in model generation: Qwen2\.5\-32B exceeds Qwen3\.5\-27B on invariance andκ¯\\bar\{\\kappa\}in every family and on original accuracy in every family but temporal, with the highest re\-sampling floors \(95–98\) and the largest deficit relative to its retest agreement \(−12\.9\-12\.9over the core,−23\-23on north/south, where original accuracy9090falls to7474on the equivalent\)\. Second, Gemma\-4\-31B does not fail the task so much as abandon the answer format: under the 20\-token cap it begins a chain of thought instead of answering on29/84%29/84\\%\(original/equivalent\) of front/behind pairs atn=128n=128and36/68%36/68\\%atn=256n=256, so its invariance over parsed pairs \(9696–9898\) hides a strict invariance of1414–4444atn=128n=128, and its accuracy columns measure format compliance; its parsed answers are9393–100%100\\%correct at everyn≤128n\\leq 128on every lexical relation, and its greedy first\-token readout is9696–97%97\\%correct atn=512n=512with the gold answer decodable from the residual stream at0\.940\.94–1\.001\.00throughn=512n=512in the probed layers\. A free\-form condition with a1,0241\{,\}024\-token budget changes almost nothing\. All four fail the nonce direction words \(κ¯≤0\.03\\bar\{\\kappa\}\\leq 0\.03\)\.

Table 8:Means over the matched nine\-relation core \(Protocol P\)\. Inv\. strict counts a pair with an unparseable answer as a disagreement; Gemma’s gap between the two columns is format abandonment under the 20\-token cap\. The last row is the correctness\-only GRPO adapter of §[6](https://arxiv.org/html/2610.22894#S6)\.
### E\.2Presentation Components, Distractors, and Negation

Table[9](https://arxiv.org/html/2610.22894#A5.T9)separates the components of the equivalent presentation for the two flagship GPT models under Protocol P, and Table[10](https://arxiv.org/html/2610.22894#A5.T10)reports the distractor\-dose and negation runs under Protocol L \(§[5\.3](https://arxiv.org/html/2610.22894#S5.SS3)\)\.

Table 9:What the presentation change is made of: Acc\.Eqv / Inv\. \(%\) on the front/behind instances of Protocol P when the equivalent presentation changes only the premise order, only the fraction of premises re\-worded with the converse relation \(order fixed\), or both\. Original\-presentation accuracy is 93\.1 \(GPT\-4\.1\) and 94\.1 \(GPT\-5\) in every row\. The first row re\-sends the original prompt unchanged\.Table 10:Per\-pair invariance \(%\) pooled overn∈\{4,…,256\}n\\in\\\{4,\\dots,256\\\}under Protocol L \(see Section[4](https://arxiv.org/html/2610.22894#S4)\)\. Left: the syntactic equivalent withround⁡\(ρ⁡\(n−1\)\)\\mathrm\{round\}\(\\rho\\,\(n\-1\)\)distractor premises over disjoint entities added to both presentations\. Right: the same items and premise subsets with each re\-worded premise expressed as the negation of its converse, and the agreement of the original prompt with itself re\-sent \(retest\)\. Theρ=0\\rho=0column is the syntactic family\.
### E\.3Output Decomposition at the Answer Position

Table[11](https://arxiv.org/html/2610.22894#A5.T11)gives the output decomposition behind §[7](https://arxiv.org/html/2610.22894#S7), and Figure[5](https://arxiv.org/html/2610.22894#A5.F5)the per\-layer gold\-probe curves\.

RelationAcc\.Yesδ\\deltassLayer*Qwen3\.5\-9B, base*front/behind55\.5/59\.596/921\.58/1\.140\.63/0\.5016/18left/right69\.5/68\.582/841\.67/1\.411\.65/1\.3813/13nonce dir\.56\.0/55\.596/881\.19/0\.720\.57/0\.1417/–*Qwen3\.5\-9B, correctness\-only GRPO*front/behind87\.5/87\.064/581\.37/0\.363\.60/3\.2213/17left/right97\.5/99\.055/53−\-0\.22/−\-0\.467\.64/7\.0913/13nonce dir\.90\.5/60\.056/36−\-0\.06/−\-1\.393\.59/1\.0016/–*Qwen3\-8B, base*front/behind64\.5/59\.582/862\.42/3\.523\.54/2\.3319/22

Table 11:Output decomposition at the answer position \(Protocol P instances,n≤32n\\leq 32, 200 pairs per relation, greedy readout\)\. Each cell is original/equivalent presentation\. Acc\. and yes\-rate \(Yes\) in %;δ\\delta= presentation\-level default toward “yes” andss= instance signal, in logits; Layer = first layer at which the cross\-validated gold probe reaches AUROC 0\.9 \(– = never; the Qwen3\-8B replicate has 36 layers, the 9B 32\)\. Nonce dir\. = nonce direction words\.Figure 5:Gold\-probe AUROC by layer at the answer position, Qwen3\.5\-9B \(Protocol P instances, greedy readout,n≤32n\\leq 32, 200 pairs per relation\): base and correctness\-only GRPO on front/behind, original \(solid\) and equivalent \(dashed\), and the equivalent presentation of the nonce direction words\. Grey band: shuffled\-label null; dotted line: AUROC 0\.9\.
### E\.4Reasoning Effort by Chain Length

Figure 6:GPT\-5\.4 by reasoning effort: per\-pair invariance \(top\) and original\-presentation accuracy \(bottom\) bynnunder front/behind \(left\) and left/right \(right\)\. Dotted: identical\-prompt self\-agreement; squares: then=2048n=2048block\. Thenonecurves use the Protocol P instances with the Protocol L equivalents;low,medium,highand then=2048n=2048block are Protocol L equivalents on the same instances\.
### E\.5Per\-nnTables

Figure[7](https://arxiv.org/html/2610.22894#A5.F7)gives chance\-corrected invariance for both base vocabularies\. Table[12](https://arxiv.org/html/2610.22894#A5.T12)gives per\-nnaccuracy on both presentations and invariance for the six Protocol P models on front/behind\. Horn\-clause profiles are in Appendix[C](https://arxiv.org/html/2610.22894#A3)\.

Figure 7:Chance\-corrected invarianceκn\\kappa\_\{n\}by chain length for the six base models under front/behind \(left\) and left/right \(right\), Protocol P\. Solid lines denote the larger and dashed lines the smaller model of each family\. The dotted reference isκn=0\\kappa\_\{n\}=0under the pooled output\-bias convention\.Table 12:Front/behind under Protocol P, pernn\(200 pairs each\): accuracy on the original and equivalent presentations and per\-pair invariance \(%\)\.

## Appendix FResults under the Release Protocol

This appendix collects the results obtained under Protocol L \(the benchmark release: eight models,400400pairs pernn,n≤2048n\\leq 2048, no re\-sampling floor; Appendix[B\.1](https://arxiv.org/html/2610.22894#A2.SS1)\)\. They are the basis of an earlier version of this work and are reported here, with the figures recomputed from the released outputs, because the eight\-model suite includes the GPT\-4o family and then=2048n=2048level, neither of which was re\-run under Protocol P\. Comparisons across the two protocols are not drawn\.

### F\.1Eight Models on the Release Protocol

Table[13](https://arxiv.org/html/2610.22894#A6.T13)summarizes accuracy and invariance pooled overn∈\{4,…,2048\}n\\in\\\{4,\\ldots,2048\\\}, Table[14](https://arxiv.org/html/2610.22894#A6.T14)the McNemar tests, and Table[15](https://arxiv.org/html/2610.22894#A6.T15)the per\-nninvariance; Figure[8](https://arxiv.org/html/2610.22894#A6.F8)plots the latter\. The qualitative picture agrees with §[5\.1](https://arxiv.org/html/2610.22894#S5.SS1): GPT\-4\.1 and GPT\-5 keep invariance above87%87\\%throughn=128n=128and fall steeply afterwards, every other model is at or below85%85\\%byn=8n=8except Qwen3\.5\-9B, and high invariance at largennfor GPT\-4o\-mini and Qwen3\.5\-27B coincides with both presentations defaulting to “no” \(accuracy4545–55%55\\%\), i\.e\., shared\-wrong mass rather than competence\. Qwen3\.5\-9B’s9191–98%98\\%agreement atn=8n=8–3232coincides with50%50\\%accuracy and is a yes\-default; atn≥256n\\geq 256its agreement falls below chance\. Six of eight models reject the marginal null at Holm\-correctedp<0\.001p<0\.001, yet the per\-nndiscordant counts change sign within rows for GPT\-4o and GPT\-4\.1\-mini \(original favoured at smallnn, equivalent atn=128n=128or512512–10241024\), which the aggregate cancels\. Seven of eight models prefer the backward query atn=2048n=2048by66–2323points, and six of eight collapse onto a “no” default there, with terminal polarity gaps from about1010points \(GPT\-4o\) to8383\(Qwen3\.5\-27B\)\.

Table 13:Per\-model accuracy and invariance on theposdataset \(N∈\{4,8,…,2048\}N\\\!\\in\\\!\\\{4,8,\\ldots,2048\\\}, 4000 questions per split\)\.Acc\. OrgandAcc\. Eqvare accuracy on original and equivalent splits;Δ\\Deltais their difference;Inv\.is the fraction of paired questions where the model gave the same yes/no on both phrasings \(over pairs with parseable answers\)\. All values are percentages\.Table 14:McNemar’s test on the original/equivalent accuracy gap, per model on theposdataset \(N=4000N=4000paired questions\)\.bbis the number of pairs with org correct and eqv wrong;ccis the reverse\. Gap = \(org\_acc−\-eqv\_acc\), with 95% paired\-bootstrap CI \(B=10000B=10000\)\.pMcNemarp\_\{\\text\{McNemar\}\}uses the continuity\-correctedχ2\\chi^\{2\}statistic;pHolmp\_\{\\text\{Holm\}\}is the Holm\-Bonferroni adjustedppacross the 8 models\.Table 15:Per\-model invariance on theposdataset, by problem sizeNN\. Invariance is the fraction of \(original, equivalent\) pairs receiving the same yes/no answer \(over pairs with parseable answers\)\. Values are percentages\.Figure 8:Per\-pair invariance bynnfor the eight models under Protocol L \(front/behind, 400 pairs pernn\)\. Dotted line: chance agreement\.
### F\.2Supervised Fine\-Tuning under the Release Protocol

The cross\-relation LoRA fine\-tune of Qwen3\.5\-9B \(Appendix[B\.3](https://arxiv.org/html/2610.22894#A2.SS3)\) evaluated on the held\-outposrelation under Protocol L reduces the marginal gap from6\.56\.5to0\.60\.6points without raising accuracy \(60\.1→57\.660\.1\\to 57\.6mean\), while its per\-pair invariance moves from64\.664\.6to57\.257\.2and sits at chance \(4848–50%50\\%\) for everyn≥128n\\geq 128; the identical recipe on Qwen3\.5\-27B changes neither accuracy \(74\.774\.7\) nor the gap \(5\.2→5\.65\.2\\to 5\.6\)\. The marginal gap and the instance\-level invariance thus move in opposite directions on the same checkpoint, which is Result[2](https://arxiv.org/html/2610.22894#Thmresult2)observed in practice: an evaluation reporting only the marginal gap would read\|Δ\|=0\.6\|\\Delta\|=0\.6as near\-perfect invariance\. Under Protocol P the same adapter is within noise of its base on every axis \(§[6](https://arxiv.org/html/2610.22894#S6)\)\.

Table 16:Cross\-relation LoRA fine\-tunes on the held\-outposrelation under Protocol L, pooled overn∈\{4,…,2048\}n\\in\\\{4,\\ldots,2048\\\}\. Values recomputed from the released outputs\.
### F\.3A Names\-List Artifact as a Diagnostic

An earlier generator emitted the names list in chain order, so that the list alone encoded the total order\. A LoRA fine\-tune of Qwen3\.5\-9B trained on such data reached≥99%\\geq 99\\%accuracy at everynnup to20482048on both presentations when evaluated with the same generator, and62\.3/66\.162\.3/66\.1\(original/equivalent\) once the names list was shuffled, about five points above the untouched base; the base Qwen3\.5\-27B itself scores90\.590\.5on the original presentation with chain\-ordered names against77\.377\.3with shuffled names, so pretrained models pick up the cue zero\-shot\. The matched\-presentation evaluation surfaced the discrepancy before the artifact was located, which is the methodological point: a presentation set does not need to know which surface feature a model exploits, only to disturb surface form while preserving logical content\. All main ordering results use shuffled names; the chain\-ordered condition here is a diagnostic control\.

Table 17:A fine\-tune trained on chain\-ordered names lists, evaluated with chain\-ordered and with shuffled names lists \(Protocol L,pos, pooled overnn\)\. The78\.178\.1invariance under shuffled names coincides with a yes\-default at largenn\(accuracy on no\-instances about18%18\\%atn=2048n=2048\)\.
### F\.4Cross\-Relation Replication \(depth\)

The same protocol applied to adepthrelation \(objects rather than people,n∈\{4,…,64\}n\\in\\\{4,\\ldots,64\\\},2,0002\{,\}000pairs per model\) reproduces the model ordering of Table[13](https://arxiv.org/html/2610.22894#A6.T13)at comparablenn, and the LoRA fine\-tune again fails to raise accuracy over its base on this second held\-out relation\.

Table 18:depthreplication under Protocol L \(n∈\{4,…,64\}n\\in\\\{4,\\ldots,64\\\}\)\.

## Appendix GAdditional GRPO Results and Reward Analysis

### G\.1Training\-Seed Replication

The main text reports seed 42, selected for higher mean invariance on the twelve clean held\-out relations\. This selection uses the reported held\-out evaluation results\. Seed 43 uses the same training and evaluation recipe\. Table[19](https://arxiv.org/html/2610.22894#A7.T19)retains both runs, including their front/behind results, and Figure[9](https://arxiv.org/html/2610.22894#A7.F9)resolves their profiles by chain length\. The selected run is not uniformly stronger on every metric: seed 43 has higher mean accuracy andκ¯\\bar\{\\kappa\}across the twelve relations\. Both runs retain substantial long\-chain failures\.

Table 19:GRPO training\-seed replication \(Protocol P,n=4n=4to10241024, 200 pairs per relation and level\)\. Both runs use the same recipe\. Main\-text results select seed 42 by mean invariance across the twelve clean held\-out relations\. Accuracy uses all prompts; paired metrics use jointly parsed answers\. Percentages exceptκ¯\\bar\{\\kappa\}\.41664256102450507575100100Chain lengthnnAccuracy \(%\)Mean accuracyBase 9BSFT \+ JSGRPO seed 42GRPO seed 43416642561024−0\.2\-0\.2000\.50\.511Chain lengthnnκn\\kappa\_\{n\}Chance\-corrected invariance

Figure 9:Training\-seed replication on held\-out front/behind \(Protocol P, 200 pairs per chain length\)\. Both GRPO seeds are compared with the base 9B and SFT with a JS consistency loss\. Mean accuracy andκn\\kappa\_\{n\}share valid\-pair support\. The main text displays seed 42, selected by mean invariance across the twelve clean held\-out relations\.#### Why the agreement bonus was inert\.

The nominal reward of the GRPO runs in Section[6](https://arxiv.org/html/2610.22894#S6)isr=correct\+λ​max⁡\(0,agree−chance\)r=\\mathrm\{correct\}\+\\lambda\\max\(0,\\mathrm\{agree\}\-\\mathrm\{chance\}\)withλ=1/2\\lambda=1/2, where agree is the share of samples on the instance’s other presentations that give the same answer andchance=q2\+\(1−q\)2\\mathrm\{chance\}=q^\{2\}\+\(1\-q\)^\{2\}withqqthe instance’s yes\-rate\. Each batch holds the four presentations of one instance \(original, equivalent, re\-shuffled, fully rewritten\) timesG=4G=4samples, but the trainer forms the advantage group per prompt, that is per presentation\. Within one group the other presentations’ answers,qqand chance are shared by all four samples, and every answer is yes or no \(no run produced an unparsed answer\), so the reward takes two values, one for the gold answer and one for the other\. The agreement term shifts the gap between them by at mostλ/2=0\.25\\lambda/2=0\.25and cannot reverse it, since the correctness gap is 1, and with group\-normalized advantages\(r−mean\)/std\(r\-\\mathrm\{mean\}\)/\\mathrm\{std\}of a two\-valued group depend only on how many of the four samples were right, not on the size of the gap\. Henceλ=1/2\\lambda=1/2andλ=0\\lambda=0give the same advantages \(up to the10−410^\{\-4\}added to the standard deviation\), and the logged cosine between the advantage vector used and theλ=0\\lambda=0vector is 1\.0000 on every step with a nonzero advantage\. The signal was also sparse: a step has a nonzero advantage only if some presentation’s four samples disagree, which happened on 28 of 600 steps for GRPO seed 42 and 31 of 600 for seed 43, and the gradient is zero on every other step\. The transfer in Section[6](https://arxiv.org/html/2610.22894#S6)is the effect of these few correctness updates, not of an agreement term\.

#### A variant in which the bonus can act\.

Rescaling the reward is not enough: the policy gradient is linear in the advantages and every nonzero GRPO step had a gradient norm of 2–38 against a clipping maximum of 1, so a bonus matters only if it changes the direction of the advantage vector\. That requires all presentations of an instance to share one advantage group and the instance’s rewards to take three or more values\. The grpo\-ig variant groups all sixteen samples of an instance, uses the instance’s mean reward as the baseline without group scaling, and rewardscorrect\+λ⁡\(agree−chance\)\\mathrm\{correct\}\+\\lambda\(\\mathrm\{agree\}\-\\mathrm\{chance\}\)without the clip at zero; everything else follows the GRPO recipe \(the same 19 held\-in relations, 600 instances,n=16n=16to128128,G=4G=4\)\. We trainedλ=0\\lambda=0andλ=0\.5\\lambda=0\.5with seeds 42 and 43\. Eachλ=0\.5\\lambda=0\.5run had 32–34 of 600 steps with a nonzero advantage, and the bonus changed the update direction on 30 of them, with a mean cosine of 0\.9987 to theλ=0\\lambda=0vector, about3∘3^\{\\circ\}: with yes/no answers and a gold label shared by all presentations, the majority answer of an informative instance is almost always the gold one, so rewarding agreement points the same way as rewarding correctness\.

Table 20:Agreement bonus versus correctness\-only GRPO on the twelve clean held\-out relations \(Protocol P,n=4n=4to10241024, 1,800 pairs per relation\)\. Acc\., Inv\. andκ¯\\bar\{\\kappa\}are means over the twelve relations; Inv\.−\-floor over the eight relations with an identical\-prompt floor\. GRPO = per\-presentation advantage groups \(correctness\-only in effect\); grpo\-ig = all sixteen samples of an instance in one advantage group, reward correct\+λ⁡\(agree−chance\)\+\\lambda\(\\text\{agree\}\-\\text\{chance\}\)\.Table[20](https://arxiv.org/html/2610.22894#A7.T20)gives the twelve\-relation evaluation\. Theλ\\lambdaeffect \(λ=0\.5\\lambda=0\.5minusλ=0\\lambda=0\) is\+1\.8\+1\.8points of invariance for seed 42 \(95% pair\-bootstrap interval\+1\.4\+1\.4to\+2\.3\+2\.3\) and\+0\.2\+0\.2for seed 43 \(−0\.2\-0\.2to\+0\.6\+0\.6\), withκ¯\\bar\{\\kappa\}up by 0\.02 for both seeds; the intervals cover evaluation noise only\. Switching the training seed within one recipe moves invariance by 1\.3–2\.8 points \(GRPO 2\.2, grpo\-igλ=0\\lambda=01\.3, grpo\-igλ=0\.5\\lambda=0\.52\.8\), so theλ\\lambdaeffect is within the seed\-to\-seed spread, and instance grouping on its own \(λ=0\\lambda=0against GRPO\) changes invariance by−0\.7\-0\.7and\+0\.2\+0\.2\. Every variant leaves the front/behind family as the weakest cells \(invariance 65–79,κ¯\\bar\{\\kappa\}0\.26–0\.55\) and the nonce direction words near zero chance\-corrected invariance \(κ¯\\bar\{\\kappa\}−0\.08\-0\.08to−0\.01\-0\.01\)\. One seed shows an evaluation\-level gain, but the two\-seed evidence does not establish a consistent improvement beyond the observed training\-seed variation\.

## Appendix HIdentical\-Prompt Repeat Agreement

### H\.1Control and Interpretation

We estimate*identical\-prompt repeat agreement*by sending each original prompt to the same model again under the same decoding settings and comparing the two answers\. This measures run\-to\-run agreement without changing the prompt\. We contrast it with invariance between the original and an equivalent presentation\. Both agreement measures condition on parseable answers in the corresponding pair\.

Earlier tables call this control the*re\-sampling floor*or self\-agreement\. It is a measured reference, not a universal lower bound\. Cross\-presentation agreement below it indicates sensitivity beyond the observed original\-prompt repeat level\. The difference does not causally isolate presentation error, since answer uncertainty can differ between presentations\. It also does not establish a competence ranking by itself\. Repeated sampling of both presentations would provide a stronger characterization of decoding variability\.

### H\.2Base Ordering Models

Table 21:Base\-model repeat controls on front/behind \(Protocol P,n=4n=4to10241024, 200 pairs per level\)\. Inv\. compares original and equivalent prompts; Repeat compares two responses to the identical original prompt\. Values are percentages; differences are computed before rounding\. Agreement conditions on parsed answers in each compared pair\.On front/behind, identical\-prompt agreement ranges from 64\.8% to 96\.4% across the six base models\. Cross\-presentation invariance is 5\.9–11\.1 percentage points lower than the corresponding repeat agreement \(Table[21](https://arxiv.org/html/2610.22894#A8.T21)\)\. For GPT\-4\.1, the comparison is 96\.4% identical\-prompt agreement against 87\.9% invariance\. The output profiles in the main text therefore accompany a measured decoding control, rather than assuming perfectly repeatable responses\.

Reshuffling\-only agreement is 2\.3–7\.9 points below the repeat reference for the GPT models\. This complements the small marginal accuracy changes reported in §[5\.3](https://arxiv.org/html/2610.22894#S5.SS3)\. Protocol L negation controls also show repeat agreement above syntactic and negated\-presentation agreement for all four tested models; Qwen3\.5\-27B gives 81\.8, 75\.2, and 60\.2%, respectively\. These observations describe sensitivity relative to the measured reference; they do not remove uncertainty induced by the altered prompts\.

### H\.3Training Comparisons

Table 22:Repeat controls for the Qwen3\.5\-9B training arms on held\-out front/behind \(Protocol P,n=4n=4to10241024, 200 pairs per level\)\. Values are percentages; differences precede rounding\. Dashes indicate an unavailable repeat control\. Both SFT and GRPO use LoRA\.GRPO improves both repeat agreement and cross\-presentation invariance, but by different amounts \(Table[22](https://arxiv.org/html/2610.22894#A8.T22)\)\. For seed\-42 GRPO, front/behind repeat agreement rises from the base model’s 64\.8% to 93\.7%, while invariance rises from 57\.3% to 73\.3%\. The difference widens from−7\.5\-7\.5to−20\.4\-20\.4points; seed 43 gives−21\.9\-21\.9\. More repeatable original\-prompt responses therefore coexist with substantial presentation failures\. This comparison supports reporting both quantities, without attributing the residual disagreement wholly to a deterministic presentation\-specific mechanism\.

### H\.4Reasoning and Additional Models

The available GPT\-5\.4 repeat controls are plotted in Figure[6](https://arxiv.org/html/2610.22894#A5.F6)\. On matched Protocol L front/behind prompts,nonegives 90\.3% repeat agreement and 79\.2% invariance;mediumgives 98\.8% and 98\.4%\. Repeat controls were not collected for every reasoning\-effort condition\. Additional\-model repeat comparisons remain in Appendix[E\.1](https://arxiv.org/html/2610.22894#A5.SS1); the GRPO agreement\-bonus comparisons are in Appendix[G](https://arxiv.org/html/2610.22894#A7)\. Horn\-clause runs have no repeat control\.

## Appendix IMechanistic Probes and Interventions

Output profiles establish the behavioral failures\. Probes and interventions in open\-weight models provide supporting evidence about how presentation dependence arises, without identifying a complete mechanism\.

#### Method\.

We study Qwen3\.5\-9B, its GRPO adapter \(§[6](https://arxiv.org/html/2610.22894#S6)\), and Qwen3\-8B with greedy yes/no readout\. Short\-chain runs use 200 matched pairs per relation atn≤32n\\leq 32; complexity runs use 30 pairs per level throughn=256n=256\. We fit cross\-validated gold\-answer probes\([Belinkov, 2022](https://arxiv.org/html/2610.22894#bib.bib1)\), apply the logit lens\([nostalgebraist, 2020](https://arxiv.org/html/2610.22894#bib.bib21)\), and patch original residual states or attention heads into equivalent runs\([Meng et al\., 2022](https://arxiv.org/html/2610.22894#bib.bib19);[Vig et al\., 2020](https://arxiv.org/html/2610.22894#bib.bib39)\)\. A balanced\-cell decomposition distinguishes the presentation\-level defaultδ\\deltatoward “yes” from instance signalss\(Table[11](https://arxiv.org/html/2610.22894#A5.T11)\)\.

#### Decodable answers and output defaults\.

At short chains, the base 9B’s gold answer is linearly decodable on lexical relations \(best probe AUROC 1\.00 on originals, 0\.92–1\.00 on equivalents\), despite nearly constant “yes” outputs\. Front/behind hasδ=1\.58\\delta=1\.58ands=0\.63s=0\.63\. Late residual patches often restore the original answer on disagreeing pairs \(Figure[5](https://arxiv.org/html/2610.22894#A5.F5), Appendix[E](https://arxiv.org/html/2610.22894#A5)\)\. These observations are consistent with a readout default overwhelming usable answer information\. They do not rule out earlier representational errors\.

#### Decodability weakens with complexity\.

At layer 26, equivalent\-presentation AUROC drops from 0\.98 atn=16n=16to 0\.75 atn=64n=64and 0\.54 atn=256n=256for the base model\. Atn=256n=256, the adapter reaches 0\.65 on equivalents, compared with 0\.90 on originals\. This demonstrates reduced linear decodability at the measured position and layer\. It does not prove that the answer is absent from the model, since other representations or nonlinear readouts may retain it\. Nonce direction terms also yield weak equivalent\-presentation probes \(base 0\.68 against a shuffled\-label null of 0\.615\)\. That evidence supports difficulty with this rendering, rather than a general claim that unfamiliar relations cannot be learned\.

#### Agreement can rise when signal is removed\.

In Qwen3\-8B, ablating four heads selected through cross\-vocabulary patching removes about 40% ofss, compared with at most 3% for the random\-head controls\. Invariance rises from 81\.0% to 93\.0% while original/equivalent accuracy falls from 64\.5/59\.5% to 59\.0/52\.0%\. The increase in agreement accompanies worse task performance, directly illustrating why invariance alone is insufficient\.

#### Training strengthens the measured signal\.

For front/behind, GRPO increasesssfrom 0\.63 to 3\.60 on originals and from 0\.50 to 3\.22 on equivalents\. The default decreases from 1\.58/1\.14 to 1\.37/0\.36, remaining nonzero on originals\. This is consistent with greater task signal relative to the default\. These probes cover one model family, small datasets, and mostly short chains; they do not explain the API models’ failures\.