Where Larger Models Excel: The Primacy of Constraint-Guided Reasoning

# The Primacy of Constraint-Guided Reasoning
Source: [https://arxiv.org/html/2606.26108](https://arxiv.org/html/2606.26108)
## Where Larger Models Excel: The Primacy of Constraint\-Guided Reasoning

Guan\-Yi Lin National Chengchi University Taipei, Taiwan 111703052@g\.nccu\.edu\.tw&Hen\-Hsen Huang Academia Sinica Taipei, Taiwan hhhuang@iis\.sinica\.edu\.tw

###### Abstract

Larger language models consistently outperform smaller ones on reasoning benchmarks, yet the reasoning differences underlying this gap remain underexplored\. Across benchmarks in mathematics, physics, chemistry, and programming, we observe stable performance gaps: averaged over datasets, Qwen3\-32B outperforms Qwen3\-8B by 6\.43%, while GPT\-OSS\-120B exceeds GPT\-OSS\-20B by 7\.38%\. To study the reasoning differences behind these gains, we developAdvCluster, an automated framework that identifies questions where the larger model shows a stable advantage, extracts fine\-grained advantage descriptions from paired reasoning traces produced by larger and smaller models, and organizes them through semantic clustering with quantitative evaluation and selection guided by a reviewer model\. Our analysis yields a systematic taxonomy of larger model reasoning advantages, spanning both common advantages that recur across domains and specialized advantages associated with particular domains\. Across these patterns, a recurring theme isConstraint\-Guided Reasoning: larger models are better at identifying explicit and implicit constraints, organizing them into structured reasoning, and using them to rule out infeasible paths and verify intermediate steps\.

## 1Introduction

While larger language models consistently outperform smaller ones on complex reasoning benchmarks, the reasoning processes underlying this advantage remain insufficiently understood\. Existing evaluations focus primarily on aggregate outcomes such as accuracy, which reveal little about where larger models excel or how their reasoning differs from that of smaller models\(Lightmanet al\.,[2024](https://arxiv.org/html/2606.26108#bib.bib23)\)\.

Prior works primarily focus on enhancing smaller models via distillation to address reasoning bottlenecks\(Hsiehet al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib22); Mitraet al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib27)\)\. While some evaluate reasoning through process supervision\(Lightmanet al\.,[2024](https://arxiv.org/html/2606.26108#bib.bib23)\)or holistic benchmarks\(Lianget al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib25)\), the field still lacks a systematic framework for empirically analyzing the qualitative reasoning advantages of larger models\.

In this work, we present a cross\-scale analysis of reasoning, specifically comparing larger models against their smaller counterparts within the same model family\. Inspired byYinet al\.\([2025](https://arxiv.org/html/2606.26108#bib.bib14)\), we developAdvCluster, an automated advantage discovery framework that first identifies benchmark questions where the larger model consistently outperforms the smaller model, and then compares their reasoning traces to extract advantage descriptions\. We then organize these descriptions into clusters through a semantic processing pipeline, and use quantitative metrics together with a reviewer model to select among candidate clustering solutions based on its consistency, distinctness, and granularity\. Figure[1](https://arxiv.org/html/2606.26108#S1.F1)illustrates the AdvCluster framework on a representative math example, showing how the reasoning advantages of larger models emerge naturally from the data\.

Our analysis categorizes these reasoning advantages into two types:common advantages, which recur across multiple domains, andspecialized advantages, which are tied to domain\-specific knowledge\. A primary common pattern identified by AdvCluster across various domains isConstraint\-Guided Reasoning\. As illustrated in Figure[1](https://arxiv.org/html/2606.26108#S1.F1), while smaller models often rely on unguided trial and error, larger models tend to reformulate problems using explicit constraints to systematically guide the solution process\.

![Refer to caption](https://arxiv.org/html/2606.26108v1/x1.png)

Figure 1:Illustration of theAdvClusterframework, exemplified by a comparative analysis of Qwen3\-32B and Qwen3\-8B on a mathematics problem\. The upper and lower panels contrast the reasoning traces of the larger and smaller models, with annotated symbols \(⋆,⋄\\star,\\diamond\) highlighting specific steps where the larger model demonstrates strategic advantages\. An LLM\-based advantage extractor converts these local differences into textual advantage descriptions\. Across many such questions, these advantages are aggregated via semantic clustering into cluster tags—such asRecasting via TransformationsandReducing to Existence Constraints—that characterize the broaderConstraint\-Guided Reasoningpattern\.
## 2Motivation and Preliminary Observations

Chain\-of\-thought prompting improves performance on complex reasoning tasks while exposing intermediate reasoning steps, making reasoning traces analyzable\(Weiet al\.,[2022b](https://arxiv.org/html/2606.26108#bib.bib1); Wanget al\.,[2023b](https://arxiv.org/html/2606.26108#bib.bib21)\)\. This increased transparency has motivated a growing body of work on reasoning trajectory analysis\(Yeoet al\.,[2024](https://arxiv.org/html/2606.26108#bib.bib20); Wanget al\.,[2023a](https://arxiv.org/html/2606.26108#bib.bib24)\)\.

Larger models generally perform better\.A separate line of work shows that language model capability generally improves with scale\.Kaplanet al\.\([2020](https://arxiv.org/html/2606.26108#bib.bib2)\)characterize these gains through neural scaling laws over parameters, data, and compute\. Subsequent studies further suggest that some complex capabilities, including multi\-step reasoning, emerge only at sufficiently large scales\.Weiet al\.\([2022a](https://arxiv.org/html/2606.26108#bib.bib3)\)describe such behaviors as emergent abilities, namely capabilities absent in smaller models but present in larger ones\.

Together, these findings indicate that larger models tend to achieve stronger reasoning performance than their smaller counterparts\. However, a central question remains: where do larger models exhibit consistent reasoning advantages over smaller ones?

Most existing work instead focuses on improving smaller models, for example by addressing specific bottlenecks such as reasoning data quality through distillation\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.26108#bib.bib4)\)\. While such efforts may indirectly suggest where larger models hold advantages, they do not provide a systematic empirical account of how reasoning behavior differs between larger and smaller models\.

Preliminary results\.We first conduct a preliminary experiment across four domains: mathematics, physics, chemistry, and programming\. Our evaluation covers a diverse set of reasoning benchmarks, including HHMT\(Balunovicet al\.,[2025](https://arxiv.org/html/2606.26108#bib.bib7)\), Omni\-MATH\(Gaoet al\.,[2025](https://arxiv.org/html/2606.26108#bib.bib6)\), JEEBench\(Aroraet al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib8)\), OlympiadBench\(Kuet al\.,[2024](https://arxiv.org/html/2606.26108#bib.bib32)\), GPQA\(Reinet al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib10)\), and CRUXEval\(Xuet al\.,[2025](https://arxiv.org/html/2606.26108#bib.bib11)\)\. We evaluate two reasoning model families at different scales: Qwen3\-8B vs\. Qwen3\-32B, and GPT\-OSS\-20B vs\. GPT\-OSS\-120B\(Team,[2025](https://arxiv.org/html/2606.26108#bib.bib12); OpenAI,[2025](https://arxiv.org/html/2606.26108#bib.bib13)\)\.

Our preliminary results show a consistent improvement from smaller models to their larger counterparts on these reasoning benchmarks\. On average across all evaluated datasets, Qwen3\-32B outperforms Qwen3\-8B by 6\.43%, while GPT\-OSS\-120B outperforms GPT\-OSS\-20B by 7\.38%\. These preliminary results confirm the performance gap between larger and smaller models in our setting and motivate a closer investigation into where these gains arise in the reasoning process\. Detailed results per data set are provided in Appendix[A\.4](https://arxiv.org/html/2606.26108#A1.SS4)\.

Table 1:Average accuracy \(%\) across all evaluated reasoning benchmarks\.
## 3Methodology

### 3\.1Dynamic Advantage Classification

Reasoning advantages are difficult to categorize using a predefined taxonomy, since the forms of model superiority vary across tasks and domains\. This makes static category assignment by an LLM judge restrictive and potentially unreliable\.Yinet al\.\([2025](https://arxiv.org/html/2606.26108#bib.bib14)\)address a related challenge in mathematical error analysis with a dynamically adaptive framework, in which categories are induced from data rather than specified in advance\. Inspired by this idea, we develop a data\-driven pipeline for reasoning advantage analysis\. Instead of imposing a fixed taxonomy, we first extract fine\-grained advantage descriptions from empirical comparisons between larger and smaller models’ reasoning traces, and then organize these descriptions through semantic clustering\. This allows recurring reasoning advantage categories to emerge naturally from the data\.

### 3\.2Our Framework: AdvCluster

AdvCluster consists of three stages: Analysis Question Set, Advantage Extraction, and Semantic Clustering\. In the first stage, we construct an analysis question set by identifying benchmark questions on which the larger model consistently outperforms the smaller model across repeated runs\. In the second stage, we construct paired comparisons between the larger and smaller models’ reasoning traces for each question in the analysis set, and use a LLM as an Advantages Extractor to produce fine\-grained advantage descriptions from these comparisons\. In the third stage, we embed and cluster the extracted descriptions to induce an interpretable taxonomy of larger model reasoning strengths\.

#### 3\.2\.1Analysis Question Set

Gap\-Based Filtering\.Because our goal is to compare reasoning differences between larger and smaller models, we first identify questions for which the larger model consistently outperforms the smaller one across repeated trials\. This step isolates analysis questions that reflect stable performance differences\.

For each questionqq, we evaluate both models overTTindependent trials\. LetcM​\(q,t\)=1c\_\{M\}\(q,t\)=1if modelMManswersqqcorrectly on trialtt, and0otherwise\. We define the pass rate of modelMMonqqas

PassRateM​\(q\)=1T​∑t=1TcM​\(q,t\),\\mathrm\{PassRate\}\_\{M\}\(q\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}c\_\{M\}\(q,t\),i\.e\., the proportion of trials on whichMManswersqqcorrectly\.

We define the performance gap for questionqqas

Δ​\(q\)=PassRateML​\(q\)−PassRateMS​\(q\),\\Delta\(q\)=\\mathrm\{PassRate\}\_\{M\_\{L\}\}\(q\)\-\\mathrm\{PassRate\}\_\{M\_\{S\}\}\(q\),whereMLM\_\{L\}andMSM\_\{S\}denote the larger and smaller models, respectively\. This quantity measures how much more often the larger model answers questionqqcorrectly than the smaller model across repeated trials\.

We retain questions with sufficiently largeΔ​\(q\)\\Delta\(q\)to form the analysis question set\. The next stage constructs reasoning comparisons only on this filtered set\.

#### 3\.2\.2Advantage Extraction

For each questionqqin the analysis question set, we construct reasoning comparisons between the larger and smaller models\. Each comparison instance consists of a pair of reasoning traces generated on the same question, one from the larger model and one from the smaller model\.

Letrq,iLr^\{L\}\_\{q,i\}andrq,iSr^\{S\}\_\{q,i\}denote the larger model and smaller model reasoning traces in theii\-th comparison instance for questionqq, respectively\. We then use an Advantages Extractor𝒥\\mathcal\{J\}to compare the paired traces and produce a set of advantage descriptions:

Zq,i=𝒥​\(q,rq,iL,rq,iS\),Z\_\{q,i\}=\\mathcal\{J\}\(q,r^\{L\}\_\{q,i\},r^\{S\}\_\{q,i\}\),where each elementz∈Zq,iz\\in Z\_\{q,i\}is a natural\-language description of a reasoning advantage exhibited by the larger model over the smaller model in that comparison\. In this way, each reasoning pair generates a small set of advantage descriptions\.

Concretely, the extractor is implemented with the following prompt:

PromptYou are an advantage extraction expert\.Model\_A is correct; Model\_B is incorrect\.TASK:Compare their reasoning ……extract 2–5 advantage objects explaining why Model\_A succeeds\.Problem: \{q\}Model\_A reasoning: \{larger\_model\_reasoning\}Model\_B reasoning: \{smaller\_model\_reasoning\}…

The complete prompt is provided in AppendixLABEL:llmanalyzeprompt

#### 3\.2\.3Semantic Clustering

After advantage extraction, we obtain a collection of natural\-language advantage descriptions from all comparison instances constructed on the analysis question set\. These descriptions summarize how the larger model outperforms the smaller model across questions from different domains\. Let

denote the set of all extracted advantage descriptions\.

Advantage Encoding and Preprocessing\.We encode each descriptionz∈𝒵z\\in\\mathcal\{Z\}into an embedding vector

e=ϕ​\(z\)∈ℝd,e=\\phi\(z\)\\in\\mathbb\{R\}^\{d\},whereϕ\\phidenotes the embedding model\.

As multiple comparison instances for the same question may yield highly similar advantage descriptions, directly clustering all descriptions can bias cluster centroids toward duplicated patterns\. We therefore perform deduplication in the embedding space using a greedy procedure with a cosine\-similarity threshold of0\.950\.95\(Guptaet al\.,[2025](https://arxiv.org/html/2606.26108#bib.bib18); Gyawaliet al\.,[2020](https://arxiv.org/html/2606.26108#bib.bib19)\)\.

We then reduce the embedding dimensionality via PCA before clustering to mitigate the difficulty of clustering in high\-dimensional vector spaces, following common embedding\-based text clustering pipelines\(Grootendorst,[2022](https://arxiv.org/html/2606.26108#bib.bib16); Eklundet al\.,[2023](https://arxiv.org/html/2606.26108#bib.bib17)\)\.

e~=ψ​\(e\)∈ℝd′,d′<d\.\\tilde\{e\}=\\psi\(e\)\\in\\mathbb\{R\}^\{d^\{\\prime\}\},\\qquad d^\{\\prime\}<d\.Let

ℰ=\{e~\}\\mathcal\{E\}=\\\{\\tilde\{e\}\\\}denote the deduplicated and low\-dimensional advantage embedding vectors\.

Clustering and Candidate Generation\.We apply K\-means clustering toℰ\\mathcal\{E\}under multiple candidate settings\(d′,K\)\(d^\{\\prime\},K\), whered′d^\{\\prime\}is the PCA dimension andKKis the number of clusters\. For each fixedd′d^\{\\prime\}, we sweep over a range ofKKvalues and quantitatively evaluate the resulting candidate clustering solutions using Davies–Bouldin Index \(DBI\) as the primary criterion and the Silhouette score as a secondary reference\. Based on this quantitative evaluation, we retain a smaller set of quantitatively favorable candidate clustering solutions\.

For each retained candidate clustering solution, we map the low\-dimensional embeddings assigned to each cluster back to their underlying natural\-language advantage descriptions\. We then use a summarization model to produce, for each cluster:

- •a shorttag;
- •a concisedefinitionof the shared reasoning advantage pattern\.

The detailed prompt for summarization model is provided in Appendix[A\.2\.2](https://arxiv.org/html/2606.26108#A1.SS2.SSS2)\.

Final Clustering Selection\.The quantitative evaluation above narrows the options to a smaller set of retained candidate clustering solutions\. However, for downstream results analysis, we seek a clustering solution that is not only quantitatively supported but also semantically well separated, interpretable, and appropriately granular\. We therefore use a reviewer model to compare the retained candidate clustering solutions and assess which one is most suitable overall, based on criteria such as internal coherence, separation between clusters, semantic interpretability, and granularity\. The prompt for reviewer model is provided in Appendix[A\.2\.3](https://arxiv.org/html/2606.26108#A1.SS2.SSS3)\.

Finally, based on both the quantitative metrics and the reviewer model’s assessment, we select the clustering solution used for downstream results analysis\. The selected clustering solution determines the final number of clustersKKand yields

\{\(Ck,tk,dk\)\}k=1K,\\\{\(C\_\{k\},t\_\{k\},d\_\{k\}\)\\\}\_\{k=1\}^\{K\},whereCkC\_\{k\}is the set of advantage descriptions assigned to thekk\-th cluster,tkt\_\{k\}is its tag, anddkd\_\{k\}is its definition\.

## 4Experiment

### 4\.1Setup

##### Models and Benchmarks\.

We study two larger–smaller model pairs from the same family: Qwen3\-32B vs\. Qwen3\-8B, and GPT\-OSS\-120B vs\. GPT\-OSS\-20B\. We evaluate them on benchmarks from four domains: mathematics \(HHMT, OMNI, JEEBench\), physics \(GPQA, JEEBench, Olympiad\-level benchmarks\), chemistry \(GPQA, JEEBench\), and programming \(CRUXEval\)\.

##### Repeated Inference and Analysis Question Set Construction\.

For each model on each dataset, we performT=10T=10independent runs\. For each questionqq, we compute the larger–smaller performance gap

Δ​\(q\)=PassRateML​\(q\)−PassRateMS​\(q\),\\Delta\(q\)=\\mathrm\{PassRate\}\_\{M\_\{L\}\}\(q\)\-\\mathrm\{PassRate\}\_\{M\_\{S\}\}\(q\),and retain questions withΔ​\(q\)≥0\.6\\Delta\(q\)\\geq 0\.6\. These retained questions form the analysis question set\.

### 4\.2Advantage Analysis Pipeline

##### Advantage Extraction and Clustering\.

Applying the above filtering procedure yields analysis question sets of 115 questions for Qwen3 and 106 questions for GPT\-OSS\. For each question in the analysis question set, we construct paired reasoning comparisons between the larger and smaller models and useGemini 3 Proas the advantage extractor to generate advantage descriptions\.

We encode the extracted advantage descriptions using OpenAI’stext\-embedding\-3\-large, then perform semantic deduplication and PCA before clustering the resulting embeddings under multiple candidate settings\(d′,K\)\(d^\{\\prime\},K\)\. After deduplication, the final analysis corpus contains 1824 advantage descriptions for Qwen3 and 1963 for GPT\-OSS\.

For each retained candidate clustering solution, we usegpt\-5\.2as the summarization model to generate cluster tags and definitions\. We then usegpt\-5\.2as a reviewer model to assess candidate clustering solutions in terms of internal coherence, inter\-cluster distinctness, interpretability, and granularity\. Based on this assessment together with quantitative clustering metrics, we select the final clustering solution used in our results analysis\.

Detailed domain level statistics and deduplication results are provided in Appendix[A\.4\.2](https://arxiv.org/html/2606.26108#A1.SS4.SSS2)\.

## 5Results and Analysis

![Refer to caption](https://arxiv.org/html/2606.26108v1/x2.png)

![Refer to caption](https://arxiv.org/html/2606.26108v1/x3.png)

Figure 2:Domain Distribution of Larger\-Model Reasoning Advantages\.Heatmaps display the advantage clusters for Qwen3 \(32B vs\. 8B, top\) and GPT\-OSS \(120B vs\. 20B, bottom\)\. Cell values are row\-normalized, so within each cluster \(row\), the proportions across domains sum to 100%\. The value in each cell shows the proportion of advantage descriptions in that cluster that come from a given domain\. Cluster tags shown alongside the y\-axis denote the semantic meaning of the clusters\.On the basis of the quantitative clustering evaluation and reviewer model assessment described in Section[3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3), we adopt the final clustering solutions ofd′=8d^\{\\prime\}=8,K=6K=6for Qwen3 andd′=4d^\{\\prime\}=4,K=6K=6for GPT\-OSS\. These settings provide the best balance between quantitative cluster quality and semantic interpretability\. The tags, definitions, and sizes of the final clusters are provided in Appendix[A\.3](https://arxiv.org/html/2606.26108#A1.SS3)\.

We then use heatmaps to examine how the resulting advantage clusters are distributed across domains\. This analysis is motivated by the possibility that different domains may nonetheless share similar underlying reasoning patterns\. As shown in Figure[2](https://arxiv.org/html/2606.26108#S5.F2), some clusters have substantial presence across multiple subjects, whereas others are concentrated in more specific domains\. Based on these distributional patterns, we organize the reasoning advantages of larger models into two broad types:

- •common advantagesrecur across multiple subjects\.
- •specialized advantagesare concentrated in a particular subject and are often tied to domain\-specific representations or verification demands\.

The former indicates that some reasoning strengths are shared across tasks in various domain, while the latter is more evident in specific domains\.

For clarity, the names shown in Figure[2](https://arxiv.org/html/2606.26108#S5.F2)are the cluster tags derived from semantic clustering[3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3)\. We use these tags throughout the following discussion as the corresponding reasoning advantage clusters\.

We further find that several clusters can be organized under a higher\-level reasoning pattern:Constraint\-Guided Reasoning\. This refers to cases where the model structures the solution process around explicit or derived constraints, reformulates them into usable conditions, and uses them to restrict the search space, rule out invalid candidates, and support intermediate verification\. We begin with this pattern, and then turn to additional reasoning advantages that are more specialized by subject\.

### 5\.1Constraint\-Guided Reasoning

The primary common advantages that emerged in our semantic clustering reveal a similar pattern: Constraint\-Guided Reasoning\. Rather than relying on trial and error over surface forms, larger models more often begin with a problem’s explicit constraints and implicit constraints and reformulate them into more usable conditions, such as feasibility conditions, boundary conditions, invariant relations, and consistency checks\. This allows the larger model to systematically narrow the solution space, remove invalid candidates, and verify intermediate steps in reasoning\.

Qwen3 exhibits this pattern most clearly in itscommon advantagessuch as\#5 Reducing to Existence Constraintsand\#2 Recasting via Transformations\. Similar pattern appears in GPT\-OSS through itscommon advantageslike\#3 Enforcing Constraint Consistencyand\#2 Deriving Invariant Parameterizations\. Here, the larger model first organizes the problem into a unified set of constraints, then uses invariant relations or parameterized forms to derive exact formulas, inequalities, or bounds\. The larger model uses them to maintain consistency with the original problem structure and to rule out conclusions that violate the constraints\.

A representation case of Constraint\-Guided reasoning is discussed earlier in Figure[1](https://arxiv.org/html/2606.26108#S1.F1)\. The larger model first performs structural reformulation, then advances the solution through boundary proof and reparameterization\. By contrast, the smaller model stays closer to local trial and error over surface cues, making ineffective search and incorrect paths more likely\.

This is also revealed inspecialized advantages, where it is expressed through the solving tools of a particular domain:

- •In chemistry, Qwen3 exhibits\#0 Constraining via Invariants\. Since chemistry reasoning process relies heavily on invariants–such as balances, conserved totals, configuration states, the model translates descriptions into a formal representation, derives quantities that must remain valid, and uses them to eliminate inconsistent candidates or revise conclusions when general heuristics conflict with harder constraints\.
- •In mathematics, GPT\-OSS naturally exhibits\#0 Deriving Tight Boundsand\#5 Deriving Modular Periods\. The former advantage allows larger model to turn the problem into a counting system based on constraint and propagates local requirements into global feasibility or contradiction\. In\#5 Deriving Modular Periods, it rewrites the problem as a modular system, tracks invariant residues, and derives periodic behavior from that constraint\-guided reformulation\.

### 5\.2Other Reasoning Advantages

Beyond Constraint\-Guided Reasoning, we observed other advantages emerge in specific domains:

Execution Tracking\.This is primarily seen in programming tasks\. Qwen3 demonstrates this in\#4 Simulating Stateful Execution\.Larger models track operations step\-by\-step and update state variables after every single action\. Throughout this process, strict control flow is maintained\. Finally, intermediate cross\-checks effectively eliminate state drift and missed updates\.

Format Control\.This reasoning advantage appears in programming tasks\. For instance, in Qwen3’s\#1 Verifying Exact Sequences, the model precisely tracks elements and boundaries; consequently, it verifies the constructed output against properties like total length and delimiter placement\. Similarly, in\#3 Enforcing Output Conformance, it identifies structural and syntax constraints early on, then seamlessly converts intermediate results into the required format\. Furthermore, GPT\-OSS exhibits a related pattern in\#4 Preserving Output Fidelity\. During complex operations, this model maintains integrity at character level and corrects minor local anomalies\. Ultimately, these clusters demonstrate that larger models excel at presenting reasoning in the exact required form\.

Representation Alignment\.In the chemistry domain, a specialized advantage demonstrated by GPT\-OSS is\#1Reconciling Representations to Outcomes\. This comprehensive ability requires the model to seamlessly fuse multiple diverse descriptions of the same system\. By jointly processing 1D symbolic stoichiometry, 2D topological connectivity, and dynamic state variables, the model effectively synthesizes these different chemical perspectives\. By successfully organizing these features, the larger model can dynamically track complex chemical transformations and accurately project intermediate states into the correct final outcome\.

### 5\.3Validation via SLM Systemic Failures

![Refer to caption](https://arxiv.org/html/2606.26108v1/mean_depth_violin_Enorm_lt_0p4.png)Figure 3:Distribution of First Error Depths for Smaller Models\.The y\-axis indicates the mean first\-error depth across runs of a question, from 1 \(Reading\) to 5 \(Encoding\)\. Violin width represents failure density, with annotated horizontal lines marking the medians\. Only consistent errors \(En​o​r​m≤0\.4E\_\{norm\}\\leq 0\.4\) are included\. Across all domains \(Chemistry, Math, Physics, Programming\), failures for smaller models heavily concentrate at 3\- Transformation\) and 4\- Process Skills\)\.Beyond identifying the advantages of larger models, we validate our findings by analyzing where smaller models systematically fail\. Using the Error Dispersion Index \(EDI, Appendix[A\.5](https://arxiv.org/html/2606.26108#A1.SS5)\) to quantify ”First\-Error” consistency, we found that both Qwen3\-8B and GPT\-OSS\-20B yield an average EDI<0\.4<0\.4\. This confirms that their failures are not stochastic, but rather stem from fundamental capability deficits\.

As shown in Figure[3](https://arxiv.org/html/2606.26108#S5.F3), these deficits heavily concentrate at theTransformation\(Depth 3\) andProcess Skills\(Depth 4\) stages\. This distribution directly mirrors the core advantage of larger models: Constraint\-Guided Reasoning\. Lacking the consistent support of this capability, smaller models either fail to abstract problem into bounded mathematical representations \(Depth 3\), or they resort to unconstrained, localized trial\-and\-error that inevitably diverges during execution \(Depth 4\)\. Ultimately, these bottlenecks provide empirical evidence that Constraint\-Guided Reasoning constitutes the primary capability gap between smaller and larger models\.

## 6Conclusion

This work aims to better understand how reasoning changes with scale within the same model family\. We develop AdvCluster, an automated analysis framework that systematically compares larger and smaller model reasoning traces, extracts advantage descriptions, and organizes them through semantic clustering\. This allows recurring patterns to emerge naturally from the data and yields a structured taxonomy of common advantages and specialized advantages\.

Our main finding is that larger models across different model families consistently exhibit Constraint\-Guided Reasoning: they are better at identifying conditions, organizing them into structured constraints, and using them to guide reasoning and eliminate infeasible paths\. We hope this work provides a clearer starting point for understanding the reasoning advantages that emerge with scale\.

## References

- D\. Arora, H\. G\. Singh, and Mausam \(2023\)Have llms advanced enough? A challenging problem solving benchmark for large language models\.See[Proceedings of the 2023 conference on empirical methods in natural language processing, EMNLP 2023, singapore, december 6\-10, 2023, Bouamoret al\.](https://arxiv.org/html/2606.26108#bib.bib31),pp\. 7527–7543\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.468),[Document](https://dx.doi.org/10.18653/V1/2023.EMNLP-MAIN.468)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- M\. Balunovic, J\. Dekoninck, I\. Petrov, N\. Jovanovic, and M\. T\. Vechev \(2025\)MathArena: evaluating llms on uncontaminated math competitions\.CoRRabs/2505\.23281\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.23281),[Document](https://dx.doi.org/10.48550/ARXIV.2505.23281),2505\.23281Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\) \(2023\)Proceedings of the 2023 conference on empirical methods in natural language processing, EMNLP 2023, singapore, december 6\-10, 2023\.Association for Computational Linguistics\.External Links:[Link](https://aclanthology.org/volumes/2023.emnlp-main/),ISBN 979\-8\-89176\-060\-8Cited by:[D\. Arora, H\. G\. Singh, and Mausam \(2023\)](https://arxiv.org/html/2606.26108#bib.bib8)\.
- N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\) \(2024\)Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation, LREC/COLING 2024, 20\-25 may, 2024, torino, italy\.ELRA and ICCL\.External Links:[Link](https://aclanthology.org/volumes/2024.lrec-main/),ISBN 978\-2\-493814\-10\-4Cited by:[Y\. Zhao, S\. Zhou, and H\. Zhu \(2024\)](https://arxiv.org/html/2606.26108#bib.bib4)\.
- W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\) \(2025\)Proceedings of the 63rd annual meeting of the association for computational linguistics \(volume 1: long papers\), ACL 2025, vienna, austria, july 27 \- august 1, 2025\.Association for Computational Linguistics\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-long),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-LONG),ISBN 979\-8\-89176\-251\-0Cited by:[R\. Xu, J\. Cao, Y\. Lu, M\. Wen, H\. Lin, X\. Han, B\. He, S\. Cheung, and L\. Sun \(2025\)](https://arxiv.org/html/2606.26108#bib.bib11)\.
- C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\) \(2025\)Findings of the association for computational linguistics: EMNLP 2025, suzhou, china, november 4\-9, 2025\.Association for Computational Linguistics\.External Links:[Link](https://aclanthology.org/volumes/2025.findings-emnlp/),ISBN 979\-8\-89176\-335\-7Cited by:[Z\. Yin, Y\. Sun, X\. Huang, X\. Qiu, and H\. Zhao \(2025\)](https://arxiv.org/html/2606.26108#bib.bib14)\.
- A\. Eklund, M\. Forsman, and F\. Drewes \(2023\)An empirical configuration study of a common document clustering pipeline\.Northern European Journal of Language Technology9\.External Links:[Link](https://aclanthology.org/2023.nejlt-1.7/),[Document](https://dx.doi.org/10.3384/nejlt.2000-1533.2023.4396)Cited by:[§3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3.p4.1)\.
- B\. Gao, F\. Song, Z\. Yang, Z\. Cai, Y\. Miao, Q\. Dong, L\. Li, C\. Ma, L\. Chen, R\. Xu, Z\. Tang, B\. Wang, D\. Zan, S\. Quan, G\. Zhang, L\. Sha, Y\. Zhang, X\. Ren, T\. Liu, and B\. Chang \(2025\)Omni\-math: A universal olympiad level mathematic benchmark for large language models\.See[22](https://arxiv.org/html/2606.26108#bib.bib29),External Links:[Link](https://openreview.net/forum?id=yaqPf0KAlN)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- M\. Grootendorst \(2022\)BERTopic: neural topic modeling with a class\-based TF\-IDF procedure\.CoRRabs/2203\.05794\.External Links:[Link](https://doi.org/10.48550/arXiv.2203.05794),[Document](https://dx.doi.org/10.48550/ARXIV.2203.05794),2203\.05794Cited by:[§3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3.p4.1)\.
- V\. Gupta, C\. Ross, D\. Pantoja, R\. J\. Passonneau, M\. Ung, and A\. Williams \(2025\)Improving model evaluation using SMART filtering of benchmark datasets\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 \- Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 \- May 4, 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),pp\. 4595–4615\.External Links:[Link](https://doi.org/10.18653/v1/2025.naacl-long.235),[Document](https://dx.doi.org/10.18653/V1/2025.NAACL-LONG.235)Cited by:[§3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3.p3.1)\.
- B\. Gyawali, L\. Anastasiou, and P\. Knoth \(2020\)Deduplication of scholarly documents using locality sensitive hashing and word embeddings\.InProceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11\-16, 2020,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),pp\. 901–910\.External Links:[Link](https://aclanthology.org/2020.lrec-1.113/)Cited by:[§3\.2\.3](https://arxiv.org/html/2606.26108#S3.SS2.SSS3.p3.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Findings of ACL,pp\. 8003–8017\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-acl.507),[Document](https://dx.doi.org/10.18653/V1/2023.FINDINGS-ACL.507)Cited by:[§1](https://arxiv.org/html/2606.26108#S1.p2.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.CoRRabs/2001\.08361\.External Links:[Link](https://arxiv.org/abs/2001.08361),2001\.08361Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p2.1)\.
- S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\) \(2022\)Advances in neural information processing systems 35: annual conference on neural information processing systems 2022, neurips 2022, new orleans, la, usa, november 28 \- december 9, 2022\.External Links:[Link](https://papers.nips.cc/paper%5C_files/paper/2022),ISBN 9781713871088Cited by:[J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022b\)](https://arxiv.org/html/2606.26108#bib.bib1)\.
- L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\) \(2024\)Proceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: long papers\), ACL 2024, bangkok, thailand, august 11\-16, 2024\.Association for Computational Linguistics\.External Links:[Link](https://aclanthology.org/volumes/2024.acl-long/),ISBN 979\-8\-89176\-094\-3Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Re, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. WANG, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. S\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. A\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Note:Featured Certification, Expert Certification, Outstanding CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=iO4LZibEqW)Cited by:[§1](https://arxiv.org/html/2606.26108#S1.p2.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[§1](https://arxiv.org/html/2606.26108#S1.p1.1),[§1](https://arxiv.org/html/2606.26108#S1.p2.1)\.
- A\. Mitra, L\. D\. Corro, S\. Mahajan, A\. Codas, C\. Simões, S\. Agrawal, X\. Chen, A\. Razdaibiedina, E\. Jones, K\. Aggarwal, H\. Palangi, G\. Zheng, C\. Rosset, H\. Khanpour, and A\. Awadallah \(2023\)Orca 2: teaching small language models how to reason\.CoRRabs/2311\.11045\.External Links:[Link](https://doi.org/10.48550/arXiv.2311.11045),[Document](https://dx.doi.org/10.48550/ARXIV.2311.11045),2311\.11045Cited by:[§1](https://arxiv.org/html/2606.26108#S1.p2.1)\.
- OpenAI \(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.CoRRabs/2508\.10925\.External Links:[Link](https://doi.org/10.48550/arXiv.2508.10925),[Document](https://dx.doi.org/10.48550/ARXIV.2508.10925),2508\.10925Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)GPQA: A graduate\-level google\-proof q&a benchmark\.CoRRabs/2311\.12022\.External Links:[Link](https://doi.org/10.48550/arXiv.2311.12022),[Document](https://dx.doi.org/10.48550/ARXIV.2311.12022),2311\.12022Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.09388),[Document](https://dx.doi.org/10.48550/ARXIV.2505.09388),2505\.09388Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- \[22\]\(2025\)The thirteenth international conference on learning representations, ICLR 2025, singapore, april 24\-28, 2025\.OpenReview\.net\.External Links:[Link](https://openreview.net/group?id=ICLR.cc/2025/Conference)Cited by:[B\. Gao, F\. Song, Z\. Yang, Z\. Cai, Y\. Miao, Q\. Dong, L\. Li, C\. Ma, L\. Chen, R\. Xu, Z\. Tang, B\. Wang, D\. Zan, S\. Quan, G\. Zhang, L\. Sha, Y\. Zhang, X\. Ren, T\. Liu, and B\. Chang \(2025\)](https://arxiv.org/html/2606.26108#bib.bib6)\.
- B\. Wang, S\. Min, X\. Deng, J\. Shen, Y\. Wu, L\. Zettlemoyer, and H\. Sun \(2023a\)Towards understanding chain\-of\-thought prompting: an empirical study of what matters\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 2717–2739\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.153),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.153)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023b\)Self\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p1.1)\.
- J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler, E\. H\. Chi, T\. Hashimoto, O\. Vinyals, P\. Liang, J\. Dean, and W\. Fedus \(2022a\)Emergent abilities of large language models\.Trans\. Mach\. Learn\. Res\.2022\.External Links:[Link](https://openreview.net/forum?id=yzkSU5zdwD)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022b\)Chain\-of\-thought prompting elicits reasoning in large language models\.See[Advances in neural information processing systems 35: annual conference on neural information processing systems 2022, neurips 2022, new orleans, la, usa, november 28 \- december 9, 2022, Koyejoet al\.](https://arxiv.org/html/2606.26108#bib.bib28),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p1.1)\.
- R\. Xu, J\. Cao, Y\. Lu, M\. Wen, H\. Lin, X\. Han, B\. He, S\. Cheung, and L\. Sun \(2025\)CRUXEVAL\-X: A benchmark for multilingual code reasoning, understanding and execution\.See[Proceedings of the 63rd annual meeting of the association for computational linguistics \(volume 1: long papers\), ACL 2025, vienna, austria, july 27 \- august 1, 2025, Cheet al\.](https://arxiv.org/html/2606.26108#bib.bib33),pp\. 23762–23779\.External Links:[Link](https://aclanthology.org/2025.acl-long.1158/)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p5.1)\.
- W\. J\. Yeo, R\. Satapathy, R\. S\. M\. Goh, and E\. Cambria \(2024\)How interpretable are reasoning explanations from prompting large language models?\.InFindings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16\-21, 2024,K\. Duh, H\. Gómez\-Adorno, and S\. Bethard \(Eds\.\),Findings of ACL,pp\. 2148–2164\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-naacl.138),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-NAACL.138)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p1.1)\.
- Z\. Yin, Y\. Sun, X\. Huang, X\. Qiu, and H\. Zhao \(2025\)Error classification of large language models on math word problems: A dynamically adaptive framework\.See[Findings of the association for computational linguistics: EMNLP 2025, suzhou, china, november 4\-9, 2025, Christodoulopouloset al\.](https://arxiv.org/html/2606.26108#bib.bib34),pp\. 338–365\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.20/)Cited by:[§1](https://arxiv.org/html/2606.26108#S1.p3.1),[§3\.1](https://arxiv.org/html/2606.26108#S3.SS1.p1.1)\.
- Y\. Zhao, S\. Zhou, and H\. Zhu \(2024\)Probe then retrieve and reason: distilling probing and reasoning capabilities into smaller language models\.See[Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation, LREC/COLING 2024, 20\-25 may, 2024, torino, italy, Calzolariet al\.](https://arxiv.org/html/2606.26108#bib.bib30),pp\. 13026–13032\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1140)Cited by:[§2](https://arxiv.org/html/2606.26108#S2.p4.1)\.

## Appendix AAppendix

### A\.1LLM Usage Summary

### A\.2Prompt

#### A\.2\.1Advantage Extractor prompt

PromptYou are an LLM advantage extraction expert\. Model\_A is correct; Model\_B is incorrect\.TASK:Compare their reasoning and do the following:1\. Identify the FIRST Newman’s Error Analysis stage where Model\_B fails\.Choose ONE: Reading \|Comprehension \|Transformation \|Process Skills \|Encoding2\. Briefly describe the specific reasoning failure at that stage\.3\. From this failure, extract 2\-5 advantage objects explaining why Model\_A succeeds\.\!\!\! RULES FOR ’advantage’ FIELD \!\!\!1\. ABSTRACTION: Generalize to universal reasoning skills \(no problem\-specific variables\)\.2\. FORMAT: Start EXACTLY with an action verb \(e\.g\., ”Identifies”, ”Applies”\)\.3\. FORBIDDEN: NEVER use ”Model\_A”, ”Model\_B”, ”The model”, or ”Correctly”\.4\. LENGTH: Maximum 10 words\.Problem: \{q\}Model\_A reasoning: \{larger\_model\_reasoning\}Model\_B reasoning: \{smaller\_model\_reasoning\}Output Format:Each object MUST follow this exact schema:\*\* ONLY OUTPUT LIST OF OBJECTS, NO OTHER TEXT\. \*\*``` [ { "type": "failure", "failure_stage": "<ONE OF: Reading | Comprehension | Transformation | Process Skills | Encoding>", "failure_description": "<Brief description of Model_B’s FIRST failure>" }, { "type": "advantage", "advantage": "<Action verb + abstract skill (max 10 words)>", "evidence": "<Specific text contrast proving the advantage>" }, { "type": "advantage", "advantage": "<another Action verb + abstract skill (max 10 words)>", "evidence": "<another specific text contrast proving the advantage>" },... ] ```

#### A\.2\.2Summarization model prompt

PromptYou are an expert in LLM Reasoning Analysis\. I will provide a cluster of similar reasoning advantages\.Please summarize the advantages and provide a tag and definition for the cluster\.INSTRUCTIONS:1\. DE\-DOMAIN: Strip all subject\-specific context \(e\.g\., replace ’chemical valency’ with ’structural constraints’\)\.2\. ACTION\-VERB TAGS: The tag MUST start with a strong action verb \(e\.g\., Mapping, Verifying, Reducing, Deriving, Resolving\)\. Avoid generic tags like ’Reasoning’ or ’Logic’\.3\. DEFINITION: Describe \*how\* the model manipulates information to reach a conclusion\. Use 2\-4 clear sentences\.4\. LENGTH: Tag \(2–4 words\)\.Advantages \(bullet list\):\{advantage\_descriptions\_within\_cluster\_k\}RETURN JSON ONLY in this format:``` { "tag": "...", "definition": "..." } ```

#### A\.2\.3Reviewer Model prompt

PromptYou are evaluating a taxonomy of reasoning advantages induced from clustering\. Do not rewrite the taxonomy\. Do not assume that fewer clusters are always better\. Focus only on taxonomy quality\. You will be given K and, for each cluster, an id, a tag, and a short definition\.Evaluate the taxonomy using these criteria:\#\#\# 1\. Mutual Exclusivity \(Distinctness\)\- \[1\-2\] Redundant: Two or more clusters describe the same concept\.\- \[3\-4\] Overlap: Significant overlap \( 40\- \[5\-6\] Moderate: Different concepts, but boundaries are fuzzy/lack exclusion\.\- \[7\-8\] Sharp: Distinct features per cluster; very low ambiguity\.\- \[9\-10\] Exclusive: Logically impossible for an instance to belong to two clusters\.\#\#\# 2\. Conceptual Precision & Depth \(Granularity\)\- \[1\-2\] Vague/Circular: Definitions offer no real insight or repeat the tag\.\- \[3\-4\] Surface\-level: Describes ONLY the outcome \(e\.g\., ”the answer is wrong”\)\.\- \[5\-6\] Functional: Describes the reasoning process/behavior in plain language\.\- \[7\-8\]Academic: Explains the underlying mechanism using formal logic terminology\.\- \[9\-10\] Research\-Ready: High\-level synthesis of unique reasoning failures/patterns; publication quality\.\#\#\# 3\. Interpretability \(Narrative Value\) \- \[1\-2\] Obscure: Hard to explain what kind of cases fit here\.\- \[3\-4\] Complex: Requires heavy mental effort to map to actual model behavior\.\- \[5\-6\] Clear: Understandable, but lacks a strong, cohesive ”story\.”\- \[7\-8\] Intuitive: A researcher can immediately picture the failure mode\.\- \[9\-10\] Insightful: Provides an ”Aha\!” moment; memorable and easy to communicate\.\#\#\# 4\. Cluster Balance & Utility \(Granularity\)\- \[1\-2\] Failed: One dominant cluster \( 80%\) or useless fragmented noise\.\- \[3\-4\] Poor: Highly skewed\- \[5\-6\] Passable: Distribution allows for basic statistical observation\.\- \[7\-8\] Healthy: Balanced enough to reveal clear performance trends across models\.\- \[9\-10\] Optimal: Ideal for high\-precision comparative analysis and scaling law research\.\#\#\# 5\. Taxonomy Resolution \(Granularity\)\- \[1\-3\] Coarse: Too broad; merges distinct behaviors, losing diagnostic power\.\- \[4\-6\] Fragmented: Too noisy; over\-splits clusters based on surface\-level wording\.\- \[7\-10\] Functional: Just the right number of clusters to cover the main reasoning stages\.Input:``` K = {k} CLUSTERS = {clusters_section} ``` Output one JSON object only\.``` {{ "Distinctness_score": [1-10], "Granularity_score": [1-10], "Interpretability_score": [1-10], "Balance_score": [1-10], "Taxonomy_resolution_score": [1-10], }} ```

### A\.3Final Clustering Solutions

Tables 2 and 3 present the tags and definitions for all clusters in the final clustering solutions \(Qwen3:d′=8d^\{\\prime\}=8,K=6K=6; GPT\-OSS:d′=4d^\{\\prime\}=4,K=6K=6\)\. These were generated during the semantic clustering stage and are included for reference\. Table 4 reports the size of each final cluster\.

#### A\.3\.1Tags and Definitions

Table 2:GPT\-OSS\-120b Advantages Cluster Tags and DefinitionTable 3:Qwen3\-32b Advantages Cluster Tags and Definition
#### A\.3\.2Cluster Sizes

Table 4:Number of advantage descriptions assigned to each cluster in the final clustering solutions\.

### A\.4Supplementary Data

#### A\.4\.1Accuracy on Benchmarks

The following four tables contain detailed data from the preliminary results across all datasets, with each table corresponding to one of the four domains: mathematical, physics, chemistry, and programming\.

Table 5:Accuracy \(%\) of various models on mathematical reasoning benchmarksTable 6:Accuracy \(%\) of various models on physics reasoning benchmarksTable 7:Accuracy \(%\) of various models on chemistry reasoning benchmarksTable 8:Accuracy \(%\) of models on programming reasoning benchmarks
#### A\.4\.2Analysis Question Set

Table 9 provides the detailed counts of the 115 \(Qwen3\) and 106 \(GPT\-OSS\) questions retained after gap\-based filtering, categorized by domain\.

Table 9:Number of analysis questions retained after the gap\-based filter, by domains\.
#### A\.4\.3Advantage Descriptions Statistics

Table[10](https://arxiv.org/html/2606.26108#A1.T10)summarizes the effect of semantic deduplication on the extracted advantage descriptions for each model pair\.

Table 10:Deduplication statistics for extracted advantage descriptions\.

### A\.5Experimental Details for First\-Error Analysis

To investigate the systemic bottlenecks of smaller models, we conducted the ”First\-Error” analysis\. Our goal is to determine whether these models make stochastic \(random\) mistakes or exhibit recurring error\.

##### First\-Error Extraction Pipeline

We filter for challenging questions for smaller models, where the performance gap between the larger and smaller models is≥0\.6\\geq 0\.6\. For each question, we prompt the Advantage Extractor to identify the exact reasoning step where the smaller model commits its First Error during the Advantage Extraction stage\. We adapt Newman’s Error Analysis framework, grouping reasoning failures into five sequential depths \(1 to 5\):

1. 1\.Reading:Failing to recognize basic text, symbols, or vocabulary\.
2. 2\.Comprehension:Misunderstanding the overall objective or meaning of the problem\.
3. 3\.Transformation:Failing to translate the problem description into a valid mathematical or logical operation\.
4. 4\.Process Skills:Committing errors during step\-by\-step calculations or rule execution\.
5. 5\.Encoding:Failing to properly format or express the final derived solution\.

##### Error Dispersion Index \(EDI\) Formulation

To quantify error consistency, we propose the Error Dispersion Index \(EDI\), a question\-level metric based on normalized empirical Shannon entropy\. For a given question, assumingNNindependent reasoning trajectories sampled from a smaller model, letcic\_\{i\}be the count of ”first errors” occurring at depthiiout of\|D\|=5\|D\|=5possible depths\. The empirical probability isP​\(di\)=ciNP\(d\_\{i\}\)=\\frac\{c\_\{i\}\}\{N\}\.

The empirical Shannon entropy \(in bits\) for this specific question is calculated as:

H=−∑P​\(di\)\>0P​\(di\)​log2⁡\(P​\(di\)\)H=\-\\sum\_\{P\(d\_\{i\}\)\>0\}P\(d\_\{i\}\)\\log\_\{2\}\(P\(d\_\{i\}\)\)\(1\)
To ensure the metric is bounded within\[0,1\]\[0,1\], we normalizeHHby the exact empirical maximum entropy,ℋm​a​x​\(N,\|D\|\)\\mathcal\{H\}\_\{max\}\(N,\|D\|\)\. Conceptually representing a state of completely random guessing,ℋm​a​x\\mathcal\{H\}\_\{max\}is defined as the entropy achieved when theNNdiscrete errors are distributed as evenly as possible across the\|D\|\|D\|reasoning depths\. The per\-question EDI is then defined as:

E​D​I=Hℋm​a​x​\(N,\|D\|\)EDI=\\frac\{H\}\{\\mathcal\{H\}\_\{max\}\(N,\|D\|\)\}\(2\)
- •E​D​I→1EDI\\to 1: Errors are uniformly scattered \(stochastic\)\.
- •E​D​I→0EDI\\to 0: Errors isolate to a single depth \(deterministic\)\.

### A\.6Summary of Error Dispersion Data

While the EDI is computed independently for each question, we average these scores across all questions within a specific domain to evaluate broader systemic trends\. As detailed in Table[11](https://arxiv.org/html/2606.26108#A1.T11), across all domains, smaller models consistently yield an average EDI below0\.40\.4\. This confirms that their reasoning failures are predominantly structural deficits rather than stochastic hallucinations\.

Table 11:Summary of Error Dispersion Index \(EDI\) across domains and lower\-capacity models\.NQN\_\{Q\}denotes the number of challenging questions analyzed\. A lower Mean EDI indicates that the model’s first\-errors are highly concentrated at specific reasoning depths\.