Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

# Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
Source: [https://arxiv.org/html/2607.26119](https://arxiv.org/html/2607.26119)
###### Abstract

Large reasoning models trained via reinforcement learning \(RL\) have been increasingly shown to outperform their supervised fine\-tuned \(SFT\) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear\. We therefore ask,what internal representational differences enable RL models’ superior performance?Our work presents two converging lines of evidence: First, linear probes trained on layer\-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations\. Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers\. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems\. Finally, we analyze token\-count variability under repeated sampling across problems to assess adaptive compute allocation\. While we observe higher variability in some RL\-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone\. We believe this token\-allocation variability reveals the spread of plausible on\-policy reasoning, highlighting which models exhibit stable policies versus those that are under\-determined, potentially non\-identifiable solution behaviour\.

Code—https://oankit\.github\.io/\-rl\-sft\-reasoning/

## Introduction

Large Reasoning Models \(LRMs\) such as OpenAI’s o1 and DeepSeek\-R1 substantially outperform traditionally finetuned large language models \(LLMs\) on reasoning and logic problems across benchmarks\(OpenAIet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib7); Shaoet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib3)\)\. Understandingwhythis is the case though, requires moving beyond performance metrics to mechanistic explanations\. While we know LRMs generate longer chains of thought and achieve higher accuracy,howthey differ internally from base LLMs remains an open challenge\.

Current research has approached this question from two mutually reinforcing but disconnected perspectives\.Mechanistic interpretabilityhas identified specific circuits for arithmetic operations\(Sachanet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib10); Hannaet al\.[2023](https://arxiv.org/html/2607.26119#bib.bib13); Zhuet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib9)\)and shown that chain\-of\-thought increases activation sparsity\(Chenet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib14)\), primarily in smaller models on elementary operations\.Behavioural studieshave revealed information\-theoretic compression limits\(Leeet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib12)\), cross\-variant sensitivity to problem phrasing\(Mirzadehet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib11)\), and RL training’s potential for long reasoning\(Yeoet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib16)\)\. However,what internal representational differencesenable LRMs’ superior performance remains unexplored\.

We seek to bridge this gapthrough integrated behavioural\-mechanistic analysis using three complementary methods: \(1\)Linear probingon layer\-wise hidden states to predict answer correctness\(Alain and Bengio[2018](https://arxiv.org/html/2607.26119#bib.bib17); Belinkov[2022](https://arxiv.org/html/2607.26119#bib.bib23)\), measuringwhenandhow stronglyrepresentations emerge across model families\. \(2\)Mean ablation interventions\(Zhang and Nanda[2024](https://arxiv.org/html/2607.26119#bib.bib24); Menget al\.[2023](https://arxiv.org/html/2607.26119#bib.bib25)\)to identify which layers are critical for mathematical reasoning across training methodologies\. \(3\)Generation consistency analysisvia multiple samples per problem, extending variance analysis\(Mirzadehet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib11)\)to within\-problem comparisons and empirically validating compression theory\(Leeet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib12)\)\.

Specifically, our contributions include: \(1\) Evidence that RL models develop stronger, earlier\-emerging representations through scalable linear probing\. \(2\) Discovery that RL training reshapes computational architecture, concentrating reasoning in deeper layers versus instruction\-tuning’s uniform distribution\. \(3\) Empirical validation of token complexity theory, revealing that superior representations manifest as consistent token usage across difficulty levels, with current RL training showing unexploited potential for adaptive allocation\. Our analysis reveals thattraining methodology fundamentally reshapes computational architecture: RL\-trained models show earlier engagement and progressive concentration in deeper layers \(r=0\.47\), whereas instruction\-tuned models distribute reasoning uniformly \(r=\-0\.11\)\. This architectural difference, combined with earlier\-emerging and stronger answer representations in RL models, provides initial mechanistic insight into performance differences, moving from descriptive benchmarking\(OpenAIet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib7); DeepSeek\-AI[2025](https://arxiv.org/html/2607.26119#bib.bib4)\)to mechanistic explanation\.

## Measuring Representation Quality via Probing

Understandingwhenandhowcorrect answer information emerges across model layers can reveal fundamental differences between LRMs and SFT models\. If LRMs develop “clearer” representations, we should be able to detect this mechanistically: problems with more linearly separable internal representations should exhibit higher discriminability between correct and incorrect answers\. To test this, we train linear probes on layer\-wise hidden states to predict final answer correctness, building on the interpretability literature using linear probes to study intermediate representations\(Alain and Bengio[2018](https://arxiv.org/html/2607.26119#bib.bib17)\), particularly for mathematical reasoning\(Zhuet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib9)\)\. We hypothesize that probe accuracy correlates with model accuracy, potentially explaining why LRMs outperform SFT models\.

### Synthetic Problem Generation

To investigate whether model failures stem from reasoning limitations or surface\-form artifacts, we generated 1,000 synthetic mathematical problems using four fixed templates covering probability, fractions, and cost calculations\. FollowingMirzadehet al\.\([2025](https://arxiv.org/html/2607.26119#bib.bib11)\)’s approach of generating synthetic GSM8K variants\(Cobbeet al\.[2021](https://arxiv.org/html/2607.26119#bib.bib2)\), our controlled generation isolates representational properties required for general reasoning from question\-specific memorization effects\. Template details and example problems are provided in Appendix[B](https://arxiv.org/html/2607.26119#A2)\.

Rationale:Synthetic generation offers three advantages: \(1\) eliminates data contamination, \(2\) enables algorithmic verification with known ground\-truth parameters, and \(3\) scales to large sample sizes for robust statistical analysis\. Prior work demonstrates that synthetic benchmarks effectively reveal model reasoning capabilities\(Mirzadehet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib11)\)while avoiding artifacts of human\-authored datasets\.

#### Completion Generation and Labeling

For each model, we generate a single completion for every problem using sampling \(temperatureT∈\[0\.6,0\.7\]T\\in\[0\.6,0\.7\],top p=0\.95=0\.95\)\. Final answers are extracted from within\\boxed\{\}delimiters\. Each completion is labeled as correct if the extracted answer matches the ground truth \(allowing a tolerance of±\\pm1 for rounding\), and incorrect otherwise\. Problems where no valid answer can be extracted are omitted from further analysis\.

Balancing procedure:To ensure fair comparison across models with different accuracy distributions, we identify the intersection of problems answered by all models, then sample equal numbers of correct and incorrect examples per model\. This produces balanced training sets where each model contributes identical sample sizes with equal class distribution, removing confounding variables\. Data is split 70/15/15 into train/validation/test sets, stratified by label\.

#### Activation Extraction

Activation extraction captures the model’s internal state at the precise moment it has completed reasoning but not yet committed to an answer\.

Probe position:We extract hidden states at the token immediately preceding\\boxed\{\}\. This position is chosen for two reasons: \(1\) all answers follow this delimiter, ensuring the model has completed reasoning before answer articulation, and \(2\) the delimiter tokenizes consistently across examples, unlike final answers which may span variable token lengths\. We tokenize the full generated text and locate\\boxed\{in the token sequence\.

Batched extraction with position preservation:We process completions in batches of 16–32\. To maintain consistent token positions across variable\-length sequences, we applyright padding\(padding appended after sequences\) rather than left padding, ensuring extraction positions remain unchanged relative to sequence start\. Each sequence receives an attention mask with 1s for real tokens and 0s for padding\.

Layer\-wise representations:For each batch, we perform a single forward pass withoutput\_hidden\_states=Trueand extract activations from allLLtransformer layers\. We collecthidden\_states\[1:L\+1\]—the outputs of transformer blocks 1 throughLL—omittinghidden\_states\[0\]\(input embeddings\) and architecture\-specific post\-normalization states that may reduce probe effectiveness\. This yields tensors of shape\[L×N×D\]\[L\\times N\\times D\], whereNNis batch size andDDis hidden dimension \(4096\)\.

#### Probe Training

For each layerℓ∈\{1,…,L\}\\ell\\in\\\{1,\\ldots,L\\\}, we train logistic regression probefℓ:ℝD→\{0,1\}f\_\{\\ell\}:\\mathbb\{R\}^\{D\}\\rightarrow\\\{0,1\\\}to predict binary answer correctness from theDD\-dimensional hidden state𝐡ℓ\\mathbf\{h\}\_\{\\ell\}\. The probe is defined as:

fℓ​\(𝐡ℓ\)=σ​\(𝐰ℓ⊤​𝐡ℓ\+bℓ\)f\_\{\\ell\}\(\\mathbf\{h\}\_\{\\ell\}\)=\\sigma\(\\mathbf\{w\}\_\{\\ell\}^\{\\top\}\\mathbf\{h\}\_\{\\ell\}\+b\_\{\\ell\}\)\(1\)where𝐰ℓ∈ℝD\\mathbf\{w\}\_\{\\ell\}\\in\\mathbb\{R\}^\{D\}is the weight vector,bℓ∈ℝb\_\{\\ell\}\\in\\mathbb\{R\}is the bias term, andσ​\(x\)=1/\(1\+e−x\)\\sigma\(x\)=1/\(1\+e^\{\-x\}\)is the sigmoid function\. Following best practices\(Belinkov[2022](https://arxiv.org/html/2607.26119#bib.bib23)\), we use 5\-fold cross\-validation to select regularization strengthCCfrom\{0\.001,0\.01,0\.1,1\.0,10\.0\}\\\{0\.001,0\.01,0\.1,1\.0,10\.0\\\}, withclass\_weight=’balanced’to handle residual class imbalance\.

We report test set accuracy as the primary metric\. Our methodology assumes the linear representation hypothesis\(Zhuet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib9)\): if correct answer information exists in a layer’s representations, a simple linear classifier should reliably extract it\. Higher probe accuracy indicates more robust linear separability between correct and incorrect answer representations\.

![Refer to caption](https://arxiv.org/html/2607.26119v1/layer_performance_regression.png)Figure 1:Layer\-wise probe accuracy for predicting answer correctness across model families\. Reasoning models \(DeepSeek\-Math\-7B\-RL, Olmo\-3\-Think\) achieve higher probe accuracy \(83–98%\) and earlier emergence compared to base and instruction\-tuned models \(DeepSeek\-Math\-Instruct, Olmo\-3\-Instruct\) \(75–90%\)\. Notable late\-layer regression appears in final layers for all models\.

### Results: Layer\-Wise Probe Accuracy

Motivation\.We hypothesize that the improved mathematical reasoning ability of reasoning\-capable models on mathematical benchmarks stems from developing*clearer*internal representations of correctness—i\.e\., representations that are more linearly separable and consistently structured across samples\. To test this, we train linear probes to classify correct vs\. incorrect solutions at each layer, using probe accuracy as a proxy for representation clarity\.

### Reasoning Models Exhibit Representation Clarity

Figure[1](https://arxiv.org/html/2607.26119#Sx2.F1)reveals substantial differences in how correctness information is encoded across model types\. Models trained with reinforcement learning from verifiable rewards—DeepSeek\-Math\-7B\-RL and Olmo\-3\-Think—achieve markedly higher probe accuracy \(83–98%\) compared to instruction\-tuned models \(75–90%\)\. This 8 percentage point gap, combined with noticeably reduced variance between individual samples \(tighter scatter in Figure[1](https://arxiv.org/html/2607.26119#Sx2.F1)\), suggests that RL and chain\-of\-thought training could lead to representations with linear separability between correct and incorrect answer states\(Zhanget al\.[2025](https://arxiv.org/html/2607.26119#bib.bib18); Parket al\.[2024](https://arxiv.org/html/2607.26119#bib.bib19)\)\.

Immediate emergence in reasoning models\.Reasoning\-capable models exhibit remarkably high probe accuracy from the very first layer, with test accuracies of 70% at layer 0 compared to 65% for Instruct models\. Both Olmo\-3\-Think and DeepSeek\-Math\-7B\-RL maintain∼\\sim94–95% accuracy throughout layers 15–29\. This immediate availability of correctness information contrasts sharply with instruction\-tuned models, where probe accuracy gradually improves from∼\\sim65% to∼\\sim66–80% and 84–90% for DeepSeek\-Math\-Instruct and Olmo\-3\-Instruct respectively over 28 layers\. Defining anemergence layerℓemerge\\ell\_\{\\text\{emerge\}\}as the first layer achieving\>80%\>80\\%test accuracy, we findℓemerge=0\\ell\_\{\\text\{emerge\}\}=0and22for DeepSeek\-Math\-RL and Olmo\-3\-Think respectively, andℓemerge=6\\ell\_\{\\text\{emerge\}\}=6and11for DeepSeek\-Math\-Instruct and Olmo\-3\-Instruct respectively\. The threshold for emergence layer must be dynamically defined based on accuracy across all models, as some thresholds are crossed by all models from layer 0\. Despite similar emergence layers for the Olmo\-3 family, the Think model achieves higher overall accuracy\.

Pre\-training and training objectives matter\.The Olmo\-3\-Instruct model, fine\-tuned from the base model using the Dolci Instruct SFT dataset, outperforms DeepSeek\-Math\-7B\-Instruct\. This demonstrates that a more robust and diverse dataset for pre\-training and SFT can still provide significant performance increases, which is supported by our probe findings where the accuracy for Olmo\-3\-Instruct is significantly higher\(Mosbachet al\.[2020](https://arxiv.org/html/2607.26119#bib.bib20); Zhou and Srikumar[2022](https://arxiv.org/html/2607.26119#bib.bib21)\)\. DeepSeek\-Math\-7B\-RL and Olmo\-3\-Think substantially outperform their instruction\-tuned counterparts, demonstrating that reasoning\-specific training objectives—not merely scale or general fine\-tuning—are key to developing clear correctness representations\.

### Representation Clarity as a Mechanism for Better Mathematical Reasoning

Our results provide a potential mechanistic explanation for why RL\-trained and chain\-of\-thought models outperform instruction\-tuned models on mathematical benchmarks: they develop fundamentally clearer internal representations of correctness\. The high probe accuracy \(75–95%\) indicates that these models encode correctness as robust, linearly\-separable features\(Parket al\.[2024](https://arxiv.org/html/2607.26119#bib.bib19); Jianget al\.[2024](https://arxiv.org/html/2607.26119#bib.bib22)\)\. This clarity likely enables more reliable access to correctness signals during autoregressive generation, leading to more consistent correct outputs\.

Caveats\.Our probing methodology has an important limitation: it requires the model to produce a sufficiently balanced distribution of correct and incorrect answers\. When the correct\-to\-incorrect ratio deviates substantially from 50:50, the probe risks learning to predict the majority class rather than genuinely detecting representational differences\. This dependency means our findings are most reliable for models operating near their capability boundaries, where both outcomes occur with reasonable frequency\.

## Layer\-Wise Mean Ablations

We aim to investigate the criticality each layer has upon the mathematical reasoning capabilities of DeepSeek\-Math models through systematic activation patching\. Our methodology employs mean ablation interventions to replace layer activations with their corresponding mean values computed from a reference dataset \(GSM8K training data\)\. This approach follows the established activation patching protocols introduced by Zhang and Nanda \(2023\)\.

### Experimental Setup

We evaluate theDeepSeek\-Math\-7B\-InstructandDeepSeek\-Math\-7B\-RLmodels on 20 GSM8K problems per model\. For each layerℓ∈\{0,1,…,L−1\}\\ell\\in\\\{0,1,\\ldots,L\-1\\\}, we replace the activationhℓh\_\{\\ell\}with its corresponding reference mean activationμℓ\\mu\_\{\\ell\}and measure the resulting degradation in accuracy\.

### Evaluation Metric

Accuracy Drop \(AD\):This metric quantifies the change in model accuracy relative to the baseline performance\. For each layerℓ\\ell, we compute:ADℓ=Accbase−Accℓabl,\\mathrm\{AD\}\_\{\\ell\}=\\mathrm\{Acc\}\_\{\\text\{base\}\}\-\\mathrm\{Acc\}\_\{\\ell\}^\{\\text\{abl\}\},whereAccbase\\mathrm\{Acc\}\_\{\\text\{base\}\}is the baseline accuracy \(without ablation\), andAccℓabl\\mathrm\{Acc\}\_\{\\ell\}^\{\\text\{abl\}\}is the accuracy measured when the activations at layerℓ\\ellare replaced byμℓ\\mu\_\{\\ell\}\. Larger values ofADℓ\\mathrm\{AD\}\_\{\\ell\}indicate higher importance of that layer in mathematical reasoning\.

Pearson Correlation Coefficient \(r\):This statistic measures the linear correlation between layer depth and accuracy drop\. A positiverrindicates that deeper layers are more critical to performance, while a value near zero implies that importance is distributed evenly across the network\.

### Implementation

We extract final answers using pattern matching on the`\\boxed\{\.\.\.\}`notation\. All generations use fixed decoding parameters \(temperature = 0\.1, top\_p = 0\.9\)\. Prompts enforce structured, step\-by\-step reasoning to ensure that mathematical problem\-solving processes are made explicit\.

### Results and Analysis

![Refer to caption](https://arxiv.org/html/2607.26119v1/figure3.png)Figure 2:Accuracy Drop \(AD\) across layers for DeepSeek\-Math\-7B\-Instruct and DeepSeek\-Math\-7B\-RL\.
### Layer Criticality Patterns

As shown in Figure[2](https://arxiv.org/html/2607.26119#Sx3.F2), the two DeepSeek variants exhibit distinct computational architectures\.DeepSeek\-Math\-7B\-RL\(baseline acc\. 70%\) exhibits a significant positive correlation between layer depth and intervention impact \(r=0\.47r=0\.47,p<0\.01p<0\.01\), with AD ranging from−0\.15\-0\.15to\+0\.15\+0\.15\. This indicates that deeper layers become increasingly critical for mathematical reasoning\. In contrast,DeepSeek\-Math\-7B\-Instruct\(baseline accuracy: 65%\) demonstrates a weak negative correlation \(r=−0\.11r=\-0\.11,p=0\.55p=0\.55\), with AD ranging from−0\.20\-0\.20to\+0\.05\+0\.05, suggesting relatively flat layer importance with slight emphasis on early layers\.

### Layer Criticality Interpretation

These contrasting patterns reflect distinct computational strategies shaped by training objectives\. The RL\-trained model shows higher early\-layer impact combined with progressive deepening, indicating hierarchical reasoning architecture with concentration in layers 9\-18 and 22\-26\. Conversely, the instruction\-tuned model displays a distributed reasoning profile, suggesting that supervision across full reasoning trajectories encourages balanced layer utilization and introduces redundancy that enhances robustness to perturbations\.

### Convergence and Divergence Points

Both models exhibit similar vulnerability across layers 0–10 \(AD≈\\approx\-0\.15 to 0\.00\), indicating shared foundational mechanisms likely responsible for arithmetic operations and core reasoning primitives\. Beyond layer 15, however, their trajectories diverge sharply, demonstrating that training methodology fundamentally reshapes higher\-order mathematical reasoning\. This divergence has implications for performance optimization and failure mode identification\.

## Token Variability in Mathematical Problem\-Solving

We investigate whether the representational differences observed in our previous findings manifest in downstream behaviours such as token variability between RL and SFT models\.

### Experimental Setup

Models:We evaluate four models spanning two architectural families:

DeepSeekMath family:DeepSeekMath\-Instruct,DeepSeekMath\-RL\(Shaoet al\.[2024](https://arxiv.org/html/2607.26119#bib.bib3)\)

Olmo 3 family:Olmo\-3\-Instruct,Olmo\-3\-Thinking\(Olmoet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib31)\)\.

Data and Methodology\.We evaluate on 50 problems from GSM8K\-Platinum\(Vendrowet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib1)\), generating 50 independent responses per problem per model \(15,000 responses per model\)\. We measure answer correctness, input tokens, and output tokens \(including reasoning tokens for LRMs\)\. Full experimental details are provided in Appendix[A](https://arxiv.org/html/2607.26119#A1)\.

Evaluation Metrics\.For each problem, we compute: \(1\)answer consistency, the proportion of runs producing correct answers; \(2\)token coefficient of variation \(CV\), computed across the 50 responses per problem asCV=σtokens/μtokens\\mathrm\{CV\}=\\sigma\_\{\\mathrm\{tokens\}\}/\\mu\_\{\\mathrm\{tokens\}\}, whereσtokens\\sigma\_\{\\mathrm\{tokens\}\}andμtokens\\mu\_\{\\mathrm\{tokens\}\}are the standard deviation and mean of output token counts respectively\. We use CV rather than raw standard deviation to enable fair comparison across model families with different baseline output lengths \(LRMs typically generate 5–10×\\timesmore tokens than SFT models\)\. Since all responses contain at least hundreds of tokens, the mean is never near zero, avoiding the instability that CV exhibits whenμ→0\\mu\\to 0; and \(3\)median output tokens, including reasoning tokens for LRMs\.

### Results: Divergent Variability Patterns

![Refer to caption](https://arxiv.org/html/2607.26119v1/bargraph_olmo3_accuracy_vs_token_cv.png)\(a\)Olmo 3 models
![Refer to caption](https://arxiv.org/html/2607.26119v1/bargraph_deepseek_accuracy_vs_token_cv.png)\(b\)DeepSeek\-Math models

Figure 3:Token coefficient of variation by accuracy range across model families\. \(a\) Olmo\-3\-Thinking and Olmo\-3\-Instruct maintain consistent low variability \(CV=0\.1\\mathrm\{CV\}=0\.1–0\.1250\.125\) across all bins where data exists\. \(b\) DeepSeek\-Math models exhibit decreasing variability, with the highest variability observed in the hardest\-difficulty region \(0–20%\) and decreasing to the lowest at 80–100%, demonstrating compression inefficiency at capability boundaries\.Model families exhibit fundamentally different variability profiles\.Figure[3\(b\)](https://arxiv.org/html/2607.26119#Sx4.F3.sf2)reveals that DeepSeek\-Math\-RL exhibits high variability across difficulty levels, peaking in the 40–60% accuracy region \(CV≈1\.3\\mathrm\{CV\}\\approx 1\.3\) and remaining elevated even at capability boundaries \(CV≈0\.95\\mathrm\{CV\}\\approx 0\.95at 0–20%,CV≈0\.35\\mathrm\{CV\}\\approx 0\.35at 80–100%\)\. DeepSeek\-Math\-Instruct shows consistently lower variability across all bins\. In contrast, Figure[3\(a\)](https://arxiv.org/html/2607.26119#Sx4.F3.sf1)shows both Olmo\-3\-Thinking and Olmo\-3\-Instruct maintain remarkably consistent low variability \(CV=0\.1\\mathrm\{CV\}=0\.1–0\.1250\.125\) in the bins where data exists \(0–20% and 80–100%\), suggesting qualitatively different generation strategies\.

Empirical evidence for compression theory\.Our findings partially supportLeeet al\.\([2025](https://arxiv.org/html/2607.26119#bib.bib12)\)’s token complexity framework\. DeepSeek\-Math\-RL exhibits high variability at capability boundaries \(CV\>0\.8\\mathrm\{CV\}\>0\.8\), meaning identical problems elicit vastly different response lengths—precisely the calibration failure predicted by information\-theoretic analysis\. Even at high accuracy where models should reliably compress, non\-trivial variability persists \(CV≈0\.35\\mathrm\{CV\}\\approx 0\.35\), indicating systematic deviation from optimal compression\. However, the Olmo\-3 family demonstrates that consistent low variability is achievable across accuracy extremes\.

RL training shows variable impact on adaptive allocation\.Contrary to expectations, DeepSeek\-Math\-RL exhibits substantiallyhighervariability than DeepSeek\-Math\-Instruct across all accuracy bins, suggesting that RL training in this case amplified rather than reduced output inconsistency\. In contrast, both Olmo\-3\-Thinking and Olmo\-3\-Instruct maintain nearly identical low\-variability profiles\. This divergence suggests that the relationship between training methodology and token allocation consistency is model\-dependent, likely influenced by the specific training pipeline and reward structure rather than RL versus SFT alone\.

## Conclusion

We investigated the mechanistic basis of reinforcement learning’s success in mathematical reasoning through integrated behavioural\-mechanistic analysis\. Our findings reveal a coherent picture: RL\-trained models develop superior representations that emerge earlier in the network; Specifically, in the DeepSeek model family, RL\-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing\. Linear probing shows RL models achieve higher accuracy in predicting answer correctness, with representations emerging in earlier layers than in SFT models, while ablation studies confirm these representations are functionally critical\. Token variability analysis reveals model\-dependent patterns: while Olmo\-3 models maintain consistent generation across difficulty levels regardless of training method, DeepSeek\-Math\-RL exhibits higher variability than its SFT counterpart—suggesting that the relationship between RL training and output consistency depends on the specific training pipeline\.

Multiple promising avenues emerge from this work\. Designing reward structures that explicitly incentivize adaptive token allocation could better exploit RL’s potential\. Our layer\-wise analysis focused on answer correctness; extending probing to intermediate reasoning steps could reveal how multi\-step solutions are constructed and validated\. Scaling this analysis to larger models and diverse reasoning domains \(code generation, scientific reasoning\) would test whether our findings generalize beyond mathematical problem\-solving\. More broadly, developing methods to detect and measure representation quality during training could enable real\-time assessment of model reliability – a critical need for deploying reasoning models in high\-stakes domains\.

## References

- G\. Alain and Y\. Bengio \(2018\)Understanding intermediate layers using linear classifier probes\.External Links:1610\.01644,[Link](https://arxiv.org/abs/1610.01644)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1),[Measuring Representation Quality via Probing](https://arxiv.org/html/2607.26119#Sx2.p1.1)\.
- Y\. Belinkov \(2022\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.External Links:[Link](https://aclanthology.org/2022.cl-1.7/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00422)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1),[Probe Training](https://arxiv.org/html/2607.26119#Sx2.SSx1.SSSx3.p1.9)\.
- X\. Chen, A\. Plaat, and N\. van Stein \(2025\)How does chain of thought think? mechanistic interpretability of chain\-of\-thought reasoning with sparse autoencoding\.External Links:2507\.22928,[Link](https://arxiv.org/abs/2507.22928)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Synthetic Problem Generation](https://arxiv.org/html/2607.26119#Sx2.SSx1.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p4.1)\.
- M\. Hanna, O\. Liu, and A\. Variengien \(2023\)How does gpt\-2 compute greater\-than?: interpreting mathematical abilities in a pre\-trained language model\.External Links:2305\.00586,[Link](https://arxiv.org/abs/2305.00586)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1)\.
- Y\. Jiang, G\. Rajendran, P\. Ravikumar, B\. Aragam, and V\. Veitch \(2024\)On the origins of linear representations in large language models\.External Links:2403\.03867,[Link](https://arxiv.org/abs/2403.03867)Cited by:[Representation Clarity as a Mechanism for Better Mathematical Reasoning](https://arxiv.org/html/2607.26119#Sx2.SSx4.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Appendix A](https://arxiv.org/html/2607.26119#A1.p3.1)\.
- A\. Lee, E\. Che, and T\. Peng \(2025\)How well do llms compress their own chain\-of\-thought? a token complexity approach\.External Links:2503\.01141,[Link](https://arxiv.org/abs/2503.01141)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1),[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1),[Results: Divergent Variability Patterns](https://arxiv.org/html/2607.26119#Sx4.SSx2.p2.2)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2023\)Locating and editing factual associations in gpt\.External Links:2202\.05262,[Link](https://arxiv.org/abs/2202.05262)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1)\.
- I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. Farajtabar \(2025\)GSM\-symbolic: understanding the limitations of mathematical reasoning in large language models\.External Links:2410\.05229,[Link](https://arxiv.org/abs/2410.05229)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1),[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1),[Synthetic Problem Generation](https://arxiv.org/html/2607.26119#Sx2.SSx1.p1.1),[Synthetic Problem Generation](https://arxiv.org/html/2607.26119#Sx2.SSx1.p2.1)\.
- M\. Mosbach, A\. Khokhlova, M\. A\. Hedderich, and D\. Klakow \(2020\)On the interplay between fine\-tuning and sentence\-level probing for linguistic knowledge in pre\-trained transformers\.InBlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,External Links:[Link](https://api.semanticscholar.org/CorpusID:222141026)Cited by:[Reasoning Models Exhibit Representation Clarity](https://arxiv.org/html/2607.26119#Sx2.SSx3.p3.1)\.
- T\. Olmo, :, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2025\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[item Olmo 3 family:](https://arxiv.org/html/2607.26119#Sx4.I1.ix2.p1.1)\.
- OpenAI, :, A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney, A\. Iftimie, A\. Karpenko, A\. T\. Passos, A\. Neitz, A\. Prokofiev, A\. Wei, A\. Tam, A\. Bennett, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Duberstein, A\. Kondrich, A\. Mishchenko, A\. Applebaum, A\. Jiang, A\. Nair, B\. Zoph, B\. Ghorbani, B\. Rossen, B\. Sokolowsky, B\. Barak, B\. McGrew, B\. Minaiev, B\. Hao, B\. Baker, B\. Houghton, B\. McKinzie, B\. Eastman, C\. Lugaresi, C\. Bassin, C\. Hudson, C\. M\. Li, C\. de Bourcy, C\. Voss, C\. Shen, C\. Zhang, C\. Koch, C\. Orsinger, C\. Hesse, C\. Fischer, C\. Chan, D\. Roberts, D\. Kappler, D\. Levy, D\. Selsam, D\. Dohan, D\. Farhi, D\. Mely, D\. Robinson, D\. Tsipras, D\. Li, D\. Oprica, E\. Freeman, E\. Zhang, E\. Wong, E\. Proehl, E\. Cheung, E\. Mitchell, E\. Wallace, E\. Ritter, E\. Mays, F\. Wang, F\. P\. Such, F\. Raso, F\. Leoni, F\. Tsimpourlas, F\. Song, F\. von Lohmann, F\. Sulit, G\. Salmon, G\. Parascandolo, G\. Chabot, G\. Zhao, G\. Brockman, G\. Leclerc, H\. Salman, H\. Bao, H\. Sheng, H\. Andrin, H\. Bagherinezhad, H\. Ren, H\. Lightman, H\. W\. Chung, I\. Kivlichan, I\. O’Connell, I\. Osband, I\. C\. Gilaberte, I\. Akkaya, I\. Kostrikov, I\. Sutskever, I\. Kofman, J\. Pachocki, J\. Lennon, J\. Wei, J\. Harb, J\. Twore, J\. Feng, J\. Yu, J\. Weng, J\. Tang, J\. Yu, J\. Q\. Candela, J\. Palermo, J\. Parish, J\. Heidecke, J\. Hallman, J\. Rizzo, J\. Gordon, J\. Uesato, J\. Ward, J\. Huizinga, J\. Wang, K\. Chen, K\. Xiao, K\. Singhal, K\. Nguyen, K\. Cobbe, K\. Shi, K\. Wood, K\. Rimbach, K\. Gu\-Lemberg, K\. Liu, K\. Lu, K\. Stone, K\. Yu, L\. Ahmad, L\. Yang, L\. Liu, L\. Maksin, L\. Ho, L\. Fedus, L\. Weng, L\. Li, L\. McCallum, L\. Held, L\. Kuhn, L\. Kondraciuk, L\. Kaiser, L\. Metz, M\. Boyd, M\. Trebacz, M\. Joglekar, M\. Chen, M\. Tintor, M\. Meyer, M\. Jones, M\. Kaufer, M\. Schwarzer, M\. Shah, M\. Yatbaz, M\. Y\. Guan, M\. Xu, M\. Yan, M\. Glaese, M\. Chen, M\. Lampe, M\. Malek, M\. Wang, M\. Fradin, M\. McClay, M\. Pavlov, M\. Wang, M\. Wang, M\. Murati, M\. Bavarian, M\. Rohaninejad, N\. McAleese, N\. Chowdhury, N\. Chowdhury, N\. Ryder, N\. Tezak, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, P\. Chao, P\. Ashbourne, P\. Izmailov, P\. Zhokhov, R\. Dias, R\. Arora, R\. Lin, R\. G\. Lopes, R\. Gaon, R\. Miyara, R\. Leike, R\. Hwang, R\. Garg, R\. Brown, R\. James, R\. Shu, R\. Cheu, R\. Greene, S\. Jain, S\. Altman, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Hernandez, S\. Baker, S\. McKinney, S\. Yan, S\. Zhao, S\. Hu, S\. Santurkar, S\. R\. Chaudhuri, S\. Zhang, S\. Fu, S\. Papay, S\. Lin, S\. Balaji, S\. Sanjeev, S\. Sidor, T\. Broda, A\. Clark, T\. Wang, T\. Gordon, T\. Sanders, T\. Patwardhan, T\. Sottiaux, T\. Degry, T\. Dimson, T\. Zheng, T\. Garipov, T\. Stasi, T\. Bansal, T\. Creech, T\. Peterson, T\. Eloundou, V\. Qi, V\. Kosaraju, V\. Monaco, V\. Pong, V\. Fomenko, W\. Zheng, W\. Zhou, W\. McCabe, W\. Zaremba, Y\. Dubois, Y\. Lu, Y\. Chen, Y\. Cha, Y\. Bai, Y\. He, Y\. Zhang, Y\. Wang, Z\. Shao, and Z\. Li \(2024\)OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.26119#Sx1.p4.1)\.
- K\. Park, Y\. J\. Choe, and V\. Veitch \(2024\)The linear representation hypothesis and the geometry of large language models\.External Links:2311\.03658,[Link](https://arxiv.org/abs/2311.03658)Cited by:[Reasoning Models Exhibit Representation Clarity](https://arxiv.org/html/2607.26119#Sx2.SSx3.p1.1),[Representation Clarity as a Mechanism for Better Mathematical Reasoning](https://arxiv.org/html/2607.26119#Sx2.SSx4.p1.1)\.
- M\. Sachan, A\. Stolfo, and Y\. Sun \(2025\)Probing for arithmetic errors in language models\.External Links:2507\.12379,[Link](https://arxiv.org/abs/2507.12379)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Vol\.abs/2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p1.1),[item DeepSeekMath family:](https://arxiv.org/html/2607.26119#Sx4.I1.ix1.p1.1)\.
- J\. Vendrow, E\. Vendrow, S\. Beery, and A\. Madry \(2025\)Do large language model benchmarks test reliability?\.External Links:2502\.03461,[Link](https://arxiv.org/abs/2502.03461)Cited by:[Appendix A](https://arxiv.org/html/2607.26119#A1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.26119#Sx4.SSx1.p2.1)\.
- E\. Yeo, Y\. Tong, M\. Niu, G\. Neubig, and X\. Yue \(2025\)Demystifying long chain\-of\-thought reasoning in llms\.External Links:2502\.03373,[Link](https://arxiv.org/abs/2502.03373)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1)\.
- A\. Zhang, Y\. Chen, J\. Pan, C\. Zhao, A\. Panda, J\. Li, and H\. He \(2025\)Reasoning models know when they’re right: probing hidden states for self\-verification\.External Links:2504\.05419,[Link](https://arxiv.org/abs/2504.05419)Cited by:[Reasoning Models Exhibit Representation Clarity](https://arxiv.org/html/2607.26119#Sx2.SSx3.p1.1)\.
- F\. Zhang and N\. Nanda \(2024\)Towards best practices of activation patching in language models: metrics and methods\.External Links:2309\.16042,[Link](https://arxiv.org/abs/2309.16042)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p3.1)\.
- Y\. Zhou and V\. Srikumar \(2022\)A closer look at how fine\-tuning changes bert\.External Links:2106\.14282,[Link](https://arxiv.org/abs/2106.14282)Cited by:[Reasoning Models Exhibit Representation Clarity](https://arxiv.org/html/2607.26119#Sx2.SSx3.p3.1)\.
- F\. Zhu, D\. Dai, and Z\. Sui \(2024\)Language models encode the value of numbers linearly\.External Links:2401\.03735,[Link](https://arxiv.org/abs/2401.03735)Cited by:[Introduction](https://arxiv.org/html/2607.26119#Sx1.p2.1),[Probe Training](https://arxiv.org/html/2607.26119#Sx2.SSx1.SSSx3.p2.1),[Measuring Representation Quality via Probing](https://arxiv.org/html/2607.26119#Sx2.p1.1)\.

## Appendix AAdditional Token Variability Experimental Details

Dataset:50 problems randomly sampled from the GSM8K\-Platinum test split\(Vendrowet al\.[2025](https://arxiv.org/html/2607.26119#bib.bib1)\)\(seed=42\)\.

Generation parameters:

- •DeepSeekMath\-Instruct: TemperatureT=0\.6T=0\.6, Max tokens = 4096
- •DeepSeekMath\-RL: TemperatureT=0\.6T=0\.6, Max tokens = 4096
- •Olmo\-3\-Instruct: TemperatureT=0\.6T=0\.6, Top\-pp=0\.95p=0\.95, Max tokens = 32768
- •Olmo\-3\-Think: TemperatureT=0\.6T=0\.6, Top\-pp=0\.95p=0\.95, Max tokens = 32768

Model Inference:We use vLLM\(Kwonet al\.[2023](https://arxiv.org/html/2607.26119#bib.bib32)\)on a single GH200 GPU for efficient execution\.

Prompt Template:

```
System: Please reason step by step, and
put your final answer within \boxed{}.
User: {question}
```

## Appendix BAdditional Details for Linear Probing

Problem templates:Each template instantiates a word problem with randomized numerical parameters while maintaining fixed logical structure:

1. 1\.Conditional probability: Calculate probability of turning in homework given sequential conditional events \(substitute teacher, class extension, personal extension\)\. Requires probability multiplication and complementary probability computation\. Answers range from 8–50%\.
2. 2\.Student demographics: Given total students, age threshold, and gender ratios stratified by age group, compute total female students\. Requires division, fraction multiplication, and subtraction\. Answers range from 220–3,986 students\.
3. 3\.Sequential growth: Given initial water flow and multiplicative/additive growth rules over days, compute final quantity\. Requires tracking state across time steps with doubling and addition\. Answers range from 7,057–25,513 gallons\.
4. 4\.Counting with unit conversion: Track brownies \(in dozens\) received and consumed across multiple events, then convert to individual items\. Requires dozen\-to\-unit conversion, fraction addition/subtraction, and summation\. Answers range from 1–231 brownies\.
5. 5\.Cost calculation: Given base price and dependent pricing rules \(e\.g\., “leather seats cost one\-third of the king cab upgrade”\), compute total cost\. Requires chained fraction operations and summation\. Answers range from$​34,490\\mathdollar 34,490–$​66,846\\mathdollar 66,846\.

Each problem includes the instruction: “Please reason step by step, and put your final answer within\\boxed\{answer\}as an integer\.” Answers range from 100 to 10,000, ensuring consistent numerical magnitude\. The following are the graphs for the question each of the question types

### B\.7 Per\-Template Probe Accuracy Results

The following figures show layer\-wise probe accuracy for each of the five synthetic problem templates\. Each figure displays test set accuracy across all transformer layers for the four models evaluated\.

![Refer to caption](https://arxiv.org/html/2607.26119v1/cost_comparison.png)Figure 4:Probing classification accuracy vs\. transformer layer\.Cost Calculation Problem: DeepSeek\-Math\-7B\-RL, DeepSeek\-Math\-7B\-Instruct, Olmo\-3\-Think, Olmo\-3\-Instruct\.![Refer to caption](https://arxiv.org/html/2607.26119v1/layer_performance_regression.png)Figure 5:Probing classification accuracy vs\. transformer layer\.Student Demographics Problem: DeepSeek\-Math\-7B\-RL, DeepSeek\-Math\-7B\-Instruct, Olmo\-3\-Think, Olmo\-3\-Instruct![Refer to caption](https://arxiv.org/html/2607.26119v1/seq_growth.png)Figure 6:Probing classification accuracy vs\. transformer layer\.Sequential Growth Problem: DeepSeek\-Math\-7B\-RL, DeepSeek\-Math\-7B\-Instruct, Olmo\-3\-Think, Olmo\-3\-Instruct![Refer to caption](https://arxiv.org/html/2607.26119v1/prob.png)Figure 7:Probing classification accuracy vs\. transformer layer\.Conditional Probability Problem: DeepSeek\-Math\-7B\-RL, DeepSeek\-Math\-7B\-Instruct, Olmo\-3\-Think, Olmo\-3\-InstructSample Questions from each of the categories:

1. 1\.Conditional probability:Yasmine is trying to decide whether they really need to do their homework\. There’s a 70% chance that tomorrow they’ll have a substitute teacher who won’t collect the homework\. Even if the normal teacher comes in, there’s a 60% chance she’ll give everyone an extension\. Even if the whole class doesn’t get an extension, there’s a 25% chance Yasmine can convince the teacher their dog ate their assignment and get a personal extension\. What is the percentage chance that Yasmine will actually have to turn in their homework tomorrow?\(Answer: 9%\)
2. 2\.Student demographics:Brook Hills High School currently enrolls 4,374 students\. Half of these students are over 18 years old, and one\-fifth of the students over 18 years old are male\. The remaining half of the students are under 18 years old, and 2/5 of the students under 18 are male\. In total, how many female students are enrolled at this school?\(Answer: 3,062 students\)
3. 3\.Sequential growth:The amount of water passing through a river at one point in time is 5,904 gallons\. After a day of heavy rain, the amount of water passing through the river doubles at the same point\. If the volume of water passing through the river at that point increases by 7,202 gallons on the third day, calculate the total amount of water passing through the river at that point\.\(Answer: 19,010 gallons\)
4. 4\.Counting with unit conversion:Quentin wanted brownies for her birthday\. She made a batch for herself; nine dozen Nut Brownies\. At her office, they threw her a party and sent her home with 9/10 dozen brownies\. When she arrived home, her friends were there to throw her a surprise party and had 4 dozen brownies waiting\. During the party, 2 2/10 dozen brownies were eaten\. How many individual brownies did Quentin have left over from the entire day?\(Answer: 140 brownies\)
5. 5\.Cost calculation:Bill is ordering a new truck\. He has decided to purchase a two\-ton truck with several added features: a king cab upgrade, a towing package, leather seats, running boards, and the upgraded exterior light package\. The base price of the truck is $42,572, and the other features are at extra cost\. The king cab is an extra $6,890, leather seats are one\-third the cost of the king cab upgrade, running boards are $500 less than the leather seats, and the upgraded exterior light package is $1,724\. What is the total cost of Bill’s new truck, in dollars?\(Answer: $55,278\)

Reproducibility:Code and data is provided in the github link