BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

# BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?
Source: [https://arxiv.org/html/2608.02867](https://arxiv.org/html/2608.02867)
Soumadeep Saha†\\dagger,Krish Sharma†\\dagger,Akshay Chaturvedi\*, Nicholas Asher†\\dagger

†\\daggerANITI, Université de Toulouse;\*LINAGORA Labs Correspondence:[soumadeep\.saha97@gmail\.com](https://arxiv.org/html/2608.02867v1/mailto:[email protected])

###### Abstract

Although reinforcement learning with verifiable rewards \(RLVR\) has improved the performance of large language models \(LLMs\) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency\. In this paper, we investigate the nature of test\-time exploration in RLVR\-trained LLMs by employing controlled maze\-solving experiments and extracting a tree structure from mathematical reasoning traces \(BODHI\-Trees\) based on semantic equivalence\. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching\. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy\. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity\.

BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

Soumadeep Saha†\\dagger, Krish Sharma†\\dagger, Akshay Chaturvedi\*,Nicholas Asher†\\dagger†\\daggerANITI, Université de Toulouse;\*LINAGORA LabsCorrespondence:[soumadeep\.saha97@gmail\.com](https://arxiv.org/html/2608.02867v1/mailto:[email protected])

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.02867v1/images/tree.png)Figure 1:Construction of mathematical reasoning trees\.We first segment reasoning traces into a chain of reasoning nodes, followed by merging said nodes based on semantic similarity\. Further details are provided in Appendix[A](https://arxiv.org/html/2608.02867#A1)\.111All associated artifacts can be found at[espressovi\.github\.io/BODHI](https://espressovi.github.io/BODHI)\.Several recent papers have pointed to a curious problem in*reinforcement learning with verifiable rewards*\(RLVR\)\-trained large language models \(LLMs\)\. These models do not seem to “discover” novel abilities beyond the base model, are less exploratory, and are reliant on a minority of high\-entropy tokens to drive reasoning diversity\(Chen et al\.,[2026a](https://arxiv.org/html/2608.02867#bib.bib4); Saha et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib28); Wang et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib34); Yue et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib42)\)\. An exploratory model with the ability to access a large number of*valid, semantically distinct*continuations is evidently desirable—it allows for the realisation of novel solutions, increased diversity, and even performance scaling with techniques such as self\-consistency\(Wang et al\.,[2023](https://arxiv.org/html/2608.02867#bib.bib35)\), tree\-of\-thought\(Yao et al\.,[2023](https://arxiv.org/html/2608.02867#bib.bib39)\), etc\. However, it is also clear that*not all exploration is desirable*\. For instance, an LLM with the capability to produce a large set of*valid continuations*, i\.e\., correct mathematical reasoning traces, which differ in superficial ways such as the naming of variables, order of commutative operations, or minor syntactic variations in its natural language \(NL\) framing does not constitute meaningful exploration, as it fails to traverse fundamentally distinct reasoning pathways or yield novel solution strategies\.

This brings us to the core tenet motivating this study:*do LLMs Branch Out and Discover Heterogeneous Inferences*, or is the exploration largely decorative? To this end, we pose threeresearch questionsin this study:

⋄\\diamondRQ1: Does RLVR limit test\-time exploration and demonstrate stronger preferences for certain trajectories?

⋄\\diamondRQ2: If so, is the policy concentration merely syntactic/stylistic?

⋄\\diamondRQ3: And how is the RLVR\-induced policy shift connected to performance?

Surprisingly, despite many recent papers making fundamental improvements to our understanding of the dynamics of RL training, especially pertaining to preventing policy collapse, encouraging exploration, etc\.\(Yu et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib40); Liu et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib22); Li et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib21); Yang et al\.,[2025b](https://arxiv.org/html/2608.02867#bib.bib38); Zhao et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib43)\), the fundamental question ofhow RLVR changes preferencesamong semantically distinct reasoning continuations remains under\-explored\.

We investigate exploration in LLMs through two modalities:mazes, where the notion of exploration is rather straightforward, andmathematical reasoning, where it is much less straightforward due to the complexities of the mathematical and linguistic expressions used\. In order to investigate exploration in mathematical reasoning, we concretize the notion of exploration by collapsing a large number of alternate traces corresponding to a problem into a tree structure \(BODHI\-Tree, see Figure[1](https://arxiv.org/html/2608.02867#S1.F1)\), wherein each node represents several semantically identical reasoning traces and its distinct children represent conceptually different continuations, thus allowing for fine\-grained analysis of the nature of exploration\.

The core contributions of our work are as follows:\(i\) Using ourBODHI\-Treedataset and mazes, we demonstrate that the policy resulting from RL\-training shows a collapse in branching behavior, \(ii\) the collapse in entropy is*not just syntactic*, and RLVR\-trained models also demonstrate a*significant collapse in the semantic\-branch preference entropy*, and \(iii\) further, we provide evidence suggesting that*RLVR drives performance by constricting the accessible state space of both invalid continuations and valid, semantically distinct ones*\. Additionally, in the course of this study we created and open\-sourced several fully\-reproducible intermediate post\-training checkpoints \(in addition to datasets and code\), to aid future work into RLVR dynamics\.

## 2Background

*RLVR*reframes next\-token prediction as a Markov Decision Process, learning a policyπθ\\pi\_\{\\theta\}for predicting next tokensoio\_\{i\}, that leads to trajectories of states\[Input\]→\[Input,o1\]→…→\[Input,o1,…​oT\]\[\\textrm\{Input\}\]\\rightarrow\[\\textrm\{Input\},o\_\{1\}\]\\rightarrow\\ldots\\rightarrow\[\\textrm\{Input\},o\_\{1\},\\ldots o\_\{T\}\], that maximize expected reward from deterministic verifiers\. This paradigm has significantly improved overall performance on several reasoning and planning benchmarks, and has pushed the frontiers of capabilities such as integration of tools and*agentic*applications\. It also enables an ancillary technique, called*Long\-CoT distillation*or simply*distillation*\(Guo et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib9); Shao et al\.,[2024](https://arxiv.org/html/2608.02867#bib.bib30)\), wherein an LLM is fine\-tuned \(SFT\) on traces generated from a more capable RL\-trained “teacher” LLM\.

Given a statests\_\{t\}, the set of continuations—every finite string withsts\_\{t\}as a prefix—can be segmented into various classes which are “*verifier equivalent*”\. However, since the verifier often only evaluates the last few tokens of the terminal state\(Shao et al\.,[2024](https://arxiv.org/html/2608.02867#bib.bib30); Guo et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib9); Yu et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib40); Liu et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib22)\), verifier equivalence does not cleanly map to semantic equivalence\. For instance, a specious continuation which happens to present the correct answer has the same utility as a semantically correct continuation, and there is no a priori guarantee that verifier\-equivalent but semantically diverse continuations have an equal propensity of realization\(Anschel et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib1)\)\. This serves as the core object of analysis in our study: the*accessibility of different verifier\-equivalent trajectories*222i\.e\., the probabilities of different \(finite\) continuations that are verifier\-equivalent\.attest time, a property we refer to asexploration\. This is different from*inter\-trace exploration*\(Jiang et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib18)\), and asks the counterfactual question:*what other continuations were likely?*

Test\-time exploration elicits the “reasoning capability” boundary\(Yue et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib42)\), since, given enough samples, a model capable of accessing diverse trajectories will arrive at novel solution approaches or techniques\. Thus, a litmus test for the efficacy of RL\-training is its ability to explore\. Additionally, exploration can amplify performance with the aid of accessory ensembling techniques such as*tree\-of\-thought*\(Yao et al\.,[2023](https://arxiv.org/html/2608.02867#bib.bib39)\)\. With the proliferation of RL\-trained LLMs, it is therefore critical to understand the*impact of the reward signal on the model’s generative policy*, especially since the assumption that RLVR consistently incentivizes semantically diverse and meaningful exploration has been widely challenged\(Saha et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib28); Wang et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib34); Yue et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib42); Anschel et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib1)\)\.

Yue et al\. \([2025](https://arxiv.org/html/2608.02867#bib.bib42)\)claimed that RL\-trained LLMs have degradedpass​@​k\\textrm\{pass\}@kscaling and do not discover fundamentally novel reasoning pathways beyond the base LLM\.Saha et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib28)\)posited that RL\-trained models are “over\-confident” compared to distilled models: they have a lower entropy policy which is correlated with a drop in performance\. Similarly,Wang et al\. \([2025](https://arxiv.org/html/2608.02867#bib.bib34)\)demonstrated that the RL\-policy is reliant on a minority \(∼20%\\sim 20\\%\) of high\-entropy “forking tokens” to drive meaningful exploration at test time\(Cheng et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib6)\), andJang et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib17)\)attributed this collapse to “over\-confidence at a small number of structurally critical decision points”\.Yuan et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib41)\), however, oppose the view that decline of high\-kpass​@​k\\textrm\{pass\}@kperformance is indicative of a decline in reasoning diversity\. They liken RLVR exploration collapse to “overtraining” and proffer re\-weighting policy updates to favor low\-success rollout groups\.Cai et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib3)\)make a similar diagnosis, and advance “boundary\-aware” curriculum learning to enhance the frontier of reasoning capabilities\.Huang et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib15)\)argue that token\-level statistics do not reflect how reasoning progresses over multi\-token semantic structures\. This debate underscores a critical gap:*existing performance\-based and token\-level metrics do not indicate if entropy collapse implies a lack of reasoning diversity*\.

The collapse of policy entropy during RL\-training has received significant attention, and several recent papers propose modifications to the standard RL\-training regimen\. Monte Carlo Tree Search\-based algorithms\(Li et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib21); Yang et al\.,[2025b](https://arxiv.org/html/2608.02867#bib.bib38); Zheng et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib44)\)that sample rollouts during training in a tree\-like structure and assign segment\-level credits have been proposed\. Other techniques explicitly include cues to enhance exploration based on properties of the rollout group\(Chen et al\.,[2026a](https://arxiv.org/html/2608.02867#bib.bib4)\)\. Techniques such as those based on occurrence frequency\(Anschel et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib1)\), embedding similarity\(Zhao et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib43)\), feedback from auxiliary models\(Mishra et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib23); Hu et al\.,[2026a](https://arxiv.org/html/2608.02867#bib.bib13)\), and many others\(Li and Li,[2026](https://arxiv.org/html/2608.02867#bib.bib20); He et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib11); Cai et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib3); Hu et al\.,[2026b](https://arxiv.org/html/2608.02867#bib.bib14); Jang et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib17); Yuan et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib41); Huang et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib15); Chen et al\.,[2026b](https://arxiv.org/html/2608.02867#bib.bib5)\)have been proposed, with varying degrees of success\(Zhao et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib43); Hao et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib10)\)\.Hao et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib10)\)study factors influencing entropy collapse during training and highlight that recent approaches rely on heuristic adjustments to some of these factors, thus limiting their efficacy\.

In related studies,Jin et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib19)\)studied the relationship between model performance and training entropy, finding a negative but task\-dependent correlation\.Wen et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib36)\)used an*LLM\-as\-a\-judge*framework to evaluate intermediate trace correctness \(CoT\-pass​@​k\\textrm\{pass\}@k\) alongside the answer, demonstrating that RLVR\-trained models have enhanced trace validity compared to the base model\. The results for CoT\-pass​@​k\\textrm\{pass\}@kscaling in mathematical reasoning tasks are mixed, and the authors speculate that the*“distilled LLM may already master major reasoning capabilities that can be learned with RLVR”*, which further underscores the need to isolate the effect of RLVR post\-training on test\-time exploration\.Jiang et al\. \([2025](https://arxiv.org/html/2608.02867#bib.bib18)\)introduce LCoT2Tree, which parses Long CoT traces into trees by mapping segments of reasoning traces to an extracted high\-level summary\. By classifying segment functions \(e\.g\., verification, exploration, backtracking\), they identify structural error patterns such as over\-branching\.

Our work, instead of focusing on token\-level metrics or performance alone, attempts to measure how RLVR changes preferences among semantically distinct reasoning continuations, and tries to ascertain if the entropy collapse is simply an artifact of stronger policy preferences for syntax and style or if the model also has calcified preferences when it comes to semantically diverse verifier\-equivalent continuations\.

## 3Methodology: Measuring Exploration

In this section, we outline the construction of the*probing datasets*that enable us to distinguish differences between the policies of various LLMs with regard to exploration\. While NL mathematical reasoning directly measures an LLM’s logical capabilities, it is inherently noisy; the vast linguistic state space allows models to generate superficial restatements or varying syntactic formulations that mimic diversity without representing genuine inferential branching\. Thus, we employ two modalities: \(i\)*maze solving*, to isolate the routing and planning capabilities and completely eliminate effects of superficial restatements; and \(ii\)*mathematical reasoning*\.

### 3\.1Maze Exploration

![Refer to caption](https://arxiv.org/html/2608.02867v1/x1.png)Figure 2:RL\-trained models are less exploratory\.We plot how often a node is visited by the distilled and RLVR\-trainedQwen3\-8B\.As a probe, maze solving serves as a rigorous, noise\-free proxy for multi\-step logical reasoning\. Maze traversal preserves the fundamental structure of step\-by\-step problem\-solving—the model must sequentially execute valid, discrete actions to reach a goal state—while stripping away the linguistic confounders that arise from NL reasoning tasks\. Because the action space is strictly constrained, alternate trace realizations reflect a shift in the model’s policy\. Furthermore, because local trace validity \(e\.g\., avoiding walls\) and distance to the goal are trivial to formalize, mazes provide an ideal, controlled environment to investigate how RL\-based post\-training alters an LLM’s policy\.

In this study, we employed the[Maze Dataset](https://understanding-search.github.io/maze-dataset/maze_dataset.html)\(Ivanitskiy et al\.,[2023](https://arxiv.org/html/2608.02867#bib.bib16)\)library to instantiate various deterministic graph traversal tasks, constraining the model’s action space to four directional moves:\{<left\>,<right\>,<up\>,<down\>\}\\\{\\texttt\{<left\>\},\\texttt\{<right\>\},\\texttt\{<up\>\},\\texttt\{<down\>\}\\\}\.

### 3\.2Exploration in Mathematical Reasoning

To study exploration in NL mathematical reasoning, given a questionQQ, we first generate several reasoning traces by sampling from a diverse set of frontier LLMs, followed by filtering based on ground truth to get “correct” reasoning responsesA\(1\),A\(2\),…,A\(n\)A^\{\(1\)\},A^\{\(2\)\},\\ldots,A^\{\(n\)\}\.333n≈90n\\approx 90per question, total API budget ofUSD 3000\.These are then segmented \(with the help of an LLM\) into individual*reasoning steps/nodes*aj\(i\)a^\{\(i\)\}\_\{j\}, such thatA\(i\)=concat​\(\[a1\(i\),a2\(i\),…,ani\(i\)\]\)A^\{\(i\)\}=\\textrm\{concat\}\(\[a^\{\(i\)\}\_\{1\},a^\{\(i\)\}\_\{2\},\\ldots,a^\{\(i\)\}\_\{n\_\{i\}\}\]\)\.

With this set of reasoning nodes, and a notion of semantic similaritysim​\(aα\(i\),aβ\(j\)\)\\textrm\{sim\}\(a^\{\(i\)\}\_\{\\alpha\},a^\{\(j\)\}\_\{\\beta\}\), we can create a*prefix tree*T=\(V,E\)T=\(V,E\)of mathematical reasoning traces, called aBODHI\-Tree, where each nodev=\{aα\(i\)\}∈Vv=\\\{a^\{\(i\)\}\_\{\\alpha\}\\\}\\in Vrepresents a set of semantically equivalent mathematical statements, and an edgeu↝v∈Eu\\leadsto v\\in Eif every statement invvcomes after some statement inuuin some reasoning traceA\(i\)A^\{\(i\)\}\(see Figure[1](https://arxiv.org/html/2608.02867#S1.F1), Algorithm[1](https://arxiv.org/html/2608.02867#alg1)\)\. Thus, every distinct path from the root to a leaf in this tree represents an alternate solution to the mathematical reasoning questionQQ\.*Creating such a structure allows us to explicitly analyze differences in exploration between different kinds of LLMs*\.235235questions from[AIME](https://huggingface.co/datasets/gneubig/aime-1983-2024), alongside∼20​K\\sim 20Kresponses and their corresponding trees constitute our probe datasetBODHI\-Tree\. Semantic similarity of two segments is gauged usingGPT\-oss\-120bin an*LLM\-as\-a\-judge*framework\. Further details on the construction ofBODHI\-Trees, like prompts, settings, validation of the*LLM\-as\-a\-judge*framework, etc\., can be found in Appendix[A](https://arxiv.org/html/2608.02867#A1)\.

### 3\.3Metrics

Since this work focuses on the probability of different continuations, the quantity of interest iscandidate preference entropy\(CPE\)\. A branch node fromBODHI\-Tree\(or the tree implicit in maze exploration\) gives us a prefixaa, and at least two alternative completionss\(1\)s^\{\(1\)\}ands\(2\)s^\{\(2\)\}\. Given an LLMPmodelP\_\{\\texttt\{model\}\}we can compute:

Smodel​\(s\(i\)\|a\)=1\|s\(i\)\|​∑k=1\|s\(i\)\|log⁡\(Pm​o​d​e​l​\(sk\(i\)\|s<k\(i\),a\)\)\\begin\{split\}&S\_\{\\texttt\{model\}\}\\big\(s^\{\(i\)\}\|a\\big\)\\\\ &=\\frac\{1\}\{\|s^\{\(i\)\}\|\}\\sum\_\{k=1\}^\{\|s^\{\(i\)\}\|\}\\log\\big\(P\_\{model\}\(s^\{\(i\)\}\_\{k\}\|s^\{\(i\)\}\_\{<k\},a\)\\big\)\\\\ \\end\{split\}\(1\)Which we can normalize to obtain

P​\(s\(i\)\|a\)=exp⁡\(Smodel​\(s\(i\)\|a\)\)∑i=1,2exp⁡\(Smodel​\(s\(i\)\|a\)\),P\(s^\{\(i\)\}\|a\)=\\frac\{\\exp\(S\_\{\\texttt\{model\}\}\(s^\{\(i\)\}\|a\)\)\}\{\\sum\_\{i=1,2\}\\exp\(S\_\{\\texttt\{model\}\}\(s^\{\(i\)\}\|a\)\)\},\(2\)and compute:

CPEmodel​\(s\(1\),s\(2\);a\)=HmodelBranch​\(s\(1\),s\(2\)\)=−∑i=1,2P​\(s\(i\)\|a\)​log⁡\(P​\(s\(i\)\|a\)\)\\begin\{split\}&\\textrm\{CPE\}\_\{\\texttt\{model\}\}\(s^\{\(1\)\},s^\{\(2\)\};a\)=H\_\{\\texttt\{model\}\}^\{\\textrm\{Branch\}\}\(s^\{\(1\)\},s^\{\(2\)\}\)\\\\ &=\-\\sum\_\{i=1,2\}P\(s^\{\(i\)\}\|a\)\\log\\big\(P\(s^\{\(i\)\}\|a\)\\big\)\\end\{split\}\(3\)whereHmodelBranch​\(s\(1\),s\(2\)\)H\_\{\\texttt\{model\}\}^\{\\textrm\{Branch\}\}\(s^\{\(1\)\},s^\{\(2\)\}\)is theconditional entropyin the choice ofs\(1\)s^\{\(1\)\},s\(2\)s^\{\(2\)\}\(orCPE\), given prefixaa, andsk\(i\)s^\{\(i\)\}\_\{k\}is thekkth token ofs\(i\)s^\{\(i\)\}\. Further, since we want to study differences resulting from post\-training, we compute:

Δ​CPEmodel​\(s\(1\),s\(2\);a\)=Δ​HmodelBranch​\(s\(1\),s\(2\)\)=HmodelBranch​\(s\(1\),s\(2\)\)−HbaseBranch​\(s\(1\),s\(2\)\)\\begin\{split\}&\\Delta\\textrm\{CPE\}\_\{\\texttt\{model\}\}\(s^\{\(1\)\},s^\{\(2\)\};a\)=\\Delta H\_\{\\texttt\{model\}\}^\{\\textrm\{Branch\}\}\(s^\{\(1\)\},s^\{\(2\)\}\)\\\\ &=H\_\{\\texttt\{model\}\}^\{\\textrm\{Branch\}\}\(s^\{\(1\)\},s^\{\(2\)\}\)\-H\_\{\\texttt\{base\}\}^\{\\textrm\{Branch\}\}\(s^\{\(1\)\},s^\{\(2\)\}\)\\end\{split\}\(4\)Δ​HmodelBranch\\Delta H\_\{\\texttt\{model\}\}^\{\\textrm\{Branch\}\}is thedifference in conditional preference entropy\(orΔ​CPE\\Delta\\textrm\{CPE\}\) and measures changes in preferences of branches arising from a certain post\-training scheme\.

### 3\.4Models

A notable challenge in analyzing differences arising out of RLVR is the dearth of intermediate post\-training checkpoints\. Popular model vendors usually do not release both the pre\-trained and RLVR checkpoints, and even if these were available, the opacity of the training data makes it hard to discern the effects of RLVR\. Thus, most of our experiments use models where post\-training \(distillation and RLVR\) was performed by us with controlled matched datasets starting from publicly available pre\-training checkpoints\.

Formazes, we distilled \(SFT\) HuggingFace’sSmolLM3\-3B\-Base\(Bakouch et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib2)\)andQwen3\-8B\-Base\(Yang et al\.,[2025a](https://arxiv.org/html/2608.02867#bib.bib37)\)on3030K oracle\-generated solutions, followed by RLVR on1212K samples\. To construct \(LLMBase\\textrm\{LLM\}\_\{\\texttt\{Base\}\},LLMDistil\\textrm\{LLM\}\_\{\\texttt\{Distil\}\},LLMRLVR\\textrm\{LLM\}\_\{\\texttt\{RLVR\}\}\) triplets formathematical reasoning, we selected Google’sgemma\-3\-12b\-pt\(Team et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib31)\)alongside three Qwen foundation models:Qwen3\-8B\-Base,Qwen3\.5\-9B\-Base\(Team,[2026](https://arxiv.org/html/2608.02867#bib.bib32)\), andQwen2\.5\-32B\-Base\(Qwen et al\.,[2024](https://arxiv.org/html/2608.02867#bib.bib27)\)\. These underwent Long\-CoT distillation using5050K samples from the[OpenThoughts\-114k\-math](https://huggingface.co/datasets/open-r1/OpenThoughts-114k-math)dataset, and RLVR employing theDAPO\-Math\-17kdataset\(Yu et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib40)\)\. We directly adoptedDAPO\-Qwen2\.5\-32B\(Yu et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib40)\)as the RL\-trained counterpart forQwen2\.5\-32B\-Basesince it is also trained onDAPO\-Math\-17k\. Further*experimental details are deferred to Appendix[B](https://arxiv.org/html/2608.02867#A2)*\.

## 4Results

### 4\.1Measuring Test\-time Exploration

Figure[2](https://arxiv.org/html/2608.02867#S3.F2)plots the number of times each node in the maze was visited by the RLVR and distillation policies \(in11K generations per maze\), and clearly shows that RL\-training results in a much less exploratory policy, i\.e\., fewer states are accessible at test time\. We empirically compute the probability of visiting a node using11K generations for5050mazes, and compute its*entropy*to find out how spread out each distribution is\. We find that forQwen3\-8B, averageHDistil=2\.3380H\_\{\\text\{Distil\}\}=2\.3380\(95%​CI​\[2\.1815,2\.4899\]95\\%\\text\{ CI \}\[2\.1815,2\.4899\]\)444Bootstrapped95%95\\%confidence intervals\.andHRLVR=1\.8138H\_\{\\text\{RLVR\}\}=1\.8138\(95%​CI​\[1\.6477,1\.9755\]95\\%\\text\{ CI \}\[1\.6477,1\.9755\]\)\.Exploration is significantly reduced in the RL\-trained modelat test time \(Δ​H=−0\.5242\\Delta H=\-0\.5242,95%​CI​\[−0\.7005,−0\.3435\]95\\%\\text\{ CI \}\[\-0\.7005,\-0\.3435\],p<0\.0001p<0\.0001\)\. Results withSmolLM3\-3Bare similar, and we have:HDistil=1\.9288H\_\{\\text\{Distil\}\}=1\.9288\(95%​CI​\[1\.7938,2\.0632\]95\\%\\text\{ CI \}\[1\.7938,2\.0632\]\),HRLVR=1\.6092H\_\{\\text\{RLVR\}\}=1\.6092\(95%​CI​\[1\.4344,1\.7756\]95\\%\\text\{ CI \}\[1\.4344,1\.7756\]\), andΔ​H=−0\.3196\\Delta H=\-0\.3196\(95%​CI​\[−0\.4252,−0\.2085\]95\\%\\text\{ CI \}\[\-0\.4252,\-0\.2085\],p<0\.0001p<0\.0001\)\.

### 4\.2Trajectory Preference in RLVR

Table 1:Effect of post\-training on LLM branching\.The table shows the change inΔ​CPE\\Delta\\textrm\{CPE\}\(Δ​CPERLVR−Δ​CPEDistil\\Delta\\textrm\{CPE\}\_\{\\text\{RLVR\}\}\-\\Delta\\textrm\{CPE\}\_\{\\text\{Distil\}\}\) between distilled and RLVR models, and the percentage of samples whereΔ​CPE\\Delta\\textrm\{CPE\}reduced in the RL\-trained model compared to the distilled one\. There is a significant drop inΔ​CPE\\Delta\\textrm\{CPE\}\(p<0\.0001p<0\.0001\) for almost all samples, suggesting that*RL\-trained models have significantly stronger trajectory preferences at branch points\.*95%95\\%CIs with paired bootstrap tests are reported\.BODHI\-Trees allow for controlled tests to investigate if there are differences in exploration patterns between different policies, and in particular, we can check whether models trained with Long\-CoTdistillationhave different trajectory preferences compared toRLVR\-trained models\. To this end, we computeΔCPE\(s\(1\)\\Delta\\textrm\{CPE\}\(s^\{\(1\)\},s\(2\);a\)s^\{\(2\)\};a\)\(Eq\.[4](https://arxiv.org/html/2608.02867#S3.E4)\) for1010K tuples fromBODHI\-Trees\. Our experiment \(see Table[1](https://arxiv.org/html/2608.02867#S4.T1)\) shows that there is asignificant dropinΔ​CPE\\Delta\\textrm\{CPE\}for RL\-trained models, i\.e\., they have lower branching entropy, and this effect isobserved in nearly every sample\. This indicates that RL\-trained models have stronger preferences when choosing between alternate completions at branch points, which reinforces test\-time compute scaling results\. The distribution is plotted in Figure[9](https://arxiv.org/html/2608.02867#A3.F9)in Appendix[C](https://arxiv.org/html/2608.02867#A3)\.

### 4\.3Exploring Invalid Continuations

![Refer to caption](https://arxiv.org/html/2608.02867v1/x2.png)Figure 3:RL\-trained models successfully learn the semantics of mazes\.We plot the entropy of the models at hall nodes \(degree 2\) to see if different training modalities produce models which correctly model the lack of ambiguity at halls\.A part of the observed entropy collapse could be explained by the model assigning less probability mass to invalid continuations\. To investigate this possibility, we turn to the*maze*task\. We choose55K“hall”nodes \(nodes with degree 2\) along maze solution paths, and since there is no ambiguity in the choice of next\-token, a semantically faithful policy should have lower entropy at these nodes\. We find that \(see Figure[3](https://arxiv.org/html/2608.02867#S4.F3)\) theRL\-trained models demonstrate statistically significant entropy collapse in halls, suggesting that the RLVR policy better delineates between different verifier\-equivalence classes\. A related notion for mathematical reasoning is “local trace coherence”, i\.e\., the ability of a model to avoid logical errors within sub\-segments of traces, and there is preliminary evidence suggesting that RLVR\-trained models perform better in this regard\(Samineni et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib29)\)\.

### 4\.4Is Entropy Collapse Stylistic?

![Refer to caption](https://arxiv.org/html/2608.02867v1/x3.png)Figure 4:Is the entropy collapse just syntactic?We observe that RLVR\-trained models usually have lower entropy \(CPE\) both when choosing between two syntactic variants of a mathematical statement and two semantically distinct continuations\. The entropy collapse in choosing between continuations with different semantics is stronger\. The*dashed*lines show the means of the distributions\.To analyze whether the entropy collapse in the RLVR policy arises due to the model developing stronger preferences for trivial mathematical representational choices \(such as how variables are named or the order of commutative operations\) alone, or also encompasses lack of semantic diversity such as the choice of themathematical approachto use, we select a branching nodeα\\alpha\(with at least 2 children\), and its two childrenβ\\betaandγ\\gammafrom aBODHI\-Tree\. We pick an anchor traceAAfrom the nodeα\\alpha, two continuationsa,ba,bfrom childβ\\betathat are semantically similar but have different syntax, and a continuationggfrom childγ\\gammawhich has different semantics froma,ba,b\. Then, as in Eq\.[4](https://arxiv.org/html/2608.02867#S3.E4), we can computeΔ​CPEmodelSyntax:=Δ​CPEmodel​\(a,b;A\)\\Delta\\textrm\{CPE\}\_\{\\texttt\{model\}\}^\{\\textrm\{Syntax\}\}:=\\Delta\\textrm\{CPE\}\_\{\\texttt\{model\}\}\(a,b;A\)andΔ​CPEmodelSemantics:=Δ​CPEmodel​\(a,g;A\)\\Delta\\textrm\{CPE\}\_\{\\texttt\{model\}\}^\{\\textrm\{Semantics\}\}:=\\Delta\\textrm\{CPE\}\_\{\\texttt\{model\}\}\(a,g;A\)\.

Since we are interested in the difference, we plot the distribution of the differences \(see Figure[4](https://arxiv.org/html/2608.02867#S4.F4)\), i\.e\.,Δ​CPERLVR∗−Δ​CPEDistil∗\\Delta\\text\{CPE\}\_\{\\texttt\{RLVR\}\}^\{\*\}\-\\Delta\\text\{CPE\}\_\{\\texttt\{Distil\}\}^\{\*\}, for2\.52\.5K sampled nodes fromBODHI\-Trees\. We observe a drop inΔ​CPESyntax\\Delta\\textrm\{CPE\}^\{\\textrm\{Syntax\}\}for all RL\-trained models compared to their distilled counterparts except forQwen3\.5\-9B, where the drop is not statistically significant\. However,all RL\-trained models in the study have a statistically significant drop inΔ​CPESemantics\\Delta\\textrm\{CPE\}^\{\\textrm\{Semantics\}\}compared to their distilled counterparts\. The difference between them is statistically significant \(p<0\.02p<0\.02except forQwen3\-8B,p=0\.11p=0\.11\)\. This suggests that the policy distribution ismore collapsed when it comes to making choices between semantically distinct continuations\. Thus, the policy collapse cannot be explained byjust syntactic/stylisticartifacts, and*RL\-trained models have a lower propensity for semantically meaningful exploration*\. This is important because this underlines the fact that RLVR doesn’t just eliminate spurious variations but also legitimate exploration\.

## 5Discussion

### 5\.1Constraints, Diversity and RLVR

The evidence presented in this paper thus far suggests that RLVR restricts the space of continuations accessible to the model \(§[4\.1](https://arxiv.org/html/2608.02867#S4.SS1)\), with reduced probability of branching \(§[4\.2](https://arxiv.org/html/2608.02867#S4.SS2)\) and invalid continuations \(§[4\.3](https://arxiv.org/html/2608.02867#S4.SS3)\)\. More importantly, the observed reduction in diversity cannot be attributed to calcification of stylistic choices alone, as there is a greater calcification in the choice of alternate semantically distinct continuations \(§[4\.4](https://arxiv.org/html/2608.02867#S4.SS4)\)\. This leads us to ask:*does RLVR achieve better sample efficiency \(pass​@​1\\textrm\{pass\}@1performance\) by learning environmental constraints, but it does so at the cost of trajectory diversity?*

To establish a more compelling connection between these, we design the following interventions on the decoding strategy in the mazes case:*\(i\)*“Legal Only” where invalid next tokens \(e\.g\., moves that result in wall collisions\) are masked out, and*\(ii\)*“Avoid Short Path” where the model is steered away from the shortest path\. TheLegal Onlyintervention sets the probability of invalid continuations to zero, and thus if a particular policy already had very low likelihood of selecting these continuations, it will be affected less\. Similarly, theAvoid Short Pathintervention sets the probability of the optimal continuation to zero \(another verifier\-equivalent continuation is guaranteed to exist\), and thus should have a larger effect on a policy which has stronger trajectory preferences\. Table[2](https://arxiv.org/html/2608.02867#S5.T2)presents our results with the distilled and RL\-trained variants ofQwen3\-8B\(T=1\.0\) with500500mazes\.

Table 2:RLVR restricts accessible states\.![Refer to caption](https://arxiv.org/html/2608.02867v1/x4.png)Figure 5:Environmental constraint\-adherence improves during RL\-training\.We plot the baseline performance \(*dashed line*\) of different RL\-training checkpoints ofQwen3\-8B, and the ratio of performance with theLegal Onlydecoding strategy to baseline\.As predicted by the hypothesis, we see that theLegal Onlyintervention has only a∼3×\\sim 3\\timesimprovement on the RL\-trained model, which is dwarfed by the∼8×\\sim 8\\timesimprovement on the distilled model\. The distilled model with theLegal Onlyconstraint shows a∼60%\\sim 60\\%improvement over the RL\-trained model, illustrating that a policy which has a flatter distribution amongst verifier\-equivalent continuations is more performant\. Figure[5](https://arxiv.org/html/2608.02867#S5.F5)plots the ratio of the performance with theLegal Onlyintervention and the baseline through the course of RL training\. As this measures the magnitude of the effect theLegal Onlyintervention has, it shows that the policy continually evolves to assign lower probabilities to invalid continuations as training progresses\. TheAvoid Short Pathintervention reduces performance by∼65%\\sim 65\\%for the RL\-trained model but only by∼50%\\sim 50\\%for the distilled model, showing that RL\-trained models have ossified preferences, i\.e\., other valid continuations are less likely to be explored\. The RL policy deprioritizes invalid moves to achieve a higher baseline \(13%13\\%\), but suffers significantly owing to a lack of trajectory diversity\.

### 5\.2Backtracking

Table 3:RLVR models have improved ability to recover from dead\-ends\.TheUnchangedcolumn lists the proportion of samples for which the distraction intervention did not change the final answer, and theΔ\\Deltacolumn shows the change between the two policies\.During reasoning, a model may reach a state where further progress is impossible, and a sound reasoning model must be able to backtrack out of these “dead\-ends” to a valid state and explore alternate trajectories\. To test this in mazes, as in §[5\.1](https://arxiv.org/html/2608.02867#S5.SS1), we design the “No Reversal” intervention, where two consecutive moves that are opposites of each other \(e\.g\.,<up\>followed by<down\>\) are masked out\. Our results show that \(see Table[2](https://arxiv.org/html/2608.02867#S5.T2)\) theNo Reversalintervention does not have a statistically significant effect on either variant\. However, this does not imply that the models cannot backtrack, as it could also be explained by the fact that the policies do not reach dead\-ends in the first place\. To investigate this further, we resort to*prefix steering*to evaluate policies, and ask:*what trajectories are available to the model at dead\-ends?*

For the maze case, we sampled2\.52\.5K dead\-end states—defined as states where the goal is only reachable if the last move is reversed—and measured the probability of the backtracking move \(pbp\_\{b\}\)\. Our results indicate that RLVR training significantly enhances backtracking capabilities compared to distillation\. ForSmolLM3\-3B, the average backtracking probability increases from𝔼​\[pb\]=0\.0647±0\.0008\\mathbb\{E\}\[p\_\{b\}\]=0\.0647\\pm 0\.0008\(Distil\) to0\.1687±0\.00350\.1687\\pm 0\.0035\(RLVR\) \(Δ=0\.1040,95%​CI​\[0\.0978,0\.1101\],p<0\.0001\\Delta=0\.1040,\\,95\\%\\text\{ CI \}\[0\.0978,0\.1101\],\\,p~<~0\.0001\)\.Qwen3\-8Bdemonstrates a similar improvement, rising from0\.0068±0\.00100\.0068\\pm 0\.0010to0\.1537±0\.00620\.1537\\pm 0\.0062\(Δ=0\.1469,95%​CI​\[0\.1348,0\.1589\],p<0\.0001\\Delta=0\.1469,\\,95\\%\\text\{ CI \}\[0\.1348,0\.1589\],\\,p~<~0\.0001\)\. Furthermore, RLVR reduces absolute failures \(pb=0p\_\{b\}=0\), with zero\-probability instances dropping from2422→18242422\\rightarrow 1824forQwen\-3\-8Band476→357476\\rightarrow 357forSmolLM\-3\-3B\. The distribution is plotted in Figure[10](https://arxiv.org/html/2608.02867#A3.F10)in Appendix[C](https://arxiv.org/html/2608.02867#A3)\.

Analyzing dead\-ends in mathematical reasoning is less straightforward, because unlike mazes, the valid action space is unconstrained\. To simulate this, we utilize a partial mathematical reasoning trajectory \(a non\-leaf node from aBODHI\-Tree\) and append a “distractor”—a spurious reasoning step sampled from a different tree—intended to derail generation \(see Figure[8](https://arxiv.org/html/2608.02867#A2.F8)\)\. We then evaluate whether the model can still arrive at the correct final answer, which measures the model’s ability to recover from missteps in reasoning\. As shown in Table[3](https://arxiv.org/html/2608.02867#S5.T3), RLVR\-trained models exhibit a significantly enhanced capability to backtrack compared to their distilled counterparts\. Since trajectories that do not involve backtracking are verifier\-invalid, these results also suggest that the RLVR policy better delineates between valid and invalid continuations\.

## 6Conclusions

We investigated the nature of the policy shift arising from RLVR training through controlled experiments using maze traversal and our mathematical exploration trees \(BODHI\-Trees\)\. We establish that RL\-trained models have stronger trajectory preferences at branch points, and that this is not merely an artifact of stylistic or syntactic preferences\. RLVR policies show a significant collapse in candidate preference entropy, suggesting that they prune semantically distinct, verifier\-equivalent trajectories\. While the policy concentration restricts trajectory diversity, it also enhances the model’s adherence to environmental constraints and ability to backtrack\. The improved trace validity is likely responsible for RLVR’s success, but it comes at the cost of genuine trajectory diversity\. Future work must address this limitation to push the boundaries of reasoning capabilities further\.

## Limitations

Mathematical Reasoning TreesTheBODHI\-Treeconstruction uses an LLM\-as\-a\-judge framework withGPT\-oss\-120bto gauge semantic equivalence, which might introduce some errors\. We performed spot checks and inter\-annotator agreement tests with more capable frontier LLMs such asdeepseek\-v4\-proand OpenAI’sgpt\-5\.6\-luna, finding near\-perfect agreement \(see Appendix[A](https://arxiv.org/html/2608.02867#A1)\); however, some errors may remain\. We could not perform rigorous human evaluations owing to resource constraints\.

Domain SpecificityThe presented empirical evidence is limited to deterministic environments \(mazes\) and mathematical reasoning \(AIME\)\. Since we study exploration, this involves sampling a large number of continuations from frontier LLMs which is prohibitively expensive for some domains \(USD 3000 for collecting AIME traces\)\. Future work that extends this to domains such as code generation would be compelling\.

## Ethics Statement

In keeping with ACL ethical guidelines, all scientific artifacts generated for this study—including code, prompts, data, and raw model outputs—are made freely available as open source under the MIT license\. Only public datasets available on the[Huggingface](https://huggingface.co/)platform were used in the study\. AI assistants were not used in ideation, coding, or writing involved in this work, and their usage was limited to copyediting tasks \(e\.g\., checking spelling, grammar, tone\)\.*We do not foresee any potential ethical, societal, or environmental risks from this work\.*

## References

- Anschel et al\. \(2025\)Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, and Gerard Medioni\. 2025\.[Group\-aware reinforcement learning for output diversity in large language models](https://doi.org/10.18653/v1/2025.emnlp-main.1649)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 32394–32415, Suzhou, China\. Association for Computational Linguistics\.
- Bakouch et al\. \(2025\)Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, and 4 others\. 2025\.SmolLM3: smol, multilingual, long\-context reasoner\.[https://huggingface\.co/blog/smollm3](https://huggingface.co/blog/smollm3)\.
- Cai et al\. \(2026\)Pengxiang Cai, Tianchen Fang, Xiaohan Li, Qingyuan Zeng, Guocong Li, and Jintai Chen\. 2026\.[Curriculum reinforcement learning can incentivize reasoning capacity in llms beyond the base model](https://arxiv.org/abs/2606.22317)\.*Preprint*, arXiv:2606\.22317\.
- Chen et al\. \(2026a\)Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, and Abolfazl Razi\. 2026a\.[DRA\-GRPO: Your GRPO needs to know diverse reasoning paths for mathematical reasoning](https://doi.org/10.18653/v1/2026.findings-acl.685)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 13995–14019, San Diego, California, United States\. Association for Computational Linguistics\.
- Chen et al\. \(2026b\)Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Xin Zhao, and Guang Shi\. 2026b\.[Pass@k training for adaptively balancing exploration and exploitation of large reasoning models](https://openreview.net/forum?id=eslxxopXTF)\.
- Cheng et al\. \(2026\)Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei\. 2026\.[Reasoning with exploration: An entropy perspective](https://doi.org/10.1609/aaai.v40i36.40290)\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 30377–30385\.
- DeepSeek\-AI et al\. \(2026\)DeepSeek\-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, and 300 others\. 2026\.[Deepseek\-v4: Towards highly efficient million\-token context intelligence](https://arxiv.org/abs/2606.19348)\.*Preprint*, arXiv:2606\.19348\.
- Dettmers et al\. \(2023\)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\. 2023\.[Qlora: Efficient finetuning of quantized llms](https://doi.org/10.52202/075280-0441)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 10088–10115\. Curran Associates, Inc\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others\. 2025\.[DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning](https://doi.org/10.1038/s41586-025-09422-z)\.*Nature*, 645\(8081\):633–638\.
- Hao et al\. \(2026\)Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen\. 2026\.[Rethinking entropy interventions in RLVR: An entropy change perspective](https://doi.org/10.18653/v1/2026.acl-long.1436)\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 31105–31133, San Diego, California, United States\. Association for Computational Linguistics\.
- He et al\. \(2026\)Xu He, Jialiang Guo, Fucheng Xiong, Haodong Zhao, Xingyang Li, Ke Zeng, and Xunliang Cai\. 2026\.[VANE: Guiding high\-value exploration in RLVR via outcome\-process novelty shaping](https://doi.org/10.18653/v1/2026.findings-acl.1434)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 28721–28739, San Diego, California, United States\. Association for Computational Linguistics\.
- Hu et al\. \(2025\)Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung\-Yeung Shum\. 2025\.[Open\-reasoner\-zero: An open source approach to scaling up reinforcement learning on the base model](https://openreview.net/forum?id=NFM8F5cV0V)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Hu et al\. \(2026a\)Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See\-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, and Bryan Hooi\. 2026a\.[Rewarding the rare: Uniqueness\-aware RL for creative problem solving in LLMs](https://doi.org/10.18653/v1/2026.findings-acl.1982)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 39765–39790, San Diego, California, United States\. Association for Computational Linguistics\.
- Hu et al\. \(2026b\)Zican Hu, Shilin Zhang, Yafu Li, Jianhao Yan, Xuyang Hu, Leyang Cui, Xiaoye Qu, Chunlin Chen, Yu Cheng, and Zhi Wang\. 2026b\.[Diversity\-incentivized exploration for versatile reasoning](https://openreview.net/forum?id=9G7AbBrd27)\.In*The Fourteenth International Conference on Learning Representations*\.
- Huang et al\. \(2026\)Fanding Huang, Guanbo Huang, Xiao Fan, Yi He, Xiao Liang, Xiao Chen, Qinting Jiang, Faisal Nadeem Khan, Jingyan Jiang, and Zhi Wang\. 2026\.[Semantic\-space exploration and exploitation in RLVR for LLM reasoning](https://doi.org/10.18653/v1/2026.findings-acl.1915)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 38402–38449, San Diego, California, United States\. Association for Computational Linguistics\.
- Ivanitskiy et al\. \(2023\)Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F\. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung\. 2023\.[A configurable library for generating and manipulating maze datasets](https://arxiv.org/abs/2309.10498)\.*Preprint*, arXiv:2309\.10498\.
- Jang et al\. \(2026\)Jaeeun Jang, Hansle Lee, and Sangmin Kim\. 2026\.[A few bad apples spoil the bunch: Preventing global entropy collapse driven by a small set of tokens in LLM reasoning](https://doi.org/10.18653/v1/2026.findings-acl.641)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 13134–13154, San Diego, California, United States\. Association for Computational Linguistics\.
- Jiang et al\. \(2025\)Gangwei Jiang, Yahui Liu, Zhaoyi Li, Wei Bi, Fuzheng Zhang, Linqi Song, Ying Wei, and Defu Lian\. 2025\.[What makes a good reasoning chain? uncovering structural patterns in long chain\-of\-thought reasoning](https://doi.org/10.18653/v1/2025.emnlp-main.329)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 6490–6514, Suzhou, China\. Association for Computational Linguistics\.
- Jin et al\. \(2026\)Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong\. 2026\.[Revisiting entropy in reinforcement learning for large reasoning models](https://doi.org/10.18653/v1/2026.findings-acl.1266)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 25300–25322, San Diego, California, United States\. Association for Computational Linguistics\.
- Li and Li \(2026\)Wendi Li and Sharon Li\. 2026\.[LAD: Learning advantage distribution for reasoning](https://doi.org/10.18653/v1/2026.findings-acl.702)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 14327–14348, San Diego, California, United States\. Association for Computational Linguistics\.
- Li et al\. \(2025\)Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang\. 2025\.[Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree\-based modeling](https://arxiv.org/abs/2508.17445)\.*Preprint*, arXiv:2508\.17445\.
- Liu et al\. \(2025\)Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\. 2025\.[Understanding r1\-zero\-like training: A critical perspective](https://arxiv.org/abs/2503.20783)\.*Preprint*, arXiv:2503\.20783\.
- Mishra et al\. \(2026\)Kshitij Mishra, Nils Lukas, and Salem Lahlou\. 2026\.[SD\-e2: Semantic exploration for reasoning under token budgets](https://doi.org/10.18653/v1/2026.findings-eacl.323)\.In*Findings of the Association for Computational Linguistics: EACL 2026*, pages 6144–6157, Rabat, Morocco\. Association for Computational Linguistics\.
- OpenAI et al\. \(2025\)OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K\. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others\. 2025\.[gpt\-oss\-120b & gpt\-oss\-20b model card](https://arxiv.org/abs/2508.10925)\.*Preprint*, arXiv:2508\.10925\.
- OpenAI \(2025\)OpenAI\. 2025\.[OpenAI o3 and o4\-mini system card](https://openai.com/index/o3-o4-mini-system-card/)\.
- OpenAI \(2026\)OpenAI\. 2026\.[GPT\-5\.6: Frontier intelligence that scales with your ambition](https://openai.com/index/gpt-5-6/)\.
- Qwen et al\. \(2024\)Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others\. 2024\.[Qwen2\.5 technical report](https://arxiv.org/abs/2412.15115)\.*ArXiv preprint*, abs/2412\.15115\.
- Saha et al\. \(2026\)Soumadeep Saha, Akshay Chaturvedi, Saptarshi Saha, Utpal Garain, and Nicholas Asher\. 2026\.[KisMATH: Do LLMs have Knowledge of Implicit Structures in mathematical reasoning?](https://doi.org/10.1162/TACL.a.729)*Transactions of the Association for Computational Linguistics*, 14:1308–1328\.
- Samineni et al\. \(2025\)Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, and Subbarao Kambhampati\. 2025\.[Local coherence or global validity? investigating rlvr traces in math domains](https://arxiv.org/abs/2510.18176)\.*Preprint*, arXiv:2510\.18176\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*Preprint*, arXiv:2402\.03300\.
- Team et al\. \(2025\)Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others\. 2025\.[Gemma 3 technical report](https://arxiv.org/abs/2503.19786)\.*Preprint*, arXiv:2503\.19786\.
- Team \(2026\)Qwen Team\. 2026\.[Qwen3\.5: Accelerating productivity with native multimodal agents](https://qwen.ai/blog?id=qwen3.5)\.
- von Werra et al\. \(2020\)Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec\. 2020\.[TRL: Transformers Reinforcement Learning](https://github.com/huggingface/trl)\.
- Wang et al\. \(2025\)Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong\-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin\. 2025\.[Beyond the 80/20 rule: High\-entropy minority tokens drive effective reinforcement learning for LLM reasoning](https://openreview.net/forum?id=yfcpdY4gMP)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V\. Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\. 2023\.[Self\-consistency improves chain of thought reasoning in language models](https://openreview.net/forum?id=1PL1NIMMrw)\.In*The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023*\. OpenReview\.net\.
- Wen et al\. \(2026\)Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang\. 2026\.[Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs](https://openreview.net/forum?id=jGbRWwIidy)\.In*The Fourteenth International Conference on Learning Representations*\.
- Yang et al\. \(2025a\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025a\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Yang et al\. \(2025b\)Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang\. 2025b\.[Treerpo: Tree relative policy optimization](https://arxiv.org/abs/2506.05183)\.*Preprint*, arXiv:2506\.05183\.
- Yao et al\. \(2023\)Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L\. Griffiths, Yuan Cao, and Karthik R Narasimhan\. 2023\.[Tree of thoughts: Deliberate problem solving with large language models](https://openreview.net/forum?id=5Xc1ecxO1h)\.In*Thirty\-seventh Conference on Neural Information Processing Systems*\.
- Yu et al\. \(2026\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others\. 2026\.[DAPO: An open\-source LLM reinforcement learning system at scale](https://openreview.net/forum?id=2a36EMSSTp)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Yuan et al\. \(2026\)Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, and Bo An\. 2026\.[Understanding diversity collapse in rlvr via the lens of overtraining](https://arxiv.org/abs/2606.15455)\.*Preprint*, arXiv:2606\.15455\.
- Yue et al\. \(2025\)Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang\. 2025\.[Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?](https://openreview.net/forum?id=upehLVgq1b)In*2nd AI for Math Workshop @ ICML 2025*\.
- Zhao et al\. \(2026\)Ziqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang, Zhiwei Xu, Xuri Ge, Zhumin Chen, Xinyu Ma, Daiting Shi, Shuaiqiang Wang, Dawei Yin, and Xin Xin\. 2026\.[Reinforced efficient reasoning via semantically diverse exploration](https://doi.org/10.18653/v1/2026.acl-long.2216)\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 47994–48007, San Diego, California, United States\. Association for Computational Linguistics\.
- Zheng et al\. \(2025\)Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma\. 2025\.[First return, entropy\-eliciting explore](https://arxiv.org/abs/2507.07017)\.*Preprint*, arXiv:2507\.07017\.

## Appendix ABODHI\-TreeConstruction

To constructBODHI\-Trees, we start with304304questions from[AIME](https://huggingface.co/datasets/gneubig/aime-1983-2024)as inSaha et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib28)\)and sample responses from a diverse set of LLMs, namely OpenAIo3\-2025\-04\-16, OpenAIo4\-mini\-2025\-04\-16\(OpenAI,[2025](https://arxiv.org/html/2608.02867#bib.bib25)\), DeepSeekr1\-0528\(Guo et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib9)\), and Qwen’sqwen3\-235b\-a22b\-07\-25\(Yang et al\.,[2025a](https://arxiv.org/html/2608.02867#bib.bib37)\), withT=1\.0T=1\.0and reasoning effort set to the highest available setting\. The number of samples was decided by the API budget, which in this case totaledUSD 3000\. All responses with incorrect answers were removed and this resulted in∼20\\sim 20K responses to235235questions, with an average of∼85\\sim 85responses per question\. These were then split into reasoning segments using OpenAIo4\-mini\-2025\-04\-16\.

Given a questionQQ, a set of “correct” reasoning tracesA\(1\),A\(2\),…,A\(n\)A^\{\(1\)\},A^\{\(2\)\},\\ldots,A^\{\(n\)\}, and their corresponding segmentationsaj\(i\)a^\{\(i\)\}\_\{j\}, such thatA\(i\)=concat​\(\[a1\(i\),a2\(i\),…,ani\(i\)\]\)A^\{\(i\)\}=\\textrm\{concat\}\(\[a^\{\(i\)\}\_\{1\},a^\{\(i\)\}\_\{2\},\\ldots,a^\{\(i\)\}\_\{n\_\{i\}\}\]\), we employ Algorithm[1](https://arxiv.org/html/2608.02867#alg1)to createBODHI\-Trees\.

Algorithm 1BODHI\-TreeConstruction1:Given a question

QQ\.

2:Given

A\(i\)=concat​\(\[a1\(i\),a2\(i\),…,ani\(i\)\]\)A^\{\(i\)\}~=~\\textrm\{concat\}\(\[a^\{\(i\)\}\_\{1\},a^\{\(i\)\}\_\{2\},\\ldots,a^\{\(i\)\}\_\{n\_\{i\}\}\]\)for

i∈\{1,…​n\}i\\in\\\{1,\\ldots n\\\}\.

3:Given

M​\(x,y\)M\(x,y\)an LLM\-based matching function,trueif

yyis equivalent to

xx, or equivalent to a part of

xx\(subsumed in

xx\)\.

4:Initialize

TTwith a single root node

vrootv\_\{\\texttt\{root\}\}
5:for

i∈1​…​ni\\in 1\\ldots ndo

6:

z​\[i\]←vrootz\[i\]\\leftarrow v\_\{\\texttt\{root\}\}
7:endfor

8:

9:while

∃A\(i\)\\exists A^\{\(i\)\}with remaining segmentsdo

10:\# Try to consume a segment into a node\.

11:forEach

A\(i\)A^\{\(i\)\}with remaining segmentsdo

12:

shead←dequeue​\(A\(i\)\);v←z​\[i\]s\_\{\\textrm\{head\}\}\\leftarrow\\textrm\{dequeue\}\(A^\{\(i\)\}\);v\\leftarrow z\[i\]
13:if

M​\(v,shead\)M\(v,s\_\{\\textrm\{head\}\}\)istruethen

14:Consume

sheads\_\{\\textrm\{head\}\}in

vv
15:else

16:

requeue​\(shead,A\(i\)\)\\textrm\{requeue\}\(s\_\{\\textrm\{head\}\},A^\{\(i\)\}\)
17:endif

18:endfor

19:

20:\# Move trace to child node or fork\.

21:forEach

A\(i\)A^\{\(i\)\}with remaining segmentsdo

22:

shead←dequeue​\(A\(i\)\);v←z​\[i\]s\_\{\\textrm\{head\}\}\\leftarrow\\textrm\{dequeue\}\(A^\{\(i\)\}\);v\\leftarrow z\[i\]
23:

matched←false\\texttt\{matched\}\\leftarrow\\texttt\{false\}
24:forEach child

vchildv\_\{\\textrm\{child\}\}of

vvin

TTdo

25:if

M​\(vchild,shead\)M\(v\_\{\\textrm\{child\}\},s\_\{\\textrm\{head\}\}\)istruethen

26:

z​\[i\]←vchildz\[i\]\\leftarrow v\_\{\\textrm\{child\}\}
27:Consume

sheads\_\{\\textrm\{head\}\}in

vchildv\_\{\\textrm\{child\}\}
28:

matched←true\\texttt\{matched\}\\leftarrow\\texttt\{true\}
29:break

30:endif

31:endfor

32:ifmatchedisfalsethen

33:Createnew node

vnewv\_\{\\text\{new\}\}
34:Add edge

v↝vnewv\\leadsto v\_\{\\text\{new\}\}
35:

z​\[i\]←vnewz\[i\]\\leftarrow v\_\{\\textrm\{new\}\}
36:Consume

sheads\_\{\\textrm\{head\}\}in

vnewv\_\{\\textrm\{new\}\}
37:endif

38:endfor

39:endwhile

40:return

TT

Algorithm[1](https://arxiv.org/html/2608.02867#alg1)starts with a root node, and each trace is assigned to this root node \(z​\[i\]←vrootz\[i\]\\leftarrow v\_\{\\texttt\{root\}\}\)\. At each iteration, for every trace we try to consume its first available segment into its assigned node, failing which we either try to match it with one of its children or spawn a new branch\. If the head of the trace matches any of the children, we consume it at the child, and assign the rest of the trace to the child\. If no such match is found, a new child is spawned, consuming the head segment, and the rest of the trace is assigned here\. The algorithm terminates when no trace has any unassigned segments\. The resulting tree \(BODHI\-Tree\) has the property that the lowest common ancestor of two traces represents their longest shared equivalent reasoning prefix\.

The matching functionM​\(x,y\)M\(x,y\)is instantiated with OpenAIGPT\-oss\-120b\(OpenAI et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib24)\)and checks ifxxis mathematically equivalent toyyoryyis mathematically equivalent to a part ofxx\. This helps align traces with differing levels of granularity\. In practice, when consuming a segment into a node, we pick a representative of the node to match against the segment\. The LLM is sampled atT=1T=1,topp=0\.99\\texttt\{top\}\_\{p\}=0\.99,reasoning\_effort = medium, using 4\-shot prompts\. A total of470~470K LLM calls are made for the construction of theBODHI\-Trees \(∼2\\sim 2K calls perBODHI\-Tree\)\. We also measured the inter\-annotator agreement betweenGPT\-oss\-120band more capable frontier LLMs, namelydeepseek\-v4\-pro\(DeepSeek\-AI et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib7)\)andgpt\-5\.6\-luna\(OpenAI,[2026](https://arxiv.org/html/2608.02867#bib.bib26)\), and the Cohen’sκ\\kappascores are0\.8150\.815and0\.8370\.837, respectively, suggesting that there is near\-perfect agreement\. The prompt is given in Figure[6](https://arxiv.org/html/2608.02867#A1.F6), and some example trees are given in Figure[7](https://arxiv.org/html/2608.02867#A1.F7)\.

<system\> \# IdentityYou are an expert mathematician who gauges similarity of mathematical statements\. Help the user detect similarity in statements by answering yes/no\.\# Instructions\* ONLY answer with ’yes’ or ’no’\. Do not produce any extra text\.\* The user will provide two statements Statement A and Statement B\.\* Answer ’yes’ if and only if Statement B is completely equivalent to Statement A or Statement B is equivalent to a part of Statement A\.\* Answer ’no’ if the statements are not equivalent, if Statement A is contained in Statement B, or if they have different mathematical meaning\.\* Ignore superficial elements like names of variables, whitespace, synonyms, phrasing, etc\.\* The only important aspect is the mathematical meaning of the statements\. \# Examples…</system\> <user\> \# Statement A:The equation of the circle is given byx2\+y2=5x^\{2\}\+y^\{2\}=5\.\# Statement B:The circle’s equation isz2\+y2=5z^\{2\}\+y^\{2\}=5\.</user\> <assistant\> yes </assistant\>

Figure 6:System prompt to instantiate theM​\(x,y\)M\(x,y\)function in Algorithm[1](https://arxiv.org/html/2608.02867#alg1)\.![Refer to caption](https://arxiv.org/html/2608.02867v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.02867v1/x6.png)

Figure 7:Examples of extractedBODHI\-Trees\.
## Appendix BExperimental Details

### B\.1Training

Formath, we performed Long\-CoT distillation employing5050K randomly chosen samples from the[OpenThoughts\-114k\-math](https://huggingface.co/datasets/open-r1/OpenThoughts-114k-math)dataset\. We perform Q\-LoRA \(r=64,α=128r=64,\\alpha=128;Dettmers et al\. \([2023](https://arxiv.org/html/2608.02867#bib.bib8)\)\) fine\-tuning for11epoch \(lr=5​e−4=5e\-4\) withmax\_tokens=16=16K, batch size of6464, a cosine learning rate scheduler with warm\-up \(5%5\\%\), andAdamWoptimizer\. RL\-training was performed with theTRLlibrary\(von Werra et al\.,[2020](https://arxiv.org/html/2608.02867#bib.bib33)\)using theDAPO\-Math\-17kdataset\(Yu et al\.,[2026](https://arxiv.org/html/2608.02867#bib.bib40)\)for11K steps withlr=1​e−6=1e\-6, group size of88, batch size of4848, andmax\_tokens=4=4K\. Following suggestions byYu et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib40)\), we setϵlow=0\.2,ϵhigh=0\.28\\epsilon\_\{\\text\{low\}\}=0\.2,\\epsilon\_\{\\text\{high\}\}=0\.28, and following suggestions byLiu et al\. \([2025](https://arxiv.org/html/2608.02867#bib.bib22)\); Yu et al\. \([2026](https://arxiv.org/html/2608.02867#bib.bib40)\); Hu et al\. \([2025](https://arxiv.org/html/2608.02867#bib.bib12)\), we setβ=0\\beta=0\. We employ the Dr\. GRPO loss\(Liu et al\.,[2025](https://arxiv.org/html/2608.02867#bib.bib22)\)to avoid length\-based biases\. We use the[Math\-Verify](https://github.com/huggingface/Math-Verify)library to compute accuracy rewards, and use a format reward which ensures that<think\>tags are present and there is exactly one\\boxed\{\}containing the answer\. Length\-based rewards are not employed\.Qwen3\.5\-9B, likeDAPO\-Qwen2\.5\-32B, was trained with Zero\-RL, whereas forQwen3\-8Bandgemma3\-12b\-ptwe perform distillation followed by RLVR\. The chat templates for the distilled and RL models are matched, and are distributed alongside the model\.All training datasets are de\-duplicated against the probing set\.

Formazes, we performed SFT with oracle\-generated traces employing the[Maze Dataset](https://understanding-search.github.io/maze-dataset/maze_dataset.html)byIvanitskiy et al\. \([2023](https://arxiv.org/html/2608.02867#bib.bib16)\)\. We used3030K mazes with sizes5×5,7×7,9×95\\times 5,\\\>7\\times 7,\\\>9\\times 9and similar settings as described above \(exceptmax\_tokens=2=2K\)\. For RL\-training,1212K mazes were used with similar settings as described above \(except group size1616, andmax\_tokens=1\.5=1\.5K\)\. We used a custom verifier to check correctness, and a format reward to ensure that the predicted moves were parsable\. Five tokens were added to the tokenizer before the distillation stage: they are<left\>, <right\>, <up\>, <down\>\(for predicting a path\), and<\-\-\>\(to denote an edge while express the maze as an adjacency list\)\.

RL\-training was performed with4×44\\times 4H1009393GB GPUs, and all other experiments used1×41\\times 4H100s\.

### B\.2Evaluation

For results in §[4\.1](https://arxiv.org/html/2608.02867#S4.SS1), we used5050mazes and sampled11K generations from the models atT=1\.0T=1\.0andtop\_k=10=10\. For results in §[4\.2](https://arxiv.org/html/2608.02867#S4.SS2), §[4\.3](https://arxiv.org/html/2608.02867#S4.SS3), and §[4\.4](https://arxiv.org/html/2608.02867#S4.SS4), we directly employ the logits, and we truncate candidate continuations to2020tokens forCPEcalculation \(see Eq\.[4](https://arxiv.org/html/2608.02867#S3.E4)\)\. The results in §[5\.1](https://arxiv.org/html/2608.02867#S5.SS1)and §[5\.2](https://arxiv.org/html/2608.02867#S5.SS2)are sampled atT=1\.0T=1\.0\.Unless explicitly mentioned, all sampling parameters are at default values\.An example distractor for the experiment in §[5\.2](https://arxiv.org/html/2608.02867#S5.SS2)is given in Figure[8](https://arxiv.org/html/2608.02867#A2.F8)\.

<system\> You are a helpful AI assistant\. Assist the user with their math query\. The final answer must be in \\boxed\{\}\.</system\> <user\> Question: For non\-negative integersaaandbbwitha\+b≤6a\+b\\leq 6, letT​\(a,b\)=\(6a\)​\(6b\)​\(6a\+b\)T\(a,b\)=\\binom\{6\}\{a\}\\binom\{6\}\{b\}\\binom\{6\}\{a\+b\}\. LetSSdenote the sum of allT​\(a,b\)T\(a,b\), whereaaandbbare non\-negative integers witha\+b≤6a\+b\\leq 6\. Find the remainder whenSSis divided by10001000\.</user\> <assistant\> \# Added prefix, BODHI\-Tree segments\.LetS=∑a=06∑b=06−a\(6a\)​\(6b\)​\(6a\+b\)S=\\sum\_\{a=0\}^\{6\}\\sum\_\{b=0\}^\{6\-a\}\\binom\{6\}\{a\}\\binom\{6\}\{b\}\\binom\{6\}\{a\+b\}\.Rewrite withk=a\+bk=a\+b\.S=∑k=06\(6k\)​∑a=0k\(6a\)​\(6k−a\)S=\\sum\_\{k=0\}^\{6\}\\binom\{6\}\{k\}\\sum\_\{a=0\}^\{k\}\\binom\{6\}\{a\}\\binom\{6\}\{k\-a\}∑a=0k\(6a\)​\(6k−a\)\\sum\_\{a=0\}^\{k\}\\binom\{6\}\{a\}\\binom\{6\}\{k\-a\}is the coefficient ofxkx^\{k\}in\(1\+x\)6​\(1\+x\)6=\(1\+x\)12\(1\+x\)^\{6\}\(1\+x\)^\{6\}=\(1\+x\)^\{12\}, so it equals\(12k\)\\binom\{12\}\{k\}\.S=∑k=06\(6k\)​\(12k\)S=\\sum\_\{k=0\}^\{6\}\\binom\{6\}\{k\}\\binom\{12\}\{k\}Compute each term:k=0:1k=0:\\,1k=1:6⋅12=72k=1:\\,6\\cdot 12=72k=2:15⋅66=990k=2:\\,15\\cdot 66=990k=3:20⋅220=4400k=3:\\,20\\cdot 220=4400k=4:15⋅495=7425k=4:\\,15\\cdot 495=7425k=5:6⋅792=4752k=5:\\,6\\cdot 792=4752Solving givesc=4±16\+2406=4±166c=\\frac\{4\\pm\\sqrt\{16\+240\}\}\{6\}=\\frac\{4\\pm 16\}\{6\}, soc=103c=\\tfrac\{10\}\{3\}orc=−2c=\-2\.\# Model continues\.\.\.k=6:\(66\)​\(126\)=1⋅924=924​…k=6:\\,\\binom\{6\}\{6\}\\binom\{12\}\{6\}=1\\cdot 924=924\\ldots

Figure 8:Example distractor \(*red*\) for experiment in §[5\.2](https://arxiv.org/html/2608.02867#S5.SS2)\.

## Appendix CAdditional Results

![Refer to caption](https://arxiv.org/html/2608.02867v1/x7.png)Figure 9:Effect of post\-training on LLM branching\.\(*top*\) We plot the distribution ofΔ​CPE\\Delta\\textrm\{CPE\}as described in §[4\.2](https://arxiv.org/html/2608.02867#S4.SS2)for distilled and RLVR\-trained models\. There is a significant leftward shift in the distribution ofΔ​CPE\\Delta\\textrm\{CPE\}suggesting that RL\-trained models have enhanced preference at branch points\. The \(*bottom*\) row shows a zoomed\-in view of the left 20\-%ile tail of each distribution\.![Refer to caption](https://arxiv.org/html/2608.02867v1/x8.png)Figure 10:RL\-trained models are more likely to backtrack\.We plot the distribution of probability of backtracking \(pbp\_\{b\}\) as described in §[5\.2](https://arxiv.org/html/2608.02867#S5.SS2)\.