RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning

# RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
Source: [https://arxiv.org/html/2606.07006](https://arxiv.org/html/2606.07006)
Yongliang Miao1,\*,Fengyuan Liu1,\*,Wei Shi2 Yanguang Liu3,Fei Sun4,Na Zou2,Mengnan Du1,† 1The Chinese University of Hong Kong, Shenzhen2Shanghai Artificial Intelligence Laboratory 3New Jersey Institute of Technology4Institute of Computing Technology, CAS r130026108@gmail\.com, mengnandu@cuhk\.edu\.cn \*Equal contribution\.†Corresponding author

###### Abstract

Supervised fine\-tuning \(SFT\) is a prevailing method for adapting large language models to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior\. However, reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model’s own reasoning distribution\. We proposeRollout\-Adaptive Supervised Fine\-Tuning \(RASFT\), a policy\-aware SFT framework that calibrates expert supervision according to problem\-level solvability estimated from verified on\-policy rollouts\. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self\-generated trajectories when the model already exhibits reliable reasoning behavior\. To preserve useful reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift\. Experiments across multiple models on six mathematical reasoning benchmarks and two code reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods\. The code is available at[https://github\.com/zjd1sq/RASFT](https://github.com/zjd1sq/RASFT)\.

RASFT: Rollout\-Adaptive Supervised Fine\-Tuning for Reasoning

Yongliang Miao1,\*, Fengyuan Liu1,\*, Wei Shi2Yanguang Liu3,Fei Sun4,Na Zou2,Mengnan Du1,†1The Chinese University of Hong Kong, Shenzhen2Shanghai Artificial Intelligence Laboratory3New Jersey Institute of Technology4Institute of Computing Technology, CASr130026108@gmail\.com, mengnandu@cuhk\.edu\.cn\*Equal contribution\.†Corresponding author\.

## 1Introduction

Supervised fine\-tuning \(SFT\) has been widely used to adapt large language models \(LLMs\) to reasoning domainsOuyanget al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib41)\); Chunget al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib38)\); Mukherjeeet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib39)\); Yueet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib40)\)\. Standard SFT relies on offline expert demonstrations and implicitly assumes that expert trajectories are the targets to be imitatedOuyanget al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib41)\); Wanget al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib42)\); Zhouet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib43)\)\. However, reasoning is not simple imitation: one problem can often be solved through multiple valid paths, while an offline expert response presents only one possible reasoning trajectoryWeiet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib16)\); Zelikmanet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib37)\); Yuanet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib36)\)\. Meanwhile, pretrained LLMs may already possess rich potential reasoning distributionsZelikmanet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib37)\); Mukherjeeet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib39)\)\. If fine\-tuning overly follows a single expert trajectory, the model may fit the surface form of the demonstration, thereby weakening its original reasoning distribution and limiting the activation of its own reasoning abilityChuet al\.\([2025](https://arxiv.org/html/2606.07006#bib.bib20)\); Mukherjeeet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib39)\)\.

This limitation has motivated recent SFT variants to reconsider how expert demonstrations should be optimized\. DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\)weakens uniform token\-level imitation by rescaling the learning signal according to the model’s confidence\. ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\)further anchors this optimization to a reference model to improve stability and reduce distributional drift\. ProFiTLiuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib3)\)reduces unnecessary fitting by emphasizing high\-value supervision signals within the expert response\. These methods make SFT less blindly imitative at the token or objective level, but their adaptivity remains largely demonstration\-internal, not policy\-aware: the expert trajectory is still optimized as the default target, leaving the objective unable to distinguish when expert imitation provides necessary correction and when it becomes rigid path fitting for the current policy\.

This motivates a policy\-adaptive view of SFT in reasoning domains:expert demonstrations should serve as problem\-level guidance whose strength is calibrated by the current policy’s ability to solve each problem, rather than only by signals within the offline trajectory\.The challenge is therefore how to determine the role of expert supervision for each problem under the current policy\. When the model still struggles, expert trajectories should provide stronger corrective guidance; when the model already exhibits reliable reasoning behavior, rigid imitation of the same expert path becomes less necessary\. Therefore, effective SFT should move beyond static demonstration fitting and adjust expert supervision according to policy\-dependent problem difficulty, while avoiding unnecessary suppression of the model’s own reasoning distribution\.

To this end, we proposeRollout\-Adaptive Supervised Fine\-Tuning \(RASFT\), a policy\-aware SFT framework that adapts expert supervision using the current model’s rollout behavior\. For each problem, RASFT constructs a local candidate pool consisting of the offline expert trajectory and verified correct trajectories generated by the current model\. It estimates a problem\-level solvability score from the success rate of model\-generated rollouts, which reflects how well the current policy can solve the problem\. Based on this score, RASFT increases the influence of expert demonstrations on difficult problems and attenuates it when the model already exhibits reliable reasoning behavior\. Furthermore, to conservatively activate the model’s reasoning ability rather than overwrite it through excessive imitation, RASFT introduces an inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift\.

Empirically, RASFT consistently outperforms SFT\-style baselines across mathematical and code reasoning tasks\. It achieves a10\.9%10\.9\\%relative gain on Qwen2\.5\-Math\-1\.5B math reasoning \(25\.00→27\.7225\.00\\rightarrow 27\.72\) and up to26\.9%26\.9\\%on Llama\-3\.2\-3B code generation \(24\.93→31\.6324\.93\\rightarrow 31\.63\), suggesting that rollout\-adaptive supervision brings benefits beyond stronger demonstration fitting\. Compared with GRPOShaoet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib31)\), RASFT further improves average math accuracy by15\.9%15\.9\\%\(20\.25→23\.4720\.25\\rightarrow 23\.47\), highlighting the robustness of combining expert guidance with rollout\-based adaptation\.

Our key contributions are summarized as follows:

- •We identify a key limitation of existing SFT variants for reasoning: their adaptivity mainly operates within offline expert demonstrations, rather than calibrating expert supervision according to the current policy’s problem\-level ability\.
- •We propose RASFT, a rollout\-adaptive SFT framework that uses verified on\-policy rollouts to estimate problem solvability and dynamically balance expert guidance with self\-generated correct reasoning trajectories\.
- •Extensive experiments on mathematical and code reasoning tasks show that RASFT outperforms SFT\-style baselines across multiple models, with ablations and RL comparisons further confirming its effectiveness and robustness\.

## 2Methodology

![Refer to caption](https://arxiv.org/html/2606.07006v1/x1.png)

Figure 1:RASFT pipeline\.\(a\) For each prompt, policy modelπθ\\pi\_\{\\theta\}samples multiple rollouts, which are verified and combined with offline expert trajectory\. \(b\) Rollout\-based solvabilityζi\\zeta\_\{i\}, which adaptively calibrates expert and rollout trajectory weights\. \(c\) RASFT updates the policy modelπθ\\pi\_\{\\theta\}by optimizing candidate trajectories with a compound weight that combines normalized trajectory weights, an inverse policy ratio, and token probability\.### 2\.1Preliminaries

We consider supervised post\-training for reasoning tasks\. Let𝒟off=\{\(xi,yie\)\}i=1N\\mathcal\{D\}\_\{\\mathrm\{off\}\}=\\\{\(x\_\{i\},y\_\{i\}^\{e\}\)\\\}\_\{i=1\}^\{N\}denote an offline reasoning dataset, wherexix\_\{i\}is a problem prompt andyiey\_\{i\}^\{e\}is an expert\-written reasoning trajectory\. A language model parameterized byθ\\thetadefines an autoregressive policy

πθ​\(y∣x\)=∏t=1\|y\|πθ​\(yt∣x,y<t\)\.\\pi\_\{\\theta\}\(y\\mid x\)=\\prod\_\{t=1\}^\{\|y\|\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\.\(1\)Conventional supervised fine\-tuning optimizes the negative log\-likelihood of the expert response:

ℒSFT​\(θ\)=−𝔼\(x,ye\)∼𝒟off\[∑t=1\|ye\|log⁡πθ​\(yte∣x,y<te\)\]\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)\{=\}\-\\\!\\mathop\{\\mathbb\{E\}\}\\limits\_\{\(x,y^\{e\}\)\\sim\\mathcal\{D\}\_\{\\mathrm\{off\}\}\}\\\!\\\!\\left\[\\sum\_\{t=1\}^\{\|y^\{e\}\|\}\\\!\\log\\pi\_\{\\theta\}\(y\_\{t\}^\{e\}\{\\mid\}x,y\_\{<t\}^\{e\}\)\\\!\\right\]\.This objective treats each expert trajectory as the desired target behavior and applies uniform token\-level imitation along the given response\.

However, reasoning differs from ordinary sequence imitation: the expert trajectoryyey^\{e\}provides only one possible solution path, while the model generates according to its own policyπθ\(⋅∣x\)\\pi\_\{\\theta\}\(\\cdot\\mid x\)\. We aim to use expert responses as supervision with less excessive imitation of trajectories that are redundant, misaligned, or already well captured by the current policy \(Figure[1](https://arxiv.org/html/2606.07006#S2.F1)\)\.

### 2\.2RASFT Framework

Policy\-aware regularization\.RASFT optimizes expert and model\-generated reasoning trajectories using policy\-aware regularization and trajectory\-level adaptive weighting, moving away from treating offline expert responses as uniformly imitative targets\. Instead of framing post\-training through an explicit reinforcement learning lens with rigid reward or advantage formulations, we view expert trajectories and self\-generated successful behaviors as an aligned candidate pool where optimization scales dynamically according to policy\-relative properties\.

The core motivation is that large language models already possess substantial reasoning capacity from pretraining\. Thus, the objective of post\-training is not to force the mechanical memorization of static demonstrations, but to leverage expert knowledge to activate and refine the model’s latent policy distribution\. Letπref\\pi\_\{\\mathrm\{ref\}\}denote the frozen initial reference model, and letπθ\\pi\_\{\\theta\}denote the current trainable policy\. For an arbitrary target trajectoryyi,j=\(yi,j,1,…,yi,j,\|yi,j\|\)y\_\{i,j\}=\(y\_\{i,j,1\},\\ldots,y\_\{i,j,\|y\_\{i,j\}\|\}\)associated with promptxix\_\{i\}, wherejjindexes a candidate trajectory, the target\-token probability under the active policy is defined as:

pi,j,tθ=πθ​\(yi,j,t∣xi,yi,j,<t\)\.p\_\{i,j,t\}^\{\\theta\}=\\pi\_\{\\theta\}\(y\_\{i,j,t\}\\mid x\_\{i\},y\_\{i,j,<t\}\)\.
To regularize how aggressively a candidate trajectory updates the policy, RASFT defines a sequence\-level inverse importance sampling ratio:

ri,j​\(θ\)=\(πref​\(yi,j∣xi\)πθ​\(yi,j∣xi\)\)1\|yi,j\|=exp⁡\(1\|yi,j\|​∑t=1\|yi,j\|log⁡πref​\(yi,j,t∣xi,yi,j,<t\)πθ​\(yi,j,t∣xi,yi,j,<t\)\)\.\\begin\{split\}r\_\{i,j\}\(\\theta\)&\{=\}\\left\(\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,j\}\\mid x\_\{i\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,j\}\\mid x\_\{i\}\)\}\\right\)^\{\\frac\{1\}\{\|y\_\{i,j\}\|\}\}\\\\ &\{=\}\\exp\\\!\\\!\\left\(\\\!\\frac\{1\}\{\|y\_\{i,j\}\|\}\\\!\\\!\\sum\_\{t=1\}^\{\|y\_\{i,j\}\|\}\\\!\\log\\\!\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,j,t\}\{\\mid\}x\_\{i\},y\_\{i,j,<t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,j,t\}\{\\mid\}x\_\{i\},y\_\{i,j,<t\}\)\}\\\!\\\!\\right\)\.\\end\{split\}Here, the numerator reflects the reference model’s support for the candidate trajectory, ensuring that updates consistent with the pretrained prior are preserved to prevent distributional drift\. Conversely, asπθ​\(yi,j∣xi\)\\pi\_\{\\theta\}\(y\_\{i,j\}\\mid x\_\{i\}\)grows relative toπref​\(yi,j∣xi\)\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,j\}\\mid x\_\{i\}\), indicating that the current policy has already shifted toward this behavior, the inverse ratio diminishes the gradient magnitude\. This ratio stabilizes optimization against length variations and helps prevent over\-imitation of trajectories\.

Rollout\-based solvability\.To augment offline guidance with online policy feedback, RASFT incorporates self\-sampled rollouts\. For a given promptxix\_\{i\}, the current model generatesKKrollouts, which are verified by a task\-specific verifier \(e\.g\., final\-answer matching\)\. Letci,k∈\{0,1\}c\_\{i,k\}\\in\\\{0,1\\\}indicate whether thekk\-th rollout is correct\. We define the problem\-level solvability index as:

ζi=1K​∑k=1Kci,k\.\\zeta\_\{i\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}c\_\{i,k\}\.Crucially, all incorrect rollouts are strictly filtered out from the optimization targets to avoid training noise; they are exclusively utilized to computeζi\\zeta\_\{i\}as an explicit proxy for problem difficulty\.

For compute efficiency, RASFT does not perform rollouts for every training instance\. Instead, it maintains a sliding window over the current expert\-sequence loss and allocates rollouts only to instances within this window\. Samples outside the window fall back to expert data\-only supervision, since very easy instances provide limited exploration value and difficult ones yield relatively few reliable successful rollouts\.

For samples within the window, we construct a local candidate pool𝒢i\\mathcal\{G\}\_\{i\}consisting of the expert trajectory and the model’s own successful rollouts:

𝒢i=\{yie\}∪\{yi,kθ∣ci,k=1\}\.\\mathcal\{G\}\_\{i\}=\\\{y\_\{i\}^\{e\}\\\}\\cup\\\{y\_\{i,k\}^\{\\theta\}\\mid c\_\{i,k\}=1\\\}\.Adaptive weighting and objective\.We assign trajectory\-level weightsWiW\_\{i\}within this pool\. For the expert trajectory, we assign a difficulty\-adaptive weight using the rollout success rate:

Wi​\(yie\)=Wemin\+\(1−ζi\)​\(Wemax−Wemin\),W\_\{i\}\(y\_\{i\}^\{e\}\)=W\_\{e\}^\{\\min\}\+\(1\-\\zeta\_\{i\}\)\(W\_\{e\}^\{\\max\}\-W\_\{e\}^\{\\min\}\),whereWeminW\_\{e\}^\{\\min\}andWemaxW\_\{e\}^\{\\max\}denote the minimum and maximum expert weights\. Whenζi\\zeta\_\{i\}is low, the expert trajectory provides stronger corrective supervision; whenζi\\zeta\_\{i\}is high, its influence is weakened to prevent rigid fitting\. Self\-generated correct rollouts receive a fixed weightWi​\(yi,kθ\)=WrW\_\{i\}\(y\_\{i,k\}^\{\\theta\}\)=W\_\{r\}\. We then perform a normalization over the candidate pool to obtain the relative trajectory coefficientW¯i,j\\bar\{W\}\_\{i,j\}:

W¯i,j=Wi​\(yi,j\)∑yi,r∈𝒢iWi​\(yi,r\)\.\\bar\{W\}\_\{i,j\}=\\frac\{W\_\{i\}\(y\_\{i,j\}\)\}\{\\sum\_\{y\_\{i,r\}\\in\\mathcal\{G\}\_\{i\}\}W\_\{i\}\(y\_\{i,r\}\)\}\.
Following token\-scaled optimization principles, we combine this trajectory coefficient with a token\-level confidence factorpi,j,tθp\_\{i,j,t\}^\{\\theta\}and the clipped inverse sequence ratio to derive the final compound weight:

ωi,j,t=W¯i,j​Clip⁡\(ri,j​\(θ\),1−ϵlow,1\+ϵhigh\)​pi,j,tθ,\\omega\_\{i,j,t\}\{=\}\\bar\{W\}\_\{i,j\}\\operatorname\{Clip\}\\\!\\left\(r\_\{i,j\}\(\\theta\),1\{\-\}\\epsilon\_\{\\mathrm\{low\}\},1\{\+\}\\epsilon\_\{\\mathrm\{high\}\}\\right\)p\_\{i,j,t\}^\{\\theta\},\(2\)whereϵlow\\epsilon\_\{\\mathrm\{low\}\}andϵhigh\\epsilon\_\{\\mathrm\{high\}\}bound the policy deviation limits\.W¯i,j\\bar\{W\}\_\{i,j\}, the clipped ratio term, and the token confidence factor are treated as stop\-gradient weights\. The final RASFT objective is minimized as a regularized, weighted sequence\-to\-sequence loss:

ℒ​\(θ\)=−𝔼i​\[∑yi,j∈𝒢i∑t=1\|yi,j\|ωi,j,t​log⁡pi,j,tθ\]\.\\mathcal\{L\}\(\\theta\)=\-\\mathbb\{E\}\_\{i\}\\left\[\\sum\_\{y\_\{i,j\}\\in\\mathcal\{G\}\_\{i\}\}\\sum\_\{t=1\}^\{\|y\_\{i,j\}\|\}\\omega\_\{i,j,t\}\\log p\_\{i,j,t\}^\{\\theta\}\\right\]\.
This objective keeps RASFT within a supervised fine\-tuning framework, but makes the supervision policy\-adaptive rather than uniformly imitative\. The rollout\-derived solvability controls the expert–rollout balance, while the inverse ratio constrains excessive drift from the pretrained policy\.

Table 1:Performance comparison of SFT and its variants on mathematical and code benchmarks\.The left block reports mathematical reasoning results, while the right block reports code reasoning results\.Boldnumbers indicate the best performance within each model group\. Blue\-shaded rows denote our RASFT method\.

## 3Experiments

In this section, we evaluate RASFT to answer the following research questions \(RQs\):RQ1: How does RASFT compare with recent SFT variants on mathematical and code reasoning tasks?RQ2: Is difficulty\-adaptive weighting beneficial for balancing expert trajectories and model\-generated rollouts?RQ3: Do on\-policy rollouts improve the training?RQ4: How much does the inverse ratio contribute to performance?RQ5: Can RASFT be effectively combined with different supervised fine\-tuning objectives?

### 3\.1Experimental Setup

#### Models and Data\.

We conduct fine\-tuning experiments on two widely adopted model families, namely LLaMAAI at Meta \([2024](https://arxiv.org/html/2606.07006#bib.bib34)\)and Qwen2\.5Qwen Team \([2024](https://arxiv.org/html/2606.07006#bib.bib33)\)\. \(i\)For mathematical reasoning experiments, we select Qwen2\.5\-Math\-1\.5B, Qwen2\.5\-Math\-7BYanget al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib26)\), and Llama\-3\.2\-3B due to their strong mathematical capabilities\. Following the experimental protocols of DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\)and ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\), we use 10k samples from NuminaMath CoTLiet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib7)\)for training\. We evaluate mathematical reasoning performance on six benchmarks: MATH\-500Lightmanet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib8)\), Minerva MathLewkowyczet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib9)\), OlympiadBenchHeet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib10)\), AIME 2024Mathematical Association of America \([2024](https://arxiv.org/html/2606.07006#bib.bib11)\), AIME 2025Mathematical Association of America \([2025](https://arxiv.org/html/2606.07006#bib.bib12)\), and AMC 2023Mathematical Association of America \([2023](https://arxiv.org/html/2606.07006#bib.bib13)\)\. \(ii\)For code reasoning experiments, we select Qwen2\.5\-Coder\-3B, Qwen2\.5\-Coder\-7BHuiet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib53)\), and Llama\-3\.2\-3B, and use 10k samples from KodCode\-V1\-SFT\-R1Xuet al\.\([2025](https://arxiv.org/html/2606.07006#bib.bib6)\)as the training dataset\. We evaluate code reasoning performance on HumanEvalChenet al\.\([2021](https://arxiv.org/html/2606.07006#bib.bib15)\)and MBPPAustinet al\.\([2021](https://arxiv.org/html/2606.07006#bib.bib14)\)\.

#### Baselines\.

We compare our method with standard SFT, and several recent variants of SFT\-based post\-training methods, including: \(i\) SFT, a classical post\-training paradigm that directly learns from expert demonstrations; \(ii\) DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\), which dynamically rescales the token\-level SFT objective according to token probabilities to improve generalization; \(iii\) ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\), which augments DFT with a lightweight KL\-based anchoring term to stabilize training and mitigate distributional drift; and \(iv\) ProFiTLiuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib3)\), which leverages probability\-guided token selection to mask low\-probability tokens and reduce overfitting to non\-essential surface expressions\.

#### Implementation Details\.

For training, all models are optimized with a learning rate of5×10−55\\times 10^\{\-5\}, a batch size of 256, a maximum input length of 2048 tokens, and 4 warm\-up steps\. For evaluation, we follow the evaluation protocols of two ICLR 2026 works, DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\)and ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\)to report the mean accuracy over 16 decoding runs for all models, using a temperature of 1\.0 and a maximum generation length of 4096 tokens\. See Appendix[A](https://arxiv.org/html/2606.07006#A1)for details\.

### 3\.2Main Results Analysis \(RQ1\)

RASFT establishes superior performance across mathematical and code reasoning benchmarks\.As shown in Table[1](https://arxiv.org/html/2606.07006#S2.T1), RASFT achieves the best average performance across all models on both mathematical and code reasoning tasks\. On math benchmarks, it improves the strongest baselines by \+2\.72, \+0\.27, and \+0\.14 average points on Qwen2\.5\-Math\-1\.5B, Qwen2\.5\-Math\-7B, and Llama\-3\.2\-3B, respectively; on code benchmarks, the corresponding gains are \+0\.40, \+1\.42, and \+6\.70 points\. These results indicate that rollout\-adaptive supervision provides benefits beyond static expert imitation: verified rollouts supply additional correct reasoning trajectories, while the solvability\-based weighting dynamically adjusts the reliance on expert demonstrations\. The gains are especially clear on benchmarks such as OlympiadBench and AIME24 for Qwen2\.5\-Math\-1\.5B, where RASFT improves the best baseline scores from 24\.11 to 27\.10 and from 4\.79 to 7\.92, respectively\.

The gains of RASFT vary across model scales and task domains\.On mathematical reasoning, the improvement over the strongest baseline becomes smaller as model capacity increases: on the MATH benchmark, the gain decreases from \+5\.50 points on Qwen2\.5\-Math\-1\.5B \(66\.39 vs\. 60\.89 for ASFT\) to \+2\.20 points on Qwen2\.5\-Math\-7B \(70\.65 vs\. 68\.45 for ProFiT\)\. The same pattern appears in the average mathematical score, where the gain decreases from \+2\.72 to \+0\.27 points\. This suggests that RASFT is especially helpful when the base model’s reasoning distribution is under\-optimized\. For code reasoning, however, the scaling trend is less monotonic: RASFT improves the average score by \+0\.40 points on Qwen2\.5\-Coder\-3B and by \+1\.42 points on Qwen2\.5\-Coder\-7B\. Thus, the effectiveness of RASFT depends on both model capacity and task domain\. Importantly, RASFT achieves the best average performance for both 7B mathematical and code models, indicating that its rollout\-adaptive weighting remains effective for stronger pretrained policies\.

Table 2:Ablation Studies on Qwen2\.5\-Math\-1\.5B\.Boldnumbers indicate the best performance within each block\.↑\\uparrowand↓\\downarrowindicate performance gain or drop relative to the baseline configuration within each block respectively\.
### 3\.3Ablation Study \(RQ2\-5\)

To evaluate the effectiveness of RASFT, we conduct ablation studies on Qwen2\.5\-Math\-1\.5B and report results on Math benchmarks, covering the following components:① Difficulty\-adaptive training,② The on\-policy rollout,③ The inverse importance sampling ratioand④ The transferability of RASFT to SFT and variants algorithms\.

①Difficulty\-adaptive training\.To examine whether RASFT effectively assigns different weights to trajectories within each group according to problem difficulty, we introduce a variant that assigns equal weights to the expert trajectory and verified rollout trajectories for all problems, regardless of their estimated difficulty\. As shown in Table[2](https://arxiv.org/html/2606.07006#S3.T2), removing difficulty adaptivity leads to a clear performance drop, with the average score decreasing from27\.7227\.72to25\.5325\.53\. The improvement brought by adaptive weighting is consistent across all six benchmarks:\+3\.35\+3\.35on Math,\+2\.35\+2\.35on Minerva Math,\+3\.49\+3\.49on OlympiadBench,\+2\.70\+2\.70on AIME24,\+0\.01\+0\.01on AIME25, and\+1\.25\+1\.25on AMC23\. The gains are especially pronounced on Math, OlympiadBench, and AIME24, indicating that treating all problems with the same expert\-rollout balance is suboptimal\.

②The on\-policy rollout\.To evaluate the significance of online exploration and dynamic solvability estimation, we remove the on\-policy rollout sampling mechanism and assign a fixed static weight to the expert trajectories\. As summarized in Table[2](https://arxiv.org/html/2606.07006#S3.T2), discarding the on\-policy rollouts triggers an across\-the\-board performance degradation on every single evaluated benchmark, with the average score decreasing from 27\.72 to 25\.05\. The drop is particularly pronounced on challenging datasets such as OlympiadBench \(\-3\.87%\) and AMC23 \(\-4\.06%\)\. This uniform regression demonstrates that pairing external expert trajectories with the model’s own attempts in a local comparison group is vital for effective post\-training\. Without this online feedback loop, the optimization lapses into rigid imitation of offline data without online adaptive calibration, failing to calibrate the training pressure according to the model’s evolving capacity and thereby limiting the activation of its latent reasoning pathways\.

③The inverse importance sampling ratio\.To examine the impact of the sequence\-level inverse importance sampling ratiori,j​\(θ\)r\_\{i,j\}\(\\theta\), we evaluate a variant by settingri,j​\(θ\)r\_\{i,j\}\(\\theta\)uniformly, which removes the reference\-relative policy regularization\. As shown in Table[2](https://arxiv.org/html/2606.07006#S3.T2), introducing the inverse ratio improves the overall performance from 27\.30% to 27\.72%, yielding advancements on primary benchmarks such as Math \(\+2\.15%\), Minerva Math \(\+1\.06%\), and OlympiadBench \(\+1\.19%\)\. This growth confirms that the sequence\-level ratio effectively modulates the update magnitude by dampening gradient updates when the current policy already aligns closely with the expert trajectories\. By preventing the model from overfitting to offline demonstrations, this regularization mechanism safeguards the model’s pretrained reasoning priors and ensures more balanced generalization across the mathematical benchmarks\.

④The transferability of RASFT to SFT and its variants\.To evaluate whether RASFT can generalize across different SFT\-style objectives, we integrate it with standard SFT and ProFiT\. For the SFT variant, we remove the token\-level probability factorpi,j,tθp\_\{i,j,t\}^\{\\theta\}from Eq\.[2](https://arxiv.org/html/2606.07006#S2.E2)\. For the ProFiT variant, we additionally apply its token masking strategy by filtering tokens whose predictive probabilities are below 0\.1\. Since Table[1](https://arxiv.org/html/2606.07006#S2.T1)reports the default RASFT built upon DFT, this ablation compares RASFT when combined with SFT, ProFiT, and DFT\. As shown in Tables[1](https://arxiv.org/html/2606.07006#S2.T1)and[2](https://arxiv.org/html/2606.07006#S3.T2), RASFT consistently improves all three SFT\-style baselines, confirming its transferability\. In particular, it improves the average score by \+0\.90 points over SFT, \+2\.91 points over ProFiT, and \+3\.53 points over DFT\. The gains are also evident on difficult competition benchmarks: RASFT raises ProFiT on AIME25 from 2\.28% to 4\.17%, and improves DFT on OlympiadBench from 22\.42% to 27\.10%\. These results support our motivation that difficult problems benefit from stronger expert guidance, while easier problems can rely more on verified model\-generated trajectories\. Moreover, the larger gains on ProFiT and DFT suggest that RASFT is complementary to token\-level rescaling or filtering methods, as it introduces sequence\-level rollout adaptivity and candidate\-pool weighting\.

## 4Analysis and Discussion

### 4\.1Comparison with RL methods

To contextualize RASFT within the landscape of recent RL alignment paradigms, we evaluate our method against two representative RL\-driven algorithms: \(i\) GRPOShaoet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib31)\), a prominent pure on\-policy reinforcement learning baseline, and \(ii\) LUFFYYanet al\.\([2025](https://arxiv.org/html/2606.07006#bib.bib32)\), a mixed\-policy framework that augments on\-policy exploration with off\-policy reasoning guidance via regularized importance sampling\. For all 3 experiments, we train on Qwen2\.5\-Math\-1\.5B\. More training details are provided in Appendix[C](https://arxiv.org/html/2606.07006#A3)\.

As illustrated in Figure[2](https://arxiv.org/html/2606.07006#S4.F2), RASFT secures the highest average accuracy of 23\.47%\. This robust performance is characterized by two primary insights: first, our solvability\-aware modulation adaptively scales up the expert trajectory weight when online exploration fails, which successfully bypasses the exploration cold\-start and optimization stagnation that cause pure on\-policy methods like GRPO to collapse on ultra\-hard tasks like AIME25; second, while unconstrained exploration enables GRPO to occasionally discover highly alternative reasoning paths on specific domains like Minerva Math, RASFT maintains a superior and more stable overall optimization trajectory by utilizing the sequence\-level inverse importance sampling ratio to dampen redundant updates and safeguard pretrained reasoning priors\.

![Refer to caption](https://arxiv.org/html/2606.07006v1/x2.png)Figure 2:Comparison between RASFT and representative RL methods \(GRPO and LUFFY\)\. RASFT demonstrates superior performance by dynamically balancing offline expert guidance and online exploration\. More training details are provided in Appendix[C](https://arxiv.org/html/2606.07006#A3)\.
### 4\.2Training Dynamics Analysis

We further analyze the training dynamics of different SFT\-style methods by tracking the training objective and parameter\-update magnitude\. For the latter, we define parameter\-update density as the fraction of parameters whose absolute update magnitude exceeds10−510^\{\-5\}at each optimization step, which reflects how broadly model parameters are modified during training\. As shown in Figure[3](https://arxiv.org/html/2606.07006#S4.F3), RASFT maintains a stable training trajectory\. Its training objective decreases smoothly and does not exhibit noticeable oscillation, showing a stability pattern consistent with strong SFT variants such as DFT, ASFT, and ProFiT\. This indicates that introducing verified rollouts and difficulty\-adaptive weighting does not destabilize supervised post\-training\. Meanwhile, the parameter\-update density of all methods increases rapidly in the early stage and then saturates, while RASFT remains comparable to or slightly lower than the other methods in the late stage\. Specifically, RASFT reaches a final update density of0\.24830\.2483, compared with0\.25070\.2507for ProFiT,0\.25400\.2540for ASFT,0\.25730\.2573for DFT, and0\.25490\.2549for SFT; the last\-five\-step average follows the same pattern, with RASFT obtaining the lowest density of0\.24800\.2480\. This supports our motivation that rollout\-adaptive weighting, together with the inverse ratio, provides targeted supervision while better preserving the pretrained reasoning prior\.

![Refer to caption](https://arxiv.org/html/2606.07006v1/x3.png)Figure 3:Training dynamics\.Training loss and parameter\-update density across SFT variants; density is the fraction of updates with magnitude above10−510^\{\-5\}\.
### 4\.3Sensitivity to the Rollout Number

To examine the sensitivity of RASFT to the rollout number, we varyKKin\{3,5,7,9\}\\\{3,5,7,9\\\}and report the average mathematical reasoning accuracy in Figure[4](https://arxiv.org/html/2606.07006#S5.F4)\. The performance first increases from 27\.72 atK=3K=3to 29\.33 atK=5K=5, yielding a gain of 1\.61 points, but then drops to 27\.01 atK=7K=7and 26\.37 atK=9K=9, corresponding to decreases of 2\.32 and 2\.96 points compared withK=5K=5\. This non\-monotonic trend shows that simply increasing the rollout budget does not necessarily improve RASFT: a small rollout number may provide insufficient on\-policy feedback, while excessive rollouts may introduce redundant self\-generated trajectories and weaken the relative contribution of expert demonstrations after candidate\-pool normalization\. Nevertheless, all rollout settings still outperform the other baselines in Table[1](https://arxiv.org/html/2606.07006#S2.T1); even the weakest setting, K=9, achieves 26\.37 average accuracy, which remains higher than the strongest baseline average of 25\.00\. The complete statistical results are provided in Appendix[B](https://arxiv.org/html/2606.07006#A2)\.

## 5Related Work

#### Objective Refinement of SFT\.

Supervised SFT with Chain\-of\-Thought \(CoT\) demonstrations is widely used to activate the reasoning abilities of large language modelsWeiet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib16)\); Chunget al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib38)\); Mukherjeeet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib39)\); Yueet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib40)\)\. However, standard cross\-entropy training treats offline demonstrations as fixed targets and uniformly maximizes the likelihood of the reference response, which can lead to overfitting to demonstration\-specific patterns and limited generalization\. Recent methods improve SFT by reweighting or selecting supervision signals from offline data\. DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\)rescales the objective according to the model’s prediction probability, ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\)adds reference anchoring to stabilize this reweighting, and ProFiTLiuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib3)\)selects high\-value tokens to reduce unnecessary fitting\. However, they still center on the expert response and overlook model\-generated reasoning trajectories\.

![Refer to caption](https://arxiv.org/html/2606.07006v1/x4.png)Figure 4:Sensitivity to the Rollout Number\.
#### Rollout\-Based Policy Optimization for Reasoning\.

Recent post\-training methods for reasoning increasingly rely on reinforcement learning with sampled trajectories and outcome feedbackCobbeet al\.\([2021](https://arxiv.org/html/2606.07006#bib.bib35)\); Yuanet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib36)\); Zelikmanet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib37)\); Gulcehreet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib30)\); DeepSeek\-AI \([2025](https://arxiv.org/html/2606.07006#bib.bib25)\)\. PPOSchulmanet al\.\([2017](https://arxiv.org/html/2606.07006#bib.bib21)\)has been widely used as a policy\-optimization algorithm, using clipped policy updates to improve training stability\. Building on this direction, GRPOShaoet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib31)\)optimizes relative advantages within sampled groups, emphasizing the importance of comparing multiple candidate reasoning trajectories rather than fitting one reference response\. More recently, LUFFYYanet al\.\([2025](https://arxiv.org/html/2606.07006#bib.bib32)\)incorporates off\-policy reasoning guidance into RL training, showing the value of combining online exploration with external trajectory information\. However, these RL methods require explicit policy optimization and extensive rollouts, whereas RASFT uses verified rollouts as adaptive supervision within an SFT objective\.

## 6Conclusions and Future Work

We introduce RASFT, a rollout\-adaptive supervised fine\-tuning method that calibrates expert supervision using the model’s own rollout behavior\. Empirical results show that RASFT improves over SFT and SFT variants across multiple models and challenging mathematical and code reasoning tasks\. Compared with RL methods, RASFT also achieves stronger overall performance\. Overall, our results suggest that expert demonstrations can serve as adaptive guidance rather than fixed imitation targets, while the inverse policy ratio helps preserve useful reasoning priors\. Looking forward, a promising direction is to extend RASFT to open\-ended reasoning tasks where reliable automatic correctness signals are harder to define\. Another direction is to study how rollout\-adaptive supervision scales with stronger models and larger training corpora across broader domains, especially when combined with more diverse verifiers and broader forms of verified, diverse self\-generated supervision\.

## Limitations

RASFT introduces additional computational cost compared with standard SFT, since it requires periodically generating on\-policy rollouts and verifying their correctness\. Although we reduce this cost by applying rollouts only to selected training instances, the overall training pipeline is still more expensive than methods that only optimize offline demonstrations\. This also makes the method sensitive to the rollout budget: using too few rollouts may provide insufficient feedback, while excessive rollouts can increase computation and introduce redundant verified trajectories\. Our method also relies on task\-specific verification signals\. In mathematical reasoning, we verify sampled responses by extracting the final answer from\\boxed​\{\}\\backslash\\texttt\{boxed\}\\\{\\\}and comparing it with the ground\-truth answer, following the common practice of evaluating mathematical problem solving through final\-answer correctnessHendryckset al\.\([2021b](https://arxiv.org/html/2606.07006#bib.bib44)\)\. In code reasoning, we use assertion\-based tests, following the common practice of evaluating generated programs with executable test casesHendryckset al\.\([2021a](https://arxiv.org/html/2606.07006#bib.bib49)\); Liet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib50)\)\. Therefore, RASFT is most directly applicable to domains where reliable automatic verification is available\. Extending the method to open\-ended reasoning tasks without clear correctness signals remains challenging\. Such tasks often rely on human preferences or learned reward models, which require costly annotation and may depend on task\-specific preference modelingStiennonet al\.\([2020](https://arxiv.org/html/2606.07006#bib.bib45)\); Baiet al\.\([2022](https://arxiv.org/html/2606.07006#bib.bib46)\)\. Recent LLM\-based judges provide a scalable alternative, but they may still introduce evaluator\-specific biases, including positional or instruction\-following failuresZhenget al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib51)\); Liuet al\.\([2023](https://arxiv.org/html/2606.07006#bib.bib52)\); Wanget al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib47)\); Zenget al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib48)\)\.

## References

- Llama 3\.2: revolutionizing edge ai and vision with open, customizable models\.Note:Meta AI BlogAccessed: 2026\-05\-21External Links:[Link](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.External Links:2108\.07732,[Document](https://dx.doi.org/10.48550/arXiv.2108.07732),[Link](https://arxiv.org/abs/2108.07732)Cited by:[Appendix A](https://arxiv.org/html/2606.07006#A1.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. El\-Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. Kaplan \(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.External Links:2204\.05862,[Document](https://dx.doi.org/10.48550/arXiv.2204.05862),[Link](https://arxiv.org/abs/2204.05862)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.External Links:2107\.03374,[Document](https://dx.doi.org/10.48550/arXiv.2107.03374),[Link](https://arxiv.org/abs/2107.03374)Cited by:[Appendix A](https://arxiv.org/html/2606.07006#A1.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. Ma \(2025\)SFT memorizes, rl generalizes: a comparative study of foundation model post\-training\.InInternational Conference on Machine Learning,External Links:2501\.17161,[Document](https://dx.doi.org/10.48550/arXiv.2501.17161),[Link](https://arxiv.org/abs/2501.17161)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1)\.
- H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma, A\. Webson, S\. S\. Gu, Z\. Dai, M\. Suzgun, X\. Chen, A\. Chowdhery, A\. Castro\-Ros, M\. Pellat, K\. Robinson, D\. Valter, S\. Narang, G\. Mishra, A\. Yu, V\. Zhao, Y\. Huang, A\. Dai, H\. Yu, S\. Petrov, E\. H\. Chi, J\. Dean, J\. Devlin, A\. Roberts, D\. Zhou, Q\. V\. Le, and J\. Wei \(2024\)Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.External Links:2210\.11416,[Link](https://arxiv.org/abs/2210.11416)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:2110\.14168,[Document](https://dx.doi.org/10.48550/arXiv.2110.14168),[Link](https://arxiv.org/abs/2110.14168)Cited by:[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.External Links:2501\.12948,[Document](https://dx.doi.org/10.48550/arXiv.2501.12948),[Link](https://arxiv.org/abs/2501.12948)Cited by:[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- C\. Gulcehre, T\. L\. Paine, S\. Srinivasan, K\. Konyushkova, L\. Weerts, A\. Sharma, A\. Siddhant, A\. Ahern, M\. Wang, C\. Gu, W\. Macherey, A\. Doucet, O\. Firat, and N\. de Freitas \(2023\)Reinforced self\-training \(ReST\) for language modeling\.arXiv preprint arXiv:2308\.08998\.External Links:2308\.08998,[Document](https://dx.doi.org/10.48550/arXiv.2308.08998),[Link](https://arxiv.org/abs/2308.08998)Cited by:[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,External Links:2402\.14008,[Document](https://dx.doi.org/10.48550/arXiv.2402.14008),[Link](https://arxiv.org/abs/2402.14008)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. Steinhardt \(2021a\)Measuring coding challenge competence with APPS\.InAdvances in Neural Information Processing Systems,External Links:2105\.09938,[Document](https://dx.doi.org/10.48550/arXiv.2105.09938),[Link](https://arxiv.org/abs/2105.09938)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems,External Links:2103\.03874,[Document](https://dx.doi.org/10.48550/arXiv.2103.03874),[Link](https://arxiv.org/abs/2103.03874)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu, K\. Dang, Y\. Fan, Y\. Zhang, A\. Yang, R\. Men, F\. Huang, B\. Zheng, Y\. Miao, S\. Quan, Y\. Feng, X\. Ren, X\. Ren, J\. Zhou, and J\. Lin \(2024\)Qwen2\.5\-Coder technical report\.arXiv preprint arXiv:2409\.12186\.External Links:2409\.12186,[Document](https://dx.doi.org/10.48550/arXiv.2409.12186),[Link](https://arxiv.org/abs/2409.12186)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra \(2022\)Solving quantitative reasoning problems with language models\.arXiv preprint arXiv:2206\.14858\.External Links:2206\.14858,[Document](https://dx.doi.org/10.48550/arXiv.2206.14858),[Link](https://arxiv.org/abs/2206.14858)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. C\. Huang, K\. Rasul, L\. Yu, A\. Jiang, Z\. Shen, Z\. Qin, B\. Dong, L\. Zhou, Y\. Fleureau, G\. Lample, and S\. Polu \(2024\)NuminaMath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Note:Hugging Face datasetDataset report:[https://github\.com/project\-numina/aimo\-progress\-prize/blob/main/report/numina\_dataset\.pdf](https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf)External Links:[Link](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. Vinyals \(2022\)Competition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Document](https://dx.doi.org/10.1126/science.abq1158),[Link](https://www.science.org/doi/10.1126/science.abq1158)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.External Links:2305\.20050,[Document](https://dx.doi.org/10.48550/arXiv.2305.20050),[Link](https://arxiv.org/abs/2305.20050)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- T\. Liu, T\. Wu, R\. Yang, S\. Sun, J\. Wang, and Y\. Yang \(2026\)ProFit: leveraging high\-value signals in sft via probability\-guided token selection\.arXiv preprint arXiv:2601\.09195\.External Links:2601\.09195,[Document](https://dx.doi.org/10.48550/arXiv.2601.09195),[Link](https://arxiv.org/abs/2601.09195)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p2.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.15.15.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.22.22.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.8.8.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- Mathematical Association of America \(2023\)2023 American Mathematics Competitions\.Note:MAA American Mathematics CompetitionsAMC 2023 competition problemsExternal Links:[Link](https://maa.org/student-programs/amc/)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- Mathematical Association of America \(2024\)2024 American Invitational Mathematics Examination\.Note:MAA American Mathematics CompetitionsAIME 2024 competition problemsExternal Links:[Link](https://maa.org/maa-invitational-competitions/)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- Mathematical Association of America \(2025\)2025 American Invitational Mathematics Examination\.Note:MAA American Mathematics CompetitionsAIME 2025 competition problemsExternal Links:[Link](https://maa.org/maa-invitational-competitions/)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- S\. Mukherjee, A\. Mitra, G\. Jawahar, S\. Agarwal, H\. Palangi, and A\. Awadallah \(2023\)Orca: progressive learning from complex explanation traces of GPT\-4\.arXiv preprint arXiv:2306\.02707\.External Links:2306\.02707,[Document](https://dx.doi.org/10.48550/arXiv.2306.02707),[Link](https://arxiv.org/abs/2306.02707)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,External Links:2203\.02155,[Document](https://dx.doi.org/10.48550/arXiv.2203.02155),[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:2412\.15115,[Document](https://dx.doi.org/10.48550/arXiv.2412.15115),[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](https://arxiv.org/abs/2402.03300)Cited by:[Appendix C](https://arxiv.org/html/2606.07006#A3.p1.1),[§1](https://arxiv.org/html/2606.07006#S1.p5.6),[§4\.1](https://arxiv.org/html/2606.07006#S4.SS1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. Christiano \(2020\)Learning to summarize from human feedback\.InAdvances in Neural Information Processing Systems,External Links:2009\.01325,[Document](https://dx.doi.org/10.48550/arXiv.2009.01325),[Link](https://arxiv.org/abs/2009.01325)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,pp\. 13484–13508\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754),[Link](https://aclanthology.org/2023.acl-long.754)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,External Links:2201\.11903,[Document](https://dx.doi.org/10.48550/arXiv.2201.11903),[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Wu, Y\. Zhou, Z\. Zhou, Y\. Peng, X\. Ye, X\. Hu, W\. Zhu, L\. Qi, M\. Yang, and X\. Yang \(2026\)On the generalization of sft: a reinforcement learning perspective with reward rectification\.InInternational Conference on Learning Representations,External Links:2508\.05629,[Document](https://dx.doi.org/10.48550/arXiv.2508.05629),[Link](https://arxiv.org/abs/2508.05629)Cited by:[Appendix A](https://arxiv.org/html/2606.07006#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.07006#S1.p2.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.13.13.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.20.20.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.6.6.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- Z\. Xu, Y\. Liu, Y\. Yin, M\. Zhou, and R\. Poovendran \(2025\)KodCode: a diverse, challenging, and verifiable synthetic dataset for coding\.arXiv preprint arXiv:2503\.02951\.External Links:2503\.02951,[Document](https://dx.doi.org/10.48550/arXiv.2503.02951),[Link](https://arxiv.org/abs/2503.02951)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Yan, Y\. Li, Z\. Hu, Z\. Wang, G\. Cui, X\. Qu, Y\. Cheng, and Y\. Zhang \(2025\)Learning to reason under off\-policy guidance\.arXiv preprint arXiv:2504\.14945\.External Links:2504\.14945,[Document](https://dx.doi.org/10.48550/arXiv.2504.14945),[Link](https://arxiv.org/abs/2504.14945)Cited by:[Appendix C](https://arxiv.org/html/2606.07006#A3.p1.1),[§4\.1](https://arxiv.org/html/2606.07006#S4.SS1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024\)Qwen2\.5\-Math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.External Links:2409\.12122,[Document](https://dx.doi.org/10.48550/arXiv.2409.12122),[Link](https://arxiv.org/abs/2409.12122)Cited by:[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. Zhou \(2023\)Scaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.External Links:2308\.01825,[Document](https://dx.doi.org/10.48550/arXiv.2308.01825),[Link](https://arxiv.org/abs/2308.01825)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- X\. Yue, X\. Qu, G\. Zhang, Y\. Fu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2023\)MAmmoTH: building math generalist models through hybrid instruction tuning\.arXiv preprint arXiv:2309\.05653\.External Links:2309\.05653,[Document](https://dx.doi.org/10.48550/arXiv.2309.05653),[Link](https://arxiv.org/abs/2309.05653)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman \(2022\)STaR: self\-taught reasoner bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems,External Links:2203\.14465,[Document](https://dx.doi.org/10.48550/arXiv.2203.14465),[Link](https://arxiv.org/abs/2203.14465)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px2.p1.1)\.
- Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. Chen \(2024\)Evaluating large language models at evaluating instruction following\.InInternational Conference on Learning Representations,External Links:2310\.07641,[Document](https://dx.doi.org/10.48550/arXiv.2310.07641),[Link](https://arxiv.org/abs/2310.07641)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,External Links:2306\.05685,[Document](https://dx.doi.org/10.48550/arXiv.2306.05685),[Link](https://arxiv.org/abs/2306.05685)Cited by:[Limitations](https://arxiv.org/html/2606.07006#Sx1.p1.1)\.
- C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu, S\. Zhang, G\. Ghosh, M\. Lewis, L\. Zettlemoyer, and O\. Levy \(2023\)LIMA: less is more for alignment\.InAdvances in Neural Information Processing Systems,External Links:2305\.11206,[Document](https://dx.doi.org/10.48550/arXiv.2305.11206),[Link](https://arxiv.org/abs/2305.11206)Cited by:[§1](https://arxiv.org/html/2606.07006#S1.p1.1)\.
- H\. Zhu, J\. Su, P\. Lai, R\. Ma, W\. Zhang, L\. Yang, and G\. Chen \(2026\)Anchored supervised fine\-tuning\.InInternational Conference on Learning Representations,External Links:2509\.23753,[Document](https://dx.doi.org/10.48550/arXiv.2509.23753),[Link](https://arxiv.org/abs/2509.23753)Cited by:[Appendix A](https://arxiv.org/html/2606.07006#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.07006#S1.p2.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.14.14.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.21.21.1),[Table 1](https://arxiv.org/html/2606.07006#S2.T1.1.1.1.1.1.1.1.7.7.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.07006#S3.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2606.07006#S5.SS0.SSS0.Px1.p1.1)\.

## Appendix ATraining and Evaluation Details

#### Training details\.

For all experiments, we train all models with AdamW and a cosine learning\-rate schedule\. We use a learning rate of5×10−55\\times 10^\{\-5\}, an effective batch size of 256, a batch size per\-device of 2, a maximum input length of 2048 tokens, 4 warm\-up steps, bf16 precision, gradient checkpointing and gradient clipping with a maximum norm of 1\.0\. All models are trained for 1 epoch with random seed 42\. For RASFT, we generate 3 rollouts per selected prompt with temperature 0\.9, top\-p 0\.95, repetition penalty 1\.1, and a maximum rollout length of 1024 tokens\. We clip the ratio to \[0\.4, 1\.1\]\.

For RASFT on mathematical tasks, we setWemin=Wr=1\.0W\_\{e\}^\{\\min\}=W\_\{r\}=1\.0andWemax=3\.0W\_\{e\}^\{\\max\}=3\.0for the two Qwen2\.5\-Math models, andWemin=Wr=1\.5W\_\{e\}^\{\\min\}=W\_\{r\}=1\.5andWemax=3\.0W\_\{e\}^\{\\max\}=3\.0for Llama\-3\.2\-3B\. The correctness of sampled responses is verified by extracting and comparing the final answer enclosed in the last\\boxed​\{\}\\backslash\\texttt\{boxed\}\\\{\\\}in the response\. For code tasks, we setWemin=1\.0W\_\{e\}^\{\\min\}=1\.0,Wr=1\.5W\_\{r\}=1\.5, andWemax=1\.8W\_\{e\}^\{\\max\}=1\.8for the two Qwen2\.5\-Coder models, andWemin=Wr=1\.5W\_\{e\}^\{\\min\}=W\_\{r\}=1\.5andWemax=3\.0W\_\{e\}^\{\\max\}=3\.0for Llama\-3\.2\-3B\. The correctness of generated code samples is verified using assertion\-based tests\.

#### Evaluation details\.

We follow the evaluation protocols of DFTWuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib1)\)and ASFTZhuet al\.\([2026](https://arxiv.org/html/2606.07006#bib.bib2)\)\. For mathematical reasoning, we evaluate all methods on the 6 mathematical reasoning benchmarks mentioned above\. For each problem, we sample 16 responses with temperature 1\.0 and a maximum generation length of 4096 tokens\. We report the average accuracy over the 16 sampled responses\. Correctness is determined by extracting the final answer and comparing it with the ground\-truth answer; for code reasoning, we evaluate all methods on HumanEvalChenet al\.\([2021](https://arxiv.org/html/2606.07006#bib.bib15)\)and MBPPAustinet al\.\([2021](https://arxiv.org/html/2606.07006#bib.bib14)\)\. We use the same decoding setting as in mathematical reasoning: temperature 1\.0, 16 sampled responses per problem, and a maximum generation length of 4096 tokens\. We report the mean accuracy over the 16 sampled responses, where the correctness is determined by executing the generated code against assertion\-based tests\.

#### Computation cost\.

We report the computational cost of different training modes in Table[3](https://arxiv.org/html/2606.07006#A1.T3)\. For the baseline methods reported in Table[1](https://arxiv.org/html/2606.07006#S2.T1), SFT, DFT, and ProFiT are expected to have nearly identical computation costs, since DFT and ProFiT do not introduce additional model components and only modify the token\-level training objective\. In contrast, ASFT introduces an additional reference model, leading to higher memory usage and different computational efficiency\. Therefore, we report the training costs of SFT, ASFT, and RASFT on the math task using 4× A100 SXM4 GPUs\.

Table 3:Computation cost comparison across different training modes\.Table 4:Detailed mathematical reasoning results of RASFT with different rollout numbersKKon Qwen2\.5\-Math\-1\.5B\. We report accuracy on each benchmark and the average accuracy across all six benchmarks\.

## Appendix BDetailed Results for Rollout Number Sensitivity

This appendix provides the complete benchmark\-level results for the rollout number sensitivity analysis\. We evaluate RASFT on Qwen2\.5\-Math\-1\.5B using 10k samples from NuminaMath CoT, varying the number of sampled rolloutsKKin\{3,5,7,9\}\\\{3,5,7,9\\\}\. The results are reported in Table[4](https://arxiv.org/html/2606.07006#A1.T4)\. Overall, the best average performance is obtained whenK=5K=5, suggesting that a moderate number of rollouts provides more informative on\-policy feedback than a smaller rollout budget\. However, further increasingKKto 7 or 9 does not lead to additional gains\. This supports the observation that excessive rollouts may introduce redundant self\-generated trajectories and reduce the relative influence of expert demonstrations after candidate\-pool normalization\. Although we useK=3K=3in the main experiments for efficiency, all evaluated rollout settings achieve higher average accuracy than the strongest non\-RASFT baseline reported in Table[1](https://arxiv.org/html/2606.07006#S2.T1)for Qwen2\.5\-Math\-1\.5B\.

## Appendix CTraining Details for RL Baselines

Table 5:Training hyperparameters used for GRPO and LUFFY\. We follow the configuration provided by LUFFY\.We compare RASFT with two representative RL\-based training methods, GRPOShaoet al\.\([2024](https://arxiv.org/html/2606.07006#bib.bib31)\)and LUFFYYanet al\.\([2025](https://arxiv.org/html/2606.07006#bib.bib32)\), on Qwen2\.5\-Math\-1\.5B\. For RASFT, we use the same training configuration as described in Appendix[A](https://arxiv.org/html/2606.07006#A1)\. For GRPO and LUFFY, we follow the configuration provided by LUFFY\. For all 3 experiments, we sample 2000 from NuminaMath CoT as training data\. The main hyperparameters are summarized in Table[5](https://arxiv.org/html/2606.07006#A3.T5)\.

For LUFFY, we keep its default mixed\-policy training configuration\. In particular, we enable the token\-level off\-policy loss, use no off\-policy normalization, and adopt thep\_div\_p\_0\.1off\-policy reshaping strategy\. For GRPO, we use the group\-relative advantage estimator under the same rollout and optimization settings\.

## Appendix DCase Studies

### D\.1Case 1

QuestionIt rained some inches on Monday\. On Tuesday, it rained0\.70\.7inches less than on Monday, and it rained0\.20\.2inches on Tuesday\. How much did it rain on Monday?

Standard ResponseLetMMdenote the amount of rain on Monday\. Since Tuesday’s rainfall was0\.70\.7inches less than Monday’s rainfall, and Tuesday’s rainfall was0\.20\.2inches, we haveM−0\.7=0\.2\.M\-0\.7=0\.2\.Solving forMM,M=0\.2\+0\.7=0\.9\.M=0\.2\+0\.7=0\.9\.Therefore, it rained0\.9\\boxed\{0\.9\}inches on Monday\.

Model Sampling Rollout 1The model denotes Monday’s rainfall asMM\. Since Tuesday’s rainfall is0\.70\.7inches less than Monday’s rainfall, it sets upM−0\.7=0\.2\.M\-0\.7=0\.2\.Solving the equation givesM=0\.2\+0\.7=0\.9\.M=0\.2\+0\.7=0\.9\.Thus, the model predicts0\.9\.\\boxed\{0\.9\}\.

Model Sampling Rollout 2The model writes the relation between Monday’s and Tuesday’s rainfall asM−0\.7=0\.2\.M\-0\.7=0\.2\.It then solves forMM:M=0\.2\+0\.7=0\.9\.M=0\.2\+0\.7=0\.9\.Therefore, the model predicts0\.9\.\\boxed\{0\.9\}\.

Model Sampling Rollout 3The model again represents Tuesday’s rainfall asM−0\.7M\-0\.7, whereMMis Monday’s rainfall\. Since Tuesday’s rainfall is0\.20\.2, it obtainsM−0\.7=0\.2\.M\-0\.7=0\.2\.Therefore,M=0\.9\.M=0\.9\.Thus, the model predicts0\.9\.\\boxed\{0\.9\}\.

### D\.2Case 2

QuestionLet\(x\+2\)n=a0\+a1​x\+a2​x2\+⋯\+an​xn,\(x\+2\)^\{n\}=a\_\{0\}\+a\_\{1\}x\+a\_\{2\}x^\{2\}\+\\cdots\+a\_\{n\}x^\{n\},wheren∈ℕ∗n\\in\\mathbb\{N\}^\{\*\},n≥2n\\geq 2, anda0a\_\{0\},a1a\_\{1\},a2a\_\{2\}form an arithmetic sequence\.\(1\) Find the middle term of the expansion of\(x\+2\)n\(x\+2\)^\{n\}\.\(2\) Find the sum of the coefficients of the terms with odd powers ofxxin the expansion of\(x\+2\)n\(x\+2\)^\{n\}\.

Standard ResponseThe coefficient ofxrx^\{r\}in\(x\+2\)n\(x\+2\)^\{n\}isar=\(nr\)​2n−r\.a\_\{r\}=\\binom\{n\}\{r\}2^\{n\-r\}\.Thus,a0=2n,a1=n​2n−1,a2=\(n2\)​2n−2\.a\_\{0\}=2^\{n\},\\qquad a\_\{1\}=n2^\{n\-1\},\\qquad a\_\{2\}=\\binom\{n\}\{2\}2^\{n\-2\}\.Sincea0,a1,a2a\_\{0\},a\_\{1\},a\_\{2\}form an arithmetic sequence,2​a1=a0\+a2\.2a\_\{1\}=a\_\{0\}\+a\_\{2\}\.Substituting the coefficients gives2​n​2n−1\\displaystyle 2n2^\{n\-1\}=2n\+\(n2\)​2n−2,\\displaystyle=2^\{n\}\+\\binom\{n\}\{2\}2^\{n\-2\},n2−9​n\+8\\displaystyle n^\{2\}\-9n\+8=0\.\\displaystyle=0\.Hencen=1n=1orn=8n=8\. Sincen≥2n\\geq 2, we haven=8n=8\.Forn=8n=8, the expansion has99terms, so the middle term is the fifth term:T5=\(84\)​24​x4=1120​x4\.T\_\{5\}=\\binom\{8\}\{4\}2^\{4\}x^\{4\}=1120x^\{4\}\.For the sum of the coefficients of odd powers ofxx, letf​\(x\)=\(x\+2\)8\.f\(x\)=\(x\+2\)^\{8\}\.Thenf​\(1\)=38,f​\(−1\)=1\.f\(1\)=3^\{8\},\\qquad f\(\-1\)=1\.Therefore,a1\+a3\+a5\+a7\\displaystyle a\_\{1\}\+a\_\{3\}\+a\_\{5\}\+a\_\{7\}=f​\(1\)−f​\(−1\)2\\displaystyle=\\frac\{f\(1\)\-f\(\-1\)\}\{2\}=38−12\\displaystyle=\\frac\{3^\{8\}\-1\}\{2\}=3280\.\\displaystyle=280\.Thus, the answers are1120​x4and3280\.\\boxed\{1120x^\{4\}\}\\qquad\\text\{and\}\\qquad\\boxed\{3280\}\.

Model Sampling Rollout 1The model correctly uses the arithmetic\-sequence condition2​a1=a0\+a2,2a\_\{1\}=a\_\{0\}\+a\_\{2\},wherea0\\displaystyle a\_\{0\}=2n,\\displaystyle=2^\{n\},a1\\displaystyle a\_\{1\}=\(n1\)​2n−1,\\displaystyle=\\binom\{n\}\{1\}2^\{n\-1\},a2\\displaystyle a\_\{2\}=\(n2\)​2n−2\.\\displaystyle=\\binom\{n\}\{2\}2^\{n\-2\}\.It obtainsn2−9​n\+8=0,n^\{2\}\-9n\+8=0,son=8n=8\. Therefore, the middle term is\(84\)​24​x4=1120​x4\.\\binom\{8\}\{4\}2^\{4\}x^\{4\}=1120x^\{4\}\.For the odd\-power coefficients, the model evaluatesf​\(1\)−f​\(−1\)2\\displaystyle\\frac\{f\(1\)\-f\(\-1\)\}\{2\}=38−12\\displaystyle=\\frac\{3^\{8\}\-1\}\{2\}=3280\.\\displaystyle=280\.Thus, this rollout predicts the correct answers:1120​x4and3280\.\\boxed\{1120x^\{4\}\}\\qquad\\text\{and\}\\qquad\\boxed\{3280\}\.

Model Sampling Rollout 2The model correctly derivesn2−9​n\+8=0,n^\{2\}\-9n\+8=0,and therefore correctly obtainsn=8n=8\. It also correctly identifies the middle term:\(84\)​24​x4=1120​x4\.\\binom\{8\}\{4\}2^\{4\}x^\{4\}=1120x^\{4\}\.However, when computing the sum of the odd\-power coefficients directly, it writesS\\displaystyle S=\(81\)​27\+\(83\)​25\\displaystyle=\\binom\{8\}\{1\}2^\{7\}\+\\binom\{8\}\{3\}2^\{5\}\+\(85\)​23\+\(87\)​2\.\\displaystyle\\quad\+\\binom\{8\}\{5\}2^\{3\}\+\\binom\{8\}\{7\}2\.This expression is correct, but the model evaluates it incorrectly asS=1056\.S=1056\.The correct value should beS=3280\.S=3280\.Thus, this rollout has a local arithmetic error in the final computation and predicts1056\.\\boxed\{1056\}\.

Model Sampling Rollout 3The model starts from the correct condition2​a1=a0\+a2,2a\_\{1\}=a\_\{0\}\+a\_\{2\},but then incorrectly simplifies the resulting equation as2​n=2\+n​\(n−1\)2,2n=2\+\\frac\{n\(n\-1\)\}\{2\},which leads ton=4\.n=4\.This is incorrect\. The correct simplification givesn2−9​n\+8=0,n^\{2\}\-9n\+8=0,son=8n=8\.Because of this early algebraic error, the model gives the wrong middle term:24​x2\.24x^\{2\}\.It then continues withn=4n=4and computesf​\(1\)−f​\(−1\)=34−1=80\.f\(1\)\-f\(\-1\)=3^\{4\}\-1=80\.This also misses the required factor of1/21/2for extracting odd\-power coefficients\. Therefore, this rollout contains both an incorrect value ofnnand an incorrect odd\-coefficient computation\. It predicts80\.\\boxed\{80\}\.

### D\.3Case 3

Question[⬇](data:text/plain;base64,ZGVmIG5leHRHcmVhdGVyRWxlbWVudChhcnIpOgogICAgIiIiCiAgICBNb2RpZnkgdGhlIGFycmF5IHN1Y2ggdGhhdCBlYWNoIGVsZW1lbnQgaW4gdGhlIGFycmF5IGlzIHJlcGxhY2VkIHdpdGggdGhlIG5leHQKICAgIGdyZWF0ZXIgZWxlbWVudCBwcmVzZW50IG9uIGl0cyByaWdodCBzaWRlIGluIHRoZSBhcnJheS4gSWYgdGhlcmUgaXMgbm8gZ3JlYXRlcgogICAgZWxlbWVudCwgcmVwbGFjZSBpdCB3aXRoIC0xLgoKICAgIFBhcmFtZXRlcnM6CiAgICBhcnIgKExpc3RbaW50XSk6IEFuIGFycmF5IG9mIGludGVnZXJzCgogICAgUmV0dXJuczoKICAgIExpc3RbaW50XTogQSBuZXcgYXJyYXkgd2l0aCBlYWNoIGVsZW1lbnQgcmVwbGFjZWQgYnkgdGhlIG5leHQgZ3JlYXRlciBlbGVtZW50CgogICAgRXhhbXBsZXM6CiAgICA+Pj4gbmV4dEdyZWF0ZXJFbGVtZW50KFs0LCA1LCAyLCAxMCwgOF0pCiAgICBbNSwgMTAsIDEwLCAtMSwgLTFdCgogICAgPj4+IG5leHRHcmVhdGVyRWxlbWVudChbMywgNywgMSwgNywgOCwgNCwgNV0pCiAgICBbNywgOCwgNywgOCwgLTEsIDUsIC0xXQogICAgIiIi)defnextGreaterElement\(arr\):"""Modifythearraysuchthateachelementinthearrayisreplacedwiththenextgreaterelementpresentonitsrightsideinthearray\.Ifthereisnogreaterelement,replaceitwith\-1\.Parameters:arr\(List\[int\]\):AnarrayofintegersReturns:List\[int\]:AnewarraywitheachelementreplacedbythenextgreaterelementExamples:\>\>\>nextGreaterElement\(\[4,5,2,10,8\]\)\[5,10,10,\-1,\-1\]\>\>\>nextGreaterElement\(\[3,7,1,7,8,4,5\]\)\[7,8,7,8,\-1,5,\-1\]"""

Standard Response[⬇](data:text/plain;base64,ZGVmIG5leHRHcmVhdGVyRWxlbWVudChhcnIpOgogICAgaWYgbm90IGFycjoKICAgICAgICByZXR1cm4gW10KCiAgICBuID0gbGVuKGFycikKICAgIHJlcyA9IFstMV0gKiBuCiAgICBzdGFjayA9IFtdCgogICAgZm9yIGkgaW4gcmFuZ2UobiAtIDEsIC0xLCAtMSk6CiAgICAgICAgY3VycmVudCA9IGFycltpXQoKICAgICAgICB3aGlsZSBzdGFjayBhbmQgc3RhY2tbLTFdIDw9IGN1cnJlbnQ6CiAgICAgICAgICAgIHN0YWNrLnBvcCgpCgogICAgICAgIGlmIHN0YWNrOgogICAgICAgICAgICByZXNbaV0gPSBzdGFja1stMV0KCiAgICAgICAgc3RhY2suYXBwZW5kKGN1cnJlbnQpCgogICAgcmV0dXJuIHJlcw==)defnextGreaterElement\(arr\):ifnotarr:return\[\]n=len\(arr\)res=\[\-1\]\*nstack=\[\]foriinrange\(n\-1,\-1,\-1\):current=arr\[i\]whilestackandstack\[\-1\]<=current:stack\.pop\(\)ifstack:res\[i\]=stack\[\-1\]stack\.append\(current\)returnres

Model Sampling Rollout 1[⬇](data:text/plain;base64,ZGVmIG5leHRHcmVhdGVyRWxlbWVudChhcnIpOgogICAgc3RhY2sgPSBbXQogICAgcmVzdWx0ID0gWy0xXSAqIGxlbihhcnIpCgogICAgZm9yIGkgaW4gcmFuZ2UobGVuKGFycikpOgogICAgICAgIHdoaWxlIHN0YWNrIGFuZCBhcnJbaV0gPiBhcnJbc3RhY2tbLTFdXToKICAgICAgICAgICAgaiA9IHN0YWNrLnBvcCgpCiAgICAgICAgICAgIHJlc3VsdFtqXSA9IGFycltpXQoKICAgICAgICBzdGFjay5hcHBlbmQoaSkKCiAgICByZXR1cm4gcmVzdWx0)defnextGreaterElement\(arr\):stack=\[\]result=\[\-1\]\*len\(arr\)foriinrange\(len\(arr\)\):whilestackandarr\[i\]\>arr\[stack\[\-1\]\]:j=stack\.pop\(\)result\[j\]=arr\[i\]stack\.append\(i\)returnresult

Model Sampling Rollout 2[⬇](data:text/plain;base64,ZGVmIG5leHRHcmVhdGVyRWxlbWVudChhcnIpOgogICAgc3RhY2sgPSBbXQogICAgcmVzdWx0ID0gW10KCiAgICBmb3IgbnVtIGluIHJldmVyc2VkKGFycik6CiAgICAgICAgd2hpbGUgc3RhY2sgYW5kIHN0YWNrWy0xXSA8PSBudW06CiAgICAgICAgICAgIHN0YWNrLnBvcCgpCgogICAgICAgIGlmIHN0YWNrOgogICAgICAgICAgICByZXN1bHQuYXBwZW5kKHN0YWNrWy0xXSkKICAgICAgICBlbHNlOgogICAgICAgICAgICByZXN1bHQuYXBwZW5kKC0xKQoKICAgICAgICBzdGFjay5hcHBlbmQobnVtKQoKICAgIHJlc3VsdC5yZXZlcnNlKCkKICAgIHJldHVybiByZXN1bHQ=)defnextGreaterElement\(arr\):stack=\[\]result=\[\]fornuminreversed\(arr\):whilestackandstack\[\-1\]<=num:stack\.pop\(\)ifstack:result\.append\(stack\[\-1\]\)else:result\.append\(\-1\)stack\.append\(num\)result\.reverse\(\)returnresult

Model Sampling Rollout 3[⬇](data:text/plain;base64,ZGVmIG5leHRHcmVhdGVyRWxlbWVudChhcnIpOgogICAgc3RhY2sgPSBbXQogICAgcmVzdWx0ID0gW10KCiAgICBmb3IgbnVtIGluIHJldmVyc2VkKGFycik6CiAgICAgICAgd2hpbGUgc3RhY2sgYW5kIHN0YWNrWy0xXSA8PSBudW06CiAgICAgICAgICAgIHN0YWNrLnBvcCgpCgogICAgICAgIGlmIHN0YWNrOgogICAgICAgICAgICByZXN1bHQuYXBwZW5kKHN0YWNrWy0xXSkKICAgICAgICBlbHNlOgogICAgICAgICAgICByZXN1bHQuYXBwZW5kKC0xKQoKICAgICAgICBzdGFjay5hcHBlbmQobnVtKQoKICAgIHJlc3VsdC5yZXZlcnNlKCkKICAgIHJldHVybiByZXN1bHQ=)defnextGreaterElement\(arr\):stack=\[\]result=\[\]fornuminreversed\(arr\):whilestackandstack\[\-1\]<=num:stack\.pop\(\)ifstack:result\.append\(stack\[\-1\]\)else:result\.append\(\-1\)stack\.append\(num\)result\.reverse\(\)returnresult

### D\.4Case 4

Question[⬇](data:text/plain;base64,ZGVmIHBhaXJfcGFydGljaXBhbnRzKG4sIHNraWxsX2xldmVscyk6CiAgICAiIiIKICAgIFBhaXJzIHBhcnRpY2lwYW50cyBzdWNoIHRoYXQgdGhlIGRpZmZlcmVuY2UgaW4gdGhlaXIgc2tpbGwgbGV2ZWxzIGlzIG1pbmltaXplZC4KICAgIElmIHRoZSBudW1iZXIgb2YgcGFydGljaXBhbnRzIGlzIG9kZCwgb25lIHBhcnRpY2lwYW50IHdpbGwgYmUgbGVmdCB1bnBhaXJlZC4KCiAgICBQYXJhbWV0ZXJzOgogICAgbiAoaW50KTogTnVtYmVyIG9mIHBhcnRpY2lwYW50cy4KICAgIHNraWxsX2xldmVscyAobGlzdCk6IExpc3Qgb2Ygc2tpbGwgbGV2ZWxzIG9mIHBhcnRpY2lwYW50cyBpbiBub24tZGVjcmVhc2luZyBvcmRlci4KCiAgICBSZXR1cm5zOgogICAgbGlzdDogQSBsaXN0IG9mIHR1cGxlcyB3aGVyZSBlYWNoIHR1cGxlIHJlcHJlc2VudHMgYSBwYWlyIG9mIHBhcnRpY2lwYW50cy4KICAgIElmIHRoZXJlIGlzIGFuIG9kZCBudW1iZXIgb2YgcGFydGljaXBhbnRzLCB0aGUgbGFzdCBlbGVtZW50IHdpbGwgYmUgYSBzaW5nbGUgc2tpbGwgbGV2ZWwuCgogICAgRXhhbXBsZXM6CiAgICA+Pj4gcGFpcl9wYXJ0aWNpcGFudHMoNiwgWzEsIDIsIDMsIDUsIDksIDEyXSkKICAgIFsoMSwgMiksICgzLCA1KSwgKDksIDEyKV0KCiAgICA+Pj4gcGFpcl9wYXJ0aWNpcGFudHMoNSwgWzEsIDEsIDMsIDQsIDldKQogICAgWygxLCAxKSwgKDMsIDQpLCAoOSwpXQogICAgIiIiCiAgICAjIEltcGxlbWVudGF0aW9uIGhlcmU=)defpair\_participants\(n,skill\_levels\):"""Pairsparticipantssuchthatthedifferenceintheirskilllevelsisminimized\.Ifthenumberofparticipantsisodd,oneparticipantwillbeleftunpaired\.Parameters:n\(int\):Numberofparticipants\.skill\_levels\(list\):Listofskilllevelsofparticipantsinnon\-decreasingorder\.Returns:list:Alistoftupleswhereeachtuplerepresentsapairofparticipants\.Ifthereisanoddnumberofparticipants,thelastelementwillbeasingleskilllevel\.Examples:\>\>\>pair\_participants\(6,\[1,2,3,5,9,12\]\)\[\(1,2\),\(3,5\),\(9,12\)\]\>\>\>pair\_participants\(5,\[1,1,3,4,9\]\)\[\(1,1\),\(3,4\),\(9,\)\]"""

Standard Response[⬇](data:text/plain;base64,ZGVmIHBhaXJfcGFydGljaXBhbnRzKG4sIHNraWxsX2xldmVscyk6CiAgICBwYWlycyA9IFtdCgogICAgZm9yIGkgaW4gcmFuZ2UoMCwgbiwgMik6CiAgICAgICAgaWYgaSArIDEgPCBuOgogICAgICAgICAgICBwYWlycy5hcHBlbmQoKAogICAgICAgICAgICAgICAgc2tpbGxfbGV2ZWxzW2ldLAogICAgICAgICAgICAgICAgc2tpbGxfbGV2ZWxzW2kgKyAxXQogICAgICAgICAgICApKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHBhaXJzLmFwcGVuZCgoc2tpbGxfbGV2ZWxzW2ldLCkpCgogICAgcmV0dXJuIHBhaXJz)defpair\_participants\(n,skill\_levels\):pairs=\[\]foriinrange\(0,n,2\):ifi\+1<n:pairs\.append\(\(skill\_levels\[i\],skill\_levels\[i\+1\]\)\)else:pairs\.append\(\(skill\_levels\[i\],\)\)returnpairs

Model Sampling Rollout 1[⬇](data:text/plain;base64,ZGVmIHBhaXJfcGFydGljaXBhbnRzKG4sIHNraWxsX2xldmVscyk6CiAgICBpZiBuICUgMiA9PSAwOgogICAgICAgIHBhaXJzID0gW10KCiAgICAgICAgZm9yIGkgaW4gcmFuZ2UoMCwgbiwgMik6CiAgICAgICAgICAgIHBhaXJzLmFwcGVuZCgoCiAgICAgICAgICAgICAgICBza2lsbF9sZXZlbHNbaV0sCiAgICAgICAgICAgICAgICBza2lsbF9sZXZlbHNbaSArIDFdCiAgICAgICAgICAgICkpCgogICAgICAgIHJldHVybiBwYWlycwogICAgZWxzZToKICAgICAgICBwYWlycyA9IFtdCgogICAgICAgIGZvciBpIGluIHJhbmdlKDAsIG4gLSAxLCAyKToKICAgICAgICAgICAgcGFpcnMuYXBwZW5kKCgKICAgICAgICAgICAgICAgIHNraWxsX2xldmVsc1tpXSwKICAgICAgICAgICAgICAgIHNraWxsX2xldmVsc1tpICsgMV0KICAgICAgICAgICAgKSkKCiAgICAgICAgcGFpcnMuYXBwZW5kKHNraWxsX2xldmVsc1stMV0pCiAgICAgICAgcmV0dXJuIHBhaXJz)defpair\_participants\(n,skill\_levels\):ifn%2==0:pairs=\[\]foriinrange\(0,n,2\):pairs\.append\(\(skill\_levels\[i\],skill\_levels\[i\+1\]\)\)returnpairselse:pairs=\[\]foriinrange\(0,n\-1,2\):pairs\.append\(\(skill\_levels\[i\],skill\_levels\[i\+1\]\)\)pairs\.append\(skill\_levels\[\-1\]\)returnpairs

Model Sampling Rollout 2[⬇](data:text/plain;base64,ZGVmIHBhaXJfcGFydGljaXBhbnRzKG4sIHNraWxsX2xldmVscyk6CiAgICBpZiBuICUgMiA9PSAwOgogICAgICAgIHBhaXJzID0gW10KCiAgICAgICAgZm9yIGkgaW4gcmFuZ2UoMCwgbiwgMik6CiAgICAgICAgICAgIHBhaXJzLmFwcGVuZCgoCiAgICAgICAgICAgICAgICBza2lsbF9sZXZlbHNbaV0sCiAgICAgICAgICAgICAgICBza2lsbF9sZXZlbHNbaSArIDFdCiAgICAgICAgICAgICkpCgogICAgICAgIHJldHVybiBwYWlycwogICAgZWxzZToKICAgICAgICBwYWlycyA9IFtdCgogICAgICAgIGZvciBpIGluIHJhbmdlKDAsIG4gLSAxLCAyKToKICAgICAgICAgICAgcGFpcnMuYXBwZW5kKCgKICAgICAgICAgICAgICAgIHNraWxsX2xldmVsc1tpXSwKICAgICAgICAgICAgICAgIHNraWxsX2xldmVsc1tpICsgMV0KICAgICAgICAgICAgKSkKCiAgICAgICAgcGFpcnMuYXBwZW5kKHNraWxsX2xldmVsc1stMV0pCiAgICAgICAgcmV0dXJuIHBhaXJz)defpair\_participants\(n,skill\_levels\):ifn%2==0:pairs=\[\]foriinrange\(0,n,2\):pairs\.append\(\(skill\_levels\[i\],skill\_levels\[i\+1\]\)\)returnpairselse:pairs=\[\]foriinrange\(0,n\-1,2\):pairs\.append\(\(skill\_levels\[i\],skill\_levels\[i\+1\]\)\)pairs\.append\(skill\_levels\[\-1\]\)returnpairs

Model Sampling Rollout 3[⬇](data:text/plain;base64,ZGVmIHBhaXJfcGFydGljaXBhbnRzKG4sIHNraWxsX2xldmVscyk6CiAgICBpZiBuICUgMiA9PSAwOgogICAgICAgIHJldHVybiBsaXN0KHppcCgKICAgICAgICAgICAgc2tpbGxfbGV2ZWxzWzo6Ml0sCiAgICAgICAgICAgIHNraWxsX2xldmVsc1sxOjoyXQogICAgICAgICkpCiAgICBlbHNlOgogICAgICAgIHJldHVybiAoCiAgICAgICAgICAgIGxpc3QoemlwKAogICAgICAgICAgICAgICAgc2tpbGxfbGV2ZWxzWzo6Ml0sCiAgICAgICAgICAgICAgICBza2lsbF9sZXZlbHNbMTo6Ml0KICAgICAgICAgICAgKSkKICAgICAgICAgICAgKyBbKHNraWxsX2xldmVsc1stMV0sKV0KICAgICAgICAp)defpair\_participants\(n,skill\_levels\):ifn%2==0:returnlist\(zip\(skill\_levels\[::2\],skill\_levels\[1::2\]\)\)else:return\(list\(zip\(skill\_levels\[::2\],skill\_levels\[1::2\]\)\)\+\[\(skill\_levels\[\-1\],\)\]\)