Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes

# Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes
Source: [https://arxiv.org/html/2609.19156](https://arxiv.org/html/2609.19156)
Qirui Chen∗ Zhejiang University The University of Hong Kong 3220103318@zju\.edu\.cn &Renjie Pi∗ The Hong Kong University of Science and Technology rpi@connect\.ust\.hk Jiahui Gao† The University of Hong Kong ggaojiahui@gmail\.com &Lingpeng Kong The University of Hong Kong lpk@cs\.hku\.hk

###### Abstract

Data\-driven fine\-tuning is widely adopted to enhance reasoning in Large Language Models \(LLMs\) due to its simplicity and efficiency\. However, mainstream imitation learning methods that rely exclusively on perfect reasoning trajectories suffer from aScaling Collapse: when the problem set is limited, increasing positive examples fails to yield continuous improvement\. However, during inference, an LLM can not guarantee that every intermediate step is correct and is therefore prone to errors\. Once such errors arise, the LLM often struggles to recover and may be further misled by the accumulation of previous mistakes\. To address this, we proposeReflective Recovery, a simple yet effective self\-supervised approach that transforms failed reasoning attempts into recovery training data\. Specifically, we extract initial segments of failed trajectories, concatenate them with prompts, and use them to guide the LLM toward valid solutions\. Because these segments from failed trajectories are likely to contain errors, this process teaches models to recognize and correct mistakes during reasoning, enabling recovery from erroneous states without relying on external critics or reward models\. Evaluated on extensive benchmarks, Reflective Recovery significantly improves performance\. On DeepSeek\-R1\-Distill\-Qwen\-7B, it boosts accuracy from 30\.0% to 37\.5% on AIME 2025 and from 37\.6% to 47\.8% on Minerva\. More importantly, analyses demonstrate that it breaks the scaling collapse barrier and enables models to develop emergent self\-correction behaviors, representing a paradigm shift from outcome\-oriented memorization to process\-oriented reflective reasoning\.

Reflective Recovery: A Self\-Supervised Method for Reasoning by Learning from Mistakes

Qirui Chen∗Zhejiang UniversityThe University of Hong Kong3220103318@zju\.edu\.cnRenjie Pi∗The Hong Kong University ofScience and Technologyrpi@connect\.ust\.hk

Jiahui Gao†The University of Hong Kongggaojiahui@gmail\.comLingpeng KongThe University of Hong Konglpk@cs\.hku\.hk

††footnotetext:∗Equal contribution\.†Corresponding author\.## 1Introduction

Large Language Models \(LLMs\) have achieved remarkable progress across diverse natural language tasks, from question answering to code generation\(Brown et al\.,[2020](https://arxiv.org/html/2609.19156#bib.bib5); Grattafiori et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib11)\)\. Among these capabilities, complex reasoning stands out as a critical frontier for tackling multi\-step mathematical problems and scientific challenges\(OpenAI et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib27); DeepSeek\-AI et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib10); Anthropic,[2023](https://arxiv.org/html/2609.19156#bib.bib3)\)\. Consequently, enhancing the reasoning capacity of LLMs has gained attention from both academia and industry\.

Among various approaches to improving reasoning ability \(e\.g\., supervised fine\-tuning, reinforcement learning\), data\-driven fine\-tuning has emerged as the most prevalent solution due to its computational efficiency and simplicity in training\. Most recent progress\(Chen et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib8); Bu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib6); Gulcehre et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib12); Yang et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib39)\)follows an imitation learning paradigm, with Rejection Sampling Fine\-Tuning \(RFT\) being the most representative\. RFT collects high\-quality, correct reasoning trajectories from large\-scale sampling as positive examples for training\(Yuan et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib42)\)\. During the initial alignment phase, this imitation approach can achieve substantial performance improvements with a moderate amount of data\.

However, our experiments reveal a fundamental efficiency bottleneck in this positive\-only imitation learning approach\. When the problem set is limited, merely increasing the sampling volume to accumulate more positive examples does not continuously translate into improved model capabilities\. For instance, on the AIME 2025 benchmark, the RFT baseline’s performance actually degrades from 35\.3% to 30\.0% as the training data volume scales from 8k to 10k\. Our experiments show that as the scale of positive training data increases, performance gains rapidly diminish and eventually stagnate—a phenomenon we termscaling collapse\(Figure[2](https://arxiv.org/html/2609.19156#S4.F2)\)\. Despite being exposed to more correct paths, models still struggle with complex, multi\-step problems\. This exposes a fundamental limitation of imitation learning: it guides models to memorize correct reasoning trajectories rather than develop robust reasoning strategies that generalize beyond memorized patterns\.

The limitation of this approach becomes particularly significant when models deviate from the correct trajectory: since the training data never includes error states, once a model makes an incorrect reasoning step, it lacks any mechanism for self\-correction\. Effective reasoning should be a dynamic process that requires not only generating correct logic, but also identifying and correcting errors when they arise\. To address this, recent work has explored mechanisms that encourage recovery or self\-correction\. However, approaches relying on expensive annotations or additional models for external feedback\(Kumar et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib19); Wang et al\.,[2025b](https://arxiv.org/html/2609.19156#bib.bib35)\)suffer from limited scalability and autonomy, while methods prompting models for intrinsic reflection after complete failure\(Shinn et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib32)\)lack the capability to perform mid\-process correction during reasoning\.

In this work, we explore a simpler alternative: rather than introducing new critics or supervision signals, we investigate whether models can learn to recover directly from their own failures\. To this end, we introduceReflective Recovery, a self\-supervised framework that constructs recovery training data from failed reasoning attempts\. Specifically, given a set of negative reasoning trajectories \(𝒟neg\\mathcal\{D\}\_\{\\text\{neg\}\}\) that lead to incorrect answers, we extract their error prefixes—partial reasoning chains\. We then pair each error prefix with a correct continuation that leads to the right solution\. This process yields a reflective dataset \(𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\) where each training instance consists of an error state and its corresponding recovery path\. By training on such data, models learn to identify and correct reasoning errors, moving from incorrect intermediate states to correct solutions\.

Despite its simplicity, Reflective Recovery does not require external reward models or reinforcement learning, making it highly practical and scalable\. Experiments on challenging benchmarks such as AIME and Minerva show that our method outperforms outcome\-based RFT baselines\. More importantly, it exhibits stable and robust scaling behavior, avoiding the performance saturation observed in standard imitation learning \(Figure[2](https://arxiv.org/html/2609.19156#S4.F2)\)\. Specifically, while the RFT baseline suffers a performance drop on AIME 2025 from 35\.3% to 30\.0% as training data scales from 8k to 10k, our method maintains a steady improvement on benchmarks like Minerva, demonstrating generalization ability\.

The main contributions of this work are:

- •We identify and characterize thescaling collapsephenomenon in imitation learning, demonstrating that its root cause lies in the absence of error recovery capability\.
- •We introduce Reflective Recovery, a simple yet effective self\-supervised framework that transforms negative reasoning trajectories into high\-value training data for learning recovery behavior\.
- •Extensive experiments demonstrate that our method significantly outperforms RFT across multiple challenging benchmarks\. Critically, unlike RFT which suffers from performance saturation, Reflective Recovery maintains robust linear scaling behavior with increasing data scale \(Figure[2](https://arxiv.org/html/2609.19156#S4.F2)\)\.
- •Through attribution analysis, we demonstrate that our method elicits dynamic reflective reasoning patterns in models—actively detecting errors and correcting reasoning paths during the inference process \(Section[4\.5](https://arxiv.org/html/2609.19156#S4.SS5)\)\.

## 2Related Work

##### External Feedback\.

A direction to enhance LLM reasoning is guiding the generation process using external supervision signals\. This typically involves the use of additional components or signals to align model outputs with correctness criteria, such as reward models or verifiers\(Kumar et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib19); Akyürek et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib1); Zhang et al\.,[2025a](https://arxiv.org/html/2609.19156#bib.bib44); Wang et al\.,[2025b](https://arxiv.org/html/2609.19156#bib.bib35); Jiang et al\.,[2024b](https://arxiv.org/html/2609.19156#bib.bib18); Bensal et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib4); Zhu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib47); An et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib2)\)\. Such feedback can be applied to evaluate reasoning steps or answers\. By leveraging these external critics, models can reject incorrect paths or be fine\-tuned by reinforcement learning to maximize rewards\.

Recent advances has moved towards providing feedback at each step of the reasoning process rather than only on the final answer\. Process Reward Models \(PRMs\) offer supervision on intermediate steps to guide the model toward correct reasoning\(Lin et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib25); Bensal et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib4); Li et al\.,[2025a](https://arxiv.org/html/2609.19156#bib.bib23)\)\. This step\-level feedback also supports the automatic generation of reasoning data, where methods like SRA\-MCTS and GFlowNet use simple verifiers to explore different reasoning paths and collect higher\-quality training examples\(Xu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib38); Bu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib6)\)\. However, these approaches still rely on carefully designed critics or large amounts of human preference data\. The need for strong external supervision and costly training procedures limits their scalability\(Zhang et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib46),[2025b](https://arxiv.org/html/2609.19156#bib.bib45)\), motivating the development for methods that can operate with minimal external guidance\.

##### Self\-Correction and Learning from Errors\.

Parallel to external supervision, intrinsic self\-correction focuses on the model’s ability to refine its own outputs\. Early inference\-time approaches such as Reflexion\(Shinn et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib32)\)prompt models to verbally critique their answers\. However, prior studies show that LLMs often have difficulty recognizing their own mistakes, particularly in multi\-step or complex reasoning tasks\(Jiang et al\.,[2024a](https://arxiv.org/html/2609.19156#bib.bib17)\), and self\-correction can even reduce performance on more challenging benchmarks\(Tie et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib33); Lee et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib21)\)\.

To overcome these limitations, recent work has shifted toward training\-based correction using negative signals\. Methods like SuperCorrect and RISE fine\-tune models on iterative refinement traces\(Yang et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib39); Qu et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib29); Wang et al\.,[2025a](https://arxiv.org/html/2609.19156#bib.bib34)\)\. Other approaches train models to avoid incorrect behaviors or learn from corrected reasoning paths, which has been shown to improve robustness\(Pi et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib28); Chen et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib8); Zhu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib47); Yu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib41); Li et al\.,[2025b](https://arxiv.org/html/2609.19156#bib.bib24); Bui et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib7); Liu et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib26); Xiong et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib37); Zelikman et al\.,[2022](https://arxiv.org/html/2609.19156#bib.bib43); Shao et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib31)\)\. Despite these advances, most approaches remain post\-hoc or outcome\-focused\. It means they intervene only after an error has fully occurred or simply discourage incorrect outputs\. As a result, they do not enable models to actively detect and recover from mistakes during the reasoning process itself\.

## 3Methodology

Our approachReflective Recoveryis based on the idea that robust reasoning requires not only learning from correct solutions, but also learning how to recover from intermediate errors\. Instead of discarding failed reasoning paths, we argue that these failures contain rich information and view them as valuable training signals\. Building on this insight, we introduce a self\-supervised Sample\-Resample paradigm that leverages the model’s own failure trajectories as a learning resource to explicitly teach the model how to correct itself\.

As illustrated in Figure[1](https://arxiv.org/html/2609.19156#S3.F1), our method consists of two phases: \(1\) extract diverse reasoning trajectories to identify failure trajectories, preserving them as potential recovery opportunities, and \(2\) transforming failed trajectories into successful, self\-corrected trajectories to build a recovery\-focused training dataset𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\.

### 3\.1Failure Trajectory Collection

To enable recovery learning, we first collect a dataset of the model’s mistakes\. Letℳ\\mathcal\{M\}denote the base language model and𝒳\\mathcal\{X\}be a set of seed problems\. In the exploratory phase, for each problemx∈𝒳x\\in\\mathcal\{X\}, we generateNNreasoning trajectories\{y1,y2,…,yN\}\\\{y\_\{1\},y\_\{2\},\\dots,y\_\{N\}\\\}using sampling decoding\. By using a high temperature and leveraging the model’s uncertainty, we collect failed trajectories that contain diverse errors\.

Next, we employ a binary outcome verifier𝒱​\(x,y\)∈\{0,1\}\\mathcal\{V\}\(x,y\)\\in\\\{0,1\\\}to assess the correctness of each trajectory\. Based on the verification result, we partition trajectories into a set of correct solutions𝒟pos=\{\(x,y\)∣𝒱​\(x,y\)=1\}\\mathcal\{D\}\_\{\\text\{pos\}\}=\\\{\(x,y\)\\mid\\mathcal\{V\}\(x,y\)=1\\\}, and a set of failed trajectories𝒟neg=\{\(x,y\)∣𝒱​\(x,y\)=0\}\\mathcal\{D\}\_\{\\text\{neg\}\}=\\\{\(x,y\)\\mid\\mathcal\{V\}\(x,y\)=0\\\}\.

Unlike standard outcome\-based methods, which typically discard𝒟neg\\mathcal\{D\}\_\{\\text\{neg\}\}, we keep these failure trajectories because they reveal the model’s specific error patterns\. By studying how the model fails, we can explicitly teach it how to recover from mistakes\.

![Refer to caption](https://arxiv.org/html/2609.19156v1/x1.png)Figure 1:The Reflective Recovery pipeline\.We proceed in two phases: \(1\)Failure Trajectory Collection\. We apply diverse sampling to extract failure trajectories \(𝒟neg\\mathcal\{D\}\_\{\\text\{neg\}\}\); \(2\)Recovery Curation\. We truncate these errors at intermediate stepttand resampling to discover valid recovery paths\. The resulting dataset \(𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\) contains self\-corrected trajectories that enable the model to learn recovery from its own mistake\.
### 3\.2Recovery Curation

The core idea of Reflective Recovery is the converting failed trajectories𝒟neg\\mathcal\{D\}\_\{\\text\{neg\}\}into a dataset of successful recovery trajectories𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\. We achieve this throughTruncate\-Resample\-Selectprocedure\.

##### Truncation\.

For each failed trajectoryy∈𝒟negy\\in\\mathcal\{D\}\_\{\\text\{neg\}\}, wherey=\(y1,…,yL\)y=\(y\_\{1\},\\dots,y\_\{L\}\), we cut it off at a intermediate stept<Lt<L\. This yields a partial prefixy<t=\(y1,…,yt−1\)y\_\{<t\}=\(y\_\{1\},\\dots,y\_\{t\-1\}\)\. This prefix represents a state where the model might have made mistakes\. By stopping here, we create a situation where the model needs to decide how to continue\. The selection ofttis a hyperparameter controlling the difficulty of recovery \(analyzed in Section[4\.4\.1](https://arxiv.org/html/2609.19156#S4.SS4.SSS1)\)\.

##### Resampling for Recovery\.

Given the prefix\(x,y<t\)\(x,y\_\{<t\}\), we sampleKKalternative continuations\{y~1,…,y~K\}\\\{\\tilde\{y\}\_\{1\},\\dots,\\tilde\{y\}\_\{K\}\\\}from the model using temperature\-based sampling\. This resampling process explores diverse continuation paths from the same intermediate state, yielding trajectories that may either reach the correct solution or persist in error\. The diversity of these samples enables the model to discover multiple recovery strategies from erroneous states\.

##### Verification and Selection\.

Finally, we verify each completionycomp=y<t⊕y~ky\_\{\\text\{comp\}\}=y\_\{<t\}\\oplus\\tilde\{y\}\_\{k\}\(where⊕\\oplusdenotes sequence concatenation\) using the verifier𝒱\\mathcal\{V\}\. Only trajectories that successfully reach correct solutions are retained to form the recovery dataset\.

The resulting recovery dataset𝒟rec\\mathcal\{D\}\_\{\\mathrm\{rec\}\}is formally defined as:

𝒟rec=\{\(x,y<t⊕y~k\)∣𝒱​\(x,y<t⊕y~k\)=1\}\\mathcal\{D\}\_\{\\mathrm\{rec\}\}=\\\{\\,\(x,y\_\{<t\}\\oplus\\tilde\{y\}\_\{k\}\)\\;\\mid\\;\\mathcal\{V\}\(x,y\_\{<t\}\\oplus\\tilde\{y\}\_\{k\}\)=1\\,\\\}Unlike regular datasets that contain only complete correct solutions,𝒟rec\\mathcal\{D\}\_\{\\mathrm\{rec\}\}consists of trajectories that transition from intermediate erroneous states to correct final answers, explicitly capturing the recovery process\.

### 3\.3Training Objective

We train the model on the recovery dataset𝒟rec\\mathcal\{D\}\_\{\\mathrm\{rec\}\}to explicitly teach it how to recover from intermediate errors\. We employ a specialized supervised fine\-tuning objective that focuses exclusively on the recovery completions\. For each recovery trajectory\(x,yrec\)∈𝒟rec\(x,y\_\{\\text\{rec\}\}\)\\in\\mathcal\{D\}\_\{\\mathrm\{rec\}\}, whereyrec=y<t⊕y~ky\_\{\\text\{rec\}\}=y\_\{<t\}\\oplus\\tilde\{y\}\_\{k\}, we treat the problem descriptionxxand the erroneous prefixy<ty\_\{<t\}as fixed context\. Gradients are only computed for resampled continuationy~\\tilde\{y\}\. The training objective minimizes the negative log\-likelihood over the tokens iny~\\tilde\{y\}\. Formally, for a completiony~\\tilde\{y\}consisting of tokens\(y~1,…,y~m\)\(\\tilde\{y\}\_\{1\},\\dots,\\tilde\{y\}\_\{m\}\), the loss is defined as:

ℒ​\(θ\)=−∑j=1mlog⁡Pθ​\(y~j∣x,y<t,y~<j\)\\mathcal\{L\}\(\\theta\)=\-\\sum\_\{j=1\}^\{m\}\\log P\_\{\\theta\}\(\\tilde\{y\}\_\{j\}\\mid x,y\_\{<t\},\\tilde\{y\}\_\{<j\}\)
By masking the loss for the prefixy<ty\_\{<t\}, the model avoids reinforcing its prior mistakes\. Instead, it learns to generate the correct continuationy~\\tilde\{y\}conditional on erroneous trajectory\. This training objective enables the model to not only learn correct solutions but also develop the ability to recognize, adapt, and recover from its own mistakes\.

## 4Experiments

### 4\.1Setup

##### Models and Data\.

Our experiments utilize the DeepSeek\-R1\-Distill\-Qwen family\(DeepSeek\-AI et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib10)\), specifically the 7B and 14B variants, to evaluate performance scalability\. For the seed prompts, we employ 3k high\-quality mathematical problems sourced from the SFT Stage 2 dataset of Light\-R1\(Wen et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib36)\)\. These problems are collected from public datasets and have been carefully filtered to remove overlap with evaluation benchmarks such as AIME and GPQA Diamond\. This dataset focuses on high\-difficulty questions, and each one needs careful thinking\.

##### Baselines\.

We benchmark our approach against Rejection Sampling Fine\-Tuning \(RFT\)\(Yuan et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib42)\)\. RFT represents the standard outcome\-based supervision paradigm, where the model is fine\-tuned exclusively on𝒟pos\\mathcal\{D\}\_\{\\text\{pos\}\}, which are trajectories yielding the correct final answer\. In contrast, our method employs process\-based recovery supervision, training on𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}derived from corrected failures\. To ensure a fair comparison, we align the training data volume for both methods in all experiments, ensuring\|𝒟pos\|=\|𝒟rec\|\|\\mathcal\{D\}\_\{\\text\{pos\}\}\|=\|\\mathcal\{D\}\_\{\\text\{rec\}\}\|\.

Table 1:Main experimental results \(pass@1accuracy\) on DeepSeek\-R1\-Distill\-Qwen 7B and 14B models\. All RFT and Ours models are trained on same number of trajectories\. The best result in each column is highlighted inbold\.MethodAIME 2024AIME 2025LiveCodeBenchGPQALiveCodeBench V4OlympiadBenchMinervaDeepSeek\-R1\-Distill\-Qwen\-7B53\.030\.034\.449\.138\.456\.437\.6\+ RFT54\.334\.735\.049\.740\.557\.146\.7\+ Reflective Recovery56\.737\.536\.550\.341\.258\.147\.8DeepSeek\-R1\-Distill\-Qwen\-14B66\.640\.048\.357\.551\.559\.043\.8\+ RFT69\.443\.353\.159\.453\.559\.147\.4\+ Reflective Recovery69\.745\.355\.360\.354\.259\.450\.7

Table 2:Ablation study on the data truncation position for the DeepSeek\-R1\-Distill\-Qwen 7B model\.LLdenotes the total length of the failed trajectory\. The best results arebold\.MethodAIME 2024AIME 2025LiveCodeBenchGPQALiveCodeBench V4OlympiadBenchMinervaDeepSeek\-R1\-Distill\-Qwen\-7B53\.030\.034\.449\.138\.456\.437\.6\+ RFT54\.334\.735\.049\.740\.557\.146\.7\+ Reflective Recovery \(t=L/2t=L/2\)56\.737\.536\.550\.341\.258\.147\.8\+ Reflective Recovery \(t=3​L/4t=3L/4\)60\.035\.835\.849\.038\.557\.147\.8

##### Implementation Details\.

The data curation process involves generatingN=8N=8initial trajectories per problem using the base model with a temperature of0\.60\.6, a top\-p value of0\.950\.95, and a maximum of 32,768 new tokens\. We employ a rule\-based verifier𝒱​\(x,y\)∈\{0,1\}\\mathcal\{V\}\(x,y\)\\in\\\{0,1\\\}to check correctness against the ground truth\. The verifier extracts answers from\\boxed\{\}expressions and compares them against ground truth using numerical tolerance \(rel\_tol=10−5\\text\{rel\\\_tol\}=10^\{\-5\}\) for floating\-point values, supporting various mathematical expressions including fractions \(\\frac\), square roots \(\\sqrt\), and standard numerical formats\. Failed trajectories are processed via our recovery re\-sampling step \(K=5K=5\) to construct𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\. For the main results, we fix the dataset size at 6k trajectories for both RFT and our method\. All models are fine\-tuned for 5 epochs using the Open\-R1 framework\(Hugging Face,[2025](https://arxiv.org/html/2609.19156#bib.bib15)\)with a learning rate of4\.0×10−54\.0\\times 10^\{\-5\}, a cosine schedule with warmup, and a maximum sequence length of 32,768 tokens\. We use Flash Attention 2\(Dao,[2024](https://arxiv.org/html/2609.19156#bib.bib9)\)for efficient training\. Inference is accelerated using vLLM\(Kwon et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib20)\)\. Complete hyperparameters are provided in Appendix[A](https://arxiv.org/html/2609.19156#A1)\.

##### Evaluation\.

We evaluate all models on a diverse suite of challenging reasoning benchmarks, reporting zero\-shot pass@1 accuracy using the lighteval framework\(Habib et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib13)\)\. The benchmarks include AIME\(Ye et al\.,[2025](https://arxiv.org/html/2609.19156#bib.bib40)\), GPQA\(Rein et al\.,[2023](https://arxiv.org/html/2609.19156#bib.bib30)\), LiveCodeBench\(Jain et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib16)\), OlympiadBench\(He et al\.,[2024](https://arxiv.org/html/2609.19156#bib.bib14)\), and Minerva Math\(Lewkowycz et al\.,[2022](https://arxiv.org/html/2609.19156#bib.bib22)\)\.

To ensure robust evaluation, we sample 32 times per problem for AIME 2024 and AIME 2025, 16 times for LiveCodeBench, 8 times for GPQA Diamond and Minerva Math, and 1 time for OlympiadBench\. This diverse suite aims to comprehensively assess the models’ generalization capabilities in complex reasoning tasks\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.19156#S4.T1)presents the comparative performance of our method and RFT baseline\. Our Reflective Recovery approach consistently surpasses the RFT baseline across complex reasoning tasks at both the 7B and 14B scales\. For example, on the DeepSeek\-R1\-Distill\-Qwen 7B model, our method reaches 37\.5% on AIME 2025, improving upon RFT by 2\.8 points\. We observe a similar improvement on Minerva, where performance increases from 46\.7% to 47\.8%\. This advantage carries over to the 14B model as well\. On Minerva Math benchmark, the improvement is particularly clear, with accuracy rising from 47\.4% to 50\.7%\.

Notably, despite training exclusively on mathematical reasoning problems, our method also yields a consistent gain of 2\.2 points on the code benchmark LiveCodeBench, demonstrating effective cross\-domain transfer\. These results suggest that learning to recover from errors is a more effective training strategy than simply imitating correct solutions\. By teaching the model to identify and rectify flaws in its reasoning chain, our method elicits an intrinsic self\-correction capability that generalizes across different domains\.

![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/gpqa_results.png)\(a\)GPQA
![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/image2.jpg)\(b\)AIME 2025
![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/m.jpg)\(c\)Minerva
![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/image6.png)\(d\)OlympiadBench

Figure 2:Scaling behavior of Pass@1 accuracy for our method versus RFT as data volume increases\. Our method exhibits robust scaling, whereas RFT shows signs of saturation or collapse\.
### 4\.3Scaling Law Analysis

To investigate the relationship between data volume and reasoning capability, we conducted a comparative study in which both the RFT baseline and Reflective Recovery were trained on datasets ranging from 2k to 10k trajectories\. The results in Figure[2](https://arxiv.org/html/2609.19156#S4.F2)reveal a distinct contrast in scaling behaviors between the two approaches\.

Our method shows a clear improvement and consistent scaling across most reasoning benchmarks as the amount of training data increases from 2k to 10k\. For example, our method achieves a steady performance gain on Minerva, suggesting the model effectively internalizes the recovery mechanism\. This indicates that our method is not just memorizing correct answers, but is learning a general ability to fix its own mistakes\.

In contrast, the RFT baseline shows more fragile scaling behavior\. On most benchmarks, its performance reaches a peak at intermediate data sizes, such as 4k or 8k examples, but then declines when the data volume increases to 10k\. For instance, accuracy on AIME 2025 drops from 35\.3% to 30\.0% as the training data volume scales from 8k to 10k, indicating adding more data actually impairs the model’s reasoning ability\. Without signal provided by error recovery, RFT appears to struggle to generalize when the data distribution expands, potentially overfitting to specific solution patterns rather than learning underlying reasoning principles\. Additional scaling behavior results on AIME 2024 and LiveCodeBench V4 are provided in Appendix[C](https://arxiv.org/html/2609.19156#A3)\.

### 4\.4Ablation Studies

#### 4\.4\.1Impact of Truncation Point

We examine the effect of the truncation point in the failed trajectories during data curation\. We compare truncating at 1/2 versus 3/4 of the Chain\-of\-Thought \(CoT\) length on DeepSeek\-R1\-Distill\-Qwen\-7B \(Table[2](https://arxiv.org/html/2609.19156#S4.T2)\)\. We observe that the 1/2 truncation generally performs better\. We posit that earlier truncation \(1/2\) creates a more effective learning signal by forcing the model to address foundational errors in the reasoning setup\. Conversely, truncating the reasoning process too late often leaves the model with a fundamentally flawed reasoning path, making it difficult to recover and introducing noise into the training signal\.

Table 3:Ablation study on recursive improvement\. We compare models trained with a mix of Iteration 1 & 2 data versus models trained solely on Iteration 1 data\. The best results in each column are highlighted inbold\.MethodAIME24AIME25LCBGPQALCB\_V4Reflective Recovery of 8k DataIter\. 1 & 257\.837\.736\.951\.341\.6\- Iter\. 256\.740\.035\.651\.540\.1Reflective Recovery of 10k DataIter\. 1 & 260\.043\.336\.152\.539\.9\- Iter\. 252\.933\.335\.051\.738\.9

![Refer to caption](https://arxiv.org/html/2609.19156v1/x2.png)Figure 3:Frequency and distribution of "Reflective Language Patterns" generated by our model versus the RFT baseline\. Our model demonstrates a notable increase\.
#### 4\.4\.2Recursive Improvement via Iterative Curation

A central question in self\-evolving systems is whether the data generation process can be applied recursively to synthesize high\-quality training signals\. To investigate this, we generated a second iteration of data \("Iter\. 2"\) by applying our recovery resampling process to instances that failed during the initial recovery attempt\.

As shown in Table[3](https://arxiv.org/html/2609.19156#S4.T3), comparing models trained with and without this second\-iteration data reveals a scale\-dependent effect\. At the 8k data level, the inclusion of Iteration 2 data yields mixed results, implying that at lower data volumes, the model benefits more from consolidating foundational recovery skills \(Iter\. 1\)\. However, at the 10k scale, the benefit of recursive data becomes clear, leading to substantial improvements across benchmarks, with gains of 7\.1 points on AIME 2024 and 10\.0 points on AIME 2025\. These results suggest that recursive curation serves as a natural curriculum: as the model scale and data volume increase, the inclusion of more challenging recovery trajectories helps iterative self\-improvement\.

#### 4\.4\.3Combining RFT and Reflective Recovery

It is natural to ask whether outcome\-based supervision and process\-based recovery supervision can complement each other\. RFT focuses on reinforcing correct reasoning trajectories from the start, while Reflective Recovery emphasizes how to recover from intermediate errors\. Since they supervise different stages of reasoning, combining them may provide additional benefits\.

To evaluate this possibility, we create a hybrid training set with 4k trajectories from RFT and 6k trajectories from Reflective Recovery, totaling 10k trajectories\. We compare this hybrid model with the RFT\-only and Reflective Recovery\-only models, keeping the total data volume the same\.

The results are shown in Table[4](https://arxiv.org/html/2609.19156#S4.T4)\. Across most datasets, the hybrid model achieves higher performance than both the RFT\-only model and the Reflective Recovery\-only model\. For example, it achieves 62\.0% accuracy on AIME 2024 and 37\.3% on LiveCodeBench\. These results indicate that combining correct reasoning demonstrations with recovery\-focused supervision leads to stronger performance than either approach alone\. Outcome\-based supervision helps stabilize reasoning trajectories, while recovery supervision improves robustness when errors occur\.

Table 4:Synergy analysis\. Performance of combining RFT data with our Reflective Recovery data\. All models are trained on a total of 10k trajectories\.MethodAIME24AIME25LCBGPQALCB\_V4RFT52\.230\.034\.948\.539\.3Reflective Recovery60\.043\.336\.152\.539\.9RFT \+ RR†62\.043\.537\.352\.041\.6†RR denotes Reflective Recovery\.

### 4\.5Analysis of Reflective Language Patterns

To determine whether the performance gains arise from genuine process resilience rather than template memorization, we examined the generated reasoning traces for metacognitive markers, meaning phrases that indicate self\-monitoring, doubt, or course correction\. As shown in Figure[3](https://arxiv.org/html/2609.19156#S4.F3), our model produces these reflective phrases more frequently and diversely than the RFT baseline\.

![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/detailed_reflective_term_comparison_AIME2024.png)Figure 4:Detailed frequency comparison of reflective language patterns on AIME 2024 training on 8k data\. The results show that our model more frequently uses phrases or language to admit mistakes and revise its reasoning trajectory, demonstrating a greater capacity for dynamic reflection and recovery\.To observe this phenomenon in detail, we provide a frequency comparison on the AIME 2024 trained on 8k data in Figure[4](https://arxiv.org/html/2609.19156#S4.F4)\. Both models use common verification phrases such as "let me verify" and "let me double\-check" at similarly high frequencies, but there is a clear gap in their use of substantial correction phrases\. Specifically, our model explicitly stated "I was wrong" 88 times, compared to 60 in the baseline\. It also used words like "reconsider" and "rethink" twice as often\. For instance, "reconsider" appeared 66 times versus 29\.

This behavior suggests that Reflective Recovery can actively monitor its reasoning process and recover from errors, indicating a more process\-oriented reasoning capability\.

## 5Conclusion

In this paper, we introduced Reflective Recovery, a self\-supervised data pipeline designed to teach language models how to fix their own mistakes during inference\. Our core contribution is the demonstration that learning to recover from errors is a more reliable and scalable training strategy than simply imitating correct answers\.

Our experiments on the DeepSeek\-R1\-Distill\-Qwen 7B and 14B models show that this method outperforms outcome\-base method Rejection Sampling Fine\-Tuning \(RFT\) on challenging reasoning benchmarks\. Moreover, our data scaling analysis shows that while RFT struggles or even fails with larger data volumes, our method continues to improve\. Furthermore, we also observe that Reflective Recovery encourages the model to reason more actively and reflectively, exhibiting more patterns like “reconsider” or “I need to correct myself” rather than just copying answers\. These findings suggest an important insight that robust reasoning is not just about avoiding errors, but about knowing how to recover from them\.

## 6Limitations and Future Work

Despite its promising results, our method faces limitations regarding computational cost and domain applicability\. Creating the training data requires generating many initial failed trajectories and resampling them for recovery\. Future work could explore ways to make the process more efficient, for example, by using curriculum learning to gradually introduce harder recovery tasks\. We also think it would be useful to balance process\-focused learning with a wider variety of data to avoid overfitting to specific domains\. Finally, this recovery\-oriented approach could be applied to other tasks that rely on verifiers, such as complex program synthesis or formal theorem proving\.

## References

- Akyürek et al\. \(2023\)Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon\. 2023\.[Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs](http://arxiv.org/abs/2305.08844)\.
- An et al\. \(2024\)Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian\-Guang Lou, and Weizhu Chen\. 2024\.[Learning from mistakes makes llm better reasoner](http://arxiv.org/abs/2310.20689)\.
- Anthropic \(2023\)Anthropic\. 2023\.[Introducing claude](https://www.anthropic.com/index/introducing-claude)\.
- Bensal et al\. \(2025\)Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mozolevskyi, Muayad Ali, and Waseem AlShikh\. 2025\.[Reflect, retry, reward: Self\-improving llms via reinforcement learning](http://arxiv.org/abs/2505.24726)\.
- Brown et al\. \(2020\)Tom B\. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert\-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M\. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei\. 2020\.[Language models are few\-shot learners](http://arxiv.org/abs/2005.14165)\.
- Bu et al\. \(2025\)Tianpeng Bu, Minying Zhang, Hongtao Duan, Shurui Li, Lulu Hu, and Yu Li\. 2025\.[Enhanced data synthesis for LLM through reasoning structures generated by hierarchical GFlowNet](https://doi.org/10.18653/v1/2025.findings-acl.821)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 15931–15958, Vienna, Austria\. Association for Computational Linguistics\.
- Bui et al\. \(2025\)Nhat\-Tan Bui, Dinh\-Hieu Hoang, Quoc\-Huy Trinh, Minh\-Triet Tran, Truong Nguyen, and Susan Gauch\. 2025\.[Nein: Telling what you don’t want](http://arxiv.org/abs/2409.06481)\.
- Chen et al\. \(2025\)Guanzheng Chen, Xin Li, Michael Qizhe Shieh, and Lidong Bing\. 2025\.[Longpo: Long context self\-evolution of large language models through short\-to\-long preference optimization](http://arxiv.org/abs/2502.13922)\.
- Dao \(2024\)Tri Dao\. 2024\.FlashAttention\-2: Faster attention with better parallelism and work partitioning\.In*International Conference on Learning Representations \(ICLR\)*\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H\. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J\. L\. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R\. J\. Chen, R\. L\. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S\. S\. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T\. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W\. L\. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X\. Q\. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y\. X\. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z\. Z\. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang\. 2025\.[Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning](http://arxiv.org/abs/2501.12948)\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia\-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El\-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala\-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching\-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric\-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina\-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean\-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L\. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma\. 2024\.[The llama 3 herd of models](http://arxiv.org/abs/2407.21783)\.
- Gulcehre et al\. \(2023\)Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas\. 2023\.[Reinforced self\-training \(rest\) for language modeling](http://arxiv.org/abs/2308.08998)\.
- Habib et al\. \(2023\)Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall\. 2023\.[Lighteval: A lightweight framework for llm evaluation](https://github.com/huggingface/lighteval)\.
- He et al\. \(2024\)Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun\. 2024\.[Olympiadbench: A challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems](http://arxiv.org/abs/2402.14008)\.
- Hugging Face \(2025\)Hugging Face\. 2025\.[Open r1: A fully open reproduction of deepseek\-r1](https://github.com/huggingface/open-r1)\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2024\.[Livecodebench: Holistic and contamination free evaluation of large language models for code](http://arxiv.org/abs/2403.07974)\.
- Jiang et al\. \(2024a\)Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi\. 2024a\.[Self\-\[in\]correct: Llms struggle with discriminating self\-generated responses](http://arxiv.org/abs/2404.04298)\.
- Jiang et al\. \(2024b\)Huchen Jiang, Yangyang Ma, Chaofan Ding, Kexin Luan, and Xinhan Di\. 2024b\.[Towards intrinsic self\-correction enhancement in monte carlo tree search boosted reasoning via iterative preference learning](http://arxiv.org/abs/2412.17397)\.
- Kumar et al\. \(2024\)Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co\-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust\. 2024\.[Training language models to self\-correct via reinforcement learning](http://arxiv.org/abs/2409.12917)\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*\.
- Lee et al\. \(2024\)Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo\. 2024\.[Volcano: Mitigating multimodal hallucination through self\-feedback guided revision](http://arxiv.org/abs/2311.07362)\.
- Lewkowycz et al\. \(2022\)Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman\-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur\-Ari, and Vedant Misra\. 2022\.[Solving quantitative reasoning problems with language models](http://arxiv.org/abs/2206.14858)\.
- Li et al\. \(2025a\)Jiaqi Li, Xinyi Dong, Yang Liu, Zhizhuo Yang, Quansen Wang, Xiaobo Wang, SongChun Zhu, Zixia Jia, and Zilong Zheng\. 2025a\.[Reflectevo: Improving meta introspection of small llms by learning self\-reflection](http://arxiv.org/abs/2505.16475)\.
- Li et al\. \(2025b\)Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding\. 2025b\.[Enhancing reasoning through process supervision with monte carlo tree search](http://arxiv.org/abs/2501.01478)\.
- Lin et al\. \(2025\)Yen\-Ting Lin, Di Jin, Tengyu Xu, Tianhao Wu, Sainbayar Sukhbaatar, Chen Zhu, Yun He, Yun\-Nung Chen, Jason Weston, Yuandong Tian, Arash Rahnama, Sinong Wang, Hao Ma, and Han Fang\. 2025\.[Step\-kto: Optimizing mathematical reasoning through stepwise binary feedback](http://arxiv.org/abs/2501.10799)\.
- Liu et al\. \(2025\)Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh\. 2025\.[Noisyrollout: Reinforcing visual reasoning with data augmentation](http://arxiv.org/abs/2504.13055)\.
- OpenAI et al\. \(2024\)OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El\-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu\-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y\. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li\. 2024\.[Openai o1 system card](http://arxiv.org/abs/2412.16720)\.
- Pi et al\. \(2024\)Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang\. 2024\.[Strengthening multimodal large language model with bootstrapped preference optimization](http://arxiv.org/abs/2403.08730)\.
- Qu et al\. \(2024\)Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar\. 2024\.[Recursive introspection: Teaching language model agents how to self\-improve](http://arxiv.org/abs/2407.18219)\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. 2023\.[Gpqa: A graduate\-level google\-proof q&a benchmark](http://arxiv.org/abs/2311.12022)\.
- Shao et al\. \(2025\)Zhihong Shao, Yuxiang Luo, Chengda Lu, Z\. Z\. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang\. 2025\.[Deepseekmath\-v2: Towards self\-verifiable mathematical reasoning](http://arxiv.org/abs/2511.22570)\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\. 2023\.[Reflexion: Language agents with verbal reinforcement learning](http://arxiv.org/abs/2303.11366)\.
- Tie et al\. \(2025\)Guiyao Tie, Zenghui Yuan, Zeli Zhao, Chaoran Hu, Tianhe Gu, Ruihang Zhang, Sizhe Zhang, Junran Wu, Xiaoyue Tu, Ming Jin, Qingsong Wen, Lixing Chen, Pan Zhou, and Lichao Sun\. 2025\.[Can llms correct themselves? a benchmark of self\-correction in llms](http://arxiv.org/abs/2510.16062)\.
- Wang et al\. \(2025a\)Qibin Wang, Pu Zhao, Shaohan Huang, Fangkai Yang, Lu Wang, Furu Wei, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang\. 2025a\.[Learning to refine: Self\-refinement of parallel reasoning in llms](http://arxiv.org/abs/2509.00084)\.
- Wang et al\. \(2025b\)Yubo Wang, Xiang Yue, and Wenhu Chen\. 2025b\.[Critique fine\-tuning: Learning to critique is more effective than learning to imitate](http://arxiv.org/abs/2501.17703)\.
- Wen et al\. \(2025\)Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang\. 2025\.[Light\-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond](http://arxiv.org/abs/2503.10460)\.
- Xiong et al\. \(2025\)Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang\. 2025\.[Self\-rewarding correction for mathematical reasoning](http://arxiv.org/abs/2502.19613)\.
- Xu et al\. \(2025\)Bin Xu, Yiguan Lin, Yinghao Li, and Yang Gao\. 2025\.[Sra\-mcts: Self\-driven reasoning augmentation with monte carlo tree search for code generation](http://arxiv.org/abs/2411.11053)\.
- Yang et al\. \(2025\)Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E\. Gonzalez, Bin Cui, and Shuicheng Yan\. 2025\.[Supercorrect: Advancing small llm reasoning with thought template distillation and self\-correction](http://arxiv.org/abs/2410.09008)\.
- Ye et al\. \(2025\)Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu\. 2025\.Aime\-preview: A rigorous and immediate evaluation framework for advanced mathematical reasoning\.[https://github\.com/GAIR\-NLP/AIME\-Preview](https://github.com/GAIR-NLP/AIME-Preview)\.GitHub repository\.
- Yu et al\. \(2025\)Jiahao Yu, Zelei Cheng, Xian Wu, and Xinyu Xing\. 2025\.[Gpo: Learning from critical steps to improve llm reasoning](http://arxiv.org/abs/2509.16456)\.
- Yuan et al\. \(2023\)Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou\. 2023\.[Scaling relationship on learning mathematical reasoning with large language models](http://arxiv.org/abs/2308.01825)\.
- Zelikman et al\. \(2022\)Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D\. Goodman\. 2022\.[Star: Bootstrapping reasoning with reasoning](http://arxiv.org/abs/2203.14465)\.
- Zhang et al\. \(2025a\)Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal\. 2025a\.[Generative verifiers: Reward modeling as next\-token prediction](http://arxiv.org/abs/2408.15240)\.
- Zhang et al\. \(2025b\)Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong\. 2025b\.[Process\-based self\-rewarding language models](http://arxiv.org/abs/2503.03746)\.
- Zhang et al\. \(2024\)Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang\. 2024\.[Small language models need strong verifiers to self\-correct reasoning](http://arxiv.org/abs/2404.17140)\.
- Zhu et al\. \(2025\)Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei\-Lin Chen, Danqi Chen, and Yu Meng\. 2025\.[The surprising effectiveness of negative reinforcement in llm reasoning](http://arxiv.org/abs/2506.01347)\.

## Appendix AHyperparameters and More Implementation Details

This section provides additional training and evaluation hyperparameters not detailed in the main text\.

### A\.1Training Configuration

All models are trained using the Open\-R1 framework\. DeepSpeed ZeRO Stage 3 is used for memory\-efficient distributed training\.\. Table[5](https://arxiv.org/html/2609.19156#A1.T5)summarizes the key training hyperparameters\.

Table 5:Training hyperparameters\.HyperparameterValueBase ModelDeepSeek\-R1\-Distill\-Qwen\-7B/14BLearning Rate4\.0×10−54\.0\\times 10^\{\-5\}LR ScheduleCosine with min LRMin LR Rate0\.1Warmup Ratio0\.03Training Epochs5Max Sequence Length32768 tokensPrecisionbfloat16AttentionFlash AttentionMax Gradient Norm0\.2OptimizerAdamW

### A\.2Evaluation Configuration

Table[6](https://arxiv.org/html/2609.19156#A1.T6)provides detailed inference configuration parameters\.

Table 6:Inference configuration\.ParameterValueMax Model Length32768 tokensGPU Memory Utilization0\.95Attention BackendFlashAttentionData Typebfloat16Temperature0\.6Top\-p0\.95

## Appendix BPrompt Templates

This section presents the prompt templates used for trajectory generation and model training\.

### B\.1Generator Prompt

For failure trajectory collection and recovery resampling, we use the following system prompt:

> “Please reason step by step, and put your final answer within\\boxed\{\}\."

##### Failure Trajectory Collection\.

To generateNNinitial trajectories, the model takes the system prompt and the problem descriptionxxas input to produce a complete reasoning trajectoryyy\.

##### Resampling for Recovery\.

To generateKKcontinuation attempts, the input consists of the system prompt, the problemxx, and the truncated trajectoryy<ty\_\{<t\}, which is served as an assistant message prefix\. The model then generates a continuationy~\\tilde\{y\}from the prefixy<ty\_\{<t\}\.

During recovery resampling, prompting the model with bothxxandy<ty\_\{<t\}encourages it to recover from the intermediate potentially erroneous state\.

### B\.2Training Data Format

We format trajectories using a standard chat template, employing the generator prompt as the system message and the problem statementxxas the user message\.

- •RFT:The problemxxserves as the sole input query\. The cross\-entropy loss is computed on the entire reasoning trajectoryyy\.
- •Reflective Recovery:The input query incorporates both the problemxxand a truncated prefixy<ty\_\{<t\}\. Althoughy<ty\_\{<t\}is formatted as the beginning part of the assistant’s response, we apply loss masking to these tokens\. Consequently, the model is trained exclusively on the continuation tokensy~\\tilde\{y\}, enabling it to learn how to learn the recovery process\.

## Appendix CAdditional Scaling Law Results

This section presents additional scaling behavior analysis on AIME 2024 and LiveCodeBench V4 benchmarks, complementing the results shown in Figure[2](https://arxiv.org/html/2609.19156#S4.F2)in Section[4\.3](https://arxiv.org/html/2609.19156#S4.SS3)\.

![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/24.png)

Figure 5:Scaling behavior on AIME 2024 Diamond benchmark\.![Refer to caption](https://arxiv.org/html/2609.19156v1/figures/44.png)

Figure 6:Scaling behavior on LiveCodeBench V4 benchmark\.### C\.1Analysis of Code Benchmark Performance

For code benchmarks, Figure[6](https://arxiv.org/html/2609.19156#A3.F6)reveals an interesting pattern\. When the data volume reaches 10k, both our method and RFT exhibit a drop in performance on LiveCodeBench V4\. This may be because the model starts to overfit to the math\-specific problems, focusing too much on mathematical reasoning at the cost of its general coding ability\. This suggests a trade\-off between domain\-specific improvement and cross\-domain generalization when scaling up training data from a single domain\.