Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

# Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation
Source: [https://arxiv.org/html/2607.05541](https://arxiv.org/html/2607.05541)
###### Abstract

Reinforcement Learning is commonly used to train large language models using environmental feedback\. In applied settings, the environment usually provides sparse or delayed feedback\. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure\. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations\. We introduce a training framework,Self\-Review Reinforcement Learning, that embeds an explicitself\-reviewstep into each RL episode\. When a first\-pass response fails, the model generates aself\-reviewto identify what went wrong, which conditions an improved second attempt\. Unlike inference\-time reflection approaches, such asReflexion, the framework optimizes self\-review with policy gradients and internalizes improvements into the base policy viaselective distillation, ensuring they persist across future episodes\. Across\-episode memorykeeps successful self\-reviews for reuse when encountering similar tasks in future episodes during training\. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3\-4B and OLMo\-3\-7B, on GSM8K benchmark\. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement\. Code github repository:[https://github\.com/ZainAmin/COMP767\-Project](https://github.com/ZainAmin/COMP767-Project)

Machine Learning, ICML

## 1Introduction

![[Uncaptioned image]](https://arxiv.org/html/2607.05541v1/uploads/SRRL.png)

Figure 1:The SRRL training framework
These days, large language models \(LLMs\) are increasingly employed as decision\-making agents in situations with delayed rewards and partial knowledge\(Bai and others,[2026](https://arxiv.org/html/2607.05541#bib.bib18); Songet al\.,[2026](https://arxiv.org/html/2607.05541#bib.bib21); Wanget al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib22); Zhanget al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib24)\)\.

Reinforcement Learning provides an intuitive framework for enhancing such agents\. After the model agent generates a first\-pass response, it typically receives environmental feedback in the form of outcome rewards\. Environmental feedback is the information or signal the model receives from the world outside itself in response to its actions\. But the problem is that the environment often provides no feedback for individual steps, making the feedback rewards delayed and sparse\. Therefore, the model is confused about how to choose the correct stepwise reasoning to fix its behaviour and ends up being trained inefficiently with respect to sampling\(Shiet al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib17); Zhanget al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib24)\)\. Especially in multistep reasoning, errors in intermediate steps can invalidate the entire solution trajectories and credit assignments, leaving the model unable to recover\.

In supervised fine\-tuning \(SFT\), the model’s base policy is trained by learning to reproduce human\-provided fixed examples\. During training, the model observes pairs of prompts and their desired responses, then adjusts its parameters to minimise the difference between the generated and desired outputs\. This kind of imitation learning allows the model to become highly effective at generating highly realistic, well\-structured responses it has seen before\. However, this strength also creates a fundamental problem\. SFT is applied to a static dataset, and the model lacks an inherent mechanism to revise its behavior\. In real world use cases, SFT doesn’t have the ability to incorporate feedback and learn from its mistakes; instead it continues to rely on the training patterns it encoded earlier\.

Reinforcement learning with verifiable rewards \(RLVR\) takes the model training into interactive contexts\. RLVR allows models to progress through a trial\-and\-error approach while receiving external validation feedback\. In the RLVR framework, agents take an action and receive a scalar reward for success, and over time, update their policy to increase the probability of high\-reward outcomes\. This process takes one step beyond imitation learning and allows models to adapt and refine their performance over time with experience\. The key limitation is that the reward signals generally provide a numerical indication of success or failure without explaining why the action was effective or incorrect\. As a result, the agents continue to make similar mistakes and adopt suboptimal strategies, without any stable corrections that persist across episodes\. For this reason, the corrective structure of the good responses taken by the agent must be deduced directly from noisy, sparse, or delayed rewards\.

A natural progression from here is to do structure learning around experience itself by transforming the feedback into intermediate reasoning representation, such as review summaries, error analysis and the strategy updates, which can help make the corrective insights straightforward\. By integrating the intermediate learning representation, agents’ learning experience can reliably consolidate the improvements, as well as adjust their behavior more accurately without totally relying on reward\-driven exploration\.

## 2Literature review

### 2\.1RLs and LLMs:

The concept of learning through experience has become widely central to the development of large language models\. While early research shows that the models refine their outputs and align their behaviour based on human\-provided scores and evaluations, they have evolved into Reinforcement Learning from Human Feedback \(RLHF\)\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.05541#bib.bib15); Christianoet al\.,[2017](https://arxiv.org/html/2607.05541#bib.bib14); Shiet al\.,[2024](https://arxiv.org/html/2607.05541#bib.bib16); Shinnet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib10)\)\.

Recent research has extended reinforcement learning beyond the idea of preference alignment in order to enhance the mathematical and symbolic reasoning of the LLMs\. Instead of just relying on human feedback, the models receive the rewards directly from the programmatically verifiable signals\. For example, a code snippet can be executed against the test cases, a mathematical answer can be checked against proven ground truth, etc\. These checks actually provide the structured supervision signals that evaluate the reasoning outcomes\. The key shift in these strategies is the replacement of subjective reward models with deterministic mechanism processes\(Guo and others,[2025](https://arxiv.org/html/2607.05541#bib.bib19); Songet al\.,[2026](https://arxiv.org/html/2607.05541#bib.bib21); OpenAI,[2024](https://arxiv.org/html/2607.05541#bib.bib7)\)\.

Similarly, the agent\-oriented research on the LLMs presents the models as an interactive policy in order to operate in an external environment instead of just generating a single static response\. In this approach, models continue to take actions, gather observations and task\-dependent rewards in order to solve problems with multitask reasoning\(Yaoet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib12); Jinet al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib20); Bai and others,[2026](https://arxiv.org/html/2607.05541#bib.bib18)\)\.

Although these approaches differ, ranging from preference alignment to formal reasoning and interactive problem solving, all of them share the same training paradigm, which is that the environmental feedback is reduced to a scalar reward, which is optimised through gradient policy updates\. This limitation requires the model to implicitly infer how to correct its mistakes through repeated exploration rather then explicitly guided towards the structured revision of its reasoning process\. As a result of this limitation, our SRRL framework departs from the reward\-driven optimisation towards the process where the feedback is transformed into structured analysis and behaviour revision before being internalised into the base policy\.

### 2\.2Learning via Trails:

A large number of research studies suggest that the next scaling era for AI will be less dependent on statistical human feedback and more on the interaction of agents, which streamlines the learning via trials\. This way, agents can easily produce experience\-rich data through continual interaction with their environment\. One of the leading studies by Silver and Sutton’s lab argues that data streams generated by agents’ interaction, combined with long\-term decision making, can help the models to go beyond imitation and adopt learning via trails mechanism\(Silver and Sutton,[2025](https://arxiv.org/html/2607.05541#bib.bib26)\)\. This study gives an inspiration to adopt strategies that can convert the failed trials into usable learning signals instead of just relying on the successful trials\.

Earlier work on simple reinforcement learning handles the challenge of sparse rewards\. The issue was that the agents only received feedback after successful trials, and it was solved by introducing a mechanism that also extracts a signal from failed trials\. A notable contribution is the goal relabeling, as proposed in Hindsight Experience Replay \(HER\)\(Andrychowiczet al\.,[2018](https://arxiv.org/html/2607.05541#bib.bib9)\)\. In the sparse\-rewards setting, an agent can attempt to succeed, but if it fails, it receives no rewards, providing very little to no guidance for improvement\. HER solves this issue by relabeling the goal of a failed trial as the outcome that was actually achieved\. It means that even if the agent did not achieve the intended goal, the trial is still relabeled as a success for a different achieved state\. So this whole process ultimately improves sample efficiency by converting failed trials into useful training data\.

Similarly, in the LLM\-agent setting environment, the author addresses the void between the reward\-driven reinforcement learning and the imitation learning by allowing the agents to train from their own interaction traces, even when the rewards are sparse or not available\. The agent, instead of relying on the external feedback, uses its generated future states and decisions as a form of self\-supervision\. Self\-reflection also plays an important role as a way to observe suboptimal or failed behaviour\(Zhanget al\.,[2025](https://arxiv.org/html/2607.05541#bib.bib24)\)\.

The inference\-time reflection methods demonstrate that LLMs can improve their responses by revising them before consolidating and producing a final response\. The authors come to the agreement that all the self\-revision techniques which give the model the ability to think twice, help the model learn from its own mistakes and correct its behaviour, often lead to improved performance\. But these approaches require the ’think twice’ step to be executed during deployment\. For this reason, the model must access memory during inference, which increases computational cost and restricts improvements in runtime framing rather than permanent changes in the policy\(Zelikmanet al\.,[2022](https://arxiv.org/html/2607.05541#bib.bib13); Madaanet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib11); Shinnet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib10)\)\.

In contrast, concurrent research aims to incorporate conditioned feedback directly into the training process rather than relying on inference\-time self\-revision\. Recently,Hübotteret al\.\([2026](https://arxiv.org/html/2607.05541#bib.bib6)\)Songet al\.\([2026](https://arxiv.org/html/2607.05541#bib.bib21)\)used reinforcement learning with textual feedback\. The feedback\-conditioned teacher policy is distilled into the student policy\. It doesn’t require direct feedback during deployment\. Through distillation, the student learns to mimic the teacher’s feedback\-informed decisions, thereby internalising the corrective direction within its parameters\.

Our proposed SRRL also aligns with this line of work\. It places focus on the self\-review during training\. After the first\-pass response fails, the model generates a structured self\-review that guides a refined retry in the same episode\. The proposed approach treats the self\-review as a structured credit assignment task\. By combining self\-review with cross\-episode memory and policy distillation, the proposed criteria transform the experience into a lasting policy improvement without requiring self\-review at inference time\.

## 3Self\-Review Reinforcement Learning

Self\-Review Reinforcement Learning is a training framework which is designed to use the environmental feedback for structured behavioral corrections\. In standard RLVR, agents usually receive a scalar reward after generating a response, and right away update their policy parameters from this signal afterwards\. The main limitation of RLVR is that the reward provides no mechanism for the agent to know which intermediate reasoning steps were responsible for the failure or the success\. This problem forces the model to rediscover corrective strategies through repeated exploration, which further causes the inefficient learning problem\. SRRL addresses these problems by proposing an explicit self\-review step within each training episode\. This process is anchored by two supporting mechanisms, cross\-episode memory and policy distillation, which ensure that the insights produced through self\-review accumulate and persist across training\.

Each SRRL training episode proceeds through several stages, as shown in Figure[1](https://arxiv.org/html/2607.05541#S1.F1)\. Given an input problemxx, the modelπθ\\pi\_\{\\theta\}generates the first\-pass responsey1y\_\{1\}\. Afterwards, the environment evaluates they1y\_\{1\}response and returns a scalar outcome rewardr1∈\{0,1\}r\_\{1\}\\in\\\{0,1\\\}along with textual feedbackf1f\_\{1\}\. If ther1r\_\{1\}falls below a reward thresholdτ\\tau\(i\.e\.,r1<τr\_\{1\}<\\tau\), then the model enters a self\-review phase, an explicit analysis of what went wrong in responsey1y\_\{1\}and what corrective strategies can be applied to correct or improve the response\. Conditioned onxx,y1y\_\{1\},r1r\_\{1\},f1f\_\{1\}, and any retrieved memorymmfrom the cross\-episode memory mechanism, the model generates a second\-pass responsey2y\_\{2\}\. The environment evaluatesy2y\_\{2\}and returnsr2r\_\{2\},f2f\_\{2\}\. If the responsey2y\_\{2\}succeeds \(i\.e\.,r2\>τr\_\{2\}\>\\tau\), the self\-review is written in the cross\-episode memorymm, and the episode ends with a distillation update that trains the base policy to directly producey2y\_\{2\}quality responses fromxxalone\. This eliminates the redundant exploration problem of corrective strategies for similar tasks in subsequent training episodes\.

The design of SRRL is beneficial as a key deployment property, as stated below:

y∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)
meaning that at inference time, the model does not require any self\-review, memory retrieval, or policy distillation\. All improvements accumulated during training through SRRL are encoded into the base policy weights, which effectively ensure zero latency at deployment\. This factor directly addresses the limitation of inference\-time self\-reflection methods, such as Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib10)\)and Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2607.05541#bib.bib11)\), which require reflection to be present during deployment\.

In SRRL, the self\-review is not triggered automatically\. It is activated only if the first\-pass response falls below the reward threshold\. This condition is very important for two reasons, first is that by applying self\-review to already correct response trajectories cause the reward hacking problem, which precisely means that the model can easily generate instance\-specific shortcuts which can guarantee the success on current sample but will be unable to generalize\. Secondly, early in training, when the first\-pass solution rewards are usually low, the self\-review heavy batches become dominated by off\-policy second\-pass attempt data that ultimately weakens the on\-policy learning signal and destablizes optimization\.

By restricting the self\-review mechanism to failed trajectories, we ensure that successful responses remain purely on\-policy and each training batch retains a stable on\-policy component\. This factor directly aligns with our model\-deployment property, since the model has no access to self\-review during deployment; training without such conditioning would widen the gap between training\-time and deployment\-time behavior\. When the condition occurs, the model generates the self\-review as

Δ∼πθ​\(x,y1,r1,f1,m\)\\Delta\\sim\\pi\_\{\\theta\}\(x,y\_\{1\},r\_\{1\},f\_\{1\},m\)
After self\-reviewΔ\\Delta, the modelπθ\\pi\_\{\\theta\}attempts the second\-pass solution as

y2∼πθ​\(x,Δ\)y\_\{2\}\\sim\\pi\_\{\\theta\}\(x,\\Delta\)
Moreover, a single self\-review only provides a learning signal within the training episode in which it is generated\. Therefore, without an additional mechanism, it is impossible to retain these insights and the model has to rediscover them independently each time for structurally similar tasks in future episodes\. To overcome this limitation, we incorporate a cross\-episode memory step and use it as retrieval context for future training episodes\.

The cross\-episode memory stores successful self\-reviews as a plain text record\. The memory itemmmis directly provided to the modelπθ\\pi\_\{\\theta\}as part of the system context during the self\-reviewΔ\\Deltastep\. The successful self\-reviewΔ\\Deltais stored inmmonly if the second\-pass attempt succeeds, as follows:

mt\+1=\{mt∪\{Δ\},if​r2\>τ,mt,otherwise\.m\_\{t\+1\}=\\begin\{cases\}m\_\{t\}\\cup\\\{\\Delta\\\},&\\text\{if \}r\_\{2\}\>\\tau,\\\\ m\_\{t\},&\\text\{otherwise\.\}\\end\{cases\}
This selectiveΔ\\Deltastoring ensures that only the validated corrective strategies persist\. Subsequently, the selective supervised policy distillation step is applied by training the base policy on the successful second\-attempt response\. We optimize the model to directly reproducey2y\_\{2\}\-level responses \(i\.e\., reproduce improved behavior directly\) fromxxwithout access to self\-reviewΔ\\Deltacontext during future training episodes for structurally similar problems and at deployment as well\.

Ldistill​\(θ\)=−𝔼\(x,y2,r2\)∼𝒟​\[𝕀\{r2\>τ\}​log⁡πθ​\(y2∣x\)\]L\_\{\\mathrm\{distill\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,y\_\{2\},r\_\{2\}\)\\sim\\mathcal\{D\}\}\\left\[\\mathbb\{I\}\_\{\\\{r\_\{2\}\>\\tau\\\}\}\\log\\pi\_\{\\theta\}\(y\_\{2\}\\mid x\)\\right\]
ThisIIfunction triggers the distillation update only to successful second attempts\. Failed second pass solution attempts do not contribute to the distillation loss\.

### 3\.1RL Objective

Both the first pass, the second pass attempts, and self\-review are optimized with the RL objective\. We adopt GRPO \(Group Relative Policy Optimization\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.05541#bib.bib4)\)as the underlying gradient policy optimizer\. The policy loss is defined as:

Lpolicy​\(θ\)=−𝔼\(x,y\)∼𝒟​\[A​\(x,y\)​log⁡πθ​\(y∣x\)\]L\_\{\\mathrm\{policy\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\\left\[A\(x,y\)\\log\\pi\_\{\\theta\}\(y\\mid x\)\\right\]
Here,yyrepresents the model output, and the conditioning context corresponds to the model inputs for each generation step\. The advantage estimateAAis calculated from associated outcome rewards using GRPO group\-relative normalization; for a group ofGGsampled responses, the advantages are normalized within the group\.

Ai=ri−meanj​\{rj\}stdj​\{rj\}A\_\{i\}=\\frac\{r\_\{i\}\-\\mathrm\{mean\}\_\{j\}\\\{r\_\{j\}\\\}\}\{\\mathrm\{std\}\_\{j\}\\\{r\_\{j\}\\\}\}
The self\-reviewΔ\\Deltais assigned the rewardr~=r2\\tilde\{r\}=r\_\{2\}; that means, the self\-review is rewarded by the downstream success it produces\.

In order to prevent the policy from drifting too far from the reference model under the combined gradient signals, we apply KL regularization with a low\-variance KL loss and a fixed coefficient ofβ=0\.001\\beta=0\.001\.

### 3\.2Combined Objective

The full SRRL training objective per episode combines policy gradient loss over all generated outputs with the selective distillation loss:

LSRRL=Lpolicy\+Ldistill\+β⋅LKLL\_\{\\mathrm\{SRRL\}\}=L\_\{\\mathrm\{policy\}\}\+L\_\{\\mathrm\{distill\}\}\+\\beta\\cdot L\_\{\\mathrm\{KL\}\}
TheLpolicyL\_\{\\mathrm\{policy\}\}term covers the first\-pass attempt unconditionally, and the self\-review and the second\-pass attempt are activated only when this condition is triggered \(i\.e\.,r1<τr\_\{1\}<\\tau\)\. TheLdistillL\_\{\\mathrm\{distill\}\}term is masked to zero in episodes wherer2<τr\_\{2\}<\\tau\. This means that failed episodes still contribute an RL signal viaLpolicyL\_\{\\mathrm\{policy\}\}, but do not contaminate the distillation process with unsuccessful strategies\. Improvements are fully encoded in the model weights, preserving the gains of self\-review\-driven training at zero additional deployment cost\.

## 4Experiments

We evaluate SRRL against a standard RLVR baseline on GSM8K agentic reasoning benchmark\.

### 4\.1Task

Task is chosen to provide meaningful reward signal for the base model\. We also ensure that the models have partial competence so that GRPO can learn from variance in rewards\. We use GSM8K, a grade\-school mathematics word\-problem reasoning benchmark\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.05541#bib.bib27)\)\. Each problem has a single numeric answer, and the agent receives reward at episode completion\.

### 4\.2Metrics

#### 4\.2\.1Reward

For GSM8K dataset, the reward is binary and defined as:

r​\(y,y∗\)=𝕀​\[match​\(y,y∗\)\]r\(y,y^\{\*\}\)=\\mathbb\{I\}\[\\mathrm\{match\}\(y,y^\{\*\}\)\]
wherematch\\mathrm\{match\}is exact equality for GSM8K\. The self\-review condition is triggered if:

maxi⁡rifirst​attempt<τ,τ=1\\max\_\{i\}r\_\{i\}^\{\\mathrm\{first\\ attempt\}\}<\\tau,\\;\\tau=1
where the maximum is taken over allGGrollouts from the first attempt\.

#### 4\.2\.2Learning Efficiency

Beyond the final evaluation reward, we also report learning efficiency as the validation reward trajectory over the training steps\. This shows how quickly each method reaches higher reward, which is a key advantage of SRRL\.

### 4\.3Models and Baselines

Table 1:Training hyperparameters for experimentsWe train two backbone models using both the RLVR and SRRL frameworks\. GRPO is used as an underlying policy gradient optimizer in all cases\. First, we used Qwen3\-4B, a 4\-billion parameter pre\-trained language model from the Qwen3 family\(Yanget al\.,[2025b](https://arxiv.org/html/2607.05541#bib.bib3)\)\. The second model used in our experimental study is OLMo\-3\-7B from the OLMo family\(Team OLMoet al\.,[2024](https://arxiv.org/html/2607.05541#bib.bib30)\)\.

The RLVR baseline trains the same models with standard reward only GRPO, and there is no self\-review, cross\-episode memory, or distillation step\. In order to ensure that we have a fair comparison of experiments, we equalize the total training compute between the RLVR and SRRL\. RLVR uses 6 rollouts per prompt, while the SRRL uses 3 rollouts per attempt across two attempts \(i\.e\. total 6 per episode\)\. Both methods share the same GRPO objective and differ only in the trajectory structure\. We apply gradient accumulation with an effective batch size of 32, accumulating gradients across 32 episodes before each optimizer update\. This improves gradient stability\. For stable training, we employed RL techniques such as clipping, and KL regularization\. We train all models using our custom SRRL training loop implemented in Pytorch with the Hugging Face Transformers library\. Table[1](https://arxiv.org/html/2607.05541#S4.T1)summarizes all hyperparameters\.

## 5Results and Discussion

![Refer to caption](https://arxiv.org/html/2607.05541v1/prism-uploads/Results-co.png)Figure 2:Learning efficiency on GSM8K for Qwen3\-4B \(left\) and OLMo\-3\-7B \(right\)\. SRRL achieves higher validation reward than RLVR throughout training and finishes with stronger final performance\.Figure[2](https://arxiv.org/html/2607.05541#S5.F2)shows the validation reward trajectories across the 219 training steps for models trained on GSM8K\. All the curves are calculated on the fixed held\-out validation set of 473 samples, and evaluated every 5 training steps using greedy decoding under the RLVR format constraints\.

### 5\.1Qwen3\-4B

Both the SRRL and the RLVR frameworks start with a shared rollout reward of approximately 0\.71 at the first step\. It reflects the model’s ability to produce correct answers under the sampled RLVR rollout protocol\. It is lower than the 4\-shot CoT benchmark, which is reported in the Qwen technical report\(Yanget al\.,[2025a](https://arxiv.org/html/2607.05541#bib.bib29)\)\. It is consistent with the known gap between the format\-constrained RLVR rollout reward and the standard greedy evaluation\. Both curves remain hard to differentiate until approximately step 50\. It indicates that both models share similar early gradient signals and the policy has not yet diverged greatly\. Afterwards, the SRRL consistently beats RLVR on the 473\-sample validation set\. After the first 65 training steps, the curve follows a natural deceleration as the model approaches the ceiling of its math\-specialized pretraining capacity\.

### 5\.2OLMo\-3\-7B

The OLMo\-3\-7B curves show a distinct pattern consistent with a general\-purpose model on a math\-focused benchmark\. Both curves begin at a rollout reward of approximately 0\.65, which is lower than Qwen3\-4B base despite OLMo\-3\-7B having more parameters\. This likely occurs because the OLMo\-3\-7B pretraining corpus is less math\-focused\. Early on, both curves remain fused until SRRL begins to diverge\. Unlike the Qwen3\-4B concave trajectory, OLMo\-3\-7B exhibits a near\-linear rise\. This suggests that the model has greater headroom to learn and has not reached a reward ceiling by step 219\. SRRL achieves a final validation reward of 0\.75, compared with RLVR’s 0\.72\. This corresponds to approximately 10 additional correct answers on the validation dataset\.

### 5\.3Effects of Self\-Review

The wider SRRL advantage on OLMo\-3\-7B compared to Qwen3\-4B is not coincidental and reveals that self\-review benefits models with weaker prior math knowledge\. Qwen3\-4B already shows strong mathematical priors from pretraining, so the policy quickly learns to generate correct answers via standard reward maximisation\. As a result, self\-review provides incremental corrections but competes with a capable base policy already\. The base policy is more prone to errors in the OLMo\-3\-7B case\. Self\-review provides a systematic error\-detection signal, and aligns with prior work that shows that iterative self\-correction is effective when the base model operates below performance saturation\.

The test rewards are closer to the corresponding validation rewards for both models\. It’s a very consistent train\-test gap that shows the absence of validation\-set overfitting\. SRRL outperforms RLVR on both models, and the OLMo\-3\-7B achieves a larger absolute improvement than the Qwen3\-4B\. Notably, Qwen3\-4B model achieves higher absolute test performance than OLMo\-3\-7B despite having fewer parameters\. This reversal is the usual parameter\-scaling intuition, demonstrated by Qwen3\-4B math focus pretraining, which gives it an excellent math reasoning prior\.

### 5\.4Implications

Together, the results support multiple conclusions\. First, SRRL surpasses standard RLVR on a general\-purpose task and matches specialised base models, which shows promise across two architectures\. Also, the magnitude of Self\-Review benefit scales inversely with the base model’s prior math capability\. This finding has practical importance for the model selection\. SRRL should be prioritised for deployment on general\-purpose base models where the RLVR’s reward signal is less informative\. Lastly, the qualitatively different learning dynamics between the two models suggest that the shape of reward curves carries significant information on how close the model is to its domain\-specific performance ceiling\. In future, it may serve as a diagnostic tool for RL training budget allocation\.

Table 2:Final test rewards on the 1,319\-sample GSM8K test set\.

## 6Conclusion

We introduced and developed a training framework, known as Self\-Review Reinforcement Learning, that embeds an explicit self\-review, cross\-episode memory and policy distillation steps into the reinforcement learning loop to transform feedback into behavioral improvements\. SRRL enables the model to diagnose its failed attempts, learn corrective strategies, and condition a refined second attempt within the same training episode\. By combining self\-review guidance with selective distillation, SRRL internalizes successful correction strategies into the base policy weights ensuring that improvements persist throughout future training episodes and eliminate the need for self\-review at deployment\. Across agentic reasoning tasks, the SRRL framework improves the learning efficiency, and produces stronger final polices as compared to standard RLVR baseline\.

We evaluated SRRL against RLVR on math reasoning\. SRRL directly addressed the latency limitation of inference time reflection methods\. Future direction suggests analysing which factor correlates the most with second attempt success the self\-review more effective, for example, specificity of error diagnosis, actionability of corrective guidelines, or the current task structure\.

## Acknowledgment

The authors would like to sincerely thank Professor Siva Reddy for his guidance, teaching, and providing a valuable learning experience throughout course\.

## References

- M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba \(2018\)Hindsight experience replay\.arXiv preprint arXiv:1707\.01495\.Cited by:[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p2.1)\.
- Y\. Baiet al\.\(2026\)Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[§1](https://arxiv.org/html/2607.05541#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p3.1)\.
- P\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.arXiv preprint arXiv:1706\.03741\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§4\.1](https://arxiv.org/html/2607.05541#S4.SS1.p1.1)\.
- D\. Guoet al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p2.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. Krause \(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.External Links:[Link](https://arxiv.org/abs/2601.20802)Cited by:[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p5.1)\.
- B\. Jin, H\. Zeng, Z\. Yue,et al\.\(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.OpenReview\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p3.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.arXiv preprint arXiv:2303\.17651\.Cited by:[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p4.1),[§3](https://arxiv.org/html/2607.05541#S3.p5.1)\.
- OpenAI \(2024\)OpenAI o1 system card\.arXiv preprint arXiv:2412\.16720\.External Links:[Link](https://arxiv.org/abs/2412.16720)Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin,et al\.\(2022\)Training language models to follow instructions with human feedback\.arXiv preprint arXiv:2203\.02155\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§3\.1](https://arxiv.org/html/2607.05541#S3.SS1.p1.1)\.
- T\. Shi, K\. Chen, and J\. Zhao \(2024\)Safer\-instruct: aligning language models with automated preference data\.arXiv preprint arXiv:2311\.08685\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p1.1)\.
- T\. Shi, Z\. Wang, L\. Yang,et al\.\(2025\)Wildfeedback: aligning llms with in\-situ user interactions and feedback\.arXiv preprint arXiv:2408\.15549\.Cited by:[§1](https://arxiv.org/html/2607.05541#S1.p2.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.arXiv preprint arXiv:2303\.11366\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p4.1),[§3](https://arxiv.org/html/2607.05541#S3.p5.1)\.
- D\. Silver and R\. S\. Sutton \(2025\)Welcome to the era of experience\.Semantic Scholar preprint\.Cited by:[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p1.1)\.
- Y\. Song, L\. Chen, F\. Tajwar,et al\.\(2026\)Expanding the capabilities of reinforcement learning via text feedback\.arXiv preprint arXiv:2602\.02482\.Cited by:[§1](https://arxiv.org/html/2607.05541#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p5.1)\.
- Team OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, and H\. Hajishirzi \(2024\)2 OLMo 2 furious\.arXiv preprint arXiv:2501\.00656\.External Links:[Link](https://arxiv.org/abs/2501.00656)Cited by:[§4\.3](https://arxiv.org/html/2607.05541#S4.SS3.p1.1)\.
- S\. Wang, Y\. Wu, and Z\. Xu \(2025\)Cogito, ergo ludo: an agent that learns to play by reasoning and planning\.arXiv preprint arXiv:2509\.25052\.Cited by:[§1](https://arxiv.org/html/2607.05541#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, Z\. Qiu,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2607.05541#S5.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025b\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.3](https://arxiv.org/html/2607.05541#S4.SS3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§2\.1](https://arxiv.org/html/2607.05541#S2.SS1.p3.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p4.1)\.
- K\. Zhang, X\. Chen, B\. Liu, T\. Xue, Z\. Liao, Z\. Liu, X\. Wang, Y\. Ning, Z\. Chen, X\. Fu, J\. Xie, Y\. Sun, B\. Gou, Q\. Qi, Z\. Meng, J\. Yang, N\. Zhang, X\. Li, A\. Shah, D\. Huynh, H\. Li, Z\. Yang, S\. Cao, L\. Jang, S\. Zhou, J\. Zhu, H\. Sun, J\. Weston, Y\. Su, and Y\. Wu \(2025\)Agent learning via early experience\.arXiv preprint\.Note:PreprintCited by:[§1](https://arxiv.org/html/2607.05541#S1.p1.1),[§1](https://arxiv.org/html/2607.05541#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.05541#S2.SS2.p3.1)\.

## Appendix AAppendix

This appendix provides supplementary material for the main paper\.

### A1Math Prompt

### A2Review Prompt

### A3GSM8K Dataset Sample

Problem:Beth bakes 4 batches of 2 dozen cookies in a week\. If these cookies are shared equally among 16 people, how many cookies does each person consume?

Solution:Beth bakes4×24\\times 2dozen cookies in total:

dozen cookies\.

Since each dozen contains 12 cookies, the total number of cookies is:

These 96 cookies are shared equally among 16 people:

Final Answer:<answer\>6</answer\>

### A4Pseudo code

Algorithm 1Self\-Review Reinforcement Learning \(SRRL\) on GSM8K benchmark1:Inputs:Language model

πθ\\pi\_\{\\theta\}; dataset of questions

xx; environment returning feedback

ffand reward

rr; reward threshold

τ\\tau; KL coefficient

β\\beta\.

2:Initialize:cross\-episode memory

m←∅m\\leftarrow\\emptyset\.

3:repeat

4:Sample question

xxfrom dataset\.

5://First attempt

6:Sample first\-pass response

y1∼πθ\(⋅∣x\)y\_\{1\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.

7:Obtain feedback and reward

\(f1,r1\)\(f\_\{1\},r\_\{1\}\)\.

8://RL update on the first attempt

9:Update

θ\\thetavia

ℒpolicy​\(θ\)\\mathcal\{L\}\_\{\\text\{policy\}\}\(\\theta\)on

y1y\_\{1\}\.

10://Gated self\-review and second attempt

11:if

r1<τr\_\{1\}<\\tauthen

12://Self\-review with cross\-episode memory

13:Retrieve relevant memory

mm\.

14:Sample self\-review

Δ∼πθ\(⋅∣x,y1,r1,f1,m\)\\Delta\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{1\},r\_\{1\},f\_\{1\},m\)\.

15:Sample second\-pass response

y2∼πθ\(⋅∣x,Δ\)y\_\{2\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\Delta\)\.

16:Obtain feedback and reward

\(f2,r2\)\(f\_\{2\},r\_\{2\}\)\.

17:Set self\-review reward

r~←r2\\tilde\{r\}\\leftarrow r\_\{2\}\.

18://Store successful self\-review only

19:if

r2≥τr\_\{2\}\\geq\\tauthen

20:Update memory:

m←m∪\{Δ\}m\\leftarrow m\\cup\\\{\\Delta\\\}\.

21:endif

22://RL update on self\-review and second attempt

23:Update

θ\\thetavia

ℒpolicy​\(θ\)\\mathcal\{L\}\_\{\\text\{policy\}\}\(\\theta\)on

Δ\\Deltaand

y2y\_\{2\}\.

24://Selective policy distillation

25:if

r2≥τr\_\{2\}\\geq\\tauthen

26:Update

θ\\thetavia

ℒdistill​\(θ\)\\mathcal\{L\}\_\{\\text\{distill\}\}\(\\theta\)to train

πθ\\pi\_\{\\theta\}to produce

y2y\_\{2\}from

xxalone\.

27:endif

28:endif

29://KL regularization

30:Apply KL regularization with coefficient

β\\beta\.

31:untilconverged