Happy to share that MURPHY has been accepted at #NeurIPS2026! 🎉

As we move toward more agentic training and iterative self-improvement, models need to learn from the full process of trying, failing, receiving feedback, and correcting themselves, not just from the final outcome. 🔁🧠

MURPHY makes GRPO feedback-aware, assigning credit to earlier attempts when their feedback helps enable a later successful fix. 🛠️✅

🎥 Quick overview below 📄 Paper: https://arxiv.org/abs/2511.07833

Murphy: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation

Source: https://arxiv.org/html/2511.07833 Chanakya Ekbote††thanks:Equal contribution and corresponding author(s). Chanakya Ekbote was an intern at AWS AI during this work.Vijay Lingam††footnotemark:Email:mailto:[email protected]@amazon.comSujay SanghaviBehrooz Omidvar-TehraniJun HuanAnoop DeorasStefano SoattoMassachusetts Institute of TechnologyAWS AIAmazon

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard recipe for post-training LLMs on reasoning tasks, with Group Relative Policy Optimization (GRPO) emerging as a leading approach. However, GRPO and its variants are inherently single-turn: they optimize from terminal rewards on isolated prompt-response pairs, leaving them poorly suited to agentic settings where models must iteratively refine solutions in response to environmental feedback. We introduceMurphy, a multi-turn extension of GRPO for self-correcting code generation.Murphyconstructs feedback-conditioned rollout trees in which failed candidate solutions are paired with executor feedback and expanded into subsequent turns, and propagates rewards backward through the tree so that later successful refinements credit earlier attempts that surfaced informative feedback. We study two propagation strategies, Max Reward (MaRS) and Mean Reward (MeRS), and introduce post-rollout pruning mechanisms that reduce multi-turn optimization cost. Across three code generation benchmarks (HumanEval, MBPP, LiveCodeBench-v6) and two model families (Qwen3-1.7B/4B, OLMo-2-7B),Murphydelivers up to6%absolute pass@1 gains over the strongest prior multi-turn execution-feedback methods. Gains are largest on the Medium/Hard subset (+4.38/+4.20at Iter-55), where iterative self-correction matters more.

1Introduction

“The road to wisdom? Well, it’s plain and simple to express: err and err and err again, but less and less and less.” —Piet Hein

Figure 1:Self-correction performance on LiveCodeBench-v6.Qwen3-4B withMurphyvs. four execution-feedback baselines (Base, GRPO-MT,μ\muCode,ReVeal) on the Easy, Medium, and Hard subsets. Base, GRPO-MT,μ\muCode, andMurphyare evaluated under the Reflexion framework(Shinn et al., 2023);ReVealis evaluated using its official generation-verification scaffold, which uses a method-specific prompt and parser. All methods are compared at 1, 3, and 5 inference turns. Bars show the mean and stdev across 3 runs.Murphyshows large gains on the Medium/Hard subset (+4.38/+4.20at Iter-55), where self-correction matters more.Large language models (LLMs) are increasingly deployed as code-generation agents that interact with their environment rather than producing a single static response(Yang et al., 2024;Wang et al., 2025). In a typical execution-feedback scaffold(Shinn et al., 2023;Zhou et al., 2024;Lingam et al., 2025), the model proposes code, the code is executed against unit tests, and failures are returned as error messages, stack traces, or failing test cases. The model is then re-prompted with the original task, its previous attempt, and the feedback, and this process repeats until the solution is accepted or a turn budget is exhausted. Such scaffolds substantially improve inference-time self-correction, but they leave the underlying model unchanged: feedback is consumed as test-time context only(Shinn et al., 2023;Setlur et al., 2026).

Recent works attempt to close this gap by training models to use execution feedback. RLEF(Gehring et al., 2025)optimizes multi-turn code generation with PPO(Schulman et al., 2017)against execution-based rewards.μ\muCode(Jain et al., 2025)recasts multi-turn correction as expert-iteration imitation learning with a learned verifier for relabeling and selection.ReVeal(Jin et al., 2026)interleaves generation and verification within a single policy, using turn-aware optimization to improve self-verification. These methods establish the value of execution feedback, but each relies on PPO-style value estimation, learned verifiers, or explicit verification policies. They leave open a simpler question:can the standard GRPO(Shao et al., 2024)post-training recipe itself be made feedback-aware in multi-turn environments?

A natural baseline is GRPO-MT, which extends GRPO by treating each feedback-conditioned interaction as a trajectory. The model samples candidates for the original prompt; passing rollouts terminate, while failed ones are paired with executor feedback and expanded for additional turns until solved or the turn budget is exhausted. Each terminated rollout’s final execution reward serves as its trajectory-level outcome for the GRPO advantage, broadcast across policy generated tokens along that rollout while executor feedback and prior-solution context are masked from the loss. Although GRPO-MT exposes the model to execution feedback, its credit assignment is coarse: all generations along a terminated trajectory inherit the same terminal signal, regardless of whether an intermediate attempt produced useful feedback, a misleading direction, or contributed little to the eventual fix. This matters in code generation, where failed attempts are not uniformly bad: an incorrect solution may reveal the precise failing case or runtime behavior that enables a later refinement. GRPO-MT can reinforce successful trajectories but cannot distinguish informative failures from uninformative ones when they share a terminal outcome.

Prior inference-time agent frameworks(Shinn et al., 2023;Zhou et al., 2024;Lingam et al., 2025)show that previous failed attempts and environment feedback provide useful signals for self-correction. In code generation, a failed program may expose an edge case, runtime error, or test failure that enables a later refinement. Thus, failed attempts should not be treated as uniformly negative; their value depends on whether the feedback they induce supports future correction.

Murphyturns this observation into a training-time objective for multi-turn GRPO. It constructs feedback-conditioned rollout trees in which successful candidates terminate and failed candidates are expanded by re-prompting on the original problem, prior output, and executor feedback. Once the rollout is complete,Murphypropagates rewards backward through the tree, allowing later successful refinements to credit earlier attempts. We instantiate this idea with Max Reward (MaRS), which backs up the best descendant outcome, and Mean Reward (MeRS), which backs up a discounted average future reward, then apply a local GRPO update using the resulting credit-adjusted rewards. This design isolates the algorithmic question from confounding system choices. Unlike RLEF,Murphyrequires no PPO critic; unlikeμ\muCode, no learned verifier or imitation reduction; unlikeReVeal, no separate verification behavior or test synthesis.Murphyis a minimal, architecture-preserving extension of GRPO to feedback-conditioned, multi-turn code generation.

Main Contributions.1.We introduceMurphy, a multi-turn GRPO algorithm that trains code LLMs to use execution feedback through feedback-conditioned rollouts and structured temporal credit assignment. (Sec. 3)2.We proposeMaRSandMeRS, two reward-propagation strategies that assign credit from later refinements to earlier attempts in the rollout tree.3.We design pruning strategies that make multi-turn GRPO training tractable by retaining informative trajectories while reducing optimization cost. (Subsec. 3.1)4.We evaluateMurphyon three code-generation benchmarks across two model families (OLMo, Qwen) and sizes (1.7B–7B), achieving up to6%absolute pass@1 gains over prior multi-turn execution-feedback methods. (Sec. 4)

2Background: GRPO

Group Relative Policy Optimization (GRPO;(Shao et al., 2024)) is a reinforcement-learning objective for LLM fine-tuning that avoids learning a separate value function. For each input promptqq, the old policyπθold\pi_{\theta_{\mathrm{old}}}samples a group ofGGcandidate responses,𝒞⁡(q)={u1,…,uG}\mathcal{C}(q)=\{u_{1},\ldots,u_{G}\}. Each responseu∈𝒞⁡(q)u\in\mathcal{C}(q)receives a scalar rewardr⁡(u)r(u). GRPO computes advantages by normalizing rewards within the response group:

A^qGRPO​(u)=r⁡(u)−meanw∈𝒞⁡(q)​r​(w)stdw∈𝒞⁡(q)​r​(w),u∈𝒞⁡(q)\displaystyle\hat{A}^{\mathrm{GRPO}}_{q}(u)=\frac{r(u)-\mathrm{mean}_{w\in\mathcal{C}(q)}r(w)}{\mathrm{std}_{w\in\mathcal{C}(q)}r(w)},\qquad u\in\mathcal{C}(q)(1)This group-relative baseline replaces the learned critic used in PPO. Letutu_{t}denote thett-th token of responseuu, and letu<tu_{<t}be its prefix. The token-level importance ratio isRθ​(q,u,t)=πθ​(ut∣q,u<t)πθold​(ut∣q,u<t)R_{\theta}(q,u,t)=\frac{\pi_{\theta}(u_{t}\mid q,u_{<t})}{\pi_{\theta_{\mathrm{old}}}(u_{t}\mid q,u_{<t})}. The GRPO clipped objective for a response group is then

𝒥GRPO​(θ,q,𝒞⁡(q),A^q)\displaystyle\mathcal{J}_{\mathrm{GRPO}}\left(\theta;q,\mathcal{C}(q),\hat{A}_{q}\right)=𝔼u∈𝒞⁡(q)[1|u|∑t=1|u|min(Rθ(q,u,t)A^q(u),clip(Rθ(q,u,t)\displaystyle=\mathbb{E}_{u\in\mathcal{C}(q)}\bigg[\frac{1}{|u|}\sum_{t=1}^{|u|}\min\!\Big(R_{\theta}(q,u,t)\hat{A}_{q}(u),\mathrm{clip}(R_{\theta}(q,u,t),1−ϵ,1+ϵ)A^q(u))]−βDKL(πθ∥πref)\displaystyle\qquad\qquad\quad,1-\epsilon,1+\epsilon)\hat{A}_{q}(u)\Big)\bigg]-\,\beta\,D_{\mathrm{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\right)(2)whereπref\pi_{\mathrm{ref}}is a fixed reference policy andβ\betacontrols the KL penalty. In standard single-turn GRPO,A^q=A^qGRPO\hat{A}_{q}=\hat{A}^{\mathrm{GRPO}}_{q}is computed from the immediate rewards of responses sampled for the same prompt.Murphykeeps this group-relative update structure but changes how the rewards used in the advantage are assigned in a multi-turn rollout tree as described inSec. 3.

3Proposed Method:Murphy

Figure 2:Overview ofMurphy.Given an input promptq(1)q_{(1)}, the model samples multiple candidate solutions. Each output is executed to obtain a scalar rewardrrand executor feedbackff. Solved outputs terminate, while failed outputs are converted into feedback-conditioned prompts by concatenating the original prompt, the failed output, and its feedback. The model then samples refinements for each failed output. After the rollout tree is generated, rewards from later turns are propagated backward through a credit-assignment ruleρ⁡(⋅)\rho(\cdot), allowing successful refinements to credit earlier failed attempts that produced useful feedback.Murphyextends GRPO to feedback-rich, multi-turn code generation. Standard GRPO optimizes one response group per prompt using rewards from that same generation step. This is insufficient for agentic code generation, where the model may execute code, observe failures, and refine its solution over multiple attempts. In this setting, an initially incorrect solution can still be valuable if it exposes useful executor feedback, such as a failing edge case or runtime error, that enables a later correction.Murphyaddresses this mismatch by constructing feedback-conditioned rollout trees and assigning credit retrospectively from later successful refinements to earlier attempts.

Feedback-conditioned rollout tree.

For each input prompt,Murphyfirst samples a group of candidate programs. Each candidate is executed against the task test suite, producing two signals: a scalar reward, such as the proportion of tests passed, and qualitative executor feedback, such as failing test cases, stack traces, compiler errors, or timeout messages. Candidates that achieve the maximum reward are treated as solved and become terminal nodes: they are kept for optimization but are not expanded in later turns.

Failed candidates are expanded in the next turn. To do so,Murphyforms a new prompt by concatenating the original task prompt, the failed output, and the corresponding executor feedback. The policy is then re-invoked on this feedback-conditioned prompt to generate a new group of candidate refinements. This expansion is local to each failed node: different failures produce different feedback-conditioned prompts and therefore induce different refinement groups. The process repeats for a fixed number of turns or until no failed nodes remain. The result is a rollout tree in which each node is a generated program and each edge corresponds to one refinement step conditioned on executor feedback from the parent node.

This tree structure preserves the causal relationship between an attempted solution, the feedback it produces, and the refinements that follow. InFig. 2, the first turn produces two outputs. The solved output terminates immediately. The failed output receives feedback, is converted into a new prompt, and is expanded into two second-turn refinements. Thus, later outputs are not independent samples from the original prompt; they are feedback-conditioned descendants of specific earlier attempts.

Retrospective credit assignment.

After the rollout tree is generated and all candidates are evaluated,Murphypropagates rewards backward using a bottom-up recursion. Starting from the leaves, each node passes reward information to its parent; the parent is then updated using its own immediate execution reward and the adjusted rewards of its children. This differs from single-turn GRPO, where a response is credited only by its own reward within the original response group. InMurphy, a failed attempt can receive positive credit if its feedback leads to a later successful refinement.

Credit-adjusted rewards.

Let𝒯\mathcal{T}denote the rollout tree for a prompt, where each nodev∈𝒯v\in\mathcal{T}corresponds to a generated program. Each node has an immediate execution rewardr⁡(v)r(v)and, if expanded, a child response group𝒞⁡(v)\mathcal{C}(v)generated by re-prompting on its feedback. Solved nodes are terminal and have𝒞⁡(v)=∅\mathcal{C}(v)=\emptyset.Murphycomputes credit-adjusted rewardsr~​(v)\tilde{r}(v)by a backward recursion over𝒯\mathcal{T}. For leaf nodes,r~​(v)=r​(v)\tilde{r}(v)=r(v). For rest nodes, we consider 2 credit-assignment rules:

MaRS:r~(v)\displaystyle\textsc{MaRS}:\quad\tilde{r}(v)=max⁡(r⁡(v),maxu∈𝒞⁡(v)⁡r~​(u)),\displaystyle=\max\left(r(v),\max_{u\in\mathcal{C}(v)}\tilde{r}(u)\right),(3)MeRS:r~(v)\displaystyle\textsc{MeRS}:\quad\tilde{r}(v)=r⁡(v)+γ⋅meanu∈𝒞⁡(v)​r~​(u)Z⁡(v),\displaystyle=\frac{r(v)+\gamma\cdot\mathrm{mean}_{u\in\mathcal{C}(v)}\tilde{r}(u)}{Z(v)},(4) whereγ∈[0,1]\gamma\in[0,1]is a discount factor andZ⁡(v)Z(v)is a normalization term that counts the reward components included in the backup. This keeps adjusted rewards comparable across nodes at different depths. In our two-turn setting,Z⁡(v)=2Z(v)=2for unsolved internal nodes, corresponding to the node’s immediate reward and the propagated reward from its children, while terminal nodes retainZ⁡(v)=1Z(v)=1andr~​(v)=r​(v)\tilde{r}(v)=r(v).MaRSperforms an optimistic backup: it propagates the best achievable refinement outcome upward through the tree, similar in spirit to max-backup variants of Monte Carlo Tree Search(Khandelwal et al., 2016). Thus, if one refinement of a failed first-turn solution solves the task,MaRScredits that success back to the first-turn node.MeRSinstead propagates an average future-refinement value, analogous to a Bellman-style backup over sampled refinement nodes. Unlike value-based RL, however,Murphyperforms this backup directly over generated refinements rather than learning a separate value function.

Node-wise GRPO objective.

After credit assignment,Murphyapplies GRPO locally to each response group in the rollout tree. Let𝒬⁡(𝒯)\mathcal{Q}(\mathcal{T})denote the set of prompts and feedback-conditioned prompts expanded in the tree. For anyq∈𝒬⁡(𝒯)q\in\mathcal{Q}(\mathcal{T}), let𝒞⁡(q)\mathcal{C}(q)be its response group. Instead of computing advantages from immediate rewards as inEq. 1,Murphynormalizes credit-adjusted rewards within the group:

A^qMurphy​(u)=r~​(u)−meanw∈𝒞⁡(q)​r~​(w)stdw∈𝒞⁡(q)​r~​(w),u∈𝒞⁡(q).\displaystyle\hat{A}^{\textsc{Murphy}}_{q}(u)=\frac{\tilde{r}(u)-\mathrm{mean}_{w\in\mathcal{C}(q)}\tilde{r}(w)}{\mathrm{std}_{w\in\mathcal{C}(q)}\tilde{r}(w)},\qquad u\in\mathcal{C}(q).(5)TheMurphyobjective is the sum of GRPO losses over the expanded response groups:

𝒥Murphy​(θ)=𝔼𝒯∼πθold​[∑q∈𝒬⁡(𝒯)𝒥GRPO​(θ,q,𝒞⁡(q),A^qMurphy)]\displaystyle\mathcal{J}_{\textsc{Murphy}}(\theta)=\mathbb{E}_{\mathcal{T}\sim\pi_{\theta_{\mathrm{old}}}}\left[\sum_{q\in\mathcal{Q}(\mathcal{T})}\mathcal{J}_{\mathrm{GRPO}}\left(\theta;q,\mathcal{C}(q),\hat{A}^{\textsc{Murphy}}_{q}\right)\right](6)Thus,Murphydoes not introduce a critic or a new policy-gradient estimator. It applies the same GRPO update fromEq. 2at each expanded response group in the rollout tree, while replacing immediate rewards with retrospectively adjusted rewards. The policy update remains token-level, but the scalar learning signal attached to each generation now reflects both its immediate execution result and its contribution to later feedback-conditioned refinements. We provide the full rollout-tree indexing, recursive credit-assignment details, and implementation details inApp. C.

3.1Pruning Strategies inMurphy

The feedback-conditioned rollout tree inMurphycan grow quickly with the number of turns. By default,Murphyuses a fixed generation budgetGGat every turn. If most generations fail and continue to be expanded, the number of generated programs can grow asGSG^{S}for a turn budgetSS, in the worst case. This increases memory use and slows optimization, since each retained sequence may contribute to advantage computation and gradient updates. To make multi-turn training tractable, we introduce pruning strategies that retain a smaller subtree before retrospective credit assignment.

Pruning is applied after rollouts are generated and rewards are computed, but before rewards are propagated backward. This order is important: the rollout procedure still collects feedback-conditioned generations and their rewards, but only the retained subtree contributes to credit assignment and optimization. Let𝒯′⊆𝒯\mathcal{T}^{\prime}\subseteq\mathcal{T}denote the retained subtree after pruning(𝒯′=Prune⁡(𝒯))(\mathcal{T}^{\prime}=\mathrm{Prune}(\mathcal{T})). Pruning changes both the children used in the backward credit recursion and the response groups included in the node-wise GRPO objective:

𝒥Murphypruned​(θ)\displaystyle\mathcal{J}_{\textsc{Murphy}}^{\mathrm{pruned}}(\theta)=𝔼𝒯∼πθold​[∑q∈𝒬⁡(𝒯′)𝒥GRPO​(θ,q,𝒞𝒯′​(q),A^qMurphy)],\displaystyle=\mathbb{E}_{\mathcal{T}\sim\pi_{\theta_{\mathrm{old}}}}\left[\sum_{q\in\mathcal{Q}(\mathcal{T}^{\prime})}\mathcal{J}_{\mathrm{GRPO}}\left(\theta;q,\mathcal{C}_{\mathcal{T}^{\prime}}(q),\hat{A}^{\textsc{Murphy}}_{q}\right)\right],(7)where𝒞𝒯′​(q)\mathcal{C}_{\mathcal{T}^{\prime}}(q)denotes the retained response group for promptqqin the pruned tree. In this sense, pruning is a compute-aware filter on the rollout tree: it does not change the rollout mechanism or the GRPO update, but determines which generated trajectories are retained for reward propagation and gradient updates.

We consider two pruning strategies with different budget granularities.IntraPuses awithin-group trajectory budget, retaining a fixed number of trajectories inside each response group.InterPuses anacross-group budget, retaining a fixed number of refinement groups among those induced by failed parent nodes. We useTopBx∈𝒳b​(s⁡(x))\mathrm{TopB}^{b}_{x\in\mathcal{X}}(s(x))to denote the subset ofbbelements from𝒳\mathcal{X}with the largest scoress⁡(x)s(x).

Intra-Group Pruning (IntraP).

IntraPprunes individual trajectories within each response group. Consider a prompt or feedback-conditioned promptqqwith a group of generated children𝒞⁡(q)\mathcal{C}(q). Given a within-group trajectory budgetbtrajb_{\mathrm{traj}},IntraPretains a subset𝒞′​(q)⊆𝒞⁡(q)\mathcal{C}^{\prime}(q)\subseteq\mathcal{C}(q)with|𝒞′​(q)|=btraj|\mathcal{C}^{\prime}(q)|=b_{\mathrm{traj}}and discards the remaining children together with their descendants. The retained trajectories are selected according to their contribution to within-group reward variance:𝒞′​(q)=TopBu∈𝒞⁡(q)btraj​(ΔVar​(u,𝒞⁡(q))),\mathcal{C}^{\prime}(q)=\mathrm{TopB}_{u\in\mathcal{C}(q)}^{b_{\mathrm{traj}}}\left(\Delta_{\mathrm{Var}}(u;\mathcal{C}(q))\right),whereΔVar​(u,𝒞​(q))\Delta_{\mathrm{Var}}(u;\mathcal{C}(q))measures how much trajectoryuucontributes to the reward variance of the group. This strategy is inspired byXu et al. (2025a), who show that retaining high-variance trajectories within a response group can reduce optimization cost while preserving the learning signal needed for group-relative policy optimization. We extend this idea to the multi-turn setting by applying it recursively over the rollout tree. Since GRPO computes advantages from relative reward differences within a group, preserving reward diversity helps retain informative comparisons while reducing the number of sequences used for optimization. If retained samples have nearly identical rewards, the normalized advantages provide little useful preference signal;IntraPtherefore keeps trajectories that maintain reward contrast within each group.

Inter-Group Pruning (InterP).

InterPprunes at the level of entire refinement groups. Each failed node at turnsscreates a feedback-conditioned prompt at turns+1s+1, and that prompt produces a group of child refinements. Instead of selecting individual trajectories within every group,InterPselects which refinement groups to retain. The intuition is that not all feedback contexts are equally useful for optimization. Some failed attempts lead to refinement groups where all children behave similarly: they may all solve the task, or they may all fail. Such groups provide limited relative learning signal because there is little contrast among candidates generated from the same feedback-conditioned prompt. Other failed attempts lead to mixed refinement groups, where some children improve substantially while others still fail. These groups are more informative because they reveal which refinements should be preferred under the same feedback context. Based on this intuition,InterPranks refinement groups by their within-group reward variance. Let𝒢s\mathcal{G}_{s}denote the set of child groups induced by failed nodes at turnss. Given an across-group budgetbgrpb_{\mathrm{grp}},InterPretains thebgrpb_{\mathrm{grp}}groups with the largest within-group reward variance:𝒢s′=TopBg∈𝒢sbgrp​(Var⁡({r⁡(u):u∈g}))\mathcal{G}^{\prime}_{s}=\mathrm{TopB}_{g\in\mathcal{G}_{s}}^{b_{\mathrm{grp}}}\left(\mathrm{Var}\big(\{r(u):u\in g\}\big)\right). The remaining groups, along with all of their descendants, are discarded.

This pruning rule is aligned with the group-relative nature of GRPO. Since advantages are computed by comparing rewards within a group, high-variance groups provide stronger preference information than groups whose rewards are nearly uniform. In contrast toIntraP, which keeps informative trajectories within each group,InterPallocates optimization budget across feedback-conditioned prompts by keeping the refinement groups that appear most informative for learning. InApp. H, we provide a discussion on the practical cost of rollout tree generation.

4Experiments

In this section, we first provide an overview of metrics and evaluation protocol used in our experiments, followed by detailed results, including ablation studies. We provide implementation details including inference hyper-parameters inApp. F.

Metrics and Evaluation Protocol.Reflexion(Shinn et al., 2023)is a widely adopted purely inference-time multi-turn framework: the model parameters are not updated, but the model iteratively refines its solution by conditioning on prior attempts, execution feedback from visible test cases, and self-generated reflections (seeApp. Efor details). To assess refinement and self-correction, we integrate all models into the same Reflexion scaffold and reportpass@1underKK-iteration evaluation. We denote these settings as Iter-KK: Iter-1 corresponds to standard input-output prompting without feedback, while at iterationk>1k>1the model revises its solution using a sliding-window history consisting of the previous output and its corresponding visible-test feedback. Unless otherwise specified, we use Iter-3 as the default multi-turn evaluation setting. The agent terminates early once all visible tests pass or when the maximum iteration budget is reached. Final solutions are evaluated on hidden test cases, and the resultingpass@1is reported. Each experiment is repeated three times, and we report the mean and standard deviation ofpass@1across runs.

HumanEval and MBPP.

We evaluate OLMo-2-1124-7B-Instruct(OLMo et al., 2025)and Qwen3 (1.7B, 4B)(Yang et al., 2025), each fine-tuned on 1,000 samples randomly drawn from KodCode(Xu et al., 2025b)and evaluated on HumanEval(Chen et al., 2021)and MBPP(Austin et al., 2021)using the evaluation protocol described above. KodCode is chosen for its minimal overlap with evaluation benchmarks111Refer to Section 3.2 of KodCode(Xu et al., 2025b)for details on contamination analysis.. We reportpass@1under single-iteration (Iter-1) and three-iteration (Iter-3) Reflexion settings inFig. 3; in all experiments,MurphyusesMaRSreward propagation.

Iter-1 evaluates one-shot generation and does not exercise feedback-conditioned refinement, so we treat it primarily as a sanity check that multi-turn training does not substantially degrade direct generation. In this setting,Murphyis competitive with the strongest baseline with one small regression on Qwen3-1.7B MBPP (-1.27 pp). The primary evaluation for self-correction is Iter-3, where models can condition on execution feedback and revise their previous attempts. Under this setting,Murphyoutperforms all baselines, with gains up to+6.20 ppover the strongest comparator. The largest improvements occur on OLMo-2-1124-7B-Instruct (+6.20 ppon HumanEval and+6.07 ppon MBPP), suggesting that weaker models benefit more from structured multi-turn credit assignment. Overall, these results indicate that retrospective reward propagation primarily improves feedback-conditioned refinement while largely preserving one-shot generation performance.

Figure 3:HumanEval and MBPP pass@1 across Reflexion iterations for three base models.Rows correspond to benchmarks (HumanEval, MBPP) and columns to base models (OLMo-2-1124-7B-Instruct, Qwen3-1.7B, Qwen3-4B). Within each cell, we compareMurphy(ours) against the Base model, the multi-turn baseline GRPO-MT, and the execution-feedback methodμ\muCode(Jain et al., 2025). Annotations aboveMurphy’s bars report the absolute percentage-point difference betweenMurphyand the strongest non-Murphybaseline within the same benchmark, model, and Reflexion iteration (green: improvement, red: regression).Murphyoffers superior performance in Iter-3 settings, where feedback-conditioned self-correction is directly evaluated. Detailed results inTab. 4.

LiveCodeBench-v6.

We additionally includeReVeal(Jin et al., 2026), a concurrent multi-turn RL framework that evolves code generation through self-verification and tool-based evaluation, in our most rigorous setting: the contamination-resistant LiveCodeBench-v6 benchmark with Qwen3-4B (Fig. 1). Base, GRPO-MT,μ\muCode, andMurphyare evaluated under the same Reflexion scaffold. ForReVeal, we instead use its official repository, hyperparameters, and native multi-turn generation-verification scaffold with 3 training turns. This givesReVealits intended evaluation setting, since its inference procedure relies on a method-specific generation prompt, output format, and parser. Although the prompt and parser differ from Reflexion, the underlying scaffolds are conceptually similar: both condition on prior incorrect responses and environment feedback over multiple turns to refine subsequent attempts. All methods are compared at 1, 3, and 5 inference turns.Murphyoutperforms bothμ\muCodeandReVealacross all difficulty subsets, with the largest gain (+4.20 pp) on the Hard subset at Iter-5, where self-correction matters most. Detailed results, including scaffold robustness analyses, are available inF.4.1(Murphy’s gains remain consistent across multiple scaffolds).

Scope ofReVealcomparison.

ReVeal’s co-trained self-verifier substantially increases per-experiment training cost: ~23 hours end-to-end for Qwen3-4B versus ~11 hours forMurphyon matched hardware (8×\timesH100s). We therefore include ReVeal as a baseline only in our most rigorous setting (LCB-v6 with Qwen3-4B,Fig. 1), where contamination resistance and difficulty stratification make the comparison most informative. ExtendingReVealacross all model–benchmark cells inFig. 3was prohibitive under our compute budget, mirroring our handling of RLEF(Gehring et al., 2025).

Additional Experiments.Further analysis is provided in the appendices: effect of training data size (App.F.4.2), and training beyond two turns (App.F.4.3). App.Dconfirms thatMurphy’s gains generalize across different iterative scaffolds.

4.1Reward Propagation Ablation:MaRSvs.MeRS

We ablate the two reward propagation strategies,MaRSandMeRS, introduced inSubsec. 3.1, training Qwen3-1.7B and OLMo-2-1124-7B-Instruct on 1K KodCode subset dataset under each strategy (Tab. 1).MaRSoffers competitive or superior performance overMeRSin iter-3 setting independent of the discount factorγ\gamma. The gap stems from how each strategy handles non-binary rewards:MeRSaverages over child rewards, diluting the signal when few refinements score highly, whereasMaRSpropagates the strongest outcome, letting rare high-reward trajectories dominate the update. This advantage is most pronounced under sparse rewards and diminishes as rewards become binary.

Table 1:MaRSvs.MeRSreward propagation.MaRSis highlighted.Δ3\Delta_{3}is relative toMeRS(γ=1\gamma=1) Iter-3 within each model block.

4.2Pruning Strategies Ablation:IntraPvs.InterP

We ablate the two pruning strategies, Intra-Group (IntraP) and Inter-Group (InterP), introduced inSubsec. 3.1, training Qwen3-1.7B on 1,000 KodCode samples under each strategy (Tab. 2). Both strategies reduce the number of retained trajectories per rollout tree (per prompt) from 72 to roughly half, lowering the per-step optimization cost without altering the rollout procedure itself. The two strategies differ in which trajectories they retain, and this leads to different multi-turn behaviors:IntraPimproves single-turn performance (Iter-1) on both benchmarks but regresses by2–2.3 ppat Iter-3, whereasInterPpreserves multi-turn performance within0.4–0.8 ppof unprunedMurphy.InterPtherefore offers the better cost-quality trade-off, retaining the multi-turn self-correction signal that pruning could otherwise discard.

Table 2:Pruned variants ofMurphyon Qwen3-1.7B. All variants generate 72 rollout trajectories per query before pruning; theRetainedcolumn reports the number of trajectories kept after pruning per prompt, which determines how many sequence-level terms enter the GRPO loss.Δ3\Delta_{3}comparesIter-3against unprunedMurphy(MaRS)Iter-3;redindicates regression.Efficiency gains.To localize the speedup, we profile per-step timing averaged across an end-to-endInterPrun against unprunedMurphyunder matched configurations (Qwen3-1.7B, 8×\timesH100 GPUs, 72 rollouts per prompt: 8 in turn-1 and up to 64 in turn-2); seeTab. 8. Generation (vLLM) and reward computation (code execution) are essentially unchanged (−0.3%-0.3\%and−8.0%-8.0\%respectively), because pruning is appliedafterrollouts are collected. The savings concentrate entirely in the optimization phase, where pruning reduces the number of sequences entering gradient computation: optimization time drops by74.1%\mathbf{74.1\%}, yielding a21.4%\mathbf{21.4\%}reduction in average per-step training time. This confirms thatInterP’s computational benefit comes from avoiding gradient updates on uninformative trajectories, not from shortening rollouts, an important property, since aggressive rollout-time pruning would risk discarding feedback signal that later turns rely on.

5Related Work

LLM Agents for Software Development.

Recent studies(Jiang et al., 2025;Zhong et al., 2024)investigate LLM agents for code generation, bug fixing, and code migration. A central factor behind their progress is inference-time iterative frameworks(Shinn et al., 2023;Lingam et al., 2025), which leverage execution feedback and self-reflection to refine candidate programs(Yang et al., 2024;Xia et al., 2025). While such methods enhance inference pipelines, they leave the base model unchanged. In contrast, our work improves the model itself through training-time optimization, strengthening the reasoning and self-correction abilities that agentic frameworks depend on.

RLVR for LLM Reasoning.GRPO(Shao et al., 2024)renewed interest in RL as an efficient alternative to PPO(Schulman et al., 2017)for post-training LLMs on verifiable objectives, and follow-up variants(Yue et al., 2025;Yu et al., 2025;Yuan et al., 2025;Zheng et al., 2025)improve stability, convergence, or shift optimization from token-level to sequence-level. These methods, however, remain single-turn. A separate line of work incorporates execution feedback during training but does so by introducing auxiliary machinery:μ\muCode(Jain et al., 2025)trains a separate verifier model that scores candidate solutions and relabels trajectories via expert iteration; RLEF(Gehring et al., 2025)optimizes multi-turn rollouts with a PPO critic over execution-based rewards; andReVeal(Jin et al., 2026)co-trains a self-verifier within the policy and uses turn-aware optimization to drive co-evolution of generation and verification. Concurrent work on multi-turn GRPO for tool-use agents(Zeng et al., 2025)introduces turn-level rewards based on predefined interaction signals (e.g., tool-execution success). In code generation, execution can provide a scalar reward at each turn, but such intermediate rewards are often sparse or myopic: a failed program may receive low reward while exposing a stack trace, edge case, or failing test that enables a later correction. Thus,Murphyuses retrospective tree-based propagation to assign credit according to downstream refinements induced by each feedback context. Concurrent to our work,TreeGRPO(Ding and Ye, 2026)also introduces tree-structured credit assignment for GRPO-style post-training, but in a different setting: visual generative models, where the tree branches within a single denoising trajectory and leaf advantages are propagated to denoising edges. In contrast,Murphyconstructs feedback-conditioned trees across multi-turn code-refinement attempts, propagates execution rewards before local group-relative normalization, and studies both optimistic (MaRS) and expectation-like (MeRS) backups. We provide a detailed comparison in AppA.1. Across these lines of work,Murphydiffers by extending GRPO to multi-turn code generation through feedback-conditioned rollout trees and retrospective reward propagation, without a learned verifier, an auxiliary critic, or predefined dense turn-level rewards. Empirically, this minimal extension matches or outperformsμ\muCodeandReVealacross our benchmarks (Sec. 4). SeeApp. Afor an extended discussion.

6Conclusion & Limitations

We introducedMurphy, a multi-turn extension of GRPO that addresses the credit-assignment mismatch in feedback-conditioned code generation.Murphyconstructs feedback-conditioned rollout trees, then propagates rewards backward so that earlier failed attempts receive credit for the informative feedback they surface. Across three benchmarks and two model families,Murphydelivers up to 6% absolute pass@1 gains over the strongest prior multi-turn execution-feedback methods, with the largest improvements on harder problems where iterative self-correction matters most. These results suggest that the standard GRPO recipe can be made feedback-aware through a structural change to credit assignment, without learned verifiers or auxiliary critics.Limitations:Murphy’s multi-turn rollouts increase training cost relative to GRPO.InterPmitigates this cost, reducing average per-step training time, but multi-turn training remains more expensive due to rollout-tree expansion. In our experiments, we use vLLM for rollout generation; shared prefixes across parent and child nodes could further enable prefix caching and KV-cache reuse, which we leave to future work. Our evaluation focuses on code generation, where execution provides clean feedback. ExtendingMurphyto noisier feedback signals, longer refinement horizons, broader agentic tasks, and adaptive turn or rollout budgets is a natural next step.

References

• [1]A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker(2024)Back to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),L. Ku, A. Martins, and V. Srikumar (Eds.),pp. 12248–12267.External Links:Link,DocumentCited by:Appendix C.
• [2]J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton(2021)Program synthesis with large language models.External Links:2108.07732,LinkCited by:§4.
• [3]T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao(2024)MEDUSA: simple llm inference acceleration framework with multiple decoding heads.InProceedings of the 41st International Conference on Machine Learning,ICML’24.Cited by:Appendix H.
• [4]M. Chen, J. Tworek, and H. J. et. al.(2021)Evaluating large language models trained on code.External Links:2107.03374,LinkCited by:§4.
• [5]M. Chen, L. Sun, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen(2025)ReSearch: learning to reason with search for llms via reinforcement learning.External Links:2503.19470,LinkCited by:Appendix A.
• [6]DeepSeek-AI, D. Guo, and D. Y. et. al.(2025)DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning.External Links:2501.12948,LinkCited by:Appendix C.
• [7]Z. Ding and W. Ye(2026)TreeGRPO: tree-advantage GRPO for online RL post-training of diffusion models.InThe Fourteenth International Conference on Learning Representations,External Links:LinkCited by:§A.1,§A.1,§5.
• [8]J. Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve(2025)RLEF: grounding code LLMs in execution feedback with reinforcement learning.InForty-second International Conference on Machine Learning,External Links:LinkCited by:Appendix A,§1,§4,§5.
• [9]A. K. Jain, G. Gonzalez-Pumariega, W. Chen, A. M. Rush, W. Zhao, and S. Choudhury(2025)Multi-turn code generation through single-step rewards.InForty-second International Conference on Machine Learning,External Links:LinkCited by:Appendix A,Table 4,Table 4,Table 4,§1,Figure 3,Figure 3,§5.
• [10]J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim(2025)A survey on large language models for code generation.ACM Trans. Softw. Eng. Methodol..External Links:ISSN 1049-331X,Link,DocumentCited by:Appendix A,§5.
• [11]B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han(2025)Search-r1: training LLMs to reason and leverage search engines with reinforcement learning.InSecond Conference on Language Modeling,External Links:LinkCited by:Appendix A.
• [12]Y. Jin, K. Xu, H. Li, X. Han, Y. Zhou, C. Li, and J. Bai(2026)ReVeal: self-evolving code agents via reliable self-verification.InThe Fourteenth International Conference on Learning Representations,External Links:LinkCited by:§1,§4,§5.
• [13]P. Khandelwal, E. Liebman, S. Niekum, and P. Stone(2016)On the analysis of complex backup strategies in monte carlo tree search.InProceedings of the 33rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol.48,pp. 1319–1328.Cited by:§A.1,§A.1,§3.
• [14]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica(2023)Efficient memory management for large language model serving with pagedattention.External Links:2309.06180,LinkCited by:Appendix H.
• [15]V. Lingam, B. O. Tehrani, S. Sanghavi, G. Gupta, S. Ghosh, L. Liu, J. Huan, and A. Deoras(2025)Enhancing language model agents using diversity of thoughts.InThe Thirteenth International Conference on Learning Representations,External Links:LinkCited by:Appendix A,Appendix D,§1,§1,§5.
• [16]T. OLMo, P. Walsh, L. Soldaini, and D. G. et. al.(2025)2 olmo 2 furious.External Links:2501.00656,LinkCited by:§4.
• [17]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov(2017)Proximal policy optimization algorithms.External Links:1707.06347,LinkCited by:Appendix A,Appendix C,§1,§5.
• [18]M. Senghaas, F. Obeid, S. Jaghouar, W. Brown, J. M. Ong, A. Baker, J. Mattern, D. Auras, J. Straube, M. Basra, A. Ismail, and J. Hagemann(2025)PRIME-RL: async & decentralized RL training at scale.InNeurIPS Workshop on GPU-Accelerated and Scalable Optimization,External Links:LinkCited by:Appendix H.
• [19]A. Setlur, M. Y. R. Yang, C. V. Snell, J. Greer, I. Wu, V. Smith, M. Simchowitz, and A. Kumar(2026)E3: learning to explore enables extrapolation of test-time compute for LLMs.InThe Fourteenth International Conference on Learning Representations,External Links:LinkCited by:§1.
• [20]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y.K. Li, Y. Wu, and D. Guo(2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.Vol.abs/2402.03300.External Links:LinkCited by:Appendix A,§1,§2,§5.
• [21]N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao(2023)Reflexion: language agents with verbal reinforcement learning.InAdvances in Neural Information Processing Systems,A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.),Vol.36,pp. 8634–8652.Cited by:Appendix A,Appendix D,Appendix E,§F.3,Figure 1,Figure 1,§1,§1,§4,§5.
• [22]F. Tajwar, G. Zeng, Y. Zhou, Y. Song, D. Arora, Y. Jiang, J. Schneider, R. Salakhutdinov, H. Feng, and A. Zanette(2026)Maximum likelihood reinforcement learning.InThe 1st Workshop on Scaling Post-training for LLMs,External Links:LinkCited by:§A.1.
• [23]L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec(2020)TRL: Transformer Reinforcement Learning.GitHub.Note:https://github.com/huggingface/trlCited by:Appendix F.
• [24]X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig(2025)OpenHands: an open platform for AI software developers as generalist agents.InThe Thirteenth International Conference on Learning Representations,External Links:LinkCited by:§1.
• [25]R. J. Williams(1992)Simple statistical gradient-following algorithms for connectionist reinforcement learning.Mach. Learn.8(3–4),pp. 229–256.External Links:ISSN 0885-6125,Link,DocumentCited by:Appendix C.
• [26]C. S. Xia, Y. Deng, S. Dunn, and L. Zhang(2025)Demystifying llm-based software engineering agents.Proc. ACM Softw. Eng.2(FSE).External Links:Link,DocumentCited by:Appendix A,§5.
• [27]Y. E. Xu, Y. Savani, F. Fang, and Z. Kolter(2025)Not all rollouts are useful: down-sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818.Cited by:§3.1.
• [28]Z. Xu, Y. Liu, Y. Yin, M. Zhou, and R. Poovendran(2025)Kodcode: a diverse, challenging, and verifiable synthetic dataset for coding.arXiv preprint arXiv:2503.02951.Cited by:§4,footnote 1.
• [29]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu(2025)Qwen3 technical report.External Links:2505.09388,LinkCited by:Appendix C,§F.2,§4.
• [30]J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press(2024)SWE-agent: agent-computer interfaces enable automated software engineering.InThe Thirty-eighth Annual Conference on Neural Information Processing Systems,External Links:LinkCited by:Appendix A,§1,§5.
• [31]Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang(2025)DAPO: an open-source llm reinforcement learning system at scale.External Links:2503.14476,LinkCited by:Appendix A,Appendix C,§5.
• [32]Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan(2025)What’s behind ppo’s collapse in long-cot? value optimization holds the secret.External Links:2503.01491,LinkCited by:Appendix A,§5.
• [33]Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, X. Wei, X. Yu, G. Liu, J. Liu, L. Liu, H. Lin, Z. Lin, B. Ma, C. Zhang, M. Zhang, W. Zhang, H. Zhu, R. Zhang, X. Liu, M. Wang, Y. Wu, and L. Yan(2025)VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks.External Links:2504.05118,LinkCited by:Appendix A,Appendix C,§5.
• [34]S. Zeng, Q. Wei, W. Brown, O. Frunza, Y. Nevmyvaka, Y. K. Zhao, and M. Hong(2025)Reinforcing multi-turn reasoning in LLM agents via turn-level credit assignment.InICML 2025 Workshop on Computer Use Agents,Cited by:§5.
• [35]S. Zhang, Y. Dong, J. Zhang, J. Kautz, B. Catanzaro, A. Tao, Q. Wu, Z. Yu, and G. Liu(2025)Nemotron-research-tool-n1: tool-using language models with reinforced reasoning.arXiv preprint arXiv:2505.00024.Cited by:Appendix A.
• [36]C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin(2025)Group sequence policy optimization.External Links:2507.18071,LinkCited by:Appendix A,§5.
• [37]L. Zhong, Z. Wang, and J. Shang(2024)Debug like a human: a large language model debugger via verifying runtime execution step by step.InFindings of the Association for Computational Linguistics ACL 2024,pp. 851–870.Cited by:Appendix A,§5.
• [38]A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang(2024)Language agent tree search unifies reasoning, acting, and planning in language models.InForty-first International Conference on Machine Learning,External Links:LinkCited by:Appendix D,§1,§1.
• [39]R. Zhuang, T. Vu, A. Dimakis, and M. Sathiamoorthy(2025)Improving multi-turn tool use with reinforcement learning.Note:Accessed: 2025-04-17Cited by:Appendix A.

1. 1Introduction
2. 2Background: GRPO
3. 3Proposed Method:Murphy1. 3.1Pruning Strategies inMurphy
4. 4Experiments1. 4.1Reward Propagation Ablation:MaRSvs.MeRS 2. 4.2Pruning Strategies Ablation:IntraPvs.InterP
5. 5Related Work
6. 6Conclusion & Limitations
7. References
8. AExtended Related Work1. A.1Relation to Tree-Structured GRPO and Backup Theory
9. BGRPO: Objective and Additional Details
10. CMulti-turn Rollout Formalism (Detailed)
11. DSensitivity to Multi-Iteration Scaffolds
12. EReflexion
13. FImplementation Details1. F.1Model Size and Compute Budget 2. F.2Hyperparameters 3. F.3Package Parameters 4. F.4Additional Experiments
14. GPrompt Examples
15. HEfficiency Gains
16. IBroader Impact

Appendix AExtended Related Work

LLM Agents for Software Development.

Recent works[10,37]have explored LLM agents for programming tasks such as code generation, bug fixing, and code migration. A key driver of progress in these domains has been inference-time iterative frameworks[21,15], which leverage execution feedback to generate self-reflections for refining candidate programs[30,26]. While these approaches underscore the value of iterative feedback and scaffolding, they primarily enhance the inference pipeline rather than the underlying model. Our work takes a complementary direction: we improve the reasoning and self-correction abilities of LLMs themselves through training-time optimization, thereby strengthening the base models that agentic frameworks depend on.

Reinforcement Learning with Verifiable Rewards for LLM Reasoning.

Reinforcement learning (RL) is a popular paradigm for post-training LLMs to improve reasoning and align outputs with verifiable objectives. GRPO[20]revived interest in RL as an efficient alternative to PPO[17], offering comparable reasoning performance with far lower computational cost. Subsequent variants[33,31,32,36]focus on stabilizing training, improving convergence, or shifting optimization from token-level to sequence-level. However, these algorithms remain tailored to single-turn tasks, optimizing models to produce one-shot completions without iterative refinement. Recent RLVR methods extend RL to multi-turn agents across domains such as search[5,11], tool use[39,35], and code generation[9,8]. These methods typically compute advantage by summing outcome and turn-level rewards, which limits temporal credit propagation. In contrast,Murphydelays credit assignment until a trajectory is complete, propagating rewards backward from successful states using a structured credit assignment criterion that preserves temporal consistency. Our work is most closely related toμ\muCode[9]and RLEF[8], which also train LLMs with execution feedback.μ\muCode jointly trains a generator and a learned verifier that scores multi-turn code solutions, while RLEF refines generations via PPO grounded in execution results. However, both approaches require auxiliary value functions or verifier LLMs, significantly increasing computational overhead and data acquisition cost.μ\muCode further depends on its verifier at inference for Best-of-N selection, introducing additional latency. These design choices make direct comparison impractical and obscure the effect of the RL formulation itself. In contrast,Murphyachieves comparable grounding in execution feedback and iterative refinement by extending GRPO to the multi-turn setting, while preserving its simplicity, efficiency, and architectural minimalism.

A.1Relation to Tree-Structured GRPO and Backup Theory

TreeGRPO[7]is closely related in spirit toMurphy: both methods replace trajectory-level credit with tree-structured credit assignment before applying a GRPO-style update. However, the two methods differ in the source of branching, the object being propagated, and the point at which normalization is applied.TreeGRPOtargets diffusion/flow post-training, where branching occurs within a single denoising trajectory. It first computes normalized leaf advantages and then propagates them backward to obtain per-edge advantages using a log-probability weighted average over child branches.Murphyinstead targets multi-turn code generation, where branching is induced by executor feedback: failed programs create feedback-conditioned prompts whose descendants are later refinements. We therefore propagate execution rewards through the tree and then compute local group-relative advantages within each feedback-conditioned response group.

Tree-structured GRPO methods differ primarily in what quantity is backed up and how descendants are aggregated.TreeGRPO[7]propagates leaf advantages backward through a denoising tree using a policy-weighted average:

A⁡(v)=∑u∈C⁡(v)wv​(u)​A​(u),wv​(u)=πθold​(u∣v)∑u′∈C⁡(v)πθold​(u′∣v).A(v)=\sum_{u\in C(v)}w_{v}(u)A(u),\qquad w_{v}(u)=\frac{\pi_{\theta_{\mathrm{old}}}(u\mid v)}{\sum_{u^{\prime}\in C(v)}\pi_{\theta_{\mathrm{old}}}(u^{\prime}\mid v)}.In contrast,Murphypropagates execution rewards through a feedback-conditioned refinement tree and computes local GRPO advantages only after this reward propagation step. This leads to two complementary backups:

MeRS:r~​(v)=r(v)+γ⋅1|C⁡(v)|∑u∈C⁡(v)r~(u)Z⁡(v),\textsc{MeRS}:\qquad\tilde{r}(v)=\frac{r(v)+\gamma\cdot\frac{1}{|C(v)|}\sum_{u\in C(v)}\tilde{r}(u)}{Z(v)},and

MaRS:r~​(v)=max⁡(r⁡(v),maxu∈C⁡(v)⁡r~​(u)).\textsc{MaRS}:\qquad\tilde{r}(v)=\max\left(r(v),\max_{u\in C(v)}\tilde{r}(u)\right). HereZ⁡(v)Z(v)denotes the local normalization factor used in the reward backup. Unlike the policy-weighted averaging rule inTreeGRPO, this normalization factor is tied to the realized refinement subtree belowvv. It may therefore depend on the number of sampled descendants, the rollout depth, and the feedback-induced branching pattern. Consequently,MeRSshould not be interpreted as the same estimator as theTreeGRPOadvantage backup. Rather, it is a sample-dependent reward propagation rule that smooths execution rewards over the refinements actually generated under a given feedback context.

MeRSis closest in spirit to the aggregation rule inTreeGRPO: both combine information from child branches before applying a GRPO-style update. The analogy, however, is only structural. InTreeGRPO, the backed-up quantity is a normalized advantage and the weights are induced by the old policy over denoising branches. InMurphy, the backed-up quantity is an execution reward, and the normalization termZ⁡(v)Z(v)is part of the sampled reward-propagation procedure.

Averaging over child branches can be viewed as a variance-reducing operation conditional on a fixed realized child set. Specifically, ifC⁡(v)C(v)and weightswvw_{v}are fixed, and if child branch valuesXuX_{u}are conditionally independent with common varianceσ2\sigma^{2}, then

Var[∑u∈C⁡(v)wv(u)Xu|C(v),wv]=σ2∑u∈C⁡(v)wv(u)2.\mathrm{Var}\!\left[\sum_{u\in C(v)}w_{v}(u)X_{u}\,\middle|\,C(v),w_{v}\right]=\sigma^{2}\sum_{u\in C(v)}w_{v}(u)^{2}.For uniform weights this becomesσ2/|C⁡(v)|\sigma^{2}/|C(v)|, conditional on the realized child set. Since the refinement tree, the number of descendants, and the normalization termZ⁡(v)Z(v)are themselves determined by the sampled executor-feedback process, this conditional calculation should not be read as an unconditional variance or unbiasedness guarantee. Instead, it provides intuition for whyMeRSbehaves as a smoothing backup over the realized refinement tree, whereasMaRSbehaves as an optimistic backup that propagates the best observed execution reward.

MaRS, by contrast, is an optimistic best-descendant backup. For binary correctness rewards,

𝔼[maxu∈C⁡(v)R(u)]=Pr(∃u∈C(v):R(u)=1),\mathbb{E}\!\left[\max_{u\in C(v)}R(u)\right]=\Pr\!\left(\exists u\in C(v):R(u)=1\right),and, under conditional independence, this equals

1−∏u∈C⁡(v)(1−pu)1-\prod_{u\in C(v)}(1-p_{u})This objective is well aligned with sparse-reward code self-correction: a failed attempt should receive credit if the feedback it induces enables at least one later refinement to solve the task. Prior analysis of Monte Carlo tree-search backups similarly shows that max-style backups can preserve rare high-value trajectories that averaging may dilute[13].

This distinction also clarifies the roles ofMeRSandMaRS.MeRSis closest toTreeGRPO’s aggregation rule: both estimate the value of an internal node by averaging over sampled descendants, and both can be interpreted as reducing variance relative to relying on a single sampled continuation.MaRS, by contrast, is an optimistic best-descendant backup. This is motivated by the sparse and existential nature of code self-correction: a failed attempt can be useful if it exposes feedback that enables at least one later refinement to solve the task. Prior work on backup strategies in Monte Carlo tree search similarly observes that averaging can drown out rare high-value trajectories, while max-style backups can be beneficial when successful trajectories are scarce[13]. Thus,MeRSprovides a lower-variance expectation-like backup, whereasMaRSis better aligned with best-of-kkand pass@kkstyle success criteria in sparse-reward code generation.

Recent work onMaxRL[22]provides a complementary perspective: in correctness-based tasks, likelihood-oriented objectives emphasize successful rollouts and higher-order pass@kkevents rather than only expected pass@1 reward. While MaxRL is developed for single-turn binary-outcome settings, it supports the intuition behindMaRS: when success is rare, training signals that preserve and amplify successful descendants can be preferable to averaging them away. Our contribution is to instantiate this principle in a feedback-conditioned multi-turn GRPO setting, where earlier failed attempts are credited according to the downstream refinements they make possible.

Appendix BGRPO: Objective and Additional Details

Notation.

We denote the model policy byπθ(⋅∣⋅)\pi_{\theta}(\cdot\mid\cdot)and the reference (older) policy byπθold(⋅∣⋅)\pi_{\theta_{\text{old}}}(\cdot\mid\cdot). LetGGbe the number of generations per prompt,𝒫⁡(Q)\mathcal{P}(Q)the distribution over input prompts/questionsQQ, andOOthe output space. For a given promptq∼𝒫⁡(Q)q\sim\mathcal{P}(Q), the reference policy produces a set ofGGresponses, forming a response group{oq,1,oq,2,…,oq,G}\{o_{q,1},o_{q,2},\ldots,o_{q,G}\}. Each generationoq,i∈Oo_{q,i}\in Ocorresponds to a full output trajectory, whereoq,i,to_{q,i,t}denotes thett-th token andoq,i,<to_{q,i,<t}the prefix up to (but excluding) tokentt. We write|oq,i||o_{q,i}|for the sequence length of theii-th generation. For a given promptqq, the reward model assigns a scalar score to each response in the group, yielding𝐫q={rq,1,rq,2,…,rq,G}\mathbf{r}_{q}=\{r_{q,1},r_{q,2},\ldots,r_{q,G}\}. Moreover, for promptqq, the advantage associated with thett-th token of theii-th generation is defined asA^q,i,t=(rq,i−μ⁡(𝐫q))/σ⁡(𝐫q)\hat{A}_{q,i,t}=(r_{q,i}-\mu(\mathbf{r}_{q}))/\sigma(\mathbf{r}_{q}), whereμ⁡(𝐫q)\mu(\mathbf{r}_{q})andσ⁡(𝐫q)\sigma(\mathbf{r}_{q})denote the mean and standard deviation of the group rewards, respectively. Finally,DKL(πθ||πθref)D_{\mathrm{KL}}(\pi_{\theta}||\pi_{\theta_{\text{ref}}})denotes the KL divergence between the current and reference policies, computed over all tokens in the generated sequences. The GRPO training objective is presented inApp. B.

Definition 1.(GRPO Objective)𝒥⁡(θ)=\displaystyle\mathcal{J}(\theta)=\𝔼q∼𝒫⁡(Q),{o(q,i)}i=1G∼πθold​(O|q)[1G∑i=1G1|oq,i|∑t=1|oq,i|min(Rθ(q,i,t)A^q,i,t,\displaystyle\mathbb{E}_{q\sim\mathcal{P}(Q),\ \{o_{(q,i)}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(O|q)}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{q,i}|}\sum_{t=1}^{|o_{q,i}|}\min\Big(R_{\theta}(q,i,t)\hat{A}_{q,i,t},clip(Rθ(q,i,t),1−ϵ,1+ϵ)A^q,i,t)]−βDKL(πθ∥πref)\displaystyle\text{clip}\Big(R_{\theta}(q,i,t),\,1-\epsilon,\,1+\epsilon\Big)\hat{A}_{q,i,t}\Big)\Big]-\beta D_{\text{KL}}(\pi_{\theta}\,\|\,\pi_{\text{ref}})Where,Rθ​(q,i,t)\displaystyle R_{\theta}(q,i,t)=πθ​(oq,i,t∣q,oq,i,<t)πθold​(oq,i,t∣q,oq,i,<t)\displaystyle=\frac{\pi_{\theta}(o_{q,i,t}\mid q,o_{q,i,<t})}{\pi_{\theta_{\text{old}}}(o_{q,i,t}\mid q,o_{q,i,<t})}A^q,i,t\displaystyle\hat{A}_{q,i,t}=(rq,i−μ⁡(𝐫q))σ⁡(𝐫q)\displaystyle=\frac{(r_{q,i}-\mu(\mathbf{r}_{q}))}{\sigma(\mathbf{r}_{q})}

Appendix CMulti-turn Rollout Formalism (Detailed)

This appendix section provides a detailed treatment of the rollout-tree construction and indexing used throughout the main paper.

We define a feedback-conditioned rollout tree that captures how model generations evolve across multiple turns of interaction with the environment. Letssdenote the turn index,SSthe total number of turns, andGsG_{s}the number of generations per prompt at turnss.

Turn 1:In the first turn (s=1s=1), the model receives an input promptq(1)q_{(1)}sampled from𝒫⁡(Q)\mathcal{P}(Q)and generatesG1G_{1}candidate programs which forms a response group{o{q(1),1},o{q(1),2},…,o{q(1),G1}}\{o_{\{q_{(1)},1\}},o_{\{q_{(1)},2\}},\ldots,o_{\{q_{(1)},G_{1}\}}\}. Note that{o{q(1),j}}j=1G1∼πθold(⋅∣q(1))\{o_{\{q_{(1)},j\}}\}_{j=1}^{G_{1}}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q_{(1)}). Each generation is executed against its associated test suite, producing a numerical rewardr{q(1),j}r_{\{q_{(1)},j\}}and environment feedbackf{q(1),j}f_{\{q_{(1)},j\}}. The reward represents the proportion of test cases passed. The environment feedback contains the specific unit tests that passed or failed, along with any corresponding error messages. TheseG1G_{1}generations form the first layer of output nodes in the rollout tree.

Turn 2:For each generationjjin turn 1 that fails to achieve the maximum reward (which equals 1 in our setting, since it represents the proportion of test cases passed), the corresponding feedback is appended to the original prompt and the prior output from turn11to form a feedback-conditioned prompt:q(2,j)=[q(1),o{q(1),j},f{q(1),j}]q_{(2,j)}=[\ q_{(1)},\ o_{\{q_{(1)},j\}},\ f_{\{q_{(1)},j\}}\ ]where[⋅][\,\cdot\,]denotes textual concatenation. The model is then re-invoked to generateG2G_{2}new candidate solutions:{o{q(2,j),k}}k=1G2∼πθold(⋅∣q(2,j))\{o_{\{q_{(2,j)},k\}}\}_{k=1}^{G_{2}}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q_{(2,j)}). Each of these generations is evaluated to obtain(r{q(2,j),k},f{q(2,j),k})(r_{\{q_{(2,j)},k\}},f_{\{q_{(2,j)},k\}}), which denote the reward and feedback. These output generations represent refinements of their corresponding parent outputso{q(1),j}o_{\{q_{(1)},j\}}and collectively form the second layer of the rollout tree.

Turnss:Building on the previous turn, this procedure extends recursively to any turns∈{1,…,S−1}s\in\{1,\dots,S-1\}. We definei[1:s]=i1,…,isi_{[1:s]}=i_{1},\dots,i_{s}as a sequence of branch indices that trace a specific path through the tree, whereiji_{j}indicates theii’th candidate that was selected at turnjj. Similarly,i[1:1]i_{[1:1]}denotesi1i_{1}. For each generationoq(s,i[1:s−1]),iso_{{q_{(s,i_{[1:s-1]})},i_{s}}}that fails to achieve the maximum reward, we construct a feedback-conditioned prompt:

q(s+1,i[1:s])=[q(s,i[1:s−1]),oq(s,i[1:s−1]),is,fq(s,i[1:s−1]),is]\displaystyle q_{(s+1,i_{[1:s]})}=[q_{(s,i_{[1:s-1]})},\,o_{{q_{(s,i_{[1:s-1]})},i_{s}}},\,f_{{q_{(s,i_{[1:s-1]})},i_{s}}}]where[⋅][\cdot]denotes textual concatenation. The model is then re-invoked to generateGs+1G_{s+1}new candidate solutions:

{o{q(s+1,i[1:s]),k}}k=1Gs+1∼πθold(⋅∣q(s+1,i[1:s]))\displaystyle\{o_{\{q_{(s+1,i_{[1:s]})},k\}}\}_{k=1}^{G_{s+1}}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q_{(s+1,i_{[1:s]})})Each candidate is evaluated to obtain its reward and feedback(r{q(s+1,i[1:s]),k},f{q(s+1,i[1:s]),k})(r_{\{q_{(s+1,i_{[1:s]})},k\}},\,f_{\{q_{(s+1,i_{[1:s]})},k\}}). The resulting generations,{o{q(s+1,i[1:s]),k}}k=1Gs+1\{o_{\{q_{(s+1,i_{[1:s]})},k\}}\}_{k=1}^{G_{s+1}}, form the child nodes of parent nodeo{q(s,i[1:s−1]),is}o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}.

A complete path from the root (the initial prompt at turn 1) to a leaf at turnSS(final output) can be expressed as:

q(1)→oq(1),i1→q(2,i[1:1])→oq(2,i[1:1]),i2→⋯→o{q(S,i[1:S−1]),iS}\displaystyle q_{(1)}\rightarrow o_{{q_{(1)},i_{1}}}\rightarrow q_{(2,i_{[1:1]})}\rightarrow o_{{q_{(2,i_{[1:1]})},i_{2}}}\rightarrow\cdots\rightarrow o_{\{q_{(S,i_{[1:S-1]})},i_{S}\}}where the indicesi1,…,iSi_{1},\dots,i_{S}specify which branch is taken at each turn. Leaf nodes at turnSSrepresent the final generations obtained after completing all refinement steps. Once rewards for all turns are computed, the rewards from these terminal nodes are propagated backward through their ancestors according to the credit assignment strategies described below.

Credit assignment formalism.After all outputs and corresponding rewards are generated and the rollout tree is constructed, we focus on assigning credit from later turns back to earlier ones. To achieve this, we explore two distinct strategies.

Max Reward Strategy (MaRS):The Max Reward Strategy is defined recursively, proceeding from the final turn back to the root. Since the final turnSShas no children, the rewards at this turn remain unchanged. We then consider turns=S−1s=S-1. Leto{q(s,i[1:s−1]),is}o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}denote a generation at turns=S−1s=S-1with an associated rewardr{q(s,i[1:s−1]),is}r_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}. If this generation already achieves the maximum reward, it has no children; otherwise, its children are defined as:

C(o{q(s,i[1:s−1]),is})={o{q(s+1,i[1:s]),1},…,o{q(s+1,i[1:s]),GS}}\displaystyle C\big(o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}\big)=\{o_{\{q_{(s+1,i_{[1:s]})},1\}},\ \ldots,\ o_{\{q_{(s+1,i_{[1:s]})},G_{S}\}}\}The corresponding set of rewards are defined as

Cr(o{q(s,i[1:s−1]),is})={r{q(s+1,i[1:s]),1},…,r{q(s+1,i[1:s]),GS}}\displaystyle C_{r}\big(o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}\big)=\{r_{\{q_{(s+1,i_{[1:s]})},1\}},\ \ldots,\ r_{\{q_{(s+1,i_{[1:s]})},G_{S}\}}\}which defaults to zero if there are no children. We then update the reward as:

r{q(s,i[1:s−1]),is}\displaystyle r_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}=max(r{q(s,i[1:s−1]),is},max(Cr(o{q(s,i[1:s−1]),is)))\displaystyle=\text{max}\Bigg(r_{\{q_{(s,i_{[1:s-1]})},i_{s}\}},\ \text{max}\Big(C_{r}(o_{\{q_{(s,i_{[1:s-1]})},i_{s}})\Big)\Bigg) This strategy assigns each node the maximum of its own reward and the best reward among its descendants. Intuitively, the descendant maximum represents the best outcome achievable through refinement, while taking the outer maximum ensures a node’s credit never decreases; even when feedback fails to improve performance. This formulation captures the maximum progress achievable from any refinement path starting at that node. The procedure operates recursively in a backward pass: rewards are first updated for all nodes at turnS−1S-1based on their children’s values at turnSS. This process continues backward through the tree, with each turnssreceiving updated rewards based on the values from turns+1s+1, until reaching the root.

Mean Reward Strategy (MeRS):The Mean Reward Strategy follows the same recursive credit assignment structure as the Max Reward Strategy (MaRS), but differs in how rewards are propagated. Inspired by the return computation in REINFORCE[25],MeRSupdates each node’s reward by incorporating the discountedmeanof its children’s rewards.

For a generationo{q(s,i[1:s−1]),is}o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}at turnss, let𝕀unsolved\mathbb{I}_{\text{unsolved}}be an indicator that equals 1 if the problem remains unsolved at this node, and 0 otherwise. The update rule is:

r{q(s,i[1:s−1]),is}=r{q(s,i[1:s−1]),is}+γ⋅C¯r(o{q(s,i[1:s−1]),is})𝕀unsolved⋅(S−s)+1\displaystyle r_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}=\frac{r_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}+\gamma\cdot\overline{C}_{r}\Big(o_{\{q_{(s,i_{[1:s-1]})},i_{s}\}}\Big)}{\mathbb{I}_{\text{unsolved}}\cdot(S-s)+1}whereγ∈[0,1]\gamma\in[0,1]is a discount factor controlling the influence of descendant rewards, andC¯r​(⋅)\overline{C}_{r}(\cdot)denotes the mean reward over children of unsolved nodes (children of solved nodes do not exist since they represent terminal states).

The denominator equalsS−s+1S-s+1for unsolved nodes and11for solved nodes. This depth-based normalization serves two purposes: (1) it prevents rewards from growing unboundedly during backward propagation, as unsolved nodes can potentially accumulate discounted contributions from up toS−sS-sfuture turns; and (2) it ensures fair comparison between nodes that solve at different turns, a node that solves immediately at turnssretains its full reward, while a node that fails but has successful descendants receives appropriately scaled credit that accounts for the additional refinement steps required. All other aspects of the recursive procedure remain identical toMaRS.

MaRSvsMeRS:MaRSpropagates the maximum descendant reward to the earlier turns, emphasizing peak performance, whereasMeRSpropagates the mean reward, emphasizing stability and overall consistency.MaRScaptures best-case improvement, whileMeRSprovides a smoother estimate of expected progress. Together, they offer complementary views of feedback-driven credit assignment.

MurphyObjective:Once the rewards are reassigned according to the chosen credit assignment strategy (MaRSorMeRS), advantages are computed similar to standard GRPO. The adjusted rewards serve as the basis for computing normalized advantages at each turn. Conditioned on a promptq~\tilde{q}and forGsG_{s}generations, we normalize each reward by subtracting the mean reward and dividing by the standard deviation of the rewards obtained across theGsG_{s}generations, yielding the normalized advantageA^q~,i,tMurphy\hat{A}^{\textsc{Murphy}}_{\tilde{q},i,t}. Additionally, as defined earlier, eachii-th generation at turnss,o{q(s,i[1:s−1]),i}∈Oo_{\{q_{(s,i_{[1:s-1]})},i\}}\in O, corresponds to a complete output trajectory, whereo{q(s,i[1:s−1]),i,t}o_{\{q_{(s,i_{[1:s-1]})},i,t\}}denotes thett-th token ando{q(s,i[1:s−1]),i,<t}o_{\{q_{(s,i_{[1:s-1]})},i,<t\}}the prefix up to (but excluding) tokentt. We denote the sequence length of theii-th generation by|o{q(s,i[1:s−1]),i}||o_{\{q_{(s,i_{[1:s-1]})},i\}}|. At each turn, the GRPO objective is applied using these credit-adjusted advantages. This per-turn optimization allowsMurphyto incorporate feedback from later turns into earlier updates, effectively extending GRPO to a multi-turn setting. Finally, the divergence between the current and reference policies is captured byDKL(πθ∥πref)D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{{\text{ref}}}), computed over all tokens in the generated sequences. The resulting optimization objective, which integrates credit-assigned rewards, normalized advantages, and KL regularization at each turn, defines theMurphyobjective, distinguishing it from the standard GRPO formulation. The fullMurphyobjective is presented inApp. C.

Definition 2.(MurphyObjective)𝒥Murphy​(θ)\displaystyle\mathcal{J}_{\textsc{Murphy}}(\theta)=𝔼q∼𝒫⁡(Q)[∑s,i1,…​iS𝒥q(s,i[1:s])(θ)]\displaystyle=\mathbb{E}_{q\sim\mathcal{P}(Q)}[\sum_{s,i_{1},\ldots i_{S}}\mathcal{J}_{q_{(s,i_{[1:s]})}}(\theta)]Where the per prompt objective at each turnssis:𝒥q(s,i[1:s−1])(θ)\displaystyle\mathcal{J}_{q_{(s,i_{[1:s-1]})}}(\theta)=𝔼{o(q~,i)}i=1Gs∼πθold​(O|q~)[∑i=1Gs1Gs​|o{q~,i}|∑t=1|o{q~,i}|(min(Rθ(q~,i,t)A^q~,i,tMurphy,\displaystyle=\mathbb{E}_{\{o_{(\tilde{q},i)}\}_{i=1}^{G_{s}}\sim\pi_{\theta_{\text{old}}}(O|\tilde{q})}\Bigg[\sum_{i=1}^{G_{s}}\frac{1}{G_{s}|o_{\{\tilde{q},i\}}|}\sum_{t=1}^{|o_{\{\tilde{q},i\}}|}\Bigg(\min\Big(R_{\theta}(\tilde{q},i,t)\hat{A}^{\textsc{Murphy}}_{\tilde{q},i,t},clip(Rθ(q~,i,t),1−ϵ,1+ϵ)⋅A^q~,i,tMurphy)−βDKL(πθ∥πref))]\displaystyle\quad\quad\quad\quad\text{clip}\left(R_{\theta}(\tilde{q},i,t),\,1-\epsilon,\,1+\epsilon\right)\cdot\hat{A}^{\textsc{Murphy}}_{\tilde{q},i,t}\Big)-\beta D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{{\text{ref}}})\Bigg)\Bigg]with,​q~\displaystyle\text{with,}\tilde{q}=q(s,i[1:s−1])\displaystyle=q_{(s,i_{[1:s-1]})}Rθ​(q,i,t)\displaystyle R_{\theta}(\tilde{q},i,t)=πθ(o{q~,i,t}∣q~,o{q~,i,<t})πθold(o{q~,i,t}∣q~,o{q~,i,<t})\displaystyle=\frac{\pi_{\theta}(o_{\{\tilde{q},i,t\}}\mid\tilde{q},o_{\{\tilde{q},i,<t\}})}{\pi_{\theta_{\text{old}}}(o_{\{\tilde{q},i,t\}}\mid\tilde{q},o_{\{\tilde{q},i,<t\}})}And,A^q~,i,tMurphy\hat{A}^{\textsc{Murphy}}_{\tilde{q},i,t}denotes the advantage.

Note.

The design ofMurphyis broadly applicable across a range of RLVR algorithms, including PPO[17]and various extensions of GRPO[1,31,33]. In this work, we focus on GRPO due to its strong empirical performance in aligning LLMs[6,29]. ExtendingMurphyto other RLVR variants is conceptually straightforward, as it builds on the same underlying principles. While our experiments primarily focus on code generation, where rich, verifiable feedback is readily available, the framework can naturally extend to other domains such as mathematics or logical reasoning, provided suitable forms of feedback are accessible.

Appendix DSensitivity to Multi-Iteration Scaffolds

Murphyis explicitly designed to train models to incorporate environment feedback and self-correct over multiple turns. In multi-iteration evaluation—typical of agentic settings where feedback is present—Murphyyields substantially larger gains. This behavior is expected and reflects the objective ofMurphy, which is to improve a model’s ability to utilize feedback across turns rather than to optimize single-pass generation.

To assess sensitivity to the choice of multi-iteration scaffold, we conduct additional experiments using the Qwen3-1.7B model. We compare the base model, the GRPO-trained model, and theMurphy-trained model under three distinct scaffolds: Reflexion[21], LATS[38], and DoT[15]. Importantly, no additional training is performed. We evaluate the same GRPO-MT checkpoint reported inFig. 3and the sameMurphy(MaRS,IntraP) checkpoint reported in Table2. All scaffolds use their standard configurations with three iterations. We report mean accuracy and standard deviation over three independent runs on HumanEval.

Table 3:Sensitivity of Qwen3-1.7B performance on HumanEval to different multi-iteration scaffolds. No additional training is performed.Across all scaffolds, theMurphy-trained model consistently outperforms both the GRPO-MT trained model (trained under a matched compute budget) and the base model. These results indicate that the gains fromMurphyare not dependent on any particular inference-time scaffold, but instead reflect improved feedback utilization learned during training.

Appendix EReflexion

Reflexion[21]is an inference-time iterative framework designed to improve reasoning through repeated interaction with feedback from an external environment. It employs three agents: an actor (MaM_{a}), an evaluator (MeM_{e}), and a self-reflection module (Ms​rM_{sr}), which operate cyclically until a termination condition is met. For code generation, the process proceeds as follows:

1. 1.Actor step:The actorMaM_{a}receives an input and generates an output (e.g., a code snippet).
2. 2.Evaluation step:The evaluatorMeM_{e}scores the output (e.g., the percentage of unit tests passed).
3. 3.Self-reflection step:If the score is insufficient, the self-reflection moduleMs​rM_{sr}diagnoses the issue, proposes a fix, and appends both the failed output and the suggested correction to the input context. The updated input is then fed back to the actor, and the cycle repeats until either the task succeeds or a maximum number of iterations is reached.

In most implementations, the actor and the self-reflection module are instantiated by the same underlying language model. The self-reflection stage thus corresponds to the model reasoning over its own prior outputs augmented with feedback from the evaluator (or executor) and the previous input, output pairs, to generate improved responses in subsequent iterations.

Appendix FImplementation Details

We implement our framework on top of TRL[23], which provides efficient distributed training and a modular implementation of GRPO. We integrate TRL with vLLM for fast inference and large-scale rollout execution, enabling scalable multi-turn training in our experiments. Prompts used to trainMurphyare listed inApp. G. All experiments use publicly available datasets. The base models (Qwen3, OLMo) are available for research use under the Apache 2.0 license.

F.1Model Size and Compute Budget

All experiments were conducted on8×8\timesNVIDIA H100 GPUs. Our implementation builds on HuggingFace’s TRL222https://huggingface.co/docs/trl/en/index. For efficiency, 2 GPUs were allocated for inference via vLLM, while the remaining 6 GPUs handled model updates. Checkpoints were saved every 50 steps, and for all baselines, we selected the checkpoint corresponding to one epoch.

F.2Hyperparameters

We set the KL regularization coefficient toβ=0.04\beta=0.04, the learning rate to10−610^{-6}, and weight decay to0.10.1for both GRPO-MT andMurphyvariants. Unless otherwise specified,Murphyuses two turns. In GRPO-MT, we sample 72 rollouts per prompt in turn 1; failed rollouts receive execution feedback and are extended in turn 2. ForMurphy, we sample 8 rollouts per prompt in turn 1 and up to 8 per prompt in turn 2, yielding at most 64 rollouts in the second turn. Forμ\muCodeandReVealwe use the author recommended hyper-parameters for training.

Following[29], we use a temperature of 0.6 and top-ppof 0.95 for all inference experiments.

F.3Package Parameters

We use the Reflexion[21]framework to evaluate all trained models. The number of iterations is swept over{1,3}\{1,3\}, andmax-tokensis set to each model’s maximum generation length. Models are hosted via vLLM. Since all models fit on a single H100 GPU, we setdata-parallel-sizeto 8 and enable prefix caching to accelerate evaluation. We use the following commands to install the appropriate packages:

pip install uv && \
uv pip install trl==0.19.1 && \
uv pip install gunicorn==20.1.0 && \
uv pip install fastapi==0.115.12
uv pip install uvicorn==0.34.2 && \
uv pip install aiohttp==3.11.18
uv pip install astunparse==1.6.3
uv pip install jsonlines tenacity && \
uv pip install vllm==0.8.5.post1

F.4Additional Experiments

Table 4:Performance of OLMo-2-1124-7B-Instruct, Qwen3-1.7B and Qwen3-4B variants on HumanEval and MBPP benchmarks. GRPO-MT corresponds to the multi-turn baseline.Murphy(Ours) is highlighted.Δ3\Delta_{3}denotes the difference between theIter-3performance of the GRPO-MT/μ\muCode/Murphy-trained models and that ofBase (Iter-3), within each model block;greenindicates improvement (darker = larger gain),redindicates regression.#### F.4.1LiveCodeBench-v6 Per-Difficulty Results

Table5reports per-difficulty pass@1 on LiveCodeBench-v6 using Qwen3-4B as the base model. Base, GRPO-MT,μ\muCode, andMurphyare evaluated under the multi-turn Reflexion framework at 1, 3, and 5 iterations, whileReVealis evaluated using its official generation-verification scaffold with the same inference-turn budgets. This givesReVealits intended evaluation setting: although its generation prompt, output format, and parser differ from Reflexion, the scaffold is structurally similar in that it conditions on prior incorrect responses and environment feedback over multiple turns to refine subsequent attempts.Murphydemonstrates superior performance in multi-turn (Iter-3/5) settings. On the Hard subset, gains over the Base model increase with iteration (+3.57, +5.10, +5.14 pp at Iter-1/3/5). The only exception is the Medium subset at Iter-1, whereμ\muCodeoutperformsMurphyby 1.67 pp; however,Murphysurpasses all baselines by Iter-3 (+3.85 pp) and further extends the margin at Iter-5 (+4.38 pp). This trend supports our claim thatMurphy’s training objective compounds across self-correction iterations.

Table 5:Per-difficulty pass@1 on LiveCodeBench-v6 with Qwen3-4B.Values are mean±std{}_{\pm\text{std}}over 3 runs, in percent.Boldmarks the best method per row;underlinemarks the second best.Murphyachieves competitive or superior performance in multi-turn setting.To test whether the LiveCodeBench-v6 gains observed on Qwen3-4B also hold for a smaller model, we evaluate Qwen3-1.7B under the same iterative execution-feedback protocol. As shown in Tab.6, Iter-1 serves primarily as a sanity check:Murphydoes not degrade one-shot generation and achieves the best aggregate performance among all methods. The main comparison is Iter-3, where models can condition on execution feedback and refine earlier attempts. In this multi-turn setting,Murphyimproves over the strongest non-Murphybaseline by +1.93 pp on the aggregate split and by +3.40 pp on the Medium split. On the Hard split, MURPHY matchesμ\muCode while exhibiting lower variance. These results show thatMurphy’s LiveCodeBench-v6 gains are not specific to Qwen3-4B and that its advantage is most apparent in the feedback-conditioned multi-turn setting targeted by our method.

Table 6:LiveCodeBench-v6 results on Qwen3-1.7B.We report pass@1 on the Easy, Medium, Hard, and aggregate splits under 1-turn and 3-turn evaluation. Iter-1 serves as a sanity check for one-shot generation, while Iter-3 is the primary multi-turn setting for evaluating feedback-conditioned self-correction.Murphyachieves the best aggregate performance at both settings, with the largest gains appearing at Iter-3. Results are averaged over three runs; standard deviations are shown after±\pm.

F.4.2Ablation: Effect of Training Dataset Size

Figure 4:Murphy’s multi-turn advantage holds across training data scales.Qwen3-1.7B trained on nested KodCode subsets (1​K⊂2​K⊂3​K⊂4​K1\text{K}\subset 2\text{K}\subset 3\text{K}\subset 4\text{K}) and evaluated under three-iteration Reflexion (Iter-3) on HumanEval (left) and MBPP (right). Markers show the mean over 3 seeds; error bars indicate one standard deviation. Shaded regions highlight whereMurphy-MaRSexceeds GRPO-MT.Murphyoutperforms compute-matched GRPO-MT at every dataset scale on both benchmarks. We focus on the multi-turn (Iter-3) setting here since it directly probes the credit-assignment mechanism that distinguishesMurphyfrom GRPO-MT.To examine whetherMurphy’s gains hold beyond the 1K training split used in our main experiments (Fig. 3), we construct three additional nested subsets of KodCode (2​K⊂3​K⊂4​K2\text{K}\subset 3\text{K}\subset 4\text{K}) and train Qwen3-1.7B with both GRPO-MT andMurphy-MaRSunder matched compute budgets.Fig. 4reports Iter-3 (multi-turn) pass@1 on HumanEval and MBPP across all four scales, including the 1K result fromFig. 3for completeness. Although neither method scales monotonically with dataset size,Murphy-MaRSoutperforms GRPO-MT at every scale on both benchmarks, with average gains of+3.9+3.9pp on HumanEval and+5.1+5.1pp on MBPP. This pattern indicates thatMurphy’s multi-turn advantage is a property of the credit-assignment mechanism rather than an artifact of the specific training subset used in our main results.

F.4.3Ablation: Effect of Pruning in Multi-TurnMurphyTraining

We noted in the main paper that increasing the number of turns inMurphycan lead to exponential growth in computational cost. To mitigate this, we design and evaluate two pruning strategies. In this experiment, we extend our setup to a 3-turn setting and study the effect of pruning on performance. Results inTab. 7show that the pruned variant achieves competitive or even superior performance compared to the non-pruned counterpart.

Table 7:Ablation study comparing pruning versus non-pruning strategies for Qwen3-1.7B trained withMurphyfor 3 turns on the KodCode dataset. Reported numbers indicatepass@1(%) over three independent runs. Pruned variant achieves competitive performance compared to non-prunedMurphy. Although the MBPP Iter-3 means round to the same values as the two-turn pruning ablation inTab. 2, these results come from separate three-turn training runs; the different standard deviations reflect different seed-level outcomes.

Appendix GPrompt Examples

To train theMurphyobjective, we employ the following prompts. The system prompt is used at each dialogue turn, and the feedback prompts are applied during every feedback turn.

System PromptYou are a Python coding AI agent. When given a Python function signature and docstring, you must provide a complete Python solution following this exact format:1.Reasoning Phase:Use<think>...</think>tags to contain your complete thought process.•Break down the problem requirements step by step•Identify key constraints, edge cases, and potential pitfalls•Plan your algorithm and data structures•Walk through examples to validate your approach•Explain your logic thoroughly as if working on scratch paper2.Implementation Phase:Use<output>...</output>tags to contain your final code.•Include the complete function with the original signature•Ensure your code directly implements the approach from your thinking•Write clean, readable code with appropriate comments if neededYour solution must be complete, correct, and handle all specified requirements.

Feedback HeaderYou have previously attempted this problem{num-attempts}time(s).

Feedback BodyPrevious Attempt #{num-attempts}Analysis: Your earlier thought process and implementation: Let me think step by step. <think> {code-generated} Test Case Results: - Passed:{passed-tests}/{total-tests}test cases - Failed:{failed-tests}/{total-tests}test cases Detailed Feedback:{feedback-string} Note: For failed test cases, your code’s output is shown followed by a ‘#‘ symbol indicating the failure.

Feedback Footer.Instructions for Your Next Attempt:1.Failure Analysis Phase:(a)Carefully examine each failed test case to understand exactly what went wrong(b)Identify the specific lines of code or logic that caused the failures(c)Look for patterns across multiple failed cases (e.g., all involve negative numbers, empty inputs, etc.)(d)Determine if failures stem from algorithmic errors, edge case handling, or implementation bugs2.Solution Refinement Phase:(a)Build upon any correct aspects of your previous solution that passed tests(b)Redesign the problematic parts of your algorithm to handle the failed cases(c)Ensure your new approach covers edge cases that weren’t properly handled before(d)Consider additional edge cases that might not be in the test suite but could break your solution3.Implementation Phase:(a)Write your improved solution that directly addresses the identified failure points(b)Test your logic mentally against the failed cases to verify it would now pass(c)Ensure your solution maintains correctness for previously passing test casesUse the concrete feedback from your{num-attempts}previous attempt(s) to create a more robust and accurate solution.

Appendix HEfficiency Gains

Practical cost of rollout expansion.

Although the number of nodes in an unpruned feedback-conditioned tree can grow with the turn budget, several factors reduce the realized cost in practice. Successful candidates terminate immediately, and only failed candidates are expanded. In addition, descendants of the same parent share substantial prompt prefixes, including the original problem, previous output, and executor feedback. In our experiments, we use vLLM for rollout generation[14]. Beyond this setup, the shared-prefix structure could be further exploited by additional systems techniques such as prefix caching, asynchronous rollout generation[18], and speculative or parallel decoding[3]. These optimizations are not used in our experiments and are complementary toMurphy’s pruning, which targets post-rollout optimization cost rather than rollout construction.

Table 8:Where pruning saves compute.Average per-step timing breakdown across an end-to-end training run for unprunedMurphyvs.MurphywithInterPpruning on Qwen3-1.7B (8×\timesH100s, up to 72 rollouts per prompt; 8 in turn-1 and up to 64 in turn-2). Pruning is applied post-rollout, so generation and reward times are essentially unchanged; the speedup concentrates in the optimization phase, where the number of sequences entering gradient computation is reduced.PhaseUnprunedInterPΔ\DeltaGeneration (vLLM)30.54s30.46s−0.3%-0.3\%Reward (code exec)4.86s4.47s−8.0%-8.0\%Rollout overhead†19.86s17.75s−10.6%-10.6\%Rollout total55.25s (76%)52.67s (92%)−4.7%-4.7\%Optimization17.49s (24%)4.53s (8%)−74.1%\mathbf{-74.1\%}Total72.74s57.20s−21.4%\mathbf{-21.4\%} †Reference log-probs, tokenization, and advantage computation.

Measured pruning speedup.

To localize the speedup from pruning, we profile per-step timing averaged across an end-to-endInterPrun against unprunedMurphyunder matched configurations (Qwen3-1.7B, 8×\timesH100 GPUs, 72 rollouts per prompt: 8 in turn 1 and up to 64 in turn 2); seeTab. 8. Generation with vLLM and reward computation via code execution are essentially unchanged (−0.3%-0.3\%and−8.0%-8.0\%, respectively), since pruning is appliedafterrollouts are collected. The savings instead concentrate in optimization, where pruning reduces the number of sequences entering gradient computation: optimization time drops by74.1%\mathbf{74.1\%}, yielding a21.4%\mathbf{21.4\%}reduction in average per-step training time. This confirms thatInterP’s computational benefit comes from avoiding gradient updates on uninformative trajectories, not from shortening rollouts. This is important because aggressive rollout-time pruning could discard feedback signals that later turns rely on.

Appendix IBroader Impact

WhileMurphyintroduces some new dynamics through iterative self-correction and reflective optimization, the associated risks appear modest overall. The main considerations involve ensuring that feedback loops remain interpretable and that reward signals do not inadvertently reinforce narrow or heuristic reasoning. There is also some potential for subtle reward hacking, where the model optimizes for easily verifiable but shallow improvements, or for mild distributional drift if reflective heuristics fail to generalize beyond training contexts. Nonetheless, becauseMurphystill relies on verifiable rewards and bounded reflection, these risks are relatively contained and can be mitigated through careful evaluation design, human oversight, and robust validation across diverse task settings.

Chanakya Ekbote (@thecekbote): [1/10]: Most RLVR methods train LLMs as if reasoning is one-shot:

🧩 Prompt → 🤖 Response → ✅ Reward

But coding agents work in loops:

write code → run tests → see failures → debug → retry 🔁

So we ask a central question:

How do we teach LLMs not just to reason, but to