When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL

# Diagnostic-Driven Refinement for Sparse Structured RL
Source: [https://arxiv.org/html/2605.28918](https://arxiv.org/html/2605.28918)
## When LLM Reward Design Fails: Diagnostic\-Driven Refinement for Sparse Structured RL

###### Abstract

For sparse, structured reinforcement\-learning tasks with semantic reward\-function interfaces, LLM\-generated reward shaping is better framed as a debugging problem than as pure one\-shot generation\. We study PPO\-trained agents, using MiniGrid as the core sparse\-structured evaluation and MuJoCo reaching/locomotion as boundary stress tests\. Our audit finds two dominant one\-shot failure modes—reward flooding and semantic/API misunderstanding—plus a rarer, less reliably labeled weak\-shaping case\. We proposediagnostic\-driven iterative refinement, where training diagnostics and a failure\-mode taxonomy guide targeted reward\-function revision\. On sparse structured tasks with diagnosable reward failures, refinement improves DoorKey\-8×\\times8 from2\.3%2\.3\\%success without shaping to97\.6%97\.6\\%, and KeyCorridor from31\.2%31\.2\\%one\-shot to86\.7%86\.7\\%with high seed\-to\-seed variance\. Controls indicate that these gains are not attributable to generic retrying or extra training alone: metrics\-only re\-prompting produces large drops \(DoorKey\-8×\\times8:97\.6%→68\.6%97\.6\\%\\to 68\.6\\%; KeyCorridor:86\.7%→11\.5%86\.7\\%\\to 11\.5\\%\), while a static\-vocabulary control recovers much of the gap \(DoorKey\-8×\\times8:87\.6%87\.6\\%; KeyCorridor:70\.7%70\.7\\%\), showing that the taxonomy prompt itself is a major mechanism and that dynamic trigger labels provide only partially isolated incremental evidence\. Budget\-matched and Best\-of\-3 comparisons help separate refinement from selection and training\-time effects\. Component\-removal stress tests, sensitivity analyses, and an audit against author labels provide converging evidence for the debugging interpretation while revealing calibration limits\. Continuous\-control results expose the boundary: success\-based diagnostics can misfire in dense\-reward locomotion, and return\-trend feedback removes one false\-positive mechanism without producing robust locomotion gains\. The low\-call protocol is a protocol\-cost contrast with population\-based reward search, not a shared\-benchmark performance comparison\. In the four environments where we run a fully crossed variance design, variance point estimates are consistent with larger gains from reward correction when LLM reward\-function variance dominates, but bootstrap intervals are wide and temper exact share claims\. The method’s scope is deliberately bounded to sparse structured tasks with reliable structured interfaces under PPO training; richer semantic fields such asevent\_textcan help, hurt, or be neutral depending on alignment with the task structure\.

## 1Introduction

Reinforcement learning in environments with sparse rewards remains a fundamental challenge\. An agent exploring randomly in a multi\-step task—such as finding a key, unlocking a door, and reaching a goal—may never encounter the terminal reward signal within feasible training budgets\. Reward shaping\(Nget al\.,[1999](https://arxiv.org/html/2605.28918#bib.bib1)\)addresses this by providing intermediate feedback, but designing effective shaping functions demands substantial domain expertise and iterative manual tuning\.

Recent work has explored LLMs as reward function generators: Eureka performs evolutionary optimization over reward code with GPU\-parallel simulation, while Text2Reward generates dense reward code from language and compact environment representations\(Maet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib2); Xieet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib3)\)\. These systems achieve impressive results, but they provide limited insight into*why*LLM\-generated rewards fail—treating reward generation primarily as a performance\-driven search or refinement problem rather than analyzing the structure of failures\.

We argue that LLM reward design for sparse, structured tasks is better understood as a*debugging*problem than a generation problem\. One\-shot LLM generation fails in recurring, identifiable ways; lightweight diagnostics can often detect and repair these failures cheaply in the studied sparse structured setting; and the approach breaks predictably when diagnostics are misaligned with the task’s evaluation signal\. This reframing leads to a practical method—diagnostic\-driven iterative refinement—that improves over one\-shot generation and non\-diagnostic re\-prompting using 1–4 LLM calls per reward\-design run, repeated across seeds for evaluation, and CPU training, without population\-based search\. This is a protocol\-level efficiency comparison, not a direct performance comparison with evolutionary reward\-search systems on shared benchmarks\.

Ourmain contributionis a diagnostic\-driven framework for identifying and repairing systematic failure modes in LLM\-generated reward shaping, supported by targeted experiments that test the framework’s core assumptions\. We provide threesupporting findings:

1. 1\.A failure taxonomy for structured reward interfaces: prompt controls show that taxonomy\-rich feedback is functionally important, but also that static taxonomy vocabulary explains a substantial part of the improvement, so automated trigger labels should not be read as the sole causal mechanism\. The taxonomy itself is grounded in an audit where reward flooding and semantic/API misunderstanding are the two dominant failure patterns, while weak or negligible shaping appears as a rarer and less reliably labeled case\. On the dominant\-mode subset, an LLM auto\-labeler agrees with author labels with94%94\\%precision \(32/34 predicted dominant labels; 71\.4% overall accuracy across all categories\)\.
2. 2\.Variance\-based evidencefor when refinement helps: LLM generation variance has the clearest dominant point estimate in representative sparse tasks \(DoorKey\-8×\\times8: LLM std47\.5%47\.5\\%vs\. RL std0\.3%0\.3\\%; crossed LLM share96%96\\%with a wide\[66,100\]%\[66,100\]\\%bootstrap interval\), making diagnostic refinement most useful in this regime; residual and RL\-training variance have larger point estimates in representative continuous\-control decompositions, limiting the benefit of reward\-function correction\.
3. 3\.A domain\-boundary stress test: success\-based diagnostics produce false positives on dense\-reward locomotion, systematically degrading the feedback loop\. We provide root\-cause analysis and evaluate a return\-trend diagnostic adaptation, showing both where the framework breaks and how one specific failure mechanism can be removed\.

The method is strongest for sparse, structured tasks with reliable semantic state fields and exposed subgoal structure, and all experiments use PPO\-trained policies\. Richer natural\-language event descriptions are not uniformly beneficial: an interface ablation shows that removingevent\_textis neutral on DoorKey\-8×\\times8 and improves a KeyCorridor rerun\. We explicitly characterize this scope: it is not yet a claim about raw\-pixel reward design, arbitrary robotics tasks, or algorithm\-agnostic reward\-function validity, and dense\-reward settings require different diagnostics \(Section[11](https://arxiv.org/html/2605.28918#S11)\)\.

Roadmap and evidence status\.Table[2](https://arxiv.org/html/2605.28918#S6.T2)provides the headline 10\-seed MiniGrid and reaching results; Section[8](https://arxiv.org/html/2605.28918#S8)tests the diagnostic story through lower\-seed stress tests and prompt controls; Sections[9\.4](https://arxiv.org/html/2605.28918#S9.SS4)–[9\.7](https://arxiv.org/html/2605.28918#S9.SS7)report boundary, budget, variance, and selection analyses\. The central positive claim is about sparse structured PPO tasks with diagnosable reward\-function failures\. MuJoCo locomotion, fixed\-coefficient RND, model\-sensitivity, and protocol\-cost comparisons are included as context and boundary evidence rather than as broad superiority claims\.

## 2Background

### 2\.1Reward Shaping

Potential\-based reward shaping\(Nget al\.,[1999](https://arxiv.org/html/2605.28918#bib.bib1)\)provides a principled framework for adding intermediate rewards without altering the optimal policy\. In practice, most reward shaping uses heuristic bonuses tied to task\-specific subgoals \(e\.g\., \+0\.2 for picking up a key\)\. While effective, these heuristics require domain knowledge and manual tuning\.

### 2\.2LLMs for Code Generation in RL

Eureka\(Maet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib2)\)demonstrated that LLMs can generate reward functions for continuous\-control tasks, using evolutionary search over hundreds of candidates evaluated in GPU\-parallel simulation \(IsaacGym\)\. Text2Reward\(Xieet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib3)\)generates dense executable reward code from language task descriptions and compact environment representations\. Our work takes a complementary, diagnostic\-driven approach: rather than searching broadly, we analyze*why*generated rewards fail and use targeted feedback to repair specific failure modes with minimal iteration\. We evaluate primarily on discrete MiniGrid tasks and include continuous MuJoCo tasks as stress tests for the diagnostic assumptions\.

### 2\.3MiniGrid Environments

We use the MiniGrid framework\(Chevalier\-Boisvertet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib6)\), which provides a suite of grid\-world environments with partial observability\. The agent receives a7×7×37\\times 7\\times 3symbolic observation encoding object type, color, and state for each visible cell\. Actions are discrete: turn left, turn right, move forward, pick up, drop, toggle, and done\.

### 2\.4MuJoCo Continuous Control

To probe the boundary beyond discrete grid worlds, we also evaluate on MuJoCo\(Todorovet al\.,[2012](https://arxiv.org/html/2605.28918#bib.bib14)\)continuous\-control tasks via Gymnasium\(Towerset al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib15)\)\. For reaching tasks:Reacher\-v4\(2\-DOF planar arm, 11\-dim observation, 2\-dim action\) andFetchReach\-v4\(7\-DOF robotic arm, 13\-dim observation, 4\-dim action\)\(Plappertet al\.,[2018](https://arxiv.org/html/2605.28918#bib.bib16)\)\. For the reaching diagnostics, we treat success as an end\-effector distance below0\.050\.05\.

We further evaluate on standard locomotion benchmarks:HalfCheetah\-v4\(2D runner, 17\-dim observation, 6\-dim action\) andHopper\-v4\(single\-legged hopper, 11\-dim observation, 3\-dim action\)\. Unlike reaching tasks with sparse binary success, these feature dense multi\-component rewards \(forward velocity\+\+alive bonus−\-control cost\), making them a boundary test for diagnostics designed around binary success signals\.

### 2\.5Intrinsic Motivation

Random Network Distillation \(RND\)\(Burdaet al\.,[2019](https://arxiv.org/html/2605.28918#bib.bib13)\)is a widely\-used intrinsic motivation method that provides exploration bonuses based on prediction error between a fixed random network and a trainable predictor\. The intrinsic reward is high for novel states and decreases as the predictor learns\. RND is domain\-agnostic and requires no task\-specific knowledge, making it a natural comparison baseline for our LLM\-guided reward design approach\.

## 3Method

### 3\.1Overview

Our pipeline has three components \(Figure[1](https://arxiv.org/html/2605.28918#S3.F1)\):

1. 1\.Reward Generation: An LLM receives a natural\-language description of the environment and produces a Python functionreward\_fn\(obs, action, reward, terminated, truncated, info, state\)that returns a shaped reward and updated state dictionary\.
2. 2\.RL Training: A standard PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.28918#bib.bib12)\)agent trains using the shaped reward\. The environment wrapper provides structured information \(agent position, carrying status, event text\) in theinfodictionary\.
3. 3\.Iterative Refinement\(optional\): After a short probe training run, automated diagnostics assess whether the reward function exhibits failure modes\. The diagnosis and training metrics are fed back to the LLM, which produces a revised function\.

Iterative Refinement Loop \(≤\\leq3 rounds\)EnvironmentDescriptionLLM\(Claude / GPT\)reward\_fnPython codePPOTrainingTrainedAgentpromptgenerateshape3000 epsProbe\(500 eps\)Auto\-Diagnosticsshort runmetricsfeedback \+prev\. codeSandboxValidationpass

Figure 1:Overview of LLM\-Guided Reward Design\. The LLM generates a reward\-shaping function from a natural\-language environment description, validated in a sandbox before use\. In iterative mode \(dashed\), short probe runs provide diagnostics for up to 3 refinement rounds\.
### 3\.2Reward Function Interface

The generated reward function receives the full transition tuple plus a mutablestatedictionary that persists across steps within an episode \(reset between episodes\)\. This enables one\-time bonuses:

1defreward\_fn\(obs,action,reward,terminated,

2truncated,info,state\):

3shaped=reward

4ifnotstate\.get\("key\_picked\_up"\):

5if"pickedup"ininfo\["event\_text"\]\.lower\(\):

6shaped\+=0\.2

7state\["key\_picked\_up"\]=True

8returnshaped,state

Theinfodictionary exposes:agent\_pos,carrying,event\_text\(natural\-language event descriptions\),step\_count, andmax\_steps\.

### 3\.3LLM Prompt Design

The generation prompt includes: \(1\) environment name and goal description, \(2\) available observation space and action space, \(3\)infodictionary fields with examples, \(4\) constraints \(reward magnitudes 0\.01–0\.5, use one\-time bonuses, avoid continuous rewards\)\. For refinement, we additionally include: \(5\) the previous reward function source code, \(6\) probe training metrics \(success rate, mean reward\), and \(7\) diagnosed failure modes\.

### 3\.4Automated Diagnostics

After each probe training run, we compute three diagnostics \(exact thresholds in Table[16](https://arxiv.org/html/2605.28918#A3.T16); supporting evidence in Section[8](https://arxiv.org/html/2605.28918#S8)\):

- •Reward hacking: If mean reward\>0\.5\>0\.5and success rate<0\.2<0\.2, the agent is accumulating shaped rewards without solving the task\. Disabled for dense\-reward locomotion tasks, which lack a binary success signal and use return\-trend analysis instead \(Section[9\.4](https://arxiv.org/html/2605.28918#S9.SS4)\)\.
- •Shaping weakness: If both success rate<0\.1<0\.1and mean reward<0\.1<0\.1, the reward bonuses are too small to guide exploration\.
- •Learning plateau: If success rate has stagnated \(improvement<5%<5\\%over the probe window\) after\>1,000\>1\{,\}000probe episodes, the agent may be stuck in a local optimum\. This gate is inactive under the default 500\-episode MiniGrid probes and is retained for longer\-probe variants rather than treated as a driver of the headline 500\-episode results\.

These diagnostics are formatted as natural\-language warnings in the refinement prompt\. The control experiments \(Section[8](https://arxiv.org/html/2605.28918#S8)\) show that taxonomy\-guided feedback is useful but imperfect: reward\-hacking detection can generate false positives on some environments, shaping\-weakness feedback is useful on moderately difficult tasks, and the plateau rule requires longer probes before it can be interpreted causally\.

### 3\.5Iterative Refinement Algorithm

Algorithm[3\.5](https://arxiv.org/html/2605.28918#S3.SS5)formalizes the iterative refinement procedure\.

Algorithm 1: LLM\-guided iterative reward design

### 3\.6Safety and Validation

Generated reward functions are executed in a sandboxed namespace with restricted\_\_builtins\_\_\(onlymath,abs,min,max,len, etc\.\)\. Functions that fail validation \(syntax errors, runtime errors on dummy inputs\) trigger up to 3 retries with error feedback\.

### 3\.7Extension to Continuous Control

For MuJoCo environments, we adapt the pipeline as follows: \(1\) theinfodictionary exposes position\-based fields \(distance\_to\_target,end\_effector\_pos,target\_pos\) and event text \(“reached target”, “moved closer”\), \(2\) the LLM prompt is adapted for continuous observations \(flat vectors\) and continuous actions, emphasizing distance\-based progress bonuses over event\-driven bonuses, \(3\) the PPO trainer uses an MLP policy with Gaussian actor, running observation normalization, and MuJoCo\-appropriate hyperparameters \(rollout length 2048, 10 PPO epochs, no entropy bonus\)\. The reward function interface remains identical, ensuring the same iterative refinement protocol applies\.

For locomotion environments \(HalfCheetah, Hopper\), the LLM designs an alternative reward function accessing velocity, position, and control cost from theinfodictionary\. The “no shaping” baseline uses the default Gymnasium reward\. Since locomotion tasks have no binary success criterion, we evaluate using mean episode return over the final 100 episodes\.

## 4Interpretation

We connect our empirical approach to the reward shaping literature as an interpretive lens for the observed effectiveness and failure modes\.

### 4\.1LLM Bonuses as Approximate Potential\-Based Shaping

Ng et al\.\(Nget al\.,[1999](https://arxiv.org/html/2605.28918#bib.bib1)\)proved that potential\-based shapingF​\(s,s′\)=γ​Φ​\(s′\)−Φ​\(s\)F\(s,s^\{\\prime\}\)=\\gamma\\Phi\(s^\{\\prime\}\)\-\\Phi\(s\)preserves the optimal policy\. The one\-time bonuses generated by our LLM \(e\.g\.,\+0\.2\+0\.2when the key is first picked up\) can be interpreted as approximations to potential\-based shaping over an augmented state space that includes event flags\.

Let𝒮\+=𝒮×\{0,1\}k\\mathcal\{S\}^\{\+\}=\\mathcal\{S\}\\times\\\{0,1\\\}^\{k\}be the state space augmented withkkbinary milestone indicators \(e\.g\., key picked up, door opened\)\. Define a potential function:

Φ​\(s,𝐦\)=∑i=1kbi⋅mi\\Phi\(s,\\mathbf\{m\}\)=\\sum\_\{i=1\}^\{k\}b\_\{i\}\\cdot m\_\{i\}\(1\)wherebib\_\{i\}is the bonus for milestoneiiandmi∈\{0,1\}m\_\{i\}\\in\\\{0,1\\\}indicates whether milestoneiihas been achieved\. The one\-time bonus\+bi\+b\_\{i\}when milestoneiiis first triggered corresponds toΦ​\(s′,𝐦′\)−Φ​\(s,𝐦\)\\Phi\(s^\{\\prime\},\\mathbf\{m\}^\{\\prime\}\)\-\\Phi\(s,\\mathbf\{m\}\)at the transition wheremim\_\{i\}changes from 0 to 1 \(neglecting the discount factor\)\. This connection is approximate because: \(1\) the LLM does not explicitly constructΦ\\Phi, \(2\) the discount correctionγ​Φ​\(s′\)−Φ​\(s\)\\gamma\\Phi\(s^\{\\prime\}\)\-\\Phi\(s\)is not enforced, and \(3\) some LLM\-generated functions include non\-potential components \(e\.g\., stagnation penalties\)\. The near\-potential structure therefore serves as intuition for why well\-designed one\-time bonuses can provide useful intermediate signal without obviously changing the task objective, not as a policy\-invariance guarantee\.

The failure mode of*reward flooding*—continuous per\-step bonuses—is not potential\-based over the original state representation\. In finite\-horizon or terminating tasks, a per\-step bonus changes the relative return of trajectories with different lengths and can favor delaying termination; dynamic\-potential variants illustrate why time\-varying shaping terms require explicit structure rather than arbitrary repeated bonuses\(Nget al\.,[1999](https://arxiv.org/html/2605.28918#bib.bib1); Devlin and Kudenko,[2012](https://arxiv.org/html/2605.28918#bib.bib28)\)\. This can alter the optimal policy by favoring longer trajectories, exactly the behavior we observe when reward flooding masks the task signal\.

### 4\.2Iterative Refinement Through a Variance\-Reduction Lens

Consider the space of reward functionsℱ\\mathcal\{F\}that an LLM can generate\. One\-shot generation samplesf∼PLLM​\(ℱ\|𝒟\)f\\sim P\_\{\\text\{LLM\}\}\(\\mathcal\{F\}\|\\mathcal\{D\}\)conditioned on the environment description𝒟\\mathcal\{D\}\. The resulting policy performanceJ​\(f\)=𝔼πf​\[R\]J\(f\)=\\mathbb\{E\}\_\{\\pi\_\{f\}\}\[R\]is a random variable with variance:

Var​\[J\]=Varf​\[𝔼π​\[R\|f\]\]\+𝔼f​\[Varπ​\[R\|f\]\]\\text\{Var\}\[J\]=\\text\{Var\}\_\{f\}\[\\mathbb\{E\}\_\{\\pi\}\[R\|f\]\]\+\\mathbb\{E\}\_\{f\}\[\\text\{Var\}\_\{\\pi\}\[R\|f\]\]\(2\)The first term is LLM generation variance; the second is RL training variance\. This equation describes the ideal marginal decomposition\. Our anchored estimates in Section[9\.6](https://arxiv.org/html/2605.28918#S9.SS6)use single\-seed conditioning as a directional approximation, while the fully crossed5×55\\times 5design in Table[14](https://arxiv.org/html/2605.28918#S9.T14)realizes the random\-effects partition directly on a representative subset\.

Iterative refinement can be viewed as a conditional sampling procedure: rather than drawing independent samples fromPLLM​\(ℱ\|𝒟\)P\_\{\\text\{LLM\}\}\(\\mathcal\{F\}\|\\mathcal\{D\}\), each refinement step conditions on the previous function and its training diagnostics, sampling fromPLLM​\(ℱ\|𝒟,fk,dk\)P\_\{\\text\{LLM\}\}\(\\mathcal\{F\}\|\\mathcal\{D\},f\_\{k\},d\_\{k\}\)\. This can be interpreted as reducing effective variability in the sampled reward functions by shifting generation toward functions that avoid diagnosed failure modes\. The approach should be most useful when LLM generation variance dominates, which is consistent with the pattern we observe for discrete MiniGrid tasks, and should provide limited benefit when RL variance dominates, as in the continuous MuJoCo stress tests\.

## 5Experimental Setup

### 5\.1Environments

We evaluate on nine MiniGrid environments and four MuJoCo environments \(Table[1](https://arxiv.org/html/2605.28918#S5.T1)\):

EnvironmentTypeKey ChallengeSub\.LavaGap\-S5GridNavigation \+ avoidance2DoorKey\-5×\\times5GridKey→\\todoor→\\togoal3DoorKey\-8×\\times8GridLarger search space3UnlockPickupGridKey→\\todoor→\\tobox4KeyCorridor\-S3R1GridCorridor navigation3BlockedUnlockPickupGridMove ball, key, door, box4FourRoomsGridMulti\-room navigation1MultiRoom\-N6GridLong\-horizon \(6 rooms\)1ObstructedMaze\-2DlhGridHidden key, obstruction4Reacher\-v4Cont\.2\-DOF arm→\\totarget1FetchReach\-v4Cont\.7\-DOF gripper→\\togoal1HalfCheetah\-v4Cont\.Forward locomotion1Hopper\-v4Cont\.Balance \+ locomotion1Table 1:Evaluation environments\. Grid = MiniGrid discrete, Cont\. = MuJoCo continuous\. Sub\. = sequential subgoals\. The three new environments \(FourRooms, MultiRoom\-N6, ObstructedMaze\) test longer planning horizons and harder exploration\. Locomotion tasks feature dense default rewards and no binary success criterion\.
### 5\.2Conditions

We compare five conditions:

1. 1\.No Shaping: Baseline PPO with only the sparse environment reward\.
2. 2\.LLM One\-Shot: A single LLM call generates the reward function\.
3. 3\.LLM Iterative: Up to 3 refinement iterations with probe runs \(500 episodes for MiniGrid, 200 for MuJoCo\)\. Dense\-reward locomotion uses the return\-aware diagnostic adaptation in the main results; the original success\-diagnostic variant is reported separately as a broken baseline\.
4. 4\.Hand\-Crafted: Human\-designed reward functions with tuned bonus magnitudes\.
5. 5\.RND: Random Network Distillation\(Burdaet al\.,[2019](https://arxiv.org/html/2605.28918#bib.bib13)\)intrinsic motivation as a domain\-agnostic exploration baseline\. The main table uses a fixed, untuned coefficient of0\.10\.1across environments; predictor architecture: 3\-layer MLP \(256–256–128\)\. We separately run a post\-hoc coefficient sweep on the two DoorKey tasks where reviewers identified this fixed coefficient as a likely confound\.

### 5\.3Training Details

All conditions use PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.28918#bib.bib12)\)\. The main one\-shot, no\-shaping, hand\-crafted, RND, and MiniGrid iterative conditions are evaluated across10 random seeds\(42, 123, 456, 789, 1024, 2048, 3141, 4096, 5555, 7777\)\. Exceptions are stated where they occur: return\-aware locomotion refinement uses 6 completed seeds, and some secondary controls have fewer successful runs due to reward\-function validation failures\.

MiniGrid hyperparameters: learning rate3×10−43\\times 10^\{\-4\},γ=0\.99\\gamma=0\.99, GAEλ=0\.95\\lambda=0\.95, clip ratio 0\.2, entropy coefficient 0\.1, 2 PPO epochs, batch size 64, rollout length 512\. Observations are normalized by dividing by 10 \(MiniGrid symbolic values range 0–10\)\. Training budget: 3,000 episodes\.

MuJoCo hyperparameters: learning rate3×10−43\\times 10^\{\-4\},γ=0\.99\\gamma=0\.99, GAEλ=0\.95\\lambda=0\.95, clip ratio 0\.2, entropy coefficient 0\.0, 10 PPO epochs, batch size 64, rollout length 2,048\. Observations use running mean/std normalization\. MLP policy with 256\-unit hidden layers and Gaussian actor \(learnable log\-std\) with tanh squashing\. Training budgets: 1,000 episodes for Reacher, HalfCheetah, and Hopper; 3,000 episodes for FetchReach \(to account for its 7\-DOF complexity\)\.

The LLM used for reward generation is Claude Haiku \(claude\-haiku\-4\-5\-20251001\) with temperature 0\.4\.

Episode budget\.The MiniGrid iterative condition uses up toK×Np\+Nf=3×500\+3,000=4,500K\\times N\_\{p\}\+N\_\{f\}=3\\times 500\+3\{,\}000=4\{,\}500total training episodes \(50% overhead\)\. The MuJoCo iterative condition uses up to3×200\+Nf3\\times 200\+N\_\{f\}episodes, whereNfN\_\{f\}is the environment\-specific full training budget\. Probe runs are short and the additional compute cost is modest relative to total training\.

## 6Results

### 6\.1Main Results

Table[2](https://arxiv.org/html/2605.28918#S6.T2)presents final performance for each condition, computed over the last 100 training episodes and averaged across random seeds\.

Table 2:Main results across 13 environments \(last 100 episodes, mean±\\pmsample std\)\. SR = success rate; Ret = mean episode return\.Boldindicates best performance per environment for MiniGrid and reaching tasks; no locomotion cell is bolded because those rows are boundary diagnostics with no robust corrected improvement\. We report 12 primary comparisons in Appendix[B](https://arxiv.org/html/2605.28918#A2); 9/12 are significant after per\-family Holm–Bonferroni correction\. The iterative column reports standard diagnostic refinement for MiniGrid and reaching tasks, and return\-aware refinement for dense\-reward locomotion\. Return\-aware locomotion iterative results use the first 6 completed canonical seeds; other main results use 10 seeds\. FetchReach trained for 3,000 episodes; all other MuJoCo environments for 1,000\.†The Hopper no\-shaping value is the 1,000\-episode main\-budget result; the budget\-matched 1,600\-episode no\-shaping result is882882in Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)\. RND uses a fixed, untuned coefficientc=0\.1c=0\.1in this table; Table[3](https://arxiv.org/html/2605.28918#S6.T3)reports a coefficient sweep for the two DoorKey rows\.We do not interpret these results as showing that LLM rewards generally dominate hand\-crafted rewards, intrinsic motivation, or continuous\-control baselines\. The central claim is narrower: in sparse structured PPO settings where one\-shot LLM rewards fail for diagnosable reasons, taxonomy\-guided diagnostic feedback improves over one\-shot generation and non\-diagnostic retrying\. The hand\-crafted, RND, and MuJoCo results are included to characterize context and boundaries, not to claim broad reward\-design superiority\.

Key findings:

- •One\-shot LLM design is unreliable: While it improves over no shaping on DoorKey\-8×\\times8 \(2\.3%→58\.3%2\.3\\%\\to 58\.3\\%\), it shows high variance \(sample std up to47\.3%47\.3\\%\) and performs comparably to no shaping on LavaGap \(64\.3%64\.3\\%vs\.64\.4%64\.4\\%\)\. On DoorKey\-5×\\times5, one\-shot drops to85\.3%85\.3\\%while all other conditions reach∼\\sim100%\.
- •Iterative refinement provides its clearest gains on sparse structured tasks: KeyCorridor improves from31\.2%31\.2\\%one\-shot to86\.7%86\.7\\%iterative, and DoorKey\-8×\\times8 reaches97\.6%97\.6\\%\(vs\.58\.3%58\.3\\%one\-shot\)\. Hand\-crafted baselines provide useful context, but the primary comparison is against one\-shot generation and non\-diagnostic controls\.
- •Fixed\-coefficient RND is mixed and coefficient\-sensitive: With the untuned coefficient used in Table[2](https://arxiv.org/html/2605.28918#S6.T2), RND intrinsic motivation*hurts*performance on DoorKey\-5×\\times5 \(13\.2%13\.2\\%vs\.99\.8%99\.8\\%no shaping\) and DoorKey\-8×\\times8 \(0\.1%0\.1\\%vs\.2\.3%2\.3\\%\)\. A reviewer\-requested sweep shows this was partly a tuning artifact on DoorKey\-5×\\times5:c=0\.01c=0\.01reaches100\.0%100\.0\\%, whilec=0\.1c=0\.1andc=0\.5c=0\.5remain much worse\. On DoorKey\-8×\\times8, however, all swept RND coefficients remain near zero\. We therefore treat RND as contextual intrinsic\-motivation evidence, not as a tuned baseline claim\.
- •The strongest positive results occur where one\-shot generation is unreliable but subgoals are semantically exposed: DoorKey\-8×\\times8 reaches97\.6%97\.6\\%, LavaGap reaches88\.8%88\.8\\%under this PPO/protocol implementation, and KeyCorridor reaches86\.7%86\.7\\%iterative success\. The KeyCorridor mean is high but unstable \(sample std28\.5%28\.5\\%\): the median seed reaches100%100\\%, but one seed reaches only19%19\\%and another50%50\\%\. Later stress tests show that diagnostic thresholds andevent\_textsignals can substantially change outcomes, so the KeyCorridor result is best read as high\-variance evidence that taxonomy\-guided refinement can improve over one\-shot generation, not as isolated proof that dynamic diagnostic labels are the sole repair mechanism\. On simpler environments \(DoorKey\-5×\\times5\) all methods converge\. On BlockedUnlockPickup, all conditions achieve≤0\.8%\\leq 0\.8\\%, indicating a task beyond the reach of reward shaping alone\. UnlockPickup improves modestly but remains challenging \(24\.3%24\.3\\%iterative success\)\.
- •Richer semantic interfaces are not always better: The headline KeyCorridor result \(86\.7%86\.7\\%,n=10n=10\) is essentially matched by a 5\-seed rerun that removesevent\_textfrom both the prompt and runtime interface \(86\.8%86\.8\\%\), while the paired with\-events rerun reaches only55\.2%55\.2\\%\. The five seed IDs are a subset of the headline canonical seeds, but the ablation uses independent LLM generations, so the near\-equality should be read as a scope warning rather than a paired replication\. The method needs structured semantic state, but not every semantic field helps\.
- •Harder environments reveal limits: On FourRooms, no shaping \(27\.2%27\.2\\%\) outperforms all reward\-shaped conditions, suggesting the LLM’s intermediate bonuses interfere with the agent’s exploration in open multi\-room layouts\. On ObstructedMaze, LLM one\-shot \(7\.4%7\.4\\%\) and iterative \(6\.6%6\.6\\%\) substantially improve over no shaping \(0\.5%0\.5\\%\), demonstrating that LLM\-designed rewards help on tasks with hidden subgoal structure\. MultiRoom\-N6 remains near 0% across all conditions, indicating a horizon beyond current reward shaping capabilities\.

Table 3:Reviewer\-requested RND coefficient sweep on the two DoorKey tasks \(10 seeds, 3,000 episodes, final\-100 success rate\)\. The fixedc=0\.1c=0\.1baseline in Table[2](https://arxiv.org/html/2605.28918#S6.T2)was badly mistuned on DoorKey\-5×\\times5, wherec=0\.01c=0\.01solves the task, but tuning over this small grid does not rescue RND on DoorKey\-8×\\times8\.
### 6\.2Learning Curves

Figure[2](https://arxiv.org/html/2605.28918#S6.F2)shows learning curves across the core sparse tasks\. The iterative condition learns faster and reaches higher final performance than one\-shot on the environments where one\-shot reward generation is most unreliable\. On DoorKey\-8×\\times8, the no\-shaping baseline never exceeds2\.3%2\.3\\%success, while iterative refinement reaches97\.6%97\.6\\%\. On LavaGap, iterative refinement achieves88\.8%88\.8\\%—more than25%25\\%above the hand\-crafted baseline \(63\.6%63\.6\\%\)\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x1.png)Figure 2:Learning curves \(success rate vs\. episode, smoothed over 100 episodes\)\. Shaded regions show±\\pm1 sample std across seeds\.
### 6\.3Post\-Refinement Training Episodes

Figure[3](https://arxiv.org/html/2605.28918#S6.F3)compares the number of full\-training episodes required to reach 80% success rate after the reward function has been selected or refined\. Probe episodes used during reward design are excluded from this figure; total environment\-interaction controls are reported separately in Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)\. Under this post\-refinement training view, the iterative condition reaches the threshold faster than the hand\-crafted baselines on DoorKey\-5×\\times5 and DoorKey\-8×\\times8\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x2.png)Figure 3:Full\-training episodes to reach 80% success after reward selection/refinement\. Probe episodes are excluded here; total\-budget controls are reported in Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)\. Hatched bars \(N/R\) indicate the threshold was never reached within 3,000 full\-training episodes\.

## 7Failure Taxonomy

The main results show that one\-shot LLM reward generation is useful but unreliable\. To make the failure structure actionable, we classify generated reward functions into two dominant failure modes and one rarer weak\-shaping case \(Table[4](https://arxiv.org/html/2605.28918#S7.T4)\)\.

Table 4:Failure modes of LLM one\-shot reward generation\. Reward flooding and semantic/API misunderstanding are the dominant modes in our audit; weak or negligible shaping is observed more rarely and is less reliably labeled\.Reward floodingis one of the most damaging observed failure modes\. In DoorKey\-8×\\times8, one generated function assigned\+0\.02\+0\.02per forward step, leading to high accumulated reward but only∼\\sim5% success rate—the agent learned to walk in circles collecting step rewards rather than solving the task\. The iterative diagnostic detects this pattern \(high reward, low success\) and explicitly warns the LLM in the refinement prompt\.

This taxonomy is not only descriptive: it defines the feedback vocabulary used in iterative refinement\. The next section tests whether those diagnostic signals are useful enough to serve as the method’s core mechanism\.

## 8Diagnostics Evidence and Controls

A strong objection to our approach is that the diagnostics*are*the method, yet aggregate performance alone does not establish that the diagnostic signals are reliable\. We therefore treat the diagnostics as empirically useful but imperfect heuristics and evaluate them through targeted controls: a component\-removal stress test \(which detector families are risky?\), prompt controls against metrics\-only and static\-vocabulary baselines \(does taxonomy\-rich feedback add value, and how much is due to the static taxonomy vocabulary?\), a taxonomy\-labeling audit against author labels \(are dominant failure\-mode labels recoverable?\), and sensitivity checks for threshold and probe design\. Table[5](https://arxiv.org/html/2605.28918#S8.T5)summarizes what each control supports and what it does not establish\. Appendix[C](https://arxiv.org/html/2605.28918#A3)lists all trigger values used in the experiments\.

Unless stated otherwise, control experiments in this section are independent lower\-seed reruns, not exact reproductions of the 10\-seed means in Table[2](https://arxiv.org/html/2605.28918#S6.T2); captions report the seed counts and protocol differences\.

Table 5:Evidence matrix for the diagnostic mechanism\. Environment, seed, and budget details are reported in the corresponding subsections; this table summarizes the reviewer\-facing interpretation\. The controls support the debugging interpretation through converging evidence, while also identifying calibration limits and boundary cases\.### 8\.1Component\-Removal Stress Test

We remove each diagnostic component one at a time—reward\-hacking detection \(RH\), shaping\-weakness detection \(SW\), and learning\-plateau detection \(LP\)—and compare against the full system on four representative environments\. These are independent 3\-seed reruns and should be read as a stress test for detector sensitivity rather than as an isolated causal decomposition of the 10\-seed headline results\.

Table 6:Component\-removal stress test: mean success rate from independent 3\-seed reruns with one detector removed at a time\.Boldmarks substantial degradation \(\>\>15 points\) outside the LP column\.†Because the plateau gate requires\>1,000\>1\{,\}000probe episodes, the−\-LP column reports an observed rerun drop rather than causal evidence about plateau warnings under the default 500\-episode probes\.The stress test reveals an asymmetric sensitivity pattern\.Shaping\-weakness detection matters on moderately difficult tasks: removing it drops DoorKey\-8×\\times8 from93%93\\%to64%64\\%\(Δ=−29\\Delta=\-29points\), where the reward function must provide sufficient exploration guidance\. The−\-LP KeyCorridor drop \(71%→18%71\\%\\to 18\\%\) is not interpreted as a clean plateau\-detector effect: local\-log inspection shows that the default 500\-episode probes do not satisfy the\>1,000\>1\{,\}000\-episode plateau eligibility gate, so plateau warnings do not fire in these runs\. We therefore treat this row as evidence of lower\-seed rerun sensitivity, not as proof that the plateau detector explains the KeyCorridor gains\.

An unexpected finding is thatremoving the reward\-hacking detector improves performanceon KeyCorridor \(71%→100%71\\%\\to 100\\%,Δ=\+28\\Delta=\+28\) and does not hurt DoorKey\-8×\\times8 \(93%→95%93\\%\\to 95\\%\)\. This suggests the reward\-hacking detector generates false positives on these environments, triggering unnecessary refinements that degrade the reward function\. We retain the detector in the headline pipeline because it provides a modest UnlockPickup safeguard and catches pathological flooding cases, but practitioners should disable it or raise its cutoff when probe traces show useful progress despite low early success\. These results suggest that shaping\-weakness and reward\-hacking diagnostics are useful but require calibration, while plateau detection should only be analyzed in protocols where its eligibility gate can actually fire\.

### 8\.2Taxonomy Prompt Controls

A natural question is whether the gains come from taxonomy\-rich diagnostic feedback or merely from giving the LLM another attempt\. We first compare the full diagnostic pipeline against a*metrics\-only baseline*that provides the LLM with identical numerical metrics \(success rate, mean reward\) but strips all dynamic diagnostic tags, failure\-mode names, and static “common failure modes to avoid” guidance\. The LLM receives only: “the results are suboptimal; write an improved reward function\.” This is a strong prompt\-level ablation of taxonomy information\.

Table 7:Full diagnostic\-guided refinement vs\. metrics\-only re\-prompting \(mean±\\pmsample std; metrics\-only uses 9–10 successful seeds, with UnlockPickup atn=9n=9\)\. The full diagnostic column reuses the 10\-seed Table[2](https://arxiv.org/html/2605.28918#S6.T2)means\. The taxonomy\-rich feedback condition provides 29–75 percentage point improvements on the hardest tasks, but this prompt\-level ablation removes both dynamic diagnostic labels and static failure\-mode vocabulary\.Removing taxonomy\-rich feedback causes large performance drops on the hardest tasks: DoorKey\-8×\\times8 drops from97\.6%97\.6\\%to68\.6%68\.6\\%\(Δ=29\.0\\Delta=29\.0points; Welchp=0\.043p=0\.043; bootstrap95%95\\%CI\[7\.9,53\.5\]\[7\.9,53\.5\]points\), KeyCorridor from86\.7%86\.7\\%to11\.5%11\.5\\%\(Δ=75\.2\\Delta=75\.2points;p=6\.7×10−6p=6\.7\{\\times\}10^\{\-6\}; CI\[56\.0,90\.7\]\[56\.0,90\.7\]\), and UnlockPickup from24\.3%24\.3\\%to6\.8%6\.8\\%\(Δ=17\.5\\Delta=17\.5points;p=6\.1×10−4p=6\.1\{\\times\}10^\{\-4\}; CI\[9\.4,24\.7\]\[9\.4,24\.7\]\)\. DoorKey\-5×\\times5 is saturated and the smaller gap is not statistically resolved \(p=0\.33p=0\.33\)\. The metrics\-only baseline is comparable to one\-shot generation on the hard sparse tasks, providing evidence thattaxonomy\-rich feedback, not mere re\-prompting, is important to iterative refinement’s effectiveness\.

To separate static taxonomy vocabulary from dynamic trigger labels, we add a stricter reviewer\-requested control on DoorKey\-8×\\times8 and KeyCorridor\. This condition keeps the staticCOMMON FAILURE MODES TO AVOIDblock and the numerical probe metrics, but removes per\-iteration trigger strings such asREWARD HACKING DETECTED,SHAPING TOO WEAK, andPLATEAU DETECTED\. Iteration\-0 reward functions are shared with the full diagnostic condition when available, so the manipulation begins at refinement\.

Table 8:Static\-vocabulary prompt control \(10 seeds\)\. Static\-vocab\-only keeps the failure\-mode vocabulary and raw metrics while removing dynamic per\-iteration diagnostic labels\. It recovers much of the metrics\-only gap, especially on KeyCorridor, so the mechanism should be framed as taxonomy\-guided prompting plus diagnostics rather than dynamic trigger labels alone\.The static\-vocabulary control changes the mechanism interpretation\. On DoorKey\-8×\\times8, full diagnostics exceed static\-vocabulary prompting by10\.010\.0points, but the interval includes small or negative gaps \(bootstrap CI\[−1\.8,30\.3\]\[\-1\.8,30\.3\]; Welchp=0\.33p=0\.33;d=0\.46d=0\.46\)\. Static vocabulary exceeds metrics\-only by19\.019\.0points, but that comparison is also not resolved atn=10n=10\(CI\[−10\.6,47\.9\]\[\-10\.6,47\.9\];p=0\.24p=0\.24;d=0\.54d=0\.54\)\. On KeyCorridor, static vocabulary captures most of the improvement over metrics\-only \(70\.7%70\.7\\%vs\.11\.5%11\.5\\%;Δ=59\.2\\Delta=59\.2points; CI\[32\.1,83\.5\]\[32\.1,83\.5\];p=0\.0017p=0\.0017;d=1\.90d=1\.90\), while full diagnostics trend higher than static vocabulary by16\.016\.0points but remain statistically unresolved \(CI\[−13\.6,46\.7\]\[\-13\.6,46\.7\];p=0\.34p=0\.34;d=0\.44d=0\.44\)\. We therefore interpret the prompt controls as evidence that the failure taxonomy is a major active ingredient; the online diagnostic labels may add value, but these experiments do not isolate them as the sole driver\. A seed\-level inspection illustrates why we avoid dismissing dynamic labels entirely: on DoorKey\-8×\\times8 seed 789, static\-vocab\-only reaches only1%1\\%final\-100 success, whereas the full diagnostic run reaches100%100\\%\. This single catastrophic static\-vocab seed explains much of the DoorKey static\-vocab variance and is suggestive, not conclusive, evidence that dynamic labels can rescue hard cases\.

### 8\.3Taxonomy Labeling Audit

We evaluate whether failure\-mode labels agree with author labels\. From the iterative refinement logs across all environments, we sample 49 reward functions that underwent refinement and manually label each with its primary failure mode: reward flooding, semantic/API misunderstanding, weak/negligible shaping, or none/mixed\. We then compare these labels against an independent LLM\-based auto\-labeler \(Claude Sonnet 4\.6, temperature 0\)\. This audit supports the taxonomy labeling used in our analysis for the two dominant modes; it does not validate the online trigger heuristics directly, which are separately evaluated through controls and sensitivity checks\.

Table 9:Taxonomy labeling audit: LLM auto\-labeler vs\. author labels on 49 reward functions\. The two dominant failure modes \(flooding, semantic/API misunderstanding\) achieve94%94\\%precision on predicted dominant labels \(32/34\)\. Overall accuracy across all categories is71\.4%71\.4\\%\(35/49\)\. Rare or mixed cases are grouped because weak/negligible shaping has support=1=1and should not be interpreted as a stable class\-level statistic\.The two dominant failure modes—reward flooding and semantic/API misunderstanding, which together account for40/4940/49author\-labeled samples—are labeled with94%94\\%precision when the auto\-labeler predicts one of those dominant modes\. In this 49\-function audit, the auto\-labeler does not miss any author\-labeled semantic/API misunderstanding cases\. Reward flooding recall is lower \(0\.650\.65\); the 8 missed flooding cases fall into rare or mixed categories, indicating a conservative boundary between continuous over\-rewarding, too\-small shaping, and ambiguous cases rather than confusion with semantic API errors\. We do not interpret rare or mixed classes as stable categories because weak/negligible shaping has only one author\-labeled example and none/mixed cases are inherently ambiguous\. A domain split emerges in the labeled data: semantic/API misunderstanding dominates MiniGrid failures, while reward flooding dominates continuous\-control failures—consistent with the structural differences between discrete event\-driven and dense continuous reward landscapes\. Because the reference labels are author labels rather than independently adjudicated human labels, this audit should be read as evidence that the dominant taxonomy categories are recoverable by an independent LLM labeler, not as a measurement of human inter\-annotator reliability\.

### 8\.4Threshold Sensitivity

We vary each diagnostic threshold independently while keeping all other components fixed, evaluating on three representative environments \(DoorKey\-8×\\times8, KeyCorridor, LavaGap\) with up to 5 successful seeds per configuration\. Table[10](https://arxiv.org/html/2605.28918#S8.T10)reports mean success rates\.

Table 10:Threshold sensitivity: mean success rate \(%\)±\\pmsample std from independent sensitivity runs \(up to 5 successful seeds per cell\) when varying one diagnostic threshold while holding others at default\. Default cells are sweep reruns rather than Table[2](https://arxiv.org/html/2605.28918#S6.T2)means\. The plateau cutoff is inactive under 500\-episode probes because the plateau gate requires\>1,000\>1\{,\}000probe episodes, so those rows should be read as rerun variability rather than a causal threshold effect\.Boldmarks the best cell per sweep\.Two actionable patterns emerge from Table[10](https://arxiv.org/html/2605.28918#S8.T10)\.Reward\-hacking threshold: increasing the cutoff \(less aggressive detection\) improves DoorKey \(80%→99%80\\%\\to 99\\%\) but slightly degrades KeyCorridor \(74%→48%74\\%\\to 48\\%\); together with the component\-removal stress test, this indicates that reward\-hacking detection is highly environment\-sensitive rather than monotonic in its threshold\.Shaping\-weakness threshold: shows non\-monotonic, environment\-dependent effects; no single direction is universally optimal\. The plateau rows are included for transparency, but because the\>1,000\>1\{,\}000\-episode gate is inactive in these 500\-episode probes, their variation should be interpreted as lower\-seed rerun and LLM stochasticity rather than evidence about the plateau cutoff itself\.

These low\-seed descriptive sweeps show that the method issensitive but diagnostically interpretable: threshold choices materially affect performance, especially on KeyCorridor, and the reward\-hacking detector in particular needs task\-specific calibration\. Bootstrap CIs on threshold differences are wide: for example, raising the reward\-hacking cutoff from0\.50\.5to1\.01\.0changes KeyCorridor by−5\.0\-5\.0points with CI\[−55\.2,47\.4\]\[\-55\.2,47\.4\], while lowering it to0\.30\.3changes KeyCorridor by\+20\.8\+20\.8points with CI\[−26\.0,64\.4\]\[\-26\.0,64\.4\]\. The only threshold\-difference CI that excludes zero is KeyCorridor’s plateau\-low row \(\+46\.8\+46\.8points, CI\[11\.2,82\.4\]\[11\.2,82\.4\]\), but the plateau gate is inactive under these probes, so we treat this as rerun sensitivity rather than a causal plateau\-threshold result\. We therefore use the sweeps as calibration evidence rather than as definitive threshold\-selection tests\.

### 8\.5Probe\-Length Sensitivity

The probe run is the diagnostic observation window: too short, and the diagnostics may lack signal; too long, and compute is wasted\. We evaluate probe lengths of 300, 500, and 800 episodes on three representative environments \(3 seeds each\)\.

Table 11:Probe\-length sensitivity: mean success rate±\\pmsample std from independent 3\-seed reruns\. Longer probes provide more reliable diagnostic observations, especially on hard environments\.Contrary to the intuition that shorter probes suffice for quick diagnosis,longer probes improve the aggregate mean and substantially stabilize KeyCorridor, though 800 episodes is not uniformly best on every environment\. The overall mean increases monotonically from62%62\\%\(300 eps\) to96%96\\%\(800 eps\), with variance dropping from±39%\\pm 39\\%to±7%\\pm 7\\%\. The effect is most pronounced on KeyCorridor, where probe==300 achieves only49%49\\%while probe==800 reaches100%100\\%across all seeds—the longer observation window gives the LLM less noisy refinement metrics and reduces premature or misleading updates\.

On DoorKey\-8×\\times8, the 300\-episode probe produces a catastrophic failure on one seed \(SR==0%\) while the other two seeds succeed, suggesting that 300 episodes is below the threshold needed for reliable reward\-function evaluation on this task\. The default of 500 episodes provides a reasonable balance, but practitioners working on hard tasks may benefit from longer probes\.

These results reinforce the paper’s central thesis: the diagnostic system’s effectiveness depends on the quality of its observations, not just the quality of its heuristics\.

### 8\.6Interface Ablation

The method depends on a curated reward\-function interface \(Section[11](https://arxiv.org/html/2605.28918#S11)\)\. To test how much the specificevent\_textfield contributes, we strip it from both the LLM prompt and the runtimeinfodictionary, forcing the LLM to rely on other semantic fields \(carrying,agent\_pos,step\_count\)\. We compare the standard pipeline \(“with events”\) against this degraded interface \(“without events”\) on DoorKey\-8×\\times8 and KeyCorridor \(5 seeds each, iterative refinement with 3 iterations\)\. The two ablation conditions use the same five canonical seed IDs and independent LLM generations; those seed IDs are a subset of the 10\-seed headline run\.

Table 12:Interface ablation: success rate \(%\)±\\pmsample std \(5 seeds\) whenevent\_textis removed from the LLM prompt and runtime info dict\. On KeyCorridor, removingevent\_text*improves*performance\.The results are nuanced\. On DoorKey\-8×\\times8, removingevent\_textcosts only2\.22\.2percentage points—within noise—because the LLM can infer subgoal completion fromcarryingtransitions \(e\.g\.,"nothing"→\\to"yellow key"implies key pickup\)\. On KeyCorridor, removingevent\_text*improves*performance by31\.631\.6points \(55\.2%→86\.8%55\.2\\%\\to 86\.8\\%\), essentially matching the 10\-seed headline mean of86\.7%86\.7\\%\. Because the ablation uses independent LLM generations, this near\-equality is not a paired replication of the headline result; it is nevertheless a strong warning against framing the KeyCorridor result as specifically event\-text\-driven\. Inspection of the generated reward functions suggests thatevent\_textstrings on KeyCorridor include misleading signals \(e\.g\., partial\-progress events for irrelevant objects\) that cause the LLM to write over\-specific reward functions; without these signals, the LLM falls back on more robust position\- and carrying\-based heuristics\.

This finding refines the paper’s scope claim: the method’s effectiveness depends on having*some*structured semantic interface \(carrying,agent\_pos\), but does not critically depend on any single field\. Rich event descriptions can help, hurt, or be neutral depending on how well they align with the task’s actual subgoal structure\.

## 9Analysis

### 9\.1Refinement Trajectory

Figure[4](https://arxiv.org/html/2605.28918#S9.F4)shows how probe success rate evolves across refinement iterations\. Key observations:

- •DoorKey\-5×\\times5: Often solved within 1–2 iterations; all methods converge to∼\\sim100% given 3,000 episodes, making this primarily a sample\-efficiency benchmark\.
- •DoorKey\-8×\\times8: Clear improvement trajectory across iterations\. Full training with the refined reward function reaches97\.6%97\.6\\%, up from58\.3%58\.3\\%with one\-shot\. Hand\-crafted achieves100%100\\%\.
- •KeyCorridor: The largest mean improvement—iterative refinement reaches86\.7%86\.7\\%vs\.31\.2%31\.2\\%one\-shot and29\.2%29\.2\\%hand\-crafted—but with high seed\-to\-seed variance\. In light of the static\-vocabulary andevent\_textcontrols, this result is best interpreted as evidence that taxonomy\-guided refinement can improve over one\-shot generation in a high\-variance sparse structured setting, not as isolated proof that dynamic diagnostic labels are the sole repair mechanism\.
- •LavaGap: Iterative refinement achieves88\.8%88\.8\\%, above the one\-shot \(64\.3%64\.3\\%\) and hand\-crafted \(63\.6%63\.6\\%\) baselines under this PPO/protocol implementation\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x3.png)Figure 4:Refinement trajectory: probe success rate at each refinement round\. Individual seed traces shown in light purple; mean in bold\. Stars indicate final full\-training results\. Dashed lines show one\-shot and hand\-crafted baselines\.
### 9\.2Continuous Control Results

To probe how the same reward\-design interface behaves beyond discrete grid worlds, we evaluate on four MuJoCo environments \(Figure[5](https://arxiv.org/html/2605.28918#S9.F5)\)\. The reward function interface remains identical—the LLM receives a natural\-language environment description and generates a Python function—but the results are best read as boundary evidence rather than a broad claim about continuous control\.

Reaching tasks\.On Reacher\-v4, LLM one\-shot achieves the highest success rate \(77\.0±4\.1%77\.0\\pm 4\.1\\%\), outperforming no shaping \(65\.0%65\.0\\%, Welch’sp<0\.001p<0\.001,d=1\.97d=1\.97; significant with per\-family correction within MuJoCo reaching\)\. On FetchReach\-v4 \(3,000 episodes\), iterative refinement achieves the highest success rate \(12\.8±8\.0%12\.8\\pm 8\.0\\%\), roughly2×2\\timesno shaping \(6\.5%6\.5\\%\), though all conditions remain far from solving this challenging sparse\-reward task\.

Locomotion tasks\.On HalfCheetah\-v4 and Hopper\-v4, the LLM designs alternative reward functions accessing velocity, position, and control cost\. Unlike reaching tasks, these environments already have dense default rewards, making them a stress test for diagnostics designed around sparse binary success\. In Table[2](https://arxiv.org/html/2605.28918#S6.T2), the locomotion iterative column reports the return\-aware diagnostic adaptation from Section[9\.4](https://arxiv.org/html/2605.28918#S9.SS4), while the original success\-diagnostic protocol is retained in Table[13](https://arxiv.org/html/2605.28918#S9.T13)as the broken iterative baseline\. The main finding is negative: iterative refinement does not robustly outperform one\-shot on locomotion tasks\. On HalfCheetah, return\-aware iterative refinement \(2,215±4852\{,\}215\\pm 485\) is statistically indistinguishable from LLM one\-shot \(2,190±4052\{,\}190\\pm 405, correctedp=0\.9795p=0\.9795\)\. On Hopper, no condition significantly outperforms the baseline, and the budget\-matched no\-shaping control in Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)is much stronger than return\-aware refinement\. These results suggest that the feedback\-driven diagnostic approach is most effective for sparse\-reward environments where the LLM can identify specific failure modes\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x4.png)Figure 5:MuJoCo learning curves \(smoothed over 50 episodes\)\. Top: reaching tasks \(success rate\)\. Bottom: locomotion tasks \(episode return\)\. Shaded regions show±\\pm1 sample std across seeds; return\-aware locomotion iterative curves use 6 seeds and other conditions use 10 seeds\.![Refer to caption](https://arxiv.org/html/2605.28918v1/x5.png)Figure 6:MuJoCo final performance: success rate for reaching tasks, episode return for locomotion tasks\. Error bars show±\\pm1 sample std across seeds; return\-aware locomotion iterative bars use 6 seeds and other conditions use 10 seeds\.
### 9\.3Secondary Comparisons

We keep the main analysis focused on the diagnostic mechanism\. Additional comparisons with RND, a naive LLM\+RND combination, exploratory model sensitivity, qualitative reward\-function patterns, and protocol\-level cost are summarized in Appendix[A](https://arxiv.org/html/2605.28918#A1)\. These analyses provide useful context, but they are not the central evidence for the debugging thesis\.

### 9\.4Dense\-Reward Boundary Case

The dense\-reward locomotion result is primarily a boundary finding\. The sharpest counterexample is Hopper: with the same total episode budget \(1,6001\{,\}600\), no\-shaping extended reaches882882while the original success\-diagnostic iterative protocol reaches only525525, and the return\-aware adaptation remains lower at472472with a shorter 1,000\-episode final training run\. Thus the refinement loop provides no advantage when the primary bottleneck is training duration rather than reward\-function quality\. HalfCheetah is less negative but still not a robust gain: success\-diagnostic iterative refinement achieves2,0402\{,\}040return compared to2,1902\{,\}190for one\-shot, and return\-aware refinement is statistically indistinguishable from one\-shot\.

Root cause\.Investigation reveals a systematic false\-positive failure in the diagnostic system\. The reward\-hacking detector fires when mean reward is high but success rate is low \(mr\>0\.5∧sr<0\.2\\text\{mr\}\>0\.5\\wedge\\text\{sr\}<0\.2\)\. On locomotion environments,is\_successis alwaysFalse\(there is no binary success criterion\), sosr=0\\text\{sr\}=0whilemr∼2,000\\text\{mr\}\\sim 2\{,\}000\. This triggers a “REWARD HACKING DETECTED” warning*every iteration*, causing the LLM to systematically strip effective shaping components from its reward function\.

Mechanism\-removal diagnostic adaptation\.We disable success\-based diagnostics for dense\-reward environments and replace them with*return\-trend analysis*: the system compares mean return in the first vs\. second half of the probe run, flagging “RETURN DECLINING” \(second half<0\.9×<0\.9\\timesfirst half, indicating destabilizing shaping\) or “RETURN STAGNATED” \(\|Δ\|<0\.05×\|\\Delta\|<0\.05\\timesfirst half, indicating insufficient signal\)\. A dedicated locomotion refinement prompt provides domain\-specific guidance: velocity milestone bonuses, stability bonuses, and magnitude constraints \(<<10% of typical episode return\)\. This prompt omits success rate \(always zero, confusing to the LLM\) and instead frames diagnostics in terms of return trajectory\.

We evaluate the adaptation on the first 6 completed canonical seeds per environment \(Table[13](https://arxiv.org/html/2605.28918#S9.T13)\)\. On HalfCheetah, the return\-aware iterative method achieves2,215±4852\{,\}215\\pm 485mean return, numerically above the broken iterative baseline \(2,040±6852\{,\}040\\pm 685\) and no shaping \(1,864±7491\{,\}864\\pm 749\) but not distinguishable from one\-shot\. On Hopper, the adapted method \(472±112472\\pm 112\) is comparable to no shaping \(453±104453\\pm 104\) but below both the broken iterative baseline \(525±161525\\pm 161\) and the budget\-matched no\-shaping control, consistent with Hopper benefiting primarily from training duration rather than reward function quality \(Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)\)\. Critically, the return\-trend diagnosticseliminate one specific systematic false\-positive failure: no run triggers the spurious “REWARD HACKING” warning that previously caused the LLM to strip effective shaping components\.

Table 13:Dense\-reward locomotion adaptation \(mean±\\pmsample std\)\. Return\-aware diagnostics remove the systematic false\-positive reward\-hacking warning, but do not establish robust locomotion gains\. Return\-aware results use the first 6 completed canonical seeds; other columns use 10 seeds\.This boundary case illustrates a central lesson:iterative refinement diagnostics must be aligned with the task’s evaluation signal\. In the sparse structured tasks studied here, where the diagnostics align with binary success signals, the framework is most effective\. When they are misaligned \(dense reward, no success criterion\), the feedback loop becomes counterproductive\. Return\-trend analysis removes the specific false\-positive reward\-hacking mechanism in locomotion, but the Hopper result shows that removing one diagnostic failure mechanism is not enough to produce metric gains in dense continuous control\.

### 9\.5Episode Budget Control

The iterative protocol uses more total training episodes than one\-shot \(3×\\times500 probe \+ 3,000 full = 4,500 for MiniGrid; 3×\\times200 \+ full for MuJoCo\)\. To control for this confound, we run extended\-budget experiments where no\-shaping and LLM one\-shot conditions train for the same total episodes as iterative \(Figure[7](https://arxiv.org/html/2605.28918#S9.F7)\)\. On the hardest MiniGrid environments, iterative refinement’s advantage persists strongly: DoorKey\-8×\\times8 iterative \(97\.6%97\.6\\%\) far exceeds both no\-shaping extended \(4\.7%4\.7\\%; raw Welchp<10−16p<10^\{\-16\},d=21\.3d=21\.3\) and LLM extended \(57\.9%57\.9\\%\)\. KeyCorridor iterative \(86\.7%86\.7\\%\) similarly outperforms no\-shaping extended \(12\.9%12\.9\\%; raw Welchp=1\.4×10−5p=1\.4\\times 10^\{\-5\},d=3\.6d=3\.6\) and LLM extended \(54\.8%54\.8\\%\)\. These two extended no\-shaping comparisons are part of the manuscript\-defined primary family in Appendix[B](https://arxiv.org/html/2605.28918#A2)\. UnlockPickup iterative \(24\.3%24\.3\\%\) also exceeds no\-shaping extended \(0\.3%0\.3\\%; raw Welchp<10−4p<10^\{\-4\},d=3\.3d=3\.3\), but we treat this budget\-control comparison as descriptive rather than part of Table[15](https://arxiv.org/html/2605.28918#A2.T15)\. These results support the interpretation that the sparse\-task gains come from reward function quality rather than additional training time\. However, on Hopper, the budget\-matched no\-shaping control \(882882\) far exceeds both standard no\-shaping \(453453\) and the original success\-diagnostic iterative protocol \(525525\)\. Even the return\-aware adaptation remains below this matched\-budget control \(472472at 1,000 episodes vs\.882882at 1,600 episodes\), indicating that Hopper’s bottleneck is training duration rather than reward design\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x6.png)Figure 7:Episode budget control: extended\-budget conditions \(4,500 episodes for MiniGrid\) compared with standard and iterative conditions\. MiniGrid extended\-budget runs use 4,500 episodes; MuJoCo extended\-budget runs use full budget\+\+600 probe\-equivalent episodes\. Seed counts match the corresponding main/budget\-control runs unless otherwise noted\. For locomotion, the iterative bar uses the original success\-diagnostic protocol, not the main Table[2](https://arxiv.org/html/2605.28918#S6.T2)return\-aware adaptation; it is retained here as the broken diagnostic baseline\.
### 9\.6Variance Decomposition

To understand when iterative refinement is most useful, we decompose observed performance variability into two components: variability from LLM reward\-function generation and variability from RL training stochasticity\. We decompose these sources by fixing one and varying the other: \(1\) fixing the LLM\-generated reward function from seed 42 and training with all 10 RL seeds to approximate RL variance, and \(2\) fixing RL seed 42 and training with each of the 10 LLM\-generated reward functions to approximate LLM\-generation variance\. This should be interpreted as a*point\-estimate decomposition*rather than a formal variance partition: because each component is anchored to a single fixed seed, the two terms are not orthogonal and need not sum to100%100\\%\.

In the sparse, structured MiniGrid environments, LLM\-function variance is typically much larger than RL variance\. LLM variance point estimates range from5959–118%118\\%of total variance, while RL variance is generally near zero \(Figure[8](https://arxiv.org/html/2605.28918#S9.F8)\)\. DoorKey\-8×\\times8 shows RL std of just0\.3%0\.3\\%vs\. LLM std of47\.5%47\.5\\%; DoorKey\-5×\\times5 shows zero RL variance vs\.31\.7%31\.7\\%LLM std\. This suggests that performance is more sensitive to*which*reward function the LLM produces than to RL training stochasticity, providing empirical support for why diagnostic refinement—improving a poor reward function—can materially change outcomes in these tasks\.

In the continuous\-control environments, the pattern shifts\. RL variance point estimates are larger \(124124–440%440\\%of total variance\), whereas LLM variance is more moderate \(4444–153%153\\%\)\. Although these ratios should not be interpreted literally as variance shares, they suggest that RL training stochasticity plays a larger role in continuous\-control settings, consistent with the weaker gains from iterative refinement observed in the locomotion domain\.

Limitations of this decomposition\.Because each marginal variance is estimated using a single anchor seed rather than a fully crossed design, the components are not statistically independent, and their sum can exceed100%100\\%as observed in several environments\. We report bootstrap95%95\\%confidence intervals \(2,0002\{,\}000resamples, reproducible seed\) alongside point estimates; these intervals are wide in several settings, especially continuous\-control environments \(e\.g\., HalfCheetah LLM std point estimate499499,95%95\\%CI\[77,722\]\[77,722\]\), indicating substantial instability in the exact ratio estimates\. Accordingly, the anchored decomposition is best read as directional evidence about which source of variability appears more important in each regime, rather than as a precise estimate of variance explained\.

Fully crossed decomposition\.To complement the anchored analysis, we run a fully crossed5×55\\times 5design \(5 LLM reward functions×\\times5 RL seeds, all 25 combinations trained\) on four representative environments spanning the variance spectrum\. This supports a two\-way random\-effects variance decomposition with the standard estimatorσ^LLM2=\(MSLLM−MSerr\)/R\\hat\{\\sigma\}^\{2\}\_\{\\text\{LLM\}\}=\(\\text\{MS\}\_\{\\text\{LLM\}\}\-\\text\{MS\}\_\{\\text\{err\}\}\)/R\(and symmetrically for RL\), clamping negative components to zero\. Because there is one observation per LLM–RL cell, the residual term includes interaction effects and unmodeled stochasticity\. Table[14](https://arxiv.org/html/2605.28918#S9.T14)reports point\-estimate variance shares and row/column\-bootstrap intervals over the LLM and RL seed axes\.

Table 14:Crossed5×55\\times 5variance decomposition via two\-way random\-effects ANOVA \(25 cells per env, one observation per cell\)\. Brackets show row/column\-bootstrap95%95\\%intervals for variance shares, using the same negative\-clamped estimator as the point estimates\. The residual includes LLM–RL interaction effects and unmodeled stochasticity\.Boldmarks the dominant variance component\. Metric: success rate for MiniGrid,final\_100\_rewardfor MuJoCo\.Table[14](https://arxiv.org/html/2605.28918#S9.T14)suggests a domain split in the crossed design that the anchored decomposition could only approximate, but the bootstrap intervals are wide and should temper any fine\-grained ranking\. On the sparse discrete tasks,LLM\-function variance has the largest point estimate: DoorKey\-8×\\times8 attributes96%96\\%of variance to the LLM reward function identity \(CI\[66,100\]%\[66,100\]\\%\), indicating that the identity of the LLM\-generated reward function accounts for most observed performance variation in this setting\. KeyCorridor shows a more balanced and uncertain split \(50%50\\%LLM, CI\[0,90\]%\[0,90\]\\%;50%50\\%residual, CI\[6,94\]%\[6,94\]\\%\), suggesting that LLM–RL interaction effects may play a larger role on this harder task\. On the continuous\-control tasks, the residual share has the largest point estimate \(5757–63%63\\%\), but the LLM and RL component intervals are broad\. The anchored design’s275\.6%275\.6\\%and439\.5%439\.5\\%RL shares on these envs \(Figure[8](https://arxiv.org/html/2605.28918#S9.F8)\) were inflated by its non\-orthogonal single\-seed anchoring; the crossed design provides cleaner descriptive point estimates, not precise variance\-share claims\.

This decomposition supports the method’s scoped interpretation most clearly for DoorKey\-8×\\times8: iterative refinement—which improves the LLM reward function—should be most useful where LLM variance appears dominant\. For KeyCorridor and the continuous\-control tasks, the crossed estimates are better read as directional diagnostics than as settled component rankings\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x7.png)Figure 8:Anchored variance ratios for RL training vs\. LLM generation per environment \(single\-seed anchored design\)\. Ratios can exceed 100% and are not variance shares because the components are non\-orthogonal; see Table[14](https://arxiv.org/html/2605.28918#S9.T14)for the fully crossed four\-environment design\.
### 9\.7Selection vs\. Refinement

An alternative to iterative refinement is*selection*: generateNNindependent reward functions, evaluate each via a short probe, and train with the best\. We implement best\-of\-3 selection with the same total episode budget as iterative \(3×\\times500 probe \+ 3,000 full for MiniGrid\)\. This approach captures some benefit of variance reduction by selecting the best from multiple independent samples\.

Figure[9](https://arxiv.org/html/2605.28918#S9.F9)compares best\-of\-3 selection with one\-shot and iterative refinement\. On DoorKey\-8×\\times8, best\-of\-3 \(92\.3%92\.3\\%\) substantially improves over one\-shot \(58\.3%58\.3\\%\) by filtering out poor candidates, though iterative \(97\.6%97\.6\\%\) still achieves the highest mean performance\. On KeyCorridor, best\-of\-3 \(60\.1%60\.1\\%\) roughly doubles one\-shot \(31\.2%31\.2\\%\) but remains below iterative \(86\.7%86\.7\\%\)\. On UnlockPickup, best\-of\-3 \(24\.3%24\.3\\%\) matches iterative \(24\.3%24\.3\\%\), both improving over one\-shot \(15\.8%15\.8\\%\)\. Seed\-level tests do not resolve the iterative\-vs\-selection differences atn=10n=10: DoorKey\-8×\\times8Δ=5\.3\\Delta=5\.3points \(Welchp=0\.51p=0\.51, bootstrap CI\[−3\.8,21\.4\]\[\-3\.8,21\.4\]\), KeyCorridorΔ=26\.6\\Delta=26\.6points \(p=0\.106p=0\.106, CI\[−2\.6,54\.8\]\[\-2\.6,54\.8\]\), and UnlockPickupΔ=0\.0\\Delta=0\.0points \(p=1\.0p=1\.0, CI\[−7\.9,6\.9\]\[\-7\.9,6\.9\]\)\. We therefore interpret Best\-of\-3 as a strong selection baseline: it captures most of the DoorKey\-8×\\times8 gain and matches UnlockPickup, while the KeyCorridor mean remains descriptively lower because independent candidates often share systematic flaws\. Note that best\-of\-3 MuJoCo locomotion results are based on fewer seeds \(HalfCheetah: 5/10, Hopper: 4/10\) due to reward function validation failures, limiting their statistical reliability\.

![Refer to caption](https://arxiv.org/html/2605.28918v1/x8.png)Figure 9:Best\-of\-3 selection vs\. LLM one\-shot vs\. iterative refinement\. MiniGrid Best\-of\-3 uses 10 seeds; MuJoCo locomotion Best\-of\-3 has incomplete validation\-successful seeds \(HalfCheetah: 5/10, Hopper: 4/10\)\.

## 10Related Work

LLMs as reward designers\.Eureka\(Maet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib2)\)uses GPT\-4 to generate reward functions for continuous\-control tasks, employing evolutionary search over hundreds of candidates evaluated in GPU\-parallel simulation \(IsaacGym\)\. Text2Reward\(Xieet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib3)\)generates dense executable reward code from language task descriptions and compact environment representations\. ELLM\(Duet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib4)\)uses LLMs to suggest exploration goals rather than reward functions\. Language to Rewards\(Yuet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib17)\)translates task specifications into reward parameters for manipulation\. More recent work extends this paradigm: Auto MC\-Reward\(Liet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib18)\)applies automated dense reward design to Minecraft, CARD\(Sunet al\.,[2025](https://arxiv.org/html/2605.28918#bib.bib19)\)refines LLM\-generated reward code using dynamic feedback and trajectory preference evaluation, and RL\-VLM\-F\(Wanget al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib20)\)uses vision\-language models for reward feedback\. Kwon et al\.\(Kwonet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib33)\)study prompting LLMs as proxy reward functions from zero\- or few\-shot task descriptions\. Our work complements these methods by providing a systematic failure\-mode taxonomy and a low\-call diagnostic refinement protocol; the comparison is protocol\-level, and direct performance comparison with population\-based methods remains a shared\-benchmark limitation\.

Reward misspecification and hacking\.A growing literature studies the risks of misspecified rewards\. Amodei et al\.\(Amodeiet al\.,[2016](https://arxiv.org/html/2605.28918#bib.bib21)\)identify reward hacking as a key AI safety concern\. Skalse et al\.\(Skalseet al\.,[2022](https://arxiv.org/html/2605.28918#bib.bib22)\)provide a formal characterization of reward hacking\. Pan et al\.\(Panet al\.,[2022](https://arxiv.org/html/2605.28918#bib.bib23)\)map the effects of reward misspecification on learned policies\. Clark and Amodei\(Clark and Amodei,[2016](https://arxiv.org/html/2605.28918#bib.bib24)\)document faulty reward functions in deployed systems\. Our failure\-mode taxonomy \(reward flooding, semantic/API misunderstanding, and rarer weak\-shaping cases\) connects to this literature by characterizing the specific ways LLM\-generated rewards become misspecified, and our diagnostic protocol provides a lightweight way to flag these failure modes during refinement\.

Reward shaping theory\.Potential\-based reward shaping\(Nget al\.,[1999](https://arxiv.org/html/2605.28918#bib.bib1)\)provides theoretical guarantees on policy invariance under additive shaping\. Devlin and Kudenko\(Devlin and Kudenko,[2012](https://arxiv.org/html/2605.28918#bib.bib28)\)extend this to dynamic potentials\. Wiewiora et al\.\(Wiewioraet al\.,[2003](https://arxiv.org/html/2605.28918#bib.bib30)\)establish connections between potential\-based shaping and Q\-value initialization\. Hu et al\.\(Huet al\.,[2020](https://arxiv.org/html/2605.28918#bib.bib29)\)propose learning to utilize shaping rewards adaptively\. Our LLM\-generated one\-time bonuses can be viewed as rough approximations to potential\-based shaping \(Section[4](https://arxiv.org/html/2605.28918#S4)\), connecting the empirical LLM approach to this theoretical reference point without implying a formal invariance guarantee\.

Intrinsic motivation\.Count\-based exploration\(Bellemareet al\.,[2016](https://arxiv.org/html/2605.28918#bib.bib7)\), curiosity\-driven methods\(Pathaket al\.,[2017](https://arxiv.org/html/2605.28918#bib.bib8)\), and Random Network Distillation\(Burdaet al\.,[2019](https://arxiv.org/html/2605.28918#bib.bib13)\)provide intrinsic rewards without domain knowledge\. Oudeyer et al\.\(Oudeyeret al\.,[2007](https://arxiv.org/html/2605.28918#bib.bib25)\)survey intrinsic motivation systems for autonomous development\. Go\-Explore\(Ecoffetet al\.,[2021](https://arxiv.org/html/2605.28918#bib.bib26)\)uses archiving and goal\-directed exploration\. Taiga et al\.\(Taigaet al\.,[2021](https://arxiv.org/html/2605.28918#bib.bib27)\)provide a systematic evaluation of bonus\-based exploration\. We include fixed\-coefficient RND as a contextual baseline and add a small DoorKey coefficient sweep\. The sweep confirms that RND is highly coefficient\-sensitive: tuning solves DoorKey\-5×\\times5 but does not rescue DoorKey\-8×\\times8 under our PPO protocol\.

LLMs for code generation\.Code\-generating LLMs\(Chenet al\.,[2021](https://arxiv.org/html/2605.28918#bib.bib9)\)have been applied broadly, with specialized models like StarCoder\(Liet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib31)\)and Code Llama\(Roziereet al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib32)\)achieving strong performance on programming benchmarks\. In RL, this capability enables generating executable reward functions\(Maet al\.,[2024](https://arxiv.org/html/2605.28918#bib.bib2)\), embodied\-agent skill code\(Wanget al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib10)\), and policy code\(Lianget al\.,[2023](https://arxiv.org/html/2605.28918#bib.bib11)\)\. Our iterative refinement with training diagnostics is analogous to the code\-test\-debug cycle in software engineering, but applied to reward function design rather than general programming\.

## 11Limitations and Future Work

Scope: structured interfaces required\.Our method depends on a curated reward\-function interface: theinfodictionary exposes semantic fields \(carrying,agent\_pos\) and event descriptions \(event\_text\)\. An interface ablation \(Section[8\.6](https://arxiv.org/html/2605.28918#S8.SS6)\) shows the method does not critically depend on any single field—removingevent\_textis neutral on DoorKey\-8×\\times8 and actually*improves*KeyCorridor—but it does require*some*structured semantic interface\. In environments without such interfaces, the LLM would need to interpret raw observations \(e\.g\., pixel arrays\), which is a fundamentally harder problem that our current framework does not address\. This domain\-specificity echoes recent LLM systems work in financial document routing and reranking\(Chenget al\.,[2026b](https://arxiv.org/html/2605.28918#bib.bib34);[a](https://arxiv.org/html/2605.28918#bib.bib35)\)and hardware\-aware vision–language deployment\(Chenet al\.,[2025](https://arxiv.org/html/2605.28918#bib.bib37)\), where task schemas, retrieval structure, or deployment substrate materially affect system behavior\.

Dense\-reward settings require different diagnostics\.As demonstrated in Section[9\.4](https://arxiv.org/html/2605.28918#S9.SS4), success\-based heuristics are counterproductive on dense\-reward tasks\. Our return\-trend adaptation addresses the specific locomotion failure mechanism, but other dense\-reward domains may require further diagnostic adaptations\. The framework is extensible, but each new domain class may need its own diagnostic design\.

Diagnostic thresholds\.The diagnostics use fixed heuristic thresholds \(Table[16](https://arxiv.org/html/2605.28918#A3.T16)\)\. Our sensitivity analysis \(Section[8\.4](https://arxiv.org/html/2605.28918#S8.SS4)\) shows that thresholds materially affect performance but in diagnostically interpretable ways, and the stress test \(Section[8\.1](https://arxiv.org/html/2605.28918#S8.SS1)\) reveals that the reward\-hacking detector can generate false positives that hurt performance on some environments\. Adaptive threshold selection—e\.g\., based on environment characteristics or early training dynamics—is a natural direction for future work\.

Environment complexity\.While we evaluate across 13 environments spanning discrete grid\-worlds, continuous reaching, and continuous locomotion, these use low\-dimensional state observations\. Extending to high\-dimensional observation spaces and long\-horizon manipulation tasks would test the scope of the approach beyond the structured setting studied here\.

Scaling to harder tasks\.BlockedUnlockPickup remains below1%1\\%success across all methods, MultiRoom\-N6 near0%0\\%, and UnlockPickup achieves only∼\\sim24% with iterative refinement\. Tasks requiring very long planning horizons or complex reward decompositions remain challenging\.

Model diversity\.We test four models across three providers with 3 seeds each \(Appendix[A](https://arxiv.org/html/2605.28918#A1)\)\. This provides exploratory evidence that the phenomenon is not unique to one model, but the sample size is insufficient for strong model\-level claims\. Open\-source models may exhibit different failure patterns\.

Algorithm and baseline scope\.All experiments use PPO, so some observed failure modes may reflect interactions between reward design and PPO optimization rather than reward functions alone\. All conclusions should therefore be read as PPO\-protocol conclusions, not algorithm\-agnostic reward\-function validity claims\. Testing an off\-policy algorithm such as SAC would help separate reward\-intrinsic failures from algorithm\-specific effects\. The main RND baseline uses a fixed intrinsic\-reward coefficient of0\.10\.1; a post\-hoc DoorKey sweep shows that this value is badly mistuned on DoorKey\-5×\\times5 but that the swept coefficients still fail on DoorKey\-8×\\times8\. The comparison with evolutionary reward\-design methods is a cost/protocol comparison rather than a shared\-benchmark head\-to\-head evaluation\.

Variance decomposition\.The anchored decomposition \(Section[9\.6](https://arxiv.org/html/2605.28918#S9.SS6)\) uses a single seed per component, producing non\-orthogonal point estimates\. We supplement it with a fully crossed5×55\\times 5design on four representative environments \(Table[14](https://arxiv.org/html/2605.28918#S9.T14)\), which provides a cleaner two\-way random\-effects partition while leaving interaction effects in the residual\. The crossed row/column\-bootstrap intervals are wide, so extending the crossed design beyond these four environments would further strengthen the analysis\.

Refinement protocol design\.The current protocol discards the probe\-trained policy and retrains from scratch, which simplifies evaluation but wastes probe computation; warm\-starting from the probe policy could improve sample efficiency\. In addition, strong diagnostic warning labels such as “REWARD HACKING DETECTED” may prime the LLM toward overcorrection\. Softer or more specific feedback is an important future prompt\-design variable\.

Temperature and per\-seed LLM calls\.LLM temperature is fixed at0\.40\.4throughout; systematic temperature tuning is left for future work\. Each seed uses its own LLM\-generated reward function, conflating LLM and RL variance; our decomposition analysis quantifies but does not eliminate this confound\.

## 12Conclusion

We have argued that LLM reward design for sparse, structured RL tasks is better understood as a debugging problem than a generation problem\. One\-shot generation fails in recurring, identifiable ways; lightweight diagnostics can often detect and repair these failures cheaply in the studied sparse structured setting; and the approach breaks predictably when diagnostics are misaligned with the task’s evaluation signal\.

The evidence supports this thesis on multiple fronts\. The failure taxonomy identifies reward flooding and semantic/API misunderstanding as dominant patterns in our audit, with weak or negligible shaping appearing as a rarer and less reliably labeled case\. Diagnostic\-driven iterative refinement achieves97\.6%97\.6\\%success on DoorKey\-8×\\times8 \(from2\.3%2\.3\\%without shaping\) and86\.7%86\.7\\%on KeyCorridor \(from31\.2%31\.2\\%one\-shot\), though KeyCorridor remains high variance across seeds and sensitivity checks\. Budget\-matched controls remain significant on the primary sparse tasks; Appendix[B](https://arxiv.org/html/2605.28918#A2)reports the correctedpp\-values\. The diagnostics are evaluated through component\-removal stress tests \(which reveal reward\-hacking false positives and lower\-seed sensitivity\), comparison against a metrics\-only baseline \(2929–7575point gaps\), and a static\-vocabulary control showing that the taxonomy prompt itself recovers much of the metrics\-only gap\. We therefore interpret the mechanism as taxonomy\-guided prompting plus diagnostic refinement, not dynamic warning labels alone\. The taxonomy audit against author labels reports94%94\\%precision \(32/34 predicted labels\) on the two dominant failure modes and71\.4%71\.4\\%overall accuracy across all categories\.

A fully crossed5×55\\times 5variance decomposition provides descriptive support for when refinement is most useful: on DoorKey\-8×\\times8, LLM\-function variance has a96%96\\%point\-estimate share with a wide bootstrap interval, consistent with reward\-function correction being especially useful in that setting\. On continuous\-control tasks, residual and RL variance have larger point estimates \(5757–63%63\\%residual,2323–30%30\\%RL\), limiting the benefit of improving any single reward function\. The locomotion boundary case demonstrates both the framework’s failure mode \(success\-based false positives on dense\-reward tasks\) and a diagnostic adaptation that removes that specific failure mechanism without producing robust locomotion gains\.

The method is strongest for sparse, structured tasks with reliable semantic interfaces—though an interface ablation shows that richer fields such asevent\_textare not uniformly helpful—and uses far fewer LLM calls than population\-based reward\-search protocols\. The failure\-mode taxonomy, diagnostic protocol, and variance analysis may inform future work on structured reward design beyond the environments studied here\.

## References

- D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Mané \(2016\)Concrete problems in AI safety\.InarXiv preprint arXiv:1606\.06565,Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p2.1)\.
- M\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. Munos \(2016\)Unifying count\-based exploration and intrinsic motivation\.Advances in Neural Information Processing Systems29\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2)\.
- Y\. Burda, H\. Edwards, A\. Storkey, and O\. Klimov \(2019\)Exploration by random network distillation\.InInternational Conference on Learning Representations,Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2),[§2\.5](https://arxiv.org/html/2605.28918#S2.SS5.p1.1),[item 5](https://arxiv.org/html/2605.28918#S5.I1.i5.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p5.1)\.
- W\. Chen, L\. Wu, Y\. Hu, Z\. Li, Z\. Cheng, Y\. Qian, L\. Zhu, Z\. Hu, L\. Liang, Q\. Tang, Z\. Liu, and H\. Yang \(2025\)AutoNeural: co\-designing vision\-language models for NPU inference\.External Links:2512\.02924,[Link](https://arxiv.org/abs/2512.02924)Cited by:[§11](https://arxiv.org/html/2605.28918#S11.p1.1)\.
- Z\. Cheng, L\. Lai, Y\. Liu, K\. Cheng, and X\. Qi \(2026a\)Enhancing financial report question\-answering: a retrieval\-augmented generation system with reranking analysis\.External Links:2603\.16877,[Link](https://arxiv.org/abs/2603.16877)Cited by:[§11](https://arxiv.org/html/2605.28918#S11.p1.1)\.
- Z\. Cheng, L\. Lai, and Y\. Liu \(2026b\)Resolving the robustness\-precision trade\-off in financial RAG through hybrid document\-routed retrieval\.External Links:2603\.26815,[Link](https://arxiv.org/abs/2603.26815)Cited by:[§11](https://arxiv.org/html/2605.28918#S11.p1.1)\.
- M\. Chevalier\-Boisvert, B\. Dai, M\. Towers, R\. de Lazcano, L\. Willems, S\. Lahlou, S\. Pal, P\. S\. Castro, and J\. Terry \(2023\)Minigrid & miniworld: modular & customizable reinforcement learning environments for goal\-oriented tasks\.Advances in Neural Information Processing Systems36\.Cited by:[§2\.3](https://arxiv.org/html/2605.28918#S2.SS3.p1.1)\.
- J\. Clark and D\. Amodei \(2016\)Faulty reward functions in the wild\.OpenAI Blog\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p2.1)\.
- S\. Devlin and D\. Kudenko \(2012\)Dynamic potential\-based reward shaping\.InProceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems,pp\. 433–440\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p3.1),[§4\.1](https://arxiv.org/html/2605.28918#S4.SS1.p3.1)\.
- Y\. Du, O\. Watkins, Z\. Wang, C\. Colas, T\. Darrell, P\. Abbeel, A\. Gupta, and J\. Andreas \(2023\)Guiding pretraining in reinforcement learning with large language models\.Proceedings of the 40th International Conference on Machine Learning\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.
- A\. Ecoffet, J\. Huizinga, J\. Lehman, K\. O\. Stanley, and J\. Clune \(2021\)First return, then explore\.Nature590,pp\. 580–586\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2)\.
- Y\. Hu, W\. Wang, H\. Jia, Y\. Wang, Y\. Chen, J\. Hao, F\. Wu, and C\. Fan \(2020\)Learning to utilize shaping rewards: a new approach of reward shaping\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p3.1)\.
- M\. Kwon, S\. M\. Xie, K\. Bullard, and D\. Sadigh \(2023\)Reward design with language models\.InInternational Conference on Learning Representations,Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.
- H\. Li, X\. Yang, Z\. Wang, X\. Zhu, J\. Zhou, Y\. Qiao, X\. Wang, H\. Li, L\. Lu, and J\. Dai \(2024\)Auto MC\-reward: automated dense reward design with large language models for Minecraft\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16426–16435\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.
- R\. Li, L\. B\. Allal, Y\. Zi, N\. Muennighoff, D\. Kocetkov, C\. Mou, M\. Marone, C\. Akiki, J\. Li, J\. Chim,et al\.\(2023\)StarCoder: may the source be with you\!\.Transactions on Machine Learning Research\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p5.1)\.
- J\. Liang, W\. Huang, F\. Xia, P\. Xu, K\. Hausman, B\. Ichter, P\. Florence, and A\. Zeng \(2023\)Code as policies: language model programs for embodied control\.IEEE International Conference on Robotics and Automation\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p5.1)\.
- Y\. J\. Ma, W\. Liang, G\. Wang, D\. Huang, O\. Bastani, D\. Jayaraman, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Eureka: human\-level reward design via coding large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.28918#S1.p2.1),[§10](https://arxiv.org/html/2605.28918#S10.p1.1),[§10](https://arxiv.org/html/2605.28918#S10.p5.1),[§2\.2](https://arxiv.org/html/2605.28918#S2.SS2.p1.1)\.
- A\. Y\. Ng, D\. Harada, and S\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.Proceedings of the 16th International Conference on Machine Learning,pp\. 278–287\.Cited by:[§1](https://arxiv.org/html/2605.28918#S1.p1.1),[§10](https://arxiv.org/html/2605.28918#S10.p3.1),[§2\.1](https://arxiv.org/html/2605.28918#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.28918#S4.SS1.p1.2),[§4\.1](https://arxiv.org/html/2605.28918#S4.SS1.p3.1)\.
- P\. Oudeyer, F\. Kaplan, and V\. V\. Hafner \(2007\)Intrinsic motivation systems for autonomous mental development\.IEEE Transactions on Evolutionary Computation11\(2\),pp\. 265–286\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2)\.
- A\. Pan, K\. Bhatia, and J\. Steinhardt \(2022\)The effects of reward misspecification: mapping and mitigating misaligned models\.International Conference on Learning Representations\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p2.1)\.
- D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell \(2017\)Curiosity\-driven exploration by self\-supervised prediction\.InInternational Conference on Machine Learning,pp\. 2778–2787\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2)\.
- M\. Plappert, M\. Andrychowicz, A\. Ray, B\. McGrew, B\. Baker, G\. Powell, J\. Schneider, J\. Tobin, M\. Chociej, P\. Welinder, V\. Kumar, and W\. Zaremba \(2018\)Multi\-goal reinforcement learning: challenging robotics environments and request for research\.External Links:1802\.09464Cited by:[§2\.4](https://arxiv.org/html/2605.28918#S2.SS4.p1.1)\.
- B\. Roziere, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.\(2023\)Code Llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p5.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[item 2](https://arxiv.org/html/2605.28918#S3.I1.i2.p1.1),[§5\.3](https://arxiv.org/html/2605.28918#S5.SS3.p1.1)\.
- J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward hacking\.Advances in Neural Information Processing Systems35\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p2.1)\.
- S\. Sun, R\. Liu, J\. Lyu, J\. Yang, L\. Zhang, and X\. Li \(2025\)A large language model\-driven reward design framework via dynamic feedback for reinforcement learning\.Knowledge\-Based Systems326,pp\. 114065\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2025.114065)Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.
- A\. A\. Taiga, W\. Fedus, M\. C\. Machado, A\. Courville, and M\. G\. Bellemare \(2021\)On bonus\-based exploration methods in the arcade learning environment\.InInternational Conference on Learning Representations,Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p4.2)\.
- E\. Todorov, T\. Erez, and Y\. Tassa \(2012\)MuJoCo: a physics engine for model\-based control\.InIEEE/RSJ International Conference on Intelligent Robots and Systems,pp\. 5026–5033\.Cited by:[§2\.4](https://arxiv.org/html/2605.28918#S2.SS4.p1.1)\.
- M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. De Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Kha, N\. Koptenko,et al\.\(2024\)Gymnasium: a standard interface for reinforcement learning environments\.arXiv preprint arXiv:2407\.17032\.Cited by:[§2\.4](https://arxiv.org/html/2605.28918#S2.SS4.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p5.1)\.
- Y\. Wang, Z\. Sun, J\. Zhang, Z\. Xian, E\. Biyik, D\. Held, and Z\. Erickson \(2024\)RL\-VLM\-F: reinforcement learning from vision language foundation model feedback\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.
- E\. Wiewiora, G\. W\. Cottrell, and C\. Elkan \(2003\)Principled methods for advising reinforcement learning agents\.Proceedings of the 20th International Conference on Machine Learning,pp\. 792–799\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p3.1)\.
- T\. Xie, S\. Zhao, C\. H\. Wu, Y\. Liu, Q\. Luo, V\. Zhong, Y\. Yang, and T\. Yu \(2024\)Text2Reward: reward shaping with language models for reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.28918#S1.p2.1),[§10](https://arxiv.org/html/2605.28918#S10.p1.1),[§2\.2](https://arxiv.org/html/2605.28918#S2.SS2.p1.1)\.
- W\. Yu, N\. Gileadi, C\. Fu, S\. Kirmani, K\. Lee, M\. G\. Arenas, H\. L\. Chiang, T\. Erez, L\. Hasenclever, J\. Humplik,et al\.\(2023\)Language to rewards for robotic skill synthesis\.arXiv preprint arXiv:2306\.08647\.Cited by:[§10](https://arxiv.org/html/2605.28918#S10.p1.1)\.

## Appendix ASecondary Analyses and Practical Takeaways

Intrinsic motivation\.With the fixed coefficient used in the main table, RND behaves differently from LLM\-designed shaping\. It is competitive on LavaGap where broad exploration is useful, but the DoorKey sweep shows high coefficient sensitivity:c=0\.01c=0\.01solves DoorKey\-5×\\times5, whilec=0\.1c=0\.1andc=0\.5c=0\.5do not, and none of the three coefficients solves DoorKey\-8×\\times8\. A naive LLM\+RND additive combination also hurts structured MiniGrid tasks \(e\.g\., DoorKey\-8×\\times8 drops from58\.3%58\.3\\%with LLM alone to0\.1%0\.1\\%\), suggesting that task\-specific and domain\-agnostic rewards require careful balancing rather than simple addition\.

Evolutionary reward design and cost\.Population\-based methods such as Eureka evaluate many reward candidates in GPU\-parallel simulation\. Our protocol instead uses 1–4 LLM calls per reward\-design run, repeated across seeds for evaluation, and CPU training\. This is a protocol\-level efficiency comparison, not a shared\-benchmark performance comparison: evolutionary search may find better functions when broad search and GPU simulation are available, while diagnostic refinement is aimed at low\-call correction when failure modes are identifiable\.

Qualitative reward\-function patterns\.Generated rewards commonly use event\-driven one\-time bonuses, position\-based progress heuristics, and bonus magnitudes comparable to hand\-crafted shaping after refinement\. These qualitative patterns support the debugging interpretation but are secondary to the controlled comparisons in the main text\.

Exploratory model sensitivity\.A small sweep across Claude Haiku, Claude Sonnet 4\.6, GPT\-5\.2, and Grok 4 on four environments with 3 seeds each suggests that one\-shot variance is not unique to one model and that refinement can help across model instances\. The sample size is too small for strong model\-level claims\.

Practical takeaways\.One\-shot generation is best suited to straightforward tasks with clear subgoals; iterative refinement is most useful for sparse structured tasks where one\-shot generation is high variance; Best\-of\-NNselection is useful when independent candidates include a good reward; dense\-reward tasks require diagnostic adaptation; and the variance analysis suggests prioritizing reward\-function quality when LLM variance dominates, while prioritizing RL optimization when training stochasticity dominates\.

## Appendix BPrimary Statistical Tests

Table[15](https://arxiv.org/html/2605.28918#A2.T15)lists the 12 manuscript\-defined primary comparisons used for the “9/12 significant” statement in Table[2](https://arxiv.org/html/2605.28918#S6.T2)\. This final submission\-prep family \(version 2026\-05\-11\) focuses on the sparse structured tasks where the method is expected to help most, includes two sparse\-task budget controls \(DoorKey\-8×\\times8 and KeyCorridor iterative vs\. extended no\-shaping\), and keeps MuJoCo reaching/locomotion comparisons in separate families as boundary checks\. The set is not an external preregistration, and it is not meant to imply that every cell in Table[2](https://arxiv.org/html/2605.28918#S6.T2)or every descriptive statistic in Section[9\.5](https://arxiv.org/html/2605.28918#S9.SS5)is part of the primary test family\. We apply Welch’stt\-test within each environment and Holm–Bonferroni correction separately within the MiniGrid, MuJoCo reaching, and MuJoCo locomotion families\. The table is generated from the same logs as Table[2](https://arxiv.org/html/2605.28918#S6.T2); locomotion uses episode return and all other rows use final\-100\-episode success rate\.

Table 15:Manuscript\-defined primary hypothesis tests used for the “9/12 significant” statement in Table[2](https://arxiv.org/html/2605.28918#S6.T2)\. Significance is assessed atpcorr<0\.05p\_\{\\mathrm\{corr\}\}<0\.05after per\-family Holm–Bonferroni correction\. pp = percentage points\.
## Appendix CThreshold Definitions

Table[16](https://arxiv.org/html/2605.28918#A3.T16)enumerates all trigger values used in the iterative refinement procedure\. All values correspond exactly to those used in our implementation\.

Table 16:All trigger values and thresholds used in the iterative refinement procedure\. No hidden defaults exist beyond this table\.
## Appendix DPrompt Templates

### D\.1Generation Prompt

The following prompt template is used for MiniGrid one\-shot reward function generation\. The\{env\_description\}placeholder is filled with a natural\-language description of the specific environment\. The supplementary code includes the MuJoCo reaching and locomotion prompt variants; these replace the discrete action description with continuous observation/action fields, and the locomotion variant uses return\-trend diagnostics rather than binary success diagnostics\.

1YouareanexpertrewardfunctiondesignerforRL\.

2

3Givenadescriptionofanenvironment,writea

4PythonrewardshapingfunctionthathelpsaPPO

5agentlearnfaster\.

6

7

8\{env\_description\}

9

10

11defreward\_fn\(obs,action,reward,terminated,

12truncated,info,state\):

13

14

15

16

17

18

19

20

21

221\.Startwiththeoriginalreward\.

232\.Addone\-timebonuses\(\+0\.1to\+0\.3\)forsubgoals\.

243\.UsestatedicttoensurebonusesgivenonlyONCE\.

254\.Checkinfo\["event\_text"\]forevents\.

265\.Keepbonusessmallvsgoalreward\(~1\.0\)\.

276\.Noimports\.Self\-containedfunction\.

### D\.2Refinement Prompt

For iterative refinement, the prompt additionally includes the previous reward function source code, training metrics, and diagnosed failure modes:

1

2\{current\_source\}

3

4

5\-Episodestrained:\{episodes\_trained\}

6\-Successrate:\{success\_rate\}

7\-Meanreward:\{mean\_reward\}

8

9

10\{diagnoses\}

11

12

131\.Rewardflooding:DoNOTaddper\-stepbonuses\.

142\.Action\-indexconfusion:MiniGridaction

150=turn\_left,2=forward,NOTdirections\.

163\.Too\-weakshaping:\+0\.1maybetoosmall\.

17Useposition\-basedprogresstracking\.

## Appendix EIterative Refinement Example

Table[17](https://arxiv.org/html/2605.28918#A5.T17)shows the iterative refinement process on DoorKey\-8×\\times8 \(seed 42\)\. The LLM progressively improves the reward function based on probe training diagnostics\.

IterProbe SRKey Changes029%Basic one\-time bonuses for key pickup \(\+0\.2\) and door open \(\+0\.25\)\. Continuous holding\-key bonus causes reward flooding\.167%Removed continuous bonus\. Added position\-based progress: bonus for reaching new maxxx\-coordinate\. Tracks exploration state\.278%Increased door bonus to \+0\.3\. Added room\-aware distance tracking\. Stagnation penalty \(60 steps\)\.Final \(3000 eps\)99%success rateTable 17:Iterative refinement on DoorKey\-8×\\times8 seed 42\. Each iteration receives the previous function’s code and probe metrics\. The LLM diagnoses issues and revises the reward function, improving from 29% to 99%\. See Appendix[F](https://arxiv.org/html/2605.28918#A6)for the actual generated code\.The progression shows the LLM addressing specific weaknesses: iteration 1 adds exploration incentives when basic milestones prove insufficient, and iteration 2 increases bonus magnitudes and adds anti\-stagnation measures when the agent plateaus at 67%\.

## Appendix FLLM\-Generated Reward Functions

We present the actual LLM\-generated reward functions from the DoorKey\-8×\\times8 iterative refinement \(seed 42\), showing how the model evolves its reward design across iterations\.

### F\.1Iteration 0: Initial One\-Shot Generation \(Probe SR: 29%\)

The initial function uses basic event\-driven bonuses but includes two problematic elements: a continuous per\-step reward for holding the key \(\+0\.01\) that can cause reward flooding, and a late\-game penalty that discourages exploration\.

1defreward\_fn\(obs,action,reward,terminated,

2truncated,info,state\):

3shaped\_reward=reward

4if"key\_picked\_up"notinstate:

5state\["key\_picked\_up"\]=False

6if"door\_opened"notinstate:

7state\["door\_opened"\]=False

8if"goal\_reached"notinstate:

9state\["goal\_reached"\]=False

10event\_text=info\.get\("event\_text",""\)

11carrying=info\.get\("carrying",""\)

12step\_count=info\.get\("step\_count",0\)

13max\_steps=info\.get\("max\_steps",1\)

14

15ifnotstate\["key\_picked\_up"\]\\

16and"pickedup"inevent\_text\.lower\(\)\\

17and"key"inevent\_text\.lower\(\):

18shaped\_reward\+=0\.2

19state\["key\_picked\_up"\]=True

20ifnotstate\["door\_opened"\]\\

21and"openeddoor"inevent\_text\.lower\(\):

22shaped\_reward\+=0\.25

23state\["door\_opened"\]=True

24

25ifstate\["key\_picked\_up"\]\\

26andnotstate\["door\_opened"\]\\

27and"key"incarrying\.lower\(\):

28shaped\_reward\+=0\.01

29

30ifnotstate\["goal\_reached"\]\\

31andstep\_count\>max\_steps\*0\.8:

32shaped\_reward\-=0\.02

33returnshaped\_reward,state

### F\.2Iteration 3: Final Refined Function \(Final SR: 99%\)

After three refinement rounds with diagnostic feedback, the function removes continuous bonuses, adds room\-aware progress tracking, and includes stagnation detection:

1defreward\_fn\(obs,action,reward,terminated,

2truncated,info,state\):

3shaped\_reward=reward

4if"key\_picked\_up"notinstate:

5state\["key\_picked\_up"\]=False

6if"door\_opened"notinstate:

7state\["door\_opened"\]=False

8if"last\_agent\_pos"notinstate:

9state\["last\_agent\_pos"\]=\(0,0\)

10if"steps\_without\_position\_change"notinstate:

11state\["steps\_without\_position\_change"\]=0

12if"max\_x\_before\_door"notinstate:

13state\["max\_x\_before\_door"\]=0

14if"max\_x\_after\_door"notinstate:

15state\["max\_x\_after\_door"\]=0

16event\_text=info\.get\("event\_text",""\)\.lower\(\)

17agent\_pos=info\.get\("agent\_pos",\(0,0\)\)

18current\_x,current\_y=agent\_pos

19

20ifnotstate\["key\_picked\_up"\]\\

21and"pickedup"inevent\_text\\

22and"key"inevent\_text:

23shaped\_reward\+=0\.2

24state\["key\_picked\_up"\]=True

25ifnotstate\["door\_opened"\]\\

26and"openeddoor"inevent\_text:

27shaped\_reward\+=0\.3

28state\["door\_opened"\]=True

29

30ifagent\_pos\!=state\["last\_agent\_pos"\]:

31state\["steps\_without\_position\_change"\]=0

32else:

33state\["steps\_without\_position\_change"\]\+=1

34

35ifnotstate\["door\_opened"\]:

36ifcurrent\_x\>state\["max\_x\_before\_door"\]:

37state\["max\_x\_before\_door"\]=current\_x

38else:

39ifcurrent\_x\>state\["max\_x\_after\_door"\]:

40state\["max\_x\_after\_door"\]=current\_x

41

42ifstate\["steps\_without\_position\_change"\]\>60\\

43andreward<0\.5:

44shaped\_reward\-=0\.05

45state\["last\_agent\_pos"\]=agent\_pos

46returnshaped\_reward,state

Key improvements the LLM made across iterations: \(1\) removed the continuous\+0\.01holding\-key bonus that caused reward flooding, \(2\) added room\-awarexx\-coordinate progress tracking to guide exploration in the 8×\\times8 grid, \(3\) replaced the coarse late\-game penalty with a position\-based stagnation detector \(60 steps without movement\), and \(4\) increased the door\-opening bonus from 0\.25 to 0\.3\. These changes resemble the kinds of local refinements a reward engineer might make after inspecting training diagnostics\.