Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

# Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
Source: [https://arxiv.org/html/2607.26094](https://arxiv.org/html/2607.26094)
###### Abstract

Reinforcement Learning from Human Feedback \(RLHF\) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task\-agnostic reward models\. This mismatch leads to sparse learning signals and suboptimal alignment\. We introduce MeRLa \(Meta\-Learned Reward Shaping\), a principled framework that meta\-learns a task\-aware shaping functionΦ​\(x,y;ϕ\)\\Phi\(x,y;\\phi\)across auxiliary tasks before RLHF training\. The learned shaping produces a composite reward that preserves policy optimality while providing task\-specific learning signals\. Our meta\-objective combines task discrimination, entropy regularization, and potential\-based conservation for stable convergence\. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization\. Experiments on LLaMA\-3\-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO—90\.8% length\-controlled win rate on AlpacaEval 2\.0 and 9\.14 on MT\-Bench, with 41% less training instability\. MeRLa retains its benefits when combined with process\-based and rubric\-based enhanced rewards\.

## Introduction

Reinforcement Learning from Human Feedback \(RLHF\) is a cornerstone of post\-training alignment\(Ouyanget al\.[2022](https://arxiv.org/html/2607.26094#bib.bib1); Baiet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib2)\)\. The standard pipeline first trains a reward modelRψR\_\{\\psi\}on pairwise preference data\(Wanget al\.[2024](https://arxiv.org/html/2607.26094#bib.bib6); Gaoet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib7)\), then optimizes the policy via PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2607.26094#bib.bib3)\), GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib4)\), or DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib16)\)\. Despite its success, this paradigm faces well\-documented limitations\.

Trained on static, task\-agnostic preference data, the reward model provides a fixed signal across diverse prompts spanning reasoning, creativity, safety, and factual accuracy\(Wanget al\.[2024](https://arxiv.org/html/2607.26094#bib.bib6); Gaoet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib7); Casperet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib8)\)\. This leads to: \(1\)reward sparsity—subtle quality differences receive near\-identical scores; \(2\)overoptimization—the policy exploits reward model blind spots; and \(3\)alignment tax—excessive optimization on a narrow signal degrades general capabilities\(Casperet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib8)\)\.

### Reward Shaping: Promise and Peril

Reward shaping augments the base reward with auxiliary terms to guide learning, with deep theoretical foundations in RL\(Nget al\.[1999](https://arxiv.org/html/2607.26094#bib.bib9)\)\. Potential\-based shaping preserves the optimal policy and can improve learning efficiency\. However, designing shaping functions for LLMs is challenging: manual engineering requires per\-task expertise\(Zouet al\.[2019](https://arxiv.org/html/2607.26094#bib.bib10)\), while learned shaping functions, including self\-adaptive approaches\(Maet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib11)\), must be carefully constrained because non\-potential\-based shaping may alter the induced optimal policy\(Nget al\.[1999](https://arxiv.org/html/2607.26094#bib.bib9)\)\. Self\-rewarding models\(Yuanet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib12)\)and AI\-generated rewards\(Leeet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib14); Xieet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib15)\)partially address this but rely on potentially flawed self\-judgments or expensive inference\.

### Our Contribution

We propose MeRLa, a framework that meta\-learns a reward shaping functionΦ​\(x,y;ϕ\)\\Phi\(x,y;\\phi\)before RLHF training\. The key insight is that experience across auxiliary tasks provides rich signal for learning what constitutes a good reward, without additional human labels\. Our contributions:

- •A meta\-learning formulation for RLHF reward shaping that preserves policy optimality via potential\-based constraints \(§4\.1\)\.
- •A composite meta\-objective balancing task discrimination, entropy regularization, and shaping conservation \(§4\.2\)\.
- •Theoretical analysis of representation drift sensitivity \(§5\.2\) and incentive misalignment from entropy maximization \(§5\.3\)\.
- •Extensive experiments on LLaMA\-3\-8B showing consistent gains over PPO, DPO, GRPO, and DAPO across four benchmarks, plus ablations with enhanced base rewards \(§6\.2, §6\.5\)\.
- •Analysis showing 37% lower reward variance and 41% less training instability while preserving generation diversity \(§7\.3\)\.

## Related Work

### RLHF and Preference Optimization

The standard RLHF pipeline trains a reward model on pairwise preferences, then optimizes the policy via PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2607.26094#bib.bib3); Ouyanget al\.[2022](https://arxiv.org/html/2607.26094#bib.bib1)\)\. Recent alternatives include DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib16)\)\(bypasses the reward model\), GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib4)\)\(group\-relative baselines\), DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.26094#bib.bib5)\)\(decoupled clipping\), RLOO\(Ahmadianet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib17)\)\(REINFORCE with online baseline\), and SimPO\(Menget al\.[2024](https://arxiv.org/html/2607.26094#bib.bib18)\)\(reference\-free reward\)\. All operate on a fixed reward signal; MeRLa complements them by learning to shape this signal\.

### Reward Shaping in RL

Potential\-based reward shaping guarantees policy invariance\(Nget al\.[1999](https://arxiv.org/html/2607.26094#bib.bib9)\)\. Prior work explores meta\-learned shaping\(Zouet al\.[2019](https://arxiv.org/html/2607.26094#bib.bib10)\), self\-adaptive shaping\(Maet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib11)\), Text2Reward\(Xieet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib15)\), and self\-rewarding models\(Yuanet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib12)\)\. MeRLa differs by meta\-learning a continuous shaping function on the LLM input\-output space with formal policy preservation and representation drift analysis\.

### Process and Enhanced Reward Models

Process reward models provide step\-level feedback for reasoning\(Lightmanet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib24); Uesatoet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib25)\); Shepherd\(Wanget al\.[2023](https://arxiv.org/html/2607.26094#bib.bib26)\)trains a critic for error detection\. RewardBench\(Lambertet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib28)\)and WARM\(Raméet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib29)\)improve evaluation and robustness\. MeRLa is complementary: it enhances any base reward signal, including process\-based and rubric\-based rewards \(§6\.5\)\.

### Meta\-Learning for Language Models

Meta\-learning has been applied to in\-context learning\(Minet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib19)\)and few\-shot adaptation, but its use in RLHF reward design remains underexplored\. We formulate reward shaping as a bilevel optimization problem: inner loop performs RLHF on individual tasks, outer loop optimizes the shaping function across tasks\.

### Representation Stability in Fine\-Tuning

Fine\-tuning can distort pretrained representations and degrade OOD performance\(Kumaret al\.[2022](https://arxiv.org/html/2607.26094#bib.bib27)\), which is particularly relevant for RLHF where the policy evolves continuously\. We address this with a frozen encoder for the shaping network and provide theoretical drift bounds \(§5\.2\)\.

## Preliminaries

#### Language Model as Policy\.

We view an autoregressive language model with parametersθ\\thetaas a stochastic policyπθ​\(y∣x\)=Πt=1\|y\|​πθ​\(yt∣x,y<t\)\\pi\_\{\\theta\}\(y\\mid x\)=\\Pi\_\{t=1\}^\{\|y\|\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)wherexxis a prompt andy=\(y1,…,y\|y\|\)y=\(y\_\{1\},\\dots,y\_\{\|y\|\}\)is the generated response\.

#### RLHF Objective\.

Given a base reward modelRbase,ψ​\(x,y\)R\_\{\\text\{base\},\\psi\}\(x,y\)trained on preference dataDprefD\_\{\\text\{pref\}\}, the standard RLHF objective optimizes:

maxθ⁡𝔼x∼D,y∼πθ\(⋅∣x\)​\[Rbase​\(x,y\)−β​KL​\(πθ∥πref\)\]\\max\_\{\\theta\}\\mathbb\{E\}\_\{x\\sim D,y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[R\_\{\\text\{base\}\}\(x,y\)\-\\beta\\,\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\right\]\(1\)whereπref\\pi\_\{\\text\{ref\}\}is the supervised fine\-tuning reference policy andβ\\betacontrols the KL penalty\.

#### Potential\-Based Reward Shaping\.

A shaping functionΦ\\Phiis potential\-based if it satisfies the formΦ​\(s,s′\)=γ​ϕ​\(s′\)−ϕ​\(s\)\\Phi\(s,s^\{\\prime\}\)=\\gamma\\phi\(s^\{\\prime\}\)\-\\phi\(s\)for some potential functionϕ:S→ℝ\\phi:S\\to\\mathbb\{R\}\(Nget al\.[1999](https://arxiv.org/html/2607.26094#bib.bib9)\)\. This guarantees that the optimal policy remains unchanged under the transformed rewardR′=R\+ΦR^\{\\prime\}=R\+\\Phi\.

## Method

MeRLa operates in two phases: a meta\-learning phase that learns a task\-aware reward shaping function, and a deployment phase that uses the learned shaping to augment standard RLHF training\. Figure[1](https://arxiv.org/html/2607.26094#Sx4.F1)provides an overview\.

![Refer to caption](https://arxiv.org/html/2607.26094v1/Figures/1.png)Figure 1:MeRLa Framework\. Meta\-Learning Phase: A task distribution provides diverse prompts for reward shaping optimization\. Deployment Phase: The learned shaping function augments the base reward model, producing a composite signal that guides the RL optimizer\.### Composite Reward Design

We define the shaped reward as:

R^​\(x,y;ϕ,ψ\)=Rbase,ψ​\(x,y\)\+α⋅Φ​\(x,y;ϕ\)\\hat\{R\}\(x,y;\\phi,\\psi\)=R\_\{\\text\{base\},\\psi\}\(x,y\)\+\\alpha\\cdot\\Phi\(x,y;\\phi\)\(2\)whereΦ​\(x,y;ϕ\)\\Phi\(x,y;\\phi\)is the learned shaping function,α\\alphaparameterizes the shaping strength, andα∈\[0,1\]\\alpha\\in\[0,1\]controls the shaping strength\.

#### Shaping Network Architecture\.

The shaping functionΦ\\Phiis implemented as a lightweight neural network that takes the prompt embeddinghxh\_\{x\}and response embeddinghyh\_\{y\}\(from the base LLM’s penultimate layer\) and outputs a scalar:

Φ​\(x,y;ϕ\)=MLPϕ​\(hx⊕hy⊕\(hy−hx\)\)\\Phi\(x,y;\\phi\)=\\text\{MLP\}\_\{\\phi\}\(h\_\{x\}\\oplus h\_\{y\}\\oplus\(h\_\{y\}\-h\_\{x\}\)\)\(3\)where⊕\\oplusdenotes concatenation and the difference term\(hy−hx\)\(h\_\{y\}\-h\_\{x\}\)encodes the response’s divergence from the prompt context\. The network is a 2\-layer MLP with hidden dimension 256 and SiLU activation, adding fewer than1​M1Mparameters\.

#### Frozen Encoder for Representation Stability\.

A critical design decision is thathxh\_\{x\}andhyh\_\{y\}are extracted from a frozen copy of the reference modelπref\\pi\_\{\\text\{ref\}\}, not from the evolving policyπθ\\pi\_\{\\theta\}\. This choice is motivated by three considerations: \(1\) it ensures the shaping function’s input distribution remains stable throughout RLHF training, preventing representation drift\(Kumaret al\.[2022](https://arxiv.org/html/2607.26094#bib.bib27)\); \(2\) it decouples the shaping signal from the shaping network’s meta\-learning from policy updates, allowingϕ\\phito be optimized independently; \(3\) it avoids the computational overhead of re\-extracting embeddings from the evolving policy at each RL step\. We provide a formal analysis of representation drift sensitivity in §5\.2, showing that the frozen encoder reduces the Lipschitz bound of the shaping function by an order of magnitude compared to the live policy encoder\.

#### Policy Invariance Guarantee\.

To ensure the shaped reward does not alter the optimal policy, we constrainΦ\\Phito approximate a potential\-based form\. Specifically, for autoregressive generation, we decompose the response\-level shaping into token\-level potentials:

Φ​\(x,y;ϕ\)=∑t=1\|y\|\[γ⋅φϕ​\(x,y≤t\)−φϕ​\(x,y<t\)\]\\Phi\(x,y;\\phi\)=\\sum\_\{t=1\}^\{\|y\|\}\\left\[\\gamma\\cdot\\varphi\_\{\\phi\}\(x,y\_\{\\leq t\}\)\-\\varphi\_\{\\phi\}\(x,y\_\{<t\}\)\\right\]\(4\)This ensures thatΦ\\Phisatisfies the conditions ofNget al\.\([1999](https://arxiv.org/html/2607.26094#bib.bib9)\), guaranteeing that any policy optimal underRbaseR\_\{\\text\{base\}\}remains optimal underR^\\hat\{R\}\. \(4\)

### Meta\-Learning Objective

Let𝒯=\{τ1,…,τM\}\\mathcal\{T\}=\\\{\\tau\_\{1\},\\dots,\\tau\_\{M\}\\\}denote a distribution ofMMauxiliary tasks\. The meta\-learning objective optimizesϕ\\phi:

minϕ⁡ℒmeta​\(ϕ\)=∑i=1M\[ℒtask\(i\)​\(ϕ\)\+λ1​ℒent​\(ϕ\)\+λ2​ℒcon​\(ϕ\)\]\\min\_\{\\phi\}\\mathcal\{L\}\_\{\\text\{meta\}\}\(\\phi\)=\\sum\_\{i=1\}^\{M\}\\left\[\\mathcal\{L\}\_\{\\text\{task\}\}^\{\(i\)\}\(\\phi\)\+\\lambda\_\{1\}\\mathcal\{L\}\_\{\\text\{ent\}\}\(\\phi\)\+\\lambda\_\{2\}\\mathcal\{L\}\_\{\\text\{con\}\}\(\\phi\)\\right\]\(5\)
#### Task Discrimination Loss\.

The shaping function should produce reward signals that discriminate response quality within each task:

ℒtask\(i\)​\(ϕ\)=−𝔼x∼τi​\[log⁡exp⁡\(R^​\(x,y\+;ϕ\)\)exp⁡\(R^​\(x,y\+;ϕ\)\)\+exp⁡\(R^​\(x,y−;ϕ\)\)\]\\mathcal\{L\}\_\{\\text\{task\}\}^\{\(i\)\}\(\\phi\)=\-\\mathbb\{E\}\_\{x\\sim\\tau\_\{i\}\}\\left\[\\log\\frac\{\\exp\(\\hat\{R\}\(x,y^\{\+\};\\phi\)\)\}\{\\exp\(\\hat\{R\}\(x,y^\{\+\};\\phi\)\)\+\\exp\(\\hat\{R\}\(x,y^\{\-\};\\phi\)\)\}\\right\]\(6\)wherey\+y^\{\+\}andy−y^\{\-\}denote preferred and rejected responses\.

#### Entropy Regularization\.

To prevent the shaping function from collapsing to a trivial constant or degenerating into a duplicate of the base reward, we add:

ℒent​\(ϕ\)=−𝔼x∼𝒯,y∼πref​\[ℋ​\(p​\(R^​\(x,y;ϕ\)\)\)\]\\mathcal\{L\}\_\{\\text\{ent\}\}\(\\phi\)=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{T\},y\\sim\\pi\_\{\\text\{ref\}\}\}\\left\[\\mathcal\{H\}\(p\(\\hat\{R\}\(x,y;\\phi\)\)\)\\right\]\(7\)whereℋ\\mathcal\{H\}denotes the entropy of the shaped reward distribution across the batch\. High entropy encourages the shaping function to spread reward values, improving signal\-to\-noise ratio\.

#### Conservation Loss\.

We regularize the shaping function to be close to potential\-based, ensuring the policy invariance property:

ℒcon​\(ϕ\)=𝔼x,y​\|\(Φ​\(x,y;ϕ\)−Φpb​\(x,y;ϕ\)\)2\|\\mathcal\{L\}\_\{\\text\{con\}\}\(\\phi\)=\\mathbb\{E\}\_\{x,y\}\\left\|\\left\(\\Phi\(x,y;\\phi\)\-\\Phi\_\{\\text\{pb\}\}\(x,y;\\phi\)\\right\)^\{2\}\\right\|\(8\)whereΦpb\\Phi\_\{\\text\{pb\}\}is the projection ofΦ\\Phionto the potential\-based space, computed via least\-squares projection onto the token\-level potential decomposition in Equation \(4\)\.

### Deployment: RLHF with Meta\-Shaped Rewards

After meta\-learning, we freeze the shaping parametersϕ∗\\phi^\{\*\}and deploy the composite reward for standard RLHF training:

maxθ⁡𝔼x∼D,y∼πθ\(⋅∣x\)​\[R^​\(x,y;ϕ∗,ψ\)−β​KL​\(πθ∥πref\)\]\\max\_\{\\theta\}\\mathbb\{E\}\_\{x\\sim D,y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\hat\{R\}\(x,y;\\phi^\{\*\},\\psi\)\-\\beta\\,\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\right\]\(9\)
MeRLa is compatible with any on\-policy RLHF algorithm\. In our experiments, we primarily use GRPO as the backbone optimizer due to its computational efficiency \(the critic model required\), and show that MeRLa also improves PPO and DAPO\.

The full procedure is summarized in Algorithm 1\.

Algorithm 1MeRLa — Meta\-Learned Reward Shaping for RLHF0:Base reward model

Rbase,ψR\_\{\\text\{base\},\\psi\}, reference policy

πref\\pi\_\{\\text\{ref\}\}, task distribution

𝒯\\mathcal\{T\}
1:Phase I: Meta\-Learning

2:forepoch =

1,…,Tmeta1,\\dots,T\_\{\\text\{meta\}\}do

3:foreach task

τi∈𝒯\\tau\_\{i\}\\in\\mathcal\{T\}do

4:Sample prompts

x∼τix\\sim\\tau\_\{i\}, responses

\(y\+,y−\)∼𝒟\(y^\{\+\},y^\{\-\}\)\\sim\\mathcal\{D\}
5:Compute shaped rewards via Eq\. \(2\) with Eq\. \(3\)

6:Accumulate

ℒtaski\+λ1​ℒent\+λ2​ℒcon\\mathcal\{L\}\_\{\\text\{task\}\}^\{i\}\+\\lambda\_\{1\}\\mathcal\{L\}\_\{\\text\{ent\}\}\+\\lambda\_\{2\}\\mathcal\{L\}\_\{\\text\{con\}\}
7:endfor

8:Update

ϕ←ϕ−ηϕ​∇ϕℒmeta\\phi\\leftarrow\\phi\-\\eta\_\{\\phi\}\\nabla\_\{\\phi\}\\mathcal\{L\}\_\{\\text\{meta\}\}via Eq\. \(5\)

9:endfor

10:Freeze shaping parameters

ϕ∗\\phi^\{\*\}
11:Phase II: Deployment

12:forepoch =

1,…,Trl1,\\dots,T\_\{\\text\{rl\}\}do

13:Sample prompts

x∼Dx\\sim D
14:Generate responses

y∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)
15:Compute shaped rewards

R^​\(x,y;ϕ∗,ψ\)\\hat\{R\}\(x,y;\\phi^\{\*\},\\psi\)
16:Update

θ\\thetavia GRPO/PPO on

𝔼D​\[R^\]\\mathbb\{E\}\_\{D\}\[\\hat\{R\}\]
17:endfor

18:returnAligned policy

πθ\\pi\_\{\\theta\}

## Theoretical Analysis

In this section, we provide formal guarantees for MeRLa\. We first establish the policy invariance theorem, then analyze the sensitivity of the shaping function to representation drift, and finally provide bounds on potential incentive misalignment from entropy maximization\.

### Policy Invariance Guarantee

###### Theorem 1\(Policy Invariance\)\.

Let𝐑^​\(x,y\)=Rbase​\(x,y\)\+α⋅Φ​\(x,y;ϕ\)\\mathbf\{\\hat\{R\}\}\(x,y\)=R\_\{\\text\{base\}\}\(x,y\)\+\\alpha\\cdot\\Phi\(x,y;\\phi\)be the shaped reward, whereΦ\\Phisatisfies the potential\-based decomposition in Equation \(4\)\. Then for any discount factorγ∈\(0,1\)\\gamma\\in\(0,1\), the optimal policy under𝐑^\\mathbf\{\\hat\{R\}\}is identical to the optimal policy underRbaseR\_\{\\text\{base\}\}, and the optimal value function transforms as:

VR∗​\(s\)=VR∗​\(s\)\+ϕ​\(s\),ΠR∗=ΠR∗V\_\{R\}^\{\*\}\(s\)=V\_\{R\}^\{\*\}\(s\)\+\\phi\(s\),\\quad\\Pi\_\{R\}^\{\*\}=\\Pi\_\{R\}^\{\*\}\(10\)

Proof sketch\.By the telescoping property of the potential\-based decomposition, the cumulative shaped reward over any trajectoryτ=\(s0,s1,…,sT\)\\tau=\(s\_\{0\},s\_\{1\},\\dots,s\_\{T\}\)reduces to∑tΦ​\(st,st\+1\)\\sum\_\{t\}\\Phi\(s\_\{t\},s\_\{t\+1\}\), which depends only on the initial and terminal states\. Since the optimal policy maximizes the discounted cumulative reward, and this additional term does not depend on the policy, the optimal policy is preserved\.□\\square

In practice, the conservation loss \(Equation 8\) ensures thatΦ\\Phiapproximates the potential\-based form with a residual errorϵ=‖Φ−Φpb‖\\epsilon=\\\|\\Phi\-\\Phi\_\{\\text\{pb\}\}\\\|\. This residual introduces a small bias bounded byα⋅ϵmax⋅‖ϕ∗‖\\alpha\\cdot\\epsilon\_\{\\max\}\\cdot\\\|\\phi^\{\*\}\\\|, which we empirically observe to be beneficial and quantify in §7\.2\.

### Representation Drift Sensitivity

A key design question is whether to compute the prompt and response embeddingshxh\_\{x\}andhyh\_\{y\}using a frozen encoder or from the evolving policy model\. We analyze this formally\.

###### Proposition 1\(Representation Drift Bound\)\.

Leth\(t\)h^\{\(t\)\}denote the embedding from the policy model at training steptt, andh\(0\)h^\{\(0\)\}denote the embedding from the frozen reference model\. Assume the shaping functionΦ\\PhiisLL\-Lipschitz\. Then the change in shaping output due to representation drift is bounded by:

‖Φ​\(hx\(t\),hy\(t\)\)−Φ​\(hx\(0\),hy\(0\)\)‖≤Lϕ⋅‖h\(t\)−h\(0\)‖\\left\\\|\\Phi\(h\_\{x\}^\{\(t\)\},h\_\{y\}^\{\(t\)\}\)\-\\Phi\(h\_\{x\}^\{\(0\)\},h\_\{y\}^\{\(0\)\}\)\\right\\\|\\leq L\_\{\\phi\}\\cdot\\left\\\|h^\{\(t\)\}\-h^\{\(0\)\}\\right\\\|\(11\)

Using the frozen encoder,h\(t\)−h\(0\)=0h^\{\(t\)\}\-h^\{\(0\)\}=0for the same input, so that‖h\(t\)−h\(0\)‖=0\\\|h^\{\(t\)\}\-h^\{\(0\)\}\\\|=0and the bound is identically zero\. Using the evolving policy encoder, parameter updates may produce nonzero representation drifts, i\.e\.,‖h\(t\)−h\(0\)‖\>0\\\|h^\{\(t\)\}\-h^\{\(0\)\}\\\|\>0\. Prior work has shown that fine\-tuning can distort pretrained represenattions\(Kumaret al\.[2022](https://arxiv.org/html/2607.26094#bib.bib27)\), while SGD\-induced representation drift can exhibit diffusion\-like dynamics with an0​\(t\)0\(\\sqrt\{t\}\)root\-mean\-square displacement in the diffusive regime\(Pashakhanloo and Koulakov[2023](https://arxiv.org/html/2607.26094#bib.bib36)\)\. To further control the Lipschitz constant, we apply spectral normalization to the shaping MLP, enforcingL≤LmaxL\\leq L\_\{\\max\}\. We also add an auxiliary Lipschitz regularization term:

ℒdrift=𝔼​\[‖∇ϕΦ‖F2\],Lϕ≤Lmax\\mathcal\{L\}\_\{\\text\{drift\}\}=\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\phi\}\\Phi\\right\\\|\_\{F\}^\{2\}\\right\],\\quad L\_\{\\phi\}\\leq L\_\{\\max\}\(12\)
#### Empirical Validation\.

With the frozen encoder, the shaping output variance across 1,000 training steps is 0\.002 \(negligible\)\. With the evolving encoder, the variance is 0\.087 — a43×43\\timesincrease\. The final AlpacaEval 2\.0 win rate drops from 90\.8% \(frozen\) to 87\.1% \(evolving\), confirming that the frozen encoder is essential for stable shaping\. The frozen encoder adds no additional memory cost, as the reference model is already maintained for KL penalty computation in standard RLHF\.

### Entropy and Incentive Alignment

A natural concern is whether maximizing the entropy of the shaped reward distribution could introduce incentive misalignment—could the entropy term encourage the shaping function to produce reward signals that diverge from the true quality ranking? We address this formally\.

###### Proposition 2\(Incentive Alignment Bound\)\.

LetΔincentive\\Delta\_\{\\text\{incentive\}\}denote the maximum change in the optimal policy’s action ranking due to the shaping function\. Then:

Δincentive≤α⋅\(ℒcon​\(ϕ\)\+ϵpb\)\\Delta\_\{\\text\{incentive\}\}\\leq\\alpha\\cdot\\left\(\\mathcal\{L\}\_\{\\text\{con\}\}\(\\phi\)\+\\epsilon\_\{\\text\{pb\}\}\\right\)\(13\)whereϵpb=‖Φ−Φ∗‖\\epsilon\_\{\\text\{pb\}\}=\\\|\\Phi\-\\Phi^\{\*\}\\\|is the residual conservation loss\. \(13\)

This bound shows that incentive misalignment is controlled by two factors: \(1\) the shaping strengthα\\alpha, and \(2\) the deviation from the potential\-based form\. Since the entropy lossℒent\\mathcal\{L\}\_\{\\text\{ent\}\}is optimized jointly withℒcon\\mathcal\{L\}\_\{\\text\{con\}\}, the conservation loss acts as a regularizer that prevents entropy maximization from distorting the reward distribution\.

#### Key insight\.

The entropy term encourages rewarddistributional richness\(spreading reward values across the batch\) rather than rewardmagnitude inflation\. Because the entropy is computed over the distributionp​\(𝐑^​\(x,y;ϕ\)\)p\(\\hat\{\\mathbf\{R\}\}\(x,y;\\phi\)\)within a batch, it cannot increase the reward for any single response without correspondingly adjusting others\. The conservation loss then ensures these adjustments remain approximately potential\-based, preserving the optimal policy ranking\.

#### Safeguards against degenerate solutions\.

Three mechanisms prevent incentive misalignment: \(1\) the conservation lossℒcon\\mathcal\{L\}\_\{\\text\{con\}\}constrainsΦ\\Phito the potential\-based subspace; \(2\) the KL penaltyβ​KL​\(πθ∥πref\)\\beta\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)provides a secondary check against reward hacking; \(3\) the task discrimination lossℒtask\\mathcal\{L\}\_\{\\text\{task\}\}ensures the shaped reward maintains the task quality ordering of the base reward\. In our experiments \(§7\.3\), we verify that MeRLa\-trained policies do not exhibit reward hacking: n\-gram diversity increases by 23%, and GPT\-4 judge scores improve across all dimensions including honesty and safety\.

## Experiments

### Experimental Setup

#### Base Model\.

We use LLaMA\-3\-8B\-Instruct\(Grattafioriet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib20)\)as the base policy\. The reward model is trained on UltraFeedback\(Cuiet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib13)\)followingOuyanget al\.\([2022](https://arxiv.org/html/2607.26094#bib.bib1)\)with LoRA\(Huet al\.[2022](https://arxiv.org/html/2607.26094#bib.bib21)\)\(rank 16\)\.

#### Baselines\.

We compare against: \(1\) SFT only, \(2\) PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2607.26094#bib.bib3); Ouyanget al\.[2022](https://arxiv.org/html/2607.26094#bib.bib1)\), \(3\) DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib16)\), \(4\) GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.26094#bib.bib4)\), and \(5\) DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.26094#bib.bib5)\)\. All share the same base model and preference data\. Methods requiring an explicit reward model use the same reward\-model checkpoint

#### Evaluation Benchmarks\.

We evaluate on: \(1\) AlpacaEval 2\.0\(Duboiset al\.[2024](https://arxiv.org/html/2607.26094#bib.bib22)\)for instruction\-following; \(2\) MT\-Bench\(Zhenget al\.[2023](https://arxiv.org/html/2607.26094#bib.bib23)\)for multi\-turn dialogue; \(3\) MATH\(Hendryckset al\.[2021](https://arxiv.org/html/2607.26094#bib.bib34)\)for mathematical reasoning; \(4\) IFEval\(Zhouet al\.[2023](https://arxiv.org/html/2607.26094#bib.bib31)\)for verifiable instruction\-following accuracy\.

#### AlpacaEval 2\.0 Protocol\.

We report length\-controlled \(LC\) win rates using GPT\-4\-Turbo as judge with the standard greedy template \(temperature = 0\)\. Responses are generated via greedy decoding \(max\_tokens = 2048\)\. Results are averaged over 3 runs with different seeds\.

#### Implementation Details\.

The shaping network is a 2\-layer MLP \(hidden dim 256, SiLU, spectral normalization\)\. We setM=64M=64meta\-tasks,α=0\.3\\alpha=0\.3,λ1=0\.05\\lambda\_\{1\}=0\.05,λ2=0\.1\\lambda\_\{2\}=0\.1\. RLHF uses GRPO \(group size 8,β=0\.004\\beta=0\.004, lr10−610^\{\-6\}, 20 epochs\) on 8×\\timesA100\-80GB GPUs\.

### Main Results

Table[1](https://arxiv.org/html/2607.26094#Sx6.T1)summarizes the main results\. MeRLa \(GRPO backbone\) achieves 90\.8% LC win rate on AlpacaEval 2\.0, outperforming DAPO by 3\.9 points\. Gains are consistent across all benchmarks: \+0\.33 on MT\-Bench, \+5\.6% on MATH, and \+3\.9% on IFEval, all statistically significant \(p < 0\.05, paired t\-test\)\.

Table 1:Main results on LLaMA\-3\-8B\.†\\daggerdenotes our reimplementation\. All numbers are mean over 3 runs\. AlpacaEval 2\.0 reports LC win rate: 95% CIs within±\\pm1\.2%\.Figure[2](https://arxiv.org/html/2607.26094#Sx6.F2)\(a\) compares model performance on benchmarks, while \(b\) plots multi\-dimensional scores in radar form\. MeRLa outperforms all baselines across six dimensions, with clear advantages in reasoning \(\+5% vs DAPO\) and safety \(\+9% vs DAPO\)\. This indicates our meta\-learned shaping function delivers superior supervision on weak points of conventional reward models\.

![Refer to caption](https://arxiv.org/html/2607.26094v1/Figures/2.png)Figure 2:\(a\) Bar chart comparing methods across four benchmarks\. \(b\) Radar chart showing per\-dimension scores from GPT\-4 evaluation\. MeRLa \(red\) dominates all dimensions\.
### Ablation Study

#### Scaling with Meta\-Tasks\.

Figure[3](https://arxiv.org/html/2607.26094#Sx6.F3)\(a\) shows how performance scales with the number of meta\-tasksMM\. MeRLa consistently outperforms DAPO across all values ofMM\. The gap widens asMMincreases, plateauing aroundM=64M=64\. This suggests that diversity in the meta\-task distribution is crucial—more tasks provide richer learning signals\.

#### Component Analysis\.

Figure[3](https://arxiv.org/html/2607.26094#Sx6.F3)\(b\) ablates each component of the meta\-objective \(Eq\. 5\)\. Removingℒtask\\mathcal\{L\}\_\{\\text\{task\}\}causes the largest drop \(∼\\sim3\.7%\), confirming its central role\. Removingℒent\\mathcal\{L\}\_\{\\text\{ent\}\}causes the second most important drop \(∼\\sim2\.6%\), validating our theoretical motivation\. The task sampler removal \(∼\\sim5\.5%\) demonstrates that intelligent task selection matters more than simply using more data\. Removingℒcon\\mathcal\{L\}\_\{\\text\{con\}\}causes a moderate drop \(∼\\sim3\.2%\), consistent with its role as a diversity regularizer\.

![Refer to caption](https://arxiv.org/html/2607.26094v1/Figures/3.png)Figure 3:\(a\) Scaling behavior: MeRLa consistently outperforms the non\-meta baseline as M increases, with diminishing returns beyond M = 64\. \(b\) Component ablation: removing any component degrades performance, with the task number being the most critical\.

### Compatibility Analysis

Table[2](https://arxiv.org/html/2607.26094#Sx6.T2)shows MeRLa is compatible with different RLHF backbones\. AlpacaEval 2\.0 LC win rate\(%\)\. Measured over 3 runs\.

Table 2:The improvement is largest with PPO \(\+6\.6%\) and GRPO \(\+6\.5%\), which lack built\-in reward signal enhancement\. With DAPO, the improvement is smaller \(\+3\.9%\) because DAPO’s dynamic sampling already partially addresses reward sparsity\. Notably, MeRLa combined with PPO outperforms standalone GRPO \(85\.2% vs\. 84\.3%\), suggesting that reward shaping can partially substitute for algorithmic complexity\.
### MeRLa with Enhanced Base Rewards

A natural question is whether MeRLa retains its advantages when the base reward is already enhanced through process\-based or rubric\-based rewards\. We investigate this with two enhanced variants:

#### Process Reward Model \(PRM\)\.

FollowingLightmanet al\.\([2023](https://arxiv.org/html/2607.26094#bib.bib24)\); Uesatoet al\.\([2022](https://arxiv.org/html/2607.26094#bib.bib25)\), we train a PRM that provides step\-level rewards for mathematical reasoning\. The final reward is the sum of step\-level rewards\.

#### Rubric\-Based Ensemble Rewards\.

We construct an ensemble of 5 reward models, each trained on a different quality dimension \(helpfulness, harmlessness, honesty, reasoning, fluency\)\. The final reward is weighted sum of the 5 dimension scores\.

Table[3](https://arxiv.org/html/2607.26094#Sx6.T3)reveals that MeRLa consistently improves upon all three base reward types\. The marginal benefit varies: \+6\.5% with the standard RM, \+4\.2% with the PRM, and \+3\.5% with the rubric ensemble\. This decreasing trend is expected—enhanced base rewards already provide richer reward signals, leaving less room for improvement\. However, improvements remain substantial and statistically significant in all cases\. This demonstrates that MeRLa is complementary to reward enhancement techniques: the meta\-learned shaping function captures task\-aware patterns that even process\-level and rubric\-based rewards miss\. The marginal benefit satisfies:

ΔMeRLa=Perf​\(Base\+Φ\)−Perf​\(Base\)≥0∀Base\\Delta\_\{\\text\{MeRLa\}\}=\\text\{Perf\}\(\\text\{Base\}\+\\Phi\)\-\\text\{Perf\}\(\\text\{Base\}\)\\geq 0\\quad\\forall\\,\\text\{Base\}\(14\)
This non\-negativity is guaranteed by the potential\-based constraint: since the shaping function preserves the optimal policy \(Theorem 1\), it cannot degrade performance in expectation\. The enhanced composite reward is:

R^enhanced​\(x,y\)=Rprocess​\(x,y\)\+α⋅Φ​\(x,y;ϕ∗\)\\hat\{R\}\_\{\\text\{enhanced\}\}\(x,y\)=R\_\{\\text\{process\}\}\(x,y\)\+\\alpha\\cdot\\Phi\(x,y;\\phi^\{\*\}\)\(15\)
Table 3:MeRLa with enhanced base rewards on LLaMA\-3\-8B\. AlpacaEval 2\.0 LC win rate \(%\) and MATH accuracy \(%\)\. Measured over 3 runs\.

## Analysis

### Reward Signal Quality

Figure[4](https://arxiv.org/html/2607.26094#Sx7.F4)\(a\) compares the distribution of reward values\. The base reward model produces a wide, bimodal distribution with high variance \(σ=0\.80\\sigma=0\.80\)\. Hand\-crafted shaping narrows this but introduces bias toward certain response styles\. MeRLa achieves the most desirable distribution: higher mean \(μ=1\.1\\mu=1\.1vs\. 0\.8\), lower variance \(σ=0\.35\\sigma=0\.35\), and better separation between high and low\-quality responses\.

Figure[4](https://arxiv.org/html/2607.26094#Sx7.F4)\(b\) shows how reward scores evolve across quality dimensions during training\. With MeRLa shaping, all five dimensions \(helpfulness, harmlessness, honesty, reasoning, fluency\) improve steadily and converge by epoch 15\. In contrast, the base reward model shows uneven progress, with reasoning and honesty lagging significantly\.

![Refer to caption](https://arxiv.org/html/2607.26094v1/Figures/4.png)Figure 4:\(a\) Distribution of reward values for the base reward model, hand\-crafted shaping, and MeRLa shaping on 2,000 held\-out pairs\. MeRLa produces a more concentrated distribution with higher mean\. \(b\) Heatmap showing RLHF training scores across five quality dimensions during RLHF training\.Figure[5](https://arxiv.org/html/2607.26094#Sx7.F5)\(a\) plots the reward during training\. MeRLa reaches 90% of its final reward within the first 150 steps, compared to 250 steps for DAPO and 350\+ steps for PPO\. The reward variance \(shaded region\) is 41% lower with MeRLa, indicating more stable training\. The reward landscape projected via t\-SNE \(Figure[5](https://arxiv.org/html/2607.26094#Sx7.F5)\(b\)\) shows that the meta\-learned shaping function provides a smoother, more informative gradient landscape\.

Figure[5](https://arxiv.org/html/2607.26094#Sx7.F5)\(b\) visualizes the reward landscape in a 2D projection\. The base reward model exhibits multiple local maxima \(blue contours\), which can trap the policy in suboptimal regions\. MeRLa shaping \(red dashed contours\) smooths the landscape, reducing the prominence of local maxima and creating a clearer path to the global optimumθ∗\\theta^\{\*\}\.

![Refer to caption](https://arxiv.org/html/2607.26094v1/Figures/5.png)Figure 5:\(a\) Average reward during RLHF training\. MeRLa converges faster and to a higher final reward, with noticeably lower variance \(shaded region\)\. \(b\) Visualization of the 2D reward landscape projected via t\-SNE\. MeRLa shaping produces a smoother landscape with a clear global optimum\.
### Training Dynamics and Stability

#### Conservation Loss and Residual Bias\.

We track the conservation lossℒcon\\mathcal\{L\}\_\{\\text\{con\}\}throughout training\. At the end of meta\-learning,ℒcon=0\.0034\\mathcal\{L\}\_\{\\text\{con\}\}=0\.0034, indicating thatΦ\\Phiclosely approximates the potential\-based form\. The resulting residual baseϵcon=0\.003\\epsilon\_\{\\text\{con\}\}=0\.003translates to a maximum policy ranking change ofα⋅ϵmax=0\.001\\alpha\\cdot\\epsilon\_\{\\max\}=0\.001, which is negligible\. This empirically validates the incentive alignment bound \(Proposition 2\) and confirms that the entropy regularization does not cause incentive misalignment in practice\.

### Why Does Meta\-Learning Help?

We identify three mechanisms through which meta\-learning improves reward shaping:

1. 1\.Task\-aware signal enrichment\.Training across diverse tasks teaches the shaping function to emphasize category\-specific quality features—logical consistency for math prompts, coherence for creative writing\.
2. 2\.Implicit reward calibration\.The task discrimination loss \(Eq\. 6\) acts as self\-calibration, amplifying reward differences where the base model is uncertain and staying conservative where it is reliable, analogous to an adaptive learning rate for the reward signal\.
3. 3\.Exploration regularization\.Entropy regularization \(Eq\. 7\) prevents reward landscape collapse, maintaining exploration pressure\. The conservation loss constrains shaping to the potential\-based subspace, avoiding incentive misalignment \(§5\.3\)\. MeRLa policies explore 23% more unique templates \(n\-gram diversity\) without reward hacking, with GPT\-4 scores improving across all dimensions \(honesty \+4%, safety \+9%\)\.

## Limitations and Future Work

Several limitations merit discussion\.First, the meta\-learning phase adds approximately 2 GPU\-hours of overhead on 8×\\timesA100 forM=64M=64tasks, though this cost is amortized across multiple RLHF runs\.Second, the policy invariance guarantee holds exactly only when the conservation loss is zero; the residual bias appears beneficial in practice but warrants formal analysis\.Third, the shaping network operates on fixed\-length embeddings; extending to variable\-length, token\-level shaping is a promising direction\.Fourth, our incentive alignment guarantees assume the base reward model is not adversarially corrupted\.Fifth, experiments use a single base model \(LLaMA\-3\-8B\); validation on larger models would strengthen generality claims\.

## Conclusion

We introduced MeRLa, a framework that meta\-learns reward shaping functions for RLHF\. By optimizing a composite meta\-objective across auxiliary task distributions, MeRLa enriches reward signals, accelerates convergence, and improves alignment—all while preserving policy optimality via potential\-based constraints\. We provided theoretical guarantees for policy invariance, analyzed representation drift sensitivity , and formally addressed incentive misalignment from entropy maximization\. Experiments on LLaMA\-3\-8B show consistent improvements over PPO, DPO, GRPO, and DAPO across four benchmarks \(90\.8% LC win rate on AlpacaEval 2\.0, 9\.14 on MT\-Bench\)\. Additional experiments with process\-based and rubric\-based rewards confirm complementary benefits\. MeRLa is compatible with any on\-policy RLHF algorithm and adds minimal parameters\.

## References

- A\. Ahmadian, C\. Cremer, M\. Gallé,et al\.\(2024\)Back to basics: revisiting reinforce style optimization for learning from human feedback in llms\.Note:arXiv:2402\.14740Cited by:[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.Note:arXiv:2204\.05862Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1)\.
- S\. Casper, X\. Davies, C\. Shi,et al\.\(2023\)Open problems and fundamental limitations of reinforcement learning from human feedback\.Note:arXiv:2307\.15217Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p2.1)\.
- G\. Cui, L\. Yuan, N\. Ding,et al\.\(2023\)UltraFeedback: boosting language models with scaled ai feedback\.Note:arXiv:2310\.01377Cited by:[Base Model\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px1.p1.1)\.
- Y\. Dubois, B\. Galambosi, P\. Liang,et al\.\(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.Note:arXiv:2404\.04475Cited by:[Evaluation Benchmarks\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px3.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2022\)Scaling laws for reward model overoptimization\.Note:arXiv:2210\.10760Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.26094#Sx1.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The llama 3 herd of models\.Note:arXiv:2407\.21783Cited by:[Base Model\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath,et al\.\(2021\)Measuring mathematical problem solving with the math dataset\.Note:arXiv:2103\.03874Cited by:[Evaluation Benchmarks\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis,et al\.\(2022\)LoRA: low\-rank adaptation of large language models\.Note:arXiv:2106\.09685Cited by:[Base Model\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px1.p1.1)\.
- A\. Kumar, A\. Raghunathan, R\. Jones, T\. Ma, and P\. Liang \(2022\)Fine\-tuning can distort pretrained features and underperform ood\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Representation Stability in Fine\-Tuning](https://arxiv.org/html/2607.26094#Sx2.SSx5.p1.1),[Frozen Encoder for Representation Stability\.](https://arxiv.org/html/2607.26094#Sx4.SSx1.SSS0.Px2.p1.5),[Representation Drift Sensitivity](https://arxiv.org/html/2607.26094#Sx5.SSx2.p2.5)\.
- N\. Lambert, V\. Pyatkin, J\. Morrison,et al\.\(2024\)RewardBench: evaluating reward models for language modeling\.Note:arXiv:2403\.13787Cited by:[Process and Enhanced Reward Models](https://arxiv.org/html/2607.26094#Sx2.SSx3.p1.1)\.
- H\. Lee, S\. Phatale, H\. Mansoor,et al\.\(2024\)RLAIF vs\. rlhf: scaling reinforcement learning from human feedback with ai feedback\.Note:arXiv:2309\.00267Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda,et al\.\(2023\)Let’s verify step by step\.Note:arXiv:2305\.20050Cited by:[Process and Enhanced Reward Models](https://arxiv.org/html/2607.26094#Sx2.SSx3.p1.1),[Process Reward Model \(PRM\)\.](https://arxiv.org/html/2607.26094#Sx6.SSx5.SSS0.Px1.p1.1)\.
- H\. Ma, Z\. Luo, T\. V\. Vo,et al\.\(2024\)Highly efficient self\-adaptive reward shaping for reinforcement learning\.Note:arXiv:2408\.03029Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1),[Reward Shaping in RL](https://arxiv.org/html/2607.26094#Sx2.SSx2.p1.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)SimPO: simple preference optimization with a reference\-free reward\.Note:arXiv:2405\.14734Cited by:[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1)\.
- S\. Min, M\. Lewis, L\. Zettlemoyer, and H\. Hajishirzi \(2022\)MetaICL: learning to learn in context\.InNorth American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 2791–2809\.Cited by:[Meta\-Learning for Language Models](https://arxiv.org/html/2607.26094#Sx2.SSx4.p1.1)\.
- A\. Y\. Ng, D\. Harada, and S\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InInternational Conference on Machine Learning \(ICML\),pp\. 278–287\.Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1),[Reward Shaping in RL](https://arxiv.org/html/2607.26094#Sx2.SSx2.p1.1),[Potential\-Based Reward Shaping\.](https://arxiv.org/html/2607.26094#Sx3.SSx5.SSS0.Px3.p1.4),[Policy Invariance Guarantee\.](https://arxiv.org/html/2607.26094#Sx4.SSx1.SSS0.Px3.p1.4)\.
- L\. Ouyang, J\. Wu, X\. Jiang,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems35,pp\. 27730–27744\.Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1),[Base Model\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px1.p1.1),[Baselines\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px2.p1.1)\.
- F\. Pashakhanloo and A\. Koulakov \(2023\)Stochastic gradient descent\-induced drift of representation in a two\-layer neural network\.InProceedings of the 40th International Conference on Machine Learning,Vol\.202,pp\. 27401–27419\.Cited by:[Representation Drift Sensitivity](https://arxiv.org/html/2607.26094#Sx5.SSx2.p2.5)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell,et al\.\(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1),[Baselines\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px2.p1.1)\.
- A\. Ramé, G\. Couairon, C\. Dancette,et al\.\(2024\)WARM: on the benefits of weight averaged reward models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Process and Enhanced Reward Models](https://arxiv.org/html/2607.26094#Sx2.SSx3.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal,et al\.\(2017\)Proximal policy optimization algorithms\.Note:arXiv:1707\.06347Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1),[Baselines\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Note:arXiv:2402\.03300Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1),[Baselines\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px2.p1.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar,et al\.\(2022\)Solving math word problems with process\- and outcome\-based feedback\.Note:arXiv:2211\.14275Cited by:[Process and Enhanced Reward Models](https://arxiv.org/html/2607.26094#Sx2.SSx3.p1.1),[Process Reward Model \(PRM\)\.](https://arxiv.org/html/2607.26094#Sx6.SSx5.SSS0.Px1.p1.1)\.
- B\. Wang, R\. Zheng, L\. Chen,et al\.\(2024\)Secrets of rlhf in large language models part ii: reward modeling\.Note:arXiv:2401\.06080Cited by:[Introduction](https://arxiv.org/html/2607.26094#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.26094#Sx1.p2.1)\.
- T\. Wang, P\. Yu, X\. E\. Tan,et al\.\(2023\)Shepherd: a critic for language model generation\.Note:arXiv:2308\.04592Cited by:[Process and Enhanced Reward Models](https://arxiv.org/html/2607.26094#Sx2.SSx3.p1.1)\.
- T\. Xie, S\. Zhao, C\. H\. Wu,et al\.\(2023\)Text2Reward: reward shaping with language models for reinforcement learning\.Note:arXiv:2309\.11489Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1),[Reward Shaping in RL](https://arxiv.org/html/2607.26094#Sx2.SSx2.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu,et al\.\(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.Note:arXiv:2503\.14476Cited by:[RLHF and Preference Optimization](https://arxiv.org/html/2607.26094#Sx2.SSx1.p1.1),[Baselines\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px2.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho,et al\.\(2024\)Self\-rewarding language models\.Note:arXiv:2401\.10020Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1),[Reward Shaping in RL](https://arxiv.org/html/2607.26094#Sx2.SSx2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36\.Cited by:[Evaluation Benchmarks\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px3.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra,et al\.\(2023\)Instruction\-following evaluation for large language models\.Note:arXiv:2311\.07911Cited by:[Evaluation Benchmarks\.](https://arxiv.org/html/2607.26094#Sx6.SSx1.SSS0.Px3.p1.1)\.
- H\. Zou, T\. Ren, D\. Yan,et al\.\(2019\)Reward shaping via meta\-learning\.Note:arXiv:1901\.09330Cited by:[Reward Shaping: Promise and Peril](https://arxiv.org/html/2607.26094#Sx1.SSx1.p1.1),[Reward Shaping in RL](https://arxiv.org/html/2607.26094#Sx2.SSx2.p1.1)\.