Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines

# 1 Introduction
Source: [https://arxiv.org/html/2605.26442](https://arxiv.org/html/2605.26442)
Alignment Tuning for Large Language Models: A Data\-Centric Lens on Alignment Data PipelinesHwanjun SongKAISTMuch of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly\. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem\. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy\. Through this lens, we identify recurring design trade\-offs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal\. Finally, we outline open challenges for alignment data pipelines, including prompt\-level alignment, agentic settings, and alignment under evolving objectives\.Date:April 6, 2026Correspondence:Hwanjun Song at[songhwanjun@kaist\.ac\.kr](https://arxiv.org/html/2605.26442v1/mailto:[email protected])Publication:Will appear at the Findings of ACL‘26\.![[Uncaptioned image]](https://arxiv.org/html/2605.26442v1/figures/dislab-icon-final.png)

The progress of large language models \(LLMs\) has been driven by scaling laws, enabled by increased model parameters\(Kaplanet al\.,[2020](https://arxiv.org/html/2605.26442#bib.bib1)\), architectural innovations\(Feduset al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib2); Wuet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib3)\), and advances in optimization\(Yuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib5)\)\. As the marginal returns of scaling plateau, performance gains have shifted toward data\-centric factors, with data quality emerging as a key driver\(Chunget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib7); Zhuanget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib6); Nazaret al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib8)\)\. Despite this shift, most prior work views data quality through*static corpora*, focusing on dataset composition and filtering for pre\-training or supervised fine\-tuning \(SFT\)\(Brownet al\.,[2020](https://arxiv.org/html/2605.26442#bib.bib10); Liuet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib18);[2025c](https://arxiv.org/html/2605.26442#bib.bib17)\)\.

However, this static perspective is insufficient to explain the safety, robustness, and preference adherence of modern LLMs\. These properties are primarily shaped during*alignment tuning*, a post\-training phase distinct from pre\-training\(Jiet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib44); Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4); Baiet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib16)\)\. Unlike supervision from fixed distributions, alignment data is inherently*dynamic*and*policy\-dependent*, generated through repeated interactions among prompts, model outputs, and feedback signals\(Liet al\.,[2025d](https://arxiv.org/html/2605.26442#bib.bib9); Yuet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib22); Liuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib23)\)\. As a result, alignment quality is governed less by static data artifacts and more by the mechanisms that iteratively construct and evaluate them\.

We therefore conceptualize alignment tuning not as a dataset curation task, but as a*pipeline design*problem\. Data quality in alignment tuning depends not only on which samples are retained, but on how candidate behaviors are generated, evaluated, and structured into learning objectives\. We introduce a unifying framework with three interacting dimensions: response synthesis, preference evaluation, and preference instantiation\. Figure[1](https://arxiv.org/html/2605.26442#S1.F1)summarizes this, highlighting how tightly coupled stages jointly construct the optimization signal\.

*\(1\) Response Synthesis:*This stage defines the behavioral support of alignment by determining how candidate responses are generated\. Key design choices include the response source \(offline distillation versus online sampling\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4); Zhanget al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib14); Yuet al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib15)\), selection strategies that prioritize informative candidates, and exploration mechanisms that preserve diversity and avoid premature mode collapse\(Wuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib13); Lanchantinet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib12)\)\.

*\(2\) Preference Evaluation:*Given synthesized responses, alignment depends on the fidelity of preference signals\. This dimension spans evaluator type, from human annotation to scalable LLM\-as\-a\-Judge frameworks\(Leeet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib25); Yuet al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib15)\), as well as judgment granularity and objective dimensionality, which determine preference fidelity and the risk of reward hacking or alignment tax under scalarized and coarse supervision\(Liet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib31); Mukherjeeet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib29)\)\.

*\(3\) Preference Instantiation:*Finally, preference instantiation determines how evaluative judgments are exposed to optimization\. This includes point\-wise rewards\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib26); Yuanet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib30)\), pair\-wise contrasts\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4); Menget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib32)\), and group\- or list\-wise formulations\(Rameshet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib35); Liuet al\.,[2025g](https://arxiv.org/html/2605.26442#bib.bib33)\), which differ in how effectively preference structure is translated into policy updates\.

From this perspective, we organize prior work into a unified data\-centric taxonomy \(Sections[4](https://arxiv.org/html/2605.26442#S4)–[6](https://arxiv.org/html/2605.26442#S6)\) and distill a set of design principles that characterize recurring trade\-offs and cross\-stage interactions across data pipeline stages \(Section[7](https://arxiv.org/html/2605.26442#S7)\)\.

Related Surveys\.Existing surveys on data\-centric LLM training primarily emphasize static data stages, such as data selection for pre\-training and SFT\(Albalaket al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib37); Wanget al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib38)\), dataset catalogs\(Liuet al\.,[2025h](https://arxiv.org/html/2605.26442#bib.bib40)\), or general training paradigms\(Minaeeet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib39)\), as well as system\-level considerations\(Xuet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib36); Zhouet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib41)\)\. In contrast, we focus on alignment tuning as a dynamic, closed\-loop pipeline, examining how response synthesis, evaluation, and instantiation jointly shape alignment outcomes\.

![Refer to caption](https://arxiv.org/html/2605.26442v1/x1.png)Figure 1:Overview of the alignment data pipeline, showing how prompts are converted into structured optimization signals through response synthesis, preference evaluation, and preference instantiation for policy optimization\.Our Scope\.Section[2\.2](https://arxiv.org/html/2605.26442#S2.SS2)provides a brief overview of alignment algorithms, but the majority of our analysis centers on the alignment data pipeline\. Detailed discussions of optimization algorithms are deferred to existing surveys on alignment techniques\(Xiaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib43)\), direct preference optimization\(Liuet al\.,[2025e](https://arxiv.org/html/2605.26442#bib.bib46)\), unified loss design\(Tanget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib127)\), and fundamental limitations such as reward hacking\(Casperet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib45)\)\.

## 2Alignment Tuning Foundation

The central challenge in developing LLMs lies in the mismatch between their training objective and human preferences\(Stiennonet al\.,[2020](https://arxiv.org/html/2605.26442#bib.bib28); Ouyanget al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib24); Baiet al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib51)\)\. Standard next\-token prediction maximizes data likelihood, which is largely orthogonal to desiderata such as helpfulness, honesty, and safety\. As a result, models trained only via pre\-training and supervised fine\-tuning may exhibit factual errors or harmful behaviors despite high likelihood performance\. Alignment tuning addresses this gap by explicitly optimizing models toward human\-valued behaviors\.

### 2\.1Problem Formulation

Letxxdenote a prompt sampled from a task distributionPP, andyybe a response generated by a policyπθ\\pi\_\{\\theta\}\. We assume an oracle reward functionr∗​\(x,y\)r^\{\*\}\(x,y\)reflecting human preferences\. Alignment tuning seeks an optimal policyπ∗\\pi^\{\*\}that maximizes expected reward while remaining close to a reference policyπref\\pi\_\{\\text\{ref\}\}, preventing reward hacking and uncontrolled drift\(Jiet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib44); Yehet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib42)\):

maxπθ⁡𝔼x,y​\[r∗​\(x,y\)−β​log⁡πθ​\(y\|x\)πref​\(y\|x\)\],\\max\_\{\\pi\_\{\\theta\}\}\\mathbb\{E\}\_\{x,y\}\\left\[r^\{\*\}\(x,y\)\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\\right\],\(1\)whereβ\\betais the regularization coefficient controlling the deviation from the reference model\.

### 2\.2Optimization Algorithms

Various approaches have been proposed to solve Eq\. \([1](https://arxiv.org/html/2605.26442#S2.E1)\)\. We review three foundational methods that represent the evolution of alignment tuning\. See Appendix[A](https://arxiv.org/html/2605.26442#A1)for details and additional algorithms\.

*Proximal Policy Optimization \(PPO\)*is the method for explicit reward maximization\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26442#bib.bib48)\)\. It operates in a two\-stage process\. First, a reward modelrϕ​\(x,y\)r\_\{\\phi\}\(x,y\)is trained to approximate oracle preferencesr∗​\(x,y\)r^\{\*\}\(x,y\)via pairwise comparison data\. Second, the policyπθ\\pi\_\{\\theta\}is optimized using reinforcement learning to maximize the learned reward\. PPO employs a clipped surrogate objective to ensure stable updates, restricting the policy update step size to prevent catastrophic forgetting of the reference distribution\.

*Direct Preference Optimization \(DPO\)*is the method that bypasses the explicit reward modeling stage by deriving the optimal policy for the KL\-regularized objective in Eq\. equation[1](https://arxiv.org/html/2605.26442#S2.E1)\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4)\)\. This derivation shows that the reward can be implicitly captured through the log\-likelihood ratio between the policy and a fixed reference model\. As a result, the optimization reduces to a classification\-style objective on paired responses\(yw,yl\)\(y\_\{w\},y\_\{l\}\), directly increasing the likelihood of the preferred outputywy\_\{w\}relative to the dispreferred oneyly\_\{l\}, without requiring a separately trained reward model\.

*Group Relative Policy Optimization \(GRPO\)*shifts from pairwise comparisons to group\-wise optimization\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib52)\)\. Instead of a learned critic, it estimates the baseline for a response by computing the mean reward of a group of outputs sampled from the current policyπθ\\pi\_\{\\theta\}for the same prompt\. The policy model is updated to increase the probability of responses that perform better than the group average\. It reduces the variance associated with external reward models and eliminates the need for a separate value network\.

## 3Understanding Alignment Tuning from a Data\-Centric Perspective

While optimization algorithms update the policy modelπθ\\pi\_\{\\theta\}, they do not by themselves determine the direction or quality of alignment\. Instead, from a data\-centric perspective, alignment outcomes are governed by the design of the alignment data pipeline, which specifies the space of candidate behaviors, the mechanism by which they are evaluated, and the structure through which preference signals are exposed to optimization\.

### 3\.1Alignment Data as Optimization Signals

Alignment tuning relies on preference signals constructed through a data pipeline rather than given a priori\. We formalize how this process yields the optimization signals that ultimately drive policy updates and shape the learned behavior\.

Formalizing the Alignment Data Pipeline\.Unlike static pre\-training corpora, alignment data is dynamically constructed through an iterative pipeline that couples response generation and preference assessment\. We formalize the resulting dataset𝒟\\mathcal\{D\}as a collection of structured training instances produced by three interacting components:

𝒟=\{\(x,𝐲,𝐬\)\|x∼P​\(x\),𝐲∼𝒮​\(𝐲∣x\),𝐬∼ℰ​\(𝐬∣x,𝐲\)\}\.\\mathcal\{D\}=\\Big\\\{\\big\(x,\\;\\mathbf\{y\},\\;\\mathbf\{s\}\\big\)\\;\\Big\|\\;x\\sim P\(x\),~\\mathbf\{y\}\\sim\\mathcal\{S\}\(\\mathbf\{y\}\\mid x\),~\\mathbf\{s\}\\sim\\mathcal\{E\}\(\\mathbf\{s\}\\mid x,\\mathbf\{y\}\)\\Big\\\}\.\(2\)Here,xxis a prompt sampled from the task distributionPP;𝐲=\{y1,…,yk\}\\mathbf\{y\}=\\\{y\_\{1\},\\ldots,y\_\{k\}\\\}is a set of candidate responses generated by a response synthesis strategy𝒮\\mathcal\{S\}, which defines the behavioral support available for alignment; and𝐬\\mathbf\{s\}represents preference signals assigned by an evaluatorℰ\\mathcal\{E\}, which are subsequently structured through preference instantiation to form training signals such as scalar scores, pairwise preferences, or rankings over𝐲\\mathbf\{y\}\.

Optimization as Margin Alignment\.Alignment algorithms optimize the policyπθ\\pi\_\{\\theta\}so that its implicit preferences match the explicit signals encoded in the alignment dataset𝒟\\mathcal\{D\}\. Across PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26442#bib.bib48)\), DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4)\), and GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib52)\), this process can be viewed as*margin alignment*, where optimization aligns policy\-induced preference margins with observed preference signals\.

Given a promptxxwith candidate responses𝐲\\mathbf\{y\}and preference signals𝐬\\mathbf\{s\}, alignment tuning aims to adjust the policyπθ\\pi\_\{\\theta\}so that its implicit preferences are consistent with the structure and magnitude of the preference information encoded in𝐬\\mathbf\{s\}, aligning model behavior with the supervision induced by the dataset𝒟\\mathcal\{D\}\. This objective can be abstractly written as:

maxθ⁡𝔼𝒟​\[f​\(Mθ​\(x,𝐲,𝐬\)\)\],\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\big\[f\\big\(M\_\{\\theta\}\(x,\\mathbf\{y\},\\mathbf\{s\}\)\\big\)\\big\],\(3\)whereMθ​\(x,𝐲,𝐬\)M\_\{\\theta\}\(x,\\mathbf\{y\},\\mathbf\{s\}\)denotes an alignment measure that quantifies how well the policy\-induced implicit preferences align with the preference signals𝐬\\mathbf\{s\}\. The functionfftransforms this alignment measure into an optimizable objective in a way that preserves the relative preference ordering\.

Pipeline Defines Optimization Signal\.Eq\. \([2](https://arxiv.org/html/2605.26442#S3.E2)\) and Eq\. \([3](https://arxiv.org/html/2605.26442#S3.E3)\) together show that alignment outcomes are determined not only by the optimization objective, but by how the alignment data pipeline constructs its inputs\. While Eq\. \([3](https://arxiv.org/html/2605.26442#S3.E3)\) frames alignment as maximizing a margin\-based objectiveMθ​\(x,𝐲,𝐬\)M\_\{\\theta\}\(x,\\mathbf\{y\},\\mathbf\{s\}\), Eq\. \([2](https://arxiv.org/html/2605.26442#S3.E2)\) specifies the response candidates𝐲\\mathbf\{y\}, preference signals𝐬\\mathbf\{s\}, and relational structure on which this margin is defined\. As a result, the pipeline does not merely provide data, but actively shapes the space, scale, and reliability of the preference margins available to optimization, thereby influencing the nature of the learned policy\.

### 3\.2Overview of Alignment Data Pipeline

As the optimization signal is determined by how alignment data is constructed, we briefly summarize the structure of the alignment data pipeline\.

The alignment data pipeline begins with*response synthesis*\(Figure[1](https://arxiv.org/html/2605.26442#S1.F1)\(a\)\), which defines the behavioral scope of alignment by generating candidate responses\. The pipeline then proceeds to*preference evaluation*\(Figure[1](https://arxiv.org/html/2605.26442#S1.F1)\(b\)\), which assigns supervisory signals to approximate latent preferences\. Finally,*preference instantiation*\(Figure[1](https://arxiv.org/html/2605.26442#S1.F1)\(c\)\) converts evaluated judgments into optimization compatible training signals\. Detailed discussions of each stage and their associated design trade\-offs are provided in Sections[4](https://arxiv.org/html/2605.26442#S4)–[7](https://arxiv.org/html/2605.26442#S7)\. Additionally, we categorizes all the methods we investigated across the three aspects in Table[5](https://arxiv.org/html/2605.26442#A0.T5)\.

## 4Response Synthesis Stage

This stage defines the exploration space of candidate responses through the synthesis strategy𝒮\\mathcal\{S\}\. Since alignment operates only on sampled responsesy∼𝒮​\(y∣x\)y\\sim\\mathcal\{S\}\(y\\mid x\), it constrains which behaviors enter alignment\. It involves three design considerations:*\(§[4\.1](https://arxiv.org/html/2605.26442#S4.SS1)\) response source*,*\(§[4\.2](https://arxiv.org/html/2605.26442#S4.SS2)\) selection strategy*, and*\(§[4\.3](https://arxiv.org/html/2605.26442#S4.SS3)\) creative exploration*\.

### 4\.1Response Source

The first design choice in response synthesis is the response source \(*i\.e\.*, where candidate responses are drawn from\)\. This choice determines the distributional relationship between the data policyπdata\\pi\_\{\\text\{data\}\}and the learned policyπθ\\pi\_\{\\theta\}, which in turn affects the effectiveness of alignment signals\. Existing approaches have evolved along two philosophies,*offline*and*online*, with techniques developed to mitigate their inherent limitations\.

Offline with Policy\-Aware Reweighting\.Offline approaches rely on fixed, high\-quality responses from stronger models \(*e\.g\.*, proprietary LLMs\) or curated datasets\. While cost\-effective, this setting induces distributional shift\(Xionget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib63); Boseet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib60)\), as the policy is optimized on responses it is unlikely to generate, resulting in biased value estimation\. To mitigate this mismatch, recent methods adopt policy\-aware reweighting, which re\-scales samples by the current policy likelihood \(*e\.g\.*,πθ/πdata\\pi\_\{\\theta\}/\\pi\_\{\\text\{data\}\}\)\. They differ primarily in the granularity of reweighting: preference\-level weighting in WPO\(Zhouet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib57)\), reward\-difference modulation in RDO\(Wanget al\.,[2024c](https://arxiv.org/html/2605.26442#bib.bib59)\), weighted reward aggregation across multiple sources in WRPO\(Yanget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib58)\), and token\-level importance weighting in TIS\-DPO\(Liuet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib61)\)\. By down\-weighting unlikely responses, policy\-aware reweighting reduces off\-policy bias and enables reliable offline alignment\.

Online with Structured Self\-Improvement\.Online approaches train directly on outputs from the current policyπθ\\pi\_\{\\theta\}, providing on\-policy supervision and reducing distributional mismatch\. Yet, they incur high computational cost from online generation and can suffer instability due to low\-quality rollouts\. Recent work addresses these issues through structured self\-improvement\. Iterative DPO\(Xionget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib63)\)and RS\-DPO\(Khakiet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib64)\)stabilize training via rejection\-based filtering, while self\-play methods such as SPIN\(Chenet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib11)\)and SPPO\(Wuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib13)\)cast alignment as a zero\-sum game without external oracles\. Further extensions improve efficiency through group\-relative feedback in GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib52)\)and active exploration in SELM\(Zhanget al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib14)\)\.

### 4\.2Selection Strategy

Another key design choice in response synthesis is the selection strategy, which determines how informative instances are selected, either post\-hoc \(offline\) or during response pool construction \(online\)\. Existing approaches differ primarily in how informativeness is quantified through*margin*\-based or*uncertainty*\-based criteria\.

Margin\-Based Selection\.Margin\-based selection identifies training instances with high alignment potential, defined as response comparisons for the same prompt \(*e\.g\.*, pairs or small ranked sets\) where preferences clearly separate better from worse behaviors\.Huanget al\.\([2025](https://arxiv.org/html/2605.26442#bib.bib50)\)distinguish explicit margins reflecting preference strength from implicit margins induced by policy likelihood differences, whereas BeeS\(Denget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib34)\)uses the two margins independently to filter informative preference instances\. Beyond filtering, MMPO\(Kimet al\.,[2024a](https://arxiv.org/html/2605.26442#bib.bib66)\)incorporates explicit margins directly into the optimization objective via soft targets, allowing the model to account for graded preference strength rather than treating all comparisons equally\. Similarly,Yaoet al\.\([2025](https://arxiv.org/html/2605.26442#bib.bib67)\)improve alignment efficiency by modifying how preference margins are integrated into the loss\.

Uncertainty\-Based Selection\.Unlike margin\-based selection, uncertainty\-based selection targets low\-confidence regions of the policy where preference signals are ambiguous or weakly specified\. Early methods use predictive entropy or probability dispersion, while recent work adopts structured uncertainty modeling\. APL\(Muldrewet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib70)\)and MAPLE\(Mahmudet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib72)\)reduce preference uncertainty via information gain maximization and Bayesian adaptive selection, respectively\. In online settings, uncertainty serves as a quality\-control signal: IUPO\(Liet al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib71)\)targets locally ambiguous reasoning steps through token\-level uncertainty, while UPO\(Wanget al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib73)\)filters unreliable samples using reward\-model uncertainty\. More recent methods like UDASA\(Sunet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib74)\)further decompose uncertainty into semantic, factual, and value\-alignment dimensions with curriculum\-style selection\.

### 4\.3Creative Exploration

Alignment objectives often overemphasize a narrow set of preferred responses, concentrating probability mass and inducing*mode collapse*that reduces semantic diversity and creativity\(Kirket al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib68); Murthyet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib69)\)\. To address this mode collapse, recent methods intervene directly in response synthesis, beyond informativeness\-driven selection\. During candidate construction, Verbalized Sampling\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib75)\)promotes exploration by eliciting multiple plausible responses and their likelihoods, while Spectrum Tuning\(Sorensenet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib76)\)trains on diverse valid outputs to control stylistic variation\. At the post\-hoc selection stage, DivPO\(Lanchantinet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib12)\)preserves high\-quality but rare responses via diversity\-aware filtering, andChunget al\.\([2025](https://arxiv.org/html/2605.26442#bib.bib77)\)reweight the objective to favor unique answers\. Finally, CRPO\(Ismayilzadaet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib78)\)explicitly rewards novelty and surprise alongside utility\.

## 5Preference Evaluation Stage

Once response candidates are generated, this stage assigns preference judgments via an adjudicator\. While early RLHF relied on human judgments, recent work increasingly adopts automated adjudicators to reduce cost and variability\(Liet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib80); Minet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib79)\)\. Accordingly, prior work can be organized along three design axes in Figure[2](https://arxiv.org/html/2605.26442#S5.F2):*\(§[5\.1](https://arxiv.org/html/2605.26442#S5.SS1)\) adjudicator type*,*\(§[5\.2](https://arxiv.org/html/2605.26442#S5.SS2)\) evaluation granularity*, and*\(§[5\.3](https://arxiv.org/html/2605.26442#S5.SS3)\) objective dimensionality*\.

![Refer to caption](https://arxiv.org/html/2605.26442v1/x2.png)Figure 2:Preference evaluation axes in alignment data pipelines, illustrating interplay among adjudicator type, judgment granularity, and objective dimensionality\.### 5\.1Human to AI Adjudicator

The first axis is the source of the preference signal\. While human evaluation was traditionally considered the gold standard, recent findings indicate that inter\-rater agreement is often insufficient for complex reasoning or coding tasks, thereby limiting the upper bound of learnable reward signals\(Kocmi and Federmann,[2023](https://arxiv.org/html/2605.26442#bib.bib81); Liet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib82)\)\.

Use of LLM\-as\-a\-Judge\.The field has shifted toward LLM\-as\-a\-Judge frameworks\(Liet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib82)\), in which LLMs are used as automated evaluators to score or compare candidate responses, improving scalability and reproducibility\. These frameworks have been instantiated in diverse way\. For example, RLAIF\(Leeet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib25)\)simply uses off\-the\-shelf LLMs as automated preference annotators, providing comparison\-based feedback for RL, while Constitutional AI\(Baiet al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib51)\)employs a critique\-and\-revise loop guided by natural language principles to generate preference labels\. More recently, Self\-Rewarding LMs\(Yuanet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib30)\)integrate the judge capability directly into the policy model, enabling it to self\-assign rewards during iterative alignment training\.

Bias Sources and Mitigation\.A critical challenge in LLM\-based automated evaluation is intrinsic bias, where the judge’s internal priors distort evaluation outcomes\(Liuet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib84); Zhenget al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib83)\)\. Recent work frames this as a calibration problem, using algorithmic interventions to neutralize specific biases\. Specifically,Wataokaet al\.\([2024](https://arxiv.org/html/2605.26442#bib.bib85)\)show that LLMs favor outputs from their own model family, exhibiting an “echo chamber” effect, and propose cross\-family evaluation using architecturally distinct LLMs as a mitigation strategy\.Wanget al\.\([2024a](https://arxiv.org/html/2605.26442#bib.bib86)\)reveal a pronounced verbosity bias, where LLM judges favor longer responses, and mitigate it through length normalization and constrained evaluation\.Ahrabianet al\.\([2025](https://arxiv.org/html/2605.26442#bib.bib87)\)further identify position bias, where models preferentially select the first option, and mitigate it through permutation\-based evaluation\.

Collective Intelligence\.Despite calibration efforts, single model judgments remain inherently unstable, as they conflate prompt sensitivity and stochastic variation into a single evaluation signal\.Yuet al\.\([2025c](https://arxiv.org/html/2605.26442#bib.bib15)\)aggregates judgments from heterogeneous LLM evaluators to average out individual noise\. Hierarchical schemes like Meta\-Rater\(Zhuanget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib6)\)further refine this by employing a superior meta\-judge to adjudicate conflicts among primary judges\. More interactive paradigms instantiate consensus\-seeking through multi\-agent debate\(Duet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib89); Chenet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib90)\), where judges iteratively critique and refine evaluations to yield stable supervision signals\. These collective mechanisms improve fidelity to the oracle reward, preventing evaluator noise from being treated as an optimization signal during alignment tuning\.

### 5\.2Judgment Granularity

The second axis concerns judgment granularity \(*i\.e\.*, how preference signals are formed within a response\)\. Early methods rely on outcome\-level supervision, collapsing response quality into a single global judgment\(Liet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib82); Liuet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib84)\)\. While sufficient for short tasks, this is inadequate for long\-form and multi\-step generation, where errors and merits are unevenly distributed\(Wanget al\.,[2024b](https://arxiv.org/html/2605.26442#bib.bib91); Yanet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib92)\)\. Recent work thus increases granularity, shifting from outcome\-level supervision to localized signals\.

Step\-Level Supervision\.Step\-level supervision increases judgment granularity by assigning preference signals along the generation process rather than only at the final outcome\(Zhenget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib95); Liet al\.,[2025e](https://arxiv.org/html/2605.26442#bib.bib96)\)\. It is most effective in domains with explicit intermediate structure, such as mathematics and coding\. Process reward models \(PRMs\) instantiate this paradigm by evaluating intermediate states\. Early PRMs relied on human annotation, but recent ones emphasizes automated supervision\. Specifically, Math\-Shepherd\(Wanget al\.,[2024b](https://arxiv.org/html/2605.26442#bib.bib91)\)estimates soft step\-level labels via Monte Carlo sampling\. OmegaPRM\(Luoet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib93)\)improves efficiency by localizing the earliest erroneous step using binary search\. GenRM\(Zhanget al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib94)\)reframes verification as next\-token prediction\. VersaPRM\(Zenget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib97)\)further extends step\-level supervision to multiple domains\.

Atomic\-Level Supervision\.For open\-ended generation, where explicit reasoning trajectories are ill\-defined, supervision shifts from global outcomes to atomic units \(sentences, spans, or tokens\) to better localize hallucinations, stylistic errors, and safety violations\. At the sentence level, Fine\-Grained RLHF\(Wuet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib98)\)provides attribute\-specific rewards, while ASPO\(Wanget al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib99)\)adaptively down\-weights hallucinated sentences in multi\-modal reasoning\. At finer resolution, token\-level methods directly integrate preference signals into optimization\. TDPO\(Zenget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib65)\)decomposes the KL regularization to the token\-level for precise diversity–quality control, and TIS\-DPO\(Liuet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib61)\)applies importance sampling to emphasize preference\-critical tokens\. Complementarily, ACPO\(Chenet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib100)\)improves factuality by decomposing responses into atomic facts and scoring them via intrinsic self\-consistency\.

### 5\.3Objective Dimensionality

The final axis is objective dimensionality: whether preference signals are scalarized or preserve the multi\-dimensional structure of human values\.

Alignment Tax and Reward Hacking\.Standard alignment pipelines often rely on scalarized objectives, encoding preference supervision as a single reward or binary label\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib24); Stiennonet al\.,[2020](https://arxiv.org/html/2605.26442#bib.bib28)\)\. As alignment objectives become conflicting, this scalarization obscures context\-dependent trade\-offs, limiting robust generalization\. One consequence is the*alignment tax*, where optimizing a unified reward favors easily optimizable sub\-objectives over others\(Linet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib103)\), as seen in trade\-offs between helpfulness and harmlessness\(Tanet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib104)\)or completeness and conciseness\(Songet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib101)\)\. Another consequence is*reward hacking*, where models exploit low\-dimensional shortcuts that inflate reward scores without improving true alignment\(Panet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib106)\)\.

Toward Multi\-Dimensional Alignment\.To overcome the limitations of scalarized objectives, recent work preserves objective dimensionality by intervening at different stages of the alignment pipeline, namely response synthesis, evaluation, and optimization\.

One line of work mitigates scalar collapse by restructuring the candidate behavior space at the*response synthesis stage*, prior to evaluation\. For example, SteerLM\(Donget al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib107)\)and HelpSteer\(Wanget al\.,[2024e](https://arxiv.org/html/2605.26442#bib.bib108);[d](https://arxiv.org/html/2605.26442#bib.bib109)\)condition generation on explicit control signals that specify which value dimension to emphasize, such as helpfulness and safety\. This design allows scalar rewards to operate over pre\-separated alignment profiles\.

A complementary direction mitigates the collapse at the*evaluation stage*by decomposing preference judgments into explicit criteria\. Early methods such as Constitutional AI\(Baiet al\.,[2022](https://arxiv.org/html/2605.26442#bib.bib51)\)and Self\-Align\(Sunet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib112)\)encode alignment objectives as normative rules guiding feedback across multiple dimensions\. More recent rubric\- and rationale\-based evaluators, including Prometheus\(Kimet al\.,[2024b](https://arxiv.org/html/2605.26442#bib.bib110)\)and OpenRubrics\(Liuet al\.,[2025f](https://arxiv.org/html/2605.26442#bib.bib111)\), operationalize this idea by evaluating responses against explicit rubrics, making multi\-dimensional trade\-offs explicit\.

At the*optimization stage*, methods address conflicts by framing alignment as a multi\-objective problem\. Early methods include sequential alignment\(Louet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib114)\), which optimizes objectives in separate stages, and post\-hoc parameter merging\(Janget al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib113)\), which combines models fine\-tuned for different goals\. More recent work jointly optimizes multiple objectives during training, as exemplified by MOPO\(Agnihotriet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib115)\), PAMA\(He and Maghsudi,[2025](https://arxiv.org/html/2605.26442#bib.bib116)\), and MO\-ODPO\(Guptaet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib118)\), enabling more principled and controllable trade\-offs\.

## 6Preference Instantiation Stage

Following preference evaluation, the pipeline proceeds to preference instantiation, the translation layer between evaluation and optimization\. This stage converts evaluated responses into structured training signals, determining how preference relations are exposed to the policy model\. Instantiation methods are categorized by the relational complexity of feedback, ranging from*\(§[6\.1](https://arxiv.org/html/2605.26442#S6.SS1)\) point\-wise*to*\(§[6\.2](https://arxiv.org/html/2605.26442#S6.SS2)\) pair\-wise*and*\(§[6\.3](https://arxiv.org/html/2605.26442#S6.SS3)\) group\-wise*\.

### 6\.1Point\-Wise Instantiation

Point\-wise instantiation assigns an independent scalar or binary signal to each prompt –response pair\(x,y\)\(x,y\), avoiding explicit comparisons and enabling simple policy training aligned with many real\-world feedback acquisition scenarios\.

Regression\-Based Supervision\.A common approach to scalar supervision trains a separate reward model to score each prompt–response pair and optimizes the policy indirectly, as in the classical PPO family\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26442#bib.bib48); Daiet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib53); Liet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib54)\)\. Recent work improves reward fidelity to provide more reliable optimization signals and strengthen downstream alignment\. For example, Critic\-RM\(Yuet al\.,[2025d](https://arxiv.org/html/2605.26442#bib.bib122)\)uses model\-generated critiques to refine reward prediction, while hybrid frameworks such as HAF\-RM\(Liuet al\.,[2025d](https://arxiv.org/html/2605.26442#bib.bib123)\)incorporate token\-level supervision during reward model training\. In contrast, A\*PO\(Brantleyet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib121)\)skips the reward model and directly trains the policy to match offline value targets via regression\.

Binary\-Based Supervision\.In contrast, binary\-based supervision relies on coarse yes/no feedback that indicates whether a response is acceptable\. KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib26)\)relies solely on binary preference signals, applying asymmetric weighting to penalize undesirable outputs more strongly than desirable ones\. Later work improves the theoretical and semantic grounding of binary supervision: BCO\(Junget al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib119)\)reframes alignment from binary feedback as a classification problem, showing that the binary ones are sufficient to recover the same response ordering as DPO\. RLBFF\(Wanget al\.,[2025c](https://arxiv.org/html/2605.26442#bib.bib120)\)improves binary supervision by clarifying what a yes/no label means, training the model to decide whether a response meets a specific criterion \(*e\.g\.*, correctness, clarity\) instead of assigning an ambiguous binary label\.

### 6\.2Pair\-Wise Instantiation

While point\-wise feedback is easy to obtain, it lacks the relational structure needed for fine\-grained alignment tuning\(Tripathiet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib124)\)\. Pair\-wise instantiation uses tuples\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\)withyw≻yly\_\{w\}\\succ y\_\{l\}to model the decision boundary between preferred and dispreferred behaviors, forming the dominant paradigm in alignment tuning\.

Contrastive Optimization\.The standard approach in this category is DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4)\), which bypasses explicit reward modeling by directly optimizing the policy with a contrastive objective comparing preferred and dispreferred responsesywy\_\{w\}andyly\_\{l\}\. Despite its effectiveness, this formulation has several limitations, including overfitting and stability\. IPO\(Azaret al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib126)\)mitigates the DPO’s overfitting by replacing the unbounded logit\-based objective with a bounded one, while GPO\(Tanget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib127)\)unifies DPO and IPO within a generalized offline preference learning framework\. In a complementary direction, sDPO\(Kimet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib128)\)improves stability on large datasets by progressively tightening the reference policy, and MRPO\(Leet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib129)\)extends the objective to multiple reference models via a stabilized virtual reference distribution\.

Reference\-Free Optimization\.While effective, DPO\-style methods rely on a reference model to anchor preference optimization, introducing additional overhead and potential instability\. To remove this dependency, recent work derives preference signals directly from the policy’s output distribution\. SimPO\(Menget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib32)\)eliminates the reference model by reinterpreting preference learning as direct optimization over the policy’s own output distribution\. ORPO\(Honget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib49)\)similarly adopts a reference\-free formulation, integrating preference contrast into supervised fine\-tuning via an odds\-ratio objective\.

Margin\-Aware Calibration\.Another limitation of standard pair\-wise objectives is that they enforce relative ordering without ensuring that confidence gaps reflect preference magnitude\. SLiC\-HF\(Zhaoet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib131)\)introduces a fixed margin via a hinge loss to better align likelihood gaps with preference strength\. MMPO\(Kimet al\.,[2024a](https://arxiv.org/html/2605.26442#bib.bib66)\)instead calibrates margins probabilistically using soft preference targets, while AlphaDPO\(Wuet al\.,[2025a](https://arxiv.org/html/2605.26442#bib.bib130)\)further generalizes this approach by adapting reward margins at the instance\-level through implicit reference reparameterization\.

### 6\.3Group\-Wise Instantiation

As alignment targets shift from subjective conversation to objective reasoning, point\- and pair\-wise feedback become bottlenecks due to their sparsity and lack of contextual normalization\. Group\-wise instantiation addresses this by optimizing the policy over a candidate set𝐲=\{y1,…,yk\}\\mathbf\{y\}=\\\{y\_\{1\},\\dots,y\_\{k\}\\\}, capturing combinatorial relationships and providing denser signals for complex preference learning\.

List\-Wise Ranking Objectives\.To realize this, methods have progressed from surrogate ranking losses to direct metric optimization\. RRHF\(Yuanet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib134)\)operates as a best\-of\-NNlearner using a hinge\-style loss to align likelihoods with preference scores, while PRO\(Songet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib135)\)refines this via iterative best\-vs\-rest decomposition to capture multi\-level preferences\. Incorporating learning\-to\-rank principles, LiPO\(Liuet al\.,[2025g](https://arxiv.org/html/2605.26442#bib.bib33)\)applies LambdaRank\-style weighting\(Burgeset al\.,[2006](https://arxiv.org/html/2605.26442#bib.bib132)\)to emphasize top\-ranked items\. Most recently, PPA\(Zhaoet al\.,[2025](https://arxiv.org/html/2605.26442#bib.bib133)\)directly optimizes a differentiable NDCG objective, closing the gap between training loss and ranking metrics\.

Group\-Relative Policy Optimization\.Beyond explicit ranking, GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib52)\)derives advantage signals from group\-level statistics rather than fixed preference orders\. While this critic\-free formulation reduces variance, later analyses identify estimation instability\. In particular, Dr\. GRPO\(Liuet al\.,[2025i](https://arxiv.org/html/2605.26442#bib.bib56)\)corrects a group\-normalization bias that conflates reasoning quality with response length\. DAPO\(Yuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib5)\)further mitigates entropy collapse in group\-wise optimization\.

## 7Key Insights and Practical Guidance

Building on the unified pipeline perspective, we analyze fundamental trade\-offs \(§[7\.1](https://arxiv.org/html/2605.26442#S7.SS1)\) and cross\-stage interactions \(§[7\.2](https://arxiv.org/html/2605.26442#S7.SS2)\) that govern how alignment signals are constructed and propagated, and distill a set of design principles \(§[7\.3](https://arxiv.org/html/2605.26442#S7.SS3)\) that characterize effective pipeline configurations\. Finally, we translate these insights into practical, scenario\-driven guidance, linking real\-world constraints to concrete end\-to\-end pipeline design choices\.

### 7\.1Core Structural Trade\-offs

Many limitations in alignment tuning arise not from individual algorithms, but from structural trade\-offs inherent to alignment data pipelines\.

∙\\bullet*Source Fidelity vs\. Distribution Shift\.*Offline supervision using high\-quality proprietary models provides a strong, reproducible starting point but lags behind policy evolution, inducing a distribution shift as the student is evaluated on responses it cannot naturally generate\. In contrast, online supervision via self\-play preserves on\-policy fidelity but introduces higher variance, making it prone to error amplification when the initial policy is weak\. This choice shapes the exploration space and propagates to evaluation and instantiation by constraining what can be effectively judged and optimized\.

∙\\bullet*Open\-Loop vs\. Closed\-Loop Alignment\.*Many alignment pipelines operate as if in an open\-loop setting, treating preference data and evaluators as static\. However, alignment is inherently dynamic and policy\-dependent, as evolving outputs continually reshape the data\. The choice of loss influences the policy’s entropy and output distribution, constraining future exploration\. Ignoring this can lead to degeneracy such as reduced diversity or reward hacking, highlighting the need for adaptive mechanisms like response resampling and evaluator recalibration\.

∙\\bullet*Evaluation Granularity vs\. Credit Assignment Precision\.*Shifting from outcome\-level to step\-level evaluation improves credit assignment for complex reasoning by providing more fine\-grained supervision\. However, this added precision comes with substantial labeling complexity and introduces the risk of over\-optimizing intermediate steps that are not strictly necessary for producing the final correct answer\.

∙\\bullet*Objective Dimensionality vs\. Optimization Tax\.*Collapsing multi\-dimensional human values into a single scalar reward simplifies the instantiation stage but introduces an alignment tax, as important nuances across dimensions are lost\. In contrast, maintaining multi\-objective rubrics better preserves performance across diverse criteria, but significantly complicates the optimization landscape, making convergence slower and less stable\.

### 7\.2Cross\-Stage Interactions

Although we categorize alignment tuning into three distinct stages—response synthesis, preference evaluation, and preference instantiation—alignment outcomes are ultimately determined by the tight coupling and interactions between them\. As in Sections[4](https://arxiv.org/html/2605.26442#S4)–[6](https://arxiv.org/html/2605.26442#S6), the output of one stage serves as the rigid input constraint for the next, and in iterative settings, the optimization outcome feeds back into the data generation process\. This section details the three critical interaction pathways that define the efficacy of the alignment pipeline\.

#### 7\.2\.1Response Synthesis Constrains Preference Evaluation

The quality of the preference signal is fundamentally bounded by the behavioral support generated during the response synthesis stage\. No matter how sophisticated an adjudicator \(in preference evaluation stage\) is, it cannot extract meaningful signals from a non\-informative candidate set\.

∙\\bullet*Informativeness and Selection Efficiency:*The synthesis stage must prioritize generating ”informative” candidates that provide high learning value\. If the synthesis strategy produces only trivial comparisons \(*e\.g\.*, a perfect response versus a nonsensical one\) or ambiguous pairs with negligible quality differences, the evaluator cannot extract a strong gradient signal\. Consequently, even a ground\-truth evaluator becomes ineffective if the candidate set lacks the margins required to distinguish better behaviors from worse ones\.

∙\\bullet*Diversity and Contrastive Information:*Alignment tuning relies on contrast\. If the synthesis strategy suffers from mode collapse or lacks exploration \(as discussed in Section[4\.3](https://arxiv.org/html/2605.26442#S4.SS3)\), the candidate set will consist of semantically identical responses\. In this scenario, even a human expert or a strong LLM\-as\-a\-Judge cannot assign a meaningful preference margin, leading to vanishing gradients during optimization\.

∙\\bullet*Distributional Shift and Evaluator Reliability:*The reliability of evaluation depends on how the responses are generated\. If synthesized responses come from a distribution that differs substantially from the data used to calibrate the evaluator, the evaluator’s confidence estimates can become misaligned\. As a result, noisy or misleading scores may be injected into the optimization signal\.

#### 7\.2\.2Preference Evaluation Dictates Instantiation Capabilities

The granularity and dimensionality of the judgments produced in the preference evaluation stage determine the upper bound of what can be modeled in the instantiation stage\. A mismatch here often leads to information loss or signal distortion\.

∙\\bullet*Granularity Mismatch:*If the evaluation stage only provides outcome\-level binary labels, attempting to use token\-level or step\-level instantiation methods \(*e\.g\.*, TIS\-DPO or ACPO\) is ill\-posed without additional approximations\. Conversely, if detailed step\-level critiques are available but the instantiation method aggregates them into a single scalar reward, the dense supervision signal is compressed, losing the ability to penalize reasoning errors\.

Table 1:Design principles for alignment data pipelines\. We distill six key principles governing response synthesis, preference evaluation, and instantiation\.∙\\bullet*Dimensionality and Alignment Tax:*When evaluation captures multi\-dimensional trade\-offs \(*e\.g\.*, helpfulness vs\. safety\), but the instantiation layer collapses these into a single scalarized value, it forces the policy to optimize a fixed trade\-off\. This often results in the ”alignment tax,” where improvements in one dimension inadvertently degrade performance in another due to the inability of the scalar loss to represent the Pareto frontier\.

#### 7\.2\.3Preference Instantiation Reshapes Future Response Synthesis

In iterative or online alignment settings, the pipeline operates as a closed loop\. The choice of loss function in the preference instantiation stage directly impacts the policy model’s entropy, which in turn defines the search space for the next round of response synthesis\.

∙\\bullet*Mode Collapse and Exploration:*Contrastive objectives like DPO effectively increase the likelihood of preferred responses but can rapidly reduce the entropy of the policy\. If the instantiation stage enforces margins too aggressively, the updated policy model may lose the diversity required for effective exploration in the subsequent synthesis step\. This creates a degenerate cycle where the model stops generating novel negatives, starving the pipeline of the informative data needed for further improvement\.

∙\\bullet*Reward Hacking Dynamics:*If the instantiation mechanism has exploitable shortcuts, the policy will learn to maximize the score without improving true quality\. Over time, it produces responses that look good according to the reward but are actually poor\. This makes evaluation increasingly difficult and often requires the evaluator to be updated or recalibrated\.

### 7\.3Pipeline Design Principles

The trade\-offs and interactions motivate design principles for alignment data pipelines\. Rather than prescribing specific algorithms, these principles characterize how response synthesis, preference evaluation, and preference instantiation should be jointly designed to manage these trade\-offs\. Based on our analysis of existing methods, we distill six principles to capture effective design patterns across response synthesis, response evaluation, and response instantiation\. We summarize the principles and their core insights in Table[1](https://arxiv.org/html/2605.26442#S7.T1), which serves as a unifying reference for prior work\.

Beyond conceptual principles above, translating these insights into practice requires addressing real\-world constraints such as cost, latency, hardware limits, and labeling budgets\. To make this survey a practical, we map alignment methods to design scenarios shaped by resource limitations and system\-level trade\-offs\. Therefore, we provide stage\-wise decision frameworks linking these practical constraints to specific alignment design choices\.

∙\\bullet*Response Synthesis Stage \(Constraints on Exploration\)\.*The primary function of this stage is to define the behavioral support of the policy\. The design choice is governed by Data Availability \(Does the data exist?\) and Inference Budget \(Can we afford to generate it?\)\. Table[2](https://arxiv.org/html/2605.26442#S7.T2)outlines how different constraints lead to distinct synthesis strategies, each with inherent trade\-offs between efficiency, distribution fidelity, and exploration\.

Table 2:Response synthesis strategies under real\-world constraints\.Table 3:Preference evaluation strategies under real\-world constraints\.∙\\bullet*Preference Evaluation Stage \(Constraints on Fidelity\)\.*The primary function of this stage is to assign preference judgments via an adjudicator\. The design choice is governed by Task Complexity \(How hard is it to evaluate?\), Labeling Budget \(Can we afford high\-quality judges?\), and Objective Dimensionality \(Are there conflicting goals like safety vs\. helpfulness?\)\. Table[3](https://arxiv.org/html/2605.26442#S7.T3)organizes these constraints into corresponding choices of adjudicator and granularity, highlighting the trade\-offs between cost and evaluation precision\.

∙\\bullet*Preference Instantiation Stage \(Constraints on Optimization\)\.*The primary function of this stage is to convert evaluated responses into structured training signals\. The design choice is governed by Hardware Constraints \(VRAM\), Task Variance \(Instabilities in optimization\), and Objective Dimensionality \(Conflicting goals\)\. Table[4](https://arxiv.org/html/2605.26442#S8.T4)organizes these constraints into corresponding feedback instantiation strategies, highlighting the trade\-offs between computational efficiency, optimization stability, and signal expressiveness\.

Additionally, Appendix[B](https://arxiv.org/html/2605.26442#A2)presents scenario\-driven pipeline configurations that jointly integrate response synthesis, preference evaluation, and instantiation, enabling end\-to\-end design under realistic deployment constraints\.

## 8Future Research Directions

Despite substantial progress in alignment tuning, several fundamental challenges remain unresolved, particularly as alignment settings become more dynamic, complex, and deployment\-driven\. To address these limitations, we highlight four key directions for future research that aim to move beyond current pipeline paradigms and enable more adaptive, robust, and scalable alignment frameworks\.

∙\\bullet*Prompt\-Level Alignment\.*Alignment must move beyond global preference models toward prompt\-level alignment\. Different prompts inherently require different evaluation criteria such as safety creativity or precision\. However current pipelines apply a shared preference function across heterogeneous prompts\. A central challenge is inferring and enforcing prompt specific alignment criteria without explicit human specification\.

∙\\bullet*Alignment for Agentic Systems\.*As agentic systems become dominant alignment must move beyond individual decisions to agentic loops\. In long horizon workflows failures arise from interactions among planning execution feedback and memory rather than single actions\. Existing methods supervise local outputs while leaving global behavior unconstrained\. A key challenge is defining trajectory\-level alignment objectives that enable agents to revise or abandon plans over time\.

Table 4:Preference instantiation strategies under real\-world constraints\.∙\\bullet*Alignment under Evolving Objectives\.*Future alignment must account for objectives that evolve over time rather than remain fixed\. In interactive and agentic settings user intent task goals and acceptable behavior change during deployment\. Current pipelines assume static training time preferences\. A key challenge is adapting alignment to evolving objectives without destabilizing learned behavior or inducing value drift\.

∙\\bullet*Multi\-Modal Alignment\.*As LLMs evolve into large multi\-modal models, alignment must extend beyond text\. Modalities such as video introduce tightly coupled temporal, spatial, and visual structures that are difficult to control with coarse supervision\. A key challenge is designing cross\-modal alignment pipelines that translate high\-level feedback into fine\-grained updates over multi\-modal representations\.

## 9Conclusion

This survey reframes alignment tuning as a data\-centric pipeline design problem\. By decomposing alignment into response synthesis, preference evaluation, and preference instantiation, we show how pipeline design governs alignment behavior\. Our analysis reveals recurring trade\-offs and interactions that guide future alignment pipelines\.

## References

- Multi\-objective preference optimization: improving human alignment of generative models\.arXiv preprint arXiv:2505\.10892\.Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p6.1)\.
- K\. Ahrabian, X\. Lin, B\. Patra, V\. Chaudhary, A\. Benhaim, J\. Pujara, and X\. Song \(2025\)A practical analysis of human alignment with\* po\.InNAACL,Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p3.1)\.
- A\. Albalak, Y\. Elazar, S\. M\. Xie, S\. Longpre, N\. Lambert, X\. Wang, N\. Muennighoff, B\. Hou, L\. Pan, H\. Jeong, C\. Raffel, S\. Chang, T\. Hashimoto, and W\. Y\. Wang \(2024\)A survey on data selection for language models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- M\. G\. Azar, Z\. D\. Guo, B\. Piot, R\. Munos, M\. Rowland, M\. Valko, and D\. Calandriello \(2024\)A general theoretical paradigm to understand learning from human preferences\.InAISTAT,Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p2.2)\.
- C\. Bai, Y\. Zhang, S\. Qiu, Q\. Zhang, K\. Xu, and X\. Li \(2025\)Online preference alignment for language models via count\-based exploration\.arXiv preprint arXiv:2501\.12735\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p2.1)\.
- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§2](https://arxiv.org/html/2605.26442#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p2.1),[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p5.1)\.
- A\. Bose, Z\. Xiong, A\. Saha, S\. S\. Du, and M\. Fazel \(2025\)Hybrid preference optimization for alignment: provably faster convergence rates by combining offline preferences with online exploration\.InICLRW,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1)\.
- K\. Brantley, M\. Chen, Z\. Gao, J\. D\. Lee, W\. Sun, W\. Zhan, and X\. Zhang \(2025\)Accelerating rl for llm reasoning with optimal advantage regression\.arXiv preprint arXiv:2505\.20686\.Cited by:[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, s\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- C\. Burges, R\. Ragno, and Q\. Le \(2006\)Learning to rank with nonsmooth cost functions\.InNeurIPS,Cited by:[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p2.1)\.
- S\. Casper, X\. Davies, C\. Shi, T\. K\. Gilbert, J\. Scheurer, J\. Rando, R\. Freedman, T\. Korbak, D\. Lindner, P\. Freire,et al\.\(2023\)Open problems and fundamental limitations of reinforcement learning from human feedback\.arXiv preprint arXiv:2307\.15217\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p9.1)\.
- J\. Chen, Y\. Lu, X\. Wang, H\. Zeng, J\. Huang, J\. Gesi, Y\. Xu, B\. Yao, and D\. Wang \(2025a\)Multi\-agent\-as\-judge: aligning llm\-agent\-based automated evaluation with multi\-dimensional human evaluation\.arXiv preprint arXiv:2507\.21028\.Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p4.1)\.
- J\. Chen, R\. Thirukovalluru, and \+\. others \(2025b\)Atomic consistency preference optimization for long\-form question answering\.arXiv preprint arXiv:2505\.09039\.Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p3.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.InICML,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1)\.
- H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.\(2024\)Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- J\. J\. Y\. Chung, V\. Padmakumar, M\. Roemmele, Y\. Sun, and M\. Kreminski \(2025\)Modifying large language model post\-training for diverse creative writing\.arXiv preprint arXiv:2503\.17126\.Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang \(2023\)Safe rlhf: safe reinforcement learning from human feedback\.arXiv preprint arXiv:2310\.12773\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p5.1),[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- X\. Deng, H\. Zhong, R\. Ai, F\. Feng, Z\. Wang, and X\. He \(2025\)Less is more: improving llm alignment via preference data selection\.arXiv preprint arXiv:2502\.14560\.Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p2.1)\.
- Y\. Dong, Z\. Wang, M\. Sreedhar, X\. Wu, and O\. Kuchaiev \(2023\)Steerlm: attribute conditioned sft as an \(user\-steerable\) alternative to rlhf\.InEMNLP,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p4.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2024\)Improving factuality and reasoning in language models through multiagent debate\.InICML,Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p4.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)Kto: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p6.1),[§1](https://arxiv.org/html/2605.26442#S1.p6.1),[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p3.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- R\. Gupta, R\. Sullivan, Y\. Li, S\. Phatale, and A\. Rastogi \(2025\)Robust multi\-objective preference alignment with online dpo\.InAAAI,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p6.1)\.
- Q\. He and S\. Maghsudi \(2025\)Pareto multi\-objective alignment for language models\.InECML\-PKDD,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p6.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)ORPO: monolithic preference optimization without reference model\.InEMNLP,Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p3.1)\.
- K\. Huang, J\. Wu, Z\. Chen, X\. Wang, J\. Gao, B\. Ding, J\. Wu, X\. He, and X\. Wang \(2025\)Larger or smaller reward margins to select preferences for llm alignment?\.InICML,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p2.1)\.
- M\. Ismayilzada, A\. Laverghetta Jr, S\. A\. Luchini, R\. Patel, A\. Bosselut, L\. van der Plas, and R\. Beaty \(2025\)Creative preference optimization\.arXiv preprint arXiv:2505\.14442\.Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- J\. Jang, S\. Kim, B\. Y\. Lin, Y\. Wang, J\. Hessel, L\. Zettlemoyer, H\. Hajishirzi, Y\. Choi, and P\. Ammanabrolu \(2023\)Personalized soups: personalized large language model alignment via post\-hoc parameter merging\.arXiv preprint arXiv:2310\.11564\.Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p6.1)\.
- J\. Ji, T\. Qiu, B\. Chen, B\. Zhang, H\. Lou, K\. Wang, Y\. Duan, Z\. He, J\. Zhou, Z\. Zhang,et al\.\(2023\)Ai alignment: a comprehensive survey\.arXiv preprint arXiv:2310\.19852\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.26442#S2.SS1.p1.7)\.
- S\. Jung, G\. Han, D\. W\. Nam, and K\. On \(2025\)Binary classifier optimization for large language model alignment\.InACL,Cited by:[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p3.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- S\. Khaki, J\. Li, L\. Ma, L\. Yang, and P\. Ramachandra \(2024\)RS\-dpo: a hybrid rejection sampling and direct preference optimization method for alignment of large language models\.InNAACL,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1)\.
- D\. Kim, Y\. Kim, W\. Song, H\. Kim, Y\. Kim, S\. Kim, and C\. Park \(2025\)Sdpo: don’t use your data all at once\.InCOLING,Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p2.2)\.
- K\. Kim, A\. Seo, H\. Liu, J\. Shin, and K\. Lee \(2024a\)Margin matching preference optimization: enhanced model alignment with granular feedback\.InEMNLP,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p2.1),[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p4.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024b\)Prometheus 2: an open source language model specialized in evaluating other language models\.InEMNLP,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p5.1)\.
- R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina, E\. Hambro, E\. Grefenstette, and R\. Raileanu \(2024\)Understanding the effects of rlhf on llm generalisation and diversity\.InICLR,Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- T\. Kocmi and C\. Federmann \(2023\)Large language models are state\-of\-the\-art evaluators of translation quality\.InEAMT,Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p1.1)\.
- J\. Lanchantin, A\. Chen, S\. Dhuliawala, P\. Yu, J\. Weston, S\. Sukhbaatar, and I\. Kulikov \(2025\)Diverse preference optimization\.arXiv preprint arXiv:2501\.18101\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p4.1),[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- H\. Le, Q\. H\. Tran, D\. Nguyen, K\. Do, S\. Mittal, K\. Ogueji, and S\. Venkatesh \(2025\)Multi\-reference preference optimization for large language models\.InAAAI,Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p2.2)\.
- H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. R\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi,et al\.\(2024\)RLAIF vs\. rlhf: scaling reinforcement learning from human feedback with ai feedback\.InICML,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p5.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p2.1)\.
- C\. Li, H\. Zhang, Y\. Xu, H\. Xue, X\. Ao, and Q\. He \(2025a\)Gradient\-adaptive policy optimization: towards multi\-objective alignment of large language models\.arXiv preprint arXiv:2507\.01915\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p5.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu,et al\.\(2025b\)From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.InEMNLP,Cited by:[§5](https://arxiv.org/html/2605.26442#S5.p1.1)\.
- H\. Li, Q\. Dong, J\. Chen, H\. Su, Y\. Zhou, Q\. Ai, Z\. Ye, and Y\. Liu \(2024\)Llms\-as\-judges: a comprehensive survey on llm\-based evaluation methods\.arXiv preprint arXiv:2412\.05579\.Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p1.1)\.
- L\. Li, H\. Liu, Y\. Zhou, Z\. Gui, X\. Weng, Y\. Yuan, Z\. Wei, and Z\. Li \(2025c\)Uncertainty\-aware iterative preference optimization for enhanced llm reasoning\.InACL,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p3.1)\.
- X\. Li, Z\. Li, Y\. Kosuga, and V\. Bian \(2025d\)Optimizing safe and aligned language generation: a multi\-objective grpo approach\.arXiv preprint arXiv:2503\.21819\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p2.1)\.
- Y\. Li, T\. Xu, Y\. Yu, X\. Zhang, X\. Chen, Z\. Ling, N\. Chao, L\. Yuan, and Z\. Zhou \(2025e\)Generalist reward models: found inside large language models\.arXiv preprint arXiv:2506\.23235\.Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- Z\. Li, T\. Xu, Y\. Zhang, Z\. Lin, Y\. Yu, R\. Sun, and Z\. Luo \(2023\)Remax: a simple, effective, and efficient reinforcement learning method for aligning large language models\.arXiv preprint arXiv:2310\.10505\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p5.1),[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- Y\. Lin, H\. Lin, W\. Xiong, S\. Diao, J\. Liu, J\. Zhang, R\. Pan, H\. Wang, W\. Hu, H\. Zhang,et al\.\(2024\)Mitigating the alignment tax of rlhf\.InEMNLP,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- A\. Liu, H\. Bai, Z\. Lu, Y\. Sun, X\. Kong, S\. Wang, J\. Shan, A\. M\. Jose, X\. Liu, L\. Wen,et al\.\(2025a\)Tis\-dpo: token\-level importance sampling for direct preference optimization with estimated weights\.InICLR,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1),[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p3.1)\.
- B\. Liu, C\. Jin, S\. Kim, W\. Yuan, W\. Zhao, I\. Kulikov, X\. Li, S\. Sukhbaatar, J\. Lanchantin, and J\. Weston \(2025b\)Spice: self\-play in corpus environments improves reasoning\.arXiv preprint arXiv:2510\.24684\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p2.1)\.
- Q\. Liu, X\. Zheng, N\. Muennighoff, G\. Zeng, L\. Dou, T\. Pang, J\. Jiang, and M\. Lin \(2025c\)RegMix: data mixture as regression for language model pre\-training\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- S\. Liu, X\. Shen, Y\. Lai, S\. Wang, S\. Yue, Z\. Huang, X\. Huang, and Z\. Wei \(2025d\)Haf\-rm: a hybrid alignment framework for reward model training\.InACL\),Cited by:[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- S\. Liu, W\. Fang, Z\. Hu, J\. Zhang, Y\. Zhou, K\. Zhang, R\. Tu, T\. Lin, F\. Huang, M\. Song,et al\.\(2025e\)A survey of direct preference optimization\.arXiv preprint arXiv:2503\.11701\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p9.1)\.
- T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. Wang \(2025f\)OpenRubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.arXiv preprint arXiv:2510\.07743\.Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p5.1)\.
- T\. Liu, Z\. Qin, J\. Wu, J\. Shen, M\. Khalman, R\. Joshi, Y\. Zhao, M\. Saleh, S\. Baumgartner, J\. Liu,et al\.\(2025g\)Lipo: listwise preference optimization through learning\-to\-rank\.InNAACL,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p6.1),[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p2.1)\.
- Y\. Liu, J\. Cao, C\. Liu, K\. Ding, and L\. Jin \(2025h\)Datasets for large language models: a comprehensive survey\.Artificial Intelligence Review58\(12\),pp\. 403\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InEMNLP,Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p1.1)\.
- Y\. Liu, S\. Tao, X\. Zhao, M\. Zhu, W\. Ma, J\. Zhu, C\. Su, Y\. Hou, M\. Zhang, M\. Zhang,et al\.\(2024\)Coachlm: automatic instruction revisions improve the data quality in llm instruction tuning\.InICDE,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025i\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p7.1),[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p3.1)\.
- X\. Lou, J\. Zhang, J\. Xie, L\. Liu, D\. Yan, and K\. Huang \(2024\)Spo: multi\-dimensional preference sequential alignment with implicit reward modeling\.arXiv preprint arXiv:2405\.12739\.Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p6.1)\.
- L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, L\. Meng, J\. Sun,et al\.\(2025\)Improve mathematical reasoning in language models with automated process supervision\.InICLR,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- S\. Mahmud, M\. Nakamura, and S\. Zilberstein \(2025\)Maple: a framework for active preference learning guided by large language models\.InAAAI,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p3.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)Simpo: simple preference optimization with a reference\-free reward\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p6.1),[§1](https://arxiv.org/html/2605.26442#S1.p6.1),[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p3.1)\.
- H\. Min, Y\. Lee, M\. Ban, J\. Deng, N\. H\. Kim, T\. Yun, H\. Su, J\. Cai, and H\. Song \(2025\)Towards multi\-dimensional evaluation of llm summarization across domains and languages\.InACL,Cited by:[§5](https://arxiv.org/html/2605.26442#S5.p1.1)\.
- S\. Minaee, T\. Mikolov, N\. Nikzad, M\. Chenaghlu, R\. Socher, X\. Amatriain, and J\. Gao \(2024\)Large language models: a survey\.arXiv preprint arXiv:2402\.06196\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- S\. Mukherjee, A\. Lalitha, S\. Sengupta, A\. Deshmukh, and B\. Kveton \(2024\)Multi\-objective alignment of large language models through hypervolume maximization\.arXiv preprint arXiv:2412\.05469\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p5.1)\.
- W\. Muldrew, P\. Hayes, M\. Zhang, and D\. Barber \(2024\)Active preference learning for large language models\.InICML,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p3.1)\.
- S\. K\. Murthy, T\. Ullman, and J\. Hu \(2025\)One fish, two fish, but not the whole sea: alignment reduces language models’ conceptual diversity\.InNAACL,Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- W\. Nazar, G\. Nazar, A\. Kamińska, and L\. Danilowicz\-Szymanowicz \(2025\)How to design, create, and evaluate an instruction\-tuning dataset for large language model training in health care: tutorial from a clinical perspective\.Journal of Medical Internet Research27\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2605.26442#S2.p1.1),[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- A\. Pan, E\. Jones, M\. Jagadeesan, and J\. Steinhardt \(2024\)Feedback loops with language models drive in\-context reward hacking\.InICML,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p2.4),[§1](https://arxiv.org/html/2605.26442#S1.p2.1),[§1](https://arxiv.org/html/2605.26442#S1.p4.1),[§1](https://arxiv.org/html/2605.26442#S1.p6.1),[§2\.2](https://arxiv.org/html/2605.26442#S2.SS2.p3.3),[§3\.1](https://arxiv.org/html/2605.26442#S3.SS1.p3.2),[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p2.2)\.
- S\. S\. Ramesh, Y\. Hu, I\. Chaimalas, V\. Mehta, P\. G\. Sessa, H\. Bou Ammar, and I\. Bogunovic \(2024\)Group robust preference optimization in reward\-free rlhf\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p6.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p1.4),[§2\.2](https://arxiv.org/html/2605.26442#S2.SS2.p2.3),[§3\.1](https://arxiv.org/html/2605.26442#S3.SS1.p3.2),[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p3.2),[§2\.2](https://arxiv.org/html/2605.26442#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2605.26442#S3.SS1.p3.2),[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1),[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p3.1)\.
- F\. Song, B\. Yu, M\. Li, H\. Yu, F\. Huang, Y\. Li, and H\. Wang \(2024\)Preference ranking optimization for human alignment\.InAAAI,Cited by:[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p2.1)\.
- H\. Song, T\. Yun, Y\. Lee, J\. Oh, G\. Lee, J\. Cai, and H\. Su \(2025\)Learning to summarize from LLM\-generated feedback\.InNAACL,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- T\. Sorensen, B\. Newman, J\. Moore, C\. Park, J\. Fisher, N\. Mireshghallah, L\. Jiang, and Y\. Choi \(2025\)Spectrum tuning: post\-training for distributional coverage and in\-context steerability\.Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2605.26442#S2.p1.1),[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- H\. Sun, Z\. Zhang, and S\. Zeng \(2025\)An uncertainty\-driven adaptive self\-alignment framework for large language models\.arXiv preprint arXiv:2507\.17477\.Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p3.1)\.
- Z\. Sun, Y\. Shen, Q\. Zhou, H\. Zhang, Z\. Chen, D\. Cox, Y\. Yang, and C\. Gan \(2023\)Principle\-driven self\-alignment of language models from scratch with minimal human supervision\.InNeurIPS,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p5.1)\.
- Y\. Tan, Y\. Jiang, Y\. Li, J\. Liu, X\. Bu, W\. Su, X\. Yue, X\. Zhu, and B\. Zheng \(2025\)Equilibrate rlhf: towards balancing helpfulness\-safety trade\-off in large language models\.arXiv preprint arXiv:2502\.11555\.Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p2.1)\.
- Y\. Tang, D\. Z\. Guo, Z\. Zheng, D\. Calandriello, R\. Munos, M\. Rowland, P\. H\. Richemond, M\. Valko, B\. Á\. Pires, and B\. Piot \(2024\)Generalized preference optimization: a unified approach to offline alignment\.InICML,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p9.1),[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p2.2)\.
- T\. Tripathi, M\. Wadhwa, G\. Durrett, and S\. Niekum \(2025\)Pairwise or pointwise? evaluating feedback protocols for bias in llm\-based evaluation\.arXiv preprint arXiv:2504\.14716\.Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p1.2)\.
- J\. Wang, Y\. Zhou, X\. Zhang, M\. Bao, and P\. Yan \(2025a\)Self\-evolutionary large language models through uncertainty\-enhanced preference optimization\.InAAAI,Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p3.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu,et al\.\(2024a\)Large language models are not fair evaluators\.InACL\),Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p3.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024b\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InACL,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- S\. Wang, Z\. Zhang, R\. Zhao, F\. Tan, and N\. Cam\-Tu \(2024c\)Reward difference optimization for sample reweighting in offline rlhf\.InEMNLP,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1)\.
- Y\. Wang, D\. Gao, R\. Long, L\. Yi, L\. Jin, L\. Yang, and X\. Cai \(2025b\)ASPO: adaptive sentence\-level preference optimization for fine\-grained multimodal reasoning\.arXiv preprint arXiv:2505\.19100\.Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p3.1)\.
- Z\. Wang, Y\. Dong, O\. Delalleau, J\. Zeng, G\. Shen, D\. Egert, J\. Zhang, M\. N\. Sreedhar, and O\. Kuchaiev \(2024d\)Helpsteer 2: open\-source dataset for training top\-performing reward models\.InNeurIPS,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p4.1)\.
- Z\. Wang, Y\. Dong, J\. Zeng, V\. Adams, M\. N\. Sreedhar, D\. Egert, O\. Delalleau, J\. Scowcroft, N\. Kant, A\. Swope,et al\.\(2024e\)Helpsteer: multi\-attribute helpfulness dataset for steerlm\.InNAACL,Cited by:[§5\.3](https://arxiv.org/html/2605.26442#S5.SS3.p4.1)\.
- Z\. Wang, J\. Zeng, O\. Delalleau, E\. Evans, D\. Egert, H\. Shin, F\. Soares, Y\. Dong, and O\. Kuchaiev \(2025c\)RLBFF: binary flexible feedback to bridge between human feedback and verifiable rewards\.arXiv preprint arXiv:2509\.21319\.Cited by:[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p3.1)\.
- Z\. Wang, W\. Zhong, Y\. Wang, Q\. Zhu, F\. Mi, B\. Wang, L\. Shang, X\. Jiang, and Q\. Liu \(2023\)Data management for training large language models: a survey\.arXiv preprint arXiv:2312\.01700\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- K\. Wataoka, T\. Takahashi, and R\. Ri \(2024\)Self\-preference bias in llm\-as\-a\-judge\.arXiv preprint arXiv:2410\.21819\.Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p3.1)\.
- J\. Wu, X\. Wang, Z\. Yang, J\. Wu, J\. Gao, B\. Ding, X\. Wang, and X\. He \(2025a\)AlphaDPO: adaptive reward margin for direct preference optimization\.InICML,Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p4.1)\.
- X\. Wu, S\. Huang, W\. Wang, S\. Ma, L\. Dong, and F\. Wei \(2024\)Multi\-head mixture\-of\-experts\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1)\.
- Y\. Wu, Z\. Sun, H\. Yuan, K\. Ji, Y\. Yang, and Q\. Gu \(2025b\)Self\-play preference optimization for language model alignment\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p4.1),[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1)\.
- Z\. Wu, Y\. Hu, W\. Shi, N\. Dziri, A\. Suhr, P\. Ammanabrolu, N\. A\. Smith, M\. Ostendorf, and H\. Hajishirzi \(2023\)Fine\-grained human feedback gives better rewards for language model training\.InNeurIPS,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p3.1)\.
- W\. Xiao, Z\. Wang, L\. Gan, S\. Zhao, Z\. Li, R\. Lei, W\. He, L\. A\. Tuan, L\. Chen, H\. Jiang,et al\.\(2024\)A comprehensive survey of direct preference optimization: datasets, theories, variants, and applications\.arXiv preprint arXiv:2410\.15595\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p9.1)\.
- W\. Xiong, H\. Dong, C\. Ye, Z\. Wang, H\. Zhong, H\. Ji, N\. Jiang, and T\. Zhang \(2024\)Iterative preference learning from human feedback: bridging theory and practice for rlhf under kl\-constraint\.InICML,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1)\.
- X\. Xu, Z\. Wu, R\. Qiao, A\. Verma, Y\. Shu, J\. Wang, X\. Niu, Z\. He, J\. Chen, Z\. Zhou,et al\.\(2024\)Data\-centric ai in the age of large language models\.arXiv preprint arXiv:2406\.14473\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- Z\. Yan, J\. Wang, J\. Chen, X\. Li, J\. Liang, R\. Li, and J\. Z\. Pan \(2025\)Atomic fact decomposition helps attributed question answering\.IEEE Transactions on Knowledge and Data Engineering\.Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p1.1)\.
- Z\. Yang, F\. Wan, L\. Zhong, T\. Shi, and X\. Quan \(2025\)Weighted\-reward preference optimization for implicit model fusion\.InICLR,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1)\.
- D\. Yao, J\. Yuan, and R\. Chen \(2025\)Enhancing small llm alignment through margin\-based objective modifications under resource constraints\.arXiv preprint arXiv:2508\.08466\.Cited by:[§4\.2](https://arxiv.org/html/2605.26442#S4.SS2.p2.1)\.
- M\. Yeh, J\. Wang, X\. Du, S\. Park, L\. Tao, S\. Im, and Y\. Li \(2025\)Position: challenges and future directions of data\-centric ai alignment\.InICML,Cited by:[§2\.1](https://arxiv.org/html/2605.26442#S2.SS1.p1.7)\.
- P\. Yu, J\. Lanchantin, T\. Wang, W\. Yuan, O\. Golovneva, I\. Kulikov, S\. Sukhbaatar, J\. Weston, and J\. Xu \(2025a\)Cot\-self\-instruct: building high\-quality synthetic prompts for reasoning and non\-reasoning tasks\.arXiv preprint arXiv:2507\.23751\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p2.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025b\)Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[Appendix A](https://arxiv.org/html/2605.26442#A1.p7.1),[§1](https://arxiv.org/html/2605.26442#S1.p1.1),[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p3.1)\.
- T\. Yu, T\. Lin, Y\. Wu, M\. Yang, F\. Huang, and Y\. Li \(2025c\)Diverse ai feedback for large language model alignment\.Transactions of the Association for Computational Linguistics13,pp\. 392–407\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p4.1),[§1](https://arxiv.org/html/2605.26442#S1.p5.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p4.1)\.
- Y\. Yu, Z\. Chen, A\. Zhang, L\. Tan, C\. Zhu, R\. Y\. Pang, Y\. Qian, X\. Wang, S\. Gururangan, C\. Zhang,et al\.\(2025d\)Self\-generated critiques boost reward modeling for language models\.InNAACL,Cited by:[§6\.1](https://arxiv.org/html/2605.26442#S6.SS1.p2.1)\.
- H\. Yuan, Z\. Yuan, C\. Tan, W\. Wang, S\. Huang, and F\. Huang \(2023\)Rrhf: rank responses to align language models with human feedback\.InEMNLP,Cited by:[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p2.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. Weston \(2024\)Self\-rewarding language models\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p6.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p2.1)\.
- T\. Zeng, S\. Zhang, S\. Wu, C\. Classen, D\. Chae, E\. Ewer, M\. Lee, H\. Kim, W\. Kang, J\. Kunde, Y\. Fan, J\. Kim, H\. I\. Koo, K\. Ramchandran, D\. Papailiopoulos, and K\. Lee \(2025\)VersaPRM: multi\-domain process reward model via synthetic reasoning data\.InICML,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- Y\. Zeng, G\. Liu, W\. Ma, N\. Yang, H\. Zhang, and J\. Wang \(2024\)Token\-level direct preference optimization\.InICML,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p3.1)\.
- J\. Zhang, S\. Yu, D\. Chong, A\. Sicilia, M\. R\. Tomz, C\. D\. Manning, and W\. Shi \(2025a\)Verbalized sampling: how to mitigate mode collapse and unlock llm diversity\.arXiv preprint arXiv:2510\.01171\.Cited by:[§4\.3](https://arxiv.org/html/2605.26442#S4.SS3.p1.1)\.
- L\. Zhang, A\. Hosseini, H\. Bansal, M\. Kazemi, A\. Kumar, and R\. Agarwal \(2025b\)Generative verifiers: reward modeling as next\-token prediction\.InICLR,Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- S\. Zhang, D\. Yu, H\. Sharma, H\. Zhong, Z\. Liu, Z\. Yang, S\. Wang, H\. H\. Awadalla, and Z\. Wang \(2025c\)Self\-exploring language models: active preference elicitation for online alignment\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p4.1),[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p3.1)\.
- Y\. Zhao, Y\. Wang, and M\. Yin \(2025\)Permutative preference alignment from listwise ranking of human judgments\.InEMNLP,Cited by:[§6\.3](https://arxiv.org/html/2605.26442#S6.SS3.p2.1)\.
- Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu \(2023\)Slic\-hf: sequence likelihood calibration with human feedback\.arXiv preprint arXiv:2305\.10425\.Cited by:[§6\.2](https://arxiv.org/html/2605.26442#S6.SS2.p4.1)\.
- C\. Zheng, J\. Zhu, Z\. Ou, Y\. Chen, K\. Zhang, R\. Shan, Z\. Zheng, M\. Yang, J\. Lin, Y\. Yu,et al\.\(2025\)A survey of process reward models: from outcome signals to process supervisions for large language models\.arXiv preprint arXiv:2510\.08049\.Cited by:[§5\.2](https://arxiv.org/html/2605.26442#S5.SS2.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InNeurIPS,Cited by:[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p3.1)\.
- W\. Zhou, R\. Agrawal, S\. Zhang, S\. R\. Indurthi, S\. Zhao, K\. Song, S\. Xu, and C\. Zhu \(2024\)WPO: enhancing rlhf with weighted preference optimization\.InEMNLP,Cited by:[§4\.1](https://arxiv.org/html/2605.26442#S4.SS1.p2.1)\.
- X\. Zhou, J\. He, W\. Zhou, H\. Chen, Z\. Tang, H\. Zhao, X\. Tong, G\. Li, Y\. Chen, J\. Zhou,et al\.\(2025\)A survey of llm x data\.arXiv preprint arXiv:2505\.18458\.Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p8.1)\.
- X\. Zhuang, J\. Peng, R\. Ma, Y\. Wang, T\. Bai, X\. Wei, Q\. Jiantao, C\. Zhang, Y\. Qian, and C\. He \(2025\)Meta\-rater: a multi\-dimensional data selection method for pre\-training language models\.InACL,Cited by:[§1](https://arxiv.org/html/2605.26442#S1.p1.1),[§5\.1](https://arxiv.org/html/2605.26442#S5.SS1.p4.1)\.

Table 5:Categorization of alignment tuning methods across three data pipeline stages\.## Appendix AAlignment Tuning Algorithms

*Proximal Policy Optimization \(PPO\)*is a representative approach for explicitly maximizing a learned reward under the KL\-regularized objective in Eq\. \([1](https://arxiv.org/html/2605.26442#S2.E1)\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26442#bib.bib48)\)\. It proceeds in two stages\. First, a reward modelrϕ​\(x,y\)r\_\{\\phi\}\(x,y\)is trained from pairwise comparisons to approximate oracle preferencesr∗​\(x,y\)r^\{\*\}\(x,y\)\. Second, the policyπθ\\pi\_\{\\theta\}is optimized with reinforcement learning to maximize the learned reward while staying close to a reference policyπref\\pi\_\{\\mathrm\{ref\}\}:

maxπθ⁡𝔼x,y​\[rϕ​\(x,y\)−β​log⁡πθ​\(y∣x\)πref​\(y∣x\)\]\.\\max\_\{\\pi\_\{\\theta\}\}\\;\\mathbb\{E\}\_\{x,y\}\\\!\\left\[r\_\{\\phi\}\(x,y\)\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\\right\]\.In practice, PPO performs stable policy updates using a clipped surrogate objective,

ℒPPO\(θ\)=𝔼x,y\[min\(ρθ\(x,y\)A^\(x,y\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{x,y\}\\\!\\Big\[\\min\\\!\\Big\(\\rho\_\{\\theta\}\(x,y\)\\,\\hat\{A\}\(x,y\),clip\(ρθ\(x,y\),1−ϵ,1\+ϵ\)A^\(x,y\)\)\]\.\\displaystyle\\qquad\\qquad\\mathrm\{clip\}\(\\rho\_\{\\theta\}\(x,y\),1\-\\epsilon,1\+\\epsilon\)\\,\\hat\{A\}\(x,y\)\\Big\)\\Big\]\.whereρθ​\(x,y\)=πθ​\(y∣x\)πθold​\(y∣x\)\\rho\_\{\\theta\}\(x,y\)=\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\\mid x\)\}is the likelihood ratio andA^​\(x,y\)\\hat\{A\}\(x,y\)is an advantage estimate constructed fromrϕ​\(x,y\)r\_\{\\phi\}\(x,y\)\(typically with a value baseline\)\. The clipping term constrains the effective step size, preventing large deviations from the reference distribution and improving training stability\.

*Direct Preference Optimization \(DPO\)*bypasses explicit reward modeling by directly optimizing the KL\-regularized objective in Eq\. \([1](https://arxiv.org/html/2605.26442#S2.E1)\) using preference pairs\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib4)\)\. Given a dataset of comparisons\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\)whereyw≻yly\_\{w\}\\succ y\_\{l\}, DPO yields a classification\-style objective that increases the relative likelihood of preferred responses underπθ\\pi\_\{\\theta\}compared toπref\\pi\_\{\\mathrm\{ref\}\}:

ℒDPO​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(\\theta\)=−𝔼\(x,yw,yl\)\[logσ\(β\(logπθ​\(yw∣x\)πref​\(yw∣x\)\\displaystyle=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\Big\[\\log\\sigma\\\!\\Big\(\\beta\\Big\(\\log\\tfrac\{\\pi\_\{\\theta\}\(y\_\{w\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\\mid x\)\}\(4\)−logπθ​\(yl∣x\)πref​\(yl∣x\)\)\)\]\.\\displaystyle\\qquad\\qquad\\qquad\-\\log\\tfrac\{\\pi\_\{\\theta\}\(y\_\{l\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\\mid x\)\}\\Big\)\\Big\)\\Big\]\.whereσ​\(⋅\)\\sigma\(\\cdot\)is the logistic sigmoid\. This objective can be interpreted as implicitly fitting a reward via the log\-likelihood ratiolog⁡πθ​\(y∣x\)πref​\(y∣x\)\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}, thereby achieving preference optimization without training a separate reward model\.

*Group Relative Policy Optimization \(GRPO\)*shifts from pairwise comparisons to group\-wise optimization by normalizing candidates sampled from the current policy\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib52)\)\. For each promptxx, GRPO samples a set of responses\{yi\}i=1k∼πθ\(⋅∣x\)\\\{y\_\{i\}\\\}\_\{i=1\}^\{k\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)and computes a group\-relative baseline \(*e\.g\.*, the mean score\):

r¯​\(x\)=1k​∑i=1kr​\(x,yi\),\\bar\{r\}\(x\)=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}r\(x,y\_\{i\}\),wherer​\(x,y\)r\(x,y\)is a scoring function \(often derived from verification\) used to rank candidates\. The policy is then updated to increase the probability of responses that outperform the group baseline:

maxπθ⁡𝔼x​𝔼y∼πθ\(⋅∣x\)​\[\(r​\(x,y\)−r¯​\(x\)\)​log⁡πθ​\(y∣x\)\]\.\\max\_\{\\pi\_\{\\theta\}\}\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\bigl\(r\(x,y\)\-\\bar\{r\}\(x\)\\bigr\)\\,\\log\\pi\_\{\\theta\}\(y\\mid x\)\\right\]\.By using group\-relative normalization, GRPO reduces variance and avoids the need for an explicit critic, value networks\.

*Other Algorithms\.*Building on these foundational approaches, the alignment literature has expanded into several related families that explore alternative design choices and practical trade\-offs\.

Methods in the PPO lineage continue to emphasize training stability and sample efficiency under explicit reward maximization, with extensions such as Safe\-RLHF\(Daiet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib53)\)incorporating safety\-aware constraints and ReMax\(Liet al\.,[2023](https://arxiv.org/html/2605.26442#bib.bib54)\)removing the need for an explicit critic\.

The DPO family investigates simplifications of preference optimization under different supervision assumptions: SimPO\(Menget al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib32)\)eliminates the reference policy to reduce computational overhead, while KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2605.26442#bib.bib26)\)relaxes the dependence on paired preference data by leveraging binary feedback\.

Finally, the GRPO family generalizes preference optimization to group\-wise settings\. DAPO\(Yuet al\.,[2025b](https://arxiv.org/html/2605.26442#bib.bib5)\)mitigates entropy collapse through decoupled clipping, whereas Dr\. GRPO\(Liuet al\.,[2025i](https://arxiv.org/html/2605.26442#bib.bib56)\)improves computational efficiency by removing biased normalization terms\.

Table 6:Recommended alignment pipeline configurations based on resource and task constraints\. We map common deployment scenarios to the most effective combination of pipeline components \(response synthesis, evaluation, and instantiation\) as discussed in the main text\.
## Appendix BPractical Guidelines for Alignment Pipeline Design

While the main body of this survey systematizes the design space of alignment tuning along response synthesis, preference evaluation, and instantiation, practitioners often face a more immediate question: ”which pipeline configuration should be selected under concrete operational constraints?” To bridge this gap between conceptual taxonomy and real\-world deployment, we provide practical guidelines for alignment pipeline design based on three governing factors: data availability, task complexity, and computational budget\.

Table[6](https://arxiv.org/html/2605.26442#A1.T6)summarizes recommended pipeline configurations for common deployment scenarios\. This section explicates the rationale behind each recommendation, clarifying how different pipeline components should be combined in practice\.

### B\.1Decision Factors

Alignment pipeline design is primarily constrained by three factors: data availability, task complexity, and computational budget\. These factors directly shape feasible choices across response synthesis, preference evaluation, and instantiation\.

##### Data Availability\.

The availability and reliability of supervision signals constrain pipeline design\. When large\-scale, high\-quality preference data exists, offline alignment using pre\-collected datasets is often sufficient and cost\-effective\. In contrast, data\-poor or cold\-start settings require on\-policy data generation and iterative self\-improvement, as the model must construct its own supervision signals to overcome distribution shift\.

##### Task Complexity\.

Task structure determines the appropriate evaluation granularity\. Tasks with explicit correctness criteria, such as mathematics, coding, or formal logic, benefit from trajectory\-level or verifiable supervision that localizes errors within intermediate steps\. Conversely, open\-ended tasks lack objective ground\-truth and require comparative judgments that preserve diversity and relative quality rather than absolute correctness\.

##### Computational Budget\.

Compute and memory constraints strongly influence feasible pipeline components\. Under limited budgets, pipelines that avoid auxiliary models and online sampling are preferred\. When sufficient compute is available, richer evaluation signals and multi\-rollout sampling can significantly improve alignment stability and sample efficiency\.

### B\.2Scenario\-Specific Design Rationale

We next explain the design rationale behind each scenario listed in Table[1](https://arxiv.org/html/2605.26442#S7.T1), clarifying why particular combinations of synthesis, evaluation, and instantiation are well suited to each setting\.

##### Resource\-Constrained Setup\.

In low\-budget general chat settings, offline response synthesis using existing off\-policy datasets minimizes data collection cost while maintaining reasonable coverage of user behaviors\. Heuristic or pre\-annotated preference evaluation avoids repeated labeling, while reference\-free pair\-wise instantiation removes the need for a separate reward or reference model\. This configuration maximizes memory efficiency and supports stable alignment under strict constraints\.

##### Complex Reasoning Tasks\.

For tasks involving structured reasoning, multiple rollouts per prompt are necessary to expose alternative solution trajectories\. Trajectory\-level or verifier\-based evaluation, such as compilers or ground\-truth checkers, provides precise supervision by identifying where reasoning succeeds or fails\. Group\-wise instantiation normalizes rewards within a cohort of candidates, reducing variance and stabilizing optimization as reasoning difficulty varies across samples\.

##### Open\-Ended Creative Generation\.

Creative tasks such as storytelling or roleplay lack objective correctness and are sensitive to mode collapse\. High\-temperature or diversity\-constrained synthesis encourages stylistic variation, while LLM\-as\-a\-Judge evaluation provides relative quality judgments beyond binary acceptability\. List\-wise or pair\-wise instantiation preserves nuanced preferences among candidates, maintaining diversity more effectively than scalar objectives\.

##### Strict Compliance and Safety Alignment\.

Safety\-critical deployment requires disentangling competing objectives such as helpfulness and harmlessness\. Targeted offline synthesis using red\-teaming prompts focuses supervision on failure modes\. Multi\-criteria evaluation separates safety and utility signals, while multi\-objective instantiation enforces explicit safety boundaries that scalar objectives often fail to maintain\.

##### Data\-Poor and Cold\-Start Adaptation\.

In new domains where external supervision is scarce, iterative online alignment becomes essential\. Self\-play from the current policy generates on\-policy data, which is labeled using a general\-purpose strong LLM as a judge\. Iterative pair\-wise instantiation allows the model to repeatedly learn from its own highest\-quality outputs, gradually bridging distribution shift and accelerating adaptation\.

### B\.3Takeaway

Across scenarios, the key insight is that effective alignment is not determined by a single algorithmic choice, but by the coherent co\-design of synthesis, evaluation, and instantiation under practical constraints\. Table[1](https://arxiv.org/html/2605.26442#S7.T1)should therefore be read not as a prescriptive checklist, but as a set of principled templates that can be adapted to the specific operational realities of a given deployment\. While these principles are not universally optimal in all settings, they provide a coarse yet informative abstraction of recurring design trade\-offs observed across alignment pipelines\.