Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

# Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
Source: [https://arxiv.org/html/2610.31862](https://arxiv.org/html/2610.31862)
Luca VianoAffiliation:EPFLChenliang LiAffiliation:Texas A&M UniversityJiaxiang LiAffiliation:University of MinnesotaVolkan CevherAffiliation:EPFLMarkus WulfmeierAffiliation:Google DeepMindStefano ErmonAffiliation:Stanford UniversityAlfredo GarciaAffiliation:Texas A&M UniversityMingyi HongNote:This work has been accepted for publication in IEEE Signal Processing Magazine\. This arXiv version includes Volkan Cevher as a coauthor in recognition of his substantial contributions\. The IEEE Signal Processing Magazine special issue limits individual authors to appearing on at most two articles within the same issue, which determined the authorship listing in the journal version\.Affiliation:University of Minnesota

###### Abstract

This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models \(LLMs\) with human intentions, preferences, and values\. We discuss standard alignment techniques, such as fine\-tuning \(SFT\), reinforcement learning with human feedback \(RLHF\), and direct preference optimization \(DPO\)\. We also explore the theoretical underpinnings of learning from human preferences, drawing connections to inverse reinforcement learning \(IRL\) and discrete choice models\. We present state\-of\-the\-art algorithms in a tutorial style, discuss their advantages and limitations, and offer insights into practical implementation\. Our exposition is intended to serve as a comprehensive resource for researchers and practitioners, providing both a foundational understanding of alignment methodologies and a framework for developing more robust and scalable alignment techniques\.

## 1Introduction

Background\.As large language models \(LLMs\) have taken the world by storm, it is clear that generative AI systems will soon become ubiquitous in our lives\. LLMs have been applied beyond chatbots and personal assistants to tackle complex challenges, including video gaming\[[8](https://arxiv.org/html/2610.31862#bib.bib32)\]and autonomous control\[[7](https://arxiv.org/html/2610.31862#bib.bib33)\]\. After the successful introduction and demonstrated efficacy of alignment training in LLMs\[[5](https://arxiv.org/html/2610.31862#bib.bib25),[38](https://arxiv.org/html/2610.31862#bib.bib51)\], the concept ofalignmenthas been playing an increasingly important role in the design and training of AI systems\. Loosely speaking, alignment refers to the performance guarantee that the AI system will generate outcomes that are intended or preferred by the human user without undesirable side effects or behaviors such as deception or manipulation\.

More technically, the LLM alignment problem involves fine\-tuning or adjusting a base model that was originally trained on extensive, diverse datasets\. While such a base model is trained to learn core behaviors and general knowledge, it often falls short when applied to specific tasks or to meeting the nuanced preferences of a human user\. To betteralignthe model, additional feedback is gathered from human experts who evaluate and/or rank the model’s performance within a specific task context\. As demonstrated in\[[38](https://arxiv.org/html/2610.31862#bib.bib51),[19](https://arxiv.org/html/2610.31862#bib.bib8)\], reinforcement learning \(RL\) holds significant promise for LLM alignment by leveraging additional feedback to enhance their performance\. This feedback embodies expert judgments regarding both the accuracy of the model’s responses and any particular preferences they may have\. The goal is to refine the model to better meet these specific requirements, ultimately enhancing its task relevance and user satisfaction\. Generally, it is observed that after the alignment process, LLMs can follow human instructions well, and can avoid providing toxic and non\-preferable responses\. Please refer to Fig\.[1](https://arxiv.org/html/2610.31862#S1.F1)as an illustration of the alignment process\.

Contributions of This Work\.Despite extensive study of the alignment problem in recent literature, this research area remains young, and it is still evolving rapidly with many unresolved theoretical challenges\. A primary contribution of this article is to introduce the signal processing community to the fundamentals of alignment, highlight state\-of\-the\-art algorithms and approaches, and clarify the intersection

![Refer to caption](https://arxiv.org/html/2610.31862v1/MSF-Demonstration.png)Figure 1:An illustration of the alignment process\.between alignment methodologies and various techniques relevant to signal processing \(SP\)\. We believe that our article will foster interdisciplinary contributions and expand the role of SP in LLM alignment research\. By understanding the alignment problem, SP researchers can leverage their expertise in mathematical modeling, noise handling, inverse problem\-solving, and optimization to address a number of key challenges\. For instance, similar to reconstructing signals from observed data in SP, many alignment approaches involve solving inverse problems, such as learning the human preference models from the data\. Therefore, we expect that introducing the alignment to SP professionals can foster innovation in AI safety, inspire novel techniques, and accelerate the development of robust, generalizable alignment frameworks that can be applied across diverse applications\.

Organization of this work\.The article is organized as follows\. In Sec\.[2](https://arxiv.org/html/2610.31862#S2), we provide the readers with an overview of the LLM training process\. In Sec\.[3](https://arxiv.org/html/2610.31862#S3), we discuss some classical notion and techniques of learning from human preference, which serves as the foundation for many alignment approaches to be discussed subsequently\. In Sec\.[4](https://arxiv.org/html/2610.31862#S4), we provide a detailed discussion of a number of state\-of\-the\-art algorithms for different stages of LLM alignment\. In Sec\.[5](https://arxiv.org/html/2610.31862#S5), we provide readers with a framework that unify a few alignment algorithms discussed in the previous section under a single formulation\. In Sec\.[6](https://arxiv.org/html/2610.31862#S6), we provide some high\-level discussion\. In Sec\.[8](https://arxiv.org/html/2610.31862#S8), we conclude with final remarks and open questions\.

## 2Preliminaries

In this section, we provide a basic description of the LLM training process, while emphasizing the basic concepts relevant for alignment problems that we will describe in the sequel\.

LLM Representation\.Consider an LLM parameterized by parametersθ\\thetaand denote the output probability byπ⁡\(y\|x;θ\)\\pi\(y\|x;\\theta\)wherex:=\[x1,…,xn\]x:=\[x\_\{1\},\\ldots,x\_\{n\}\]is the sequence of input prompts andy:=\[y1,…,ym\]y:=\[y\_\{1\},\\ldots,y\_\{m\}\]is the sequence of output continuation, wherennandmmare their sequence length, correspondingly\. In practice, each one ofyiy\_\{i\}andxjx\_\{j\}is a token, which is one element of a token vocabulary\. A token is a discrete unit that represents certain semantic information\. For example, a word “learning” itself could be one token in the vocabulary, however more commonly two elements “learn” and “\-ing” are both tokens where “\-ing” represents the semantic information of tense\. A tokenizer is commonly utilized to transform an input natural language sentence into sequence of tokensx=\[x1,…,xn\]x=\[x\_\{1\},\\ldots,x\_\{n\}\]andy=\[y1,…,ym\]y=\[y\_\{1\},\\ldots,y\_\{m\}\]\. Given sequencesxxandyy, we use\|x\|\|x\|and\|y\|\|y\|to denote the length of the sequencexxandyy\.

Typical LLM is an auto\-regressive model, meaning that it predicts the output probability of theyjy\_\{j\}given all tokens inxxandy<j:=\[y1,…,yj−1\]y\_\{<j\}:=\[y\_\{1\},\\ldots,y\_\{j\-1\}\]\(y<1y\_\{<1\}is null\)\. More precisely, the

![Refer to caption](https://arxiv.org/html/2610.31862v1/autoregressive_model.png)Figure 2:An illustration of Auto\-regressive LLMs\. Here <BOS\> and <EOS\> are two special tokens representing the start and the end of the sentence, respectively\. The tokens at the bottom are the input token sequence, and those at the top are the output tokens\. In this example, the token sequence “<BOS\> All transformer models ” is given to the LLM and “are generative models <EOS\>” is the prediction\. Each of the previous tokens will be used to predict the next token, corresponding to \([1](https://arxiv.org/html/2610.31862#S2.E1)\)\.probability of producing a sequenceyycan be defined as:

π⁡\(y\|x;θ\)=∏j=1mπ⁡\(yj\|x,y<j;θ\)\.\\pi\(y\|x;\\theta\)=\\prod\_\{j=1\}^\{m\}\\pi\(y\_\{j\}\|x,y\_\{<j\};\\theta\)\.\(1\)The state\-of\-the\-art LLMs are built upon the attention mechanism\[[55](https://arxiv.org/html/2610.31862#bib.bib28)\]andθ\\thetarepresents all the trainable parameters, including the attention matrices and weights of the feed\-forward layers\.

Model Architecture\.Before going to the details, we briefly discuss the standard architectural design adopted by modern large language models\. Most LLMs are built upon the Transformer\[[55](https://arxiv.org/html/2610.31862#bib.bib28)\]architecture, which consists of a stack of self\-attention and feed\-forward \(MLP\)\[[46](https://arxiv.org/html/2610.31862#bib.bib66)\]layers interleaved with normalization\[[4](https://arxiv.org/html/2610.31862#bib.bib68)\]and residual connections\[[50](https://arxiv.org/html/2610.31862#bib.bib69)\]\. Each input tokenxix\_\{i\}is first embedded into a continuous representation and then processed through multiple attention heads, where self\-attention\[[55](https://arxiv.org/html/2610.31862#bib.bib28)\]computes contextualized token representations by weighting all previous tokens according to their learned relevance\. Formally, each attention head computesAttn​\(Q,K,V\)=softmax​\(Q​K⊤/dk\)​V\\text\{Attn\}\(Q,K,V\)=\\text\{softmax\}\(QK^\{\\top\}/\\sqrt\{d\_\{k\}\}\)V, whereQ,K,VQ,K,Vare linear projections of the hidden states\. The output of each attention block is then passed through the MLP layers or normalization layers such as LayerNorm\[[66](https://arxiv.org/html/2610.31862#bib.bib67)\], forming one Transformer block\. StackingLLsuch blocks yields a deep auto\-regressive model that predicts the next\-token probabilityπ⁡\(yj∣x,y<j;θ\)\\pi\(y\_\{j\}\\mid x,y\_\{<j\};\\theta\)as defined in \(1\)\. These components together define the function classπ⁡\(⋅,θ\)\\pi\(\\cdot;\\theta\)over which subsequent alignment and optimization problems are formulated\.

Next, we provide an overview of the different stages of LLM training\. Before delving into the alignment\-specific stages, it is worth noting that fine\-tuning is widely used in LLM training beyond alignment tasks\. Depending on the supervision signal, fine\-tuning can take the form of supervised fine\-tuning \(SFT\) or RL\-based optimization\. These approaches have been extensively applied to a variety of downstream objectives, including domain adaptation, instruction following\[[77](https://arxiv.org/html/2610.31862#bib.bib70)\], reasoning enhancement\[[57](https://arxiv.org/html/2610.31862#bib.bib22),[19](https://arxiv.org/html/2610.31862#bib.bib8)\], controllable generation\[[74](https://arxiv.org/html/2610.31862#bib.bib73)\], and agent training\[[41](https://arxiv.org/html/2610.31862#bib.bib71),[51](https://arxiv.org/html/2610.31862#bib.bib72)\]\. For completeness, we briefly review both stages below\.

The LLM pre\-training\.The first stage of the training is calledpre\-training, where an LLM model is initialized randomly and trained over a vast corpus of documents, known as the pre\-training dataset\. As an example, Llama 3 model is pre\-trained on more than 15T token of documents collected from all over the Internet, with the entire token vocabulary size \(i\.e\., the total number of distinct tokens\) of 128K\[[14](https://arxiv.org/html/2610.31862#bib.bib48)\]\. The pre\-training dataset is an unsupervised dataset, denoted as𝒟pre:=\{xi\}i=1Npre\\mathcal\{D\}\_\{\\text\{pre\}\}:=\\\{x^\{i\}\\\}\_\{i=1\}^\{N\_\{\\text\{pre\}\}\}, where each samplexi=\[x1i,…,xmii\]x^\{i\}=\[x^\{i\}\_\{1\},\\ldots,x^\{i\}\_\{m\_\{i\}\}\]is a sequence ofmim\_\{i\}tokens, andmim\_\{i\}denotes the length of the sequence\.𝒟\\mathcal\{D\}is a dataset without labels usually collected from the internet, such as Wikipedia articles; the learning task at the pre\-training stage is typically formulated as the so\-callednext\-token predictiontask, which predicts the next token given the current and all previous tokens shown to the LLM, and can be mathematically formulated as a negative log\-likelihood minimization problem:

minθ−1Npre∑i=1Nprelogπ\(xi;θ\)=−1Npre∑i=1Npre∑j=1milogπ\(xji\|x<ji;θ\)\.\\min\_\{\\theta\}\-\\frac\{1\}\{N\_\{\\text\{pre\}\}\}\\sum\_\{i=1\}^\{N\_\{\\text\{pre\}\}\}\\log\\pi\(x^\{i\};\\theta\)=\-\\frac\{1\}\{N\_\{\\text\{pre\}\}\}\\sum\_\{i=1\}^\{N\_\{\\text\{pre\}\}\}\\sum\_\{j=1\}^\{m\_\{i\}\}\\log\\pi\(x\_\{j\}^\{i\}\|x\_\{<j\}^\{i\};\\theta\)\.\(2\)This task is commonly understood as encouraging the LLM to memorize all the given texts and enabling the LLM for more fine\-grained tasks\. \([2](https://arxiv.org/html/2610.31862#S2.E2)\) is further solved by first order optimizers such as SGD, RMSProp\[[22](https://arxiv.org/html/2610.31862#bib.bib77)\]and AdamW\[[26](https://arxiv.org/html/2610.31862#bib.bib75),[32](https://arxiv.org/html/2610.31862#bib.bib76)\]\. In practice, AdamW is the default optimizer for pre\-training, fine\-tuning and aligning LLMs due to its efficiency and stability\. We use AdamW to solve all the alignment protocols in this work unless otherwise mentioned\.

The LLM Alignment\.Once the pre\-training is done, the next step is calledalignmentorfine\-tuning\. The fine\-tuning process is a supervised process that aims at improving the instruction following capabilities of LLMs to better align with human behaviors and values\. It usually consists of two main steps: the supervised fine\-tuning \(SFT\) step and the reinforcement learning with human feedback \(RLHF\) step\.

The SFT step utilizes a demonstration dataset𝒟demo:=\{\(xi,yi\)\}i=1Ndemo\\mathcal\{D\}\_\{\\text\{demo\}\}:=\\\{\(x^\{i\},y^\{i\}\)\\\}\_\{i=1\}^\{N\_\{\\text\{demo\}\}\}, wherexxis the input prompt \(such as, “Where is the capital of the US?”\) andyyis the expert response \(such as, “Washington, DC”\)\. We assume that the demonstration continuationsyyare collected from anexpert\(typically a human expert, but sometimes it can also be a very large and well\-aligned model such as GPT4 which produces high quality responses\), thus we also denote\(x,y\)∼𝒟demo\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\}asx∼ρ,y∼πE\(⋅\|x\)x\\sim\\rho,y\\sim\\pi^\{E\}\(\\cdot\|x\)as their population distributions,

![Refer to caption](https://arxiv.org/html/2610.31862v1/illustration_rlhf.png)Figure 3:An illustration of the standard LLM Alignment process\. The standard RLHF pipeline first trains a supervised model on demonstrations, then builds a reward model from preference data, and finally optimizes the policy with RL guided by the reward model\.whereρ\\rhois the distribution of the input prompts when collecting the data, andπE\\pi^\{E\}is the expert policy\.

Meanwhile the RLHF step first utilizes apreference datasetto learn from human preference about the quality of the answers to a particular questionxx\. This dataset is denoted as𝒟pref:=\{\(x,yw≻yl\)\}\\mathcal\{D\}\_\{\\text\{pref\}\}:=\\\{\(x,y\_\{w\}\\succ y\_\{l\}\)\\\}, whereywy\_\{w\}is preferred \(i\.e\., ‘wins’\) overyly\_\{l\}\(i\.e\., ‘loses’\) by a human labeler, and will be succinctly denoted as\(yw≻yl\)\(y\_\{w\}\\succ y\_\{l\}\)throughout the paper\. We use the notationx∼ρ,\(yw≻yl\)∼πP\(⋅\|x\)x\\sim\\rho,\(y\_\{w\}\\succ y\_\{l\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)to denote the population distribution of the preference dataset, whereπP\\pi^\{P\}is the preference distribution\. The preference data is typically used to learn a parameterizedreward functionr⁡\(x,y,ϕ\)r\(x,y;\\phi\)whereϕ\\phiis the reward parameters\. The reward model provides a score for a given prompt\-response pair\(x,y\)\(x,y\)\. Once such a reward function is learned, the RLHF step leverages this reward function, together with aprompt dataset𝒟prompt:=\{x\}\\mathcal\{D\}\_\{\\rm prompt\}:=\\\{x\\\}, to evaluate and further improve the quality of the generated responses, through a reinforcement learning \(RL\) process\. The whole process of RLHF is summarized and illustrated in Fig\.[3](https://arxiv.org/html/2610.31862#S2.F3)\.

Below we provide an example of the demonstration and the preference data\.

rgb\]0\.95,0\.95,0\.95Example 1: Data of LLM Alignment\.Given a question prompt “How should I respond to an email from my professor asking for a delayed assignment?” The demonstration data includes apreferredanswer “You can politely acknowledge the delay and provide a reasonable explanation\. For example: ‘Dear Professor, I apologize for the delay in submitting my assignment\. I encountered unexpected challenges but will submit it by \[new deadline\]\. Thank you for your understanding\.” Meanwhile, the preference dataset includes an additionalnon\-preferredanswer, e\.g\., “Just say you were busy and didn’t have time\. Professors have to deal with it\.” Clearly, the preferred response aligns with human values of politeness and professionalism\.

We will provide a detailed account of popular methods to learn from these different kinds of datasets in the subsequent sections\. In a high level, the SFT step can be used to directly teach LLM to imitate the expert behavior, while the RLHF step teaches LLM to distinguish the better behavior over the worse ones, in order to achieve a better generalization\.

Based on the above discussion, we can now define the LLM alignment problem as follows:

rgb\]0\.95,0\.95,0\.95LLM alignment problem:Given a diverse dataset representing human demonstrations and preferences, how can we effectively train an LLM to consistently generate outputs that align with human values, ethical principles, and intended use cases?

The LLM Evaluation\.Before closing this introductory section, let us briefly discuss how the LLMs are evaluated after different stages of training\. At the pre\-training stage, the model learns to predict the next token probabilistically\. The model is thus evaluated on how well it predicts the next token, and the common measure is the perplexity \(PPL\)\.

Mathematically, given a sequencex=\[x1,…,xm\]x=\[x\_\{1\},\\ldots,x\_\{m\}\], perplexity is defined as the exponential of the average negative log\-likelihood of the tokens:

PPL\(x;θ\):=exp\(−1m∑j=1mlogπ\(xj∣x<j;θ\)\),\\text\{PPL\}\(x;\\theta\):=\\exp\\\!\\left\(\-\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\log\\pi\(x\_\{j\}\\mid x\_\{<j\};\\theta\)\\right\),which assigns a single score to each sequence\. A lower perplexity indicates that the model is more confident and accurate in predicting the next tokens\.

In the fine\-tuning stage, multiple metrics are typically used\. After the SFT step, the metric is usually determined by the nature of the task\. For classification tasks such as sentiment analysis \(i\.e\., given a text of words, determine whether the sentiment of the text is positive or negative\), traditional metrics, such as accuracy, precision and recall are used; For translation tasks, BLEU \(BiLingual Evaluation Understudy\) score is the most popular metric, which measures how similar a machine\-translated text is to a human\-written reference translation\. After the RLHF step, the aligned model could be evaluated by win\-rates from human raters or some judge models \(typically takes the form of a specialized LLM\), which calculate the favorability of the model response over other models/ground truth response\.

In the next section, we will begin our technical discussion by delving into classical techniques on learning from human feedback data\. This will serve as mathematical foundation for various alignment techinques to be discuss subsequently\.

## 3Theoretical Foundations of Learning from Human Feedback\.

Given the background above, it is clear that the alignment problem is closely related to the problem of modeling human behaviors through the observed choices\. Indeed, in the context of LLM, the responseyyin the demonstration data as well as the labels that distinguishywy\_\{w\}’s withyly\_\{l\}’s can both be viewed as human choices\. It turns out that there is a vast literature about developing models for discrete choices, and these models were first used to describe discrimination between perceptual stimuli\[[52](https://arxiv.org/html/2610.31862#bib.bib12)\]and choices for urban transportation modes\[[36](https://arxiv.org/html/2610.31862#bib.bib11)\]\. In this section, we will provide a brief overview of this literature\.

### 3\.1Classical models of discrete choice

The choice models are used to describe observed choices \(or actions\)a∈Aa\\in Amade by a decision maker given a relevant state variables∈Ss\\in S\. A parallel to LLM alignment problem described in the previous section can be made by having the statesscorrespond to the promptxxand the actionaacorresponds to the possible responsesyy’s to the prompt\.

As in the previous section, we use the notation\(s,a\)≻\(s,a′\)\(s,a\)\\succ\(s,a^\{\\prime\}\)to indicate that in statessactiona∈Aa\\in Ais preferred overa′∈Aa^\{\\prime\}\\in A\. An assumedstructureof preferences over action setAAis a relation≻\\succ, which is ∙\\bulletComplete:\(s,a\)≻\(s,a′\)\(s,a\)\\succ\(s,a^\{\\prime\}\)ora′≻aa^\{\\prime\}\\succ afor alla,a′∈Aa,a^\{\\prime\}\\in Aands∈Ss\\in S\.

∙\\bulletTransitive:\(s,a\)≻\(s,a′\)\(s,a\)\\succ\(s,a^\{\\prime\}\)and\(s,a′\)≻\(s,a′′\)\(s,a^\{\\prime\}\)\\succ\(s,a^\{\\prime\\prime\}\)implies\(s,a\)≻\(s,a′′\)\(s,a\)\\succ\(s,a^\{\\prime\\prime\}\), for alla,a′,a′′∈Aa,a^\{\\prime\},a^\{\\prime\\prime\}\\in A,s∈Ss\\in S\.

A structure of preferences can be modeled by a reward \(or utility\) functionr:S×A↦ℝr:S\\times A\\mapsto\\mathbb\{R\}, such that\(s,a\)≻\(s,a′\)\(s,a\)\\succ\(s,a^\{\\prime\}\)if and only ifr⁡\(s,a\)\>r⁡\(s,a′\)r\(s,a\)\>r\(s,a^\{\\prime\}\)\. This implies when in statess, the decision maker will always preferaatoa′a^\{\\prime\}\. However in practice, one may observe a decision maker engages inmixing, i\.e\. implementing different actions when in the same states∈Ss\\in S\. In the context of LLM alignment, mixing corresponds to observing different responses to the same prompt\.

There are several ways of modeling mixing in the literature of discrete choice:

Random Utility: In this model, the decision maker perceives a random reward \(or utility\)r⁡\(s,a\)\+ϵ⁡\(a\)r\(s,a\)\+\\epsilon\(a\)\(whereϵ⁡\(a\)\\epsilon\(a\)is a random variable\) given the pair\(s,a\)\(s,a\)\. A pair\(s,a\)\(s,a\)is preferred to\(s,a′\)\(s,a^\{\\prime\}\)if:

r⁡\(s,a\)\+ϵ⁡\(a\)\>r⁡\(s,a′\)\+ϵ⁡\(a′\)\.r\(s,a\)\+\\epsilon\(a\)\>r\(s,a^\{\\prime\}\)\+\\epsilon\(a^\{\\prime\}\)\.For a given states∈𝒮s\\in\\mathcal\{S\}and a rewardr⁡\(⋅\)r\(\\cdot\), the probability of such preference can be modeled as follows:

Pr​\(\(s,a\)≻\(s,a′\)\)\\displaystyle P\_\{r\}\\Big\(\(s,a\)\\succ\(s,a^\{\\prime\}\)\\Big\):=Prob​\(r⁡\(s,a\)\+ϵ⁡\(a\)\>r⁡\(s,a′\)\+ϵ⁡\(a′\)\)=Prob​\(ϵ⁡\(a\)−ϵ⁡\(a′\)\>r⁡\(s,a′\)−r⁡\(s,a\)\)\.\\displaystyle:=\\mbox\{Prob\}\\Big\(r\(s,a\)\+\\epsilon\(a\)\>r\(s,a^\{\\prime\}\)\+\\epsilon\(a^\{\\prime\}\)\\Big\)=\\mbox\{Prob\}\\Big\(\\epsilon\(a\)\-\\epsilon\(a^\{\\prime\}\)\>r\(s,a^\{\\prime\}\)\-r\(s,a\)\\Big\)\.When the random variables\{ϵ⁡\(a\):a∈A\}\\\{\\epsilon\(a\):a\\in A\\\}are assumed i\.i\.d and standard Gumbel we obtain the following logit model\[[33](https://arxiv.org/html/2610.31862#bib.bib37),[9](https://arxiv.org/html/2610.31862#bib.bib31)\]\(also known as the Bradley\-Terry\-Luce model \(BTL\)\):

Pr​\(\(s,a\)≻\(s,a′\)\)\\displaystyle P\_\{r\}\\Big\(\(s,a\)\\succ\(s,a^\{\\prime\}\)\\Big\)=exp⁡r⁡\(s,a\)exp⁡r⁡\(s,a\)\+exp⁡r⁡\(s,a′\)=σ⁡\(r⁡\(s,a\)−r⁡\(s,a′\)\),\\displaystyle=\\frac\{\\exp r\(s,a\)\}\{\\exp r\(s,a\)\+\\exp r\(s,a^\{\\prime\}\)\}=\\sigma\\Big\(r\(s,a\)\-r\(s,a^\{\\prime\}\)\\Big\),\(3\)whereσ⁡\(x\):=11\+e−x\\sigma\(x\):=\\frac\{1\}\{1\+e^\{\-x\}\}\. In\[[20](https://arxiv.org/html/2610.31862#bib.bib64)\], the authors summarize the various motivations and use\-cases of the BT model\. We will see later that the logit model expressed above has been heavily used in the RLHF step to learn a reward function from preference data\.

Perturbed Utility: Stochastic choice can also arise as the solution to expected reward maximization subject to an additive perturbation\[[35](https://arxiv.org/html/2610.31862#bib.bib10)\]which may describe limited information processing capacity\[[53](https://arxiv.org/html/2610.31862#bib.bib7)\]or aversion to ambiguity\[[34](https://arxiv.org/html/2610.31862#bib.bib6)\]\. Assuming that the reward functionr⁡\(⋅\)r\(\\cdot\)is known, then the optimal stochastic choice can be modeled as the following utility maximization problem

πr∗\(⋅\|s\):=argmaxπ∈Δ\|A\|\{𝔼a∼π\[r\(s,a\)\]−β×c\(π\)\},\\displaystyle\\pi^\{\*\}\_\{r\}\(\\cdot\|s\):=\\arg\\max\_\{\\pi\\in\\Delta^\{\|A\|\}\}\\\{\\mathbb\{E\}\_\{a\\sim\\pi\}\[r\(s,a\)\]\-\\beta\\times c\(\\pi\)\\\},\(4\)whereΔ\|A\|⊂ℝ\|A\|\\Delta^\{\|A\|\}\\subset\\mathbb\{R\}^\{\|A\|\}is the simplex;c:Δ\|A\|↦ℝc:\\Delta^\{\|A\|\}\\mapsto\\mathbb\{R\}is a convex function;β\>0\\beta\>0is a constant\. For example, whenccis the \(forward\) Kullback\-Leibler \(KL\) divergence with respect to a default policyπ0\\pi^\{0\}defined as

DKL\(π∥π0\):=∑a∈Aπ\(a\)logπ⁡\(a\)π0​\(a\),\\displaystyle D\_\{\\rm KL\}\(\\pi\\\|\\pi^\{0\}\):=\\sum\_\{a\\in A\}\\pi\(a\)\\log\\frac\{\\pi\(a\)\}\{\\pi^\{0\}\(a\)\},\(5\)then the optimal stochastic choice can be expressed as follows:

πr∗​\(a\|s\)=π0​\(a\|s\)​exp⁡\(β−1​r​\(s,a\)\)∑a~∈Aπ0​\(a~\|s\)​exp⁡\(β−1​r​\(s,a~\)\)\.\\displaystyle\\pi^\{\*\}\_\{r\}\(a\|s\)=\\frac\{\\pi^\{0\}\(a\|s\)\\exp\(\\beta^\{\-1\}r\(s,a\)\)\}\{\\sum\_\{\\tilde\{a\}\\in A\}\\pi^\{0\}\(\\tilde\{a\}\|s\)\\exp\(\\beta^\{\-1\}r\(s,\\tilde\{a\}\)\)\}\.\(6\)It should be noted that in order to precisely calculate the optimal policy, one has to assume that all available actions inAAare known\. Nevertheless, the perturbed utility model described here is instrumental for establishing a number of LLM alignment approaches, assuming that a reward model is available\.

### 3\.2The reward learning problem

Based upon the observed preferences𝒟pref:=\{\(s,a\)≻\(s,a′\)\}\\mathcal\{D\}\_\{\\rm pref\}:=\\\{\(s,a\)\\succ\(s,a^\{\\prime\}\)\\\}by an agent and the model \([3](https://arxiv.org/html/2610.31862#S3.E3)\), the estimation problem consists of finding a reward functionr⁡\(s,a\)r\(s,a\)thatrationalizesthe data, so that observed preferences areconsistentwith a reward model\. Specifically, the likelihood of𝒟pref\\mathcal\{D\}\_\{\\rm pref\}can be written as

ℓpref​\(r\)=𝔼𝒟pref​\[log⁡Pr​\(\(s,a\)≻\(s,a′\)\)\]\.\\displaystyle\\ell\_\{\\rm pref\}\(r\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\rm pref\}\}\\Big\[\\log P\_\{r\}\\Big\(\(s,a\)\\succ\(s,a^\{\\prime\}\)\\Big\)\\Big\]\.\(7\)In general, there is no unique reward function that maximizes likelihoodℓpref​\(r\)\\ell\_\{\{\\rm pref\}\}\(r\)\. However, assuming the rewardr⁡\(s,a0\)r\(s,a\_\{0\}\)for a reference actiona0∈Aa\_\{0\}\\in Ais known, it can be shown a unique reward function that maximizes likelihood functionℓpref​\(r\)\\ell\_\{\\rm pref\}\(r\)\.

Alternatively, based upon the observed choices𝒟demo:=\{\(s,a\)\}\\mathcal\{D\}\_\{\\rm demo\}:=\\\{\(s,a\)\\\}and assuming these choices areconsistentwith the model \([6](https://arxiv.org/html/2610.31862#S3.E6)\)\. Then the estimation problem consists of finding a reward functionr⁡\(s,a\)r\(s,a\)that maximizes the likelihood of𝒟demo\\mathcal\{D\}\_\{\\rm demo\}defined as:

ℓdemo​\(r\)=𝔼𝒟demo​\[log⁡πr∗​\(a\|s\)\]\.\\displaystyle\\ell\_\{\\rm demo\}\(r\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\rm demo\}\}\\Big\[\\log\\pi^\{\*\}\_\{r\}\(a\|s\)\\Big\]\.\(8\)As pointed out before, a limitation of this model is the assumption of complete information on the menu of available choicesAA\. When the reward is linearly parametrized, i\.e\.r⁡\(s,a\)=ϕ​\(s,a\)⊤​θr\(s,a\)=\\phi\(s,a\)^\{\\top\}\\thetawhereϕ⁡\(s,a\),θ∈ℝp\\phi\(s,a\),\\theta\\in\\mathbb\{R\}^\{p\}andϕ⁡\(s,a\)\\phi\(s,a\)is a vector of features, problem \([8](https://arxiv.org/html/2610.31862#S3.E8)\) is the Lagrangian dual of themaximum entropyestimation problem \(see Theorem 2 in\[[69](https://arxiv.org/html/2610.31862#bib.bib16)\]\):

maxπ\(⋅\|s\)∈Δ\|A\|\\displaystyle\\max\_\{\\pi\(\\cdot\|s\)\\in\\Delta^\{\|A\|\}\}𝔼s∼𝒟demo\[ℋ\(π\(⋅\|s\)\)\],s\.t\.𝔼s∼𝒟demo,a∼π\(⋅\|s\)\[ϕ\(s,a\)\]\.\\displaystyle\\quad\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\rm demo\}\}\[\\mathcal\{H\}\(\\pi\(\\cdot\|s\)\)\],\\quad\{\\rm s\.t\.\}\\quad\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\rm demo\},a\\sim\\pi\(\\cdot\|s\)\}\[\\phi\(s,a\)\]\.

## 4State\-of\-the\-art Algorithms for LLM Alignment

In this section, we provide an overview of the mathematical formulation of the alignment problem, and a number of state\-of\-the\-art algorithms\. To facilitate discussion, this section is organized according to the*structure*of the available data\.

### 4\.1Learning from demonstrations data

We begin with the discussion on how to model and learn from*demonstration data*𝒟demo:=\{\(x,y\)\}\\mathcal\{D\}\_\{\\text\{demo\}\}:=\\\{\(x,y\)\\\}\. Given the promptxx, the responseyyis viewed as agolden answer\. A natural approach to learning from such a dataset is supervised learning, leading to the supervised fine\-tuning \(SFT\) paradigm\. To this end, we formulate the following optimization problem over the LLM parameters classΘ\\Theta:

minθ∈ΘℓSFT\(θ\):=−𝔼x∼ρ,y∼πE\(⋅\|x\)\[logπ\(y\|x;θ\)\]\\min\_\{\\theta\\in\\Theta\}\\ \\ell\_\{\\mathrm\{SFT\}\}\(\\theta\):=\-\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi^\{\\rm E\}\(\\cdot\|x\)\}\\left\[\\log\\pi\\left\(y\|x;\\theta\\right\)\\right\]\(9\)whereπE\\pi^\{\\rm E\}is the expert policy\. It is easy to see the above problem shares the same optimal solutions as

minθ∈Θ𝔼x∼ρ\[DKL\(πE\(⋅\|x\)∥π\(⋅\|x;θ\)\)\],\\min\_\{\\theta\\in\\Theta\}~\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi^\{\\mathrm\{E\}\}\\left\(\\cdot\|x\\right\)\\\|\\pi\\left\(\\cdot\|x;\\theta\\right\)\)\],\(10\)whereDK​L​\(⋅\)D\_\{KL\}\(\\cdot\)is the \(forward\) KL\-divergence defined in \([5](https://arxiv.org/html/2610.31862#S3.E5)\)\. The latter shows that SFT aims at imitating the demonstration dataset via minimizing the KL divergence\. Given a dataset𝒟demo\\mathcal\{D\}\_\{\\rm demo\}of expert demonstrations, we show the SFT method in Protocol[1](https://arxiv.org/html/2610.31862#alg1)whereℓ^SFT​\(θ\)\\widehat\{\\ell\}\_\{\\mathrm\{SFT\}\}\(\\theta\)is a finite sample version ofℓSFT​\(θ\)\\ell\_\{\\mathrm\{SFT\}\}\(\\theta\)in \([9](https://arxiv.org/html/2610.31862#S4.E9)\)\.

It is worth noting that the SFT stage described here is closely related to behavioral cloning\[[39](https://arxiv.org/html/2610.31862#bib.bib42),[44](https://arxiv.org/html/2610.31862#bib.bib43)\], whose goal is tomimicthe policy of an expert, we describe the details in the next paragraph\.

#### 4\.1\.1Behavioral cloning approaches

Behavioural cloning \(BC, see Protocol[1](https://arxiv.org/html/2610.31862#alg1)\) it is anofflineproblem where the learner cannot collect new data using the current policy, but should learn relying exclusively on the expert dataset\[[39](https://arxiv.org/html/2610.31862#bib.bib42)\]\. This can be done via a reduction to supervised learning where the actions are the labels to be predicted and the states are the features\. Very recently,\[[16](https://arxiv.org/html/2610.31862#bib.bib14)\]show that exact minimization of the lossℓ^SFT\\widehat\{\\ell\}\_\{\\mathrm\{SFT\}\}allows to output a policy which enjoys suboptimality guarantees compared to the expert\. Their experiments and theoretical analysis also show that the guarantees and practical performance of the method do not deteriorate for long horizon problems\. Unfortunately, exact minimization of the lossℓ^SFT\\widehat\{\\ell\}\_\{\\mathrm\{SFT\}\}is not possible for non\-convex policy classes, and to our knowledge, there are currently no studies investigating how those optimization errors affect performance\. In\[[60](https://arxiv.org/html/2610.31862#bib.bib60)\], the authors proposed offline inverse reinforcement learning \(IRL\)\-based imitation leraning methods in the SFT setting to effectively utilize the sequential structure underlying autoregressive generation\. Compared with standard SFT method, the IRL methods can extract both reward and policy from the demonstration dataset\. Moreover, the authors find that offline IRL\-based imitation retains diversity while maximizing task performance\. In this sense, IRL can also be regarded as a strong alternative on fixed SFT datasets even without online data generation\. In Sec\.[5](https://arxiv.org/html/2610.31862#S5), we will discuss the details of IRL in LLM alignment\. It is also important to notice that in the alignment of LLM, the environment is simpler than what generally assumed in analyzing and empirically testing BC\. Indeed, in language tasks LLMs are tree shaped and deterministic\.

Protocol 1SFT \(Supervised Fine Tuning\) via Behavioral Cloning \(BC\)1:The learner receives: \(i\) A dataset

𝒟demo=\{\(xi,yi\)\}i=1Ndemo\\mathcal\{D\}\_\{\\text\{demo\}\}=\\\{\(x^\{i\},y^\{i\}\)\\\}^\{N\_\{\\mathrm\{demo\}\}\}\_\{i=1\}, \(ii\) A policy function parameters class

Θ\\Theta\.

2:The learner computes the loss:

ℓ^SFT​\(θ\):=𝔼\(x,y\)∼𝒟demo​\[−log⁡π⁡\(yi\|xi;θ\)\]\.\\widehat\{\\ell\}\_\{\\mathrm\{SFT\}\}\(\\theta\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\}\}\\left\[\-\\log\\pi\\left\(y^\{i\}\|x^\{i\};\\theta\\right\)\\right\]\.
3:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}with

θ⋆=arg​minθ∈Θ⁡ℓ^SFT​\(θ\)\\theta^\{\\star\}=\\argmin\_\{\\theta\\in\\Theta\}\\widehat\{\\ell\}\_\{\\mathrm\{SFT\}\}\(\\theta\)\.

#### 4\.1\.2Self generation approaches

An alternative approach, dubbed SPIN \(Self\-Play fIne\-tuNing\), presented in\[[10](https://arxiv.org/html/2610.31862#bib.bib15)\]casts the fine\-tuning with demonstration problem as a two\-player game\. On the one hand, the first player tries to generate answers as similar as possible as the observed ones in𝒟demo\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\. On the other hand, the second player’s aims at distinguish artificially generated answers from the answers in the demonstrations dataset𝒟demo\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\. The algorithm employs alternating updates between the first and second player\. In particular, at iterationtt, the first player observes the second players parametersθt\\theta\_\{t\}and updates its weights via the following minimization over potentially time\-varying parameters classes\{ℱt\}\\\{\\mathcal\{F\}\_\{t\}\\\}as

ft\+1=arg​minf∈ℱt𝔼x,y∼𝒟demo𝔼y′∼π\(⋅\|x;θt\)\[ℓ\(f\(x,y\)−f\(x,y′\)\)\],f\_\{t\+1\}=\\argmin\_\{f\\in\\mathcal\{F\}\_\{t\}\}\\mathbb\{E\}\_\{x,y\\sim\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\}\\mathbb\{E\}\_\{y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\left\[\{\\ell\(f\(x,y\)\-f\(x,y^\{\\prime\}\)\)\}\\right\],\(11\)whereℓ⁡\(⋅\)\\ell\(\\cdot\)is a monotonically decreasing, non negative, smooth and convex function\. A common choice in practice satisfying the above properties is the logistic loss functionℓ⁡\(x\)=log⁡\(1\+exp⁡\(−x\)\)\\ell\(x\)=\\log\\big\(1\+\\exp\(\-x\)\\big\)\. All these properties are imposed to make the optimization problem easier to solve via first order methods\. At this point, the second player can update its weights exploiting knowledge offt\+1f\_\{t\+1\}\. This is done by solving the following problem:θt\+1=arg​minθ∈Θ−𝔼x∼ρ,y∼π\(⋅\|x;θt\)\[ft\+1\(x,y\)\+βDKL\(π\(⋅\|x;θ\)\|\|π\(⋅\|x;θt\)\)\],\\theta\_\{t\+1\}=\\argmin\_\{\\theta\\in\\Theta\}\-\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\left\[\{f\_\{t\+1\}\(x,y\)\+\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\(\\cdot\|x;\\theta\)\|\|\\pi\(\\cdot\|x;\\theta\_\{t\}\)\)\}\\right\],which is an application of[Equation4](https://arxiv.org/html/2610.31862#S3.E4)forc\(π\)=−DKL\(π\(⋅\|x\)\|\|π\(⋅\|x;θt\)\)c\(\\pi\)=\-D\_\{\\mathrm\{KL\}\}\(\\pi\(\\cdot\|x\)\|\|\\pi\(\\cdot\|x;\\theta\_\{t\}\)\)\. The analytical solution of the above problem is surprisingly easy to obtain \(analogously to \([6](https://arxiv.org/html/2610.31862#S3.E6)\)\):π⁡\(y\|x;θt\+1\)∝π⁡\(y\|x;θt\)​exp⁡\(ft\+1​\(x,y\)β\),\\pi\(y\|x;\\theta\_\{t\+1\}\)\\propto\\pi\(y\|x;\\theta\_\{t\}\)\\exp\\left\(\{\\frac\{f\_\{t\+1\}\(x,y\)\}\{\\beta\}\}\\right\),if the classΘ\\Thetais expressive enough compared to the classℱt\+1\\mathcal\{F\}\_\{t\+1\}\. To ensure that this is the case, in\[[10](https://arxiv.org/html/2610.31862#bib.bib15)\]the classℱt\+1\\mathcal\{F\}\_\{t\+1\}is chosen to be coupled withΘ\\Thetaas follows,ℱt\+1=\{β​log⁡\(π⁡\(y\|x;θt\+1\)π⁡\(y\|x;θt\)\)\}\\mathcal\{F\}\_\{t\+1\}=\\left\\\{\\beta\\log\\left\(\{\\frac\{\\pi\(y\|x;\\theta\_\{t\+1\}\)\}\{\\pi\(y\|x;\\theta\_\{t\}\)\}\}\\right\)\\right\\\}\. Another benefit of this choice is that the two updates can now be expressed as single update over the weight classΘ\\Theta\. Indeed, \([11](https://arxiv.org/html/2610.31862#S4.E11)\) is at this point equivalent to

θt\+1=arg​minθ∈Θℓ^SPIN\(θ\)=𝔼\(x,y\)∼𝒟demo𝔼y′∼π\(⋅\|x;θt\)\[ℓ\(βlog\(π⁡\(y\|x;θ\)π⁡\(y\|x;θt\)\)−βlog\(π⁡\(y′\|x;θ\)π⁡\(y′\|x;θt\)\)\)\]\.\\displaystyle\\theta\_\{t\+1\}=\\argmin\_\{\\theta\\in\\Theta\}\\widehat\{\\ell\}\_\{\\mathrm\{SPIN\}\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\}\\mathbb\{E\}\_\{y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\left\[\{\\ell\\left\(\\beta\\log\\left\(\{\\frac\{\\pi\(y\|x;\\theta\)\}\{\\pi\(y\|x;\\theta\_\{t\}\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\(y^\{\\prime\}\|x;\\theta\)\}\{\\pi\(y^\{\\prime\}\|x;\\theta\_\{t\}\)\}\}\\right\)\\right\)\}\\right\]\.\(12\)
![Refer to caption](https://arxiv.org/html/2610.31862v1/figures/mode_seeking_covering.png)Figure 4:True distribution in blue and estimated distribution in green\. When the estimated distribution has a smaller support than the true one, as in the left plot, we say that it has a mode seeking\. Vice versa, the estimated distribution has a mode covering behavior if its support is larger, as in the right plot\. Image from[https://sander\.ai/2020/03/24/audio\-generation\.html](https://sander.ai/2020/03/24/audio-generation.html)\.The practical algorithm can be obtained via a stochastic estimator of this loss constructed with the dataset𝒟demo\\mathcal\{D\}\_\{\\mathrm\{demo\}\}, as described in Protocol[2](https://arxiv.org/html/2610.31862#alg2)\. We note that SPIN shares similarity with Direct Preference Optimization \(DPO\), to be described in Section[4\.3](https://arxiv.org/html/2610.31862#S4.SS3), however there are also important differences that we will highlight in Section[5](https://arxiv.org/html/2610.31862#S5)\.

In SPIN, the adversarial imitation procedure leverages self\-generated data to recover a demonstration\-level policy \(see Equation \([11](https://arxiv.org/html/2610.31862#S4.E11)\)\)\. Notably, both adversarial imitation frameworks like TextGAIL\[[59](https://arxiv.org/html/2610.31862#bib.bib29)\]\(see Protocol[3](https://arxiv.org/html/2610.31862#alg3)\) and online IRL approaches\[[72](https://arxiv.org/html/2610.31862#bib.bib47)\]similarly incorporate self\-generated rollouts into their training pipelines to enable effective learning from demonstrations\. We discuss these details of IRL approaches and their connections with other baselines in Section[5](https://arxiv.org/html/2610.31862#S5)\.

Protocol 2SFT \(Supervised Fine Tuning\) via SPIN \(Self\-Play fIne tuNing\)1:The learner receives: \(i\) A dataset

𝒟demo=\{\(xi,yi\)\}i=1Ndemo\\mathcal\{D\}\_\{\\text\{demo\}\}=\\\{\(x^\{i\},y^\{i\}\)\\\}^\{N\_\{\\mathrm\{demo\}\}\}\_\{i=1\}, \(ii\) A policy function parameters class

Θ\\Theta\.

2:for

t=1,…,Tt=1,\\dots,Tdo

3:The learner samples

y′,i∼π\(⋅\|xi;θt\)y^\{\{\}^\{\\prime\},i\}\\sim\\pi\(\\cdot\|x^\{i\};\\theta\_\{t\}\)and computes the loss:

ℓ^SPIN\(θ\):=𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅∣x;θt\)\[ℓ\(βlog\(π⁡\(y∣x;θ\)π⁡\(y∣x;θt\)\)−βlog\(π⁡\(y′∣x;θ\)π⁡\(y′∣x;θt\)\)\)\]\.\\widehat\{\\ell\}\_\{\\mathrm\{SPIN\}\}\(\\theta\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{demo\}\},\\;y^\{\\prime\}\\sim\\pi\(\\cdot\\mid x;\\theta\_\{t\}\)\}\\left\[\\ell\\left\(\\beta\\log\\left\(\\frac\{\\pi\(y\\mid x;\\theta\)\}\{\\pi\(y\\mid x;\\theta\_\{t\}\)\}\\right\)\-\\beta\\log\\left\(\\frac\{\\pi\(y^\{\\prime\}\\mid x;\\theta\)\}\{\\pi\(y^\{\\prime\}\\mid x;\\theta\_\{t\}\)\}\\right\)\\right\)\\right\]\.
4:

θt\+1:=arg​minθ∈Θ⁡ℓ^SPIN​\(θ\)\\theta\_\{t\+1\}:=\\argmin\_\{\\theta\\in\\Theta\}\\widehat\{\\ell\}\_\{\\mathrm\{SPIN\}\}\(\\theta\)
5:endfor

6:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}

Protocol 3GAIL \(Generative Adversarial Imitation Learning\)\[[59](https://arxiv.org/html/2610.31862#bib.bib29)\]1:The learner receives: \(i\) A dataset

𝒟demo=\{\(xi,yi\)\}i=1Ndemo\\mathcal\{D\}\_\{\\text\{demo\}\}=\\\{\(x^\{i\},y^\{i\}\)\\\}^\{N\_\{\\mathrm\{demo\}\}\}\_\{i=1\}, \(ii\) A policy function parameter class

Θ\\Theta, \(iii\) A discriminator parameter class

Φ\\Phi\.

2:The learner optimizes the following problem:

minθmaxϕℓ^GAIL\(θ,ϕ\):=𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅∣x;θ\)\[log\(1−Dϕ\(x,y\)\)\+log\(D\(x,y′\)\)\+λlogπθ\(y′\|x\)\]\.\\min\_\{\\theta\}\\max\_\{\\phi\}\\widehat\{\\ell\}\_\{\\mathrm\{GAIL\}\}\(\\theta,\\phi\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{demo\}\},\\;y^\{\\prime\}\\sim\\pi\(\\cdot\\mid x;\\theta\)\}\\Big\[\\log\(1\-D\_\{\\phi\}\(x,y\)\)\+\\log\(D\(x,y^\{\\prime\}\)\)\+\\lambda\\log\\pi\_\{\\theta\}\(y^\{\\prime\}\|x\)\\Big\]\.
3:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}

#### 4\.1\.3Information theoretic approaches

Recent works\[[49](https://arxiv.org/html/2610.31862#bib.bib46),[12](https://arxiv.org/html/2610.31862#bib.bib38)\]considered matching general information theoretic divergences between answers distribution\.

As mentioned, minimizing the forward KL divergence between demonstrated and learner answer distribution is equivalent to the SFT objective in \([10](https://arxiv.org/html/2610.31862#S4.E10)\)\. As commonly noticed, the forward KL minimization outputs a mode covering distribution\. That is, the learned distribution has support which contains the support of the demonstrations distribution\.

However, a mode seeking behavior can be more desirable if one wants to perfectly imitate some aspects of the demonstrations while discarding others\. We refer the reader to[Figure4](https://arxiv.org/html/2610.31862#S4.F4)for a visual illustration\. To this end,\[[49](https://arxiv.org/html/2610.31862#bib.bib46)\]suggested to minimize the reversed KL divergenceminθ∈Θ𝔼x∼ρ\[DKL\(π\(⋅\|x;θ\)∥πE\(⋅\|x\)\)\]\\min\_\{\\theta\\in\\Theta\}~\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta\\right\)\\\|\\pi^\{\\mathrm\{E\}\}\\left\(\\cdot\|x\\right\)\)\]\. This method however requires additional prompts and answers generated by the learner policy\. As a last example of divergence minimization,\[[61](https://arxiv.org/html/2610.31862#bib.bib41)\]considersχ2\\chi^\{2\}divergence minimization\.

### 4\.2Learning from preferences

Learning from preferences is usually performed in two steps: \(i\) reward learning using the preference dataset \(ii\) based on the reward model learned in the first step, further fine\-tune the LLM by conducting reinforcement learning using a prompt dataset\. Note that these two steps are closely related to the two types of stochastic choice models discussed in Section[3\.1](https://arxiv.org/html/2610.31862#S3.SS1)\.

Protocol 4RLHF \(Reinforcement Learning from Human Feedback\)1:The learner receives: \(i\) A preference dataset

𝒟pref=\{\(xi,ywi,yℓi\)\}i=1Npref\\mathcal\{D\}\_\{\\rm pref\}=\\\{\(x^\{i\},y^\{i\}\_\{w\},y^\{i\}\_\{\\ell\}\)\\\}^\{N\_\{\\rm pref\}\}\_\{i=1\}, \(ii\) A prompt dataset

𝒟prompts=\{xi\}i=1Nprompts\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}=\\\{x^\{i\}\\\}^\{N\_\{\\mathrm\{prompts\}\}\}\_\{i=1\}, \(iii\) A reward function parameter class

Φ\\Phi, \(iv\) A policy function parameter class

Θ\\Theta\.

2:The learner estimates the reward as

r⁡\(⋅,⋅,ϕ⋆\)r\\left\(\{\\cdot,\\cdot;\\phi^\{\\star\}\}\\right\)with

ϕ⋆=arg​minϕ∈Φ⁡ℓ^RM​\(ϕ\):=−𝔼\(x,yw,yl\)∼𝒟pref​\[log⁡\(σ⁡\(r⁡\(xi,ywi,ϕ\)−r⁡\(xi,yli,ϕ\)\)\)\]\.\\phi^\{\\star\}=\\argmin\_\{\\phi\\in\\Phi\}\\ \\widehat\{\\ell\}\_\{\\rm RM\}\(\\phi\):=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\_\{\\rm pref\}\}\\Big\[\\log\\Big\(\\sigma\\big\(r\(x^\{i\},y^\{i\}\_\{w\};\\phi\)\-r\(x^\{i\},y^\{i\}\_\{l\};\\phi\)\\big\)\\Big\)\\Big\]\.
3:The learner finds

θ⋆=arg​minθ∈Θ⁡ℓRL​\(θ\)\\theta^\{\\star\}=\\argmin\_\{\\theta\\in\\Theta\}\\ell\_\{\\mathrm\{RL\}\}\(\\theta\), using RL trainer\.

4:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}\.

Protocol 5PPO \(Proximal Policy Optimization\)\[[47](https://arxiv.org/html/2610.31862#bib.bib17)\]1:Receive policy class parameters

Θ\\Theta\. Initialize

θ1\\theta\_\{1\}\.

2:for

t=1,…,Tt=1,\\dots,Tdo

3:Sample

y∼π\(⋅\|x;θt\)y\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)and

x∈𝒟promptsx\\in\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}\.

4:Update the value function

5:Compute advantages

\{A^h\}h=1\|y\|\\\{\\hat\{A\}\_\{h\}\\\}\_\{h=1\}^\{\|y\|\}according to generalized advantage estimation at the token level

6:The learner computes

ℓ^PPO​\(θ\)\\displaystyle\\widehat\{\\ell\}\_\{\\rm PPO\}\(\\theta\)=−𝔼x∼𝒟prompts,y∼π\(⋅\|x;θt\)\[1\|y\|∑h=1\|y\|\{min\[π⁡\(yh\|x,y<h;θ\)π⁡\(yh\|x,y<h;θt\)A^h,\\displaystyle=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{prompts\}\},y\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Bigg\[\\frac\{1\}\{\|y\|\}\\sum\_\{h=1\}^\{\|y\|\}\\Bigg\\\{\\min\\Bigg\[\\frac\{\\pi\(y\_\{h\}\|x,y\_\{<h\};\\theta\)\}\{\\pi\(y\_\{h\}\|x,y\_\{<h\};\\theta\_\{t\}\)\}\\hat\{A\}\_\{h\},clip\(π⁡\(yh\|x,y<h;θ\)π⁡\(yh\|x,y<h;θt\),1−ϵ,1\+ϵ\)A^h\]\}\]\\displaystyle\\quad\\quad\\quad\\quad\\text\{clip\}\\Bigg\(\\frac\{\\pi\(y\_\{h\}\|x,y\_\{<h\};\\theta\)\}\{\\pi\(y\_\{h\}\|x,y\_\{<h\};\\theta\_\{t\}\)\},1\-\\epsilon,1\+\\epsilon\\Bigg\)\\hat\{A\}\_\{h\}\\Bigg\]\\Bigg\\\}\\Bigg\]
7:

θt\+1=arg​minθ∈Θ⁡ℓPPO​\(θ\)\\theta\_\{t\+1\}=\\argmin\_\{\\theta\\in\\Theta\}\\ell\_\{\\mathrm\{PPO\}\}\(\\theta\)
8:endfor

9:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}\.

Protocol 6REINFORCE\[[58](https://arxiv.org/html/2610.31862#bib.bib24)\]1:Receive policy class parameters

Θ\\Theta\. Initialize

θ1\\theta\_\{1\}, step size

η\\eta
2:for

t=1,…,Tt=1,\\dots,Tdo

3:Sample outputs

y∼π\(⋅\|x;θt\)y\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)and

x∈𝒟promptsx\\in\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}\.

4:The learner computes the gradient estimator

∇ℓ^REINFORCE​\(θt\)\\displaystyle\\nabla\\widehat\{\\ell\}\_\{\\rm REINFORCE\}\(\\theta\_\{t\}\)=−𝔼x∼𝒟prompts,y∼π\(⋅\|x;θt\)\[r\(x,y\)∇θlogπ\(y\|x;θt\)\]\\displaystyle=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{prompts\}\},y\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\big\[r\(x,y\)\\nabla\_\{\\theta\}\\log\\pi\(y\|x;\\theta\_\{t\}\)\\big\]
5:

θt\+1=θt−η∇ℓ^REINFORCE\(θt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla\\widehat\{\\ell\}\_\{\\rm REINFORCE\}\(\\theta\_\{t\}\)
6:endfor

7:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}\.

##### Reward learning

To find an appropriate reward model, RLHF \(see e\.g\.,\[[11](https://arxiv.org/html/2610.31862#bib.bib34)\]\) leverages a set ofpreference dataset𝒟pref:=\{\(x,yw,yl\)\}\\mathcal\{D\}\_\{\\rm pref\}:=\\\{\(x,y\_\{w\},y\_\{l\}\)\\\}, where each data contains a pair of outputyw,yly\_\{w\},y\_\{l\}\. The outputywy\_\{w\}is preferred overyly\_\{l\}by human labeler \(denoted asyw≻yly\_\{w\}\\succ y\_\{l\}\)\. Leveraging the the Bradley\-Terry model \(BT\) discussed in \([3](https://arxiv.org/html/2610.31862#S3.E3)\) and the likelihood of the preference data \([7](https://arxiv.org/html/2610.31862#S3.E7)\), one could formulate the following problem to find the reward model \(parameterized byϕ\\phi\):

ϕ⋆=arg​minϕ∈Φ⁡ℓRM​\(ϕ\):=−𝔼x∼ρ,\(yw≻yl\)∼πP​\[log⁡\(σ⁡\(r⁡\(x,yw,ϕ\)−r⁡\(x,yl,ϕ\)\)\)\]\.\\phi^\{\\star\}=\\argmin\_\{\\phi\\in\\Phi\}\\ \\ell\_\{\\rm RM\}\(\\phi\):=\-\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{w\}\\succ y\_\{l\}\)\\sim\\pi^\{P\}\}\\Big\[\\log\\Big\(\\sigma\\big\(r\(x,y\_\{w\};\\phi\)\-r\(x,y\_\{l\};\\phi\)\\big\)\\Big\)\\Big\]\.\(13\)where\(yw≻yl\)∼πP\(y\_\{w\}\\succ y\_\{l\}\)\\sim\\pi^\{P\}denotes sampling from the distribution of the preference dataset\. Protocol[4](https://arxiv.org/html/2610.31862#alg4)describes the procedure wherer⁡\(⋅,ϕ\)r\(\\cdot;\\phi\)is learned via minimization of the empirical lossℓ^RM\\widehat\{\\ell\}\_\{\\mathrm\{RM\}\}\. In the empirical lossℓ^RM\\widehat\{\\ell\}\_\{\\mathrm\{RM\}\}, we replace\(yw≻yl\)∼πP\(y\_\{w\}\\succ y\_\{l\}\)\\sim\\pi^\{P\}by sampling from a finite preference datset\(yw≻yl\)∼𝒟prefs\(y\_\{w\}\\succ y\_\{l\}\)\\sim\\mathcal\{D\}\_\{\\rm prefs\}\.

##### Policy Learning

Once a reward modelr⁡\(x,y,ϕ⋆\)r\(x,y;\\phi^\{\\star\}\)is learned, the LLM can be fine tuned on the top of SFT by leveraging the perturbed utility model \([4](https://arxiv.org/html/2610.31862#S3.E4)\), which results in the following problem:

minθℓRL\(θ\):=−𝔼x∼ρ,y∼π\(⋅\|x;θ\)\[r\(x,y;ϕ\)\]\+β𝔼x∼ρ\[DKL\(π\(⋅\|x;θ\)∥πref\(⋅\|x\)\)\],\\min\_\{\\theta\}\\ \\ell\_\{\\mathrm\{RL\}\}\(\\theta\):=\-\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x;\\theta\)\}\\left\[r\(x,y;\\phi\)\\right\]\+\\beta\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\],\(14\)whereπref\\pi\_\{\\mathrm\{ref\}\}is a reference model such as the SFT\-ed model, andβ\\betais the regularization coefficient of the KL regularizer\. Here, the optimal policy in \([14](https://arxiv.org/html/2610.31862#S4.E14)\) under the reward parameterϕ\\phihas the following closed\-form expression:

πrϕ⋆​\(y\|x\)=πref​\(y\|x\)​exp⁡\(β−1​r​\(x,y,ϕ\)\)Z⁡\(x,ϕ,β\),\\pi\_\{r\_\{\\phi\}\}^\{\\star\}\(y\|x\)=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\\exp\(\\beta^\{\-1\}r\(x,y;\\phi\)\)\}\{Z\(x,\\phi,\\beta\)\},\(15\)where we haveZ⁡\(x,ϕ,β\):=∑y′πref​\(y′\|x\)​exp⁡\(β−1​r​\(x,y′,ϕ\)\)Z\(x,\\phi,\\beta\):=\\sum\_\{y^\{\\prime\}\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\|x\)\\exp\(\\beta^\{\-1\}r\(x,y^\{\\prime\};\\phi\)\)\. Note that both the expected reward and the KL in \([14](https://arxiv.org/html/2610.31862#S4.E14)\) arenotcomputable given the large cardinality of𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}, so \([14](https://arxiv.org/html/2610.31862#S4.E14)\) is usually solved approximately by minimizing an empirical loss, using policy optimization techniques such as REINFORCE\[[58](https://arxiv.org/html/2610.31862#bib.bib24)\], RLOO\[[2](https://arxiv.org/html/2610.31862#bib.bib21)\], PPO\[[47](https://arxiv.org/html/2610.31862#bib.bib17)\]and GRPO\[[48](https://arxiv.org/html/2610.31862#bib.bib53)\]; see Protocol[4](https://arxiv.org/html/2610.31862#alg4)\-[8](https://arxiv.org/html/2610.31862#alg8)\.

*Pros and Cons: PPO or REINFORCE?*PPO was proposed as an improvement over the REINFORCE algorithm in environment where the gradient estimates generated by REINFORCE suffer from high variance\. As a result, PPO enforces conservative policy updates to counterbalance the high variance effect\. However, as argued in\[[2](https://arxiv.org/html/2610.31862#bib.bib21)\], for the LLM fine\-tuning task the gradient variance is not as high to justify the PPO conservative mechanism\. Since both options are used across the community, we describe both algorithms in Protocols[5](https://arxiv.org/html/2610.31862#alg5)and[6](https://arxiv.org/html/2610.31862#alg6)respectively\. As a remark, PPO requires a critic model to estimate the advantage function which can be used to compute the value function score\{V⁡\(x,y<h\)\}h=1\|y\|\\\{V\(x,y\_\{<h\}\)\\\}\_\{h=1\}^\{\|y\|\}at each token level and compute the token\-level advantage estimation\. Consequently, training the critic model for token\-level advantage estimation will also incur additional memory overhead\. REINFORCE does not require training a critic, therefore it is more memory efficient than PPO\. RLOO\[[2](https://arxiv.org/html/2610.31862#bib.bib21)\]is a modern REINFORCE modification that retains the advantage of not keeping a critic in memory and achieves a variance reduction effect subtracting the average of the other rewards \(see Protocol[7](https://arxiv.org/html/2610.31862#alg7)\)\. It can be shown that the RLOO estimator in an unbiased estimator of the advantage\. Recently, a new policy optimization method \(GRPO\) which does not require an estimated reward model has been introduced and has been proven successful in reasoning applications \(see Protocol[8](https://arxiv.org/html/2610.31862#alg8)and Section[4\.5](https://arxiv.org/html/2610.31862#S4.SS5)\)\.

Protocol 7RLOO \(REINFORCE Leave\-One\-Out\)\[[2](https://arxiv.org/html/2610.31862#bib.bib21)\]1:Receive policy class parameters

Θ\\Thetaand reward model parameters

ϕ⋆\\phi^\{\\star\}\. Initialize

θ1\\theta\_\{1\}, step size

η\\eta
2:for

t=1,…,Tt=1,\\dots,Tdo

3:Sample

GGoutputs

\{yj\}j=1G∼π\(⋅\|x;θt\)\\\{y^\{j\}\\\}\_\{j=1\}^\{G\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)and

x∈𝒟promptsx\\in\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}\.

4:The learner computes rewards

\{rj\}j=1,…,G\\\{r^\{j\}\\\}\_\{j=1,\.\.\.,G\}for each output

yjy^\{j\}\.

5:The learner computes the estimated advantage function

A^j=rj−1G−1​∑i≠jri\.\\hat\{A\}\_\{j\}=r^\{j\}\-\\frac\{1\}\{G\-1\}\\sum\_\{i\\neq j\}r^\{i\}\.
6:The learner computes

∇ℓ^RLOO​\(θt\)\\displaystyle\\nabla\\widehat\{\\ell\}\_\{\\rm RLOO\}\(\\theta\_\{t\}\)=−𝔼x∼𝒟prompts,\{yj\}j=1G∼π\(⋅\|x;θt\)\[1G∑j=1GA^j∇θlogπ\(yj\|x;θt\)\]\\displaystyle=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{prompts\}\},\\\{y^\{j\}\\\}\_\{j=1\}^\{G\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\\hat\{A\}^\{j\}\\nabla\_\{\\theta\}\\log\\pi\(y^\{j\}\|x;\\theta\_\{t\}\)\\Bigg\]
7:

θt\+1=θt−η∇ℓRLOO\(θt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla\\ell\_\{\\mathrm\{RLOO\}\}\(\\theta\_\{t\}\)
8:endfor

9:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}\.

*Pros and Cons: Learning from demonstrations vs learning from preferences\.*It is widely observed that the models learnedonlyusing SFT stage do not perform as well as those that are trained via episodically learning the policy \([14](https://arxiv.org/html/2610.31862#S4.E14)\) and learning the reward \([13](https://arxiv.org/html/2610.31862#S4.E13)\) on top of the SFT models\[[38](https://arxiv.org/html/2610.31862#bib.bib51)\]\. This is partly because the reward model and the RL algorithms together guide the LLMs to explore high\-quality responses that can go beyond those defined by the SFT data, thus improving their generalization capabilities\[[68](https://arxiv.org/html/2610.31862#bib.bib74)\]\. Moreover, obtaining the SFT dataset is typically more expensive than obtaining preference data, therefore standard alignment process leveragesboththe SFT and the preference data\.

Protocol 8GRPO \(Group Relative Policy Optimization\)\[[48](https://arxiv.org/html/2610.31862#bib.bib53)\]1:Receive policy class parameters

Θ\\Theta, and reward model parameters

ϕ⋆\\phi^\{\\star\}\. Initialize

θ1\\theta\_\{1\}\.

2:for

t=1,…,Tt=1,\\dots,Tdo

3:Sample

GGoutputs

\{yj\}j=1G∼π\(⋅\|x;θt\)\\\{y^\{j\}\\\}\_\{j=1\}^\{G\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)and

x∈𝒟promptsx\\in\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}\.

4:The learner computes rewards

\{rj\}j=1,…,G\\\{r^\{j\}\\\}\_\{j=1,\.\.\.,G\}for each output

yjy^\{j\}\.

5:The learner computes the estimated advantage function

A^j=rj−mean\(\{rj\}j=1G\)std\(\{rj\}j=1G\),j=1,⋯,G\\hat\{A\}^\{j\}=\\frac\{r^\{j\}\-\\operatorname\*\{mean\}\(\\\{r^\{j\}\\\}\_\{j=1\}^\{G\}\)\}\{\\operatorname\*\{std\}\(\\\{r^\{j\}\\\}\_\{j=1\}^\{G\}\)\},\\quad j=1,\\cdots,G
6:The learner computes

ℓGRPO​\(θ\)\\displaystyle\\ell\_\{\\rm GRPO\}\(\\theta\)=−𝔼x∼𝒟prompts,\{yj\}j=1G∼π\(⋅\|x;θt\)\[1G∑j=1G1\|yj\|∑h=1\|yj\|\{min\[π⁡\(yhj\|x,y<hj;θ\)π⁡\(yhj\|x,y<hj;θt\)A^j,\\displaystyle=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{prompts\}\},\\\{y^\{j\}\\\}\_\{j=1\}^\{G\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\\frac\{1\}\{\|y^\{j\}\|\}\\sum\_\{h=1\}^\{\|y^\{j\}\|\}\\Bigg\\\{\\min\\Bigg\[\\frac\{\\pi\(y^\{j\}\_\{h\}\|x,y^\{j\}\_\{<h\};\\theta\)\}\{\\pi\(y^\{j\}\_\{h\}\|x,y^\{j\}\_\{<h\};\\theta\_\{t\}\)\}\\hat\{A\}\_\{j\},clip\(π⁡\(yhj\|x,y<hj;θ\)π⁡\(yhj\|x,y<hj;θt\),1−ϵ,1\+ϵ\)A^j\]−β𝔻KL\[πθ∥πref\]\}\]\\displaystyle\\quad\\quad\\quad\\quad\\text\{clip\}\\Bigg\(\\frac\{\\pi\(y^\{j\}\_\{h\}\|x,y^\{j\}\_\{<h\};\\theta\)\}\{\\pi\(y^\{j\}\_\{h\}\|x,y^\{j\}\_\{<h\};\\theta\_\{t\}\)\},1\-\\epsilon,1\+\\epsilon\\Bigg\)\\hat\{A\}\_\{j\}\\Bigg\]\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\[\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\]\\Bigg\\\}\\Bigg\]
7:

θt\+1=arg​minθ∈Θ⁡ℓGRPO​\(θ\)\\theta\_\{t\+1\}=\\argmin\_\{\\theta\\in\\Theta\}\\ell\_\{\\mathrm\{GRPO\}\}\(\\theta\)
8:endfor

9:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}\.

### 4\.3Learning from preferences without online generations: Direct Preference Optimization

r

Figure 5:Estimated memory consumption of running thePythia\-1Bmodel onTL;DRdataset with batch\-size 2 on a single deviceDirect Preference Optimization \(DPO\)\[[43](https://arxiv.org/html/2610.31862#bib.bib13)\]bypasses the need to perform the RL, by noticing that minimizingℓRL​\(⋅\)\\ell\_\{\\mathrm\{RL\}\}\(\\cdot\)admits an analytical solution if we optimize over the whole Marvov stationary policy space denoted byΠ\\Pi\. In fact, notice that from our discussion on stochastic choice model \([6](https://arxiv.org/html/2610.31862#S3.E6)\), for a fixedr⁡\(⋅,⋅,ϕ\)r\(\\cdot,\\cdot;\\phi\), the solution to the policy optimization problemπϕ⋆\(⋅,x\)=arg​minπ∈Π−𝔼y∼π\(⋅\|x\)\[r\(x,y;ϕ\)\]\+βDKL\(π\(⋅\|x\)\|\|πref\(⋅\|x\)\)\\pi\_\{\\phi\}^\{\\star\}\(\\cdot,x\)=\\argmin\_\{\\pi\\in\\Pi\}\-\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\{r\(x,y;\\phi\)\}\\right\]\+\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\(\\cdot\|x\)\|\|\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)is defined in \([15](https://arxiv.org/html/2610.31862#S4.E15)\)\. This implies that for allyy, it holds thatlog⁡\(πϕ⋆​\(y\|x\)\)=log⁡\(πref​\(y\|x\)\)\+β−1​r​\(x,y,ϕ\)−log⁡\(Z⁡\(x,ϕ,β\)\)\.\\log\(\\pi\_\{\\phi\}^\{\\star\}\(y\|x\)\)=\\log\(\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\)\+\\beta^\{\-1\}r\(x,y;\\phi\)\-\\log\(Z\(x,\\phi,\\beta\)\)\.Such a quantity cannot be computed in closed form because computingZ⁡\(x,ϕ,β\)Z\(x,\\phi,\\beta\)is intractable\. However, for two possible answersyy,y′y^\{\\prime\}to thesamequestionxx, it holds thatlog⁡\(πϕ⋆​\(y\|x\)\)−log⁡\(πϕ⋆​\(y′\|x\)\)=log⁡\(πref​\(y\|x\)\)−log⁡\(πref​\(y′\|x\)\)\+β−1​r​\(x,y,ϕ\)−β−1​r​\(x,y′,ϕ\)\.\\log\(\\pi\_\{\\phi\}^\{\\star\}\(y\|x\)\)\-\\log\(\\pi\_\{\\phi\}^\{\\star\}\(y^\{\\prime\}\|x\)\)=\\log\(\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\)\-\\log\(\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\|x\)\)\+\\beta^\{\-1\}r\(x,y;\\phi\)\-\\beta^\{\-1\}r\(x,y^\{\\prime\};\\phi\)\.Therefore, the difference of reward functions for different answersy,y′y,y^\{\\prime\}can be computed efficiently as the normalization constantZ⁡\(x,ϕ,β\)Z\(x,\\phi,\\beta\)does not appear, i\.e\.

r⁡\(x,y,ϕ\)−r⁡\(x,y′,ϕ\)=β​log⁡\(πϕ⋆​\(y\|x\)πref​\(y\|x\)\)−β​log⁡\(πϕ⋆​\(y′\|x\)πref​\(y′\|x\)\)\.\\displaystyle r\(x,y;\\phi\)\-r\(x,y^\{\\prime\};\\phi\)=\\beta\\log\\left\(\{\\frac\{\\pi\_\{\\phi\}^\{\\star\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\_\{\\phi\}^\{\\star\}\(y^\{\\prime\}\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\|x\)\}\}\\right\)\.\(16\)It follows that we can plug in the above analytical solution into the lossℓRM\\ell\_\{\\mathrm\{RM\}\}\([13](https://arxiv.org/html/2610.31862#S4.E13)\) to obtain the following bi\-level optimization problem that finds the optimal reward parameterization:

minϕ∈Φ⁡ℓ⁡\(ϕ\)\\displaystyle\\min\_\{\\phi\\in\\Phi\}\\ \\ell\(\\phi\):=−𝔼x∼ρ,\(yl≺yw\)∼πP\(⋅\|x\)\[log\(σ\(βlog\(πϕ⋆​\(yw\|x\)πref​\(yw\|x\)\)−βlog\(πϕ⋆​\(yl\|x\)πref​\(yl\|x\)\)\)\)\]\\displaystyle:=\-\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{l\}\\prec y\_\{w\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)\}\\Big\[\\log\\Big\(\\sigma\\big\(\\beta\\log\\left\(\{\\frac\{\\pi^\{\\star\}\_\{\\phi\}\(y\_\{w\}\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi^\{\\star\}\_\{\\phi\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\}\\right\)\\big\)\\Big\)\\Big\]\(17a\)s\.t\.πϕ⋆\\displaystyle\{\\rm s\.t\.\}\\quad\\pi^\{\\star\}\_\{\\phi\}\\:=arg​maxπ∈Π𝔼x∼ρ,y∼π\(⋅\|x\)\[r\(x,y;ϕ\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x\)∥πref\(⋅\|x\)\)\],\\displaystyle:=\\argmax\_\{\\pi\\in\\Pi\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x\)\}\\left\[r\(x,y;\\phi\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\],\(17b\)whereπϕ⋆\\pi\_\{\\phi\}^\{\\star\}is an optimal policy defined in \([15](https://arxiv.org/html/2610.31862#S4.E15)\) under a certain reward model parameterized byϕ\\phi\. If one faithfully follows the above approach, then one would first compute the optimal rewardr⁡\(⋅,ϕ∗\)r\(\\cdot;\\phi^\{\*\}\), then use it to identify the optimal policy using \([15](https://arxiv.org/html/2610.31862#S4.E15)\)\. To avoid training two separate models,\[[43](https://arxiv.org/html/2610.31862#bib.bib13)\]proposes to directly parameterize the policy and find the parametersθ⋆\\theta^\{\\star\}that solve the following problem:

arg​minθ∈ΘℓDPO\(θ\):=−𝔼x∼ρ,\(yl≺yw\)∼πP\(⋅\|x\)\[log\(σ\(βlog\(π⁡\(yw\|x;θ\)πref​\(yw\|x\)\)−βlog\(π⁡\(yl\|x;θ\)πref​\(yl\|x\)\)\)\)\]\.\\displaystyle\\argmin\_\{\\theta\\in\\Theta\}\\ \\ell\_\{\\rm DPO\}\(\\theta\):=\-\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{l\}\\prec y\_\{w\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)\}\\left\[\{\\log\\left\(\{\\sigma\\left\(\{\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{w\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{l\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\}\\right\)\}\\right\)\}\\right\)\}\\right\]\.\(18\)Notice that, as shown in Protocol[9](https://arxiv.org/html/2610.31862#alg9), an empirical estimate of this loss can be computed without the need of the dataset𝒟prompts\\mathcal\{D\}\_\{\\rm prompts\}and online generations\. A first clear advantage of DPO over RLHF is that only the policy parameter is kept in memory\. The reward network does not need to be stored, therefore, as shown in the Fig\.[5](https://arxiv.org/html/2610.31862#S4.F5), DPO is more memory efficient than RLHF\. Further, no self\-generation such as line 3 of Protocol[5](https://arxiv.org/html/2610.31862#alg5)and[6](https://arxiv.org/html/2610.31862#alg6)is needed, so DPO is much easier to implement as compared to RL based algorithms\.

Table 1:The policyπDPO\\pi\_\{\\mathrm\{DPO\}\}minimizesℓDPO\\ell\_\{\\mathrm\{DPO\}\}for anyα∈\[0,1\)\\alpha\\in\[0,1\)\. Hence, forα\>0\\alpha\>0, it produces a policy with support which is not a subset ofπref\\pi\_\{\\mathrm\{ref\}\}\. The same can not happen forπRLHF\\pi\_\{\\mathrm\{RLHF\}\}that minimizesℓRLHF\\ell\_\{\\mathrm\{RLHF\}\}\.However, dropping the analytical form ofπϕ⋆​\(y\|x\)\\pi\_\{\\phi\}^\{\\star\}\(y\|x\)in the last step of the DPO derivation creates practical differences between the performance of DPO and RLHF\. Therefore, DPO and RLHF are not equivalent, in particular, since we dropped a constraint, the solution set for DPO is a superset of the solution set for RLHF \(Theorem 4\.1 in\[[67](https://arxiv.org/html/2610.31862#bib.bib18)\]\)\. Recent studies suggest that RL\-based fine\-tuning has greater potential to enhance the performance of LLMs compared to the DPO\[[67](https://arxiv.org/html/2610.31862#bib.bib18)\]

Protocol 9DPO \(Direct Preference Optimization\)1:The learner receives as input:

- •A preference dataset𝒟pref=\{\(xi,ywi,yℓi\)\}i=1N\\mathcal\{D\}\_\{\\rm pref\}=\\\{\(x^\{i\},y^\{i\}\_\{w\},y^\{i\}\_\{\\ell\}\)\\\}^\{N\}\_\{i=1\}\.
- •A policy function parameters classΘ\\Theta\.

2:The learner computes the stochastic loss\.

ℓ^DPO​\(θ\):=−𝔼\(x,yw,yl\)∈𝒟pref​\[log⁡\(σ⁡\(β​log⁡\(π⁡\(yw\|x;θ\)πref​\(yw\|x\)\)−β​log⁡\(π⋆​\(yl\|x;θ\)πref​\(yl\|x\)\)\)\)\]\\widehat\{\\ell\}\_\{\\mathrm\{DPO\}\}\(\\theta\):=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\_\{\\rm pref\}\}\\left\[\{\\log\\left\(\{\\sigma\\left\(\{\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{w\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi^\{\\star\}\(y\_\{l\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\}\\right\)\}\\right\)\}\\right\)\}\\right\]
3:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}with

θ⋆=arg​minθ∈Θ⁡ℓ^DPO​\(θ\)\\theta^\{\\star\}=\\argmin\_\{\\theta\\in\\Theta\}\\widehat\{\\ell\}\_\{\\mathrm\{DPO\}\}\(\\theta\)\.

Table 2:On HH\-RLHF, GPT\-4 is used to decide the winner between PPO and DPO models’ output\. Taken from\[[67](https://arxiv.org/html/2610.31862#bib.bib18)\]\.From the formulation of DPO in Eq\. \([18](https://arxiv.org/html/2610.31862#S4.E18)\), we see that DPO only fits the given preference data and does not leverage online generations from its own model \(see line 3 of Protocol[5](https://arxiv.org/html/2610.31862#alg5)and[6](https://arxiv.org/html/2610.31862#alg6)\)\. Due to the lack of the optimal policy constraint, the model trained by the DPO method can suffer from the limited generalization capability and can encounter the distribution shift issue when the coverage of the preference dataset is limited\.

Furthermore, when there exists distribution shift between the model outputs and the preference dataset, the performance of DPO can be significantly affected\. A possible way to circumvent this problem is to use DPO in an online fashion\[[40](https://arxiv.org/html/2610.31862#bib.bib1)\]\. In contrast, in the RLHF training pipeline, the policy model is trained by RL and has leveraged online generations from the model itself, which can alleviate the distribution shift issue between model outputs and the preference dataset through leveraging the generalization power from the explicit estimated reward model\. Similar empirical observations for the potential drawbacks of DPO compared with RLHF / PPO has also been observed in recent study\[[67](https://arxiv.org/html/2610.31862#bib.bib18)\]\. Hence, to unlock the full potential of LLMs, it appears that RL is a necessary step in the post\-training pipeline\.

The toy example in\[[67](https://arxiv.org/html/2610.31862#bib.bib18)\]that we report in Table[1](https://arxiv.org/html/2610.31862#S4.T1)shows that the solution set of DPO is a strict superset of the solution set of RLHF\. In particular, there are minimizers of the DPO loss assign positive probability to answers for which no information is available \(y3y\_\{3\}in this example\)\. This example shows that DPO should not be applied when the preference dataset has poor coverage of the answers set𝒴\\mathcal\{Y\}\. For more practical problems, it is also observed that DPO can be less performant in terms of winrate against the model fine tuned via RLHF; see Table[2](https://arxiv.org/html/2610.31862#S4.T2)\.

### 4\.4Learning from preference without the BT assumption\.

Regardless of using RLHF and the DPO approach to perform alignment, one key step is to use the BT model \([13](https://arxiv.org/html/2610.31862#S4.E13)\) to model human preference\. The BT assumption implies transitivity\. This is restrictive because the preference dataset collected from different individuals might not be transitive even if each person follows a transitive model in generating the preference\. As an example, let us consider33individualsh1,h2,h3h\_\{1\},h\_\{2\},h\_\{3\}and33possible answersy1,y2,y3y\_\{1\},y\_\{2\},y\_\{3\}\. Let us denotePhP\_\{h\}the preference model of individualshh\. Suppose that they follow the following preference models:

𝒫h1​\(y1≻y2\)=1𝒫h1​\(y2≻y3\)=0𝒫h1​\(y3≻y1\)=1\\displaystyle\\mathcal\{P\}\_\{h\_\{1\}\}\(y\_\{1\}\\succ y\_\{2\}\)=1\\quad\\mathcal\{P\}\_\{h\_\{1\}\}\(y\_\{2\}\\succ y\_\{3\}\)=0\\quad\\mathcal\{P\}\_\{h\_\{1\}\}\(y\_\{3\}\\succ y\_\{1\}\)=1𝒫h2​\(y1≻y2\)=0𝒫h2​\(y2≻y3\)=1𝒫h2​\(y3≻y1\)=1\\displaystyle\\mathcal\{P\}\_\{h\_\{2\}\}\(y\_\{1\}\\succ y\_\{2\}\)=0\\quad\\mathcal\{P\}\_\{h\_\{2\}\}\(y\_\{2\}\\succ y\_\{3\}\)=1\\quad\\mathcal\{P\}\_\{h\_\{2\}\}\(y\_\{3\}\\succ y\_\{1\}\)=1𝒫h3​\(y1≻y2\)=1𝒫h3​\(y2≻y3\)=1𝒫h3​\(y3≻y1\)=0\.\\displaystyle\\mathcal\{P\}\_\{h\_\{3\}\}\(y\_\{1\}\\succ y\_\{2\}\)=1\\quad\\mathcal\{P\}\_\{h\_\{3\}\}\(y\_\{2\}\\succ y\_\{3\}\)=1\\quad\\mathcal\{P\}\_\{h\_\{3\}\}\(y\_\{3\}\\succ y\_\{1\}\)=0\.Notice, that each of these models is transitive\. However, the average model defined as𝒫⁡\(y≻y′\)=13​∑h∈\{h1,h2,h3\}𝒫h​\(y≻y′\)\\mathcal\{P\}\(y\\succ y^\{\\prime\}\)=\\frac\{1\}\{3\}\\sum\_\{h\\in\\\{h\_\{1\},h\_\{2\},h\_\{3\}\\\}\}\\mathcal\{P\}\_\{h\}\(y\\succ y^\{\\prime\}\)satisfies𝒫⁡\(y1≻y2\)=𝒫⁡\(y2≻y3\)=𝒫⁡\(y3≻y1\)=2/3\\mathcal\{P\}\(y\_\{1\}\\succ y\_\{2\}\)=\\mathcal\{P\}\(y\_\{2\}\\succ y\_\{3\}\)=\\mathcal\{P\}\(y\_\{3\}\\succ y\_\{1\}\)=2/3\. That is, the average model is non transitive and can not be modeled by the BT assumption\. In the sense, the BT assumption is data wasteful\. In this example, one should consider preferences only from a single individual in order to make the BT assumption valid\. Not enforcing the BT assumption, allows to use more data, i\.e\. preferences from all the three individuals\. Given this motivation, recent works\[[37](https://arxiv.org/html/2610.31862#bib.bib9)\]investigated learning from preferences directly\. The training happens in two stages as in RLHF: the first step uses maximum likelihood to learn a preference model rather than a reward function, and a second step identifies a Nash equilibrium rather than an optimal policy\.

##### Preference learning

First, we learn via maximum likelihood, a preference model𝒫:𝒳×𝒴×𝒴→\[0,1\]\\mathcal\{P\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\\times\\mathcal\{Y\}\\rightarrow\[0,1\]such that𝒫⁡\(y≻y′\|x\)\\mathcal\{P\}\(y\\succ y^\{\\prime\}\|x\)is the probability that certain oracle, which produced the preferences in the dataset𝒟pref\\mathcal\{D\}\_\{\\rm pref\}, prefers answeryyovery′y^\{\\prime\}following the promptxx\.

Protocol 10NLHF \(Nash Learning from Human Feedback\)1:The learner receives as input: \(i\) A preference dataset

𝒟pref=\{\(xi,ywi,yℓi\)\}i=1N\\mathcal\{D\}\_\{\\rm pref\}=\\\{\(x^\{i\},y^\{i\}\_\{w\},y^\{i\}\_\{\\ell\}\)\\\}^\{N\}\_\{i=1\}, \(ii\) a prompt dataset

𝒟prompts=\{xi\}i=1Nprompts\\mathcal\{D\}\_\{\\mathrm\{prompts\}\}=\\\{x^\{i\}\\\}^\{N\_\{\\mathrm\{prompts\}\}\}\_\{i=1\}, \(iii\) a preference model function parameters class

Φ\\Phiand \(iv\) a policy function parameters class

Θ\\Theta\.

2:The learner estimates the preference model as

𝒫\(⋅≻⋅\|⋅;ϕ⋆\)\\mathcal\{P\}\\left\(\{\\cdot\\succ\\cdot\|\\cdot;\\phi^\{\\star\}\}\\right\)with

ϕ⋆=arg​minϕ∈Φ⁡ℓ^PM​\(ϕ\):=−𝔼\(x,yw,yl\)∈𝒟pref​log⁡\(𝒫⁡\(yw≻yl\|x;ϕ\)\)\.\\phi^\{\\star\}=\\argmin\_\{\\phi\\in\\Phi\}\\ \\widehat\{\\ell\}\_\{\\rm PM\}\(\\phi\):=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\_\{\\rm pref\}\}\\log\\Big\(\\mathcal\{P\}\(y\_\{w\}\\succ y\_\{l\}\|x;\\phi\)\\Big\)\.
3:The learner computes the stochastic objective

ℓ^NLHF​\(θ,θ′\):=\\displaystyle\\widehat\{\\ell\}\_\{\\mathrm\{NLHF\}\}\(\\theta,\\theta^\{\\prime\}\):=−𝔼x∈𝒟prompts,y∼π\(⋅\|x;θ\),y′∼π\(⋅\|x;θ′\)𝒫\(y≻y′\|x;ϕ⋆\)\\displaystyle\-\\mathbb\{E\}\_\{x\\in\\mathcal\{D\}\_\{\\rm prompts\},y\\sim\\pi\(\\cdot\|x;\\theta\),y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta^\{\\prime\}\)\}\\mathcal\{P\}\(y\\succ y^\{\\prime\}\|x;\\phi^\{\\star\}\)\+βDKL\(π\(⋅\|x;θ\)∥πref\(⋅\|x\)\)−βDKL\(π\(⋅\|x;θ′\)∥πref\(⋅\|x\)\),\\displaystyle\+\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\-\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta^\{\\prime\}\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\),
4:The learner outputs

πθ⋆\\pi\_\{\\theta^\{\\star\}\}with

θ⋆=arg​minθ∈Θ⁡maxθ′∈Θ​ℓ^NLHF​\(θ,θ′\)\\theta^\{\\star\}=\\argmin\_\{\\theta\\in\\Theta\}\\max\_\{\\theta^\{\\prime\}\\in\\Theta\}\\widehat\{\\ell\}\_\{\\mathrm\{NLHF\}\}\(\\theta,\\theta^\{\\prime\}\)\.

Mirroring the procedure in RLHF, let us consider a preference model class with parameters classΦ\\Phi, i\.e\.\{𝒫\(⋅≻⋅\|⋅;ϕ\)\}ϕ∈Φ\\left\\\{\{\\mathcal\{P\}\(\\cdot\\succ\\cdot\|\\cdot;\\phi\)\}\\right\\\}\_\{\\phi\\in\\Phi\}, consider the following minimization problem \(wherePM\{\\rm PM\}stands for preference modelling\): ϕ⋆=arg​minϕ∈ΦℓPM\(ϕ\):=−𝔼x∼ρ,\(yl≺yw\)∼πP\(⋅\|x\)\[log\(𝒫\(yw≻yl\|x;ϕ\)\)\]\.\\phi^\{\\star\}=\\argmin\_\{\\phi\\in\\Phi\}\\ \\ell\_\{\\rm PM\}\(\\phi\):=\-\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{l\}\\prec y\_\{w\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)\}\\Big\[\\log\\Big\(\\mathcal\{P\}\(y\_\{w\}\\succ y\_\{l\}\|x;\\phi\)\\Big\)\\Big\]\.Protocol[10](https://arxiv.org/html/2610.31862#alg10)shows how to do this practically, minimizing an empirical unbiased estimate forℓPM\\ell\_\{\\rm PM\}\.

##### Nash equilibrium learning

After learning the preference model,\[[37](https://arxiv.org/html/2610.31862#bib.bib9)\]proposes to find the unique Nash equilibrium of the objectiveℓ^NLHF\\widehat\{\\ell\}\_\{\\rm NLHF\}in Protocol[10](https://arxiv.org/html/2610.31862#alg10)\. Due to the min max structure of this problem, the optimization algorithm design is more delicate and require careful conservative updates to ensure last iterate convergence\. However, this additional complexity on the optimization side comes at the benefit of dropping the BT assumption and therefore accommodating a larger number of preference data\.

Experiments in\[[37](https://arxiv.org/html/2610.31862#bib.bib9), Table 1\]shows that the NLHF method dubbed MD1 achieves win rate of0\.5980\.598against the same model aligned via RLHF on a text summarization task\. The drawback of this method is that it requires solving a minmax optimization problem rather than the simpler minimization problem required in RLHF\.

### 4\.5Learning from data verifiers via Rule\-Based Reinforcement Learning

Before closing this section, we would like to mention a recently developed alignment strategy, which is fundamental for training powerful reasoning models such as DeepSeek\-R1\[[19](https://arxiv.org/html/2610.31862#bib.bib8)\]\. The general idea is simple: To teach the model to solve many challenging math, STEM or coding related problems, a reward model is no longer needed\. Rather, one can simply use a*solution verifier*, which simply verifies whether the solution is correct or not\. In\[[19](https://arxiv.org/html/2610.31862#bib.bib8)\], the authors proposed to use LeetCode tests case as verifier for programming questions, while creating a dataset of mathematical problems from OpenWebMath for math verifiers\. In particular, in the reproduction of the Deepseek R1,\[[64](https://arxiv.org/html/2610.31862#bib.bib59)\]adopted the rule\-based reward system that comprises two types of rewards: Format Reward and Answer Reward\. Here, we show the detailed reward designed in\[[64](https://arxiv.org/html/2610.31862#bib.bib59)\]as below:

rformat=\{1,if format is correct−1,if format is incorrect,ranswer=\{2,if the answer fully matches the ground truth−1\.5,if the answer partially mismatches the ground truth−2,if the answer cannot be parsed or is missing\\displaystyle r\_\{\\rm format\}=\\begin\{cases\}1,&\\text\{if format is correct\}\\\\ \-1,&\\text\{if format is incorrect\}\\end\{cases\},\\hskip 9\.24994ptr\_\{\\rm answer\}=\\begin\{cases\}2,&\\text\{if the answer fully matches the ground truth\}\\\\ \-1\.5,&\\text\{if the answer partially mismatches the ground truth\}\\\\ \-2,&\\text\{if the answer cannot be parsed or is missing\}\\end\{cases\}whererformatr\_\{\\rm format\}is the format reward andranswerr\_\{\\rm answer\}is the answer reward\.

However in\[[19](https://arxiv.org/html/2610.31862#bib.bib8)\], it is noted that the model which optimizes purely the rule based reward, i\.e\. DeepSeek\-R1\-Zero leads to poor language skills suffering for example from language mixing\. To avoid this,\[[19](https://arxiv.org/html/2610.31862#bib.bib8)\]uses multi\-stage training with learning with verifiable rewards at the initial stage and SFT/preference based learning used at the final stages to improve the language skills of the model\. In the initial stage, the rule based reward is optimized via GRPO in Algorithm[8](https://arxiv.org/html/2610.31862#alg8)\.

## 5A Unified Perspective for LLM Alignment

So far we have discussed a number of state\-of\-the\-art approaches for LLM alignment\. Generally speaking, they can be divided into two categories: alignment using the demonstration data, and alignment using the preference data\. One might wonder why algorithms belonging to these two categories are so different \(e\.g\., supervised learning vs\. reward learning \+ RL\)\. After all, both families of algorithms are trying to learn from some forms of human preference\. In this section, we provide a perspective derived from the stochastic choice model discussed in Section[3](https://arxiv.org/html/2610.31862#S3), which serves to unify different approaches of alignment\.

Recall that in Section[3](https://arxiv.org/html/2610.31862#S3), we have introduced two estimation problems \([7](https://arxiv.org/html/2610.31862#S3.E7)\) and \([8](https://arxiv.org/html/2610.31862#S3.E8)\)\. Although they use different datasets \(i\.e\., the former uses the preference dataset𝒟​pref\\mathcal\{D\}\{\\rm pref\}, while the latter uses the demonstration dataset𝒟​demo\\mathcal\{D\}\{\\rm demo\}\), they share a few common properties: \(1\) the reward function can be learned from both datasets and \(2\) both maximize some form of the likelihood function\. These observations lead to the following questions:

- •\(Q1\) Given that the formulation of the demonstration\-based problem also allows for reward learning, why do existing alignment algorithms only learn rewards from the preference dataset but not from the demonstration dataset?
- •\(Q2\) Since both problems maximize the likelihood functions, why do we need to separate the alignment process into different stages \(i\.e\., SFT and RLHF\)?

In the following, we will first address question \(Q1\) by formulating a new problem that learnsboththe reward and the policy from𝒟demon\\mathcal\{D\}\_\{\\rm demon\}\. We will show that this formulation is closely related to a class of RL problem called inverse RL \(IRL\), and we will derive efficient methods from the IRL literature\. Somewhat unexpectedly, it turns out that a number of algorithms discussed in the previous section, such as SFT and SPIN, can be viewed as special cases of the IRL\-based approach\.

Furthermore, leveraging the development in \(Q1\), we address \(Q2\) by introducing a framework that unifies the two stages of alignment problem into asinglestage\. The new framework, named Alignment with Integrated Human Feedback \(AIHF\), jointly learns a reward and a policy by maximizing the likelihood ofboth𝒟demon\\mathcal\{D\}\_\{\\rm demon\}and𝒟pref\\mathcal\{D\}\_\{\\rm pref\}\. We will discuss the connection of AIHF with various alignment algorithms introduced in the previous section\.

### 5\.1Jointly learning policy and reward from the demonstrations data

To jointly learn a reward and a policy from the demonstration dataset𝒟demon\\mathcal\{D\}\_\{\\rm demon\}, let us leverage the estimation problem \([8](https://arxiv.org/html/2610.31862#S3.E8)\) derived from the stochastic choice theory\. Given the demonstration\(x,y\)\(x,y\)sampled from an expert policyπE\\pi^\{\\rm E\}, we can directly translate \([8](https://arxiv.org/html/2610.31862#S3.E8)\) with the optimal reward defined in \([4](https://arxiv.org/html/2610.31862#S3.E4)\), into the LLM alignment setting and obtain the following problem\[[71](https://arxiv.org/html/2610.31862#bib.bib36)\]:

maxϕ\\displaystyle\\max\_\{\\phi\}\\ℓIRL\(ϕ\):=𝔼x∼ρ,y∼πE\(⋅\|x\)\[logπrϕ⋆\(y\|x\)\]\\displaystyle\\ell\_\{\\mathrm\{IRL\}\}\(\\phi\):=\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi^\{\\rm E\}\(\\cdot\|x\)\}\\left\[\\log\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\\left\(y\|x\\right\)\\right\]\(19a\)πrϕ⋆\\displaystyle\\pi^\{\\star\}\_\{r\_\{\\phi\}\}:=arg​maxπ𝔼x∼ρ,y∼π\(⋅\|x\)\[r\(x,y;ϕ\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x\)∥πref\(⋅\|x\)\)\]\.\\displaystyle:=\\argmax\_\{\\pi\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x\)\}\\left\[r\(x,y;\\phi\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\]\.\(19b\)Compared with the SFT formulation in \([9](https://arxiv.org/html/2610.31862#S4.E9)\), we see that \([19](https://arxiv.org/html/2610.31862#S5.E19)\) shares the same objective, but has an additional constraint which ensures that the policy has to be an optimal policy under a certain reward model\.

The problem \([19](https://arxiv.org/html/2610.31862#S5.E19)\) falls under the class of inverse reinforcement learning \(IRL\) problems, where the objective is to jointly learn an optimal policy and the underlying reward model, assuming access to expert policy samples\. Similarly, as noted in Sec\.[4\.1\.1](https://arxiv.org/html/2610.31862#S4.SS1.SSS1), the conventional SFT formulation \([9](https://arxiv.org/html/2610.31862#S4.E9)\) is a form of BC that aims to replicate an expert’s policy\. However, BC generally exhibits poorer generalization than IRL\. Indeed,\[[45](https://arxiv.org/html/2610.31862#bib.bib27)\]shows that BC is a suboptimal approach to learning from demonstrations, leading to regret bounds that scale quadratically with the problem horizon\. This arises due to distribution shift – the learned policy may encounter states not covered in the demonstration data, causing compounding errors that increasingly deviate from the expert trajectory over time\.

In contrast, IRL mitigates distribution shift and achieves better generalization by explicitly modeling the expert’s reward function, rather than simply mimicking actions\. By recovering an estimate of the reward structure that drives expert behavior, IRL enables the agent to make goal\-directed decisions even in unseen states\. This contrasts with BC, which lacks an underlying objective thus often struggles in novel situations\. Additionally, optimizing policies with a learned reward function allows for adaptive behavior, where the agent can deviate from expert demonstrations in a way that still aligns with the inferred intent\. As a result, IRL\-based policies are generally more robust in complex environments where state distributions differ from training data\[[56](https://arxiv.org/html/2610.31862#bib.bib58)\]\. Recognizing the limitations of standard SFT methods, particularly regarding distribution shift, researchers have proposed a range of IRL\-based techniques\[[59](https://arxiv.org/html/2610.31862#bib.bib29),[29](https://arxiv.org/html/2610.31862#bib.bib4),[60](https://arxiv.org/html/2610.31862#bib.bib60),[72](https://arxiv.org/html/2610.31862#bib.bib47)\]to more effectively leverage demonstration data\.

Unfortunately, the IRL problem \([19](https://arxiv.org/html/2610.31862#S5.E19)\) appears to be much more involved compared to the SFT problem \([9](https://arxiv.org/html/2610.31862#S4.E9)\), as it involvestwolevels of optimization problem stacked in hierarchy\. It turns out that there is a way to simplify it\. In\[[70](https://arxiv.org/html/2610.31862#bib.bib3),[73](https://arxiv.org/html/2610.31862#bib.bib49)\], the authors show that when a finite demonstration dataset𝒟demo:=\{\(x,y\)\}\\mathcal\{D\}\_\{\\text\{demo\}\}:=\\\{\(x,y\)\\\}is available, one can consider the following surrogate objective:

maxϕminθL~\(ϕ,θ;𝒟demo\):=𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅\|x;θ\)\[r\(x,y;ϕ\)−r\(x,y′;ϕ\)\+βDKL\(π\(⋅\|x;θ\)∥πref\(⋅\|x\)\)\]\.\\displaystyle\\max\_\{\\phi\}\\min\_\{\\theta\}~\\widetilde\{L\}\(\\phi,\\theta;\\mathcal\{D\}\_\{\\rm demo\}\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\},y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\)\}\\Big\[r\(x,y;\\phi\)\-r\(x,y^\{\\prime\};\\phi\)\+\\beta\{D\}\_\{\\rm KL\}\\Big\(\\pi\(\\cdot\|x;\\theta\)\\\|\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\\Big\)\\Big\]\.\(20\)As a remark, one can plug the closed\-form expression of the optimal policyπrϕ∗\\pi^\{\*\}\_\{r\_\{\\phi\}\}into the maximum likelihood objective \([19a](https://arxiv.org/html/2610.31862#S5.E19.1)\), then one can see that the surrogate objective \([20](https://arxiv.org/html/2610.31862#S5.E20)\) is a finite\-sample version of the IRL problem \([19](https://arxiv.org/html/2610.31862#S5.E19)\)\. Further, as the number of demonstration data becomes large, problem \([20](https://arxiv.org/html/2610.31862#S5.E20)\) becomes asymptotically close to problem \([19](https://arxiv.org/html/2610.31862#S5.E19)\), in the sense that their optimal solutions become asymptotically close to each other \(with provable bounds\)\[[70](https://arxiv.org/html/2610.31862#bib.bib3)\]\. The surrogate objective in \([20](https://arxiv.org/html/2610.31862#S5.E20)\) offers an alternative interpretation of the IRL problem formulated in \([19](https://arxiv.org/html/2610.31862#S5.E19)\) from the adversarial training perspective\. Specifically, the reward estimator is optimized to distinguish between expert demonstrations and self\-generated trajectories \(the max problem\), while the policy is optimized to generate trajectories which can best mimic the expert demonstrations and the self\-generated trajectories \(the min problem\)\. Therefore, it is reasonable to develop practical algorithms based on \([20](https://arxiv.org/html/2610.31862#S5.E20)\), which is apparently much easier to optimize\. As suggested in\[[73](https://arxiv.org/html/2610.31862#bib.bib49)\], one can use the following alternating algorithm to optimize the reward and the policy:

ϕt\+1\\displaystyle\\phi\_\{t\+1\}:=arg​minϕ−𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅\|x;θt\)\[ℓ\(r\(x,y;ϕ\)−r\(x,y′;ϕ\)\)\]\.\\displaystyle:=\\argmin\_\{\\phi\}~\-\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\},y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Big\[\\ell\\Big\(r\(x,y;\\phi\)\-r\(x,y^\{\\prime\};\\phi\)\\Big\)\\Big\]\.\(21a\)θt\+1\\displaystyle\\theta\_\{t\+1\}:=arg​maxθ𝔼x∼ρ,y∼π\(⋅\|x;θ\)\[r\(x,y;ϕt\+1\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x;θ\)∥πref\(⋅\|x\)\)\]\.\\displaystyle:=\\argmax\_\{\\theta\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x;\\theta\)\}\\left\[r\(x,y;\\phi\_\{t\+1\}\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\]\.\(21b\)Note that if we directly apply the above gradient descent/ascent algorithm to \([20](https://arxiv.org/html/2610.31862#S5.E20)\), thenℓ⁡\(⋅\)\\ell\(\\cdot\)in \([21a](https://arxiv.org/html/2610.31862#S5.E21.1)\) should be a linear functionℓ⁡\(x\)=x\\ell\(x\)=x\. Considering that the use of a linear loss function results in an unbounded objective value, one can typically choose some easy\-to\-optimize loss functions such as log sigmoid functionℓ⁡\(z\):=log⁡σ⁡\(z\)\\ell\(z\):=\\log\\sigma\(z\)\. Next, let us make some remarks about the algorithm \([21](https://arxiv.org/html/2610.31862#S5.E21)\)\.

*Connection to the RLHF\.*We observe that \([21](https://arxiv.org/html/2610.31862#S5.E21)\) is an iterative update scheme whichintegratesthe reward and policy updates\. This closely resembles the RLHF pipeline, where both the reward and policy are updated\. There are two major differences\. First, in RLHF the reward and the policy are updated in twoseparatestages, while here they are updatediteratively\. Second, since only demonstration data𝒟demon\\mathcal\{D\}\_\{\\rm demon\}are available, when learning a reward model in \([21a](https://arxiv.org/html/2610.31862#S5.E21.1)\), the demonstration is always treated as the preferred data, while the non\-preferred data are the model\-generated responses\. This is reasonable, under the assumption that expert data are of high quality\. In practice, RLHF\-based and IRL\-based methods are complementary to each other\. It’s common to leverage both demonstration and preference datasets jointly in the training pipeline to achieve better performance\[[23](https://arxiv.org/html/2610.31862#bib.bib40),[54](https://arxiv.org/html/2610.31862#bib.bib23)\]\. In\[[28](https://arxiv.org/html/2610.31862#bib.bib5),[62](https://arxiv.org/html/2610.31862#bib.bib61)\], the connections has been shown that both IRL and RLHF are imitating a target policy using different dataset \(demonstrations and preferences\)\. Leveraging this insight, the alignment pipeline can incorporate both demonstration and preference datasets, improving model performance by integrating RLHF with IRL techniques\. In general, it can be shown that RLHF could be a special form of IRL\. Here, we show that IRL models the preference of a single action over all possible actions in \([19a](https://arxiv.org/html/2610.31862#S5.E19.1)\) when maximizing the likelihood of one demonstration action, whereas RLHF models the relative preference between pairs of actions\.

Below we show that the reward modeling in RLHF can be derived from the optimal policy expression in \([15](https://arxiv.org/html/2610.31862#S4.E15)\)\. As we have defined in \([15](https://arxiv.org/html/2610.31862#S4.E15)\), given the reward parameterϕ\\phi, the optimal policy of the regularized RL problem defined in \([14](https://arxiv.org/html/2610.31862#S4.E14)\) has the closed\-formπrϕ⋆​\(y\|x\)=πref​\(y\|x\)​exp⁡\(β−1​r​\(x,y,ϕ\)\)Z⁡\(x,ϕ,β\)\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\(y\|x\)=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\\exp\(\\beta^\{\-1\}r\(x,y;\\phi\)\)\}\{Z\(x,\\phi,\\beta\)\}\. With this expression, we can now obtain a model for the likelihood that sequenceywy\_\{w\}is preferred overyly\_\{l\}\. With a pair of actions\{yl,yw\}\\\{y\_\{l\},y\_\{w\}\\\}, by theindependence of irrelevant alternativesproperty\[[17](https://arxiv.org/html/2610.31862#bib.bib35)\]of the optimal choiceπrϕ∗\\pi^\{\*\}\_\{r\_\{\\phi\}\}, the likelihood that sequenceywy\_\{w\}is preferred overyly\_\{l\}is given byℙπrϕ∗​\(yw≻yl\|x\):=πrϕ∗​\(yw\|x\)πrϕ∗​\(yw\|x\)\+πrϕ∗​\(yl\|x\)\.\\mathbb\{P\}\_\{\\pi^\{\*\}\_\{r\_\{\\phi\}\}\}\(y\_\{w\}\\succ y\_\{l\}\|x\):=\\frac\{\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{w\}\|x\)\}\{\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{w\}\|x\)\+\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{l\}\|x\)\}\.This motivates the choice model as the followinglikelihood function:

L⁡\(R⁡\(⋅,ϕ\)\)\\displaystyle L\(R\(\\cdot;\\phi\)\)=𝔼\(yw≻yl\)​\[log⁡πrϕ∗​\(yw\|x\)πrϕ∗​\(yw\|x\)\+πrϕ∗​\(yl\|x\)\]\\displaystyle=\\mathbb\{E\}\_\{\(y\_\{w\}\\succ y\_\{l\}\)\}\\Big\[\\log\\frac\{\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{w\}\|x\)\}\{\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{w\}\|x\)\+\\pi^\{\*\}\_\{r\_\{\\phi\}\}\(y\_\{l\}\|x\)\}\\Big\]=𝔼\(yw≻yl\)​\[log⁡πref​\(yw\|x\)​exp⁡\(β−1​r​\(x,yw,ϕ\)\)πref​\(yw\|x\)​exp⁡\(β−1​r​\(x,yw,ϕ\)\)\+πref​\(yl\|x\)​exp⁡\(β−1​r​\(x,yl,ϕ\)\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\(y\_\{w\}\\succ y\_\{l\}\)\}\\Big\[\\log\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\\exp\(\\beta^\{\-1\}r\(x,y\_\{w\};\\phi\)\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\\exp\(\\beta^\{\-1\}r\(x,y\_\{w\};\\phi\)\)\+\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\\exp\(\\beta^\{\-1\}r\(x,y\_\{l\};\\phi\)\)\}\\Big\]\.Whenπref\\pi\_\{\\mathrm\{ref\}\}equals to the uniform distribution, this model is equivalent to the BTL model in RLHF:

L⁡\(R⁡\(⋅,ϕ\)\)=ℓRM​\(ϕ\):=−𝔼\(yw≻yl\)​\[log⁡\(σ⁡\(r⁡\(x,yw,ϕ\)−r⁡\(x,yl,ϕ\)\)\)\]\.\\displaystyle L\(R\(\\cdot;\\phi\)\)=\\ell\_\{\\rm RM\}\(\\phi\):=\-\\mathbb\{E\}\_\{\(y\_\{w\}\\succ y\_\{l\}\)\}\\Big\[\\log\\Big\(\\sigma\\big\(r\(x,y\_\{w\};\\phi\)\-r\(x,y\_\{l\};\\phi\)\\big\)\\Big\)\\Big\]\.\(22\)For the IRL formulation defined in \([19](https://arxiv.org/html/2610.31862#S5.E19)\), we see that it models the preference of a single action over the entire action space\. When the action space consists of only two actions, we see that RLHF is a special case of IRL and the reward modeling between IRL and RLHF is equivalent\.

*Connection to SPIN\.*There is also an interesting connection between \([20](https://arxiv.org/html/2610.31862#S5.E20)\) and the SPIN introduced in Sec\.[4\.1\.2](https://arxiv.org/html/2610.31862#S4.SS1.SSS2)\. Note that in \([16](https://arxiv.org/html/2610.31862#S4.E16)\), we have shown that the reward difference can be expressed by using the definition of the optimal policy\. By plugging \([16](https://arxiv.org/html/2610.31862#S4.E16)\) into \([21a](https://arxiv.org/html/2610.31862#S5.E21.1)\), we obtain the following problem:

minϕ∈Φ⁡ℓ⁡\(ϕ\)\\displaystyle\\min\_\{\\phi\\in\\Phi\}\\ \\ell\(\\phi\):=−𝔼\(x,y\)∼𝒟,y′∼π\(⋅\|x;θt\)\[ℓ\(βlog\(πrϕ⋆​\(y\|x\)πref​\(y\|x\)\)−βlog\(πrϕ⋆​\(y′\|x\)πref​\(y′\|x\)\)\)\]\\displaystyle:=\-\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\},y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Big\[\\ell\\Big\(\\beta\\log\\left\(\{\\frac\{\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\(y^\{\\prime\}\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\|x\)\}\}\\right\)\\Big\)\\Big\]s\.t\.πrϕ⋆\\displaystyle s\.t\.\\quad\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\\:=arg​maxπ𝔼x∼ρ,y∼π\(⋅\|x\)\[r\(x,y;ϕ\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x\)∥πref\(⋅\|x\)\)\]\.\\displaystyle:=\\argmax\_\{\\pi\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x\)\}\\left\[r\(x,y;\\phi\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\]\.Similarly, as discussed in the DPO approach, one can make two simplifications \(1\) remove the constraints that the policy has to be optimal w\.r\.t\. some reward; \(2\) directly parameterize the resulting policy usingθ\\theta\. Then the above problem will be reduced precisely to the SPIN formulation \([12](https://arxiv.org/html/2610.31862#S4.E12)\)\.

To further compare the SPIN and IRL, note that at each iteration, once the synthetic preference dataset has been constructed where the preferred data are the demonstration / SFT data and the nonpreferred data are the model\-generated data, SPIN directly leverages the synthetic preference dataset and runs DPO to update the policy, while the IRL approach considerstwoseparate steps to updateboththe reward model and the policy model\. Similarly to the comparison of DPO and RLHF, here we expect that IRL enjoys better generalization and can alleviate the distribution shift issue through leveraging online generations in the RL training step, at the cost of a heavier memory burden and additional online generations\.

In summary, the method discussed in this section addressed the question \(Q1\) we raised at the beginning of this section – instead of directly performing SFT on the demonstration data𝒟demon\\mathcal\{D\}\_\{\\rm demon\}, it is also possible to learn a reward function from such a dataset, and we expect that the resulting model will generalize better\. Later we will use numerical results to confirm such an advantage\.

### 5\.2Jointly learning policy and reward from Integrated Human Feedback

Next, let us address question \(Q2\), by developing an approach thatintegratesdifferent steps of the alignment process\. Toward this end, we observe that by leveraging the IRL\-based formulation \([19](https://arxiv.org/html/2610.31862#S5.E19)\), learning from the demonstration data𝒟demon\\mathcal\{D\}\_\{\\rm demon\}becomes very similar to the standard RLHF approach discussed in Sec\.[4\.2](https://arxiv.org/html/2610.31862#S4.SS2), which learns preference data𝒟pref\\mathcal\{D\}\_\{\\rm pref\}– both maximize the likelihood of observing the data, while explicitly learning a reward function\. Therefore, using reward learning as a common component, it is straightforward to integrate these problems together\. In\[[28](https://arxiv.org/html/2610.31862#bib.bib5)\]the authors considered the following problem, which extends the IRL problem \([19](https://arxiv.org/html/2610.31862#S5.E19)\) by adding the BT loss \([13](https://arxiv.org/html/2610.31862#S4.E13)\):

maxϕ\\displaystyle\\max\_\{\\phi\}\\ℓAIHF\(ϕ\):=γ𝔼x∼ρ,y∼πE\(⋅\|x\)\[logπrϕ⋆\(y\|x\)\]\\displaystyle\\ell\_\{\\mathrm\{AIHF\}\}\(\\phi\):=\\gamma\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi^\{\\rm E\}\(\\cdot\|x\)\}\\left\[\\log\\pi^\{\\star\}\_\{r\_\{\\phi\}\}\\left\(y\|x\\right\)\\right\]\+α𝔼x∼ρ,\(yl≺yw\)∼πP\(⋅\|x\)\[ℓ\(r\(x,yw;ϕ\)−r\(x,yl;ϕ\)\)\]\\displaystyle\\quad\\quad\\quad\\quad\\quad\+\\alpha\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{l\}\\prec y\_\{w\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)\}\\Big\[\\ell\\Big\(r\(x,y\_\{w\};\\phi\)\-r\(x,y\_\{l\};\\phi\)\\Big\)\\Big\]\(23a\)πrϕ⋆\\displaystyle\\pi^\{\\star\}\_\{r\_\{\\phi\}\}:=arg​maxπ𝔼x∼ρ,y∼π\(⋅\|x\)\[r\(x,y;ϕ\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x\)∥πref\(⋅\|x\)\)\],\\displaystyle:=\\argmax\_\{\\pi\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x\)\}\\left\[r\(x,y;\\phi\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\],\(23b\)whereγ≥0\\gamma\\geq 0andα≥0\\alpha\\geq 0are two tuneable parameters\. The authors refer to this approach as AIHF \(Alignment with Integrated Human Feedback\)\.

To design an algorithm to solve problem \([23](https://arxiv.org/html/2610.31862#S5.E23)\), one can combine the RLHF Protocol[4](https://arxiv.org/html/2610.31862#alg4)and the IRL algorithm in \([21](https://arxiv.org/html/2610.31862#S5.E21)\), to obtain the following iterative algorithm \(wherettdenotes the iteration number\):

ϕt\+1\\displaystyle\\phi\_\{t\+1\}:=arg​minϕ−γ𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅\|x;θt\)\[ℓ\(r\(x,y;ϕ\)−r\(x,y′;ϕ\)\)\]\\displaystyle:=\\argmin\_\{\\phi\}~\-\\gamma\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\},y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Big\[\\ell\\Big\(r\(x,y;\\phi\)\-r\(x,y^\{\\prime\};\\phi\)\\Big\)\\Big\]−α​𝔼\(x,yw,yl\)∼𝒟pref​\[ℓ⁡\(r⁡\(x,yw,ϕ\)−r⁡\(x,yl,ϕ\)\)\],\\displaystyle\\quad\\quad\\quad\\quad\\quad\\quad\\quad\\,\\,\\,\-\\alpha\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\_\{\\rm pref\}\}\\Big\[\\ell\\big\(r\(x,y\_\{w\};\\phi\)\-r\(x,y\_\{l\};\\phi\)\\big\)\\Big\],\(24a\)θt\+1\\displaystyle\\theta\_\{t\+1\}:=arg​maxθ𝔼x∼ρ,y∼π\(⋅\|x;θ\)\[r\(x,y;ϕt\+1\)\]−β𝔼x∼ρ\[DKL\(π\(⋅\|x;θ\)∥πref\(⋅\|x\)\)\]\.\\displaystyle:=\\argmax\_\{\\theta\}~~\\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi\(\\cdot\|x;\\theta\)\}\\left\[r\(x,y;\\phi\_\{t\+1\}\)\\right\]\-\{\\beta\}\\mathbb\{E\}\_\{x\\sim\\rho\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\\left\(\\cdot\|x;\\theta\\right\)\\\|\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\|x\\right\)\)\]\.\(24b\)In fact, compared to the IRL algorithm \([21](https://arxiv.org/html/2610.31862#S5.E21)\), the only difference is that the reward is learned frombothpreference dataset and the demonstration dataset\. In\[[28](https://arxiv.org/html/2610.31862#bib.bib5)\], the authors observed that learning reward function from a diverse set of data can be more robust to situations, e\.g\. when there is an imbalance between the demonstration and preference data, resulting in a more robust policy model; see\[[28](https://arxiv.org/html/2610.31862#bib.bib5), Sec\.3\.4\]for detailed discussion and a few examples showcasing such an advantage\.

Finally, we note that AIHF has several notable special cases\. First, if one choosesγ=0\\gamma=0, then reward learning and policy learning becomecompletely decoupled, reducing to the orignal two\-step RLHF approach discussed in Section[4\.2](https://arxiv.org/html/2610.31862#S4.SS2)\. Second, similarly to DPO, one can plug in the optimal solution of the lower\-level problem \(as given in \([16](https://arxiv.org/html/2610.31862#S4.E16)\)\) to the upper\-level, remove the lower\-level constraint \([23b](https://arxiv.org/html/2610.31862#S5.E23.2)\), and directly parameterize the policy\. The resulting problem becomes:

maxθ𝔼x∼ρ,y∼πE\(⋅\|x;θ\)\[logπ\(y\|x\)\]\+α𝔼x∼ρ,\(yl≺yw\)∼πP\(⋅\|x\)\[ℓ\(βlog\(π⁡\(yw\|x;θ\)πref​\(yw\|x\)\)−βlog\(π⁡\(yl\|x;θ\)πref​\(yl\|x\)\)\)\]\.\\max\_\{\\theta\}\\ \\mathbb\{E\}\_\{x\\sim\\rho,y\\sim\\pi^\{\\rm E\}\(\\cdot\|x;\\theta\)\}\\left\[\\log\\pi\\left\(y\|x\\right\)\\right\]\+\\alpha\\mathbb\{E\}\_\{x\\sim\\rho,\(y\_\{l\}\\prec y\_\{w\}\)\\sim\\pi^\{P\}\(\\cdot\|x\)\}\\Big\[\\ell\\Big\(\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{w\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{l\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\}\\right\)\\Big\)\\Big\]\.This corresponds to the Regularized Preference Optimization \(RPO\) approach proposed in\[[31](https://arxiv.org/html/2610.31862#bib.bib50)\]\. Moreover, if we drop \([24b](https://arxiv.org/html/2610.31862#S5.E24.2)\) and apply the DPO trick to \([24](https://arxiv.org/html/2610.31862#S5.E24)\) to optimize:

maxθ\\displaystyle\\max\_\{\\theta\}𝔼\(x,y\)∼𝒟demo,y′∼π\(⋅\|x;θt\)\[ℓ\(βlog\(π⁡\(y\|x;θ\)πref​\(y\|x\)\)−βlog\(π⁡\(y′\|x;θ\)πref​\(y′\|x\)\)\)\]\\displaystyle\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{\\rm demo\},y^\{\\prime\}\\sim\\pi\(\\cdot\|x;\\theta\_\{t\}\)\}\\Big\[\\ell\\Big\(\\beta\\log\\left\(\{\\frac\{\\pi\(y\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\(y^\{\\prime\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\|x\)\}\}\\right\)\\Big\)\\Big\]\+α​𝔼\(x,yw,yl\)∼𝒟pref​\[ℓ⁡\(β​log⁡\(π⁡\(yw\|x;θ\)πref​\(yw\|x\)\)−β​log⁡\(π⁡\(yl\|x;θ\)πref​\(yl\|x\)\)\)\]\.\\displaystyle\+\\alpha\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\_\{\\rm pref\}\}\\Big\[\\ell\\Big\(\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{w\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\}\}\\right\)\-\\beta\\log\\left\(\{\\frac\{\\pi\(y\_\{l\}\|x;\\theta\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\}\\right\)\\Big\)\\Big\]\.Then it corresponds to the Self\-Play with Adversarial Critic \(SPAC\) approach proposed in\[[24](https://arxiv.org/html/2610.31862#bib.bib45)\]\.

Finally, we conclude this section with a tabular comparison of different types of alignment algorithms\.

Table 3:Comparison of alignment algorithms by reward requirement, dataset type, and value\-function \(critic\) dependence\. Here since AIHF can use different kinds of RL algorithms such as PPO and DPO, we denote it as ‘flexible’ in the last column\.Table[3](https://arxiv.org/html/2610.31862#S5.T3)summarizes different alignment algorithms according to their data sources and optimization requirements\. Demonstration\-based methods \(e\.g\., SFT, SPIN, IRL\) rely solely on expert trajectories, whereas preference\-based methods \(e\.g\., DPO, PPO, RLOO\) depend on pairwise comparisons or on demonstration data when initialized from an SFT model\. Among these, PPO\-style variants explicitly require a value function to compute advantages, while most direct preference\-modeling approaches remove the need for critic estimation\. Hybrid frameworks such as AIHF integrate both demonstration and preference data, effectively bridging supervised and reinforcement\-based alignment paradigms\.

## 6Dataset, Benchmarks, and Evaluation

In this section, we will introduce a few datasets and benchmarks that are often used to evaluate alignment algorithms\. Further, we will present some evaluation results comparing a number of algorithms we have discussed so far\.

### 6\.1Dataset

Typically, datasets used in LLM alignment can be categorized into two types: \(1\) datasets containing multiple responses for a given prompt, with human \(or AI\-generated\) rankings indicating preference, and \(2\) datasets containing a single demonstration response per prompt\.

Preference Dataet\. Some of the most well\-known preference datasets include: \(1\) HH\-RLHF111Available at[https://huggingface\.co/datasets/Anthropic/hh\-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)\.\(Anthropic’s Helpful\-Harmless RLHF\), a dataset collected by Anthropic that includes human preference annotations over pairs of responses, focusing on helpfulness \(how well the response answers the query\) and harmlessness \(avoiding toxic, biased, or dangerous content\)\. \(2\) UltraFeedback222Available at[https://huggingface\.co/datasets/openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback)\., which consists of 64K prompts, each accompanied by four model completions from a diverse set of open and proprietary models\. Preference labels are provided by GPT\-4 based on criteria such as helpfulness and honesty\. This dataset has been used to train state\-of\-the\-art chat models such as Zephyr\-7B\-β\\beta\.

Demonstration Dataset\. Some of the most well\-known demonstration datasets include: \(1\) the TL;DR dataset333Available at[https://huggingface\.co/datasets/CarperAI/openai\_summarize\_tldr](https://huggingface.co/datasets/CarperAI/openai_summarize_tldr)\., a dataset collected from Reddit where users provide short summaries \(TL;DRs\) of their posts\. \(2\) UltraChat444Available at[https://huggingface\.co/datasets/stingning/ultrachat](https://huggingface.co/datasets/stingning/ultrachat)\., an open\-source, large\-scale, multi\-turn dialogue dataset generated using Turbo APIs\. It contains 774K training samples and is designed to enhance instruction\-following capabilities\.

Self\-Generated DatasetIn addition to using public datasets, another line of research leverages self\-generated data for policy optimization\. For example, given a prompt,\[[1](https://arxiv.org/html/2610.31862#bib.bib62)\]samples one or more responses from a policyπ\\pi, evaluates each input\-generation pair using a evaluation function \(e\.g\., a reward model in RLHF\), labels them as preferred or dispreferred, and then updates the policy using the EM algorithm based on these individual preferences\. Similarly,\[[65](https://arxiv.org/html/2610.31862#bib.bib63)\]also employs self\-generated data to iteratively improve the policy model\. This approach can lead to more uniform coverage of the prompt\-response space\.

### 6\.2Benchmark datasets and benchmark models

AlpacaEval\. AlpacaEval\[[15](https://arxiv.org/html/2610.31862#bib.bib54)\]is an LLM\-based automated evaluation metric – it operates on a fixed set of 805 instructions chosen to be representative of user interactions on the Alpaca web demo\. A GPT\-4 turbo\-based evaluator then compares the responses head\-to\-head and outputs the probability of preferring the evaluated model\. The win rate is then computed as the expected probability that the auto\-evaluator prefers the evaluated model’s output on the 805 instructions\. This win rate serves as a performance measure of the evaluated LM chatbot\.

Open LLM Leaderboard\. The HuggingFace Open LLM Leaderboard framework\[[6](https://arxiv.org/html/2610.31862#bib.bib19)\]is widely adopted for evaluating LLMs\. This comprehensive evaluation suite assesses LLM performance across six key tasks: commonsense reasoning, including ARC, HellaSwag, and Winogrande; multi\-task language understanding, as measured by MMLU; truthfulness assessment, which evaluates a model’s tendency to mimic human falsehoods using TruthfulQA; and mathematical problem\-solving, assessed via GSM8K\.

MT\-Bench\. MT\-Bench\[[75](https://arxiv.org/html/2610.31862#bib.bib55)\]is a benchmark consisting of 80 high\-quality multi\-turn questions, specifically designed to evaluate multi\-turn conversation and instruction\-following capabilities in LLMs\. It focuses on common real\-world use cases while incorporating challenging questions to effectively differentiate model performance\. To ensure comprehensive coverage, the benchmark is structured around eight key categories of user prompts: writing, roleplay, extraction, reasoning, mathematics, coding, knowledge I \(STEM\), and knowledge II \(humanities/social sciences\)\.

HELMHolistic Evaluation of Language Models \(HELM\)\[[30](https://arxiv.org/html/2610.31862#bib.bib30)\]consists of two levels: \(i\) an abstract taxonomy of scenarios and metrics that defines the design space for language model evaluation, and \(ii\) a concrete set of implemented scenarios and metrics selected to prioritize coverage \(e\.g\., different English varieties\), value \(e\.g\., user\-facing applications\), and feasibility \(e\.g\., limited engineering resources\)\. HELM currently includes a core set of 16 scenarios and 7 metrics\. These scenarios cover six user\-facing tasks \(e\.g\., question answering, information retrieval, summarization, toxicity detection\) and span multiple domains \(e\.g\., news, books\)\.

Artificial Analysis555Available at[https://artificialanalysis\.ai/leaderboards/models](https://artificialanalysis.ai/leaderboards/models)\.Artificial Analysis provides objective evaluations and insights to assist developers, researchers, businesses, and other AI users in selecting the most suitable AI technologies for their specific use cases\. This benchmark compares and ranks the performance of over 30 LLMs across key metrics, including output quality, cost, performance, and speed, as well as context window size and other relevant factors\.

Reasoning Benchmark\. Several challenging benchmarks are also used to evaluate reasoning ability\. For example, mathematical benchmarks such as AIME \(American Invitational Mathematics Examination\), AMC \(American Mathematics Competitions\), and OlympiadBench\[[21](https://arxiv.org/html/2610.31862#bib.bib65)\]assess problem\-solving skills in competitive mathematics\. Additionally, datasets like the Knights and Knaves \(K&K\) puzzles\[[63](https://arxiv.org/html/2610.31862#bib.bib56)\]and LeetCode are commonly used for evaluating logical reasoning and coding proficiency\.

RewardBench\. The RewardBench\[[27](https://arxiv.org/html/2610.31862#bib.bib52)\]offers a comprehensive set of evaluations for reward models, encompassing key aspects such as chat, instruction\-following, coding, safety, and other critical metrics for fine\-tuned language models\. The RwardBench dataset includes a combination of pre\-existing evaluation prompt\-completion pairs as well as newly curated examples specifically for this project\.

### 6\.3Numerical Results

In this subsection, we will present some evaluations results of different alignment approaches\. First, we will compare IRL approach \([21](https://arxiv.org/html/2610.31862#S5.E21)\) with other learning\-from\-demonstration approaches like SFT \(Protocol[1](https://arxiv.org/html/2610.31862#alg1)\) and SPIN \(Protocol[2](https://arxiv.org/html/2610.31862#alg2)\) to illustrate their differences\. Then we will compare a number of algorithms that leverage preference data to further improve the LLM alignment\.

#### 6\.3\.1Demonstration\-only setting

We conduct experiments on the UltraChat dataset\[[13](https://arxiv.org/html/2610.31862#bib.bib26)\]\. Both the policy and reward models are initialized from the public checkpoint Mistral\-7b\-SFT\-Beta666HuggingFaceH4/mistral\-7b\-sft\-beta,[https://huggingface\.co/HuggingFaceH4/mistral\-7b\-sft\-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta), which is an SFT model fine\-tuned on the UltraChat dataset using the base model Mistral\-7B\-v0\.1777mistralai/Mistral\-7B\-v0\.1,[https://huggingface\.co/mistralai/Mistral\-7B\-v0\.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)\.

Since UltraChat contains only demonstration data, we construct a synthetic preference dataset to train the reward model for IRL approaches\. In each IRL iteration, we treat the demonstration data from UltraChat as preferred responses and the outputs generated by the IRL policy model as rejected responses\. This method enables the creation of preference pairs without explicit human annotations\.

\(a\)Policy Evaluations on Open LLM Leaderboard\(b\)Reward Evaluations on Reward Bench
Figure 6:Left:Evaluation of policy models on the Open LLM Leaderboard\.Right:Evaluation of reward models using RewardBench, where reward accuracy measures the proportion of preference pairs in which the reward model assigns a higher score to the preferred response\. Results show that high\-quality demonstration datasets can effectively enhance both reward and policy models via the IRL method\.We assess the performance of reward models derived from several baselines using RewardBench\[[27](https://arxiv.org/html/2610.31862#bib.bib52)\], focusing on categories pertinent to language understanding and generation\. Since SPIN lacks an explicit reward model, we evaluate its performance based on an implicit reward formulated asr⁡\(s,a\)=log⁡π⁡\(a\|s\)πref​\(a\|s\)r\(s,a\)=\\log\\frac\{\\pi\(a\|s\)\}\{\\pi\_\{\\rm ref\}\(a\|s\)\}, motivated by \([16](https://arxiv.org/html/2610.31862#S4.E16)\)\. In our experiments,πref\\pi\_\{\\rm ref\}represents the SFT model\. For the SFT model itself, we measure performance using the implicit rewardr⁡\(s,a\)=log⁡πSFT​\(a\|s\)r\(s,a\)=\\log\\pi\_\{\\rm SFT\}\(a\|s\)\.

The results, summarized in Table[4](https://arxiv.org/html/2610.31862#S6.T4), indicate that the IRL\-trained reward model outperforms both the base model \(initialized from the SFT model\) and the policy model trained with SPIN\[[10](https://arxiv.org/html/2610.31862#bib.bib15)\]\. This demonstrates

Table 4:Evaluation of Policy Models in MT\-Bench\. The results show that the IRL method can effectively improve the quality of policy models on multi\-turn questions\.that integrating high\-quality demonstration datasets with IRL methods, which generate synthetic preference pairs by combining demonstrations and model outputs, can significantly enhance reward model quality\. Furthermore, Figure[7](https://arxiv.org/html/2610.31862#S6.F7)illustrates the comparative evaluation of alignment methods when both demonstrations and preference dataset are available\. The numerical results show that the IRL\-based method is compatible with RLHF method to to enhance model alignment through integrating both demonstration and preference dataset\.

Figure 7:Performance comparison of alignment methods\. The AIHF\-trained model demonstrates superior alignment across various downstream tasks\.We evaluate various LLM policy models using the Open LLM Leaderboard\[[18](https://arxiv.org/html/2610.31862#bib.bib20)\]and MT\-Bench\[[75](https://arxiv.org/html/2610.31862#bib.bib55)\]\. As shown in Fig\.[6](https://arxiv.org/html/2610.31862#S6.F6)and Table[4](https://arxiv.org/html/2610.31862#S6.T4), the IRL\-based method consistently outperforms both the mistral\-7b\-sft\-beta checkpoint and the SPIN method \(regardless of whether self\-generation is performed once or twice\)\. These results demonstrate the effectiveness of leveraging high\-quality demonstrations and synthetic preferences to improve language model performance in dialogue generation tasks\.

#### 6\.3\.2Aligning with Both Demonstrations and Pairwise Comparisons

In this section, we evaluate several alignment algorithms, categorized as follows: \(1\) Preference\-only: DPO\[[42](https://arxiv.org/html/2610.31862#bib.bib39)\], IPO\[[3](https://arxiv.org/html/2610.31862#bib.bib57)\], and SPAC\[[24](https://arxiv.org/html/2610.31862#bib.bib45)\], which optimize policies solely based on preference data; \(2\) Demonstration\-only: SPIN\[[10](https://arxiv.org/html/2610.31862#bib.bib15)\]; \(3\) Preference and demonstration: RPO\[[31](https://arxiv.org/html/2610.31862#bib.bib50)\]and AIHF, which leverage both preference data and demonstrations\. For this experiment, we use UltraFeedback\. as the preference dataset and UltraChat200k888Available at[https://huggingface\.co/datasets/HuggingFaceH4/ultrachat\_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)\.as the demonstration dataset\. The base model for our experiments is mistral\-7b\-sft\-beta999Available at[https://huggingface\.co/HuggingFaceH4/mistral\-7b\-sft\-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta)\.\[[25](https://arxiv.org/html/2610.31862#bib.bib44)\], which is also used to initialize the reward model\.

Table 5:Evaluation of reward models in Reward\-Bench\. It can be seen that the reward model estimated by AIHF which incorporatesbothdemonstrations and preference achieves better performance\.Fig\.[7](https://arxiv.org/html/2610.31862#S6.F7)evaluates the quality of the aligned models trained on different methods\. Furthermore, in Table[5](https://arxiv.org/html/2610.31862#S6.T5), we also evaluate the reward models estimated using different methods \(DPO, standard preference learning and AIHF\) over the popular benchmark RewardBench\[[27](https://arxiv.org/html/2610.31862#bib.bib52)\]\. The BTL reward model is trained in the RLHF pipeline using the loss function defined in \([13](https://arxiv.org/html/2610.31862#S4.E13)\)\. The DPO reward is obtained by training a DPO loss on preference data \(UltraFeedback\) and use the following implicit rewardr⁡\(s,a\)=log⁡π⁡\(a\|s\)πref​\(a\|s\)r\(s,a\)=\\log\\frac\{\\pi\(a\|s\)\}\{\\pi\_\{\\rm ref\}\(a\|s\)\}whereπref\\pi\_\{\\rm ref\}is the SFT model\. For AIHF, we use the demonstration data UltraChat and generate the non\-preferred sample using the base model \(mistral\-7b\-stf\-full\), then train a reward with UltraFeedback and UltraChat data combined\. It can be seen that the reward model estimated by AIHF which incorporatesbothdemonstrations and preference achieves better performance\.

## 7Connection of Alignment to Signal Processing

Before we conclude this article, we would like to point out that the SP community has a wealth of expertise that can contribute significantly to alignment research\. Given the complexity of aligning LLMs with human preferences, SP methodologies offer valuable tools and frameworks that can improve both the theoretical and practical aspects of alignment\.

The connection between large\-scale model alignment and SP can be made precise once we recognize that the core challenge of alignment is*inference from indirect, noisy, and often incomplete measurements of human intent*\. Many mathematical tools originally developed in SP, such as inverse problems, denoising and robust estimation all provide rigorous frameworks for tackling precisely these challenges\. Below, we elaborate on a few of these connections\.

First, alignment can be viewed as a broad class of*inverse problems*, in which both the human intent and the model’s reasoning trajectory act as latent variables that must be inferred from indirect and noisy observations\. In this view, human feedback \(e\.g\., pairwise preferences or rankings\) constitutes a noisy measurement of an underlying latent*reward signal*that reflects human values, while the model’s internal reasoning process represents a second latent structure that mediates how these preferences are expressed in behavior\. This setting is analogous to classical SP problems such as image deblurring or tomography, where one seeks to recover a clean signalxxfrom degraded observationsy=A​x\+εy=Ax\+\\varepsilon\. Here, both the human reward signal and the model’s internal reasoning can be regarded as hidden components ofxx, and the alignment algorithm attempts to reconstruct them from limited feedback datayy\.

Recent work has begun to explicitly exploit this latent\-variable perspective\. For example,\[[76](https://arxiv.org/html/2610.31862#bib.bib2)\]formulates reasoning\-based alignment as a joint inference problem: it treats an LLM’s internal rationales as hidden variables and alternates between \(i\) inferring plausible reasoning chains using reward\-guided sampling, and \(ii\) updating model parameters to maximize the joint log\-likelihood of the latent rationales and final outputs\. This two\-stage optimization is formally equivalent to an*Expectation–Maximization \(EM\)*procedure widely used in SP and statistics to solve inverse problems with hidden variables\. In this context, the E\-step corresponds to estimating latent model reasoning signals, while the M\-step updates model parameters given these inferred signals\. Such a formulation provides a principled link between SP\-style latent\-variable estimation and alignment objectives, and clarifies how human preferences or reasoning traces can be reconstructed and aligned through iterative inference\.

Such latent variable perspectives also extend to the IRL problem discussed earlier, which can be viewed as solving inverse problems over high\-dimensional function spaces\. Classical SP regularization techniques such as Tikhonov penalties, total\-variation priors, or sparsity constraints can provide principled ways to stabilize these ill\-posed estimation problems, particularly when human feedback is sparse or inconsistent\.

Moreover, as alignment research expands to multi\-modal domains involving text, images, speech, and video, SP expertise in multi\-modal signal fusion, feature extraction, and cross\-modal learning becomes increasingly relevant\. The SP community’s long\-standing experience in designing architectures that integrate heterogeneous signal sources can directly inform the development of more effective and interpretable multi\-modal alignment mechanisms\.

## 8Concluding Remarks

In this work, we provided a tutorial on aligning LLMs with human feedback, exploring both mathematical foundations and state\-of\-the\-art algorithmic approaches\. Despite significant progress in aligning LLMs with human values and preferences, several open challenges remain\.

LLM alignment methods continue to face challenges related to scalability and effectiveness\. Learning robust and high\-quality policies from reward models is challenging and requires extensive human annotation and computation\. Existing RL\-based fine\-tuning methods, such as PPO, demand careful hyperparameter tuning and require a huge amount of compute resources\. Moreover, reward hacking remains a significant problem – where models find unintended shortcuts to maximize reward scores without genuinely improving response alignment\. Instead of producing responses that align with human intent, models can generate outputs optimized purely for high reward function scores, potentially leading to unreliable behavior\. Addressing these concerns requires the development of more efficient RL algorithms, robust and diverse reward functions \(e\.g\., a combination of trained reward functions and the verifiable ones mentioned in Sec\.[4\.5](https://arxiv.org/html/2610.31862#S4.SS5)\), and hybrid approaches that integrate supervised learning with RL to improve scalability and alignment fidelity\.

## Acknowledgments

Luca Viano was supported by the Swiss Data Science Center under fellowship number P22\_03\. This work was supported as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre \(CSCS\) under the Alps infrastructure\. Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF\-24\-1\-0048\. This work was funded by the Swiss National Science Foundation \(SNSF\) under grant number 2000\-1\-240094\.

## References

- \[1\]A\. Abdolmaleki, B\. Piot, B\. Shahriari, J\. T\. Springenberg, T\. Hertweck, R\. Joshi, J\. Oh, M\. Bloesch, T\. Lampe, N\. Heess,et al\.\(2024\)Preference optimization as probabilistic inference\.arXiv preprint arXiv:2410\.04166\.Cited by:[§6\.1](https://arxiv.org/html/2610.31862#S6.SS1.p4.1)\.
- \[2\]A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, A\. Üstün, and S\. Hooker\(2024\)Back to basics: revisiting reinforce style optimization for learning from human feedback in llms\.arXiv preprint arXiv:2402\.14740\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p1.3),[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p2.1),[Protocol 7](https://arxiv.org/html/2610.31862#alg7)\.
- \[3\]M\. G\. Azar, Z\. D\. Guo, B\. Piot, R\. Munos, M\. Rowland, M\. Valko, and D\. Calandriello\(2024\)A general theoretical paradigm to understand learning from human preferences\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 4447–4455\.Cited by:[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[4\]J\. L\. Ba, J\. R\. Kiros, and G\. E\. Hinton\(2016\)Layer normalization\.arXiv preprint arXiv:1607\.06450\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p5.1)\.
- \[5\]Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§1](https://arxiv.org/html/2610.31862#S1.p1.1)\.
- \[6\]E\. Beeching, C\. Fourrier, N\. Habib, S\. Han, N\. Lambert, N\. Rajani, O\. Sanseviero, L\. Tunstall, and T\. Wolf\(2023\)Open llm leaderboard\.Hugging Face\.Note:[https://huggingface\.co/spaces/HuggingFaceH4/open\_llm\_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p2.1)\.
- \[7\]M\. G\. Bellemare, S\. Candido, P\. S\. Castro, J\. Gong, M\. C\. Machado, S\. Moitra, S\. S\. Ponda, and Z\. Wang\(2020\)Autonomous navigation of stratospheric balloons using reinforcement learning\.Nature588\(7836\),pp\. 77–82\.Cited by:[§1](https://arxiv.org/html/2610.31862#S1.p1.1)\.
- \[8\]C\. Berner, G\. Brockman, B\. Chan, V\. Cheung, P\. Dębiak, C\. Dennison, D\. Farhi, Q\. Fischer, S\. Hashme, C\. Hesse,et al\.\(2019\)Dota 2 with large scale deep reinforcement learning\.arXiv preprint arXiv:1912\.06680\.Cited by:[§1](https://arxiv.org/html/2610.31862#S1.p1.1)\.
- \[9\]R\. A\. Bradley and M\. E\. Terry\(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p6.3)\.
- \[10\]Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu\(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.arXiv preprint arXiv:2401\.01335\.Cited by:[§4\.1\.2](https://arxiv.org/html/2610.31862#S4.SS1.SSS2.p1.1),[§4\.1\.2](https://arxiv.org/html/2610.31862#S4.SS1.SSS2.p1.2),[§6\.3\.1](https://arxiv.org/html/2610.31862#S6.SS3.SSS1.p4.1),[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[11\]P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei\(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px1.p1.1)\.
- \[12\]C\. Cundy and S\. Ermon\(2023\)Sequencematch: imitation learning for autoregressive sequence modelling with backtracking\.arXiv preprint arXiv:2306\.05426\.Cited by:[§4\.1\.3](https://arxiv.org/html/2610.31862#S4.SS1.SSS3.p1.1)\.
- \[13\]N\. Ding, Y\. Chen, B\. Xu, Y\. Qin, Z\. Zheng, S\. Hu, Z\. Liu, M\. Sun, and B\. Zhou\(2023\)Enhancing chat language models by scaling high\-quality instructional conversations\.arXiv preprint arXiv:2305\.14233\.Cited by:[§6\.3\.1](https://arxiv.org/html/2610.31862#S6.SS3.SSS1.p1.1)\.
- \[14\]A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p7.1)\.
- \[15\]Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto\(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p1.1)\.
- \[16\]D\. J\. Foster, A\. Block, and D\. Misra\(2024\)Is behavior cloning all you need? understanding horizon in imitation learning\.arXiv preprint arXiv:2407\.15007\.Cited by:[§4\.1\.1](https://arxiv.org/html/2610.31862#S4.SS1.SSS1.p1.1)\.
- \[17\]D\. Fudenberg, R\. Iijima, and T\. Strzalecki\(2015\)Stochastic choice and revealed perturbed utility\.Econometrica83\(6\),pp\. 2371–2409\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p6.1)\.
- \[18\]L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou\(2023\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§6\.3\.1](https://arxiv.org/html/2610.31862#S6.SS3.SSS1.p6.1)\.
- \[19\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2610.31862#S1.p2.1),[§2](https://arxiv.org/html/2610.31862#S2.p6.1),[§4\.5](https://arxiv.org/html/2610.31862#S4.SS5.p1.1),[§4\.5](https://arxiv.org/html/2610.31862#S4.SS5.p2.1)\.
- \[20\]I\. Hamilton, N\. Tawn, and D\. Firth\(2023\)The many routes to the ubiquitous bradley\-terry model\.arXiv preprint arXiv:2312\.13619\.Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p6.4)\.
- \[21\]C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.arXiv preprint arXiv:2402\.14008\.Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p6.1)\.
- \[22\]G\. Hinton\(2012\)Rmsprop: divide the gradient by a running average of its recent magnitude\.External Links:[Link](https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p7.2)\.
- \[23\]B\. Ibarz, J\. Leike, T\. Pohlen, G\. Irving, S\. Legg, and D\. Amodei\(2018\)Reward learning from human preferences and demonstrations in atari\.Advances in neural information processing systems31\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p5.1)\.
- \[24\]X\. Ji, S\. Kulkarni, M\. Wang, and T\. Xie\(2024\)Self\-play with adversarial critic: provable and scalable offline alignment for language models\.arXiv preprint arXiv:2406\.04274\.Cited by:[§5\.2](https://arxiv.org/html/2610.31862#S5.SS2.p3.3),[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[25\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[26\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),San Diego, CA, USA\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p7.2)\.
- \[27\]N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi,et al\.\(2024\)Rewardbench: evaluating reward models for language modeling\.arXiv preprint arXiv:2403\.13787\.Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p7.1),[§6\.3\.1](https://arxiv.org/html/2610.31862#S6.SS3.SSS1.p3.1),[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p2.1)\.
- \[28\]C\. Li, S\. Zeng, Z\. Liao, J\. Li, D\. Kang, A\. Garcia, and M\. Hong\(2024\)Joint demonstration and preference learning improves policy alignment with human feedback\.arXiv preprint arXiv:2406\.06874\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p5.1),[§5\.2](https://arxiv.org/html/2610.31862#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.31862#S5.SS2.p2.2)\.
- \[29\]J\. Li, S\. Zeng, H\. Wai, C\. Li, A\. Garcia, and M\. Hong\(2024\)Getting more juice out of the sft data: reward learning from human demonstration improves sft for llm alignment\.Advances in Neural Information Processing Systems\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p3.1)\.
- \[30\]P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. A\. Cosgrove, C\. D\. Manning, C\. Re, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. WANG, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. S\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. A\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Note:Featured Certification, Expert CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=iO4LZibEqW)Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p4.1)\.
- \[31\]Z\. Liu, M\. Lu, S\. Zhang, B\. Liu, H\. Guo, Y\. Yang, J\. Blanchet, and Z\. Wang\(2024\)Provably mitigating overoptimization in RLHF: your SFT loss is implicitly an adversarial regularizer\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=2cQ3lPhkeO)Cited by:[§5\.2](https://arxiv.org/html/2610.31862#S5.SS2.p3.2),[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[32\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p7.2)\.
- \[33\]R\. D\. Luce\(1959\)Individual choice behavior: a theoretical analysis\.Wiley,New York, NY, USA\.Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p6.3)\.
- \[34\]F\. Maccheroni, M\. Marinacci, and A\. Rustichini\(2006\)Ambiguity aversion, robustness, and the variational representation of preferences\.Econometrica74\(6\),pp\. 1447–1498\.Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p7.1)\.
- \[35\]M\. J\. Machina\(1985\)Stochastic choice functions generated from deterministic preferences over lotteries\.The Economic Journal95\(379\),pp\. 575–594\.External Links:[Document](https://dx.doi.org/10.2307/2233028),[Link](https://doi.org/10.2307/2233028)Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p7.1)\.
- \[36\]D\. McFadden\(1974\)The measurement of urban travel demand\.Journal of Public Economics3\(4\),pp\. 303–328\.External Links:[Document](https://dx.doi.org/10.1016/0047-2727%2874%2990003-6)Cited by:[§3](https://arxiv.org/html/2610.31862#S3.p1.1)\.
- \[37\]R\. Munos, M\. Valko, D\. Calandriello, M\. G\. Azar, M\. Rowland, Z\. D\. Guo, Y\. Tang, M\. Geist, T\. Mesnard, A\. Michi,et al\.\(2023\)Nash learning from human feedback\.arXiv preprint arXiv:2312\.00886\.Cited by:[§4\.4](https://arxiv.org/html/2610.31862#S4.SS4.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2610.31862#S4.SS4.SSS0.Px2.p2.1),[§4\.4](https://arxiv.org/html/2610.31862#S4.SS4.p1.2)\.
- \[38\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2610.31862#S1.p1.1),[§1](https://arxiv.org/html/2610.31862#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p3.1)\.
- \[39\]D\. A\. Pomerleau\(1988\)Alvinn: an autonomous land vehicle in a neural network\.Advances in neural information processing systems1\.Cited by:[§4\.1\.1](https://arxiv.org/html/2610.31862#S4.SS1.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2610.31862#S4.SS1.p2.1)\.
- \[40\]B\. Qi, P\. Li, F\. Li, J\. Gao, K\. Zhang, and B\. Zhou\(2024\)Online dpo: online direct preference optimization with fast\-slow chasing\.arXiv preprint arXiv:2406\.05534\.Cited by:[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p4.1)\.
- \[41\]Z\. Qi, X\. Liu, I\. L\. Iong, H\. Lai, X\. Sun, W\. Zhao, Y\. Yang, X\. Yang, J\. Sun, S\. Yao,et al\.\(2024\)Webrl: training llm web agents via self\-evolving online curriculum reinforcement learning\.arXiv preprint arXiv:2411\.02337\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p6.1)\.
- \[42\]R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.arXiv preprint arXiv:2305\.18290\.Cited by:[§6\.3\.2](https://arxiv.org/html/2610.31862#S6.SS3.SSS2.p1.1)\.
- \[43\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2024\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p1.1),[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p1.3)\.
- \[44\]S\. Ross and D\. Bagnell\(2010\)Efficient reductions for imitation learning\.InProceedings of the thirteenth international conference on artificial intelligence and statistics,pp\. 661–668\.Cited by:[§4\.1](https://arxiv.org/html/2610.31862#S4.SS1.p2.1)\.
- \[45\]S\. Ross, G\. Gordon, and D\. Bagnell\(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p2.1)\.
- \[46\]D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams\(1986\)Learning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p5.1)\.
- \[47\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p1.3),[Protocol 5](https://arxiv.org/html/2610.31862#alg5)\.
- \[48\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p1.3),[Protocol 8](https://arxiv.org/html/2610.31862#alg8)\.
- \[49\]H\. Sun and M\. van der Schaar\(2024\)Inverse\-rlignment: inverse reinforcement learning from demonstrations for llm alignment\.arXiv preprint arXiv:2405\.15624\.Cited by:[§4\.1\.3](https://arxiv.org/html/2610.31862#S4.SS1.SSS3.p1.1),[§4\.1\.3](https://arxiv.org/html/2610.31862#S4.SS1.SSS3.p3.1)\.
- \[50\]C\. Szegedy, S\. Ioffe, V\. Vanhoucke, and A\. Alemi\(2017\)Inception\-v4, inception\-resnet and the impact of residual connections on learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.31\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p5.1)\.
- \[51\]K\. Team, Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen,et al\.\(2025\)Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p6.1)\.
- \[52\]L\. L\. Thurstone\(1927\)A law of comparative judgment\.Psychological Review34\(4\),pp\. 273–286\.Cited by:[§3](https://arxiv.org/html/2610.31862#S3.p1.1)\.
- \[53\]N\. Tishby and D\. Polani\(2011\)Information theory of decisions and actions\.InPerception\-Action Cycle: Models, Architectures, and Hardware,V\. Cutsuridis, A\. Hussain, and J\. G\. Taylor \(Eds\.\),pp\. 601–636\.External Links:[Document](https://dx.doi.org/10.1007/978-1-4419-1452-1%5F19),[Link](https://link.springer.com/chapter/10.1007/978-1-4419-1452-1_19)Cited by:[§3\.1](https://arxiv.org/html/2610.31862#S3.SS1.p7.1)\.
- \[54\]L\. Tunstall, E\. Beeching, N\. Lambert, N\. Rajani, K\. Rasul, Y\. Belkada, S\. Huang, L\. von Werra, C\. Fourrier, N\. Habib,et al\.\(2023\)Zephyr: direct distillation of lm alignment\.arXiv preprint arXiv:2310\.16944\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p5.1)\.
- \[55\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p4.2),[§2](https://arxiv.org/html/2610.31862#S2.p5.1)\.
- \[56\]L\. Viano, Y\. Huang, P\. Kamalaruban, A\. Weller, and V\. Cevher\(2021\)Robust inverse reinforcement learning under transition dynamics mismatch\.Advances in Neural Information Processing Systems34,pp\. 25917–25931\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p3.1)\.
- \[57\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p6.1)\.
- \[58\]R\. J\. Williams\(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8,pp\. 229–256\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p1.3),[Protocol 6](https://arxiv.org/html/2610.31862#alg6)\.
- \[59\]Q\. Wu, L\. Li, and Z\. Yu\(2021\)Textgail: generative adversarial imitation learning for text generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 14067–14075\.Cited by:[§4\.1\.2](https://arxiv.org/html/2610.31862#S4.SS1.SSS2.p3.1),[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p3.1),[Protocol 3](https://arxiv.org/html/2610.31862#alg3)\.
- \[60\]M\. Wulfmeier, M\. Bloesch, N\. Vieillard, A\. Ahuja, J\. Bornschein, S\. Huang, A\. Sokolov, M\. Barnes, G\. Desjardins, A\. Bewley,et al\.\(2024\)Imitating language via scalable inverse reinforcement learning\.Advances in Neural Information Processing Systems37,pp\. 90714–90735\.Cited by:[§4\.1\.1](https://arxiv.org/html/2610.31862#S4.SS1.SSS1.p1.1),[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p3.1)\.
- \[61\]M\. Wulfmeier, P\. Ondruska, and I\. Posner\(2015\)Maximum entropy deep inverse reinforcement learning\.arXiv preprint arXiv:1507\.04888\.Cited by:[§4\.1\.3](https://arxiv.org/html/2610.31862#S4.SS1.SSS3.p3.1)\.
- \[62\]T\. Xiao, Y\. Yuan, M\. Li, Z\. Chen, and V\. G\. Honavar\(2025\)On a connection between imitation learning and rlhf\.arXiv preprint arXiv:2503\.05079\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p5.1)\.
- \[63\]C\. Xie, Y\. Huang, C\. Zhang, D\. Yu, X\. Chen, B\. Y\. Lin, B\. Li, B\. Ghazi, and R\. Kumar\(2024\)On memorization of large language models in logical reasoning\.arXiv preprint arXiv:2410\.23123\.Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p6.1)\.
- \[64\]T\. Xie, Z\. Gao, Q\. Ren, H\. Luo, Y\. Hong, B\. Dai, J\. Zhou, K\. Qiu, Z\. Wu, and C\. Luo\(2025\)Logic\-rl: unleashing llm reasoning with rule\-based reinforcement learning\.arXiv preprint arXiv:2502\.14768\.Cited by:[§4\.5](https://arxiv.org/html/2610.31862#S4.SS5.p1.1)\.
- \[65\]W\. Xiong, H\. Dong, C\. Ye, Z\. Wang, H\. Zhong, H\. Ji, N\. Jiang, and T\. Zhang\(2023\)Iterative preference learning from human feedback: bridging theory and practice for rlhf under kl\-constraint\.arXiv preprint arXiv:2312\.11456\.Cited by:[§6\.1](https://arxiv.org/html/2610.31862#S6.SS1.p4.1)\.
- \[66\]J\. Xu, X\. Sun, Z\. Zhang, G\. Zhao, and J\. Lin\(2019\)Understanding and improving layer normalization\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p5.1)\.
- \[67\]S\. Xu, W\. Fu, J\. Gao, W\. Ye, W\. Liu, Z\. Mei, G\. Wang, C\. Yu, and Y\. Wu\(2024\)Is dpo superior to ppo for llm alignment? a comprehensive study\.arXiv preprint arXiv:2404\.10719\.Cited by:[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p2.1),[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p4.1),[§4\.3](https://arxiv.org/html/2610.31862#S4.SS3.p5.1),[Table 2](https://arxiv.org/html/2610.31862#S4.T2)\.
- \[68\]T\. Xu, Z\. Li, and Y\. Yu\(2020\)Error bounds of imitating policies and environments\.Advances in Neural Information Processing Systems33,pp\. 15737–15749\.Cited by:[§4\.2](https://arxiv.org/html/2610.31862#S4.SS2.SSS0.Px2.p3.1)\.
- \[69\]S\. Zeng, M\. Hong, and A\. Garcia\(2022\)Structural estimation of markov decision processes in high\-dimensional state space with finite\-time guarantees\.arXiv preprint arXiv:2210\.01282\.Cited by:[§3\.2](https://arxiv.org/html/2610.31862#S3.SS2.p2.2)\.
- \[70\]S\. Zeng, M\. Hong, and A\. Garcia\(2024\)Structural estimation of markov decision processes in high\-dimensional state space with finite\-time guarantees\.Operations Research\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p4.1),[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p4.2)\.
- \[71\]S\. Zeng, C\. Li, A\. Garcia, and M\. Hong\(2022\)Maximum\-likelihood inverse reinforcement learning with finite\-time guarantees\.Advances in Neural Information Processing Systems35,pp\. 10122–10135\.Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p1.1)\.
- \[72\]S\. Zeng, Y\. Liu, H\. Rangwala, G\. Karypis, M\. Hong, and R\. Fakoor\(2025\)From demonstrations to rewards: alignment without explicit human preferences\.arXiv preprint arXiv:2503\.13538\.Cited by:[§4\.1\.2](https://arxiv.org/html/2610.31862#S4.SS1.SSS2.p3.1),[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p3.1)\.
- \[73\]S\. Zeng, Y\. Liu, H\. Rangwala, G\. Karypis, M\. Hong, and R\. Fakoor\(2025\)From demonstrations to rewards: alignment without explicit human preferences\.External Links:[Link](https://openreview.net/forum?id=mltelO89Ve)Cited by:[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p4.1),[§5\.1](https://arxiv.org/html/2610.31862#S5.SS1.p4.2)\.
- \[74\]X\. Zhang, C\. Li, S\. Zeng, J\. Li, Z\. Wang, K\. Lin, S\. Lu, A\. Garcia, and M\. Hong\(2025\)Aligning frozen llms by reinforcement learning: an iterative reweight\-then\-optimize approach\.arXiv preprint arXiv:2506\.17828\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p6.1)\.
- \[75\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36,pp\. 46595–46623\.Cited by:[§6\.2](https://arxiv.org/html/2610.31862#S6.SS2.p3.1),[§6\.3\.1](https://arxiv.org/html/2610.31862#S6.SS3.SSS1.p6.1)\.
- \[76\]H\. Zhong, Y\. Yin, S\. Zhang, X\. Xu, Y\. Liu, Y\. Zuo, Z\. Liu, B\. Liu, S\. Zheng, H\. Guo,et al\.BRiTE: bootstrapping reinforced thinking process to enhance language model reasoning\.InForty\-second International Conference on Machine Learning,Cited by:[§7](https://arxiv.org/html/2610.31862#S7.p4.1)\.
- \[77\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§2](https://arxiv.org/html/2610.31862#S2.p6.1)\.