Robust Nash Alignment under Preference Uncertainty

# Robust Nash Alignment under Preference Uncertainty
Source: [https://arxiv.org/html/2610.00715](https://arxiv.org/html/2610.00715)
Debamita GhoshAffiliation:University of Central FloridaEmail:[debamita\.ghosh@ucf\.edu](mailto:)David TangAffiliation:University of Central FloridaEmail:[david\.tang@ucf\.edu](mailto:)Yudan WangAffiliation:Arizona State UniversityEmail:[ywan1645@asu\.edu](mailto:)Alvaro VelasquezAffiliation:University of Colorado BoulderEmail:[alvaro\.velasquez@colorado\.edu](mailto:)Yue WangAffiliation:University of Central FloridaEmail:[yue\.wang@ucf\.edu](mailto:)

###### Abstract

Preference\-based alignment methods typically optimize against a single preference model, and can therefore be brittle when pairwise preferences are uncertain: noisy, heterogeneous, or shift after deployment\. To address these issues, we propose Robust Nash Alignment, a game\-theoretic framework for alignment to uncertain pairwise preferences\. Our formulation has a major learner seeking a policy with a large worst\-case win rate against both an adversarial competitor and any preference kernel lying in an ambiguity set around a nominal preference\. When the ambiguity set captures the uncertainty in preferences, the resulting robust objective of the game directly yields a certified lower bound on worst\-case performance\. However, we note this problem is computationally challenging to optimize, and to address this, we introduce a four\-player primal\-dual proxy game involving the leader policy, follower policy, adversarial kernel, and dual variable, and develop a single\-loop optimistic mirror descent\-ascent algorithm for it\. We show that the proxy always lower\-bounds the truncated hard\-constrained objective, quantify the proxy\-to\-hard gap, and characterize an exactness condition under which the proxy recovers the robust objective\. We then prove an𝒪⁡\(1/T\)\\mathcal\{O\}\(1/\\sqrt\{T\}\)average\-iteration convergence for the proxy\-game duality gap, which implies a near\-optimal robust policy for the original robust objective\. Experiments on controlled tabular games and LLM alignment with uncertain preference further validate the convergence theory and show improved performance over nominal baselines\.

## 1Introduction

Aligning large language models \(LLMs\) with preferences is a central challenge in modern post\-training\. The dominant paradigm, reinforcement learning from human feedback \(RLHF\)\[[9](https://arxiv.org/html/2610.00715#bib.bib9),[41](https://arxiv.org/html/2610.00715#bib.bib41),[33](https://arxiv.org/html/2610.00715#bib.bib33)\], collects pairwise comparisons, fits a scalar reward model, and optimizes policy against that reward with a proximity constraint to a reference model\. More recent direct alignment methods, including Direct Preference Optimization \(DPO\), SLiC\-HF, Contrastive Preference Learning \(CPL\), and Soft Preference Optimization \(SPO\)\[[35](https://arxiv.org/html/2610.00715#bib.bib35),[55](https://arxiv.org/html/2610.00715#bib.bib55),[15](https://arxiv.org/html/2610.00715#bib.bib15),[40](https://arxiv.org/html/2610.00715#bib.bib40)\], bypass the explicit reward model by optimizing preference\-derived surrogate losses directly\. Despite their practical success, all of these methods share a common structural assumption: they model preferences through a latent scalar reward function, typically via the Bradley\-Terry \(BT\)\[[45](https://arxiv.org/html/2610.00715#bib.bib45),[4](https://arxiv.org/html/2610.00715#bib.bib4)\]or Thurstone framework\. This assumption is, however, restrictive\. Preferences may be context\-dependent, cyclic, or heterogeneous across annotators, and cannot always be represented by a single scalar reward\[[3](https://arxiv.org/html/2610.00715#bib.bib3),[29](https://arxiv.org/html/2610.00715#bib.bib29),[54](https://arxiv.org/html/2610.00715#bib.bib54),[52](https://arxiv.org/html/2610.00715#bib.bib52),[8](https://arxiv.org/html/2610.00715#bib.bib8),[14](https://arxiv.org/html/2610.00715#bib.bib14)\]\.

A growing line of work addresses this limitation by modeling preferences directly through pairwise comparison probabilities and framing alignment as a two\-player zero\-sum game\. In particular, Nash Learning from Human Feedback \(NLHF\) and its extensions\[[29](https://arxiv.org/html/2610.00715#bib.bib29),[54](https://arxiv.org/html/2610.00715#bib.bib54),[52](https://arxiv.org/html/2610.00715#bib.bib52),[5](https://arxiv.org/html/2610.00715#bib.bib5)\]seek a policy that is hard to beat in the induced preference game, rather than one that maximizes a learned scalar reward\. This general\-preference perspective resolves the modeling mismatch: it can represent non\-transitive, cyclic, and population\-heterogeneous preferences that no scalar reward can capture\. However, it introduces a different and equally important challenge that has been largely overlooked:*the preference kernel itself is uncertain\.*

The true pairwise preference probabilityP⁡\(yi≻yj∣x\)P\(y\_\{i\}\\succ y\_\{j\}\\mid x\)is never observed or fixed\. In practice, it must be estimated from finite, noisy, and potentially biased comparisons, often aggregated across heterogeneous annotator populations and changing deployment environments\[[9](https://arxiv.org/html/2610.00715#bib.bib9),[41](https://arxiv.org/html/2610.00715#bib.bib41),[33](https://arxiv.org/html/2610.00715#bib.bib33),[43](https://arxiv.org/html/2610.00715#bib.bib43)\]\. Existing general\-preference methods, including NLHF, treat the learned preference kernel as a fixed ground truth and optimize against this single point estimate\. This can be brittle: small errors in preference estimation, misspecification of the comparison model, or deployment\-time shifts in who provides feedback may materially change which responses the policy favors\[[36](https://arxiv.org/html/2610.00715#bib.bib36),[38](https://arxiv.org/html/2610.00715#bib.bib38),[8](https://arxiv.org/html/2610.00715#bib.bib8),[14](https://arxiv.org/html/2610.00715#bib.bib14)\]\. We refer to this vulnerability as the*fragile consensus*problem—a policy that appears optimal under the nominal preference data may concentrate on responses whose advantages rest on slim margins that are statistically indistinguishable from noise, and a small perturbation to the preference kernel can reverse the dominance ordering entirely\.

To address this challenge, we propose to treat the preference kernel as fundamentally uncertain and to optimize for the worst case over a set of plausible kernels\. Inspired by the literature on distributionally robust optimization\[[36](https://arxiv.org/html/2610.00715#bib.bib36),[20](https://arxiv.org/html/2610.00715#bib.bib20),[31](https://arxiv.org/html/2610.00715#bib.bib31)\], where ambiguity sets, divergence balls, and worst\-case guarantees are standard tools for hedging estimation error and distribution shift, we formulate*alignment under uncertain preferences*as a robust two\-player game over a pairwise preference kernel\. At each prompt, the main player seeks to maximize a Kullback\-Leibler \(KL\)\-regularized win rate, while an adversarial player simultaneously chooses the hardest competitor*and*the worst\-case preference kernel from an ambiguity set centered at the nominal estimate\. The resulting objective directly provides a lower bound on the policy’s alignment performance under all plausible preference kernels in the ambiguity set, effectively addressing the fragile consensus problem\.

Our contributions are summarized as follows\.

Robust preference\-game formulation\.We formulate alignment under preference uncertainty as a robust game over a pairwise preference kernel with a Bernoulli\-KL ambiguity set\. Unlike standard NLHF, which optimizes against a single nominal kernel, our objective maximizes the worst\-case game value over all kernels within a prescribed divergence ball\. We show that this formulation yields a principled worst\-case performance guarantee: the learned policy’s win rate is lower\-bounded against every plausible preference kernel in the ambiguity set, improving the robustness and reliability of preference\-based alignment\.

Four\-player algorithm with convergence guarantees\.To avoid the computational burden of solving the hard\-constrained distributionally robust problem directly, we pass to its Lagrangian as a proxy and derive a four\-player primal\-dual formulation involving the leader policy, follower policy, adversarial preference kernel, and dual variable\. We further provide a detailed duality and convergence analysis: we characterize when the four\-player game exactly or approximately recovers the original hard\-constrained robust objective, and we prove that the proposed algorithm converges to a saddle point of the proxy robust game at a rate of𝒪⁡\(1/T\)\\mathcal\{O\}\(1/\\sqrt\{T\}\)\. Under the stated exactness conditions, this yields a certified worst\-case game\-value guarantee\.

Experimental validation\.We validate our algorithm on controlled tabular games, a best\-of\-mmreranking task on IMDb continuations with heterogeneous annotator groups, and a summarization task on TL;DR using LLMs\. Across all settings, the robust policy achieves substantially higher worst\-case win rates than the non\-robust NashMD baseline, and performs better alignment under perturbed preferences from the ambiguity set in the IMDb task, and enjoys better generalizability to external LLM judges in the summarization task, while maintaining comparable nominal performance\.

## 2Preliminaries

Notation\.Let𝒳\\mathcal\{X\}denote the prompt space, and letD0D\_\{0\}be the deployment\-time prompt distribution\. For each promptx∈𝒳x\\in\\mathcal\{X\}, letY⁡\(x\)=\{y1,…,ym\}Y\(x\)=\\\{y\_\{1\},\\dots,y\_\{m\}\\\}be the finite response set\. A policy is a mappingπ:𝒳→Δ⁡\(Y\)\\pi:\\mathcal\{X\}\\to\\Delta\(Y\), whereΔ⁡\(Y⁡\(x\)\)\\Delta\(Y\(x\)\)denotes the simplex over responses for promptxx\. Givenxx, a responseyyis sampled fromπ\(⋅∣x\)\\pi\(\\cdot\\mid x\)\. We also fix a reference policyπref\(⋅∣x\)\\pi\_\{\\rm ref\}\(\\cdot\\mid x\)with full support, which can be derived from Supervised Fine\-Tuning \(SFT\), and a KL regularization parameterτ\>0\\tau\>0\.

Reward\-based alignment\.A standard approach to preference\-based alignment assumes that pairwise human preferences are generated from an underlying scalar reward functionR⋆​\(x,y\)R^\{\\star\}\(x,y\)\. Under the classical Bradley\-Terry / logistic preference model\[[4](https://arxiv.org/html/2610.00715#bib.bib4),[9](https://arxiv.org/html/2610.00715#bib.bib9),[33](https://arxiv.org/html/2610.00715#bib.bib33)\], the probability that responseyyis preferred toy′y^\{\\prime\}under promptxxis

ℙ⁡\(y≻y′∣x\)=exp⁡\(R⋆​\(x,y\)\)exp⁡\(R⋆​\(x,y\)\)\+exp⁡\(R⋆​\(x,y′\)\)\.\\mathbb\{P\}\(y\\succ y^\{\\prime\}\\mid x\)=\\frac\{\\exp\(R^\{\\star\}\(x,y\)\)\}\{\\exp\(R^\{\\star\}\(x,y\)\)\+\\exp\(R^\{\\star\}\(x,y^\{\\prime\}\)\)\}\.\(1\)
Given a dataset with pairwise responses and preferences, one can first learn a reward modelRRthrough regression, and maximize the classical KL\-regularized RLHF objective

JRB\(π;R\):=𝔼x∼D0\[𝔼y∼π\(⋅∣x\)\[R\(x,y\)\]−τDKL\(π\(⋅∣x\)∥πref\(⋅∣x\)\)\]\.J\_\{\\rm RB\}\(\\pi;R\):=\\mathbb\{E\}\_\{x\\sim D\_\{0\}\}\\left\[\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid x\)\}\[R\(x,y\)\]\-\\tau D\_\{\\mathrm\{KL\}\}\\bigl\(\\pi\(\\cdot\\mid x\)\\,\\\|\\,\\pi\_\{\\rm ref\}\(\\cdot\\mid x\)\\bigr\)\\right\]\.\(2\)The reward\-based alignment problem is thenπR⋆∈arg⁡maxπ∈Π​JRB​\(π,R\),\\pi\_\{R\}^\{\\star\}\\in\\arg\\max\_\{\\pi\\in\\Pi\}J\_\{\\rm RB\}\(\\pi;R\),whereΠ\\Piis the policy class of interest\. The KL term discourages the learned policy from drifting arbitrarily far from the reference policy\[[41](https://arxiv.org/html/2610.00715#bib.bib41),[33](https://arxiv.org/html/2610.00715#bib.bib33),[35](https://arxiv.org/html/2610.00715#bib.bib35)\], and it results in a closed\-form solution to \([2](https://arxiv.org/html/2610.00715#S2.E2)\) as

πR⋆​\(y∣x\)∝πref​\(y∣x\)​exp⁡\(1τ​R​\(x,y\)\),\\pi\_\{R\}^\{\\star\}\(y\\mid x\)\\propto\\pi\_\{\\rm ref\}\(y\\mid x\)\\exp\\left\(\\frac\{1\}\{\\tau\}R\(x,y\)\\right\),\(3\)which underlies the derivation of the direct preference optimization method\[[35](https://arxiv.org/html/2610.00715#bib.bib35)\]\.

#### General preference alignment\.

A limitation of \([1](https://arxiv.org/html/2610.00715#S2.E1)\) is that it assumes all pairwise preferences are induced by a single scalar reward function\. More generally, one can directly treat pairwise preference probabilities as primitive objects, without assuming the latent reward\[[29](https://arxiv.org/html/2610.00715#bib.bib29),[52](https://arxiv.org/html/2610.00715#bib.bib52),[54](https://arxiv.org/html/2610.00715#bib.bib54)\]\.

###### Definition 1\.

A general preference oracle is a mappingP:𝒳×Y×Y→\[0,1\]P:\\mathcal\{X\}\\times Y\\times Y\\to\[0,1\]such that, for every promptx∈𝒳x\\in\\mathcal\{X\}and responsesy,y′∈Y⁡\(x\)y,y^\{\\prime\}\\in Y\(x\), there exists a preferencez∼Ber⁡\(P⁡\(y≻y′∣x\)\),z\\sim\\mathrm\{Ber\}\\bigl\(P\(y\\succ y^\{\\prime\}\\mid x\)\\bigr\),wherez=1z=1indicates thatyyis preferred toy′y^\{\\prime\}, andz=0z=0indicates the opposite\.

###### Assumption 1\(Antisymmetric preference model\)\.

For any fixed promptxx, definePx​\(y≻y′\):=P⁡\(y≻y′∣x\)P\_\{x\}\(y\\succ y^\{\\prime\}\):=P\(y\\succ y^\{\\prime\}\\mid x\)\. We assume that the preference probabilities are antisymmetric, i\.e\.,

Px​\(yi≻yj\)\+Px​\(yj≻yi\)=1,∀yi≠yj\.P\_\{x\}\(y\_\{i\}\\succ y\_\{j\}\)\+P\_\{x\}\(y\_\{j\}\\succ y\_\{i\}\)=1,\\qquad\\forall\\,y\_\{i\}\\neq y\_\{j\}\.\(4\)

Given two policiesπ,ν∈Π\\pi,\\nu\\in\\Pi, Nash learning from human feedback \(NLHF\) considers the KL\-regularized game payoff under the general preference kernelPP:

J⁡\(π,ν,P\)\\displaystyle J\(\\pi,\\nu;P\):=𝔼x∼D0\[𝔼y∼π\(⋅∣x\),y′∼ν\(⋅∣x\)\[Px\(y≻y′\)\]−τDKL\(π\(⋅∣x\)∥πref\(⋅∣x\)\)\\displaystyle:=\\mathbb\{E\}\_\{x\\sim D\_\{0\}\}\\Big\[\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid x\),\\;y^\{\\prime\}\\sim\\nu\(\\cdot\\mid x\)\}\\big\[P\_\{x\}\(y\\succ y^\{\\prime\}\)\\big\]\-\\tau D\_\{\\mathrm\{KL\}\}\\bigl\(\\pi\(\\cdot\\mid x\)\\,\\\|\\,\\pi\_\{\\rm ref\}\(\\cdot\\mid x\)\\bigr\)\+τDKL\(ν\(⋅∣x\)∥πref\(⋅∣x\)\)\]\.\\displaystyle\\quad\+\\tau D\_\{\\mathrm\{KL\}\}\\bigl\(\\nu\(\\cdot\\mid x\)\\,\\\|\\,\\pi\_\{\\rm ref\}\(\\cdot\\mid x\)\\bigr\)\\Big\]\.\(5\)Here,π\\piis the max\-player andν\\nuis the min\-player\. The interpretation is that both players seek a policy that is consistently preferred to any competitor, while both players are regularized toward the same reference policy\. The nominal general\-preference alignment problem is therefore formulated as a regularized minimax game

\(π⋆,ν⋆\)∈arg⁡maxπ∈Π​arg​minν∈Π⁡J⁡\(π,ν,P\)\.\(\\pi^\{\\star\},\\nu^\{\\star\}\)\\in\\arg\\max\_\{\\pi\\in\\Pi\}\\arg\\min\_\{\\nu\\in\\Pi\}J\(\\pi,\\nu;P\)\.\(6\)Due to the symmetric and convex\-concave game structure, \([6](https://arxiv.org/html/2610.00715#S2.E6)\) has a unique symmetric Nash equilibrium\(π⋆,π⋆\)\(\\pi^\{\\star\},\\pi^\{\\star\}\)\[[29](https://arxiv.org/html/2610.00715#bib.bib29)\]\. NLHF then aims to find this policyπ⋆\\pi^\{\\star\}as the alignment policy\.

## 3Robust Game Formulation for Uncertain Preference Alignment

In this section, we introduce our formulation of a robust game to address uncertainty in preferences\. For notational simplicity, we fix a prompt and suppress the prompt index\. The global prompt\-averaged objective is obtained by taking expectations under a prompt\-rectangular ambiguity model\.

LetY=\{1,…,m\}Y=\\\{1,\\dots,m\\\}be a finite response set, andd=\(m2\)d=\\binom\{m\}\{2\}, and setE:=\{\(i,j\):1≤i<j≤m\}\.E:=\\\{\(i,j\):1\\leq i<j\\leq m\\\}\.Forp=\(pi​j\)\(i,j\)∈E∈\[0,1\]dp=\(p\_\{ij\}\)\_\{\(i,j\)\\in E\}\\in\[0,1\]^\{d\}, we define the preference kernel that satisfies Assumption[1](https://arxiv.org/html/2610.00715#Thmassumption1)as

Pp​\(i≻j\)=\{pi​j,i<j,1−pj​i,i\>j,0\.5,i=j\.P\_\{p\}\(i\\succ j\)=\\begin\{cases\}p\_\{ij\},&i<j,\\\\ 1\-p\_\{ji\},&i\>j,\\\\ 0\.5,&i=j\.\\end\{cases\}
We model the potential uncertainty in the preference kernel through the Bernoulli\-KL divergence\. Namely, centered at a nominal kernelp⋆∈\(0,1\)dp^\{\\star\}\\in\(0,1\)^\{d\}, the Bernoulli\-KL divergence is defined as

DKLBern\(p∥p⋆\):=∑\(i,j\)∈E\[pi​jlogpi​jpi​j⋆\+\(1−pi​j\)log1−pi​j1−pi​j⋆\],D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\):=\\sum\_\{\(i,j\)\\in E\}\\left\[p\_\{ij\}\\log\\frac\{p\_\{ij\}\}\{p^\{\\star\}\_\{ij\}\}\+\(1\-p\_\{ij\}\)\\log\\frac\{1\-p\_\{ij\}\}\{1\-p^\{\\star\}\_\{ij\}\}\\right\],\(7\)measuring the difference between the two kernels\. We then set the ambiguity set as

𝒫:=\{p:DKLBern\(p∥p⋆\)≤ρ\},\\displaystyle\\mathcal\{P\}:=\\\{p:D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\\},\(8\)whereρ\\rhois some uncertainty radius, quantifying the uncertainty level\. Namely, the ambiguity set contains the kernels whose differences from the nominal one are bounded byρ\\rho\.

#### Objective\.

With the ambiguity set defined in \([8](https://arxiv.org/html/2610.00715#S3.E8)\), we propose our robust formulation as

maxπ∈Δm⁡minν∈Δm​minp∈𝒫⁡J⁡\(π,ν,p\),\\displaystyle\\max\_\{\\pi\\in\\Delta\_\{m\}\}\\min\_\{\\nu\\in\\Delta\_\{m\}\}\\min\_\{p\\in\\mathcal\{P\}\}J\(\\pi,\\nu;p\),\(9\)whereJ⁡\(π,ν,p\)J\(\\pi,\\nu;p\)is the game value under the specific kernelpp:

J\(π,ν;p\):=∑i=1m∑j=1mπiνjPp\(i≻j\)−τDKL\(π∥πref\)\+τDKL\(ν∥πref\)\.J\(\\pi,\\nu;p\):=\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{m\}\\pi\_\{i\}\\nu\_\{j\}\\,P\_\{p\}\(i\\succ j\)\-\\tau D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{\\mathrm\{ref\}\}\)\+\\tau D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\pi\_\{\\mathrm\{ref\}\}\)\.\(10\)Namely, we enable the adversarial player to further choose a preference kernel as part of its strategy, and the main player needs to maximize its winning rate under the worst\-case, i\.e\., the least favorable preference kernel\. This consideration of the worst\-case then provides a lower bound guarantee\. Namely, ifπ⋆\\pi^\{\\star\}solves \([9](https://arxiv.org/html/2610.00715#S3.E9)\), then for anyp∈𝒫p\\in\\mathcal\{P\}and any adversarial policyν\\nu, one has \(ignoring the regularizer\) an optimized lower bound on the winning rate underppand againstν\\nu:

Pp​\(π⋆≻ν\)≥minp∈𝒫,ν⁡Pp​\(π⋆≻ν\)=maxπ⁡minp∈𝒫,ν​Pp​\(π≻ν\)=optimal value of \([9](https://arxiv.org/html/2610.00715#S3.E9)\)\.\\displaystyle P\_\{p\}\(\\pi^\{\\star\}\\succ\\nu\)\\geq\\min\_\{p\\in\\mathcal\{P\},\\nu\}P\_\{p\}\(\\pi^\{\\star\}\\succ\\nu\)=\\max\_\{\\pi\}\\min\_\{p\\in\\mathcal\{P\},\\nu\}P\_\{p\}\(\\pi\\succ\\nu\)=\\text\{ optimal value of \\eqref\{eq:obj\}\}\.\(11\)Therefore, the maximizerπ⋆\\pi^\{\\star\}of our robust value \([9](https://arxiv.org/html/2610.00715#S3.E9)\) provides a baseline performance of uncertain alignment, thus improving the robustness and reliability against preference uncertainty\.

## 4Four\-Player Proxy Game

In this section, we aim to design a concrete algorithm to solve the robust game \([9](https://arxiv.org/html/2610.00715#S3.E9)\)\.

For standard, non\-robust NLHF, where the kernel is fixed and the only strategic interaction is between the two players, the optimization landscape is that of a regularized two\-player game, which can be handled by no\-regret and optimistic first\-order methods\. However, our robust formulation is qualitatively more difficult\. The major challenge arises from the fact that, even for a fixed leader policyπ\\pi, the inner valuegH​\(π\):=minν,p⁡J⁡\(π,ν,p\)g\_\{H\}\(\\pi\):=\\min\_\{\\nu,p\}J\(\\pi,\\nu;p\)is difficult to compute exactly becauseν\\nuandppare coupled through the win\-rate term\. If one first fixesπ,p\\pi,pand eliminates the adversarial policyν⋆\\nu^\{\\star\}using the Gibbs response, the reduced objective becomes

minp∈𝒫\{−τDKL\(π∥πref\)−τlog∑j=1mπref\(j\)exp\(−b⁡\(j,π,p\)τ\)\},\\min\_\{p\\in\\mathcal\{P\}\}\\Big\\\{\-\\tau D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{\\rm ref\}\)\-\\tau\\log\\sum\_\{j=1\}^\{m\}\\pi\_\{\\rm ref\}\(j\)\\exp\\big\(\-\\frac\{b\(j;\\pi,p\)\}\{\\tau\}\\big\)\\Big\\\},\(12\)whereb⁡\(j,π,p\)=∑i=1mπ⁡\(i∣x\)​Pp​\(i≻j\)b\(j;\\pi,p\)=\\sum\_\{i=1\}^\{m\}\\pi\(i\\mid x\)P\_\{p\}\(i\\succ j\)is affine inpp\. Since the log\-sum\-exp is convex and appears with a negative sign, the reduced objective is concave inpp\. However, minimizing a concave function over a convex uncertainty set is NP\-hard in general\[[18](https://arxiv.org/html/2610.00715#bib.bib18)\]\. On the other hand, if one fixesπ,ν\\pi,\\nufirst and solves for the worst\-case kernelp⋆p^\{\\star\}, since the uncertainty set𝒫\\mathcal\{P\}is defined with correlation among response pairs,𝒫\\mathcal\{P\}is not rectangular over all\(i,j\)\(i,j\)pairs; thus, solving the distributionally robust optimization problem can still be inefficient or NP\-hard\[[48](https://arxiv.org/html/2610.00715#bib.bib48)\]\. Moreover, standard NLHF involves only two strategic blocks\(π,ν\)\(\\pi,\\nu\), yet our robust game has another player controllingpp\. Although one can combine the two minimizing players as a single adversary who takes the joint action\(ν,p\)\(\\nu,p\), it breaks the standard two\-player saddle problem structure in NLHF, losing all of its properties like the minimax theorem or unique Nash equilibrium\[[29](https://arxiv.org/html/2610.00715#bib.bib29)\]\. The resulting game need not be monotone in the Euclidean sense, which makes naive simultaneous gradient methods unstable\.

Due to these challenges, we develop a proxy game formulation that is easier to solve and develop detailed studies on the connection between the proxy and the original robust game\.

For regularity, we first assume the main player policy is constrained to the floor\-truncated simplexΠϵ:=\{π∈Δm:πi≥ϵπref,i∀i\}\\Pi\_\{\\epsilon\}:=\\left\\\{\\pi\\in\\Delta\_\{m\}:\\ \\pi\_\{i\}\\geq\\epsilon\\,\\pi\_\{\\mathrm\{ref\},i\}\\ \\ \\forall i\\right\\\}for some positive valueϵ\\epsilon\. Specifically, one can chooseϵ\\epsilonarbitrarily small to approximately recover the probability simplexΔm\\Delta\_\{m\}\. For eachπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}, denote the original inner problem value with the hard constraintp∈𝒫p\\in\\mathcal\{P\}as

gH​\(π,ρ\):=minν∈Δm⁡minp∈𝒫⁡J⁡\(π,ν,p\),g\_\{H\}\(\\pi;\\rho\):=\\min\_\{\\nu\\in\\Delta\_\{m\}\}\\min\_\{p\\in\\mathcal\{P\}\}J\(\\pi,\\nu;p\),\(13\)and the corresponding optimal robust value as

V⁡\(ρ\):=maxπ∈Πϵ⁡gH​\(π,ρ\)\.V\(\\rho\):=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}\(\\pi;\\rho\)\.\(14\)
We further define an interior boxPδ:=\[δ,1−δ\]dP\_\{\\delta\}:=\[\\delta,1\-\\delta\]^\{d\}\. with someδ∈\(0,1/2\)\\delta\\in\(0,1/2\)due to regularity, and consider the truncated hard objective:

gHδ​\(π,ρ\):=minν∈Δm⁡minp∈Pδ∩𝒫⁡J⁡\(π,ν,p\),g\_\{H\}^\{\\delta\}\(\\pi;\\rho\):=\\min\_\{\\nu\\in\\Delta\_\{m\}\}\\min\_\{\\begin\{subarray\}\{c\}p\\in P\_\{\\delta\}\\cap\\mathcal\{P\}\\end\{subarray\}\}J\(\\pi,\\nu;p\),\(15\)and denote its optimal value as

Vδ​\(ρ\):=maxπ∈Πϵ⁡gHδ​\(π,ρ\)\.V^\{\\delta\}\(\\rho\):=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\.\(16\)
We first show that the truncation error is explicit and uniformly controlled\.

###### Theorem 1\(Uniform approximation by the truncated hard objective\)\.

Assumep⋆∈Pδp^\{\\star\}\\in P\_\{\\delta\}\. Then, for everyπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\},0≤gHδ​\(π,ρ\)−gH​\(π,ρ\)≤δ0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{H\}\(\\pi;\\rho\)\\leq\\deltaand0≤Vδ​\(ρ\)−V⁡\(ρ\)≤δ\.0\\leq V^\{\\delta\}\(\\rho\)\-V\(\\rho\)\\leq\\delta\.Moreover, ifπδ⋆∈arg​maxπ∈Πϵ⁡gHδ​\(π,ρ\)\\pi\_\{\\delta\}^\{\\star\}\\in\\argmax\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}^\{\\delta\}\(\\pi;\\rho\), thengH​\(πδ⋆,ρ\)≥V⁡\(ρ\)−δ\.g\_\{H\}\(\\pi\_\{\\delta\}^\{\\star\};\\rho\)\\geq V\(\\rho\)\-\\delta\.

This theorem shows that the error introduced by the regularity relaxation is bounded byδ\\delta\. Thus, one can chooseδ\\deltasmall enough to ensure regularity and approximate the original solution arbitrarily\. Therefore, we will focus on solving this problem,arg⁡maxπ∈Πϵ​gHδ​\(π,ρ\)\\arg\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\. SincegHδg^\{\\delta\}\_\{H\}is a hard\-constrained problem, which can be inefficient to solve directly, we introduce a Lagrangian dual variable and consider the Lagrangian function\. Specifically, with an additional dual variableγ\\gamma, we define and consider the four\-player payoff as

F\(π,γ,ν,p\):=J\(π,ν;p\)\+γ\(DKLBern\(p∥p⋆\)−ρ\)\.F\(\\pi,\\gamma,\\nu,p\):=J\(\\pi,\\nu;p\)\+\\gamma\\bigl\(D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho\\bigr\)\.\(17\)Moreover, fix a compact positive intervalΓ:=\[γmin,γmax\]⊂\(0,∞\)\\Gamma:=\[\\gamma\_\{\\min\},\\gamma\_\{\\max\}\]\\subset\(0,\\infty\)and𝒩β:=\{ν∈Δm:νj≥βπref,j∀j\}\\mathcal\{N\}\_\{\\beta\}:=\\left\\\{\\nu\\in\\Delta\_\{m\}:\\ \\nu\_\{j\}\\geq\\beta\\,\\pi\_\{\\mathrm\{ref\},j\}\\ \\ \\forall j\\right\\\}forβ∈\(0,1\)\\beta\\in\(0,1\), we define the proxy objective as

gproxyδ​\(π,ρ\):=maxγ∈Γ⁡minν∈𝒩β,p∈Pδ⁡F⁡\(π,γ,ν,p\),g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\):=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\},\\;p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\),\(18\)and its optimal value as

Vproxyδ​\(ρ\):=maxπ∈Πϵ⁡gproxyδ​\(π,ρ\)\.V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\):=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\.\(19\)We then study the proxy’s connection to the original problem\. Firstly, we introduce a regularity assumption\.

###### Assumption 2\.

We assume the following regularization conditions hold: \(1\)p⋆∈\(δ,1−δ\)dp^\{\\star\}\\in\(\\delta,1\-\\delta\)^\{d\}; \(2\)0<β≤e−1/τ0<\\beta\\leq e^\{\-1/\\tau\}; \(3\)Γ=\[γmin,γmax\]⊂\(0,∞\)\\Gamma=\[\\gamma\_\{\\min\},\\gamma\_\{\\max\}\]\\subset\(0,\\infty\)withγmax≥1/ρ\\gamma\_\{\\max\}\\geq 1/\\rho; \(4\)\.τ​γmin≥14\\tau\\gamma\_\{\\min\}\\geq\\frac\{1\}\{4\}\.

Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(1\)–\(3\) are mild regularity conditions that can be enforced by suitable algorithmic choices\. Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(4\) is a curvature condition needed for the proxy\-game convergence analysis\.

We then derive the following results for the connections\.

###### Theorem 2\.

Under \(1\)\-\(3\) of Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2), for everyπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}, the proxy value always lower bounds the hard\-constrained value:

gHδ​\(π,ρ\)=minν∈𝒩β⁡maxγ∈\[0,γmax\]​minp∈Pδ⁡F⁡\(π,γ,ν,p\)≥gproxyδ​\(π,ρ\)\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)=\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)\\geq g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\.\(20\)If we additionally assume \(4\) of Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2), then for everyπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\},

0≤gHδ​\(π,ρ\)−gproxyδ​\(π,ρ\)≤ρ​γmin,and​0≤Vδ​\(ρ\)−Vproxyδ​\(ρ\)≤ρ​γmin\.0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\leq\\rho\\gamma\_\{\\min\},\\text\{ and \}0\\leq V^\{\\delta\}\(\\rho\)\-V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\leq\\rho\\gamma\_\{\\min\}\.\(21\)Moreover, if Assumption[3](https://arxiv.org/html/2610.00715#Thmassumption3)\(in Appendix\) additionally holds, then the two objectives are equivalent:

gHδ​\(π,ρ\)=gproxyδ​\(π,ρ\),and​Vδ​\(ρ\)=Vproxyδ​\(ρ\)\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)=g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\),\\text\{ and \}V^\{\\delta\}\(\\rho\)=V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\.\(22\)

This theorem characterizes the comprehensive connections between our proxy valuegHδg^\{\\delta\}\_\{H\}and the hard\-constrained ones\. Firstly, we showed that under the mildest conditions \(1\)\-\(3\) of Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2), the proxy value is always a lower bound ofgHδg^\{\\delta\}\_\{H\}\. Thus, any solution to the proxy also provides an optimized lower bound to the hard\-constrained problem\. Moreover, if Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(4\) holds, the error betweengHδg^\{\\delta\}\_\{H\}andgproxyδg\_\{\\mathrm\{proxy\}\}^\{\\delta\}is then bounded byρ​γmin\\rho\\gamma\_\{\\min\}\. Thus, one can set the values ofγmin\\gamma\_\{\\min\}and the temperatureτ\\tauto satisfy Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(4\) and to guarantee that the proxy is an accurate approximation of the original problem: recall the error betweengproxyδg\_\{\\mathrm\{proxy\}\}^\{\\delta\}and the original problemgHg\_\{H\}is up toδ\+ρ​γmin\\delta\+\\rho\\gamma\_\{\\min\}, which can be properly small\. Moreover, Assumption[3](https://arxiv.org/html/2610.00715#Thmassumption3)is a stronger exactness condition used only to upgrade the proxy guarantee to the truncated hard\-constrained objective\. If it holds, which assumes a uniformly lower bound of the maximizerγ⋆\\gamma^\{\\star\}under any\(π,ν\)\(\\pi,\\nu\), the errorρ​γmin\\rho\\gamma\_\{\\min\}diminishes and the proxy is exactly equivalent to the hard\-constrainedgHδg^\{\\delta\}\_\{H\}\.

The advantage of studying the proxy is that the proxy has a much better structure and is easier to design convergent algorithms\. Specifically, the hard\-constrained problemgHδg^\{\\delta\}\_\{H\}is amin−max−min\\min\-\\max\-\\minproblem per \([20](https://arxiv.org/html/2610.00715#S4.E20)\), and maximizing overπ\\pifurther makes it a four\-layermax−min−max−min\\max\-\\min\-\\max\-\\minproblem, which is significantly challenging and unstable to solve\. The proxy exchanges the order ofmaxγ\\max\_\{\\gamma\}andminν\\min\_\{\\nu\}, makingmaxπ⁡gproxyδ\\max\_\{\\pi\}g\_\{\\mathrm\{proxy\}\}^\{\\delta\}a bi\-levelmaxπ,γ−minp,ν\\max\_\{\\pi,\\gamma\}\-\\min\_\{p,\\nu\}problem\. Such amax−min\\max\-\\minis much easier to solve, and we can view it as a two\-player zero\-sum game and seek for its saddle point\. In the next section, we design a concrete algorithm based on this understanding, and develop its convergence analysis\.

## 5Algorithm Design and Convergence Analysis

In this section, we develop our concrete algorithm for solving the proxy game\. Specifically, we treat each parameter as a player, and develop a four\-player gradient\-based algorithm\. As we discussed previously, the structure of the proxy enables us to view it as amax−min\\max\-\\minproblem\. Also, since the robust games are non\-monotone and naive simultaneous gradient approaches can be unstable, we thus propose to adapt optimistic or extra\-gradient approaches\[[7](https://arxiv.org/html/2610.00715#bib.bib7),[37](https://arxiv.org/html/2610.00715#bib.bib37),[42](https://arxiv.org/html/2610.00715#bib.bib42),[27](https://arxiv.org/html/2610.00715#bib.bib27),[28](https://arxiv.org/html/2610.00715#bib.bib28),[47](https://arxiv.org/html/2610.00715#bib.bib47)\]\. Specifically, we aim to solve the proxymax−min\\max\-\\minproblem

maxx∈X⁡miny∈Y⁡F⁡\(x,y\),x=\(π,γ\)∈X:=Πϵ×Γ,y=\(ν,p\)∈Y:=𝒩β×Pδ,\\max\_\{x\\in X\}\\min\_\{y\\in Y\}F\(x,y\),\\quad x=\(\\pi,\\gamma\)\\in X:=\\Pi\_\{\\epsilon\}\\times\\Gamma,\\ y=\(\\nu,p\)\\in Y:=\\mathcal\{N\}\_\{\\beta\}\\times P\_\{\\delta\},by a single\-loop optimistic mirror descent/ascent method\.

We use entropy geometry for the simplex blocks, Euclidean geometry forγ\\gamma, and binary entropy geometry forpp\. The associated Bregman divergences are:Dπ\(π,π′\)=DKL\(π∥π′\),D\_\{\\pi\}\(\\pi,\\pi^\{\\prime\}\)=D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi^\{\\prime\}\),Dγ​\(γ,γ′\)=12​\(γ−γ′\)2D\_\{\\gamma\}\(\\gamma,\\gamma^\{\\prime\}\)=\\frac\{1\}\{2\}\(\\gamma\-\\gamma^\{\\prime\}\)^\{2\},Dν\(ν,ν′\)=DKL\(ν∥ν′\)D\_\{\\nu\}\(\\nu,\\nu^\{\\prime\}\)=D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu^\{\\prime\}\), andDp\(p,p′\)=DKLBern\(p∥p′\)\.D\_\{p\}\(p,p^\{\\prime\}\)=D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\prime\}\)\.At iterationtt, let

zt=\(πt,γt,νt,pt\)∈X×Y,gt∙:=∇∙F​\(zt\),z\_\{t\}=\(\\pi\_\{t\},\\gamma\_\{t\},\\nu\_\{t\},p\_\{t\}\)\\in X\\times Y,\\qquad g\_\{t\}^\{\\bullet\}:=\\nabla\_\{\\bullet\}F\(z\_\{t\}\),and defineg^t∙:=2​gt∙−gt−1∙,\\widehat\{g\}\_\{t\}^\{\\bullet\}:=2g\_\{t\}^\{\\bullet\}\-g\_\{t\-1\}^\{\\bullet\},with∙∈\{π,γ,ν,p\}\.\\bullet\\in\\\{\\pi,\\gamma,\\nu,p\\\}\.Then, we update each player through mirror descent \(exact update rules are deferred to Appendix \([25](https://arxiv.org/html/2610.00715#A2.E25)\)\-\([28](https://arxiv.org/html/2610.00715#A2.E28)\)\) and present our algorithm in Algorithm[1](https://arxiv.org/html/2610.00715#alg1)\. Since the mirror maps are separable across blocks, these coordinate\-wise updates are exactly optimistic mirror descent/ascent on the grouped saddle problemmaxx∈X⁡miny∈Y⁡F⁡\(x,y\)\\max\_\{x\\in X\}\\min\_\{y\\in Y\}F\(x,y\)\.

Algorithm 1Four\-player optimistic gradient descent/ascent \(OGDA\)1:step size

η\>0\\eta\>0, iteration budget

TT, initial point

z1=\(π1,γ1,ν1,p1\)∈X×Yz\_\{1\}=\(\\pi\_\{1\},\\gamma\_\{1\},\\nu\_\{1\},p\_\{1\}\)\\in X\\times Y
2:Compute

g1∙=∇∙F​\(z1\)g\_\{1\}^\{\\bullet\}=\\nabla\_\{\\bullet\}F\(z\_\{1\}\)for all blocks, and set

g0∙←g1∙g\_\{0\}^\{\\bullet\}\\leftarrow g\_\{1\}^\{\\bullet\}
3:for

t=1,…,Tt=1,\\dots,Tdo

4:Form optimistic gradients

g^t∙=2​gt∙−gt−1∙\\widehat\{g\}\_\{t\}^\{\\bullet\}=2g\_\{t\}^\{\\bullet\}\-g\_\{t\-1\}^\{\\bullet\}
5:Update

\(πt\+1,γt\+1,νt\+1,pt\+1\)\(\\pi\_\{t\+1\},\\gamma\_\{t\+1\},\\nu\_\{t\+1\},p\_\{t\+1\}\)by \([25](https://arxiv.org/html/2610.00715#A2.E25)\)\-\([28](https://arxiv.org/html/2610.00715#A2.E28)\)

6:if

t<Tt<Tthen

7:Compute

gt\+1∙=∇∙F​\(zt\+1\)g\_\{t\+1\}^\{\\bullet\}=\\nabla\_\{\\bullet\}F\(z\_\{t\+1\}\)for all blocks

8:endif

9:endfor

10:Return the averaged iterate

z¯T=\(π¯T,γ¯T,ν¯T,p¯T\):=1T​∑t=1T\(πt,γt,νt,pt\)\.\\bar\{z\}\_\{T\}=\(\\bar\{\\pi\}\_\{T\},\\bar\{\\gamma\}\_\{T\},\\bar\{\\nu\}\_\{T\},\\bar\{p\}\_\{T\}\):=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\(\\pi\_\{t\},\\gamma\_\{t\},\\nu\_\{t\},p\_\{t\}\)\.

We then develop the convergence analysis of our algorithm in the following theorem\.

###### Theorem 3\(Convergence and robust guarantee of four\-player OGDA\)\.

Assume Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2), and run Algorithm[1](https://arxiv.org/html/2610.00715#alg1)with the constant step sizeη=18​Lδ,\\eta=\\frac\{1\}\{8L\_\{\\delta\}\},whereLδ=𝒪⁡\(1/δ\)L\_\{\\delta\}=\\mathcal\{O\}\(1/\\delta\)is the constant from Lemma[3](https://arxiv.org/html/2610.00715#Thmlemma3)\. DefineDXdiam≜sup\(π,γ\)∈X\{DKL\(π∥π1\)\+12\(γ−γ1\)2\},D\_\{X\}^\{\\mathrm\{diam\}\}\\triangleq\\sup\_\{\(\\pi,\\gamma\)\\in X\}\\left\\\{D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{1\}\)\+\\frac\{1\}\{2\}\(\\gamma\-\\gamma\_\{1\}\)^\{2\}\\right\\\},andDYdiam≜sup\(ν,p\)∈Y\{DKL\(ν∥ν1\)\+DKLBern\(p∥p1\)\}\.D\_\{Y\}^\{\\mathrm\{diam\}\}\\triangleq\\sup\_\{\(\\nu,p\)\\in Y\}\\left\\\{D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu\_\{1\}\)\+D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p\_\{1\}\)\\right\\\}\.Then the duality gap of the averaged iteratez¯T=\(x¯T,y¯T\)=\(\(π¯T,γ¯T\),\(ν¯T,p¯T\)\)\\bar\{z\}\_\{T\}=\(\\bar\{x\}\_\{T\},\\bar\{y\}\_\{T\}\)=\\bigl\(\(\\bar\{\\pi\}\_\{T\},\\bar\{\\gamma\}\_\{T\}\),\(\\bar\{\\nu\}\_\{T\},\\bar\{p\}\_\{T\}\)\\bigr\)satisfies

Gap⁡\(z¯T\):=maxx′∈X⁡F⁡\(x′,y¯T\)−miny′∈Y⁡F⁡\(x¯T,y′\)≤8​Lδ​\(DXdiam\+DYdiam\)T,\\Gap\(\\bar\{z\}\_\{T\}\):=\\max\_\{x^\{\\prime\}\\in X\}F\(x^\{\\prime\},\\bar\{y\}\_\{T\}\)\-\\min\_\{y^\{\\prime\}\\in Y\}F\(\\bar\{x\}\_\{T\},y^\{\\prime\}\)\\leq\\frac\{8L\_\{\\delta\}\\bigl\(D\_\{X\}^\{\\mathrm\{diam\}\}\+D\_\{Y\}^\{\\mathrm\{diam\}\}\\bigr\)\}\{T\},and the averaged main policyπ¯T\\bar\{\\pi\}\_\{T\}satisfiesV⁡\(ρ\)−gH​\(π¯T,ρ\)≤8​Lδ​\(DXdiam\+DYdiam\)T\+ρ​γmin\+δ\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq\\frac\{8L\_\{\\delta\}\\bigl\(D\_\{X\}^\{\\mathrm\{diam\}\}\+D\_\{Y\}^\{\\mathrm\{diam\}\}\\bigr\)\}\{T\}\+\\rho\\,\\gamma\_\{\\min\}\+\\delta\.Furthermore, settingδ=𝒪⁡\(1/T\)\\delta=\\mathcal\{O\}\(1/\\sqrt\{T\}\), yields an𝒪⁡\(1T\+ρ​γmin\)\\mathcal\{O\}\\left\(\\frac\{1\}\{\\sqrt\{T\}\}\+\\rho\\,\\gamma\_\{\\min\}\\right\)\-optimal policy for the original robust game\. Additionally, under Assumption[3](https://arxiv.org/html/2610.00715#Thmassumption3),π¯T\\bar\{\\pi\}\_\{T\}improves to𝒪⁡\(1/T\)\\mathcal\{O\}\\big\(1/\\sqrt\{T\}\\big\)\-optimality\.

Our OGDA algorithm thus converges to a saddle point of the proxy game, which is also a near\-optimal solution to our robust objective \([9](https://arxiv.org/html/2610.00715#S3.E9)\)\. Therefore, our algorithm effectively solves our robust alignment problem, enhancing the robustness and reliability of Nash learning based alignment\. A detailed comparison of our approach with standard NLHF is developed in Appendix[C](https://arxiv.org/html/2610.00715#A3)\.

## 6Experiment

We then evaluate our four\-player OGDA algorithm across three settings of increasing scale: \(i\) a synthetic tabular game that isolates the fragile\-consensus phenomenon, \(ii\) a best\-of\-mmreranking task on the IMDb dataset with heterogeneous annotator groups, and \(iii\) a summarization task with LLM\-generated responses and external LLM judgments\.

### 6\.1Tabular Game

We first consider a collection of independently generated tabular prompts, each with a finite response set of sizem=3m=3\. For each promptxx, we construct a*fragile\-consensus*preference kernel by drawing a slim marginsx∼Unif⁡\[0\.015,0\.040\]s\_\{x\}\\sim\\mathrm\{Unif\}\[0\.015,0\.040\]and a solid marginSx∼Unif⁡\[0\.25,0\.40\]S\_\{x\}\\sim\\mathrm\{Unif\}\[0\.25,0\.40\], and setting

Px⋆=\(1/21/2\+sx1/2−Sx1/2−sx1/21/2\+Sx1/2\+Sx1/2−Sx1/2\)\.P^\{\\star\}\_\{x\}\\;=\\;\\begin\{pmatrix\}1/2&1/2\+s\_\{x\}&1/2\-S\_\{x\}\\\\\[3\.0pt\] 1/2\-s\_\{x\}&1/2&1/2\+S\_\{x\}\\\\\[3\.0pt\] 1/2\+S\_\{x\}&1/2\-S\_\{x\}&1/2\\end\{pmatrix\}\.\(23\)We then apply a random permutation of the response labels so that the identity of the fragile action varies across prompts\. This construction produces a cyclic preference game with one easily reversible edge \(KL cost≈2​sx2≤0\.003\\approx 2s\_\{x\}^\{2\}\\leq 0\.003\) and two expensive\-to\-reverse edges \(KL cost≈2​Sx2/\(Sx​\(1−Sx\)\)≥0\.5\\approx 2S\_\{x\}^\{2\}/\(S\_\{x\}\(1\-S\_\{x\}\)\)\\geq 0\.5\), which is precisely the regime in which robustness matters most: the adversary can flip the cheap edge within a small budgetρ\\rho, reversing the dominance ordering, while the expensive edges are untouchable\. We use a uniform reference policy and setτ=0\.05\\tau=0\.05\.

We compare: \(i\)NashMD, the nominal two\-player mirror\-descent baseline operating on the fixed kernelPx⋆P^\{\\star\}\_\{x\}; \(ii\)Four\-Player OGDAwith training radiiρ∈\{0\.10,0\.20,0\.40\}\\rho\\in\\\{0\.10,0\.20,0\.40\\\}; and \(iii\) theoptimal robust valueV⁡\(ρ=0\.2\)V\(\\rho=0\.2\), computed via low\-dimensional global search\. Both NashMD and OGDA are initialized atπref\\pi\_\{\\text\{ref\}\}and trained for 2000 iterations\. At each steptt, we evaluategH​\(πt,ρ=0\.2\)g\_\{H\}\(\\pi\_\{t\};\\rho=0\.2\)—the worst\-case game value under the Bernoulli\-KL ball of radius0\.20\.2\.

Figure[1](https://arxiv.org/html/2610.00715#S6.F1)\(left\) shows the convergence ofgH​\(πt,0\.2\)g\_\{H\}\(\\pi\_\{t\};0\.2\)over training iterations\. NashMD converges to a fixed value that is strictly below the optimal robust valueV⁡\(0\.2\)V\(0\.2\), confirming that the nominal Nash policy is suboptimal for the robust objective\. The four\-player OGDA atρ=0\.2\\rho=0\.2converges to a value close toV⁡\(0\.2\)V\(0\.2\), validating the convergence guarantee\. Atρ=0\.1\\rho=0\.1\(under\-budgeted\), the algorithm converges but to a lower value, since it optimizes for a smaller perturbation set\. Atρ=0\.4\\rho=0\.4\(over\-budgeted\), the policy is more conservative and also falls below theρ=0\.2\\rho=0\.2optimum\.

Figure[1](https://arxiv.org/html/2610.00715#S6.F1)\(right\) showsgH​\(π,ρeval\)g\_\{H\}\(\\pi;\\rho\_\{\\mathrm\{eval\}\}\)as a function of the evaluation radius for the final policies\. The four\-player OGDA policies uniformly dominate NashMD at every positiveρeval\\rho\_\{\\mathrm\{eval\}\}, with the advantage growing as the evaluation radius increases\. This further confirms the enhanced robustness of our approach against preference uncertainty\.

![Refer to caption](https://arxiv.org/html/2610.00715v1/figures/game1.png)

![Refer to caption](https://arxiv.org/html/2610.00715v1/figures/game2.png)

Figure 1:Experiments under Tabular Games
### 6\.2Best\-of\-mmIMDb Continuation Task

We next validate our algorithm on a best\-of\-mmreranking task, where candidate responses are generated by a pretrained LLM and preference kernels are constructed from heterogeneous annotator groups\. Details of our training procedure, hyperparameters, and additional results are deferred to Appendix[E](https://arxiv.org/html/2610.00715#A5)\.

For each ofN=8000N=8000prompts from the IMDb review dataset\[[23](https://arxiv.org/html/2610.00715#bib.bib23)\], we pre\-generatem=8m=8candidate continuations usinggoogle/flan\-t5\-large\[[11](https://arxiv.org/html/2610.00715#bib.bib11)\]\. We generate the preferences based on three annotator groups:Comprehensive/analytical\(p1\)\(p\_\{1\}\)prefers detailed, well\-structured responses \(favors length 300–600 characters\);Efficient/concise\(p2\)\(p\_\{2\}\)prefers short, direct responses \(favors length 50–120 characters\);Balanced/nuanced\(p3\)\(p\_\{3\}\)prefers hedged, multi\-perspective responses \(favors hedging keywords\)\. We set the nominal kernel as the uniform mixture of the three preferences:Pp⋆​\(a≻b\)=\(∑i=13Pi​\(a≻b\)\)/3P\_\{p^\{\\star\}\}\(a\\succ b\)=\(\\sum^\{3\}\_\{i=1\}P\_\{i\}\(a\\succ b\)\)/3, and construct the uncertainty set centered at it as𝒫ρeval≜\{\(w1,w2,w3\)∈Δ3:DKL\(\(w1,w2,w3\)\|\|\(1/3,1/3,1/3\)\)≤ρeval\}\\mathcal\{P\}\_\{\\rho\_\{\\mathrm\{eval\}\}\}\\triangleq\\\{\(w\_\{1\},w\_\{2\},w\_\{3\}\)\\in\\Delta\_\{3\}:D\_\{\\text\{KL\}\}\(\(w\_\{1\},w\_\{2\},w\_\{3\}\)\|\|\(1/3,1/3,1/3\)\)\\leq\\rho\_\{\\mathrm\{eval\}\}\\\}to model preference uncertainty\.

We then train and compare the robust policyπR\\pi\_\{R\}\(trained at radiusρtrain\\rho\_\{\\mathrm\{train\}\}\) against the NashMD baselineπN\\pi\_\{N\}by their pairwise*win rate*under perturbed kernels\. For each evaluation radiusρeval\\rho\_\{\\mathrm\{eval\}\}, we sampleM=600M=600kernels\{Pj\}j=1M\\\{P\_\{j\}\\\}\_\{j=1\}^\{M\}uniformly from𝒫ρeval\\mathcal\{P\}\_\{\\rho\_\{\\mathrm\{eval\}\}\}and compute the win rate:

WinRate⁡\(πR​vs​πN,ρeval\):=\|\{j∈\[M\]:Ppj​\(πR≻πN\)≥0\.5\}\|/M,\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)\\;:=\\;\\bigl\|\\\{j\\in\[M\]:P\_\{p\_\{j\}\}\(\\pi\_\{R\}\\succ\\pi\_\{N\}\)\\geq 0\.5\\\}\\bigr\|\\big/M,\(24\)wherePpj​\(πR≻πN\)=∑i,kπR​\(i\)​πN​\(k\)​pj​\(i≻k\)P\_\{p\_\{j\}\}\(\\pi\_\{R\}\\succ\\pi\_\{N\}\)=\\sum\_\{i,k\}\\pi\_\{R\}\(i\)\\,\\pi\_\{N\}\(k\)\\,p\_\{j\}\(i\\succ k\)is the probability that a response drawn fromπR\\pi\_\{R\}is preferred over one drawn fromπN\\pi\_\{N\}under kernelpjp\_\{j\}\. A win rate above50%50\\%means the robust policy is preferred to NashMD under the majority of plausible kernels\. Table[1](https://arxiv.org/html/2610.00715#S6.T1)reports the win rate of the robust policy against NashMD, averaged over all80008000prompts, for training radiiρtrain∈\{0\.01,0\.02,0\.05,0\.10,0\.15\}\\rho\_\{\\mathrm\{train\}\}\\in\\\{0\.01,0\.02,0\.05,0\.10,0\.15\\\}and evaluation radiiρeval∈\{0\.05,0\.10,0\.20,0\.30,0\.50\}\\rho\_\{\\mathrm\{eval\}\}\\in\\\{0\.05,0\.10,0\.20,0\.30,0\.50\\\}\.

Table 1:Win rate against NashMD under random kernel perturbations, averaged overN=8000N=8000IMDb prompts withm=8m=8candidates\. Each entry reportsWinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)from \([24](https://arxiv.org/html/2610.00715#S6.E24)\)\. Values above0\.500\.50indicate the robust policy is preferred under the majority of sampled kernels\.As the results shown, our robust policy outperforms standard NashMD uniformly, indicating that the robust policy is preferred under most sampled kernels that are different from the nominal training preference\. Therefore, our method substantially improves robustness to preference uncertainty and out\-of\-distribution generalization\.

### 6\.3Summarization Task with LLMs

We further evaluate our performance on the TL;DR summarization task\[[41](https://arxiv.org/html/2610.00715#bib.bib41)\]\. The nominal preference kernel is a non\-Bradley\-Terry \(non\-BT\) pairwise judge,Pϕpair​\(ya≻yb∣x\)=σ⁡\(Lpair​\(x,ya,yb\)\)P^\{\\mathrm\{pair\}\}\_\{\\phi\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)=\\sigma\\big\(L^\{\\mathrm\{pair\}\}\(x,y\_\{a\},y\_\{b\}\)\\big\), instantiated with PairRM\[[21](https://arxiv.org/html/2610.00715#bib.bib21)\]: since each candidate’s score is conditioned on its opponent, the kernel is not constrained to admit any scalar reward decomposition\. On real model outputs, the closest Bradley\-Terry fit to this kernel leaves a mean residual of0\.3840\.384logits per edge \(3\.103\.10percentage points\), with1\.4%1\.4\\%of triples outright cyclic, confirming the nominal kernel used here is measurably non\-BT \(construction and diagnostic in Appendix[F\.6](https://arxiv.org/html/2610.00715#A6.SS6)\)\. To evaluate robustness under preference uncertainty, we use three external LLMs from different model families:DeepSeek\-R1\-Distill\-Qwen\-32B\[[12](https://arxiv.org/html/2610.00715#bib.bib12)\],Gemma\-4\-31B\[[44](https://arxiv.org/html/2610.00715#bib.bib44)\], andNemotron\-3\-Nano\-Omni\[[32](https://arxiv.org/html/2610.00715#bib.bib32)\], as judgment at evaluation time\. These external LLMs are instructed to judge the summary with focus on accuracy, coherence, conciseness, and helpfulness to the reader\. Since each external judge will induce its own pairwise preference \(from their own reasoning\) that differs from the nominal one, these judgments simulate a deployment scenario where the user population’s preferences diverge from the training\-time annotation distribution\. Details of the experiments are deferred to Appendix[F](https://arxiv.org/html/2610.00715#A6)\.

The results are presented in Table[2](https://arxiv.org/html/2610.00715#S6.T2)\. Most external judges \(five of six configurations\) yield a win rate above0\.500\.50over NashMD, confirming that the robust policy remains competitive under this non\-BT nominal kernel even though, unlike prior work, our framework does not require a Bradley\-Terry reward model\. Appendix[F\.7](https://arxiv.org/html/2610.00715#A6.SS7)reports additional results with a Bradley\-Terry nominal kernel across both fine\-tuning backbones\.

Table 2:Win rate against NashMD under external LLM judges, evaluated on10001000TL;DR prompts, with a non\-BT \(PairRM\) nominal kernel\.Fine\-Tuning Modelρtrain\\rho\_\{\\text\{train\}\}Win Rate under Evaluation ModelDeepSeek\-R1Gemma\-4Nemotron\-3Qwen\-1\.5B0\.010\.7080\.5770\.4920\.050\.6250\.6150\.732Summary\.Across all three settings, the experiments confirm our central theoretical prediction:*the four\-player OGDA algorithm produces policies that are strictly more robust than the non\-robust NashMD baseline, as the preference differs from the training ones\.*

## 7Conclusion

We addressed the sensitivity of preference\-based alignment to uncertainty in the pairwise preference kernel by proposing Robust Nash Alignment, which optimizes the worst\-case win rate over a divergence\-based ambiguity set and provides a principled robustness guarantee\. To overcome the computational challenges of the resulting hard\-constrained robust objective, we introduced a tractable four\-player primal\-dual proxy game and established its approximation properties and convergence via a single\-loop optimistic mirror descent\-ascent algorithm\. Empirically, our method improves robustness to preference perturbations across tabular and LLM settings while maintaining strong nominal performance\. Overall, our results demonstrate that explicitly modeling preference uncertainty is a principled and practical approach to achieving reliable Nash\-style alignment\.

## References

- \[1\]M\. Aghassi and D\. Bertsimas\.Robust game theory\.Mathematical programming, 107\(1\):231–273, 2006\.
- \[2\]S\. M\. Ananthanarayanan and C\. Kroer\.Computing the Optimal Distributionally\-Robust Strategy to Commit To\.arXiv preprint arXiv:2209\.07647, 2022\.
- \[3\]M\. G\. Azar, Z\. D\. Guo, B\. Piot, R\. Munos, M\. Rowland, M\. Valko, and D\. Calandriello\.A General Theoretical Paradigm to Understand Learning from Human Preferences\.InInternational Conference on Artificial Intelligence and Statistics, pages 4447–4455\. PMLR, 2024\.
- \[4\]R\. A\. Bradley and M\. E\. Terry\.Rank analysis of incomplete block designs: I\. the method of paired comparisons\.Biometrika, 39\(3/4\):324–345, 1952\.
- \[5\]D\. Calandriello, D\. Guo, R\. Munos, M\. Rowland, Y\. Tang, B\. A\. Pires, P\. H\. Richemond, C\. L\. Lan, M\. Valko, T\. Liu, et al\.Human Alignment of Large Language Models through Online Preference Optimisation\.arXiv preprint arXiv:2403\.08635, 2024\.
- \[6\]S\. Chakraborty, J\. Qiu, H\. Yuan, A\. Koppel, F\. Huang, D\. Manocha, A\. S\. Bedi, and M\. Wang\.MaxMin\-RLHF: Alignment with Diverse Human Preferences\.arXiv preprint arXiv:2402\.08925, 2024\.
- \[7\]C\.\-K\. Chiang, T\. Yang, C\.\-J\. Lee, M\. Mahdavi, C\.\-J\. Lu, R\. Jin, and S\. Zhu\.Online Optimization with Gradual Variations\.InConference on Learning Theory, pages 6–1\. JMLR Workshop and Conference Proceedings, 2012\.
- \[8\]K\. Chidambaram, K\. V\. Seetharaman, and V\. Syrgkanis\.Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences\.arXiv preprint arXiv:2510\.15716, 2025\.
- \[9\]P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei\.Deep Reinforcement Learning from Human Preferences\.Advances in neural information processing systems, 30, 2017\.
- \[10\]X\. Chu, Z\. Zhang, T\. Jia, and Y\. Jin\.Stackelberg Self\-Annotation: A Robust Approach to Data\-Efficient LLM Alignment\.arXiv preprint arXiv:2502\.18099, 2025\.
- \[11\]H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, E\. Li, X\. Wang, M\. Dehghani, S\. Brahma, A\. Webson, S\. S\. Gu, Z\. Dai, M\. Suzgun, X\. Chen, A\. Chowdhery, S\. Narang, G\. Mishra, A\. Yu, V\. Zhao, Y\. Huang, A\. Dai, H\. Yu, S\. Petrov, E\. H\. Chi, J\. Dean, J\. Devlin, A\. Roberts, D\. Zhou, Q\. V\. Le, and J\. Wei\.Scaling Instruction\-Finetuned Language Models, 2022\.
- \[12\]DeepSeek\-AI\.DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025\.
- \[13\]K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela\.KTO: Model Alignment as Prospect Theoretic Optimization\.arXiv preprint arXiv:2402\.01306, 2024\.
- \[14\]H\. Furuta, K\.\-H\. Lee, S\. S\. Gu, Y\. Matsuo, A\. Faust, H\. Zen, and I\. Gur\.Geometric\-Averaged Preference Optimization for Soft Preference Labels\.Advances in Neural Information Processing Systems, 37:57076–57114, 2024\.
- \[15\]J\. Hejna, R\. Rafailov, H\. Sikchi, C\. Finn, S\. Niekum, W\. B\. Knox, and D\. Sadigh\.Contrastive Preference Learning: Learning from Human Feedback without RL\.arXiv preprint arXiv:2310\.13639, 2023\.
- \[16\]I\. Hong, Z\. Li, A\. Bukharin, Y\. Li, H\. Jiang, T\. Yang, and T\. Zhao\.Adaptive Preference Scaling for Reinforcement Learning with Human Feedback\.Advances in Neural Information Processing Systems, 37:107249–107269, 2024\.
- \[17\]J\. Hong, N\. Lee, and J\. Thorne\.ORPO: Monolithic Preference Optimization without Reference Model\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024\.
- \[18\]R\. Horst and H\. Tuy\.Global Optimization: Deterministic Approaches\.Springer Science & Business Media, 2013\.
- \[19\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\.LoRA: Low\-Rank Adaptation of Large Language Models, 2021\.
- \[20\]G\. N\. Iyengar\.Robust Dynamic Programming\.Mathematics of Operations Research, 30\(2\):257–280, 2005\.
- \[21\]D\. Jiang, X\. Ren, and B\. Y\. Lin\.LLM\-Blender: Ensembling large language models with pairwise ranking and generative fusion\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 14165–14178, Toronto, Canada, 2023\. Association for Computational Linguistics\.
- \[22\]Y\. Liu, H\. Xu, S\.\-J\. S\. Yang, and J\. Zhang\.Distributionally robust equilibrium for continuous games: Nash and Stackelberg models\.European Journal of Operational Research, 265\(2\):631–643, 2018\.
- \[23\]A\. L\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. Potts\.Learning Word Vectors for Sentiment Analysis\.InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011\. Association for Computational Linguistics\.
- \[24\]D\. Mandal, P\. Sasnauskas, and G\. Radanovic\.Distributionally Robust Reinforcement Learning with Human Feedback\.arXiv preprint arXiv:2503\.00539, 2025\.
- \[25\]J\. Mcmahan, G\. Artiglio, and Q\. Xie\.Roping in uncertainty: Robustness and regularization in markov games\.InInternational Conference on Machine Learning, pages 35267–35295\. PMLR, 2024\.
- \[26\]Y\. Meng, M\. Xia, and D\. Chen\.SimPO: Simple Preference Optimization with a Reference\-Free Reward \.Advances in Neural Information Processing Systems, 37:124198–124235, 2024\.
- \[27\]P\. Mertikopoulos, B\. Lecouat, H\. Zenati, C\.\-S\. Foo, V\. Chandrasekhar, and G\. Piliouras\.Optimistic mirror descent in saddle\-point problems: Going the extra \(gradient\) mile\.arXiv preprint arXiv:1807\.02629, 2018\.
- \[28\]A\. Mokhtari, A\. Ozdaglar, and S\. Pattathil\.A Unified Analysis of Extra\-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach\.InInternational Conference on Artificial Intelligence and Statistics, pages 1497–1507\. PMLR, 2020\.
- \[29\]R\. Munos, M\. Valko, D\. Calandriello, M\. G\. Azar, M\. Rowland, Z\. D\. Guo, Y\. Tang, M\. Geist, T\. Mesnard, C\. Fiegel, et al\.Nash Learning from Human Feedback\.InForty\-first International Conference on Machine Learning, 2024\.
- \[30\]A\. Nayak, T\. Yang, O\. Yagan, G\. Joshi, and Y\. Chi\.Achieving Logarithmic Regret in KL\-Regularized Zero\-Sum Markov Games\.arXiv preprint arXiv:2510\.13060, 2025\.
- \[31\]A\. Nilim and L\. El Ghaoui\.Robust control of markov decision processes with uncertain transition matrices\.Operations Research, 53\(5\):780–798, 2005\.
- \[32\]NVIDIA, :, A\. S\. Deshmukh, K\. Chumachenko, T\. Rintamaki, M\. Le, T\. Poon, D\. M\. Taheri, I\. Karmanov, G\. Liu, J\. Seppanen, A\. Goel, M\. Ranzinger, G\. Heinrich, G\. Chen, L\. Voegtle, P\. Fischer, T\. Roman, K\. Sapra, C\. McCarthy, S\. Zhang, F\. Liu, H\. Ye, Y\. Dong, M\. Liu, Y\. Peng, P\. Zelasko, Z\. Chen, N\. R\. Koluguri, N\. Tadevosyan, L\. Grigoryan, E\. H\. Asl, P\. Biswas, L\. Tavabi, Y\. Su, Z\. Yu, P\. Jin, A\. Milesi, N\. Haber, Y\. Xu, S\. Amiraslani, N\. Mulepati, E\. Tramel, J\. Jung, X\. Lu, B\. Cui, J\. Xu, Z\. Li, S\. Wang, Y\. Kuang, S\. Zhang, H\. Yang, B\. Li, H\. Yin, S\. Han, P\. Molchanov, A\. Renduchintala, C\. Wang, D\. Mosallanezhad, S\. Singhal, L\. Vega, K\. Cheung, S\. Ghosh, Y\. Zhang, A\. Bukharin, V\. Srinivasan, J\. Greco, A\. Manoel, M\. V\. Segbroeck, S\. Panguliri, R\. Watve, D\. Kakwani, S\. Pachori, J\. Glick, R\. Sri\-Tharan, A\. Zaman, K\. Nguyen, S\. Chen, J\. Fang, Q\. Miao, W\. Zhou, Y\. Wang, Z\. P\. Bhat, V\. Praveen, A\. Jain, R\. Arunachalam, T\. Kornuta, A\. Sharabiani, A\. Shen, W\. Huang, Y\.\-F\. Wu, A\. R\. Ghias, H\. Li, B\. Yu, N\. Tajbakhsh, C\. Cui, W\. Gao, L\. Ding, T\. Kong, M\. Kilaru, A\. Bhiwandiwalla, M\. Wawrzos, D\. Korzekwa, P\. Ribalta, G\. Chlebus, B\. Nushi, E\. Dobrowolska, M\. J\. Mikulski, K\. Dhawan, S\. Huang, J\. Balam, Y\. Wang, N\. Karpov, V\. Mendelev, G\. Zelenfroynd, M\. Mkrtchyan, Q\. Miao, O\. Almog, B\. Pawar, R\. Shivbhakta, S\. Sabnis, A\. Sharabiani, N\. Habibi, G\. Venkataramani, P\. Peng, P\. Rodney, S\. Panev, R\. Mazzarese, N\. Liu, M\. Fukuyama, A\. Skliar, R\. Waleffe, D\. Riach, Y\. Zou, J\. Hu, H\. Zhang, B\. Xu, Y\. Yang, Z\. Ahmed, A\. Milesi, C\. del Mundo, C\. Voegele, Z\. Cheng, N\. Assaf, A\. Skliar, D\. Afrimi, N\. Bagrov, R\. Zilberstein, O\. Masad, E\. Khvedchenia, N\. Bagrov, B\. Tymchenko, T\. Asida, D\. Afrimi, P\. Mannan, V\. Cui, M\. Evans, K\. Luna, J\. Lou, P\. Xu, G\. Huang, N\. Habibi, M\. Boone, P\. Thalasta, A\. Adesoba, D\. Yared, C\. Parisien, L\. Derczynski, S\. Ghosh, W\. Feely, M\. Schaffer, R\. Sri\-Tharan, J\. Glick, B\. Simkin, G\. Zelenfroynd, T\. Grzegorzek, R\. Garg, A\. Jhunjhunwala, S\. Kolchenko, F\. Memarian, H\. Kumar, S\. Kumar, I\. Hulseman, A\. Shah, K\. Briski, P\. Subramanian, J\. Conway, U\. Karpas, J\. P\. Scowcroft, A\. Surla, S\. Ammireddy, E\. Evans, J\. Oliver, T\. Balough, C\.\-C\. Chen, S\. Bhaskar, A\. Rico, B\. Sadeghi, S\. Mard, K\. Cheung, M\. Price, L\. Sleiman, S\. Kaji, W\. Helmholz, W\. Quan, M\. Lightstone, J\. Cohen, J\. Zhang, O\. Kuchaiev, B\. Ginsburg, J\. Kautz, E\. Long, M\. Shoeybi, M\. Patwary, O\. Olabiyi, A\. Tao, B\. Catanzaro, and U\. Karpas\.Nemotron 3 nano omni: Efficient and open multimodal intelligence, 2026\.
- \[33\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems, 35:27730–27744, 2022\.
- \[34\]B\. Pásztor, T\. K\. Buening, and A\. Krause\.Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game\.InNeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2025\.
- \[35\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\.Direct Preference Optimization: Your Language Model is Secretly a Reward Model\.Advances in neural information processing systems, 36:53728–53741, 2023\.
- \[36\]H\. Rahimian and S\. Mehrotra\.Distributionally Robust Optimization: A Review\.arXiv preprint arXiv:1908\.05659, 2019\.
- \[37\]A\. Rakhlin and K\. Sridharan\.Online Learning with Predictable Sequences\.InConference on Learning Theory, pages 993–1019\. PMLR, 2013\.
- \[38\]S\. Sagawa, P\. W\. Koh, T\. Hashimoto, and P\. Liang\.Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst\-Case Generalization\.InProc\. International Conference on Learning Representations \(ICLR\), 2020\.
- \[39\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\.DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.ArXiv, abs/1910\.01108, 2019\.
- \[40\]A\. Sharifnassab, S\. Salehkaleybar, S\. Ghiassian, S\. Kanoria, and D\. Schuurmans\.Soft Preference Optimization: Aligning Language Models to Expert Distributions\.arXiv preprint arXiv:2405\.00747, 2024\.
- \[41\]N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano\.Learning to summarize with Human Feedback\.Advances in neural information processing systems, 33:3008–3021, 2020\.
- \[42\]V\. Syrgkanis, A\. Agarwal, H\. Luo, and R\. E\. Schapire\.Fast Convergence of Regularized Learning in Games\.Advances in Neural Information Processing Systems, 28, 2015\.
- \[43\]Y\. Tang, Z\. D\. Guo, Z\. Zheng, D\. Calandriello, R\. Munos, M\. Rowland, P\. H\. Richemond, M\. Valko, B\. Á\. Pires, and B\. Piot\.Generalized Preference Optimization: A Unified Approach to Offline Alignment\.arXiv preprint arXiv:2402\.05749, 2024\.
- \[44\]G\. Team, T\. Mesnard, C\. Hardin, R\. Dadashi, S\. Bhupatiraju, S\. Pathak, L\. Sifre, M\. Rivière, M\. S\. Kale, J\. Love, P\. Tafti, L\. Hussenot, P\. G\. Sessa, A\. Chowdhery, A\. Roberts, A\. Barua, A\. Botev, A\. Castro\-Ros, A\. Slone, A\. Héliou, A\. Tacchetti, A\. Bulanova, A\. Paterson, B\. Tsai, B\. Shahriari, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Crepy, D\. Cer, D\. Ippolito, D\. Reid, E\. Buchatskaya, E\. Ni, E\. Noland, G\. Yan, G\. Tucker, G\.\-C\. Muraru, G\. Rozhdestvenskiy, H\. Michalewski, I\. Tenney, I\. Grishchenko, J\. Austin, J\. Keeling, J\. Labanowski, J\.\-B\. Lespiau, J\. Stanway, J\. Brennan, J\. Chen, J\. Ferret, J\. Chiu, J\. Mao\-Jones, K\. Lee, K\. Yu, K\. Millican, L\. L\. Sjoesund, L\. Lee, L\. Dixon, M\. Reid, M\. Mikuła, M\. Wirth, M\. Sharman, N\. Chinaev, N\. Thain, O\. Bachem, O\. Chang, O\. Wahltinez, P\. Bailey, P\. Michel, P\. Yotov, R\. Chaabouni, R\. Comanescu, R\. Jana, R\. Anil, R\. McIlroy, R\. Liu, R\. Mullins, S\. L\. Smith, S\. Borgeaud, S\. Girgin, S\. Douglas, S\. Pandya, S\. Shakeri, S\. De, T\. Klimenko, T\. Hennigan, V\. Feinberg, W\. Stokowiec, Y\. hui Chen, Z\. Ahmed, Z\. Gong, T\. Warkentin, L\. Peran, M\. Giang, C\. Farabet, O\. Vinyals, J\. Dean, K\. Kavukcuoglu, D\. Hassabis, Z\. Ghahramani, D\. Eck, J\. Barral, F\. Pereira, E\. Collins, A\. Joulin, N\. Fiedel, E\. Senter, A\. Andreev, and K\. Kenealy\.Gemma: Open Models Based on Gemini Research and Technology, 2024\.
- \[45\]L\. L\. Thurstone\.Psychophysical Analysis\.The American journal of psychology, 38\(3\):368–389, 1927\.
- \[46\]D\. Tiapkin, D\. Calandriello, D\. Belomestny, E\. Moulines, A\. Naumov, K\. Rasul, M\. Valko, and P\. Menard\.Proximal Point Nash Learning from Human Feedback\.arXiv preprint arXiv:2505\.19731, 2025\.
- \[47\]C\.\-Y\. Wei, C\.\-W\. Lee, M\. Zhang, and H\. Luo\.Linear Last\-iterate Convergence in Constrained Saddle\-point Optimization\.arXiv preprint arXiv:2006\.09517, 2020\.
- \[48\]W\. Wiesemann, D\. Kuhn, and B\. Rustem\.Robust Markov Decision Processes\.Mathematics of Operations Research, 38\(1\):153–183, 2013\.
- \[49\]J\. Wu, Y\. Xie, Z\. Yang, J\. Wu, J\. Chen, J\. Gao, B\. Ding, X\. Wang, and X\. He\.Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization\.arXiv preprint arXiv:2407\.07880, 2024\.
- \[50\]Z\. Xu, S\. Vemuri, K\. Panaganti, D\. Kalathil, R\. Jain, and D\. Ramachandran\.Robust LLM Alignment via Distributionally Robust Direct Preference Optimization\.arXiv preprint arXiv:2502\.01930, 2025\.
- \[51\]A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Fan\.Qwen2 technical report\.arXiv preprint arXiv:2407\.10671, 2024\.
- \[52\]C\. Ye, W\. Xiong, Y\. Zhang, H\. Dong, N\. Jiang, and T\. Zhang\.Online Iterative Reinforcement Learning from Human Feedback with General Preference Model\.Advances in Neural Information Processing Systems, 37:81773–81807, 2024\.
- \[53\]K\. Zhang, T\. Sun, Y\. Tao, S\. Genc, S\. Mallya, and T\. Basar\.Robust multi\-agent reinforcement learning with model uncertainty\.InProc\. Advances in Neural Information Processing Systems \(NeurIPS\), volume 33, 2020\.
- \[54\]Y\. Zhang, D\. Yu, B\. Peng, L\. Song, Y\. Tian, M\. Huo, N\. Jiang, H\. Mi, and D\. Yu\.Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No\-Regret Learning\.InInternational Conference on Learning Representations \(ICLR\), 2025\.General\-preference RLHF as a two\-player game; OMD\-based last\-iterate convergence; KL\-regularized formulation\.
- \[55\]Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu\.SLiC\-HF: Sequence Likelihood Calibration with Human Feedback\.arXiv preprint arXiv:2305\.10425, 2023\.
- \[56\]R\. Zhou, M\. Fazel, and S\. S\. Du\.Extragradient Preference Optimization \(EGPO\): Beyond Last\-Iterate Convergence for Nash Learning from Human Feedback\.arXiv preprint arXiv:2503\.08942, 2025\.
- \[57\]B\. Zhu, M\. Jordan, and J\. Jiao\.Principled Reinforcement Learning with Human Feedback from Pairwise or K\-wise Comparisons\.InInternational Conference on Machine Learning, pages 43037–43067\. PMLR, 2023\.

Appendix

Table of Contents

[E\.4 Results](https://arxiv.org/html/2610.00715#A5.SS4)\.[E\.4](https://arxiv.org/html/2610.00715#A5.SS4)

[F\.4 Losses](https://arxiv.org/html/2610.00715#A6.SS4)\.[F\.4](https://arxiv.org/html/2610.00715#A6.SS4)

## Appendix ARelated Work

Reward\-based RLHF and direct alignment\.A large body of work on alignment starts from the RLHF pipeline: collect pairwise comparisons, fit a reward model, and optimize a policy against that reward subject to a regularization term that keeps the policy close to a reference model\[[9](https://arxiv.org/html/2610.00715#bib.bib9),[41](https://arxiv.org/html/2610.00715#bib.bib41),[33](https://arxiv.org/html/2610.00715#bib.bib33)\]\. This paradigm has been highly influential in large\-scale language\-model alignment, but it typically treats the preference signal as a nominal target and often relies, either explicitly or implicitly, on a latent\-utility view of pairwise comparisons\. Motivated by the computational and modeling limitations of reward\-model\-based RLHF, a broad line of direct or reward\-free preference optimization methods has emerged, including DPO, SLiC\-HF, CPL, KTO, ORPO, SimPO, SPO, and adaptive preference scaling\[[35](https://arxiv.org/html/2610.00715#bib.bib35),[55](https://arxiv.org/html/2610.00715#bib.bib55),[15](https://arxiv.org/html/2610.00715#bib.bib15),[13](https://arxiv.org/html/2610.00715#bib.bib13),[17](https://arxiv.org/html/2610.00715#bib.bib17),[26](https://arxiv.org/html/2610.00715#bib.bib26),[40](https://arxiv.org/html/2610.00715#bib.bib40),[16](https://arxiv.org/html/2610.00715#bib.bib16)\]\. These methods substantially improve the practical pipeline of preference optimization, but they generally continue to optimize against a fixed nominal preference estimate rather than an ambiguity set of plausible preference kernels\.

General\-preference and game\-theoretic alignment\.Recent work has argued that pairwise human preferences should be modeled directly rather than compressed into a scalar reward function\[[3](https://arxiv.org/html/2610.00715#bib.bib3)\]\. This viewpoint leads naturally to game\-theoretic formulations of alignment\. In particular, Nash Learning from Human Feedback \(NLHF\) formulates preference alignment as a regularized two\-player game and introduces Nash\-MD as a no\-regret algorithm for the induced general\-preference game\[[29](https://arxiv.org/html/2610.00715#bib.bib29)\]\. Subsequent works have further developed this perspective, including online and offline algorithms under general preference models and policy\-loss implementations based on no\-regret learning\[[52](https://arxiv.org/html/2610.00715#bib.bib52),[54](https://arxiv.org/html/2610.00715#bib.bib54),[30](https://arxiv.org/html/2610.00715#bib.bib30),[56](https://arxiv.org/html/2610.00715#bib.bib56)\]\. More recently, sequential leader–follower formulations of preference optimization have also begun to appear, including Stackelberg\-style alignment methods and Stackelberg game variants for data\-efficient alignment\[[34](https://arxiv.org/html/2610.00715#bib.bib34),[10](https://arxiv.org/html/2610.00715#bib.bib10)\]\. Our work is closest in spirit to this line of research, but differs in one central respect: we do not assume that the learned preference game is exact\. Instead, we explicitly model uncertainty in preference kernels and optimize against the worst kernel in an ambiguity set\. This brings significant challenges and hardness to our studies, and we discuss them in detail in Section[C](https://arxiv.org/html/2610.00715#A3)\.

Soft labels, heterogeneous preferences, and population shift\.Another line of work emphasizes that preference data are often soft, noisy, and heterogeneous across annotators or user groups\. Recent studies incorporate soft preference labels or pairwise confidence information into direct alignment objectives\[[16](https://arxiv.org/html/2610.00715#bib.bib16),[40](https://arxiv.org/html/2610.00715#bib.bib40),[14](https://arxiv.org/html/2610.00715#bib.bib14)\], while other work explicitly models hidden annotator types or heterogeneous preference populations\[[8](https://arxiv.org/html/2610.00715#bib.bib8)\]\. These papers highlight an important limitation of nominal preference optimization: even if one optimizes the nominal objective correctly, the aggregated preference signal may still be unstable because it hides substantial variation across annotators or deployment populations\. Our formulation is complementary to this line of work\. Rather than only modifying the loss to account for label softness or latent subgroups, we directly treat the learned pairwise preference kernel as uncertain and seek a policy with a worst\-case performance guarantee over that uncertainty set\.

Robust alignment and distributionally robust preference optimization\.Our formulation is also related to the broader literature on distributionally robust optimization and robust sequential decision making, where ambiguity sets are used to hedge estimation error and distribution shift\[[38](https://arxiv.org/html/2610.00715#bib.bib38),[20](https://arxiv.org/html/2610.00715#bib.bib20),[31](https://arxiv.org/html/2610.00715#bib.bib31)\]\. In the alignment literature, recent work has started to robustify reward\-based RLHF or DPO\-style objectives against prompt\-distribution shift, noisy preference data, or uncertain preference strength\[[16](https://arxiv.org/html/2610.00715#bib.bib16),[6](https://arxiv.org/html/2610.00715#bib.bib6),[49](https://arxiv.org/html/2610.00715#bib.bib49),[50](https://arxiv.org/html/2610.00715#bib.bib50),[24](https://arxiv.org/html/2610.00715#bib.bib24)\]\. These methods are close in motivation to ours, but they robustify different objects: prompt distributions, training examples, or nominal pairwise losses\. More importantly, all these works are based on the reward and BT\-model assumption\. By contrast, our approach directly robustifies the*pairwise preference kernel itself*inside a general\-preference game\. This distinction is important because kernel uncertainty directly captures ambiguity in preferences, which is the primitive object in general\-preference alignment\.

Robust game theory and robust Markov games\.A separate line of work studies robustness to payoff or model uncertainty directly in game\-theoretic solution concepts, rather than in a single\-agent decision problem\. This includes robust game theory under set\-based payoff uncertainty\[[1](https://arxiv.org/html/2610.00715#bib.bib1)\], distributionally robust Nash and Stackelberg equilibrium models under exogenous distributional uncertainty\[[22](https://arxiv.org/html/2610.00715#bib.bib22)\], and distributionally robust commitment to a strategy when the follower’s utility is known only up to a finite set of candidate models\[[2](https://arxiv.org/html/2610.00715#bib.bib2)\]\. Robust Markov games extend this to multi\-stage settings with uncertain rewards or transitions\[[53](https://arxiv.org/html/2610.00715#bib.bib53),[25](https://arxiv.org/html/2610.00715#bib.bib25)\]\. Our setting differs from this literature in two respects\. First, robust Markov games typically seek a robust Nash equilibrium between symmetric players, whereas our objective is asymmetric, as we have a single learned policy against a strategic follower and a shared adversarial kernel\. Second, a single Bernoulli\-KL budget couples every response pair within a prompt, ruling out the per\-action or per\-player decomposition, and the multi\-stage Bellman recursion that robust Markov\-game algorithms rely on\. Existing robust\-game and robust\-Markov\-game results therefore do not directly provide our proxy formulation or its convergence guarantee\.

## Appendix BMain results

### B\.1Algorithm Update Rules

πt\+1\\displaystyle\\pi\_\{t\+1\}=arg​maxπ∈Πϵ\{⟨g^tπ,π⟩−1ηDKL\(π∥πt\)\},\\displaystyle=\\argmax\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}\\left\\\{\\langle\\widehat\{g\}\_\{t\}^\{\\pi\},\\pi\\rangle\-\\frac\{1\}\{\\eta\}D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{t\}\)\\right\\\},\(25\)γt\+1\\displaystyle\\gamma\_\{t\+1\}=ProjΓ⁡\(γt\+η​g^tγ\),\\displaystyle=\\proj\_\{\\Gamma\}\\bigl\(\\gamma\_\{t\}\+\\eta\\widehat\{g\}\_\{t\}^\{\\gamma\}\\bigr\),\(26\)νt\+1\\displaystyle\\nu\_\{t\+1\}=arg​minν∈𝒩β\{⟨g^tν,ν⟩\+1ηDKL\(ν∥νt\)\},\\displaystyle=\\argmin\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\left\\\{\\langle\\widehat\{g\}\_\{t\}^\{\\nu\},\\nu\\rangle\+\\frac\{1\}\{\\eta\}D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu\_\{t\}\)\\right\\\},\(27\)pt\+1\\displaystyle p\_\{t\+1\}=arg​minp∈Pδ\{⟨g^tp,p⟩\+1ηDKLBern\(p∥pt\)\}\.\\displaystyle=\\argmin\_\{p\\in P\_\{\\delta\}\}\\left\\\{\\langle\\widehat\{g\}\_\{t\}^\{p\},p\\rangle\+\\frac\{1\}\{\\eta\}D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p\_\{t\}\)\\right\\\}\.\(28\)
###### Assumption 3\(Dual\-optimum inclusion\)\.

For every\(π,ν\)∈Πϵ×𝒩β\(\\pi,\\nu\)\\in\\Pi\_\{\\epsilon\}\\times\\mathcal\{N\}\_\{\\beta\}, the scalar dual problem

maxγ≥0minp∈Pδ\{J\(π,ν;p\)\+γ\(DKLBern\(p∥p⋆\)−ρ\)\}\\max\_\{\\gamma\\geq 0\}\\min\_\{p\\in P\_\{\\delta\}\}\\left\\\{J\(\\pi,\\nu;p\)\+\\gamma\\bigl\(D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho\\bigr\)\\right\\\}admits a maximizer in the algorithmic intervalΓ=\[γmin,γmax\]\\Gamma=\[\\gamma\_\{\\min\},\\gamma\_\{\\max\}\]\.

### B\.2Proof of Theorem[1](https://arxiv.org/html/2610.00715#Thmtheorem1)

###### Proof\.

Fixπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}\. Since the feasible set in \([15](https://arxiv.org/html/2610.00715#S4.E15)\) is a subset of the feasible set in \([13](https://arxiv.org/html/2610.00715#S4.E13)\), the lower bound

gHδ​\(π,ρ\)≥gH​\(π,ρ\)g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\\geq g\_\{H\}\(\\pi;\\rho\)is immediate\. It remains to show the upper bound\. Fixν∈Δm\\nu\\in\\Delta\_\{m\}, and let

K\(ρ\):=\{p∈\[0,1\]d:DKLBern\(p∥p⋆\)≤ρ\}\.K\(\\rho\):=\\left\\\{p\\in\[0,1\]^\{d\}:\\ D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\right\\\}\.Take anyp∈K⁡\(ρ\)p\\in K\(\\rho\), and clip it coordinatewise intoPδP\_\{\\delta\}:p~:=ProjPδ⁡\(p\)\.\\widetilde\{p\}:=\\proj\_\{P\_\{\\delta\}\}\(p\)\.Becausep⋆∈Pδp^\{\\star\}\\in P\_\{\\delta\}, each scalarp~e\\widetilde\{p\}\_\{e\}lies on the line segment joiningpep\_\{e\}andpe⋆p^\{\\star\}\_\{e\}\. For each coordinateee, the scalar function

u↦u​log⁡upe⋆\+\(1−u\)​log⁡1−u1−pe⋆u\\mapsto u\\log\\frac\{u\}\{p\_\{e\}^\{\\star\}\}\+\(1\-u\)\\log\\frac\{1\-u\}\{1\-p\_\{e\}^\{\\star\}\}is convex and has its unique minimum atu=pe⋆u=p\_\{e\}^\{\\star\}\. Hence clipping towardpe⋆p^\{\\star\}\_\{e\}cannot increase the coordinatewise Bernoulli\-KL divergence, so

DKLBern\(p~∥p⋆\)≤DKLBern\(p∥p⋆\)≤ρ\.D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(\\widetilde\{p\}\\\|p^\{\\star\}\)\\leq D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\.Thusp~\\widetilde\{p\}is feasible for the truncated problem\.

NowJ⁡\(π,ν,p\)J\(\\pi,\\nu;p\)depends onpponly through the linear term∑\(i,j\)∈Ewi​j​\(π,ν\)​pi​j\.\\sum\_\{\(i,j\)\\in E\}w\_\{ij\}\(\\pi,\\nu\)\\,p\_\{ij\}\.Therefore

\|J⁡\(π,ν,p~\)−J⁡\(π,ν,p\)\|\\displaystyle\|J\(\\pi,\\nu;\\widetilde\{p\}\)\-J\(\\pi,\\nu;p\)\|=\|∑\(i,j\)∈Ewi​j​\(π,ν\)​\(p~i​j−pi​j\)\|≤‖p~−p‖∞​∑\(i,j\)∈E\|wi​j​\(π,ν\)\|\.\\displaystyle=\\left\|\\sum\_\{\(i,j\)\\in E\}w\_\{ij\}\(\\pi,\\nu\)\\,\(\\widetilde\{p\}\_\{ij\}\-p\_\{ij\}\)\\right\|\\leq\\\|\\widetilde\{p\}\-p\\\|\_\{\\infty\}\\sum\_\{\(i,j\)\\in E\}\|w\_\{ij\}\(\\pi,\\nu\)\|\.Since each coordinate is clipped by at mostδ\\delta, we have

‖p~−p‖∞≤δ\.\\\|\\widetilde\{p\}\-p\\\|\_\{\\infty\}\\leq\\delta\.Also,

∑\(i,j\)∈E\|wi​j​\(π,ν\)\|\\displaystyle\\sum\_\{\(i,j\)\\in E\}\|w\_\{ij\}\(\\pi,\\nu\)\|≤∑\(i,j\)∈E\(πi​νj\+πj​νi\)=∑i≠jπi​νj=1−∑i=1mπi​νi≤1\.\\displaystyle\\leq\\sum\_\{\(i,j\)\\in E\}\(\\pi\_\{i\}\\nu\_\{j\}\+\\pi\_\{j\}\\nu\_\{i\}\)=\\sum\_\{i\\neq j\}\\pi\_\{i\}\\nu\_\{j\}=1\-\\sum\_\{i=1\}^\{m\}\\pi\_\{i\}\\nu\_\{i\}\\leq 1\.Hence

J⁡\(π,ν,p~\)≤J⁡\(π,ν,p\)\+δ\.J\(\\pi,\\nu;\\widetilde\{p\}\)\\leq J\(\\pi,\\nu;p\)\+\\delta\.Taking the minimum first overp∈K⁡\(ρ\)p\\in K\(\\rho\), then overν∈Δm\\nu\\in\\Delta\_\{m\}, yields

gHδ​\(π,ρ\)≤gH​\(π,ρ\)\+δ\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\\leq g\_\{H\}\(\\pi;\\rho\)\+\\delta\.This proves the claim0≤gHδ​\(π,ρ\)−gH​\(π,ρ\)≤δ0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{H\}\(\\pi;\\rho\)\\leq\\delta\. Maximizing overπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}gives0≤Vδ​\(ρ\)−V⁡\(ρ\)≤δ0\\leq V^\{\\delta\}\(\\rho\)\-V\(\\rho\)\\leq\\delta\.

Using the above two claims, we have

V⁡\(ρ\)≤Vδ​\(ρ\)=gHδ​\(πδ⋆,ρ\)≤gH​\(πδ⋆,ρ\)\+δ\.V\(\\rho\)\\leq V^\{\\delta\}\(\\rho\)=g\_\{H\}^\{\\delta\}\(\\pi\_\{\\delta\}^\{\\star\};\\rho\)\\leq g\_\{H\}\(\\pi\_\{\\delta\}^\{\\star\};\\rho\)\+\\delta\.Rearranging gives the claim\.∎

### B\.3Proof of Theorem[2](https://arxiv.org/html/2610.00715#Thmtheorem2)

###### Proof\.

Fixπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}, and define

ϕπ​\(ν,γ\):=minp∈Pδ⁡F⁡\(π,γ,ν,p\)\.\\phi\_\{\\pi\}\(\\nu,\\gamma\):=\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)\.
Step 1: reduction to𝒩β\\mathcal\{N\}\_\{\\beta\}and nested strong duality\.For fixed\(π,p\)\(\\pi,p\), define

bj\(π,p\):=∑i=1mπ\(i\)Pp\(i≻j\),j=1,…,m\.b\_\{j\}\(\\pi,p\):=\\sum\_\{i=1\}^\{m\}\\pi\(i\)\\,P\_\{p\}\(i\\succ j\),\\qquad j=1,\\dots,m\.SincePp​\(i≻j\)∈\[0,1\]P\_\{p\}\(i\\succ j\)\\in\[0,1\]and∑iπ⁡\(i\)=1\\sum\_\{i\}\\pi\(i\)=1, we havebj​\(π,p\)∈\[0,1\]b\_\{j\}\(\\pi,p\)\\in\[0,1\]for alljj\.

For fixed\(π,p\)\(\\pi,p\), the follower subproblem is

minν∈Δm\{∑j=1mν\(j\)bj\(π,p\)\+τDKL\(ν∥πref\)\}\.\\min\_\{\\nu\\in\\Delta\_\{m\}\}\\left\\\{\\sum\_\{j=1\}^\{m\}\\nu\(j\)\\,b\_\{j\}\(\\pi,p\)\+\\tau D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\pi\_\{\\rm ref\}\)\\right\\\}\.Its unique minimizer is the Gibbs distribution

νjbr​\(π,p\)=πref\(j\)e−bj\(π,p\)/τ∑ℓ=1mπref\(ℓ\)e−bℓ\(π,p\)/τ\.\\nu\_\{j\}^\{\\mathrm\{br\}\}\(\\pi,p\)=\\frac\{\\pi\_\{\\rm ref\}\(j\)e^\{\-b\_\{j\}\(\\pi,p\)/\\tau\}\}\{\\sum\_\{\\ell=1\}^\{m\}\\pi\_\{\\rm ref\}\(\\ell\)e^\{\-b\_\{\\ell\}\(\\pi,p\)/\\tau\}\}\.Becausebj​\(π,p\)∈\[0,1\]b\_\{j\}\(\\pi,p\)\\in\[0,1\], we havee−1/τ≤e−bj\(π,p\)/τ≤1e^\{\-1/\\tau\}\\leq e^\{\-b\_\{j\}\(\\pi,p\)/\\tau\}\\leq 1\. Let

Z\(π,p\):=∑ℓ=1mπref\(ℓ\)e−bℓ\(π,p\)/τ\.Z\(\\pi,p\):=\\sum\_\{\\ell=1\}^\{m\}\\pi\_\{\\rm ref\}\(\\ell\)e^\{\-b\_\{\\ell\}\(\\pi,p\)/\\tau\}\.Since∑ℓπref​\(ℓ\)=1\\sum\_\{\\ell\}\\pi\_\{\\rm ref\}\(\\ell\)=1and each exponential factor is at most11, we haveZ⁡\(π,p\)≤1Z\(\\pi,p\)\\leq 1\. Therefore

νjbr\(π,p\)=πref\(j\)e−bj\(π,p\)/τZ⁡\(π,p\)≥πref\(j\)e−1/τ\.\\nu\_\{j\}^\{\\mathrm\{br\}\}\(\\pi,p\)=\\pi\_\{\\rm ref\}\(j\)\\frac\{e^\{\-b\_\{j\}\(\\pi,p\)/\\tau\}\}\{Z\(\\pi,p\)\}\\geq\\pi\_\{\\rm ref\}\(j\)e^\{\-1/\\tau\}\.Under Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(2\),β≤e−1/τ\\beta\\leq e^\{\-1/\\tau\}, this implies

νbr​\(π,p\)∈𝒩β\.\\nu^\{\\mathrm\{br\}\}\(\\pi,p\)\\in\\mathcal\{N\}\_\{\\beta\}\.Hence, for every\(π,p\)\(\\pi,p\),

minν∈Δm⁡J⁡\(π,ν,p\)=minν∈𝒩β⁡J⁡\(π,ν,p\)\.\\min\_\{\\nu\\in\\Delta\_\{m\}\}J\(\\pi,\\nu;p\)=\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}J\(\\pi,\\nu;p\)\.\(29\)The same identity holds withJ⁡\(π,ν,p\)J\(\\pi,\\nu;p\)replaced byF⁡\(π,γ,ν,p\)F\(\\pi,\\gamma,\\nu,p\), since the dual termγ\(DKLBern\(p∥p⋆\)−ρ\)\\gamma\(D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho\)does not depend onν\\nu\.

Now fixν∈𝒩β\\nu\\in\\mathcal\{N\}\_\{\\beta\}\. Thepp\-subproblem

minp∈PδDKLBern\(p∥p⋆\)≤ρ⁡J⁡\(π,ν,p\)\\min\_\{\\begin\{subarray\}\{c\}p\\in P\_\{\\delta\}\\\\ D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\end\{subarray\}\}J\(\\pi,\\nu;p\)is a convex program: the objective is affine inpp, the feasible set is convex, and Slater’s condition holds because Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(1\) givesp⋆∈\(δ,1−δ\)dp^\{\\star\}\\in\(\\delta,1\-\\delta\)^\{d\}andDKLBern\(p⋆∥p⋆\)=0<ρD\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p^\{\\star\}\\\|p^\{\\star\}\)=0<\\rho\. Therefore strong duality yields

minp∈PδDKLBern\(p∥p⋆\)≤ρ⁡J⁡\(π,ν,p\)=maxγ≥0⁡minp∈Pδ⁡F⁡\(π,γ,ν,p\)\.\\min\_\{\\begin\{subarray\}\{c\}p\\in P\_\{\\delta\}\\\\ D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\end\{subarray\}\}J\(\\pi,\\nu;p\)=\\max\_\{\\gamma\\geq 0\}\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)\.Under Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(3\),γmax≥1/ρ\\gamma\_\{\\max\}\\geq 1/\\rho, and the standard dual upper\-bound argument implies that a maximizer can be chosen in\[0,γmax\]\[0,\\gamma\_\{\\max\}\]\. Consequently,

minp∈PδDKLBern\(p∥p⋆\)≤ρ⁡J⁡\(π,ν,p\)=maxγ∈\[0,γmax\]⁡minp∈Pδ⁡F⁡\(π,γ,ν,p\)\.\\min\_\{\\begin\{subarray\}\{c\}p\\in P\_\{\\delta\}\\\\ D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\end\{subarray\}\}J\(\\pi,\\nu;p\)=\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)\.Combining this with \([29](https://arxiv.org/html/2610.00715#A2.E29)\), we obtain

gHδ​\(π,ρ\)=minν∈𝒩β⁡maxγ∈\[0,γmax\]​minp∈Pδ⁡F⁡\(π,γ,ν,p\)\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)=\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)\.\(30\)
Step 2: proof of the lower bound \([20](https://arxiv.org/html/2610.00715#S4.E20)\)\.By definition,

gproxyδ​\(π,ρ\)=maxγ∈Γ⁡minν∈𝒩β,p∈Pδ⁡F⁡\(π,γ,ν,p\)=maxγ∈Γ⁡minν∈𝒩β​ϕπ​\(ν,γ\)\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\},\\;p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)\.SinceΓ=\[γmin,γmax\]⊆\[0,γmax\]\\Gamma=\[\\gamma\_\{\\min\},\\gamma\_\{\\max\}\]\\subseteq\[0,\\gamma\_\{\\max\}\],

gproxyδ​\(π,ρ\)≤maxγ∈\[0,γmax\]⁡minν∈𝒩β​ϕπ​\(ν,γ\)\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\leq\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)\.By weak minimax,

maxγ∈\[0,γmax\]⁡minν∈𝒩β​ϕπ​\(ν,γ\)≤minν∈𝒩β⁡maxγ∈\[0,γmax\]​ϕπ​\(ν,γ\)\.\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)\\leq\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)\.Using \([30](https://arxiv.org/html/2610.00715#A2.E30)\), the right\-hand side equalsgHδ​\(π,ρ\)g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\. Therefore

gproxyδ​\(π,ρ\)≤gHδ​\(π,ρ\),g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\),which proves \([20](https://arxiv.org/html/2610.00715#S4.E20)\)\.

Step 3: excluding the interval\[0,γmin\)\[0,\\gamma\_\{\\min\}\)costs at mostρ​γmin\\rho\\gamma\_\{\\min\}\.Fixν∈𝒩β\\nu\\in\\mathcal\{N\}\_\{\\beta\}, and define

qπ,ν​\(γ\):=ϕπ​\(ν,γ\)\.q\_\{\\pi,\\nu\}\(\\gamma\):=\\phi\_\{\\pi\}\(\\nu,\\gamma\)\.We claim that for any0≤a<b≤γmax0\\leq a<b\\leq\\gamma\_\{\\max\},

qπ,ν​\(a\)≤qπ,ν​\(b\)\+ρ⁡\(b−a\)\.q\_\{\\pi,\\nu\}\(a\)\\leq q\_\{\\pi,\\nu\}\(b\)\+\\rho\(b\-a\)\.\(31\)Indeed, letpb∈arg⁡minp∈Pδ⁡F⁡\(π,b,ν,p\)p\_\{b\}\\in\\arg\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,b,\\nu,p\)\. Then

qπ,ν​\(a\)\\displaystyle q\_\{\\pi,\\nu\}\(a\)≤F⁡\(π,a,ν,pb\)\\displaystyle\\leq F\(\\pi,a,\\nu,p\_\{b\}\)=F\(π,b,ν,pb\)\+\(a−b\)\(DKLBern\(pb∥p⋆\)−ρ\)\\displaystyle=F\(\\pi,b,\\nu,p\_\{b\}\)\+\(a\-b\)\\bigl\(D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{b\}\\\|p^\{\\star\}\)\-\\rho\\bigr\)=qπ,ν\(b\)\+\(b−a\)\(ρ−DKLBern\(pb∥p⋆\)\)\\displaystyle=q\_\{\\pi,\\nu\}\(b\)\+\(b\-a\)\\bigl\(\\rho\-D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{b\}\\\|p^\{\\star\}\)\\bigr\)≤qπ,ν​\(b\)\+ρ⁡\(b−a\),\\displaystyle\\leq q\_\{\\pi,\\nu\}\(b\)\+\\rho\(b\-a\),sinceDKLBern\(pb∥p⋆\)≥0D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{b\}\\\|p^\{\\star\}\)\\geq 0\. This proves \([31](https://arxiv.org/html/2610.00715#A2.E31)\)\.

Now letγ♯∈\[0,γmax\]\\gamma^\{\\sharp\}\\in\[0,\\gamma\_\{\\max\}\]be a maximizer ofqπ,νq\_\{\\pi,\\nu\}over\[0,γmax\]\[0,\\gamma\_\{\\max\}\]\. Ifγ♯∈Γ\\gamma^\{\\sharp\}\\in\\Gamma, then

maxγ∈\[0,γmax\]⁡qπ,ν​\(γ\)=maxγ∈Γ⁡qπ,ν​\(γ\)\.\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}q\_\{\\pi,\\nu\}\(\\gamma\)=\\max\_\{\\gamma\\in\\Gamma\}q\_\{\\pi,\\nu\}\(\\gamma\)\.Ifγ♯<γmin\\gamma^\{\\sharp\}<\\gamma\_\{\\min\}, then applying \([31](https://arxiv.org/html/2610.00715#A2.E31)\) witha=γ♯a=\\gamma^\{\\sharp\}andb=γminb=\\gamma\_\{\\min\}yields

qπ,ν​\(γ♯\)≤qπ,ν​\(γmin\)\+ρ⁡\(γmin−γ♯\)≤maxγ∈Γ⁡qπ,ν​\(γ\)\+ρ​γmin\.q\_\{\\pi,\\nu\}\(\\gamma^\{\\sharp\}\)\\leq q\_\{\\pi,\\nu\}\(\\gamma\_\{\\min\}\)\+\\rho\(\\gamma\_\{\\min\}\-\\gamma^\{\\sharp\}\)\\leq\\max\_\{\\gamma\\in\\Gamma\}q\_\{\\pi,\\nu\}\(\\gamma\)\+\\rho\\gamma\_\{\\min\}\.Hence, in all cases,

maxγ∈\[0,γmax\]⁡qπ,ν​\(γ\)≤maxγ∈Γ⁡qπ,ν​\(γ\)\+ρ​γmin\.\\max\_\{\\gamma\\in\[0,\\gamma\_\{\\max\}\]\}q\_\{\\pi,\\nu\}\(\\gamma\)\\leq\\max\_\{\\gamma\\in\\Gamma\}q\_\{\\pi,\\nu\}\(\\gamma\)\+\\rho\\gamma\_\{\\min\}\.Taking the minimum overν∈𝒩β\\nu\\in\\mathcal\{N\}\_\{\\beta\}, we obtain

gHδ​\(π,ρ\)≤minν∈𝒩β⁡maxγ∈Γ​ϕπ​\(ν,γ\)\+ρ​γmin\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\\leq\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\max\_\{\\gamma\\in\\Gamma\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)\+\\rho\\gamma\_\{\\min\}\.\(32\)
Step 4: under Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(4\), the minimax defect onΓ\\Gammavanishes\.Assume now Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\-\(4\), i\.e\.

τ​γmin≥14\.\\tau\\gamma\_\{\\min\}\\geq\\frac\{1\}\{4\}\.We show thatϕπ​\(⋅,⋅\)\\phi\_\{\\pi\}\(\\cdot,\\cdot\)is convex inν\\nuand concave inγ\\gammaon𝒩β×Γ\\mathcal\{N\}\_\{\\beta\}\\times\\Gamma\.

*Concavity inγ\\gamma\.*For fixedν\\nu, the function

γ↦F⁡\(π,γ,ν,p\)\\gamma\\mapsto F\(\\pi,\\gamma,\\nu,p\)is affine for every fixedpp, soϕπ​\(ν,⋅\)\\phi\_\{\\pi\}\(\\nu,\\cdot\), being the pointwise infimum of affine functions, is concave onΓ\\Gamma\.

*Convexity inν\\nu\.*Fixγ∈Γ\\gamma\\in\\Gamma\. We claim that\(ν,p\)↦F⁡\(π,γ,ν,p\)\(\\nu,p\)\\mapsto F\(\\pi,\\gamma,\\nu,p\)is jointly convex on𝒩β×Pδ\\mathcal\{N\}\_\{\\beta\}\\times P\_\{\\delta\}\. Write

W⁡\(π,ν,p\)=∑j=1mν⁡\(j\)​bj​\(π,p\),bj​\(π,p\)=∑i=1mπ⁡\(i\)​Pp​\(i≻j\)\.W\(\\pi,\\nu,p\)=\\sum\_\{j=1\}^\{m\}\\nu\(j\)\\,b\_\{j\}\(\\pi,p\),\\qquad b\_\{j\}\(\\pi,p\)=\\sum\_\{i=1\}^\{m\}\\pi\(i\)P\_\{p\}\(i\\succ j\)\.Let\(ν,p\)\(\\nu,p\)and\(ν′,p′\)\(\\nu^\{\\prime\},p^\{\\prime\}\)be two points in𝒩β×Pδ\\mathcal\{N\}\_\{\\beta\}\\times P\_\{\\delta\}\. SinceWWis bilinear in\(ν,p\)\(\\nu,p\),

W⁡\(π,ν′,p′\)−W⁡\(π,ν,p\)−⟨∇νW​\(π,ν,p\),ν′−ν⟩−⟨∇pW​\(π,ν,p\),p′−p⟩=⟨ν′−ν,b⁡\(π,p′\)−b⁡\(π,p\)⟩\.W\(\\pi,\\nu^\{\\prime\},p^\{\\prime\}\)\-W\(\\pi,\\nu,p\)\-\\langle\\nabla\_\{\\nu\}W\(\\pi,\\nu,p\),\\nu^\{\\prime\}\-\\nu\\rangle\-\\langle\\nabla\_\{p\}W\(\\pi,\\nu,p\),p^\{\\prime\}\-p\\rangle=\\langle\\nu^\{\\prime\}\-\\nu,\\;b\(\\pi,p^\{\\prime\}\)\-b\(\\pi,p\)\\rangle\.For each coordinatejj,

\|bj​\(π,p′\)−bj​\(π,p\)\|≤∑i=1mπ⁡\(i\)​\|Pp′​\(i≻j\)−Pp​\(i≻j\)\|≤‖p′−p‖2,\|b\_\{j\}\(\\pi,p^\{\\prime\}\)\-b\_\{j\}\(\\pi,p\)\|\\leq\\sum\_\{i=1\}^\{m\}\\pi\(i\)\\,\|P\_\{p^\{\\prime\}\}\(i\\succ j\)\-P\_\{p\}\(i\\succ j\)\|\\leq\\\|p^\{\\prime\}\-p\\\|\_\{2\},hence

\|⟨ν′−ν,b⁡\(π,p′\)−b⁡\(π,p\)⟩\|≤‖ν′−ν‖1​‖p′−p‖2\.\\bigl\|\\langle\\nu^\{\\prime\}\-\\nu,\\;b\(\\pi,p^\{\\prime\}\)\-b\(\\pi,p\)\\rangle\\bigr\|\\leq\\\|\\nu^\{\\prime\}\-\\nu\\\|\_\{1\}\\,\\\|p^\{\\prime\}\-p\\\|\_\{2\}\.Therefore,

F⁡\(π,γ,ν′,p′\)\\displaystyle F\(\\pi,\\gamma,\\nu^\{\\prime\},p^\{\\prime\}\)≥F⁡\(π,γ,ν,p\)\+⟨∇νF​\(π,γ,ν,p\),ν′−ν⟩\+⟨∇pF​\(π,γ,ν,p\),p′−p⟩\\displaystyle\\geq F\(\\pi,\\gamma,\\nu,p\)\+\\langle\\nabla\_\{\\nu\}F\(\\pi,\\gamma,\\nu,p\),\\nu^\{\\prime\}\-\\nu\\rangle\+\\langle\\nabla\_\{p\}F\(\\pi,\\gamma,\\nu,p\),p^\{\\prime\}\-p\\rangle\+τDKL\(ν′∥ν\)\+γDKLBern\(p′∥p\)−∥ν′−ν∥1∥p′−p∥2\.\\displaystyle\\quad\+\\tau D\_\{\\mathrm\{KL\}\}\(\\nu^\{\\prime\}\\\|\\nu\)\+\\gamma D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p^\{\\prime\}\\\|p\)\-\\\|\\nu^\{\\prime\}\-\\nu\\\|\_\{1\}\\,\\\|p^\{\\prime\}\-p\\\|\_\{2\}\.Using Pinsker’s inequalities

DKL\(ν′∥ν\)≥12∥ν′−ν∥12,DKLBern\(p′∥p\)≥2∥p′−p∥22,D\_\{\\mathrm\{KL\}\}\(\\nu^\{\\prime\}\\\|\\nu\)\\geq\\frac\{1\}\{2\}\\\|\\nu^\{\\prime\}\-\\nu\\\|\_\{1\}^\{2\},\\qquad D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p^\{\\prime\}\\\|p\)\\geq 2\\\|p^\{\\prime\}\-p\\\|\_\{2\}^\{2\},we obtain

τDKL\(ν′∥ν\)\+γDKLBern\(p′∥p\)−∥ν′−ν∥1∥p′−p∥2≥τ2s2\+2γt2−st,\\tau D\_\{\\mathrm\{KL\}\}\(\\nu^\{\\prime\}\\\|\\nu\)\+\\gamma D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p^\{\\prime\}\\\|p\)\-\\\|\\nu^\{\\prime\}\-\\nu\\\|\_\{1\}\\,\\\|p^\{\\prime\}\-p\\\|\_\{2\}\\geq\\frac\{\\tau\}\{2\}s^\{2\}\+2\\gamma t^\{2\}\-st,wheres=‖ν′−ν‖1s=\\\|\\nu^\{\\prime\}\-\\nu\\\|\_\{1\}andt=‖p′−p‖2t=\\\|p^\{\\prime\}\-p\\\|\_\{2\}\. Sinceγ≥γmin\\gamma\\geq\\gamma\_\{\\min\}andτ​γmin≥1/4\\tau\\gamma\_\{\\min\}\\geq 1/4,

τ2​s2\+2​γ​t2−s​t=\(τ2​s−2​γ​t\)2\+2​\(τ​γ−12\)​s​t≥0\.\\frac\{\\tau\}\{2\}s^\{2\}\+2\\gamma t^\{2\}\-st=\\left\(\\sqrt\{\\frac\{\\tau\}\{2\}\}\\,s\-\\sqrt\{2\\gamma\}\\,t\\right\)^\{2\}\+2\\left\(\\sqrt\{\\tau\\gamma\}\-\\frac\{1\}\{2\}\\right\)st\\geq 0\.Thus\(ν,p\)↦F⁡\(π,γ,ν,p\)\(\\nu,p\)\\mapsto F\(\\pi,\\gamma,\\nu,p\)is jointly convex\. Taking the infimum overp∈Pδp\\in P\_\{\\delta\}shows thatϕπ​\(⋅,γ\)\\phi\_\{\\pi\}\(\\cdot,\\gamma\)is convex on𝒩β\\mathcal\{N\}\_\{\\beta\}\.

Since𝒩β\\mathcal\{N\}\_\{\\beta\}andΓ\\Gammaare convex and compact, andϕπ\\phi\_\{\\pi\}is continuous, Sion’s minimax theorem applies:

minν∈𝒩β⁡maxγ∈Γ​ϕπ​\(ν,γ\)=maxγ∈Γ⁡minν∈𝒩β​ϕπ​\(ν,γ\)=gproxyδ​\(π,ρ\)\.\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\max\_\{\\gamma\\in\\Gamma\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\phi\_\{\\pi\}\(\\nu,\\gamma\)=g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\.Substituting this into \([32](https://arxiv.org/html/2610.00715#A2.E32)\) yields

gHδ​\(π,ρ\)≤gproxyδ​\(π,ρ\)\+ρ​γmin\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\\leq g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\+\\rho\\gamma\_\{\\min\}\.Combining with \([20](https://arxiv.org/html/2610.00715#S4.E20)\), we conclude

0≤gHδ​\(π,ρ\)−gproxyδ​\(π,ρ\)≤ρ​γmin,0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\leq\\rho\\gamma\_\{\\min\},which proves the first inequality in \([21](https://arxiv.org/html/2610.00715#S4.E21)\)\.

Step 5: pass from pointwise values to optimal values\.By definition,

Vδ​\(ρ\)=maxπ∈Πϵ⁡gHδ​\(π,ρ\),Vproxyδ​\(ρ\)=maxπ∈Πϵ⁡gproxyδ​\(π,ρ\)\.V^\{\\delta\}\(\\rho\)=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}^\{\\delta\}\(\\pi;\\rho\),\\qquad V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\.For any two functionsf,gf,gon the same domain,

maxπ⁡f⁡\(π\)−maxπ⁡g⁡\(π\)≤maxπ⁡\(f⁡\(π\)−g⁡\(π\)\)\.\\max\_\{\\pi\}f\(\\pi\)\-\\max\_\{\\pi\}g\(\\pi\)\\leq\\max\_\{\\pi\}\(f\(\\pi\)\-g\(\\pi\)\)\.Applying this withf⁡\(π\)=gHδ​\(π,ρ\)f\(\\pi\)=g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)andg⁡\(π\)=gproxyδ​\(π,ρ\)g\(\\pi\)=g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\), and using the pointwise bound just proved, we obtain

Vδ​\(ρ\)−Vproxyδ​\(ρ\)≤maxπ∈Πϵ⁡\(gHδ​\(π,ρ\)−gproxyδ​\(π,ρ\)\)≤ρ​γmin\.V^\{\\delta\}\(\\rho\)\-V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\leq\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}\\bigl\(g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\bigr\)\\leq\\rho\\gamma\_\{\\min\}\.This proves the second inequality in \([21](https://arxiv.org/html/2610.00715#S4.E21)\)\.

Step 6: exactness onΓ\\Gamma\.Assume now Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)and Assumption[3](https://arxiv.org/html/2610.00715#Thmassumption3)\. Since any maximizerγ⋆\\gamma^\{\\star\}lies inΓ\\Gamma, \([30](https://arxiv.org/html/2610.00715#A2.E30)\) then implies that

gHδ​\(π,ρ\)=maxγ∈Γ⁡minν∈𝒩β​minp∈Pδ⁡F⁡\(π,γ,ν,p\)=gproxyδ​\(π,ρ\)\.g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{\\nu\\in\\mathcal\{N\}\_\{\\beta\}\}\\min\_\{p\\in P\_\{\\delta\}\}F\(\\pi,\\gamma,\\nu,p\)=g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\.Finally, maximizing overπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\}gives

Vproxyδ​\(ρ\)=maxπ∈Πϵ⁡gproxyδ​\(π,ρ\)=maxπ∈Πϵ⁡gHδ​\(π,ρ\)=Vδ​\(ρ\),V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)=V\_\{\\delta\}\(\\rho\),which proves \([22](https://arxiv.org/html/2610.00715#S4.E22)\)\.

∎

### B\.4Proof of Theorem[3](https://arxiv.org/html/2610.00715#Thmtheorem3)

###### Theorem 4\(Convergence and robust guarantee of four\-player OGDA with explicitδ\\delta\-dependence\)\.

Assume Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2), and let

Lδ:=max⁡\{1,τϵ​πref,min,τβ​πref,min,γmaxδ⁡\(1−δ\),d,2​d​log⁡1−δδ\},L\_\{\\delta\}:=\\max\\left\\\{1,\\,\\frac\{\\tau\}\{\\epsilon\\,\\pi\_\{\\rm ref,min\}\},\\,\\frac\{\\tau\}\{\\beta\\,\\pi\_\{\\rm ref,min\}\},\\,\\frac\{\\gamma\_\{\\max\}\}\{\\delta\(1\-\\delta\)\},\\,\\sqrt\{d\},\\,2\\sqrt\{d\}\\,\\log\\frac\{1\-\\delta\}\{\\delta\}\\right\\\},where

πref,min:=min1≤i≤m⁡πref,i\.\\pi\_\{\\rm ref,min\}:=\\min\_\{1\\leq i\\leq m\}\\pi\_\{\{\\rm ref\},i\}\.In particular,

Lδ=O⁡\(max⁡\{1δ⁡\(1−δ\),log⁡1−δδ\}\)as​δ↓0\.L\_\{\\delta\}=O\\\!\\left\(\\max\\left\\\{\\frac\{1\}\{\\delta\(1\-\\delta\)\},\\ \\log\\frac\{1\-\\delta\}\{\\delta\}\\right\\\}\\right\)\\qquad\\text\{as \}\\delta\\downarrow 0\.
Run Algorithm 1 with the constant step size

η=18​Lδ\.\\eta=\\frac\{1\}\{8L\_\{\\delta\}\}\.Define

DXdiam:=sup\(π,γ\)∈X\{DKL\(π∥π1\)\+12\(γ−γ1\)2\},DYdiam:=sup\(ν,p\)∈Y\{DKL\(ν∥ν1\)\+DKLBern\(p∥p1\)\},D\_\{X\}^\{\\mathrm\{diam\}\}:=\\sup\_\{\(\\pi,\\gamma\)\\in X\}\\left\\\{D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{1\}\)\+\\frac\{1\}\{2\}\(\\gamma\-\\gamma\_\{1\}\)^\{2\}\\right\\\},\\qquad D\_\{Y\}^\{\\mathrm\{diam\}\}:=\\sup\_\{\(\\nu,p\)\\in Y\}\\left\\\{D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu\_\{1\}\)\+D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p\_\{1\}\)\\right\\\},and

BT,δ:=8​Lδ​\(DXdiam\+DYdiam\)T\.B\_\{T,\\delta\}:=\\frac\{8L\_\{\\delta\}\\bigl\(D\_\{X\}^\{\\mathrm\{diam\}\}\+D\_\{Y\}^\{\\mathrm\{diam\}\}\\bigr\)\}\{T\}\.
Then the duality gap of the averaged iterate

z¯T=\(x¯T,y¯T\)=\(\(π¯T,γ¯T\),\(ν¯T,p¯T\)\)\\bar\{z\}\_\{T\}=\(\\bar\{x\}\_\{T\},\\bar\{y\}\_\{T\}\)=\\bigl\(\(\\bar\{\\pi\}\_\{T\},\\bar\{\\gamma\}\_\{T\}\),\(\\bar\{\\nu\}\_\{T\},\\bar\{p\}\_\{T\}\)\\bigr\)satisfies

Gap⁡\(z¯T\):=maxx′∈X⁡F⁡\(x′,y¯T\)−miny′∈Y⁡F⁡\(x¯T,y′\)≤BT,δ\.\\Gap\(\\bar\{z\}\_\{T\}\):=\\max\_\{x^\{\\prime\}\\in X\}F\(x^\{\\prime\},\\bar\{y\}\_\{T\}\)\-\\min\_\{y^\{\\prime\}\\in Y\}F\(\\bar\{x\}\_\{T\},y^\{\\prime\}\)\\leq B\_\{T,\\delta\}\.\(28\)Moreover, the averaged main policy satisfies

V⁡\(ρ\)−gH​\(π¯T,ρ\)≤BT,δ\+ρ​γmin\+δ\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq B\_\{T,\\delta\}\+\\rho\\gamma\_\{\\min\}\+\\delta\.\(29\)Equivalently,

V⁡\(ρ\)−gH​\(π¯T,ρ\)=O⁡\(LδT\+ρ​γmin\+δ\)\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)=O\\\!\\left\(\\frac\{L\_\{\\delta\}\}\{T\}\+\\rho\\gamma\_\{\\min\}\+\\delta\\right\)\.
Additionally, under Assumption[3](https://arxiv.org/html/2610.00715#Thmassumption3),

V⁡\(ρ\)−gH​\(π¯T,ρ\)≤BT,δ\+δ,V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq B\_\{T,\\delta\}\+\\delta,\(30\)soπ¯T\\bar\{\\pi\}\_\{T\}is an

O⁡\(LδT\+δ\)O\\\!\\left\(\\frac\{L\_\{\\delta\}\}\{T\}\+\\delta\\right\)\-optimal policy for the original robust game\.

###### Proof\.

We divide the proof into four steps\.

Step 1: proxy\-game convergence with explicitLδL\_\{\\delta\}\.Let

xt:=\(πt,γt\)∈X,yt:=\(νt,pt\)∈Y,zt=\(xt,yt\),x\_\{t\}:=\(\\pi\_\{t\},\\gamma\_\{t\}\)\\in X,\\qquad y\_\{t\}:=\(\\nu\_\{t\},p\_\{t\}\)\\in Y,\\qquad z\_\{t\}=\(x\_\{t\},y\_\{t\}\),and define the averaged iterates

x¯T:=1T​∑t=1Txt,y¯T:=1T​∑t=1Tyt\.\\bar\{x\}\_\{T\}:=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}x\_\{t\},\\qquad\\bar\{y\}\_\{T\}:=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}y\_\{t\}\.
For anyx∈Xx\\in X, define

AT​\(x\):=∑t=1T\(F⁡\(x,yt\)−F⁡\(xt,yt\)\),A\_\{T\}\(x\):=\\sum\_\{t=1\}^\{T\}\\bigl\(F\(x,y\_\{t\}\)\-F\(x\_\{t\},y\_\{t\}\)\\bigr\),and for anyy∈Yy\\in Y, define

BT​\(y\):=∑t=1T\(F⁡\(xt,yt\)−F⁡\(xt,y\)\)\.B\_\{T\}\(y\):=\\sum\_\{t=1\}^\{T\}\\bigl\(F\(x\_\{t\},y\_\{t\}\)\-F\(x\_\{t\},y\)\\bigr\)\.
By the blockwise optimistic mirror\-descent inequalities and the convexity–concavity structure of Lemma[3](https://arxiv.org/html/2610.00715#Thmlemma3), for everyx∈Xx\\in Xand everyy∈Yy\\in Y,

AT​\(x\)\+BT​\(y\)≤DX​\(x,x1\)\+DY​\(y,y1\)η\+VT−ΞT,A\_\{T\}\(x\)\+B\_\{T\}\(y\)\\leq\\frac\{D\_\{X\}\(x,x\_\{1\}\)\+D\_\{Y\}\(y,y\_\{1\}\)\}\{\\eta\}\+V\_\{T\}\-\\Xi\_\{T\},\(33\)where

DX\(x,x1\):=DKL\(π∥π1\)\+12\(γ−γ1\)2,DY\(y,y1\):=DKL\(ν∥ν1\)\+DKLBern\(p∥p1\),D\_\{X\}\(x,x\_\{1\}\):=D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{1\}\)\+\\frac\{1\}\{2\}\(\\gamma\-\\gamma\_\{1\}\)^\{2\},\\qquad D\_\{Y\}\(y,y\_\{1\}\):=D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu\_\{1\}\)\+D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p\_\{1\}\),and

VT:=η​∑t=1T\(‖gtπ−gt−1π‖∞2\+\|gtγ−gt−1γ\|2\+‖gtν−gt−1ν‖∞2\+‖gtp−gt−1p‖22\),V\_\{T\}:=\\eta\\sum\_\{t=1\}^\{T\}\\Bigl\(\\\|g\_\{t\}^\{\\pi\}\-g\_\{t\-1\}^\{\\pi\}\\\|\_\{\\infty\}^\{2\}\+\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|^\{2\}\+\\\|g\_\{t\}^\{\\nu\}\-g\_\{t\-1\}^\{\\nu\}\\\|\_\{\\infty\}^\{2\}\+\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}^\{2\}\\Bigr\),ΞT:=12​η∑t=1T\(DKL\(πt∥πt\+1\)\+\|γt−γt\+1\|2\+DKL\(νt∥νt\+1\)\+DKLBern\(pt∥pt\+1\)\)\.\\Xi\_\{T\}:=\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}\\Bigl\(D\_\{\\mathrm\{KL\}\}\(\\pi\_\{t\}\\\|\\pi\_\{t\+1\}\)\+\|\\gamma\_\{t\}\-\\gamma\_\{t\+1\}\|^\{2\}\+D\_\{\\mathrm\{KL\}\}\(\\nu\_\{t\}\\\|\\nu\_\{t\+1\}\)\+D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{t\}\\\|p\_\{t\+1\}\)\\Bigr\)\.
Now, by Lemma[5](https://arxiv.org/html/2610.00715#Thmlemma5), all gradient\-variation terms are bounded using the constantLδL\_\{\\delta\}; hence Lemma[6](https://arxiv.org/html/2610.00715#Thmlemma6)applies withL=LδL=L\_\{\\delta\}\. Sinceη=1/\(8​Lδ\)\\eta=1/\(8L\_\{\\delta\}\), Lemma[6](https://arxiv.org/html/2610.00715#Thmlemma6)yields

Therefore \([33](https://arxiv.org/html/2610.00715#A2.E33)\) simplifies to

AT​\(x\)\+BT​\(y\)≤DX​\(x,x1\)\+DY​\(y,y1\)η,∀x∈X,∀y∈Y\.A\_\{T\}\(x\)\+B\_\{T\}\(y\)\\leq\\frac\{D\_\{X\}\(x,x\_\{1\}\)\+D\_\{Y\}\(y,y\_\{1\}\)\}\{\\eta\},\\qquad\\forall x\\in X,\\ \\forall y\\in Y\.\(34\)
Now choose

x^∈arg⁡maxx∈X⁡F⁡\(x,y¯T\),y^∈arg⁡miny∈Y⁡F⁡\(x¯T,y\)\.\\hat\{x\}\\in\\arg\\max\_\{x\\in X\}F\(x,\\bar\{y\}\_\{T\}\),\\qquad\\hat\{y\}\\in\\arg\\min\_\{y\\in Y\}F\(\\bar\{x\}\_\{T\},y\)\.Then

Gap⁡\(z¯T\)=F⁡\(x^,y¯T\)−F⁡\(x¯T,y^\)\.\\Gap\(\\bar\{z\}\_\{T\}\)=F\(\\hat\{x\},\\bar\{y\}\_\{T\}\)\-F\(\\bar\{x\}\_\{T\},\\hat\{y\}\)\.BecauseF⁡\(⋅,y\)F\(\\cdot,y\)is concave andF⁡\(x,⋅\)F\(x,\\cdot\)is convex by Lemma[3](https://arxiv.org/html/2610.00715#Thmlemma3), Jensen’s inequality gives

F⁡\(x^,y¯T\)≤1T​∑t=1TF⁡\(x^,yt\),F⁡\(x¯T,y^\)≥1T​∑t=1TF⁡\(xt,y^\)\.F\(\\hat\{x\},\\bar\{y\}\_\{T\}\)\\leq\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}F\(\\hat\{x\},y\_\{t\}\),\\qquad F\(\\bar\{x\}\_\{T\},\\hat\{y\}\)\\geq\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}F\(x\_\{t\},\\hat\{y\}\)\.Hence

T​Gap⁡\(z¯T\)\\displaystyle T\\,\\Gap\(\\bar\{z\}\_\{T\}\)≤∑t=1T\(F⁡\(x^,yt\)−F⁡\(xt,y^\)\)\\displaystyle\\leq\\sum\_\{t=1\}^\{T\}\\bigl\(F\(\\hat\{x\},y\_\{t\}\)\-F\(x\_\{t\},\\hat\{y\}\)\\bigr\)=∑t=1T\(F⁡\(x^,yt\)−F⁡\(xt,yt\)\)\+∑t=1T\(F⁡\(xt,yt\)−F⁡\(xt,y^\)\)\\displaystyle=\\sum\_\{t=1\}^\{T\}\\bigl\(F\(\\hat\{x\},y\_\{t\}\)\-F\(x\_\{t\},y\_\{t\}\)\\bigr\)\+\\sum\_\{t=1\}^\{T\}\\bigl\(F\(x\_\{t\},y\_\{t\}\)\-F\(x\_\{t\},\\hat\{y\}\)\\bigr\)=AT​\(x^\)\+BT​\(y^\)\.\\displaystyle=A\_\{T\}\(\\hat\{x\}\)\+B\_\{T\}\(\\hat\{y\}\)\.Applying \([34](https://arxiv.org/html/2610.00715#A2.E34)\) withx=x^x=\\hat\{x\}andy=y^y=\\hat\{y\}, and then using the definitions ofDXdiamD\_\{X\}^\{\\mathrm\{diam\}\}andDYdiamD\_\{Y\}^\{\\mathrm\{diam\}\}, gives

T​Gap⁡\(z¯T\)≤DX​\(x^,x1\)\+DY​\(y^,y1\)η≤DXdiam\+DYdiamη\.T\\,\\Gap\(\\bar\{z\}\_\{T\}\)\\leq\\frac\{D\_\{X\}\(\\hat\{x\},x\_\{1\}\)\+D\_\{Y\}\(\\hat\{y\},y\_\{1\}\)\}\{\\eta\}\\leq\\frac\{D\_\{X\}^\{\\mathrm\{diam\}\}\+D\_\{Y\}^\{\\mathrm\{diam\}\}\}\{\\eta\}\.Substitutingη=1/\(8​Lδ\)\\eta=1/\(8L\_\{\\delta\}\)yields

Gap⁡\(z¯T\)≤8​Lδ​\(DXdiam\+DYdiam\)T=BT,δ,\\Gap\(\\bar\{z\}\_\{T\}\)\\leq\\frac\{8L\_\{\\delta\}\\bigl\(D\_\{X\}^\{\\mathrm\{diam\}\}\+D\_\{Y\}^\{\\mathrm\{diam\}\}\\bigr\)\}\{T\}=B\_\{T,\\delta\},which proves\(28\)\(28\)\.

Step 2: proxy\-policy guarantee\.By definition of the duality gap,

Gap⁡\(z¯T\)=maxx′∈X⁡F⁡\(x′,y¯T\)−miny′∈Y⁡F⁡\(x¯T,y′\)\.\\Gap\(\\bar\{z\}\_\{T\}\)=\\max\_\{x^\{\\prime\}\\in X\}F\(x^\{\\prime\},\\bar\{y\}\_\{T\}\)\-\\min\_\{y^\{\\prime\}\\in Y\}F\(\\bar\{x\}\_\{T\},y^\{\\prime\}\)\.Sincemaxx′⁡F⁡\(x′,y¯T\)≥F⁡\(x¯T,y¯T\)=F⁡\(z¯T\)\\max\_\{x^\{\\prime\}\}F\(x^\{\\prime\},\\bar\{y\}\_\{T\}\)\\geq F\(\\bar\{x\}\_\{T\},\\bar\{y\}\_\{T\}\)=F\(\\bar\{z\}\_\{T\}\), we obtain

Gap⁡\(z¯T\)≥F⁡\(z¯T\)−miny′∈Y⁡F⁡\(x¯T,y′\)\.\\Gap\(\\bar\{z\}\_\{T\}\)\\geq F\(\\bar\{z\}\_\{T\}\)\-\\min\_\{y^\{\\prime\}\\in Y\}F\(\\bar\{x\}\_\{T\},y^\{\\prime\}\)\.Rearranging,

miny′∈Y⁡F⁡\(x¯T,y′\)≥F⁡\(z¯T\)−Gap⁡\(z¯T\)\.\\min\_\{y^\{\\prime\}\\in Y\}F\(\\bar\{x\}\_\{T\},y^\{\\prime\}\)\\geq F\(\\bar\{z\}\_\{T\}\)\-\\Gap\(\\bar\{z\}\_\{T\}\)\.\(35\)
Now

gproxyδ​\(π¯T,ρ\)=maxγ∈Γ⁡miny∈Y⁡F⁡\(\(π¯T,γ\),y\)\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)=\\max\_\{\\gamma\\in\\Gamma\}\\min\_\{y\\in Y\}F\(\(\\bar\{\\pi\}\_\{T\},\\gamma\),y\)\.Sincex¯T=\(π¯T,γ¯T\)∈X\\bar\{x\}\_\{T\}=\(\\bar\{\\pi\}\_\{T\},\\bar\{\\gamma\}\_\{T\}\)\\in X,

gproxyδ​\(π¯T,ρ\)≥miny∈Y⁡F⁡\(x¯T,y\)\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq\\min\_\{y\\in Y\}F\(\\bar\{x\}\_\{T\},y\)\.Combining with \([35](https://arxiv.org/html/2610.00715#A2.E35)\) and then using\(28\)\(28\),

gproxyδ​\(π¯T,ρ\)≥F⁡\(z¯T\)−Gap⁡\(z¯T\)≥F⁡\(z¯T\)−BT,δ\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq F\(\\bar\{z\}\_\{T\}\)\-\\Gap\(\\bar\{z\}\_\{T\}\)\\geq F\(\\bar\{z\}\_\{T\}\)\-B\_\{T,\\delta\}\.\(36\)
Step 3: transfer from the proxy game to the original hard objective\.By Theorem[2](https://arxiv.org/html/2610.00715#Thmtheorem2), under Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)we have the pointwise bounds

0≤gHδ​\(π,ρ\)−gproxyδ​\(π,ρ\)≤ρ​γmin,0≤Vδ​\(ρ\)−Vproxyδ​\(ρ\)≤ρ​γmin,∀π∈Πϵ\.0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)\\leq\\rho\\gamma\_\{\\min\},\\qquad 0\\leq V^\{\\delta\}\(\\rho\)\-V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\leq\\rho\\gamma\_\{\\min\},\\qquad\\forall\\pi\\in\\Pi\_\{\\epsilon\}\.By Theorem 1, the truncation error satisfies

0≤gHδ​\(π,ρ\)−gH​\(π,ρ\)≤δ,0≤Vδ​\(ρ\)−V⁡\(ρ\)≤δ,∀π∈Πϵ\.0\\leq g\_\{H\}^\{\\delta\}\(\\pi;\\rho\)\-g\_\{H\}\(\\pi;\\rho\)\\leq\\delta,\\qquad 0\\leq V^\{\\delta\}\(\\rho\)\-V\(\\rho\)\\leq\\delta,\\qquad\\forall\\pi\\in\\Pi\_\{\\epsilon\}\.
From \([36](https://arxiv.org/html/2610.00715#A2.E36)\),

gproxyδ​\(π¯T,ρ\)≥F⁡\(z¯T\)−BT,δ\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq F\(\\bar\{z\}\_\{T\}\)\-B\_\{T,\\delta\}\.Hence

gH​\(π¯T,ρ\)≥gproxyδ​\(π¯T,ρ\)−δ≥F⁡\(z¯T\)−BT,δ−δ\.g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\-\\delta\\geq F\(\\bar\{z\}\_\{T\}\)\-B\_\{T,\\delta\}\-\\delta\.This gives the explicit robust\-value certificate\.

Next, usingV⁡\(ρ\)≤Vδ​\(ρ\)V\(\\rho\)\\leq V^\{\\delta\}\(\\rho\)andgH​\(π¯T,ρ\)≥gproxyδ​\(π¯T,ρ\)−δg\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\-\\delta, we get

V⁡\(ρ\)−gH​\(π¯T,ρ\)\\displaystyle V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)≤Vδ​\(ρ\)−gproxyδ​\(π¯T,ρ\)\+δ\\displaystyle\\leq V^\{\\delta\}\(\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\+\\delta=\(Vδ​\(ρ\)−Vproxyδ​\(ρ\)\)\+\(Vproxyδ​\(ρ\)−gproxyδ​\(π¯T,ρ\)\)\+δ\.\\displaystyle=\\bigl\(V^\{\\delta\}\(\\rho\)\-V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\bigr\)\+\\bigl\(V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\bigr\)\+\\delta\.By Theorem[2](https://arxiv.org/html/2610.00715#Thmtheorem2),

Vδ​\(ρ\)−Vproxyδ​\(ρ\)≤ρ​γmin\.V^\{\\delta\}\(\\rho\)\-V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\leq\\rho\\gamma\_\{\\min\}\.Also,

Vproxyδ​\(ρ\)=maxπ∈Πϵ⁡gproxyδ​\(π,ρ\)=maxx∈X⁡miny∈Y⁡F⁡\(x,y\),V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)=\\max\_\{\\pi\\in\\Pi\_\{\\epsilon\}\}g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)=\\max\_\{x\\in X\}\\min\_\{y\\in Y\}F\(x,y\),so

Vproxyδ​\(ρ\)≤maxx∈X⁡F⁡\(x,y¯T\)\.V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\\leq\\max\_\{x\\in X\}F\(x,\\bar\{y\}\_\{T\}\)\.And from the definition ofgproxyδg\_\{\\mathrm\{proxy\}\}^\{\\delta\},

gproxyδ​\(π¯T,ρ\)≥miny∈Y⁡F⁡\(x¯T,y\)\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\geq\\min\_\{y\\in Y\}F\(\\bar\{x\}\_\{T\},y\)\.Subtracting the last two displays,

Vproxyδ​\(ρ\)−gproxyδ​\(π¯T,ρ\)≤maxx∈X⁡F⁡\(x,y¯T\)−miny∈Y⁡F⁡\(x¯T,y\)=Gap⁡\(z¯T\)≤BT,δ\.V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq\\max\_\{x\\in X\}F\(x,\\bar\{y\}\_\{T\}\)\-\\min\_\{y\\in Y\}F\(\\bar\{x\}\_\{T\},y\)=\\Gap\(\\bar\{z\}\_\{T\}\)\\leq B\_\{T,\\delta\}\.Combining the last three inequalities proves

V⁡\(ρ\)−gH​\(π¯T,ρ\)≤BT,δ\+ρ​γmin\+δ\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq B\_\{T,\\delta\}\+\\rho\\gamma\_\{\\min\}\+\\delta\.SinceBT,δ=𝒪⁡\(Lδ/T\)B\_\{T,\\delta\}=\\mathcal\{O\}\(L\_\{\\delta\}/T\), this also yields

V⁡\(ρ\)−gH​\(π¯T,ρ\)=O⁡\(LδT\+ρ​γmin\+δ\)\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)=O\\\!\\left\(\\frac\{L\_\{\\delta\}\}\{T\}\+\\rho\\gamma\_\{\\min\}\+\\delta\\right\)\.
Step 4: sharper guarantee under Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)\.Assume now Assumption[2](https://arxiv.org/html/2610.00715#Thmassumption2)in addition\. Then by Theorem[2](https://arxiv.org/html/2610.00715#Thmtheorem2),

gproxyδ​\(π,ρ\)=gHδ​\(π,ρ\),Vproxyδ​\(ρ\)=Vδ​\(ρ\),∀π∈Πϵ\.g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\pi;\\rho\)=g\_\{H\}^\{\\delta\}\(\\pi;\\rho\),\\qquad V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)=V^\{\\delta\}\(\\rho\),\\qquad\\forall\\pi\\in\\Pi\_\{\\epsilon\}\.Therefore,

V⁡\(ρ\)−gH​\(π¯T,ρ\)≤Vδ​\(ρ\)−gH​\(π¯T,ρ\)≤Vδ​\(ρ\)−gHδ​\(π¯T,ρ\)\+δ\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq V^\{\\delta\}\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq V^\{\\delta\}\(\\rho\)\-g\_\{H\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\+\\delta\.Using exact recovery and the proxy optimality bound already established,

Vδ​\(ρ\)−gHδ​\(π¯T,ρ\)=Vproxyδ​\(ρ\)−gproxyδ​\(π¯T,ρ\)≤BT,δ\.V^\{\\delta\}\(\\rho\)\-g\_\{H\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)=V\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\rho\)\-g\_\{\\mathrm\{proxy\}\}^\{\\delta\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq B\_\{T,\\delta\}\.Hence

V⁡\(ρ\)−gH​\(π¯T,ρ\)≤BT,δ\+δ\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)\\leq B\_\{T,\\delta\}\+\\delta\.SinceBT,δ=𝒪⁡\(Lδ/T\)B\_\{T,\\delta\}=\\mathcal\{O\}\(L\_\{\\delta\}/T\), this gives the sharper rate

V⁡\(ρ\)−gH​\(π¯T,ρ\)=O⁡\(LδT\+δ\)\.V\(\\rho\)\-g\_\{H\}\(\\bar\{\\pi\}\_\{T\};\\rho\)=O\\\!\\left\(\\frac\{L\_\{\\delta\}\}\{T\}\+\\delta\\right\)\.∎

## Appendix CComparisons with Standard NLHF

Compared with standard NLHF\[[29](https://arxiv.org/html/2610.00715#bib.bib29)\]and its following studies\[[54](https://arxiv.org/html/2610.00715#bib.bib54),[46](https://arxiv.org/html/2610.00715#bib.bib46),[30](https://arxiv.org/html/2610.00715#bib.bib30),[56](https://arxiv.org/html/2610.00715#bib.bib56),[52](https://arxiv.org/html/2610.00715#bib.bib52)\], the convergence analysis in our setting is substantially more involved\. In nominal NLHF, the preference kernel is fixed, and the optimization problem is a regularized two\-player zero\-sum game between the main policy and the adversarial policy\. Once the game is written in saddle\-point form, the KL regularizers provide strong concavity in the leader policy and strong convexity in the follower policy, so one can analyze mirror descent or optimistic no\-regret dynamics directly on a fixed convex–concave game\. By contrast, our robust formulation introduces uncertainty in the preference kernel itself\. The resulting objective is no longer a standard two\-player nominal game, but a distributionally robust max\-min problem in which the adversary chooses both a competing policy and a plausible preference kernel\. This creates two new difficulties that do not appear in standard NLHF\.

First, the original hard\-constrained problem does not retain the clean convex\-concave structure of nominal NLHF\. If one eliminates the follower using the Gibbs best response, the reduced objective becomes a concave minimization problem over the kernel ambiguity set, which is globally hard in general\. On the other hand, if one keeps both the adversarial policy and the preference kernel as optimization variables, the inner minimization is jointly nonconvex because the win\-rate term is bilinear in the pair\(ν,p\)\(\\nu,p\)\. Therefore, unlike in standard NLHF, one cannot analyze the original hard\-constrained robust objective directly using a standard minimax or saddle\-point argument\.

A further difficulty is that the kernel block acquires its curvature through the dual variableγ\\gamma\. In the nominal two\-player game, the KL regularizers already provide fixed curvature in the policy blocks\. In the four\-player robust game, by contrast, thepp\-block is strongly convex only with modulus proportional toγ\\gamma\. This is why the lower boundγmin\>0\\gamma\_\{\\min\}\>0and the conditionτ​γmin≥1/4\\tau\\gamma\_\{\\min\}\\geq 1/4enter the analysis: they ensure that the KL curvature in the kernel block is strong enough to dominate the mixed bilinear interaction between the follower policy and the adversarial kernel\. Without such a condition, the grouped\(π,γ\)\(\\pi,\\gamma\)\-vs\.\-\(ν,p\)\(\\nu,p\)game need not be convex\-concave, and the standard optimistic mirror\-descent analysis does not go through\.

Our gradient\-variation analysis is also substantially more coupled than in NLHF\. In nominal NLHF, once the preference kernel is fixed, the gradient of each player depends only on the other policy\. In our setting, the gradients evolve through a chain of dependencies:γ→p→\(π,ν\),\\gamma\\rightarrow p\\rightarrow\(\\pi,\\nu\),so the variation of one block can depend on the movement of several other blocks\. In particular, thepp\-gradient depends on the current policies and on the dual variable, while theγ\\gamma\-gradient is exactly the KL\-budget slack of the current kernel\. Establishing convergence therefore requires a sharper self\-bounding argument that simultaneously controls all four regret terms and shows that the total gradient variation can be absorbed by the negative Bregman stability terms\. This absorption step has no analogue in the standard NLHF proof\.

## Appendix DAuxiliary Lemmas

We prove several auxiliary lemmas required for the theorems\.

###### Lemma 1\(Universal upper bound on the dual multiplier\)\.

Fix\(π,ν\)∈Πϵ×Δm\(\\pi,\\nu\)\\in\\Pi\_\{\\epsilon\}\\times\\Delta\_\{m\}, and define

qπ,ν\(γ\):=minp∈Pδ\{J\(π,ν;p\)\+γ\(DKLBern\(p∥p⋆\)−ρ\)\},γ≥0\.q\_\{\\pi,\\nu\}\(\\gamma\):=\\min\_\{p\\in P\_\{\\delta\}\}\\left\\\{J\(\\pi,\\nu;p\)\+\\gamma\\bigl\(D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho\\bigr\)\\right\\\},\\qquad\\gamma\\geq 0\.Then every maximizer ofqπ,νq\_\{\\pi,\\nu\}over\[0,∞\)\[0,\\infty\)lies in\[0,1/ρ\]\[0,1/\\rho\]\. In particular, one may choose the algorithmic upper boundγmax\\gamma\_\{\\max\}so thatγmax≥1/ρ\\gamma\_\{\\max\}\\geq 1/\\rho\.

###### Proof\.

Becausep⋆∈Pδp^\{\\star\}\\in P\_\{\\delta\}, for everyγ≥0\\gamma\\geq 0,

qπ,ν​\(γ\)≤J⁡\(π,ν,p⋆\)−ρ​γ\.q\_\{\\pi,\\nu\}\(\\gamma\)\\leq J\(\\pi,\\nu;p^\{\\star\}\)\-\\rho\\gamma\.Atγ=0\\gamma=0,

qπ,ν​\(0\)=minp∈Pδ⁡J⁡\(π,ν,p\)\.q\_\{\\pi,\\nu\}\(0\)=\\min\_\{p\\in P\_\{\\delta\}\}J\(\\pi,\\nu;p\)\.For fixed\(π,ν\)\(\\pi,\\nu\), the only part ofJ⁡\(π,ν,p\)J\(\\pi,\\nu;p\)that depends onppis the win\-rate term, which always lies in\[0,1\]\[0,1\]\. Hence

J⁡\(π,ν,p⋆\)−qπ,ν​\(0\)≤1\.J\(\\pi,\\nu;p^\{\\star\}\)\-q\_\{\\pi,\\nu\}\(0\)\\leq 1\.Therefore, ifγ\>1/ρ\\gamma\>1/\\rho, then

qπ,ν​\(γ\)≤J⁡\(π,ν,p⋆\)−ρ​γ<J⁡\(π,ν,p⋆\)−1≤qπ,ν​\(0\),q\_\{\\pi,\\nu\}\(\\gamma\)\\leq J\(\\pi,\\nu;p^\{\\star\}\)\-\\rho\\gamma<J\(\\pi,\\nu;p^\{\\star\}\)\-1\\leq q\_\{\\pi,\\nu\}\(0\),so such aγ\\gammacannot be optimal\. ∎

###### Lemma 2\(Blockwise OMD inequalities\)\.

For any comparatorsπ∈Πϵ\\pi\\in\\Pi\_\{\\epsilon\},γ∈Γ\\gamma\\in\\Gamma,ν∈𝒩β\\nu\\in\\mathcal\{N\}\_\{\\beta\}, andp∈Pδp\\in P\_\{\\delta\}, the iterates produced by Algorithm[1](https://arxiv.org/html/2610.00715#alg1)satisfy

∑t=1T⟨gtπ,π−πt⟩\\displaystyle\\sum\_\{t=1\}^\{T\}\\langle g\_\{t\}^\{\\pi\},\\pi\-\\pi\_\{t\}\\rangle≤DKL\(π∥π1\)η\+η∑t=1T∥gtπ−gt−1π∥∞2−12​η∑t=1TDKL\(πt∥πt\+1\),\\displaystyle\\leq\\frac\{D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{1\}\)\}\{\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\\\|g\_\{t\}^\{\\pi\}\-g\_\{t\-1\}^\{\\pi\}\\\|\_\{\\infty\}^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{t\}\\\|\\pi\_\{t\+1\}\),\(37\)∑t=1Tgtγ​\(γ−γt\)\\displaystyle\\sum\_\{t=1\}^\{T\}g\_\{t\}^\{\\gamma\}\(\\gamma\-\\gamma\_\{t\}\)≤\(γ−γ1\)22​η\+η​∑t=1T\|gtγ−gt−1γ\|2−12​η​∑t=1T\|γt−γt\+1\|2,\\displaystyle\\leq\\frac\{\(\\gamma\-\\gamma\_\{1\}\)^\{2\}\}\{2\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}\|\\gamma\_\{t\}\-\\gamma\_\{t\+1\}\|^\{2\},\(38\)∑t=1T⟨gtν,νt−ν⟩\\displaystyle\\sum\_\{t=1\}^\{T\}\\langle g\_\{t\}^\{\\nu\},\\nu\_\{t\}\-\\nu\\rangle≤DKL\(ν∥ν1\)η\+η∑t=1T∥gtν−gt−1ν∥∞2−12​η∑t=1TDKL\(νt∥νt\+1\),\\displaystyle\\leq\\frac\{D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\nu\_\{1\}\)\}\{\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\\\|g\_\{t\}^\{\\nu\}\-g\_\{t\-1\}^\{\\nu\}\\\|\_\{\\infty\}^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}D\_\{\\mathrm\{KL\}\}\(\\nu\_\{t\}\\\|\\nu\_\{t\+1\}\),\(39\)∑t=1T⟨gtp,pt−p⟩\\displaystyle\\sum\_\{t=1\}^\{T\}\\langle g\_\{t\}^\{p\},p\_\{t\}\-p\\rangle≤DKLBern\(p∥p1\)η\+η∑t=1T∥gtp−gt−1p∥22−12​η∑t=1TDKLBern\(pt∥pt\+1\)\.\\displaystyle\\leq\\frac\{D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p\_\{1\}\)\}\{\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{t\}\\\|p\_\{t\+1\}\)\.\(40\)

###### Proof\.

Apply Lemma[5](https://arxiv.org/html/2610.00715#Thmlemma5)blockwise:

- •to theπ\\pi\-block with entropy geometry and maximization;
- •to theγ\\gamma\-block with Euclidean geometry and maximization;
- •to theν\\nu\-block with entropy geometry and minimization;
- •to thepp\-block with binary\-entropy geometry and minimization\.

The dual norms areℓ∞\\ell\_\{\\infty\}forπ\\piandν\\nu, the Euclidean norm forγ\\gamma, andℓ2\\ell\_\{2\}forpp\. ∎

The constant\-step OGDA proof requires a uniform bound on blockwise gradient variation\.

###### Lemma 3\(Gradient\-variation bounds\)\.

Let

πref,min:=min1≤i≤m⁡πref,i,Lπ:=τϵ​πref,min,Lν:=τβ​πref,min,Lp:=γmaxδ⁡\(1−δ\),\\pi\_\{\\mathrm\{ref\},\\min\}:=\\min\_\{1\\leq i\\leq m\}\\pi\_\{\\mathrm\{ref\},i\},\\qquad L\_\{\\pi\}:=\\frac\{\\tau\}\{\\epsilon\\pi\_\{\\mathrm\{ref\},\\min\}\},\\qquad L\_\{\\nu\}:=\\frac\{\\tau\}\{\\beta\\pi\_\{\\mathrm\{ref\},\\min\}\},\\qquad L\_\{p\}:=\\frac\{\\gamma\_\{\\max\}\}\{\\delta\(1\-\\delta\)\},and

Mδ:=2​log⁡1−δδ\.M\_\{\\delta\}:=2\\log\\frac\{1\-\\delta\}\{\\delta\}\.Fort≥2t\\geq 2, let

zt=\(πt,γt,νt,pt\)∈X×Y\.z\_\{t\}=\(\\pi\_\{t\},\\gamma\_\{t\},\\nu\_\{t\},p\_\{t\}\)\\in X\\times Y\.Then the block gradients satisfy

‖gtπ−gt−1π‖∞\\displaystyle\\\|g\_\{t\}^\{\\pi\}\-g\_\{t\-1\}^\{\\pi\}\\\|\_\{\\infty\}≤Lπ​‖πt−πt−1‖1\+‖νt−νt−1‖1\+‖pt−pt−1‖2,\\displaystyle\\leq L\_\{\\pi\}\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\},\(41\)\|gtγ−gt−1γ\|\\displaystyle\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|≤d​Mδ​‖pt−pt−1‖2,\\displaystyle\\leq\\sqrt\{d\}\\,M\_\{\\delta\}\\,\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\},\(42\)‖gtν−gt−1ν‖∞\\displaystyle\\\|g\_\{t\}^\{\\nu\}\-g\_\{t\-1\}^\{\\nu\}\\\|\_\{\\infty\}≤‖πt−πt−1‖1\+Lν​‖νt−νt−1‖1\+‖pt−pt−1‖2,\\displaystyle\\leq\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+L\_\{\\nu\}\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\},\(43\)‖gtp−gt−1p‖2\\displaystyle\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}≤d​‖πt−πt−1‖1\+d​‖νt−νt−1‖1\+Lp​‖pt−pt−1‖2\+d​Mδ​\|γt−γt−1\|\.\\displaystyle\\leq\\sqrt\{d\}\\,\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+\\sqrt\{d\}\\,\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+L\_\{p\}\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\}\+\\sqrt\{d\}\\,M\_\{\\delta\}\\,\|\\gamma\_\{t\}\-\\gamma\_\{t\-1\}\|\.\(44\)Consequently, if we define fort≥2t\\geq 2

Stπ:=‖πt−πt−1‖12,Stγ:=\|γt−γt−1\|2,Stν:=‖νt−νt−1‖12,Stp:=‖pt−pt−1‖22,S\_\{t\}^\{\\pi\}:=\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}^\{2\},\\qquad S\_\{t\}^\{\\gamma\}:=\|\\gamma\_\{t\}\-\\gamma\_\{t\-1\}\|^\{2\},\\qquad S\_\{t\}^\{\\nu\}:=\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}^\{2\},\\qquad S\_\{t\}^\{p\}:=\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\}^\{2\},and

L:=max⁡\{1,Lπ,Lν,Lp,d,d​Mδ\},L:=\\max\\left\\\{1,\\,L\_\{\\pi\},\\,L\_\{\\nu\},\\,L\_\{p\},\\,\\sqrt\{d\},\\,\\sqrt\{d\}\\,M\_\{\\delta\}\\right\\\},then

‖gtπ−gt−1π‖∞2\\displaystyle\\\|g\_\{t\}^\{\\pi\}\-g\_\{t\-1\}^\{\\pi\}\\\|\_\{\\infty\}^\{2\}≤3​L2​\(Stπ\+Stν\+Stp\),\\displaystyle\\leq 3L^\{2\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\),\(45\)\|gtγ−gt−1γ\|2\\displaystyle\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|^\{2\}≤L2​Stp,\\displaystyle\\leq L^\{2\}S\_\{t\}^\{p\},\(46\)‖gtν−gt−1ν‖∞2\\displaystyle\\\|g\_\{t\}^\{\\nu\}\-g\_\{t\-1\}^\{\\nu\}\\\|\_\{\\infty\}^\{2\}≤3​L2​\(Stπ\+Stν\+Stp\),\\displaystyle\\leq 3L^\{2\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\),\(47\)‖gtp−gt−1p‖22\\displaystyle\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}^\{2\}≤4​L2​\(Stπ\+Stγ\+Stν\+Stp\)\.\\displaystyle\\leq 4L^\{2\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\gamma\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\)\.\(48\)

###### Proof\.

We prove the first\-order bounds and then square them\.

Theπ\\pi\-block\.From \([51](https://arxiv.org/html/2610.00715#A4.E51)\) of Lemma[6](https://arxiv.org/html/2610.00715#Thmlemma6),

gπ​\(i\)=∑j=1mνj​Pp​\(i≻j\)−τ⁡\(log⁡πiπref,i\+1\)\.g^\{\\pi\}\(i\)=\\sum\_\{j=1\}^\{m\}\\nu\_\{j\}P\_\{p\}\(i\\succ j\)\-\\tau\\left\(\\log\\frac\{\\pi\_\{i\}\}\{\\pi\_\{\\mathrm\{ref\},i\}\}\+1\\right\)\.For the win\-rate component, since∑jνj=1\\sum\_\{j\}\\nu\_\{j\}=1,

\|∑jνt,j​Ppt​\(i≻j\)−∑jνt−1,j​Ppt−1​\(i≻j\)\|\\displaystyle\\left\|\\sum\_\{j\}\\nu\_\{t,j\}P\_\{p\_\{t\}\}\(i\\succ j\)\-\\sum\_\{j\}\\nu\_\{t\-1,j\}P\_\{p\_\{t\-1\}\}\(i\\succ j\)\\right\|≤\|∑j\(νt,j−νt−1,j\)​Ppt​\(i≻j\)\|\+\|∑jνt−1,j​\(Ppt​\(i≻j\)−Ppt−1​\(i≻j\)\)\|\\displaystyle\\qquad\\leq\\left\|\\sum\_\{j\}\(\\nu\_\{t,j\}\-\\nu\_\{t\-1,j\}\)P\_\{p\_\{t\}\}\(i\\succ j\)\\right\|\+\\left\|\\sum\_\{j\}\\nu\_\{t\-1,j\}\\bigl\(P\_\{p\_\{t\}\}\(i\\succ j\)\-P\_\{p\_\{t\-1\}\}\(i\\succ j\)\\bigr\)\\right\|≤‖νt−νt−1‖1\+‖pt−pt−1‖∞≤‖νt−νt−1‖1\+‖pt−pt−1‖2\.\\displaystyle\\qquad\\leq\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{\\infty\}\\leq\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\}\.For the KL term, since everyπt​\(i\)≥ϵ​πref,i≥ϵ​πref,min\\pi\_\{t\}\(i\)\\geq\\epsilon\\pi\_\{\\mathrm\{ref\},i\}\\geq\\epsilon\\pi\_\{\\mathrm\{ref\},\\min\},

\|log⁡πt​\(i\)πref,i−log⁡πt−1​\(i\)πref,i\|=\|log⁡πt​\(i\)−log⁡πt−1​\(i\)\|≤\|πt​\(i\)−πt−1​\(i\)\|ϵ​πref,min\\left\|\\log\\frac\{\\pi\_\{t\}\(i\)\}\{\\pi\_\{\\mathrm\{ref\},i\}\}\-\\log\\frac\{\\pi\_\{t\-1\}\(i\)\}\{\\pi\_\{\\mathrm\{ref\},i\}\}\\right\|=\|\\log\\pi\_\{t\}\(i\)\-\\log\\pi\_\{t\-1\}\(i\)\|\\leq\\frac\{\|\\pi\_\{t\}\(i\)\-\\pi\_\{t\-1\}\(i\)\|\}\{\\epsilon\\pi\_\{\\mathrm\{ref\},\\min\}\}by the mean\-value theorem\. Taking the maximum overiiproves \([41](https://arxiv.org/html/2610.00715#A4.E41)\)\.

Theγ\\gamma\-block\.By \([52](https://arxiv.org/html/2610.00715#A4.E52)\) of Lemma[6](https://arxiv.org/html/2610.00715#Thmlemma6),

gγ=DKLBern\(p∥p⋆\)−ρ\.g^\{\\gamma\}=D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho\.For each coordinatee=\(i,j\)∈Ee=\(i,j\)\\in E, the derivative of the scalar Bernoulli\-KL term with respect topep\_\{e\}is

he′​\(u\)=log⁡u⁡\(1−pe⋆\)pe⋆​\(1−u\)\.h\_\{e\}^\{\\prime\}\(u\)=\\log\\frac\{u\(1\-p\_\{e\}^\{\\star\}\)\}\{p\_\{e\}^\{\\star\}\(1\-u\)\}\.Since bothuuandpe⋆p\_\{e\}^\{\\star\}lie in\[δ,1−δ\]\[\\delta,1\-\\delta\], we have

\|he′​\(u\)\|≤\|logit⁡\(u\)\|\+\|logit⁡\(pe⋆\)\|≤2​log⁡1−δδ=Mδ\.\|h\_\{e\}^\{\\prime\}\(u\)\|\\leq\|\\logit\(u\)\|\+\|\\logit\(p\_\{e\}^\{\\star\}\)\|\\leq 2\\log\\frac\{1\-\\delta\}\{\\delta\}=M\_\{\\delta\}\.Therefore, by the mean\-value theorem and Cauchy–Schwarz,

\|gtγ−gt−1γ\|≤∑e=1dMδ​\|pt,e−pt−1,e\|≤d​Mδ​‖pt−pt−1‖2,\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|\\leq\\sum\_\{e=1\}^\{d\}M\_\{\\delta\}\\,\|p\_\{t,e\}\-p\_\{t\-1,e\}\|\\leq\\sqrt\{d\}\\,M\_\{\\delta\}\\,\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\},which proves \([42](https://arxiv.org/html/2610.00715#A4.E42)\)\.

Theν\\nu\-block\.The proof is identical to theπ\\pi\-block, now using the floorνt,j≥β​πref,j≥β​πref,min\\nu\_\{t,j\}\\geq\\beta\\pi\_\{\\mathrm\{ref\},j\}\\geq\\beta\\pi\_\{\\mathrm\{ref\},\\min\}\. This gives \([43](https://arxiv.org/html/2610.00715#A4.E43)\)\.

Thepp\-block\.From \([54](https://arxiv.org/html/2610.00715#A4.E54)\) of Lemma[6](https://arxiv.org/html/2610.00715#Thmlemma6),

gi​jp=wi​j​\(π,ν\)\+γ​hi​j​\(pi​j\),hi​j​\(u\):=log⁡u⁡\(1−pi​j⋆\)pi​j⋆​\(1−u\)\.g^\{p\}\_\{ij\}=w\_\{ij\}\(\\pi,\\nu\)\+\\gamma\\,h\_\{ij\}\(p\_\{ij\}\),\\qquad h\_\{ij\}\(u\):=\\log\\frac\{u\(1\-p\_\{ij\}^\{\\star\}\)\}\{p\_\{ij\}^\{\\star\}\(1\-u\)\}\.For the antisymmetric weights, fix\(i,j\)∈E\(i,j\)\\in E\. Then

\|wi​j​\(πt,νt\)−wi​j​\(πt−1,νt−1\)\|\\displaystyle\|w\_\{ij\}\(\\pi\_\{t\},\\nu\_\{t\}\)\-w\_\{ij\}\(\\pi\_\{t\-1\},\\nu\_\{t\-1\}\)\|=\|πt,i​νt,j−πt,j​νt,i−πt−1,i​νt−1,j\+πt−1,j​νt−1,i\|\\displaystyle=\|\\pi\_\{t,i\}\\nu\_\{t,j\}\-\\pi\_\{t,j\}\\nu\_\{t,i\}\-\\pi\_\{t\-1,i\}\\nu\_\{t\-1,j\}\+\\pi\_\{t\-1,j\}\\nu\_\{t\-1,i\}\|≤\|πt,i−πt−1,i\|\+\|πt,j−πt−1,j\|\+\|νt,i−νt−1,i\|\+\|νt,j−νt−1,j\|\\displaystyle\\leq\|\\pi\_\{t,i\}\-\\pi\_\{t\-1,i\}\|\+\|\\pi\_\{t,j\}\-\\pi\_\{t\-1,j\}\|\+\|\\nu\_\{t,i\}\-\\nu\_\{t\-1,i\}\|\+\|\\nu\_\{t,j\}\-\\nu\_\{t\-1,j\}\|≤‖πt−πt−1‖1\+‖νt−νt−1‖1\.\\displaystyle\\leq\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\.Hence

‖w⁡\(πt,νt\)−w⁡\(πt−1,νt−1\)‖2≤d​‖πt−πt−1‖1\+d​‖νt−νt−1‖1\.\\\|w\(\\pi\_\{t\},\\nu\_\{t\}\)\-w\(\\pi\_\{t\-1\},\\nu\_\{t\-1\}\)\\\|\_\{2\}\\leq\\sqrt\{d\}\\,\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+\\sqrt\{d\}\\,\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\.
For the KL\-log term,

\|hi​j′​\(u\)\|=1u⁡\(1−u\)≤1δ⁡\(1−δ\),\|h\_\{ij\}^\{\\prime\}\(u\)\|=\\frac\{1\}\{u\(1\-u\)\}\\leq\\frac\{1\}\{\\delta\(1\-\\delta\)\},so

‖h⁡\(pt\)−h⁡\(pt−1\)‖2≤1δ⁡\(1−δ\)​‖pt−pt−1‖2\.\\\|h\(p\_\{t\}\)\-h\(p\_\{t\-1\}\)\\\|\_\{2\}\\leq\\frac\{1\}\{\\delta\(1\-\\delta\)\}\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\}\.Also\|hi​j​\(u\)\|≤Mδ\|h\_\{ij\}\(u\)\|\\leq M\_\{\\delta\}on\[δ,1−δ\]\[\\delta,1\-\\delta\], so

‖h⁡\(pt−1\)‖2≤d​Mδ\.\\\|h\(p\_\{t\-1\}\)\\\|\_\{2\}\\leq\\sqrt\{d\}\\,M\_\{\\delta\}\.Therefore

‖gtp−gt−1p‖2\\displaystyle\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}≤‖w⁡\(πt,νt\)−w⁡\(πt−1,νt−1\)‖2\+γmax​‖h⁡\(pt\)−h⁡\(pt−1\)‖2\+\|γt−γt−1\|​‖h⁡\(pt−1\)‖2\\displaystyle\\leq\\\|w\(\\pi\_\{t\},\\nu\_\{t\}\)\-w\(\\pi\_\{t\-1\},\\nu\_\{t\-1\}\)\\\|\_\{2\}\+\\gamma\_\{\\max\}\\\|h\(p\_\{t\}\)\-h\(p\_\{t\-1\}\)\\\|\_\{2\}\+\|\\gamma\_\{t\}\-\\gamma\_\{t\-1\}\|\\,\\\|h\(p\_\{t\-1\}\)\\\|\_\{2\}≤d​‖πt−πt−1‖1\+d​‖νt−νt−1‖1\+Lp​‖pt−pt−1‖2\+d​Mδ​\|γt−γt−1\|,\\displaystyle\\leq\\sqrt\{d\}\\,\\\|\\pi\_\{t\}\-\\pi\_\{t\-1\}\\\|\_\{1\}\+\\sqrt\{d\}\\,\\\|\\nu\_\{t\}\-\\nu\_\{t\-1\}\\\|\_\{1\}\+L\_\{p\}\\\|p\_\{t\}\-p\_\{t\-1\}\\\|\_\{2\}\+\\sqrt\{d\}\\,M\_\{\\delta\}\\,\|\\gamma\_\{t\}\-\\gamma\_\{t\-1\}\|,which proves \([44](https://arxiv.org/html/2610.00715#A4.E44)\)\.

Finally, \([45](https://arxiv.org/html/2610.00715#A4.E45)\)–\([48](https://arxiv.org/html/2610.00715#A4.E48)\) follow from

\(a\+b\+c\)2≤3​\(a2\+b2\+c2\),\(a\+b\+c\+d\)2≤4​\(a2\+b2\+c2\+d2\)\.\(a\+b\+c\)^\{2\}\\leq 3\(a^\{2\}\+b^\{2\}\+c^\{2\}\),\\qquad\(a\+b\+c\+d\)^\{2\}\\leq 4\(a^\{2\}\+b^\{2\}\+c^\{2\}\+d^\{2\}\)\.∎

###### Lemma 4\(Gradient variation is absorbed by the stability terms\)\.

Let

VT:=η​∑t=1T\(‖gtπ−gt−1π‖∞2\+\|gtγ−gt−1γ\|2\+‖gtν−gt−1ν‖∞2\+‖gtp−gt−1p‖22\),V\_\{T\}:=\\eta\\sum\_\{t=1\}^\{T\}\\left\(\\\|g\_\{t\}^\{\\pi\}\-g\_\{t\-1\}^\{\\pi\}\\\|\_\{\\infty\}^\{2\}\+\|g\_\{t\}^\{\\gamma\}\-g\_\{t\-1\}^\{\\gamma\}\|^\{2\}\+\\\|g\_\{t\}^\{\\nu\}\-g\_\{t\-1\}^\{\\nu\}\\\|\_\{\\infty\}^\{2\}\+\\\|g\_\{t\}^\{p\}\-g\_\{t\-1\}^\{p\}\\\|\_\{2\}^\{2\}\\right\),and

ΞT:=12​η∑t=1T\(DKL\(πt∥πt\+1\)\+\|γt−γt\+1\|2\+DKL\(νt∥νt\+1\)\+DKLBern\(pt∥pt\+1\)\)\.\\Xi\_\{T\}:=\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}\\left\(D\_\{\\mathrm\{KL\}\}\(\\pi\_\{t\}\\\|\\pi\_\{t\+1\}\)\+\|\\gamma\_\{t\}\-\\gamma\_\{t\+1\}\|^\{2\}\+D\_\{\\mathrm\{KL\}\}\(\\nu\_\{t\}\\\|\\nu\_\{t\+1\}\)\+D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{t\}\\\|p\_\{t\+1\}\)\\right\)\.If

η≤18​L,\\eta\\leq\\frac\{1\}\{8L\},\(49\)whereLLis the constant from Lemma[3](https://arxiv.org/html/2610.00715#Thmlemma3), then

VT≤ΞT\.V\_\{T\}\\leq\\Xi\_\{T\}\.\(50\)

###### Proof\.

Becauseg0∙=g1∙g\_\{0\}^\{\\bullet\}=g\_\{1\}^\{\\bullet\}, thet=1t=1contribution toVTV\_\{T\}is zero\. Thus we sum fromt=2t=2onward\. By Lemma[3](https://arxiv.org/html/2610.00715#Thmlemma3),

VT≤11​L2​η​∑t=2T\(Stπ\+Stγ\+Stν\+Stp\)\.V\_\{T\}\\leq 11L^\{2\}\\eta\\sum\_\{t=2\}^\{T\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\gamma\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\)\.On the other hand, Pinsker’s inequality and the Bernoulli Pinsker bound give

DKL\(πt∥πt\+1\)≥12∥πt−πt\+1∥12=12St\+1π,D\_\{\\mathrm\{KL\}\}\(\\pi\_\{t\}\\\|\\pi\_\{t\+1\}\)\\geq\\frac\{1\}\{2\}\\\|\\pi\_\{t\}\-\\pi\_\{t\+1\}\\\|\_\{1\}^\{2\}=\\frac\{1\}\{2\}S\_\{t\+1\}^\{\\pi\},DKL\(νt∥νt\+1\)≥12∥νt−νt\+1∥12=12St\+1ν,D\_\{\\mathrm\{KL\}\}\(\\nu\_\{t\}\\\|\\nu\_\{t\+1\}\)\\geq\\frac\{1\}\{2\}\\\|\\nu\_\{t\}\-\\nu\_\{t\+1\}\\\|\_\{1\}^\{2\}=\\frac\{1\}\{2\}S\_\{t\+1\}^\{\\nu\},DKLBern\(pt∥pt\+1\)≥2∥pt−pt\+1∥22=2St\+1p\.D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{t\}\\\|p\_\{t\+1\}\)\\geq 2\\\|p\_\{t\}\-p\_\{t\+1\}\\\|\_\{2\}^\{2\}=2S\_\{t\+1\}^\{p\}\.Hence

ΞT\\displaystyle\\Xi\_\{T\}≥14​η​∑t=1TSt\+1π\+12​η​∑t=1TSt\+1γ\+14​η​∑t=1TSt\+1ν\+1η​∑t=1TSt\+1p\\displaystyle\\geq\\frac\{1\}\{4\\eta\}\\sum\_\{t=1\}^\{T\}S\_\{t\+1\}^\{\\pi\}\+\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}S\_\{t\+1\}^\{\\gamma\}\+\\frac\{1\}\{4\\eta\}\\sum\_\{t=1\}^\{T\}S\_\{t\+1\}^\{\\nu\}\+\\frac\{1\}\{\\eta\}\\sum\_\{t=1\}^\{T\}S\_\{t\+1\}^\{p\}≥14​η​∑t=2T\+1\(Stπ\+Stγ\+Stν\+Stp\)\\displaystyle\\geq\\frac\{1\}\{4\\eta\}\\sum\_\{t=2\}^\{T\+1\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\gamma\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\)≥14​η​∑t=2T\(Stπ\+Stγ\+Stν\+Stp\)\.\\displaystyle\\geq\\frac\{1\}\{4\\eta\}\\sum\_\{t=2\}^\{T\}\\bigl\(S\_\{t\}^\{\\pi\}\+S\_\{t\}^\{\\gamma\}\+S\_\{t\}^\{\\nu\}\+S\_\{t\}^\{p\}\\bigr\)\.ThereforeVT≤ΞTV\_\{T\}\\leq\\Xi\_\{T\}whenever

11​L2​η≤14​η,i\.e\.η≤12​11​L\.11L^\{2\}\\eta\\leq\\frac\{1\}\{4\\eta\},\\qquad\\text\{i\.e\.\}\\qquad\\eta\\leq\\frac\{1\}\{2\\sqrt\{11\}\\,L\}\.The stronger condition \([49](https://arxiv.org/html/2610.00715#A4.E49)\) is sufficient\. ∎

###### Lemma 5\(Standard optimistic mirror\-descent bound\)\.

Lethhbe a mirror map with Bregman divergenceDhD\_\{h\}, letη\>0\\eta\>0, and letg^t=2​gt−gt−1\\widehat\{g\}\_\{t\}=2g\_\{t\}\-g\_\{t\-1\}\. For minimization, the update

zt\+1=arg​minz∈Z⁡\{⟨g^t,z⟩\+1η​Dh​\(z,zt\)\}z\_\{t\+1\}=\\argmin\_\{z\\in Z\}\\left\\\{\\langle\\widehat\{g\}\_\{t\},z\\rangle\+\\frac\{1\}\{\\eta\}D\_\{h\}\(z,z\_\{t\}\)\\right\\\}satisfies, for every comparatoru∈Zu\\in Z,

∑t=1T⟨gt,zt−u⟩≤Dh​\(u,z1\)η\+η​∑t=1T‖gt−gt−1‖∗2−12​η​∑t=1TDh​\(zt\+1,zt\)\.\\sum\_\{t=1\}^\{T\}\\langle g\_\{t\},z\_\{t\}\-u\\rangle\\leq\\frac\{D\_\{h\}\(u,z\_\{1\}\)\}\{\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\\\|g\_\{t\}\-g\_\{t\-1\}\\\|\_\{\*\}^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}D\_\{h\}\(z\_\{t\+1\},z\_\{t\}\)\.For maximization, the analogous update witharg​max\\argmaxsatisfies

∑t=1T⟨gt,u−zt⟩≤Dh​\(u,z1\)η\+η​∑t=1T‖gt−gt−1‖∗2−12​η​∑t=1TDh​\(zt\+1,zt\)\.\\sum\_\{t=1\}^\{T\}\\langle g\_\{t\},u\-z\_\{t\}\\rangle\\leq\\frac\{D\_\{h\}\(u,z\_\{1\}\)\}\{\\eta\}\+\\eta\\sum\_\{t=1\}^\{T\}\\\|g\_\{t\}\-g\_\{t\-1\}\\\|\_\{\*\}^\{2\}\-\\frac\{1\}\{2\\eta\}\\sum\_\{t=1\}^\{T\}D\_\{h\}\(z\_\{t\+1\},z\_\{t\}\)\.

###### Lemma 6\(Partial gradients\)\.

Forz=\(π,γ,ν,p\)∈X×Yz=\(\\pi,\\gamma,\\nu,p\)\\in X\\times Y, the block gradients ofFFare:

gπ​\(i\)\\displaystyle g^\{\\pi\}\(i\)=∑j=1mνjPp\(i≻j\)−τ\(logπiπref,i\+1\),i=1,…,m,\\displaystyle=\\sum\_\{j=1\}^\{m\}\\nu\_\{j\}\\,P\_\{p\}\(i\\succ j\)\-\\tau\\left\(\\log\\frac\{\\pi\_\{i\}\}\{\\pi\_\{\\mathrm\{ref\},i\}\}\+1\\right\),\\qquad i=1,\\dots,m,\(51\)gγ\\displaystyle g^\{\\gamma\}=DKLBern\(p∥p⋆\)−ρ,\\displaystyle=D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\-\\rho,\(52\)gν​\(j\)\\displaystyle g^\{\\nu\}\(j\)=∑i=1mπiPp\(i≻j\)\+τ\(logνjπref,j\+1\),j=1,…,m,\\displaystyle=\\sum\_\{i=1\}^\{m\}\\pi\_\{i\}\\,P\_\{p\}\(i\\succ j\)\+\\tau\\left\(\\log\\frac\{\\nu\_\{j\}\}\{\\pi\_\{\\mathrm\{ref\},j\}\}\+1\\right\),\\qquad j=1,\\dots,m,\(53\)gi​jp\\displaystyle g^\{p\}\_\{ij\}=wi​j​\(π,ν\)\+γ​log⁡pi​j​\(1−pi​j⋆\)pi​j⋆​\(1−pi​j\),\(i,j\)∈E\.\\displaystyle=w\_\{ij\}\(\\pi,\\nu\)\+\\gamma\\log\\frac\{p\_\{ij\}\(1\-p^\{\\star\}\_\{ij\}\)\}\{p^\{\\star\}\_\{ij\}\(1\-p\_\{ij\}\)\},\\qquad\(i,j\)\\in E\.\(54\)

###### Proof\.

Differentiate \([17](https://arxiv.org/html/2610.00715#S4.E17)\)\. Theπ\\pi\- andν\\nu\-gradients are obtained from the linear win\-rate term and the KL regularizers\. Theγ\\gamma\-gradient is immediate becauseFFis affine inγ\\gamma\. For \([54](https://arxiv.org/html/2610.00715#A4.E54)\), the coefficient ofpi​jp\_\{ij\}in the win\-rate decomposition iswi​j​\(π,ν\)w\_\{ij\}\(\\pi,\\nu\), and

∂∂pi​jDKLBern\(p∥p⋆\)=logpi​j​\(1−pi​j⋆\)pi​j⋆​\(1−pi​j\)\.∎\\frac\{\\partial\}\{\\partial p\_\{ij\}\}D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)=\\log\\frac\{p\_\{ij\}\(1\-p^\{\\star\}\_\{ij\}\)\}\{p^\{\\star\}\_\{ij\}\(1\-p\_\{ij\}\)\}\.\\qed

## Appendix EIMDb Continuation Task: Experiment Details

### E\.1Nominal Preference and Annotator Groups

We use the IMDb review corpus\[[23](https://arxiv.org/html/2610.00715#bib.bib23)\]as a source of continuation prompts\. Each prompt is the first sentence of a review; the task is to continue it\. We sampleN=8,000N=8\{,\}000prompts and generatem=8m=8candidate continuations per prompt usinggoogle/flan\-t5\-large\[[11](https://arxiv.org/html/2610.00715#bib.bib11)\]with diverse instruction templates and sampling temperatures\. The full set of prompt\-level preference kernels is pre\-computed and cached so that NashMD and all robust variants are trained and evaluated on identical finite action sets\.

To model heterogeneous human preferences we construct three annotator groups:Group A\(comprehensive/analytical\) prefers longer, more detailed continuations with stronger evidence of analysis;Group B\(efficient/concise\) prefers short, direct, and decisive continuations; andGroup C\(balanced/nuanced\) prefers responses that acknowledge both strengths and weaknesses\. For promptxxand groupg∈\{A,B,C\}g\\in\\\{A,B,C\\\}, each candidateiireceives a scalar scoresi\(g\)s\_\{i\}^\{\(g\)\}\. These scores are converted into a group\-specific Bradley–Terry pairwise preference kernel:

Pi​j\(g\)=σ⁡\(β⁡\(si\(g\)−sj\(g\)\)\),i<j,P\_\{ij\}^\{\(g\)\}=\\sigma\\\!\\bigl\(\\beta\\,\(s\_\{i\}^\{\(g\)\}\-s\_\{j\}^\{\(g\)\}\)\\bigr\),\\quad i<j,\(55\)withPj​i\(g\)=1−Pi​j\(g\)P\_\{ji\}^\{\(g\)\}=1\-P\_\{ij\}^\{\(g\)\}andPi​i\(g\)=12P\_\{ii\}^\{\(g\)\}=\\tfrac\{1\}\{2\}\. Writingp\(g\)∈\[0,1\]dp^\{\(g\)\}\\in\[0,1\]^\{d\},d=\(m2\)=28d=\\binom\{m\}\{2\}=28, for the upper\-triangular vector, the nominal preference kernel isp⋆=∑g∈\{A,B,C\}αg​p\(g\)p^\{\\star\}=\\sum\_\{g\\in\\\{A,B,C\\\}\}\\alpha\_\{g\}\\,p^\{\(g\)\}with fixed mixture weightsα\\alpha\. Perturbing the mixture weights away fromα\\alphavia a Dirichlet draw produces*group\-shift*kernels that lie in the convex hull of the group kernels and serve as a model of structured population\-level preference shift \(Section[E\.4](https://arxiv.org/html/2610.00715#A5.SS4)\)\.

### E\.2Optimization Algorithms

#### NashMD baseline\.

The NashMD baseline solves the nominal KL\-regularized game

maxπ∈Δmminν∈ΔmJ\(π,ν;P⋆\)=π⊤P⋆ν−τDKL\(π∥πref\)\+τDKL\(ν∥πref\)\\max\_\{\\pi\\in\\Delta^\{m\}\}\\min\_\{\\nu\\in\\Delta^\{m\}\}J\(\\pi,\\nu;P^\{\\star\}\)=\\pi^\{\\top\}P^\{\\star\}\\nu\-\\tau\\,D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{\\mathrm\{ref\}\}\)\+\\tau\\,D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\pi\_\{\\mathrm\{ref\}\}\)\(56\)with nominal kernelP⋆=P⁡\(p⋆\)P^\{\\star\}=P\(p^\{\\star\}\)fixed throughout training\. We use simultaneous Optimistic Gradient Descent\-Ascent \(OGDA\) on the policy pair\(π,ν\)\(\\pi,\\nu\)with step sizeηNash=0\.30\\eta\_\{\\mathrm\{Nash\}\}=0\.30\.

#### Four\-player robust algorithm\.

The robust variant hedges against kernel misspecification within the Bernoulli\-KL ball𝒰ρ=\{p:DKLBern\(p∥p⋆\)≤ρ\}\\mathcal\{U\}\_\{\\rho\}=\\\{p:D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\\}by solving

maxπ∈Δmminp:DKLBern\(p∥p⋆\)≤ρp∈\[δ,1−δ\]dminν∈ΔmJ\(π,ν;P\(p\)\)\.\\max\_\{\\pi\\in\\Delta^\{m\}\}\\min\_\{\\begin\{subarray\}\{c\}p:\\,D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\\\|p^\{\\star\}\)\\leq\\rho\\\\ p\\in\[\\delta,1\-\\delta\]^\{d\}\\end\{subarray\}\}\\min\_\{\\nu\\in\\Delta^\{m\}\}J\(\\pi,\\nu;P\(p\)\)\.\(57\)The inner minimization is solved approximately by alternating between the closed\-form follower best\-response and the kernel updatepi​j←σ⁡\(logit⁡\(pi​j⋆\)−wi​j​\(π,ν\)/max⁡\(γ,γmin\)\)p\_\{ij\}\\leftarrow\\sigma\\\!\\bigl\(\\logit\(p^\{\\star\}\_\{ij\}\)\-w\_\{ij\}\(\\pi,\\nu\)/\\\!\\max\(\\gamma,\\gamma\_\{\\min\}\)\\bigr\)forK=8K=8inner steps, then the outer\(π,γ\)\(\\pi,\\gamma\)update is performed with OMD at step sizes\(ηπ,ηγ\)=\(0\.25,0\.15\)\(\\eta\_\{\\pi\},\\eta\_\{\\gamma\}\)=\(0\.25,0\.15\)\.

### E\.3Hyperparameters

Table 3:Hyperparameters for the IMDb continuation experiments\.
### E\.4Results

Given the robust policyπR\\pi\_\{R\}and the NashMD baselineπN\\pi\_\{N\}, we measure the win rate using[24](https://arxiv.org/html/2610.00715#S6.E24)\. The sampled kernels for evaluation are \(i\) the KL\-ball boundary𝒰ρeval\\mathcal\{U\}\_\{\\rho\_\{\\mathrm\{eval\}\}\}sampled uniformly at random, used for Table 1 results, and \(ii\) a Dirichlet\-perturbed annotator mixturep~=∑gα~g​p\(g\)\\tilde\{p\}=\\sum\_\{g\}\\tilde\{\\alpha\}\_\{g\}p^\{\(g\)\},α~∼Dirichlet⁡\(κ​α\)\\tilde\{\\alpha\}\\sim\\mathrm\{Dirichlet\}\(\\kappa\\alpha\), projected to the KL\-ball boundary at radiusρeval\\rho\_\{\\mathrm\{eval\}\}, used in Table[4](https://arxiv.org/html/2610.00715#A5.T4)\. Annotator group\-shift kernels remain in the low\-dimensional mixture subspace, making them a more structured perturbation family\.

Table 4:WinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)under group\-shift kernels atτ=0\.01\\tau=0\.01, averaged over8,0008\{,\}000prompts\. Bold marks the highest value in each column\.Table[5](https://arxiv.org/html/2610.00715#A5.T5)reportsWinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)at the best training radius for each KL\-regularization temperatureτ∈\{0\.001,0\.005,0\.01,0\.05,0\.1\}\\tau\\in\\\{0\.001,0\.005,0\.01,0\.05,0\.1\\\}\. The strongest overall regime isτ=0\.05\\tau=0\.05: the robust policy wins with probability above97%97\\%forρeval≤0\.30\\rho\_\{\\mathrm\{eval\}\}\\leq 0\.30\. At very smallτ\\tauthe NashMD baseline is already highly concentrated, reducing the marginal gain from robust training; at largeτ\\tauboth policies approach the uniform distribution and the adversary has limited room to exploit kernel perturbations\.

Table 5:WinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)at the bestρtrain\\rho\_\{\\mathrm\{train\}\}for eachτ\\tau\(KL\-random kernels\)\.Figure[2](https://arxiv.org/html/2610.00715#A5.F2)shows the full win\-rate curves atτ=0\.05\\tau=0\.05\. Training radiiρtrain≤0\.10\\rho\_\{\\mathrm\{train\}\}\\leq 0\.10are statistically indistinguishable from one another at every evaluation radius \(their pairwise differences never exceed the 95% confidence half\-width\), whileρtrain=0\.15\\rho\_\{\\mathrm\{train\}\}=0\.15is significantly worse, with the gap widening asρeval\\rho\_\{\\mathrm\{eval\}\}grows\.

![Refer to caption](https://arxiv.org/html/2610.00715v1/tau_sweep/tau_0.05/probwin_kl_rand.png)\(a\)KL\-random perturbations
![Refer to caption](https://arxiv.org/html/2610.00715v1/tau_sweep/tau_0.05/probwin_group_shift.png)\(b\)Group\-shift perturbations

Figure 2:WinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)atτ=0\.05\\tau=0\.05, for each training radiusρtrain∈\{0\.005,0\.01,0\.02,0\.05,0\.10,0\.15\}\\rho\_\{\\mathrm\{train\}\}\\in\\\{0\.005,0\.01,0\.02,0\.05,0\.10,0\.15\\\}\. Radiiρtrain∈\[0\.005,0\.10\]\\rho\_\{\\mathrm\{train\}\}\\in\[0\.005,0\.10\]\(gray band and mean\) are statistically tied at every evaluation radius;ρtrain=0\.15\\rho\_\{\\mathrm\{train\}\}=0\.15\(red\) is significantly worse, and increasingly so asρeval\\rho\_\{\\mathrm\{eval\}\}grows\.Table[6](https://arxiv.org/html/2610.00715#A5.T6)gives the full\(τ,ρtrain\)\(\\tau,\\rho\_\{\\mathrm\{train\}\}\)interaction atρeval=0\.30\\rho\_\{\\mathrm\{eval\}\}=0\.30\. Forτ≤0\.01\\tau\\leq 0\.01the smallest training radius is uniformly best\. Forτ≥0\.05\\tau\\geq 0\.05the optimum shifts toρtrain=0\.10\\rho\_\{\\mathrm\{train\}\}=0\.10, though the improvement over smaller radii is modest\.

Table 6:WinRate⁡\(πR​vs​πN,ρeval\)\\mathrm\{WinRate\}\(\\pi\_\{R\}\\text\{ vs \}\\pi\_\{N\};\\,\\rho\_\{\\mathrm\{eval\}\}\)atρeval=0\.30\\rho\_\{\\mathrm\{eval\}\}=0\.30, by\(τ,ρtrain\)\(\\tau,\\rho\_\{\\mathrm\{train\}\}\)\(KL\-random kernels\)\. Bold marks the row\-wise maximum\.

## Appendix FTL;DR Summarization Task: Experiment Details

### F\.1Fine\-Tuning Models\.

We initialize the models fromQwen2\.5\-1\.5B\-Instruct\[[51](https://arxiv.org/html/2610.00715#bib.bib51)\]andGemma\-2\-2B\[[44](https://arxiv.org/html/2610.00715#bib.bib44)\]as both the initial and the reference model across the respective experiments and fine\-tune with LoRA\[[19](https://arxiv.org/html/2610.00715#bib.bib19)\]for 5000 gradient steps with KL regularizationτ=0\.005\\tau=0\.005\. The four\-player game formulation was compared with adversarial budgetρ∈\{0\.01,0\.05\}\\rho\\in\\\{0\.01,0\.05\\\}\. The leader and follower policies carry separate LoRA adapters attached to the same frozen backbone\.

### F\.2Nominal Preference Proxy

We train a Bradley\-Terry\(BT\) preference judge model to serve as the nominal preference proxy\. The judge is a scalar score modelsϕ:\(x,y\)→ℝs\_\{\\phi\}:\(x,y\)\\to\\mathbb\{R\}built onDistilBERT\-base\-uncase\[[39](https://arxiv.org/html/2610.00715#bib.bib39)\]backbone with a linear scalar head, and fine\-tuned with the standard BT cross\-entropy loss on the TL;DR dataset:

ℒBT​\(ϕ\)=−𝔼\(x,yw,yl\)​\[log⁡σ⁡\(sϕ​\(x,yw\)−sϕ​\(x,yl\)\)\],\\mathcal\{L\}\_\{\\mathrm\{BT\}\}\(\\phi\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\left\[\\log\\sigma\\big\(s\_\{\\phi\}\(x,y\_\{w\}\)\-s\_\{\\phi\}\(x,y\_\{l\}\)\\big\)\\right\],\(58\)
where\(yw,yl\)\(y\_\{w\},y\_\{l\}\)denotes the preferred and dispreferred responses respectively\. Training uses 83,573 comparison pairs\. After 3 epochs the judge achieves 72% validation accuracy\. During training with both NashMD and Four\-Player the pairwise preference is calculated as:

P∗​\(ya≻yb∣x\)=σ⁡\(sϕ​\(x,ya\)−sϕ​\(x,yb\)\)P^\{\*\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)=\\sigma\\big\(s\_\{\\phi\}\(x,y\_\{a\}\)\-s\_\{\\phi\}\(x,y\_\{b\}\)\\big\)\(59\)

### F\.3Adversary Proxy

The kernel\-adversary in the four\-player game is parameterized by an*antisymmetric*perturbation headδψ​\(ya,yb\)\\delta\_\{\\psi\}\(y\_\{a\},y\_\{b\}\)of the nominal preference logit\. The head computes

δψ​\(ya,yb\)=gψ​\(ua​b\)−gψ​\(ub​a\),gψ:ℝ7→ℝ,\\delta\_\{\\psi\}\(y\_\{a\},y\_\{b\}\)\\;=\\;g\_\{\\psi\}\(u\_\{ab\}\)\-g\_\{\\psi\}\(u\_\{ba\}\),\\qquad g\_\{\\psi\}:\\mathbb\{R\}^\{7\}\\to\\mathbb\{R\},\(60\)wheregψg\_\{\\psi\}is a two\-layer MLP:ℝ7→W1ℝ64→tanhℝ64→W2ℝ\\mathbb\{R\}^\{7\}\\xrightarrow\{\\;W\_\{1\}\\;\}\\mathbb\{R\}^\{64\}\\xrightarrow\{\\;\\tanh\\;\}\\mathbb\{R\}^\{64\}\\xrightarrow\{\\;W\_\{2\}\\;\}\\mathbb\{R\}, with the output layerW2W\_\{2\}initialized to zero so thatδψ​\(ya,yb\)=0\\delta\_\{\\psi\}\(y\_\{a\},y\_\{b\}\)=0\(no perturbation\) at initialization\. The algorithm therefore initializes at the nominal preference and gradually grows the adversarial perturbation\. Antisymmetry in \([60](https://arxiv.org/html/2610.00715#A6.E60)\) ensuresδψ​\(ya,yb\)=−δψ​\(yb,ya\)\\delta\_\{\\psi\}\(y\_\{a\},y\_\{b\}\)=\-\\delta\_\{\\psi\}\(y\_\{b\},y\_\{a\}\)and hencePψ​\(ya≻yb\)\+Pψ​\(yb≻ya\)=1P\_\{\\psi\}\(y\_\{a\}\\succ y\_\{b\}\)\+P\_\{\\psi\}\(y\_\{b\}\\succ y\_\{a\}\)=1at all iterates\.

The perturbed preference is

Pψ​\(ya≻yb∣x\)=σ⁡\(logit⁡P∗​\(ya≻yb∣x\)\+δψ​\(ya,yb\)\)\.P\_\{\\psi\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)\\;=\\;\\sigma\\\!\\bigl\(\\logit P^\{\*\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)\+\\delta\_\{\\psi\}\(y\_\{a\},y\_\{b\}\)\\bigr\)\.\(61\)
The Bernoulli\-KL ambiguity constraint𝔼\[DKLBern\(pψ∥p⋆\)\]≤ρ\\mathbb\{E\}\\bigl\[D\_\{\\mathrm\{KL\}\}^\{\\mathrm\{Bern\}\}\(p\_\{\\psi\}\\,\\\|\\,p^\{\\star\}\)\\bigr\]\\leq\\rhois enforced softly through the dual variableγ\\gammain the Lagrangian payoff\. We additionally clip the predicted logit shift before applying the sigmoid in[61](https://arxiv.org/html/2610.00715#A6.E61)to preventPψP\_\{\\psi\}from saturating to\{0,1\}\\\{0,1\\\}to stabilize the dual ascent\.

### F\.4Losses\.

For a mini\-batch with samplesyL∼πLy\_\{L\}\\sim\\pi\_\{\\mathrm\{L\}\}andyF∼πFy\_\{F\}\\sim\\pi\_\{\\mathrm\{F\}\}, we definepψ=Pψ​\(yL≻yF∣x\)p\_\{\\psi\}=P\_\{\\psi\}\(y\_\{L\}\\succ y\_\{F\}\\mid x\)and KL proxiesκL=clip⁡\(log⁡πL​\(yL\)/πref​\(yL\),−c,c\)\\kappa\_\{L\}=\\mathrm\{clip\}\(\\log\\pi\_\{\\mathrm\{L\}\}\(y\_\{L\}\)/\\pi\_\{\\mathrm\{ref\}\}\(y\_\{L\}\),\-c,c\),κF=clip⁡\(log⁡πF​\(yF\)/πref​\(yF\),−c,c\)\\kappa\_\{F\}=\\mathrm\{clip\}\(\\log\\pi\_\{\\mathrm\{F\}\}\(y\_\{F\}\)/\\pi\_\{\\mathrm\{ref\}\}\(y\_\{F\}\),\-c,c\)\. The three player losses are

ℒψ\\displaystyle\\mathcal\{L\}\_\{\\psi\}=𝔼\[pψ\]\+γ\(𝔼\[DKLB​e​r​n\(pψ∥P∗\)\]−ρ\),\\displaystyle=\\mathbb\{E\}\\bigl\[p\_\{\\psi\}\\bigr\]\+\\gamma\\,\\bigl\(\\mathbb\{E\}\[D\_\{\\mathrm\{KL\}\}^\{Bern\}\(p\_\{\\psi\}\\,\\\|\\,P^\{\*\}\)\]\-\\rho\\bigr\),\(62\)ℒπ\\displaystyle\\mathcal\{L\}\_\{\\pi\}=−𝔼⁡\[\(pψ−τ​κL−bL\)​log⁡πL​\(yL\)\],\\displaystyle=\-\\mathbb\{E\}\\bigl\[\(p\_\{\\psi\}\-\\tau\\kappa\_\{L\}\-b\_\{L\}\)\\,\\log\\pi\_\{\\mathrm\{L\}\}\(y\_\{L\}\)\\bigr\],\(63\)ℒν\\displaystyle\\mathcal\{L\}\_\{\\nu\}=−𝔼⁡\[\(−pψ−τ​κF−bF\)​log⁡πF​\(yF\)\],\\displaystyle=\-\\mathbb\{E\}\\bigl\[\(\-p\_\{\\psi\}\-\\tau\\kappa\_\{F\}\-b\_\{F\}\)\\,\\log\\pi\_\{\\mathrm\{F\}\}\(y\_\{F\}\)\\bigr\],\(64\)wherebLb\_\{L\},bFb\_\{F\}are EMA baselines for variance reduction, and the adversaryψ\\psi*maximizes*via \([62](https://arxiv.org/html/2610.00715#A6.E62)\) \(gradient ascent\) while the dual variable is updated by clipped projected gradient ascent:

γ←clip⁡\(γ\+ηγ​\(𝔼⁡\[DKLB​e​r​n\]−ρ\),0,γmax\)\.\\gamma\\;\\leftarrow\\;\\mathrm\{clip\}\\\!\\Bigl\(\\gamma\+\\eta\_\{\\gamma\}\\,\\bigl\(\\mathbb\{E\}\[D\_\{\\mathrm\{KL\}\}^\{Bern\}\]\-\\rho\\bigr\),\\;0,\\;\\gamma\_\{\\max\}\\Bigr\)\.\(65\)
Each outer step interleavesKKadversary updates per\(πL,πF,γ\)\(\\pi\_\{L\},\\pi\_\{F\},\\gamma\)update\. We default toK=3K=3as we obsevedK=1K=1is too weak \(the constraint is far from active\) whileK≥3K\\geq 3tracks the constraint budget closely\.

### F\.5Evaluation with LLM Judges

We employ three comparatively larger open\-source models—DeepSeek\-R1\-Distill\-Qwen\-32B\[[12](https://arxiv.org/html/2610.00715#bib.bib12)\], Gemma\-4\-31B\[[44](https://arxiv.org/html/2610.00715#bib.bib44)\], and Nemotron\-3\-Nano\-Omni\[[32](https://arxiv.org/html/2610.00715#bib.bib32)\]—to obtain preference signals querying with the following prompt template:

> You are an expert judge evaluating the quality of two generated summaries\. Post to summarize: ⟨text⟩ Summary A: ⟨summary1⟩ Summary B: ⟨summary2⟩ Evaluate which summary is better based on: \(1\) Accuracy, \(2\) Coherence, \(3\) Conciseness, \(4\) Helpfulness\.

where⟨text⟩\\langle\\text\{text\}\\ranglecorresponds to the Reddit postxx,⟨summary1⟩\\langle\\text\{summary1\}\\rangleto the policy responsey∼π\(⋅∣x\)y\\sim\\pi\(\\cdot\\mid x\), and⟨summary2⟩\\langle\\text\{summary2\}\\rangleto the baseline responsey′∼πbase\(⋅∣x\)y^\{\\prime\}\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid x\)\. We estimate the preference probabilityPjext​\(π≻πbase\)P^\{\\text\{ext\}\}\_\{j\}\(\\pi\\succ\\pi\_\{\\text\{base\}\}\)as the fraction of comparisons in which the policy output is preferred over10001000prompts from the TL;DR dataset\. Following Munos et al\.\[[29](https://arxiv.org/html/2610.00715#bib.bib29)\], each comparison is treated as a Bernoulli trial and uncertainty is quantified using Clopper–Pearson 95% confidence intervals; forn=1000n=1000, the worst\-case half\-width is approximately±0\.032\\pm 0\.032\. Four player policies preserve strong nominal performance against the NashMD baseline\. Training against the preference uncertainty has not collapsed into overly conservative solutions, rather, the robust policies still improve the nominal pairwise objective with the pairwise judges across evaluation criteria\.

### F\.6Non\-BT Nominal Kernel: Construction and Diagnostic

The TL;DR experiment in Section[6\.3](https://arxiv.org/html/2610.00715#S6.SS3)uses a non\-Bradley–Terry \(non\-BT\) nominal kernel rather than the standard, reproducible Bradley–Terry \(BT\) construction \(BT judge, Appendix[F\.2](https://arxiv.org/html/2610.00715#A6.SS2)\)\[[41](https://arxiv.org/html/2610.00715#bib.bib41)\], which is antisymmetric by design:P⋆​\(ya≻yb∣x\)=σ⁡\(sϕ​\(x,ya\)−sϕ​\(x,yb\)\)P^\{\\star\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)=\\sigma\\big\(s\_\{\\phi\}\(x,y\_\{a\}\)\-s\_\{\\phi\}\(x,y\_\{b\}\)\\big\)\. The ambiguity set𝒫\\mathcal\{P\}in Eq\. \([8](https://arxiv.org/html/2610.00715#S3.E8)\) is a Bernoulli\-KL ball over thed=\(m2\)d=\\binom\{m\}\{2\}pairwise probabilities themselves \(imposed in expectation over sampled pairs in TL;DR, Appendix[F\.4](https://arxiv.org/html/2610.00715#A6.SS4)\), with no constraint of the formlogit⁡\(pi​j\)=si−sj\\mathrm\{logit\}\(p\_\{ij\}\)=s\_\{i\}\-s\_\{j\}: even from a BT center, the worst\-case kernel found by the adversary may already be cyclic or otherwise not representable by any scalar reward\. We go further and construct a nominal kernel that is itself non\-BT, described next, together with a diagnostic confirming it is measurably so\.

Non\-BT nominal kernel\.We replace the BT judge with a directly parameterized pairwise kernel,

Pϕpair​\(ya≻yb∣x\)=σ⁡\(Lpair​\(x,ya,yb\)\),Lpair​\(x,ya,yb\)=c2​\[qϕ​\(x,ya,yb\)−qϕ​\(x,yb,ya\)\],P^\{\\mathrm\{pair\}\}\_\{\\phi\}\(y\_\{a\}\\succ y\_\{b\}\\mid x\)=\\sigma\\big\(L^\{\\mathrm\{pair\}\}\(x,y\_\{a\},y\_\{b\}\)\\big\),\\qquad L^\{\\mathrm\{pair\}\}\(x,y\_\{a\},y\_\{b\}\)=\\tfrac\{c\}\{2\}\\big\[q\_\{\\phi\}\(x,y\_\{a\},y\_\{b\}\)\-q\_\{\\phi\}\(x,y\_\{b\},y\_\{a\}\)\\big\],whereqϕq\_\{\\phi\}is a single joint forward pass over the post and both candidate summaries, instantiated with PairRM\[[21](https://arxiv.org/html/2610.00715#bib.bib21)\]\(a 0\.4B DeBERTa\-v3\-large cross\-encoder\)\. Each candidate’s representation is conditioned on its opponent, so its score is not a function of\(x,y\)\(x,y\)alone and the kernel is not constrained to be BT\. Becauseqϕq\_\{\\phi\}is order\-dependent, the difference of the two orderings is what givesLpair​\(a,b\)=−Lpair​\(b,a\)L^\{\\mathrm\{pair\}\}\(a,b\)=\-L^\{\\mathrm\{pair\}\}\(b,a\), hencePpair​\(a≻b\)\+Ppair​\(b≻a\)=1P^\{\\mathrm\{pair\}\}\(a\\succ b\)\+P^\{\\mathrm\{pair\}\}\(b\\succ a\)=1exactly\. Herec\>0c\>0is a scalar temperature fit by maximum likelihood on TL;DR human comparisons\. The policy is Qwen2\.5\-1\.5B\-Instruct\. The adversary head, Bernoulli\-KL constraint, dual update and NashMD baseline are as in Appendix[F\.3](https://arxiv.org/html/2610.00715#A6.SS3)–[F\.4](https://arxiv.org/html/2610.00715#A6.SS4); the only change is that three of the adversary’s seven input features, which used per\-response rewards, now carry the pairwise logit\.

The kernel is not BT\.LetLi​j=logit​Pϕpair​\(yi≻yj∣x\)L\_\{ij\}=\\mathrm\{logit\}\\,P^\{\\mathrm\{pair\}\}\_\{\\phi\}\(y\_\{i\}\\succ y\_\{j\}\\mid x\)\. A BT kernel hasLi​j=si−sjL\_\{ij\}=s\_\{i\}\-s\_\{j\}, so for every tripleC:=Li​j\+Lj​k\+Lk​i=0C:=L\_\{ij\}\+L\_\{jk\}\+L\_\{ki\}=0: the scores telescope away, for any reward model\. The least\-squares BT fit to a triple removesC/3C/3from each edge, so\|C\|/3\|C\|/3is the per\-edge distance to the nearest BT kernel\. On 1,000 triples of real model outputs \(per prompt, one summary from each of NashMD\-PG and the two robust policies trained under this kernel\), the mean best\-BT projection error is0\.3840\.384logits per edge \(3\.103\.10percentage points in preference probability\), the median\|C\|\|C\|is0\.960\.96, and1\.4%1\.4\\%of triples are outright cyclic\. Edges are confident \(\|L\|≈3\|L\|\\approx 3logits on average\), so a0\.380\.38\-logit residual rarely flips a sign: cycles are rare, yet the departure from BT is present in nearly every triple\. The nominal kernel used in training is therefore measurably non\-BT\.

### F\.7Additional Results with a Bradley\-Terry Nominal Kernel

For completeness, we also report results under the standard Bradley–Terry \(BT\) nominal kernel \(Appendix[F\.2](https://arxiv.org/html/2610.00715#A6.SS2)\) that Section[6\.3](https://arxiv.org/html/2610.00715#S6.SS3)’s non\-BT experiment departs from\. Table[7](https://arxiv.org/html/2610.00715#A6.T7)reports the win rate of the robust policy against NashMD on the same protocol as Table[2](https://arxiv.org/html/2610.00715#S6.T2)\(three external judges,10001000TL;DR prompts, Clopper–Pearson half\-width≈±0\.032\\approx\\pm 0\.032atn=1000n=1000\), and additionally covers a second fine\-tuning backbone, Gemma\-2B, alongside Qwen\-1\.5B\.

Table 7:Win rate against NashMD under external LLM judges, evaluated on10001000TL;DR prompts, with a Bradley\-Terry nominal kernel \(cf\. Table[2](https://arxiv.org/html/2610.00715#S6.T2), which uses a non\-BT kernel\)\.Fine\-Tuning Modelρtrain\\rho\_\{\\text\{train\}\}Win Rate under Evaluation ModelDeepSeek\-R1Gemma\-4Nemotron\-3Qwen\-1\.5B0\.010\.4960\.5060\.7640\.050\.5740\.5930\.568Gemma\-2B0\.010\.5420\.5180\.4620\.050\.6400\.7200\.583

## Appendix GLLM Judge Reasoning Examples

### Example 1 \(prompt\_id57\)

Disagreement:Nemotron\-3\-Nano→\\toA; Gemma\-4\-31B→\\toB; DeepSeek\-R1\-32B→\\toB

Post \(r/relationships\)Title:Me \[20m\] just got out of an 8 month relationship with my gf \[19F\]\. I am very emotional, she wasn’t\.I am far from the stereotypical macho guy\. I love romance \[…\] She never speaks about her feelings, never told me she loved me without me having to say it first \[…\] My question aimed at both genders: Are girls generally emotional, do you ladies like to speak about emotions? Do you generally like when the guy is cute with you and calls you babe, and says he loves you? \[…\]

Summary A \(NashMD\-PG\)OP is seeking advice on whether girls generally like to be romantic and express emotions, and if they prefer a partner who is affectionate and expresses love\.

Summary B \(Four\-Playerρ=0\.05\\rho\{=\}0\.05\)The OP is seeking to understand if girls generally express emotions and appreciate romantic displays of affection, as he is a romantic guy who enjoys being cute and expressing his love\. He’s questioning if this is something most girls find appealing\.

Nemotron\-3\-Nano\(verdict:A, 14\.4 s\)\[Conciseness\]A is more concise \(fewer words\)\. B includes extra info about his own traits—possible unnecessary padding\. So A wins on conciseness\. \[…\]\[Helpfulness\]A directly states the request for advice\. B adds context that may help reader understand OP’s perspective, but the key point—seeking advice about girls’ preferences—is stated more succinctly in A\. \[…\]Overall, Summary A seems better due to higher conciseness and directness while still accurate\.

Gemma\-4\-31B\(verdict:B, 67\.3 s\)\[Helpfulness\]Summary B captures the “why” of the post whereas A only captures the “what”\. The OP is not asking a generic question about gender; he is questioning his own experience in light of his personality and recent breakup\. This makes B a more complete reflection of the source text\. \[…\]Summary A is more concise; Summary B is more contextually complete without becoming wordy\. The motivation behind the question is a key part of the post, so B is slightly better overall\.

DeepSeek\-R1\-Distill\-Qwen\-32B\(verdict:B, 58\.6 s\)\[Helpfulness\]Summary A captures the main question but lacks some context\. Summary B provides more background, making it clearer why OP is asking this and what he has experienced\. \[…\]Summary B includes more details about OP’s romantic nature and his girlfriend’s behaviour, which are crucial for understanding his perspective\.

### Example 2 \(prompt\_id125\)

Disagreement:Nemotron\-3\-Nano→\\toA; Gemma\-4\-31B→\\toB; DeepSeek\-R1\-32B→\\toB

Post \(r/dating\_advice\)Title:I \(21,M\) need help getting a girl \(22\) who used to like me interested again\.I recently started messaging a girl I knew from high school and it seemed to be going somewhere, but a couple of bad decisions and me not being honest enough in my replies have led me into a rut\. She seems to have lost interest and only replies when I message first\. How do I get from that to a position where I can ask her out? \[…\]

Summary A \(NashMD\-PG\)OP \(21M\) is trying to rekindle a past connection with a 22F he knows from high school\. He’s lost some mojo and is unsure how to move from a passive messaging pattern to asking her out\.

Summary B \(Four\-Playerρ=0\.05\\rho\{=\}0\.05\)A 21\-year\-old man is trying to rekindle a connection with a 22\-year\-old woman he knew from high school, but after some miscommunications, she’s lost interest\. He’s seeking advice on how to get back in her good graces and ask her out\.

Nemotron\-3\-Nano\(verdict:A, 16\.9 s\)\[Helpfulness\]Summary A mentions the “passive messaging pattern”, capturing the specific current dynamic \(she only replies when prompted\)\. Summary B says “lost interest” but does not identify this nuance\. \[…\]Summary A captures the specific dynamic slightly better; final verdict A\.

Gemma\-4\-31B\(verdict:B, 74\.8 s\)\[Accuracy\]Summary B captures the cause\-effect relationship: bad decisions→\\tolost interest→\\toneeds advice\. Summary A’s “lost some mojo” reflects a side detail from the Fun Facts section rather than the primary driver of the conflict in the main text\. \[…\]\[Helpfulness\]Summary B provides a more complete narrative arc \(the attempt, the mistake, the result, the goal\), giving better context for advice\.

DeepSeek\-R1\-Distill\-Qwen\-32B\(verdict:B, 161\.7 s\)\[Accuracy\]Summary B includes specific reasons for her losing interest \(miscommunications / bad decisions\), which aligns with the post\. Summary A is vague, saying only he “lost some mojo”, which does not capture the cause of her disengagement\. \[…\]\[Helpfulness\]B explainswhyshe lost interest, providing context that would help readers offer more targeted advice\.

### Example 3 \(prompt\_id114\)

Disagreement:Nemotron\-3\-Nano→\\toB; Gemma\-4\-31B→\\toB; DeepSeek\-R1\-32B→\\toA

Post \(r/relationships\)Title:\[m24\] Arranged a date and it was postponed\. Am I wasting my time?A few weeks ago I met a girl, got her number and arranged to meet up\. The day before, she asked to postpone because of an exam\. I said sure; she then said she had exams for another couple of weeks\. After they finished, I texted her on Saturday night \[…\] and got the response “no but \[…\]” Am I wasting my time?

Summary A \(NashMD\-PG\)OP is confused and questioning if he’s wasting time after postponing a date with a girl who initially agreed to meet up but then said she’d be busy for a few more weeks\.

Summary B \(Four\-Playerρ=0\.05\\rho\{=\}0\.05\)The OP arranged a date with a girl but she kept postponing it due to exams, leading him to question if he wasted his time\. He’s confused by her lack of clear communication and is wondering if she’s not interested\.

Nemotron\-3\-Nano\(verdict:B, 12\.0 s\)\[Accuracy\]Summary A says she “initially agreed” and was “busy for a few more weeks”—neither is quite right\. She did not explicitly agree before the postponement, and the exams are now finished\. Summary B accurately states she postponeddue to examsand captures his uncertainty about her interest\. \[…\]\[Helpfulness\]B conveys both the postponement reason and his underlying concern \(is she interested?\), which is the real question driving the post\.

Gemma\-4\-31B\(verdict:B, 80\.1 s\)\[Accuracy\]Summary A says “after postponing a date”—grammatically ambiguous, implying OP did the postponing, whereas the girl was the one who postponed\. Summary B correctly identifies thatshekept postponing, and also captures his underlying concern about her interest level\. \[…\]\[Helpfulness\]B explicitly states the core of OP’s dilemma: whether she is actually interested\. Summary A misses this psychological dimension entirely\.

DeepSeek\-R1\-Distill\-Qwen\-32B\(verdict:A, 70\.8 s\)\[Accuracy\]Summary A captures the postponement and his confusion closely\. Summary B adds that he is confused by “a lack of clear communication”, which is an interpretation not explicitly stated in the post—the OP is confused about whether she is interested, not about communication clarity per se\. \[…\]\[Conciseness\]Summary A sticks strictly to the facts from the post\. Summary B introduces an extra interpretive layer which might be considered redundant\.

### Example 4 \(prompt\_id71\)

Disagreement:Nemotron\-3\-Nano→\\toB; Gemma\-4\-31B→\\toB; DeepSeek\-R1\-32B→\\toA

Post \(r/relationships\)Title:I \[F21\] feel frustrated with my boyfriend’s \[M24\] laziness\. Am I justified?I have a well\-paying part\-time job and am a full\-time student; he works full time but doesn’t make ends meet and I often cover his expenses\. He plays video games while I’m trying to work on important assignments\. When I asked him for help with housework he got mad, left without saying goodbye, and saidwe should just break up\[…\] Am I justified in feeling frustrated?

Summary A \(NashMD\-PG\)OP \(F21\) is frustrated with her boyfriend’s \(M24\) lack of help and appreciation, especially during her stressful school deadlines\. She feels unappreciated andwants to break upbecause he doesn’t contribute to household chores or offer support\.

Summary B \(Four\-Playerρ=0\.05\\rho\{=\}0\.05\)The poster is frustrated with her boyfriend’s lack of effort in contributing to household responsibilities and feeling unappreciated despite her hard work and stress\. She feels he’s only concerned with his own leisure activities and doesn’t offer support during her school stress\.

Nemotron\-3\-Nano\(verdict:B, 45\.7 s\)\[Accuracy\]Summary A says she “wants to break up”; however, it washewho suggested breaking up, not her\. She expressed frustration and felt underappreciated, but never stated she wants to end the relationship\. Summary A therefore introduces a claim not directly supported by the post\. \[…\]Summary B stays strictly within what is stated: frustration, lack of effort in household duties, feeling unappreciated, his focus on leisure\. Accuracy is the primary criterion; B is more faithful to the post\.

Gemma\-4\-31B\(verdict:B, 72\.2 s\)\[Accuracy\]Summary A contains a significant factual error\. It states that OP “wants to break up,” whereas the text specifies that it was theboyfriendwho said “we should just break up” during the argument\. Summary B is accurate and faithfully reflects the content of the post\. \[…\]Summary A’s inaccuracy regarding who suggested the break\-up makes it unreliable; B is the clear winner on accuracy\.

DeepSeek\-R1\-Distill\-Qwen\-32B\(verdict:A, 58\.6 s\)\[Accuracy\]Summary A mentions that she wants to break up—a specific, emotionally significant detail from the post\. Although it was he who voiced the suggestion, the context implies she may be considering it given her frustration; this is arguably a reasonable inference rather than a fabrication\. \[…\]\[Helpfulness\]Including the break\-up detail gives a clearer picture of the severity of her feelings, making Summary A more useful for understanding how serious the situation is\.