Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

# Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
Source: [https://arxiv.org/html/2607.26358](https://arxiv.org/html/2607.26358)
\\undefine@key

newfloatplacement\\undefine@keynewfloatname\\undefine@keynewfloatfileext\\undefine@keynewfloatwithin

Brian W\. LeeUniversity of California, BerkeleyIan Waudby\-SmithUniversity of California, Berkeley Philip AmortilaUniversity of California, BerkeleyNika HaghtalabUniversity of California, BerkeleyMichael I\. JordanUniversity of California, BerkeleyInria & École Normale Supérieure

###### Abstract

Reinforcement learning \(RL\) fine\-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy\. A standard way to balance this trade\-off is via a KL\-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient\. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward–retention trade\-offs\. We instead propose a game\-theoretic framework that gives this trade\-off an explicit statistical interpretation\. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy\. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL\-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability\. Drawing on classical results from concave\-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL\-regularized RL objective, thus allowing for flexible integration into standard fine\-tuning pipelines\. In experiments withQwen3\-8BandLlama\-3\.2\-1B, we demonstrate that our methods result in competitive reward\-retention trade\-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open\-source models\.

## 1Introduction

A common goal in large language model \(LLM\) fine\-tuning is to improve performance on a target objective while preserving useful behaviors inherited from a reference policy\. These objectives may be at odds with one another, and there are many possible ways to formalize the resulting trade\-off\. One of the most widely used approaches is Kullback–Leibler \(KL\)\-regularized reinforcement learning \(RL\), which augments the reward objective with a penalty for deviating from the reference policy\(jaques2017sequence;jaques2019way;neu2017unified;ziegler2019fine;stiennon2020learning;ouyang2022training\)\. In this formulation, the regularization coefficientβ≥0\\beta\\geq 0determines how aggressively the fine\-tuned policy is penalized for deviating from the reference policy \(e\.g\., a pre\-trained LLM\)\. Whenβ\\betais too small, fine\-tuning may increase reward at the cost of significantly changing the model’s behavior\. Whenβ\\betais too large, the model remains close to the reference policy, but may fail to adequately learn the target task\. In practice, this coefficient is often selected by manual tuning or via grid search\(see, e\.g\.,ouyang2022training;zhang2023wisdom;lin2024mitigating;tang2024understanding\)\. However this approach can waste compute and lead to undesirable reward–retention trade\-offs\. Some implementations instead adaptβ\\betaonline to match a prescribed KL target, but this still requires the learner to specify the target divergence and update heuristic in advance\(schulman2017proximal;ziegler2019fine\)\.

We consider an alternative notion of deviation from the reference policy\. Rather than measuring deviations directly through a distance or similarity metric, we ask how difficult it is to distinguish the fine\-tuned policy from the reference policy based on its outputs\. This naturally leads to a game between an agent who seeks to maximize reward, and a monitor attempting to detect deviations from the reference policy\. In this game, an agent deploys a policy to maximize their utility, while a monitor observes policy outputs as they are generated and aims to detect whether those outputs have been generated by the \(intended\) reference policy or by some other \(unintended\) one\. If the monitor detects a deviation from the reference policy, deployment is terminated\. The agent therefore faces a trade\-off between increasing reward and remaining statistically indistinguishable from the reference policy\. We call this the*sequential detection game*\.

### 1\.1Our Contributions

#### Game\-theoretic formulation for RL fine\-tuning \(Section[3](https://arxiv.org/html/2607.26358#S3)\)\.

We provide a new perspective on RL fine\-tuning through the lens of sequential detection\. Starting from a game\-theoretic formulation, we prove that the agent’s Nash equilibrium strategy in the sequential detection game is to fine\-tune the reference policy with the standard KL\-regularized RL objective\. Thus, rather than introducing a new fine\-tuning objective, our game\-theoretic formulation recovers one that is already used in practice, which allows existing RL fine\-tuning algorithms to be applied directly\. At the same time, it gives the KL penalty an operational interpretation as the statistical cost of remaining difficult to distinguish from the reference policy under sequential monitoring\. Moreover, the equilibrium identifies the regularization coefficientβ⋆\\beta^\{\\star\}that optimally trades off between reward maximization and statistical distinguishability\. Unlike conventional RL fine\-tuning, whereβ\\betais chosen heuristically, the equilibrium coefficient is determined solely by the reward function, prompt distribution, and reference policy\.

#### Learning the optimal regularization strength \(Section[4](https://arxiv.org/html/2607.26358#S4)\)\.

Building on our equilibrium analysis, we present a stochastic bisection algorithm \(Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\) that estimatesβ⋆\\beta^\{\\star\}toϵ\\epsilon\-precision by solvingO​\(log⁡\(1/ϵ\)\)O\(\\log\(1/\\epsilon\)\)RL sub\-problems\. Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)may be interpreted as a game\-theoretically principled approach to RL fine\-tuning: rather than relying on a separate hyperparameter search, it adaptively computes the equilibrium regularization strength\. The analysis of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)combines classical techniques from the literature on fractional programming\(dinkelbach1967nonlinear;schaible1976fractional\)and sequential hypothesis testing\(robbins1974expected\)\.

#### Continual learning and model auditing experiments \(Section[5](https://arxiv.org/html/2607.26358#S5)\)\.

We evaluate our framework in both continual learning and model auditing settings usingQwen3\-8B\(yang2025qwen3\)andLlama\-3\.2\-1B\(meta2024llama321b\)\. In continual learning, we find preliminary evidence that the equilibrium regularization coefficient yields better reward–retention trade\-offs than compute\-matched manual tuning\. In model auditing, we show that the monitor’s equilibrium\-prescribed test detects hidden model modifications by open\-source API providers more quickly than natural baselines while controlling the type\-I error rate\.

## 2Preliminaries

We now review the key ingredients from RL fine\-tuning, sequential hypothesis testing, and game theory that will later be combined in our analysis\.

#### RL fine\-tuning\.

Consider a prompt space𝒳\\mathcal\{X\}and response space𝒴\\mathcal\{Y\}\. A policyπ\\piis a mapping from prompts to distributions over responses, i\.e\.,π:𝒳→Δ​\(𝒴\)\\pi:\\mathcal\{X\}\\rightarrow\\Delta\(\\mathcal\{Y\}\)\. We useπ​\(y\|x\)\\pi\(y\|x\)to denote the probability of receiving responseyygiven promptxx, andy∼π\(⋅\|x\)y\\sim\\pi\(\\cdot\|x\)to denote a sample from the conditional distribution over responses givenxx\.111LLMs generate tokens auto\-regressively, but for our purposes it suffices to consider only the induced distribution over complete responses\.

Given a reference policyπref\\pi\_\{\\mathrm\{ref\}\}, a reward functionr:𝒳×𝒴→ℝr:\\mathcal\{X\}\\times\\mathcal\{Y\}\\rightarrow\\mathbb\{R\}, the KL\-regularized RL fine\-tuning objective for a regularization parameterβ≥0\\beta\\geq 0is defined by

maxπ\{𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]−β⋅𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\},\\max\_\{\\pi\}\\left\\\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\cdot\\mathbb\{E\}\\left\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\\right\]\\right\\\},\(1\)whereKL\(π\(⋅\|x\),πref\(⋅\|x\)\):=∑y∈𝒴π\(y\|x\)logπ​\(y\|x\)πref​\(y\|x\)\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\):=\\sum\_\{y\\in\\mathcal\{Y\}\}\\pi\(y\|x\)\\log\\frac\{\\pi\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}is the KL divergence betweenπ\(⋅\|x\)\\pi\(\\cdot\|x\)andπref\(⋅\|x\)\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\), and expectations are \(also\) taken over the prompt distribution𝒟\\mathcal\{D\}\. This optimization admits a closed\-form optimal solution\(donsker1975variational;rafailov2023direct\)of the form

πβ​\(y\|x\):=πref​\(y\|x\)​exp⁡\(r​\(x,y\)/β\)Zβ​\(x\),\\pi\_\{\\beta\}\(y\|x\):=\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\\frac\{\\exp\(r\(x,y\)/\\beta\)\}\{Z\_\{\\beta\}\(x\)\},\(2\)whereZβ​\(x\):=𝔼y∼πref\(⋅\|x\)​\[exp⁡\(r​\(x,y\)/β\)\]Z\_\{\\beta\}\(x\):=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[\\exp\(r\(x,y\)/\\beta\)\]is the partition function\. We callπβ\\pi\_\{\\beta\}the tilt ofπref\\pi\_\{\\mathrm\{ref\}\}\. SinceZβ​\(x\)Z\_\{\\beta\}\(x\)is generally intractable to compute due to the size of𝒴\\mathcal\{Y\}, RL methods like Proximal Policy Optimization\(schulman2017proximal\)and Group Relative Policy Optimization\(shao2024deepseekmath\)are typically used to approximately solve Equation \([1](https://arxiv.org/html/2607.26358#S2.E1)\)\.

#### Sequential hypothesis testing\.

Sequential hypothesis testing is a paradigm of statistical inference for which type\-I errors \(false positive rates\) are controlled not only at pre\-determined sample sizes, but also at stopping times\(wald1945sequential;wald1947sequential\)\. Informally, sequential hypothesis tests allow analysts to “peek” at their data routinely to adaptively stop experiments and make conclusions for data\-dependent reasons\.

In the simplest case of simple nulls and simple alternatives, one sees a sequence of observationsz1,z2,…z\_\{1\},z\_\{2\},\\ldotsfrom a distributionPzP\_\{z\}and the goal is to determine whetherPz=PP\_\{z\}=P\(the null\) orPz=QP\_\{z\}=Q\(the alternative\)\. A sequential hypothesis test,ϕ≡\(ϕt\)t∈ℕ\\phi\\equiv\(\\phi\_\{t\}\)\_\{t\\in\\mathbb\{N\}\}, is a sequence of functionsϕt≡ϕ​\(z1,…,zt\)\\phi\_\{t\}\\equiv\\phi\(z\_\{1\},\\dots,z\_\{t\}\)outputting either “reject” or “do not reject” at each time stept∈ℕt\\in\\mathbb\{N\}\. For a fixedα∈\(0,1\)\\alpha\\in\(0,1\), a test is said to be*α\\alpha\-correct*if it controls the type\-I error at all sample sizes simultaneously; i\.e\.,P\(∃t∈ℕ:ϕtrejects\)≤αP\(\\exists\\;t\\in\\mathbb\{N\}:\\phi\_\{t\}\\text\{ rejects\}\)\\leq\\alpha\. Lettingτα≡τα​\(ϕ\)=inf\{t∈ℕ:ϕt​rejects\}\\tau\_\{\\alpha\}\\equiv\\tau\_\{\\alpha\}\(\\phi\)=\\inf\\\{t\\in\\mathbb\{N\}:\\phi\_\{t\}\\text\{ rejects\}\\\}be the*stopping time*, notice thatα\\alpha\-correctness is equivalent to the conditionP​\(τα<∞\)≤αP\(\\tau\_\{\\alpha\}<\\infty\)\\leq\\alpha\. A test is said to be*power\-one*ifQ​\(τ<∞\)=1Q\(\\tau<\\infty\)=1\.

It is a classical result that for anyα\\alpha\-correct, power\-one sequential test,𝔼Q​\[τα\]≥log⁡\(1/α\)KL​\(Q,P\)\\mathbb\{E\}\_\{Q\}\[\\tau\_\{\\alpha\}\]\\geq\\frac\{\\log\(1/\\alpha\)\}\{\\mathrm\{KL\}\(Q,P\)\}; seewald1945sequentialandrobbins1974expected\. This lower bound is known to be tight in the high\-confidence regime, in the sense that there exist validα\\alpha\-correct, power\-one tests for which

limα↓0𝔼Q​\[τα\]log⁡\(1/α\)=1KL​\(Q,P\)\.\\lim\_\{\\alpha\\downarrow 0\}\\frac\{\\mathbb\{E\}\_\{Q\}\[\\tau\_\{\\alpha\}\]\}\{\\log\(1/\\alpha\)\}=\\frac\{1\}\{\\mathrm\{KL\}\(Q,P\)\}\.\(3\)A canonical test with this leading\-order behavior is the sequential probability ratio test \(SPRT\)\(wald1945sequential\)given by

ϕt=𝟙​\{∏i=1td​Q​\(zi\)d​P​\(zi\)≥1α\}\.\\phi\_\{t\}=\\mathbbm\{1\}\\left\\\{\\prod\_\{i=1\}^\{t\}\\frac\{dQ\(z\_\{i\}\)\}\{dP\(z\_\{i\}\)\}\\geq\\frac\{1\}\{\\alpha\}\\right\\\}\.

#### Nash equilibria\.

A two\-player game consists of strategy spaces𝒮1\\mathcal\{S\}\_\{1\},𝒮2\\mathcal\{S\}\_\{2\}and utility functionsu1,u2:𝒮1×𝒮2→ℝu\_\{1\},u\_\{2\}:\\mathcal\{S\}\_\{1\}\\times\\mathcal\{S\}\_\{2\}\\rightarrow\\mathbb\{R\}\. Given a strategys2∈𝒮2s\_\{2\}\\in\\mathcal\{S\}\_\{2\}for player 2, a strategys1∈𝒮1s\_\{1\}\\in\\mathcal\{S\}\_\{1\}is a*best response*for player 1 ifs1∈arg⁡maxs1′∈𝒮1⁡u1​\(s1′,s2\)\.s\_\{1\}\\in\\arg\\max\_\{s\_\{1\}^\{\\prime\}\\in\\mathcal\{S\}\_\{1\}\}u\_\{1\}\(s\_\{1\}^\{\\prime\},s\_\{2\}\)\.Best responses for player 2 are defined analogously\.

A strategy profile\(s1∗,s2∗\)\(s\_\{1\}^\{\*\},s\_\{2\}^\{\*\}\)is a*Nash equilibrium*if both players’ strategies are simultaneously best responses:

u1​\(s1∗,s2∗\)≥u1​\(s1,s2∗\)​for all​s1∈𝒮1​and​u2​\(s1∗,s2∗\)≥u2​\(s1∗,s2\)​for all​s2∈𝒮2\.u\_\{1\}\(s\_\{1\}^\{\*\},s\_\{2\}^\{\*\}\)\\geq u\_\{1\}\(s\_\{1\},s\_\{2\}^\{\*\}\)\\;\\;\\text\{for all \}s\_\{1\}\\in\\mathcal\{S\}\_\{1\}\\quad\\text\{and \}\\quad u\_\{2\}\(s\_\{1\}^\{\*\},s\_\{2\}^\{\*\}\)\\geq u\_\{2\}\(s\_\{1\}^\{\*\},s\_\{2\}\)\\;\\;\\text\{for all \}s\_\{2\}\\in\\mathcal\{S\}\_\{2\}\.Thus, at a Nash equilibrium, neither player can improve their utility by unilaterally changing their strategy; i\.e\., each player’s behavior is optimal given the behavior of the other\.

## 3The Sequential Detection Game

We explore differences between reference policies and target policies from the perspective of a monitor aiming to quickly distinguish one from the other in a two\-player \(general sum\) game against a strategic agent\.

Sequential detection game:Consider a setup where a monitor intends to deploy a policyπref\\pi\_\{\\mathrm\{ref\}\}while an agent potentially deploys a policyπ~≠πref\\tilde\{\\pi\}\\neq\\pi\_\{\\mathrm\{ref\}\}in place ofπref\\pi\_\{\\mathrm\{ref\}\}\. The monitor intends to detect such a change from prompt\-response outputs while controlling the type\-I error rate at a desired levelα∈\(0,1\)\\alpha\\in\(0,1\)\.Concretely, for each time stept∈ℕt\\in\\mathbb\{N\}:1\.A promptxt∼𝒟x\_\{t\}\\sim\\mathcal\{D\}is generated from the prompt distribution𝒟\\mathcal\{D\}and a responseyt∼π~\(⋅∣xt\)y\_\{t\}\\sim\\tilde\{\\pi\}\(\\cdot\\mid x\_\{t\}\)is generated from the agent’s deployed policy\.2\.The monitor updates a sequential testϕt≡ϕ​\(\(x1,y1\),…,\(xt,yt\)\)\\phi\_\{t\}\\equiv\\phi\(\(x\_\{1\},y\_\{1\}\),\\dots,\(x\_\{t\},y\_\{t\}\)\)subject to the type\-I error constraint:ℙy∼πref\(∃t∈ℕ:ϕtrejects\)≤α\\mathbb\{P\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\}\\left\(\\exists\\;t\\in\\mathbb\{N\}:\\phi\_\{t\}\\text\{ rejects\}\\right\)\\leq\\alpha\.3\.Ifϕt\\phi\_\{t\}rejects, then the monitor stops deployment, and the agent receives utility∑i=1tr​\(xi,yi\)\\sum\_\{i=1\}^\{t\}r\(x\_\{i\},y\_\{i\}\)according to reward functionr:𝒳×𝒴→ℝr:\\mathcal\{X\}\\times\\mathcal\{Y\}\\rightarrow\\mathbb\{R\}, while the monitor receives utility−t\-t\.

In this sequential detection game, the monitor stops deployment if and only ifϕt\\phi\_\{t\}rejects\. In other words, their stopping rule is characterized by the stopping timeτα=inf\{t∈ℕ:ϕt​rejects\}\\tau\_\{\\alpha\}=\\inf\\\{t\\in\\mathbb\{N\}:\\phi\_\{t\}\\text\{ rejects\}\\\}, with the convention thatτα=∞\\tau\_\{\\alpha\}=\\inftyifϕt\\phi\_\{t\}never rejects\. From this vantage point, the agent wishes to deploy a policyπ~\\tilde\{\\pi\}to maximize their expected cumulative reward before detection,𝔼y∼π~\(⋅∣x\)​\[∑i=1ταr​\(xi,yi\)\]\\mathbb\{E\}\_\{y\\sim\\tilde\{\\pi\}\(\\cdot\\mid x\)\}\[\\sum\_\{i=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{i\},y\_\{i\}\)\], while the monitor wishes to minimize𝔼y∼π~\(⋅∣x\)​\[τα\]\\mathbb\{E\}\_\{y\\sim\\tilde\{\\pi\}\(\\cdot\\mid x\)\}\[\\tau\_\{\\alpha\}\]\.

The agent’s optimal choice ofπ~\\tilde\{\\pi\}turns out to be precisely a KL\-regularized tilt of the reference policy, while the monitor’s best response turns out to be a likelihood ratio test betweenπref\\pi\_\{\\mathrm\{ref\}\}andπ~\\tilde\{\\pi\}\. To state these facts formally, we first require two assumptions on the agent’s reward functionrr\.

###### Assumption 3\.1\(The agent can and must strategize to benefit\)\.

It is possible for the agent to achieve positive expected utility\. That is, there exists a policyπ\\pisuch that𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\>0\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\>0\. Moreover, they must strategize in order to do so, i\.e\.,μref:=𝔼y∼πref\(⋅\|x\)​\[r​\(x,y\)\]≤0\\mu\_\{\\mathrm\{ref\}\}:=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\\leq 0\.

###### Assumption 3\.2\(The monitor can always devise a detection strategy\)\.

For everyπ′≠πref\\pi^\{\\prime\}\\neq\\pi\_\{\\mathrm\{ref\}\}, there exists a test\(ϕt′\)t∈ℕ\(\\phi\_\{t\}^\{\\prime\}\)\_\{t\\in\\mathbb\{N\}\}with a finite expected stopping time, i\.e\.,𝔼π′​\[τα​\(ϕ′\)\]<∞\\mathbb\{E\}\_\{\\pi^\{\\prime\}\}\[\\tau\_\{\\alpha\}\(\\phi^\{\\prime\}\)\]<\\infty\.

Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)provides sufficient conditions for the agent to want to participate in the game and to learn a non\-trivial strategy\.222If the latter half of[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)does not hold, the agent can deploy policyπref\\pi\_\{\\mathrm\{ref\}\}and collect infinite reward\.For the necessity of strategization in Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1), it is mathematically equivalent to assume thatμref−c≤0\\mu\_\{\\mathrm\{ref\}\}\-c\\leq 0, wherec≥0c\\geq 0is the minimum improvement in expected reward required for the agent to want to update the reference policy\. The constantccmay also be interpreted as capturing the cost associated with model training \(e\.g\., monetary costs, time spent, effort exerted, etc\.\)\. Assumption[3\.2](https://arxiv.org/html/2607.26358#S3.Thmtheorem2)is an analogous assumption on the monitor’s incentive to play the game\. It is a weak assumption that is satisfied for likelihood ratios in all but pathological cases which we eschew for the purposes of this paper\.

In what follows, we characterize the equilibrium of the sequential detection game\. These statements should be interpreted as holding in the so\-called “high\-confidence” regime, i\.e\., whereα↓0\\alpha\\downarrow 0\. This is because the sequential hypothesis tests we consider take place in discrete time: they observe one \(prompt, response\) pair at a time and can only stop after an integer number of observations\.robbins1974expectedprovide an information\-theoretic lower boundlog⁡\(1/α\)/KL​\(Q,P\)\\log\(1/\\alpha\)/\\mathrm\{KL\}\(Q,P\)for the expected stopping time of anyα\\alpha\-correct sequential test\. This bound is tight at the leadinglog⁡\(1/α\)\\log\(1/\\alpha\)order, but the corresponding upper bounds can include lower\-order boundary\-crossing effects as an artifact of the discrete\-time nature of the test\(siegmund1985sequential\)\. Therefore the following equilibrium characterization can be thought of as stating that unilateral deviations cannot improve either player’s utility*at the leadinglog⁡\(1/α\)\\log\(1/\\alpha\)order*\.

#### Equilibrium characterization\.

We begin with the monitor’s equilibrium strategy\. Suppose the agent is playing policyπ~\\tilde\{\\pi\}\. Targeting the true agent policyπ~\\tilde\{\\pi\}maximizes the expected evidence accumulated per sample and therefore minimizes the expected stopping time\. Moreover, it is known that SPRTs exhibit the optimal leading order behavior of Equation[3](https://arxiv.org/html/2607.26358#S2.E3)\. Therefore the monitor’s best\-response to agent policyπ~\\tilde\{\\pi\}is to test for it using an SPRT\.

The agent’s goal is to pick their policy to maximize𝔼​\[∑t=1ταr​\(xt,yt\)\]\\mathbb\{E\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\], which can be written as𝔼​\[τα\]⋅𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\mathbb\{E\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]by Wald’s equation, since the expected stopping time will be finite under Assumptions[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)and[3\.2](https://arxiv.org/html/2607.26358#S3.Thmtheorem2)\. By therobbins1974expectedlower bound, the agent can guarantee themselves utility at leastlog\(1/α\)𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]/𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\log\(1/\\alpha\)\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]/\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\], which is also an \(asymptotic\) upper bound on their utility under the monitor’s best\-response SPRT\. Since neither𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]nor𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]depend onα\\alpha, the agent’s optimization takes the form

maxπ⁡𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\],\\max\_\{\\pi\}\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\}\{\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\},\(4\)where we use the convention that0/0=00/0=0\. Equation \([4](https://arxiv.org/html/2607.26358#S3.E4)\) is a fractional program, with a concave numerator and a convex, strictly positive denominator \(excluding the singular pointπ=πref\\pi=\\pi\_\{\\mathrm\{ref\}\}, which cannot be optimal under Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)\)\. As such, we can draw on the rich literature on concave\-convex fractional programming\(see, e\.g\.,dinkelbach1967nonlinear;schaible1976fractional\), which shows that while such fractional optimization problems are not concave, they are quasi\-concave and can be solved iteratively through a sequence of concave optimization problems\.

###### Theorem 3\.3\.

Suppose that for each iterationkk, the quantitiesπ\(k\)\\pi^\{\(k\)\}andβ\(k\)\\beta^\{\(k\)\}are chosen byπ\(k\)∈argmaxπ\{𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]−β\(k\)𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\}\\pi^\{\(k\)\}\\in\\operatorname\*\{argmax\}\_\{\\pi\}\\left\\\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta^\{\(k\)\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\\right\\\}andβ\(k\+1\)←𝔼y∼π\(k\)\(⋅\|x\)​\[r​\(x,y\)\]𝔼\[KL\(π\(k\)\(⋅\|x\),πref\(⋅\|x\)\)\]\\beta^\{\(k\+1\)\}\\leftarrow\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi^\{\(k\)\}\(\\cdot\|x\)\}\[r\(x,y\)\]\}\{\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi^\{\(k\)\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\}\. Then it holds thatβ\(k\)→β⋆\\beta^\{\(k\)\}\\to\\beta^\{\\star\}, where

πβ⋆∈argmaxπ\{𝔼y∼π\(⋅∣x\)\[r\(x,y\)\]−β⋆𝔼\[KL\(π\(⋅∣x\),πref\(⋅∣x\)\]\}\\pi\_\{\\beta^\{\\star\}\}\\in\\operatorname\*\{argmax\}\_\{\\pi\}\\left\\\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid x\)\}\[r\(x,y\)\]\-\\beta^\{\\star\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\\mid x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\]\\right\\\}andπβ⋆\\pi\_\{\\beta^\{\\star\}\}solves the fractional program \([4](https://arxiv.org/html/2607.26358#S3.E4)\)\.

Notice thatπβ⋆\\pi\_\{\\beta^\{\\star\}\}is precisely the optimizer of the RL fine\-tuning objective in Equation \([1](https://arxiv.org/html/2607.26358#S2.E1)\) with regularization coefficientβ⋆\\beta^\{\\star\}\. In other words, the optimal policy for the agent is the reference policyπref\\pi\_\{\\mathrm\{ref\}\}tilted byβ⋆\\beta^\{\\star\}as in \([2](https://arxiv.org/html/2607.26358#S2.E2)\)\.

#### Discussion\.

Taken together, the results in this section imply that the agent playing policyπβ⋆\\pi\_\{\\beta^\{\\star\}\}and the monitor testing for policyπβ⋆\\pi\_\{\\beta^\{\\star\}\}is a Nash equilibrium in the sequential detection game\. The equilibrium characterization yields a game\-theoretic derivation for KL\-regularized RL, and provides an operational interpretation of the optimal regularization coefficient as the one which maximizes reward per unit of statistical distinguishability from the reference policy\. In this sense, the KL penalty is not just a convenient proxy for behavioral preservation, but the “shadow price” of remaining difficult to detect under sequential monitoring\.

This equilibrium characterization is also robust to tie\-breaking\. In Appendix[A\.1](https://arxiv.org/html/2607.26358#A1.SS1), we show that whileπβ⋆\\pi\_\{\\beta^\{\\star\}\}is not a unique agent best\-response, the monitor’s SPRT withℋ1:π=πβ⋆\\mathcal\{H\}\_\{1\}:\\pi=\\pi\_\{\\beta^\{\\star\}\}is still anα\\alpha\-correct, power\-one test for*any*agent best response\. Therefore, the monitor’s test remains valid even if the agent breaks ties in an unknown or arbitrary way\.

## 4Reduction to KL\-Regularized RL

We now turn our attention from characterizing the equilibrium policyπβ⋆\\pi\_\{\\beta^\{\\star\}\}to learning it\. Traditional fractional programming approaches cannot be applied off\-the\-shelf, as they rely on the ability to evaluate quantities like𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]and𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]exactly for an arbitrary policyπ\\pi, which is generally not feasible whenπ\\piis an LLM\. Instead, we show thatβ⋆\\beta^\{\\star\}can be characterized as the unique root of a one\-dimensional function and estimate it from data using a stochastic bisection procedure, in the spirit of stochastic root\-finding methods\(robbins1951stochastic\)\.

Consider the functionM\(β\):=𝔼y∼πβ\(⋅\|x\)\[r\(x,y\)\]−β⋅𝔼\[KL\(πβ\(⋅\|x\),πref\(⋅\|x\)\)\]\.M\(\\beta\):=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\cdot\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\_\{\\beta\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\.Intuitively,M​\(β\)M\(\\beta\)measures whether the reward that is obtained by deviating from the reference policy exceeds the statistical cost imposed by the KL penalty for the givenβ\\beta\. Whenβ\\betais too small, the reward dominates andM​\(β\)\>0M\(\\beta\)\>0\. Whenβ\\betais too large, the KL penalty dominates andM​\(β\)<0M\(\\beta\)<0\. The equilibrium coefficientβ⋆\\beta^\{\\star\}is precisely the point where these forces balance\. Therefore, computingβ⋆\\beta^\{\\star\}reduces to finding the unique root ofM​\(β\)M\(\\beta\)\. This justification follows from an analysis ofdinkelbach1967nonlinear\(see Lemma[A\.1](https://arxiv.org/html/2607.26358#A1.Thmtheorem1)\)\.

SinceM​\(β\)M\(\\beta\)is not available in closed form, we use𝔼^\\widehat\{\\mathbb\{E\}\}to denote the sample average and instead consider the empirical estimate ofM​\(β\)M\(\\beta\)usingn≥1n\\geq 1samples:

M^n\(β\):=𝔼^y∼πβ\(⋅\|x\)\[r\(x,y\)\]−β⋅𝔼^\[KL\(πβ\(⋅\|x\),πref\(⋅\|x\)\)\]\.\\widehat\{M\}\_\{n\}\(\\beta\):=\\widehat\{\\mathbb\{E\}\}\_\{y\\sim\\pi\_\{\\beta\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\cdot\\widehat\{\\mathbb\{E\}\}\[\\mathrm\{KL\}\(\\pi\_\{\\beta\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\.Our algorithm proceeds as follows: Setβlow=0\\beta\_\{\\mathrm\{low\}\}=0\. Given an upper boundβhigh≥β⋆\\beta\_\{\\mathrm\{high\}\}\\geq\\beta^\{\\star\}, setβ^:=0\.5​\(βhigh\+βlow\)\\widehat\{\\beta\}:=0\.5\(\\beta\_\{\\mathrm\{high\}\}\+\\beta\_\{\\mathrm\{low\}\}\)and collect enough samplesnnsuch thatM^n​\(β^\)±rad​\(β^,n,δ\)\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)\\pm\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)is bounded away from zero, whererad​\(β^,n,δ\)\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)is a confidence radius that holds with probability1−δ1\-\\delta\. Depending on the sign ofM^n​\(β^\)\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\), set eitherβhigh←β^\\beta\_\{\\mathrm\{high\}\}\\leftarrow\\widehat\{\\beta\}orβlow←β^\\beta\_\{\\mathrm\{low\}\}\\leftarrow\\widehat\{\\beta\}, and repeat this process until\|βhigh−βlow\|\|\\beta\_\{\\mathrm\{high\}\}\-\\beta\_\{\\mathrm\{low\}\}\|is within the desired precisionϵ\\epsilon\. The full procedure is outlined in Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\.

Input:

ϵ\>0\\epsilon\>0,

δ∈\(0,1\)\\delta\\in\(0,1\),

βhigh∈ℝ\+\\beta\_\{\\mathrm\{high\}\}\\in\\mathbb\{R\}\_\{\+\}
Set

βlow=0\\beta\_\{\\mathrm\{low\}\}=0
while*βhigh−βlow\>ϵ\\beta\_\{\\mathrm\{high\}\}\-\\beta\_\{\\mathrm\{low\}\}\>\\epsilon*do

β^=12​\(βhigh\+βlow\)\\widehat\{\\beta\}=\\frac\{1\}\{2\}\(\\beta\_\{\\mathrm\{high\}\}\+\\beta\_\{\\mathrm\{low\}\}\),

n=100n=100
πβ^←RL\-Oracle​\(πref,β^\)\\pi\_\{\\widehat\{\\beta\}\}\\leftarrow\\textup\{\{RL\-Oracle\}\}\(\\pi\_\{\\mathrm\{ref\}\},\\widehat\{\\beta\}\)
while*True\\mathrm\{True\}*do

if*M^n​\(β^\)\+rad​\(β^,n,δ\)<0\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)\+\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)<0*then

βhigh←β^\\beta\_\{\\mathrm\{high\}\}\\leftarrow\\widehat\{\\beta\}; break

else if*M^n​\(β^\)−rad​\(β^,n,δ\)\>0\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)\-\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)\>0*then

βlow←β^\\beta\_\{\\mathrm\{low\}\}\\leftarrow\\widehat\{\\beta\}; break

else

n←2​nn\\leftarrow 2n
end if

end while

end while

return

RL\-Oracle​\(πref,βlow\)\\textup\{\{RL\-Oracle\}\}\(\\pi\_\{\\mathrm\{ref\}\},\\beta\_\{\\mathrm\{low\}\}\)

ALGORITHM 1Stochastic bisection method###### Theorem 4\.1\.

Ifβhigh≥β⋆\\beta\_\{\\mathrm\{high\}\}\\geq\\beta^\{\\star\}and for everyβ\\betaqueried by Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1),

ℙ\(∃n≥1:\|M^n\(β\)−M\(β\)\|\>rad\(β,n,δ\)\)≤δ,\\mathbb\{P\}\\left\(\\exists\\;n\\geq 1:\\left\|\\widehat\{M\}\_\{n\}\(\\beta\)\-M\(\\beta\)\\right\|\>\\mathrm\{rad\}\(\\beta,n,\\delta\)\\right\)\\leq\\delta,then with probability at least1−δ​log2⁡\(βhigh/ϵ\)1\-\\delta\\log\_\{2\}\\left\(\\beta\_\{\\mathrm\{high\}\}/\\epsilon\\right\), Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)returns a policyπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}satisfying0≤β⋆−βlow≤ϵ0\\leq\\beta^\{\\star\}\-\\beta\_\{\\mathrm\{low\}\}\\leq\\epsiloninlog2⁡\(βhigh/ϵ\)\\log\_\{2\}\\left\(\\beta\_\{\\mathrm\{high\}\}/\\epsilon\\right\)bisection iterations\.

We instantiate Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)with specific choices of\(βhigh,rad​\(⋅,⋅,⋅\)\)\(\\beta\_\{\\mathrm\{high\}\},\\mathrm\{rad\}\(\\cdot,\\cdot,\\cdot\)\)in the Appendix \(Corollaries[B\.5](https://arxiv.org/html/2607.26358#A2.Thmtheorem5)and[B\.7](https://arxiv.org/html/2607.26358#A2.Thmtheorem7)\)\. In either instantiation, Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)needsn=O​\(M​\(β^\)−2\)n=O\(M\(\\widehat\{\\beta\}\)^\{\-2\}\)samples in each iteration in order to determine the sign ofM​\(β^\)M\(\\widehat\{\\beta\}\)with high probability\. Therefore, asβ^→β⋆\\widehat\{\\beta\}\\rightarrow\\beta^\{\\star\}and\|M​\(β^\)\|→0\|M\(\\widehat\{\\beta\}\)\|\\rightarrow 0, we should expect the number of samples we need in each iteration of stochastic bisection to increase\. As a practical relaxation, we forego the confidence sequence in our experiments and bisect based on the sign ofM^n​\(β^\)\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)after observing a sufficiently large number of samples \(e\.g\.,n=4096n=4096\)\.

Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)posits access to an “RL oracle” which, given a reference policyπref\\pi\_\{\\mathrm\{ref\}\}and regularization parameterβ\\beta, returns the KL\-regularized tiltπβ\\pi\_\{\\beta\}\. While exact, polynomial\-time oracles generally do not exist due to the difficulty of computing the partition function, there is a rich literature of approximate oracles that have been designed to solve the KL\-regularized RL problem \(e\.g\., PPO, GRPO, and variants thereof\)\. Consequently, our theoretical guarantees for Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)should be interpreted as characterizing the complexity of identifying the equilibrium coefficient, assuming that each RL subproblem can be solved sufficiently accurately\.

#### Warm\-start version of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\.

It is possible to warm start Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)given an arbitrary setting ofβhigh\\beta\_\{\\mathrm\{high\}\}\(i\.e\., one that is not guaranteed to satisfyβhigh\>β⋆\\beta\_\{\\mathrm\{high\}\}\>\\beta^\{\\star\}\) as follows: ComputeM^n​\(βhigh\)\+rad​\(βhigh,n,δ\)\\widehat\{M\}\_\{n\}\(\\beta\_\{\\mathrm\{high\}\}\)\+\\mathrm\{rad\}\(\\beta\_\{\\mathrm\{high\}\},n,\\delta\)\. If it is bounded below zero, run Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)as normal\. Otherwise setβlow←βhigh\\beta\_\{\\mathrm\{low\}\}\\leftarrow\\beta\_\{\\mathrm\{high\}\},βhigh←2​βhigh\\beta\_\{\\mathrm\{high\}\}\\leftarrow 2\\beta\_\{\\mathrm\{high\}\}, and repeat\. This procedure will produce anε\\varepsilon\-approximate solution with high probability\. Furthermore, it makeslog2⁡\(βhigh/ϵ\)\\log\_\{2\}\(\\beta\_\{\\mathrm\{high\}\}/\\epsilon\)calls to the RL oracle ifβhigh\\beta\_\{\\mathrm\{high\}\}is a valid upper bound onβ⋆\\beta^\{\\star\}\. If it is not, the number of oracle calls is stillO​\(log2⁡\(σ2/\(\|μref\|​ϵ\)\)\)O\\left\(\\log\_\{2\}\\left\(\\sigma^\{2\}/\(\|\\mu\_\{\\mathrm\{ref\}\}\|\\epsilon\)\\right\)\\right\)if rewards areσ\\sigma\-sub\-Gaussian by Lemma[B\.3](https://arxiv.org/html/2607.26358#A2.Thmtheorem3)\.

Finally, it is worth noting that in our game formulation, even if the agent and the monitor run separate copies of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1), the agent and monitor utilities will still be close to their equilibrium values\. See Appendix[B\.1](https://arxiv.org/html/2607.26358#A2.SS1)for more details\.

## 5Experiments

We evaluate our methods empirically in two settings: a continual learning task and a model auditing task\. In the continual learning task, the goal is to improve performance on a new target reward while preserving existing behavioral properties of the reference policy\. In the model auditing task, an auditor tries to detect whether a model has been strategically fine\-tuned from its outputs\. Taken together, these experiments test the two main operational interpretations of our framework: stochastic bisection as a replacement for manual KL coefficient search, and the equilibrium likelihood\-ratio test as a practical auditing rule\.

#### Experiment setup\.

The two sets of experiments use the same setup\. In each round, the language model is given the same prompt \(“Once upon a time”\) and generates a completion, with the agent receiving reward equal to the number of characters in the response\.

Before bisection, the reward is calibrated by sampling completions from the reference policy and subtracting off the raw mean reward, plus a fixed marginρ\\rho\. Under the reference policy, the calibrated reward therefore has mean approximately equal to−ρ\-\\rho\. The calibrated reward is used throughout KL\-constrained fine\-tuning and bisection; because adding a constant to the reward does not change the optimal policy at any fixedβ\\beta, this shift does not alter the fine\-tuning objective\. Evaluation metrics on Pareto plots are reported in the original, uncalibrated units\.

### 5\.1Continual Learning

![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/qwen_mini.png)Figure 1:Reward–retention trade\-offs forQwen3\-8Bwith shiftρ=0\.1\\rho=0\.1\. The vertical axis reports response length in tokens, and the horizontal axes report narrative and grammatical coherence, as judged byGPT\-5\-mini\(singh2025openai\)\. Blue circles show policies trained on the compute\-matched exponential grid overβ\\beta, the gray square denotes the reference policy; and the orange circle and green diamond denote two independent stochastic\-bisection runs\. Each point is the average of400400independent responses\. Higher values are better on both axes\.![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p1_mini_cropped.png)\(a\)Llama\-3\.2\-1Bwith shiftρ=0\.1\\rho=0\.1\.
![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p2_mini_cropped.png)\(b\)Llama\-3\.2\-1Bwith shiftρ=0\.2\\rho=0\.2\.

Figure 2:Reward–retention trade\-offs forLlama\-3\.2\-1Bwith shiftρ=0\.1\\rho=0\.1\(left\) andρ=0\.2\\rho=0\.2\(right\)\. Plotting conventions, axis labels, and plot titles are the same as in Figure[1](https://arxiv.org/html/2607.26358#S5.F1)\. In both settings, Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)selects policies near the elbow of the empirical trade\-off curve traced out by the compute\-matchedβ\\betagrid\.In our continual learning setup, the response\-length reward function plays the role of the skill to be improved, while the reference policy’s ability to produce coherent responses is the behavior to be retained\. Although this is a single model update rather than a multi\-task setting, it isolates the central continual learning trade\-off: aggressively optimizing a new objective can alter or degrade useful behaviors inherited from the reference policy\.333This is commonly observed in the literature on RLHF and is often referred to as the “alignment tax”\(e\.g\.,ouyang2022training;lin2024mitigating;zhang2024cppo;jang2024degeneration\)\.For each trained policy, we measure the target behavior using token count and retention using both narrative and grammatical coherence\.444We usedGPT\-5\-miniandGPT\-5\-nanoas judges to evaluate narrative and grammatical coherence\(zheng2023judging\)\. All main body plots useGPT\-5\-mini\. Results usingGPT\-5\-nanoare very similar and are in Appendix[C](https://arxiv.org/html/2607.26358#A3)\.

We compare Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)against the standard practice of manually sweeping over the regularization coefficient\. This baseline trains one policy for each value in an exponentially spaced grid overβ\\beta, using the same number of fine\-tuning runs as Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\. After inspecting the results, a practitioner could select whichever grid point best matches their desired balance between length and coherence\.

Our results are summarized in Figures[1](https://arxiv.org/html/2607.26358#S5.F1)and[2](https://arxiv.org/html/2607.26358#S5.F2)\. Higher values are desirable on both axes, so policies toward the upper\-right corner of each panel achieve more favorable reward–retention trade\-offs\. The points labeledAgentandMonitorare produced by two independent runs of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\. We retain these labels for consistency with the auditing experiment in Section[5\.2](https://arxiv.org/html/2607.26358#S5.SS2); in this experiment, the two points illustrate the variation induced by GRPO RL oracle calls\. Blue circles correspond to points on an exponential grid between0andβhigh\\beta\_\{\\mathrm\{high\}\}, and the reference policy is denoted by a gray square\.

Figure[1](https://arxiv.org/html/2607.26358#S5.F1)shows the results forQwen3\-8Bwith shiftρ=0\.1\\rho=0\.1\. Policies that generate the longest responses exhibit lower coherence, while policies that preserve the highest narrative and grammatical coherence produce shorter responses\. Interestingly, narrative coherence actually increases with token count up to a point, before decreasing once the token count gets too large\. Each run of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)selects an intermediate policy near the bends of the empirical trade\-off curves, in contrast to the compute\-matched grid, which results in models closer to either extreme\.

Figure[2](https://arxiv.org/html/2607.26358#S5.F2)shows the corresponding results forLlama\-3\.2\-1Bwith shiftsρ=0\.1\\rho=0\.1andρ=0\.2\\rho=0\.2\. The same qualitative pattern holds in both settings: the bisection policies lie near the “elbow” of the curve traced out by the grid, avoiding both the high\-retention regime in which little progress is made on the length objective, and the high\-length regime in which coherence falls sharply\.555Llama\-3\.2\-1Bscores considerably lower thanQwen3\-8Bin terms of narrative and grammatical coherence for all values ofβ\\beta, and does not exhibit the same improvement in narrative coherence as token count increases\. This is likely due to it being a much smaller model\.

#### Takeaways\.

Across all settings, we find that Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)selects a single policy near the elbow of the empirical reward–retention frontier\. In some instances \(e\.g\., Figure[1](https://arxiv.org/html/2607.26358#S5.F1), left\), Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)finds desirable regions of the frontier that are not reached by grid search\. However even when Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)and grid search find similar parts of the frontier \(e\.g\., Figure[2\(b\)](https://arxiv.org/html/2607.26358#S5.F2.sf2)\), grid search requires an additional post\-hoc selection step, unlike Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)which resolves this trade\-off by adaptively concentrating its fine\-tuning runs around the equilibrium coefficient\.

### 5\.2Model Auditing

Table 1:Model\-auditing results at levelα=0\.05\\alpha=0\.05\.πA\\pi\_\{A\}SPRT uses the true deployed policy and is thus an oracle comparator;πM\\pi\_\{M\}SPRT uses the monitor’s independently trained equilibrium policy\. Lower stopping times and false positive rates are better\. Averages are taken over100100samples\.We now turn our attention to the auditing interpretation of the sequential detection game\. Consider a model\-hosting platform that sets inference costs such that they are profitable on a per\-token basis\. Such a platform has an economic incentive to fine\-tune its models to produce unnecessarily long responses, while remaining difficult to distinguish from the advertised model\(velasco2026test;cao2026pay\)\.

A third\-party auditor \(e\.g\., a watchdog organization\) can easily observe the model’s outputs, but verifying the parameters of the deployed model is significantly more difficult\. Nevertheless, the auditor may want to detect this strategic behavior when it exists, while controlling the probability of falsely flagging the model when it does not\. Our theory precisely characterizes the equilibrium of this game, and gives the auditor a way to empirically test for this type of bad behavior\.

Starting from the same reference policy and using the token length reward described above, the agent and monitor independently run Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)with the same RL oracle budget\. LetπA\\pi\_\{A\}denote the policy obtained by the agent and letπM\\pi\_\{M\}denote the policy obtained \(independently\) by the monitor/auditor\. The agent deploysπA\\pi\_\{\\mathrm\{A\}\}, whereas the monitor usesπM\\pi\_\{\\mathrm\{M\}\}as the alternative hypothesis in its sequential test\. In other words, the auditor knows the reference policy and the incentive being optimized, but they do not know the agent’s realized policy or share its fine\-tuning randomness\.

Each auditing trial uses a source policyπsrc∈\{πA,πref\}\\pi\_\{\\mathrm\{src\}\}\\in\\\{\\pi\_\{A\},\\pi\_\{\\mathrm\{ref\}\}\\\}\. Sampling fromπA\\pi\_\{A\}represents the presence of strategic fine\-tuning, whereasπref\\pi\_\{\\mathrm\{ref\}\}represents honest deployment\. At timett, a completionyt∼πsrc\(⋅\|x\)y\_\{t\}\\sim\\pi\_\{\\mathrm\{src\}\}\(\\cdot\|x\)is generated and scored under bothπM\\pi\_\{M\}andπref\\pi\_\{\\mathrm\{ref\}\}\. The monitor uses the SPRT which rejects ifLt\>log⁡\(1/α\)L\_\{t\}\>\\log\(1/\\alpha\)for anyt≥1t\\geq 1, whereLt:=∑s=1tlog⁡\(πM​\(ys\|x\)/πref​\(ys\|x\)\)L\_\{t\}:=\\sum\_\{s=1\}^\{t\}\\log\(\\pi\_\{M\}\(y\_\{s\}\|x\)/\\pi\_\{\\mathrm\{ref\}\}\(y\_\{s\}\|x\)\)and the log likelihood is computed by summing the token\-level log probabilities along the realized completion\.

We compare the equilibrium test against a uniform mixture likelihood ratio test \(described in Appendix[C](https://arxiv.org/html/2607.26358#A3)\)\. We also compare against the “skyline” SPRT which uses the agent’s actual deployed policy as its alternative\. While the agent’s policy is not available to the auditor, it gives a lower bound on the detection speed attainable by any SPRT\. Finally, all tests are evaluated on the same sequence of completions fromπsrc\\pi\_\{\\mathrm\{src\}\}; the candidate policies are used only to score these observations\.

#### Takeaways\.

Table[1](https://arxiv.org/html/2607.26358#S5.T1)summarizes the results\. ForQwen3\-8B, the strategic modification is sufficiently pronounced that every test detects it after approximately one observation\. The agent SPRT stops after exactly one completion on average, while the independently trained monitor stops after1\.081\.08completions\. The uniform mixture test obtains a slightly better stopping time of1\.021\.02, albeit at the cost of a false positive rate that is higher than the0\.050\.05threshold\. The differences between testing strategies are clearer forLlama\-3\.2\-1B\. As expected, the oracle agent SPRT detects the deployed policy fastest\. Nevertheless, the monitor test is significantly faster than the uniform mixture test at every calibration margin\. The composite grid tests also reject more frequently under the reference policy, with empirical false\-positive rates between0\.080\.08and0\.20\.2, compared to the monitor’s false positive rate which is between0\.010\.01and0\.060\.06\.

## 6Related Work

#### Reward–KL trade\-offs in RL fine\-tuning\.

KL regularization is a central component of modern LLM fine\-tuning, particularly in reinforcement learning from human feedback, where it is used to improve a target reward while limiting drift from a reference model\. Several works study the trade\-off between reward and KL regularization empirically\(see, e\.g\.,bai2022training;gao2023scaling\)\.korbak2022rlprovide a complementary theoretical interpretation of KL\-regularized RL through the lens of variational inference, where the reference model is analogous to a prior and the reward function provides evidence\.

#### Game\-theoretic applications of hypothesis testing\.

A small but growing line of work uses statistical hypothesis testing procedures as strategic instruments in games\.gauthier2026bettinguse sequential tests to monitor whether agents in a multi\-agent system are behaving consistently with equilibrium play\.capitaine2026testuse sequential testing in infinitely repeated games, where they show a “test\-then\-punish strategy” can sustain cooperation by detecting deviations and triggering punishment\.hu2024gameare closer in spirit, as they also study game\-theoretic hypothesis testing against strategic evasion\. They model an attacker who manipulates observations to evade a Neyman–Pearson test and characterize equilibrium distortion strategies and detector responses in Stackelberg and signaling games\. Our settings differ in both the object being manipulated and the resulting equilibrium structure\.

#### Continual learning\.

Our experiments in Section[5\.1](https://arxiv.org/html/2607.26358#S5.SS1)are instances of continual learning, where the goal is to learn new tasks while avoiding degradation on previously learned tasks\.sun2019lamoluses LLMs to generate pseudo\-samples of previous tasks for training alongside data for the new task\.razdaibiedina2023progressiveavoid modifying the reference policy by learning task\-specific “soft prompts” and concatenating newly learned prompts with earlier ones\.zhang2024cppostudy continual learning in RLHF and use sample\-wise “balance weights” to regulate the trade\-off between policy learning and knowledge retention\. Like us,zhang2023copfexplicitly consider policies that are tilted from a previous task’s policy, but they treat the KL coefficient as an externally chosen hyperparameter\. Seewu2024continualandshi2025continualfor surveys on continual learning for LLMs\.

#### AI auditing\.

Our experiments in Section[5\.2](https://arxiv.org/html/2607.26358#S5.SS2)are related to work on language model auditing, where the goal is to provide independent assurance that LLMs operate safely, ethically, and in compliance with legal or organizational standards\(mokander2024auditing\)\. Several recent works formalize auditing as a statistical testing problem over model outputs\.gao2025modelintroduce model equality testing, which asks whether a black\-box API is serving the same model as a claimed reference model\.richter2025auditingframe auditing for behavioral shifts in LLMs as a sequential testing problem\. Our auditing setup is aligned with this statistical view, but differs in that the audited model is itself trained strategically to avoid detection\.

#### Strategic learning\.

More broadly, our work belongs to the literature on strategic machine learning, where the deployment of a learning algorithm influences the behavior of other agents\. Strategic classification studies settings in which individuals manipulate their features to receive favorable decisions\(hardt2016strategic\), while performative prediction\(perdomo2020performative\)analyzes learning problems where the underlying data distribution changes in response to the deployed model\. Subsequent work extends these ideas to multi\-step settings, where strategic responses can accumulate over time\(harris2021stateful;brown2022performative\), like in our setting\.

## 7Conclusions and Future Research

We have introduced the sequential detection game, a game\-theoretic framework for RL fine\-tuning in which an agent seeks to maximize reward while remaining difficult to distinguish from a reference policy\. We have shown that the Nash equilibrium of this game recovers the standard KL\-regularized RL objective while determining the regularization coefficient endogenously, giving it an explicit operational interpretation rather than treating it as a tunable hyperparameter\. We then developed a stochastic bisection algorithm for estimating the equilibrium coefficient and present preliminary empirical evidence that it yields competitive reward–retention trade\-offs in a continual learning setting, while also enabling principled auditing of language model APIs\. There are several further directions that are worth pursuing in this vein\.

#### Unknown agent motives\.

Our framework provides a principled way to do black\-box auditing of model API platforms when their reward function is known to the auditor\. While knowledge of the reward function may be reasonable in settings where the platform’s incentives are public knowledge, it would be interesting to extend our equilibrium characterization to settings where the auditor is uncertain about the underlying reward function\.

#### Antidistillation sampling\.

Another promising application of our framework is antidistillation sampling\(savani2026antidistillation\), where the goal is to modify a model’s next\-token probabilities in order to make it harder to distill the model’s capabilities\.savani2026antidistillationdo this by tilting the reference policy with a specially designed reward function that makes model distillation more difficult\. They treat the regularization coefficient as a tunable parameter to control the trade\-off between accuracy and \(anti\-\)distillability, but it would be interesting to use our framework to automatically balance between the two\.

#### Further empirical work

is needed to close the gap between our equilibrium analysis and large\-scale RL fine\-tuning in practice\. More broadly, we hope that this work encourages the development of practical fine\-tuning methods whose hyperparameters are derived from principled operational objectives, rather than chosen heuristically\.

## Acknowledgments

KH was supported in part by the Simons Institute for the Theory of Computing, and part of this work was conducted when he was visiting the Institute\. IW\-S gratefully acknowledges support from the Miller Institute for Basic Research in Science\. PA gratefully acknowledges the support of DARPA through award No\. HR00112520022\. This work was also funded by the European Union \(ERC\-2022\-SYG\-OCEAN\-101071601\), the NSF Institute for Foundations of Machine Learning under grant CCF\-2505865, the National Science Foundation under grants CCF\-2145898, the Office of Naval Research under grant N00014\-24\-1\-2159, a Google Research Scholar Award, an Alfred P\. Sloan fellowship, and a Schmidt Science AI2050 fellowship\. Views and opinions expressed are however those of the author\(s\) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency\. Neither the European Union nor the granting authority can be held responsible for them\.

## References

## Appendix AAppendix for Section[3](https://arxiv.org/html/2607.26358#S3): The Sequential Detection Game

###### Lemma A\.1\(dinkelbach1967nonlinear\)\.

The following properties are true forM​\(β\)M\(\\beta\):

1. 1\.M​\(β⋆\)=0M\(\\beta^\{\\star\}\)=0
2. 2\.M​\(β\)<0M\(\\beta\)<0for allβ\>β⋆\\beta\>\\beta^\{\\star\}
3. 3\.M​\(β\)\>0M\(\\beta\)\>0for allβ<β⋆\\beta<\\beta^\{\\star\}
4. 4\.M​\(β\)M\(\\beta\)is convex\.

###### Lemma A\.2\.

Letτα:=inf\{τ:∑t=1τlog⁡d​Qd​P​\(Xt\)≥log⁡\(1/α\)\}\\tau\_\{\\alpha\}:=\\inf\\\{\\tau\\;:\\;\\sum\_\{t=1\}^\{\\tau\}\\log\\frac\{dQ\}\{dP\}\(X\_\{t\}\)\\geq\\log\(1/\\alpha\)\\\}and assume thatQ<<PQ<<P,𝔼R​\[τα\]<∞\\mathbb\{E\}\_\{R\}\[\\tau\_\{\\alpha\}\]<\\infty, and𝔼R​\[log⁡d​Qd​P​\(X\)\]<∞\\mathbb\{E\}\_\{R\}\[\\log\\frac\{dQ\}\{dP\}\(X\)\]<\\infty\. Then

𝔼R​\[τα\]≥log⁡\(1/α\)𝔼R​\[log⁡d​Qd​P​\(X\)\]\.\\mathbb\{E\}\_\{R\}\[\\tau\_\{\\alpha\}\]\\geq\\frac\{\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{R\}\[\\log\\frac\{dQ\}\{dP\}\(X\)\]\}\.

###### Proof\.

By Wald’s identity,

𝔼R​\[∑t=1ταlog⁡d​Qd​P​\(Xt\)\]=𝔼R​\[τα\]⋅𝔼R​\[log⁡d​Qd​P​\(X\)\]\.\\mathbb\{E\}\_\{R\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}\\log\\frac\{dQ\}\{dP\}\(X\_\{t\}\)\\right\]=\\mathbb\{E\}\_\{R\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{R\}\\left\[\\log\\frac\{dQ\}\{dP\}\(X\)\\right\]\.Our definition ofτα\\tau\_\{\\alpha\}implies that

𝔼R​\[τα\]⋅𝔼R​\[log⁡d​Qd​P​\(X\)\]≥log⁡\(1/α\)\.\\mathbb\{E\}\_\{R\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{R\}\\left\[\\log\\frac\{dQ\}\{dP\}\(X\)\\right\]\\geq\\log\(1/\\alpha\)\.Dividing both sides by𝔼R​\[log⁡d​Qd​P​\(X\)\]\\mathbb\{E\}\_\{R\}\[\\log\\frac\{dQ\}\{dP\}\(X\)\]obtains the result\. ∎

### A\.1Robustness to tie\-breaking

Whileπβ⋆\\pi\_\{\\beta^\{\\star\}\}is an optimal policy for the agent to play against the monitor’s equilibrium SPRT, it is not the only one\.

###### Theorem A\.3\.

Letd\(π\):=𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)−KL\(π\(⋅\|x\),πβ⋆\(⋅\|x\)\)\]d\(\\pi\):=\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\-\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\beta^\{\\star\}\}\(\\cdot\|x\)\)\]\. Suppose thatτα:=inf\{τ:∑t=1τlog⁡πβ⋆​\(xt,yt\)πref​\(xt,yt\)≥log⁡\(1/α\)\}\\tau\_\{\\alpha\}:=\\inf\\left\\\{\\tau\\;:\\;\\sum\_\{t=1\}^\{\\tau\}\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(x\_\{t\},y\_\{t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(x\_\{t\},y\_\{t\}\)\}\\geq\\log\(1/\\alpha\)\\right\\\}and the agent plays policyπ\\pisuch thatd​\(π\)\>0d\(\\pi\)\>0\. We have that

maxπ⁡𝔼xt∼𝒟,yt∼π\(⋅\|xt\)​\[∑t=1ταr​\(xt,yt\)\]≥β⋆​log⁡\(1/α\)\.\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\_\{t\}\\sim\\mathcal\{D\},y\_\{t\}\\sim\\pi\(\\cdot\|x\_\{t\}\)\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\]\\geq\\beta^\{\\star\}\\log\(1/\\alpha\)\.

###### Proof\.

We have that

maxπ⁡𝔼xt∼𝒟,yt∼π\(⋅\|xt\)​\[∑t=1ταr​\(xt,yt\)\]=maxπ⁡𝔼xt∼𝒟,yt∼π\(⋅\|xt\)​\[τα\]⋅𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\],\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\_\{t\}\\sim\\mathcal\{D\},y\_\{t\}\\sim\\pi\(\\cdot\|x\_\{t\}\)\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\]=\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\_\{t\}\\sim\\mathcal\{D\},y\_\{t\}\\sim\\pi\(\\cdot\|x\_\{t\}\)\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\],
where we know thatd​\(π\)\>0d\(\\pi\)\>0\(and hence𝔼y∼π\(⋅\|x\)​\[τα\]<∞\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[\\tau\_\{\\alpha\}\]<\\infty\) by Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)\. Applying Lemma[A\.2](https://arxiv.org/html/2607.26358#A1.Thmtheorem2), we have that

maxπ⁡𝔼xt∼𝒟,yt∼π\(⋅\|xt\)​\[∑t=1ταr​\(xt,yt\)\]≥maxπ⁡𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)𝔼y∼π\(⋅\|x\)​\[log⁡πβ⋆​\(y\|x\)πref​\(y\|x\)\]\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\_\{t\}\\sim\\mathcal\{D\},y\_\{t\}\\sim\\pi\(\\cdot\|x\_\{t\}\)\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\]\\geq\\max\_\{\\pi\}\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}The joint distribution induced byπβ⋆\\pi\_\{\\beta^\{\\star\}\}is absolutely continuous with respect toπref\\pi\_\{\\mathrm\{ref\}\}sinceπβ⋆\\pi\_\{\\beta^\{\\star\}\}is a tilt ofπref\\pi\_\{\\mathrm\{ref\}\}\. Focusing on the denominator, we have that

𝔼y∼π\(⋅\|x\)​\[log⁡πβ⋆​\(y\|x\)πref​\(y\|x\)\]=1β⋆​𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]−𝔼​\[log⁡Zβ⋆​\(x\)\]\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]=\\frac\{1\}\{\\beta^\{\\star\}\}\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\mathbb\{E\}\[\\log Z\_\{\\beta^\{\\star\}\}\(x\)\]Setting this aside for a moment, recall that

M​\(β⋆\)\\displaystyle M\(\\beta^\{\\star\}\)=𝔼y∼πβ⋆\(⋅\|x\)\[r\(x,y\)\]−β⋆𝔼\[KL\(πβ⋆\(⋅\|x\),πref\(⋅\|x\)\)\]\\displaystyle=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta^\{\\star\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta^\{\\star\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\_\{\\beta^\{\\star\}\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]=β⋆​𝔼​\[log⁡Zβ⋆​\(x\)\]=0\\displaystyle=\\beta^\{\\star\}\\mathbb\{E\}\[\\log Z\_\{\\beta^\{\\star\}\}\(x\)\]=0where the last equality follows from Lemma[A\.1](https://arxiv.org/html/2607.26358#A1.Thmtheorem1)\. Since we know thatβ⋆\>0\\beta^\{\\star\}\>0by Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1), we can conclude that𝔼​\[log⁡Zβ⋆​\(x\)\]=0\\mathbb\{E\}\[\\log Z\_\{\\beta^\{\\star\}\}\(x\)\]=0\. Substituting this all back in, we can conclude that

𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)𝔼y∼π\(⋅\|x\)​\[log⁡πβ⋆​\(y\|x\)πref​\(y\|x\)\]=β⋆​log⁡\(1/α\),\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}=\\beta^\{\\star\}\\log\(1/\\alpha\),which is independent of the agent’s policyπ\\pias long asd​\(π\)\>0d\(\\pi\)\>0\. ∎

Therefore, we can conclude that if the monitor is deploying a sequential likelihood ratio test betweenπβ⋆\\pi\_\{\\beta^\{\\star\}\}andπref\\pi\_\{\\mathrm\{ref\}\}, the agent can guarantee themselves utility at leastβ⋆​log⁡\(1/α\)\\beta^\{\\star\}\\log\(1/\\alpha\)by playing any policyπ\\pisuch thatd​\(π\)\>0d\(\\pi\)\>0\(includingπ=πβ⋆\\pi=\\pi\_\{\\beta^\{\\star\}\}\)\. Moreover asα↓0\\alpha\\downarrow 0, this is the optimal utility attainable for the agent under this test\.

The test with stopping timeτα:=inf\{τ:∑t=1τlog⁡πβ⋆​\(xt,yt\)πref​\(xt,yt\)≥log⁡\(1/α\)\}\\tau\_\{\\alpha\}:=\\inf\\\{\\tau\\;:\\;\\sum\_\{t=1\}^\{\\tau\}\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(x\_\{t\},y\_\{t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(x\_\{t\},y\_\{t\}\)\}\\geq\\log\(1/\\alpha\)\\\}is also a valid sequential test for testing the null hypothesisℋ0:π=πref\\mathcal\{H\}\_\{0\}\\;:\\;\\pi=\\pi\_\{\\mathrm\{ref\}\}versus the*composite*alternativeℋ1:π​is s\.t\.​d​\(π\)\>0\\mathcal\{H\}\_\{1\}\\;:\\;\\pi\\text\{ is s\.t\. \}d\(\\pi\)\>0\.

To see this, letLt=log⁡πβ⋆​\(yt\|xt\)πref​\(yt\|xt\)L\_\{t\}=\\log\\frac\{\\pi\_\{\\beta^\{\\star\}\}\(y\_\{t\}\|x\_\{t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{t\}\|x\_\{t\}\)\}and note that underℋ0\\mathcal\{H\}\_\{0\},exp⁡\(∑s=1tLs\)\\exp\\left\(\\sum\_\{s=1\}^\{t\}L\_\{s\}\\right\)is a non\-negative martingale with mean11\. By Ville’s inequality, we have that

ℙℋ0​\(supt\>0exp⁡\(∑s=1tLs\)≥1α\)≤α,\\mathbb\{P\}\_\{\\mathcal\{H\}\_\{0\}\}\\left\(\\sup\_\{t\>0\}\\exp\\left\(\\sum\_\{s=1\}^\{t\}L\_\{s\}\\right\)\\geq\\frac\{1\}\{\\alpha\}\\right\)\\leq\\alpha,which implies thatℙℋ0​\(τα<∞\)≤α\\mathbb\{P\}\_\{\\mathcal\{H\}\_\{0\}\}\(\\tau\_\{\\alpha\}<\\infty\)\\leq\\alpha\.

Underℋ1\\mathcal\{H\}\_\{1\},𝔼y∼π\(⋅\|x\)​\[Lt\]=d​\(π\)\>0\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[L\_\{t\}\]=d\(\\pi\)\>0, so by the law of large numbers we have that∑s=1tLs→∞\\sum\_\{s=1\}^\{t\}L\_\{s\}\\rightarrow\\inftyalmost surely, and soτα<∞\\tau\_\{\\alpha\}<\\inftyalmost surely\.

Therefore, even if the agent breaks ties among best\-response policies in an arbitrary or unknown way, this remains a valid power\-1 sequential test for the composite alternativeℋ1:π​is s\.t\.​d​\(π\)\>0\\mathcal\{H\}\_\{1\}\\;:\\;\\pi\\text\{ is s\.t\. \}d\(\\pi\)\>0\.

## Appendix BAppendix for Section[4](https://arxiv.org/html/2607.26358#S4): Reduction to KL\-Regularized RL

###### Assumption B\.1\(Sub\-Gaussian rewards\)\.

Rewards are sub\-Gaussian with parameterσ\\sigma, i\.e\., for anyπ\\piand allλ∈ℝ\\lambda\\in\\mathbb\{R\},

log⁡𝔼y∼π\(⋅\|x\)​\[exp⁡\(λ​\(r​\(x,y\)−μ\)\)\]≤σ2​λ22,\\log\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\exp\(\\lambda\(r\(x,y\)\-\\mu\)\)\\right\]\\leq\\frac\{\\sigma^\{2\}\\lambda^\{2\}\}\{2\},whereμ:=𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\mu:=\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\.

###### Lemma B\.2\(howard2021time\)\.

Given a sequence of i\.i\.d\. observations\(X\)n=1∞\(X\)\_\{n=1\}^\{\\infty\}from aσ\\sigma\-sub\-Gaussian distribution, we have that

\|1n​∑i=1nXi−𝔼​\[X\]\|≤1\.7​σ​0\.72​log⁡\(10\.4/δ\)\+log⁡log⁡\(2​n\)n\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}X\_\{i\}\-\\mathbb\{E\}\[X\]\\right\|\\leq 1\.7\\sigma\\sqrt\{\\frac\{0\.72\\log\(10\.4/\\delta\)\+\\log\\log\(2n\)\}\{n\}\}with probability at least1−δ1\-\\delta, simultaneously for everyn≥1n\\geq 1\.

###### Lemma B\.3\.

Under Assumption[B\.1](https://arxiv.org/html/2607.26358#A2.Thmtheorem1),β⋆≤σ22​\|μref\|\\beta^\{\\star\}\\leq\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\}\.

###### Proof\.

Consider the Donsker\-Varadhan variational formula\[donsker1975variational\], which says that for two distributionsPP,QQsuch thatP≪QP\\ll Qand functionffwhich is measurable,

𝔼P​\[f\]≤log⁡𝔼Q​\[exp⁡\(f\)\]\+KL​\(P,Q\)\.\\mathbb\{E\}\_\{P\}\[f\]\\leq\\log\\mathbb\{E\}\_\{Q\}\[\\exp\(f\)\]\+\\mathrm\{KL\}\(P,Q\)\.LetPPbe the joint distribution over\(x,y\)\(x,y\)such thatx∼𝒟x\\sim\\mathcal\{D\},y∼π\(⋅\|x\)y\\sim\\pi\(\\cdot\|x\)for some arbitrary policyπ\\pi,QQbe the joint distribution such thatx∼𝒟x\\sim\\mathcal\{D\},y∼πref\(⋅\|x\)y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\), andf=λ​\(r​\(x,y\)−μref\)f=\\lambda\(r\(x,y\)\-\\mu\_\{\\mathrm\{ref\}\}\)for some arbitraryλ∈ℝ\\lambda\\in\\mathbb\{R\}\. Plugging this in, we get that

𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]−μref≤1λ\(log𝔼y∼πref\(⋅\|x\)\[exp\(λ\(r\(x,y\)−μref\)\)\]\+𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\)\.\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\mu\_\{\\mathrm\{ref\}\}\\leq\\frac\{1\}\{\\lambda\}\\left\(\\log\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[\\exp\(\\lambda\(r\(x,y\)\-\\mu\_\{\\mathrm\{ref\}\}\)\)\]\+\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\\right\)\.Observe thatP≪QP\\ll Qis implicit in our setting as if this fails to hold, thenKL​\(π∥πref\)=∞\\mathrm\{KL\}\(\\pi\\\|\\pi\_\{\\mathrm\{ref\}\}\)=\\inftyso such a policyπ\\picould never be optimal\. Applying Assumption[B\.1](https://arxiv.org/html/2607.26358#A2.Thmtheorem1), we have that

𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]−μref\\displaystyle\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\mu\_\{\\mathrm\{ref\}\}≤1λ\(σ2​λ22\+𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\)\\displaystyle\\leq\\frac\{1\}\{\\lambda\}\\left\(\\frac\{\\sigma^\{2\}\\lambda^\{2\}\}\{2\}\+\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\\right\)=σ2​λ2\+1λ𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\.\\displaystyle=\\frac\{\\sigma^\{2\}\\lambda\}\{2\}\+\\frac\{1\}\{\\lambda\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\.Sinceλ\\lambdais arbitrary, we can pick it to minimize the right hand side\. Applying first\-order conditions, we get that the optimal choice ofλ\\lambdais

λ=2𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]σ2\.\\lambda=\\sqrt\{\\frac\{2\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\}\{\\sigma^\{2\}\}\}\.Plugging this into our bound, we get that

𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]−μref≤2σ2𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\.\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\mu\_\{\\mathrm\{ref\}\}\\leq\\sqrt\{2\\sigma^\{2\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\}\.Rearranging terms and subtractingβ𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\beta\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]from each side of the inequality, we get

𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]−β𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\displaystyle\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]≤μref\+2​σ​𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\displaystyle\\leq\\mu\_\{\\mathrm\{ref\}\}\+\\sqrt\{2\}\\sigma\\sqrt\{\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\}−β𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\.\\displaystyle\-\\beta\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\.Our choice ofπ\\piwas arbitrary, so we take the max overπ\\pion both sides to get

M​\(β\)\\displaystyle M\(\\beta\)≤μref\+maxπ\(2σ𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]−β𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\)\\displaystyle\\leq\\mu\_\{\\mathrm\{ref\}\}\+\\max\_\{\\pi\}\\left\(\\sqrt\{2\}\\sigma\\sqrt\{\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\}\-\\beta\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\\right\)=μref\+maxk≥0⁡\(2​σ​k−β​k\)\.\\displaystyle=\\mu\_\{\\mathrm\{ref\}\}\+\\max\_\{k\\geq 0\}\(\\sqrt\{2\}\\sigma\\sqrt\{k\}\-\\beta k\)\.Since2​σ​k−β​k\\sqrt\{2\}\\sigma\\sqrt\{k\}\-\\beta kis concave inkk, we can apply the first order condition to compute the maximum, which is obtained atk=σ22​β2k=\\frac\{\\sigma^\{2\}\}\{2\\beta^\{2\}\}\. Substituting this back in, we get that

M​\(β\)≤μref\+σ22​β\.M\(\\beta\)\\leq\\mu\_\{\\mathrm\{ref\}\}\+\\frac\{\\sigma^\{2\}\}\{2\\beta\}\.Using this inequality, we can see thatM​\(β\)<0M\(\\beta\)<0ifμref\+σ22​β<0\\mu\_\{\\mathrm\{ref\}\}\+\\frac\{\\sigma^\{2\}\}\{2\\beta\}<0, or equivalently, ifβ\>σ2−2​μref\\beta\>\\frac\{\\sigma^\{2\}\}\{\-2\\mu\_\{\\mathrm\{ref\}\}\}\. Finally, taking the contrapositive, we get that ifM​\(β\)≥0M\(\\beta\)\\geq 0, thenβ≤σ2−2​μref\\beta\\leq\\frac\{\\sigma^\{2\}\}\{\-2\\mu\_\{\\mathrm\{ref\}\}\}\. SinceM​\(β⋆\)=0M\(\\beta^\{\\star\}\)=0, we can conclude thatβ⋆≤σ2−2​μref=σ22​\|μref\|\\beta^\{\\star\}\\leq\\frac\{\\sigma^\{2\}\}\{\-2\\mu\_\{\\mathrm\{ref\}\}\}=\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\}\. ∎

See[4\.1](https://arxiv.org/html/2607.26358#S4.Thmtheorem1)

###### Proof\.

The proof consists of two separate parts:

1. 1\.Showing the correctness of Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1), conditioned on the clean event
2. 2\.Bounding the number of loop iterations in Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\(and therefore the number of high probability events\)

Correctness:βlow=0\\beta\_\{\\mathrm\{low\}\}=0is a valid lower bound onβ⋆\\beta^\{\\star\}, since Assumption[3\.1](https://arxiv.org/html/2607.26358#S3.Thmtheorem1)implies that it is possible for the agent to get non\-negative cumulative expected reward\.βhigh=σ22​\|μref\|\\beta\_\{\\mathrm\{high\}\}=\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\}is a valid upper bound onβ⋆\\beta^\{\\star\}by Lemma[B\.3](https://arxiv.org/html/2607.26358#A2.Thmtheorem3)\.

Under the clean event,M​\(β^\)<0M\(\\widehat\{\\beta\}\)<0ifM^n​\(β^\)\+rad​\(β^,n,δ\)<0\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)\+\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)<0, so we can conclude thatβhigh\>β^\>β⋆\\beta\_\{\\mathrm\{high\}\}\>\\widehat\{\\beta\}\>\\beta^\{\\star\}by parts 1 and 2 of Lemma[A\.1](https://arxiv.org/html/2607.26358#A1.Thmtheorem1)\. On the other hand, ifM^n​\(β^\)−rad​\(β^,n,δ\)\>0\\widehat\{M\}\_\{n\}\(\\widehat\{\\beta\}\)\-\\mathrm\{rad\}\(\\widehat\{\\beta\},n,\\delta\)\>0thenM​\(β^\)\>0M\(\\widehat\{\\beta\}\)\>0under the clean event, so we can conclude thatβlow<β^<β⋆\\beta\_\{\\mathrm\{low\}\}<\\widehat\{\\beta\}<\\beta^\{\\star\}by parts 1 and 3 of Lemma[A\.1](https://arxiv.org/html/2607.26358#A1.Thmtheorem1)\. Therefore sinceM​\(β\)M\(\\beta\)is a continuous function on the interval\[βlow,βhigh\]\[\\beta\_\{\\mathrm\{low\}\},\\beta\_\{\\mathrm\{high\}\}\], under the clean event,βhigh−βlow≤ϵ\\beta\_\{\\mathrm\{high\}\}\-\\beta\_\{\\mathrm\{low\}\}\\leq\\epsilonandβ⋆∈\[βlow,βhigh\]\\beta^\{\\star\}\\in\[\\beta\_\{\\mathrm\{low\}\},\\beta\_\{\\mathrm\{high\}\}\]once the while loop terminates\.

Number of iterations:Our bisection method takeslog2⁡\(βhigh−βlowϵ\)=log2⁡\(σ22​\|μref\|​ϵ\)\\log\_\{2\}\(\\frac\{\\beta\_\{\\mathrm\{high\}\}\-\\beta\_\{\\mathrm\{low\}\}\}\{\\epsilon\}\)=\\log\_\{2\}\(\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\\epsilon\}\)iterations to ensure that the final parameter returned is within an additive error ofϵ\\epsilon\. Assuming a confidence sequence onM^n​\(β\)\\widehat\{M\}\_\{n\}\(\\beta\)that is valid for alln≥1n\\geq 1and any fixedβ\\beta, our high probability statement requires taking the union bound over each bisection method iteration\. ∎

###### Lemma B\.4\.

Under Assumption[B\.1](https://arxiv.org/html/2607.26358#A2.Thmtheorem1), ifπref\\pi\_\{\\mathrm\{ref\}\}is such thatπref​\(ι\|h\)≥γ\\pi\_\{\\mathrm\{ref\}\}\(\\iota\|h\)\\geq\\gammafor any tokenι∈𝒴\\iota\\in\\mathcal\{Y\}and token prefixhh, then for any fixedβ\\beta,

\|M^n​\(β\)−M​\(β\)\|≤1\.7​\(σ\+β​m​log⁡\(1/γ\)\)​0\.72​log⁡\(20\.8/δ\)\+log⁡log⁡\(2​n\)n,\|\\widehat\{M\}\_\{n\}\(\\beta\)\-M\(\\beta\)\|\\leq 1\.7\\left\(\\sigma\+\\beta m\\log\(1/\\gamma\)\\right\)\\sqrt\{\\frac\{0\.72\\log\(20\.8/\\delta\)\+\\log\\log\(2n\)\}\{n\}\},with probability at least1−δ1\-\\deltasimultaneously for alln≥1n\\geq 1, wheremmis the maximum token output sequence length\. Therefore, it suffices to set

rad​\(β,n,δ\)=1\.7​\(σ\+β​m​log⁡\(1/γ\)\)​0\.72​log⁡\(20\.8/δ\)\+log⁡log⁡\(2​n\)n\\mathrm\{rad\}\(\\beta,n,\\delta\)=1\.7\\left\(\\sigma\+\\beta m\\log\(1/\\gamma\)\\right\)\\sqrt\{\\frac\{0\.72\\log\(20\.8/\\delta\)\+\\log\\log\(2n\)\}\{n\}\}in Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)\.

###### Proof\.

For an arbitrary policyπ\\piwithx1,…,xn∼𝒟x\_\{1\},\\ldots,x\_\{n\}\\sim\\mathcal\{D\}andyi∼π\(⋅\|xi\)y\_\{i\}\\sim\\pi\(\\cdot\|x\_\{i\}\), we have that

\|1n​∑i=1nr​\(xi,yi\)−𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\|≤1\.7​σ​0\.72​log⁡\(10\.4/δ\)\+log⁡log⁡\(2​n\)n,\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}r\(x\_\{i\},y\_\{i\}\)\-\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\\right\|\\leq 1\.7\\sigma\\sqrt\{\\frac\{0\.72\\log\(10\.4/\\delta\)\+\\log\\log\(2n\)\}\{n\}\},with probability at least1−δ1\-\\delta, simultaneously for everyn≥1n\\geq 1by Lemma[B\.2](https://arxiv.org/html/2607.26358#A2.Thmtheorem2)\. Turning our attention to the KL term, we have that

𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\\displaystyle\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]=𝔼y∼π\(⋅\|x\)​\[∑j=1mlog⁡\(π​\(yj\|y1:j−1,x\)πref​\(yj\|y1:j−1,x\)\)\]\\displaystyle=\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\sum\_\{j=1\}^\{m\}\\log\\left\(\\frac\{\\pi\(y\_\{j\}\|y\_\{1:j\-1\},x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{j\}\|y\_\{1:j\-1\},x\)\}\\right\)\\right\]=𝔼​\[∑j=1m𝔼y1:j−1∼π\(⋅\|x\)​\[∑ι∈𝒴π​\(ι\|y1:j−1,x\)​log⁡\(π​\(ι\|y1:j−1,x\)πref​\(ι\|y1:j−1,x\)\)\]\]\.\\displaystyle=\\mathbb\{E\}\\left\[\\sum\_\{j=1\}^\{m\}\\mathbb\{E\}\_\{y\_\{1:j\-1\}\\sim\\pi\(\\cdot\|x\)\}\\left\[\\sum\_\{\\iota\\in\\mathcal\{Y\}\}\\pi\(\\iota\|y\_\{1:j\-1\},x\)\\log\\left\(\\frac\{\\pi\(\\iota\|y\_\{1:j\-1\},x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\iota\|y\_\{1:j\-1\},x\)\}\\right\)\\right\]\\right\]\.LetGj​\(x\)=𝔼y1:j−1∼π\(⋅\|x\)​\[∑ι∈𝒴π​\(ι\|y1:j−1,x\)​log⁡\(π​\(ι\|y1:j−1,x\)πref​\(ι\|y1:j−1,x\)\)\]G\_\{j\}\(x\)=\\mathbb\{E\}\_\{y\_\{1:j\-1\}\\sim\\pi\(\\cdot\|x\)\}\\left\[\\sum\_\{\\iota\\in\\mathcal\{Y\}\}\\pi\(\\iota\|y\_\{1:j\-1\},x\)\\log\\left\(\\frac\{\\pi\(\\iota\|y\_\{1:j\-1\},x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\iota\|y\_\{1:j\-1\},x\)\}\\right\)\\right\], and observe that0≤Gj​\(x\)≤log⁡\(1/γ\)0\\leq G\_\{j\}\(x\)\\leq\\log\(1/\\gamma\)\.

Consider the one\-sample estimatorG^j​\(x\):=∑ι∈𝒴π​\(ι\|y1:j−1,x\)​log⁡\(π​\(ι\|y1:j−1,x\)πref​\(ι\|y1:j−1,x\)\)\\widehat\{G\}\_\{j\}\(x\):=\\sum\_\{\\iota\\in\\mathcal\{Y\}\}\\pi\(\\iota\|y\_\{1:j\-1\},x\)\\log\\left\(\\frac\{\\pi\(\\iota\|y\_\{1:j\-1\},x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\iota\|y\_\{1:j\-1\},x\)\}\\right\)and note that it is unbiased and computable given access toπ\\pi,πref\\pi\_\{\\mathrm\{ref\}\},xx, andy∼π\(⋅\|x\)y\\sim\\pi\(\\cdot\|x\)\. We can estimate𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]=𝔼\[∑j=1mGj\(x\)\]\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]=\\mathbb\{E\}\[\\sum\_\{j=1\}^\{m\}G\_\{j\}\(x\)\]as1n​∑i=1n∑j=1mG^j​\(xi\)\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{m\}\\widehat\{G\}\_\{j\}\(x\_\{i\}\)\. Observing that∑j=1mGj​\(x\)\\sum\_\{j=1\}^\{m\}G\_\{j\}\(x\)is sub\-Gaussian with parameter at mostm​log⁡\(1/γ\)m\\log\(1/\\gamma\), we can apply Lemma[B\.2](https://arxiv.org/html/2607.26358#A2.Thmtheorem2)to get

\|1n∑i=1n∑j=1mG^j\(xi\)−𝔼\[KL\(π\(⋅\|x\),πref\(⋅\|x\)\)\]\|≤1\.7mlog\(1/γ\)0\.72​log⁡\(10\.4/δ\)\+log⁡log⁡\(2​n\)n,\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{m\}\\widehat\{G\}\_\{j\}\(x\_\{i\}\)\-\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\\right\|\\leq 1\.7m\\log\(1/\\gamma\)\\sqrt\{\\frac\{0\.72\\log\(10\.4/\\delta\)\+\\log\\log\(2n\)\}\{n\}\},with probability at least1−δ1\-\\delta, simultaneously for alln≥1n\\geq 1\. Combining this with our high\-probability guarantee for𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]and taking a union bound gets us the desired result\. ∎

###### Corollary B\.5\.

Under Assumption[B\.1](https://arxiv.org/html/2607.26358#A2.Thmtheorem1), ifβhigh=σ22​\|μref\|\\beta\_\{\\mathrm\{high\}\}=\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\},πref​\(ι\|h\)≥γ\>0\\pi\_\{\\mathrm\{ref\}\}\(\\iota\|h\)\\geq\\gamma\>0for any tokenι∈𝒴\\iota\\in\\mathcal\{Y\}and token prefixhh, and

rad​\(β,n,δ\)=1\.7​\(σ\+β​m​log⁡\(1/γ\)\)​\(0\.72​log⁡\(20\.8/δ\)\+log⁡log⁡\(2​n\)\)/n,\\mathrm\{rad\}\(\\beta,n,\\delta\)=1\.7\\left\(\\sigma\+\\beta m\\log\(1/\\gamma\)\\right\)\\sqrt\{\(0\.72\\log\(20\.8/\\delta\)\+\\log\\log\(2n\)\)/n\},then with probability at least1−δ​log2⁡\(σ22​\|μref\|​ϵ\)1\-\\delta\\log\_\{2\}\\left\(\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\\epsilon\}\\right\), Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)returns a policyπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}satisfying0≤β⋆−βlow≤ϵ0\\leq\\beta^\{\\star\}\-\\beta\_\{\\mathrm\{low\}\}\\leq\\epsiloninlog2⁡\(σ22​\|μref\|​ϵ\)\\log\_\{2\}\\left\(\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\\epsilon\}\\right\)bisection iterations\.

###### Proof\.

Corollary[B\.5](https://arxiv.org/html/2607.26358#A2.Thmtheorem5)follows immediately from Theorem[4\.1](https://arxiv.org/html/2607.26358#S4.Thmtheorem1)and Lemma[B\.4](https://arxiv.org/html/2607.26358#A2.Thmtheorem4)\. ∎

###### Assumption B\.6\(Conditional sub\-Gaussianity\)\.

Rewards satisfy Assumption[B\.1](https://arxiv.org/html/2607.26358#A2.Thmtheorem1)\. Moreover, under the reference policy, rewards are uniformly conditionally sub\-Gaussian, i\.e\., for every promptx∈𝒳x\\in\\mathcal\{X\}and allλ∈ℝ\\lambda\\in\\mathbb\{R\},

log⁡𝔼y∼πref\(⋅\|x\)​\[exp⁡\(λ​\(r​\(x,y\)−μref​\(x\)\)\)\]≤σ2​λ22,\\log\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\\left\[\\exp\\left\(\\lambda\\bigl\(r\(x,y\)\-\\mu\_\{\\rm ref\}\(x\)\\bigr\)\\right\)\\right\]\\leq\\frac\{\\sigma^\{2\}\\lambda^\{2\}\}\{2\},whereμref​\(x\):=𝔼y∼πref\(⋅\|x\)​\[r​\(x,y\)\]\\mu\_\{\\mathrm\{ref\}\}\(x\):=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\.

###### Corollary B\.7\.

Under Assumption[B\.6](https://arxiv.org/html/2607.26358#A2.Thmtheorem6), if

rad​\(β,n,δ\)=1\.7​σ​\(2\+σ28​β2\)​0\.72​log⁡\(10\.4/δ\)\+log⁡log⁡\(2​n\)n,\\mathrm\{rad\}\(\\beta,n,\\delta\)=1\.7\\sigma\\sqrt\{\\left\(2\+\\frac\{\\sigma^\{2\}\}\{8\\beta^\{2\}\}\\right\)\\frac\{0\.72\\log\(10\.4/\\delta\)\+\\log\\log\(2n\)\}\{n\}\},andβhigh=σ22​\|μref\|\\beta\_\{\\mathrm\{high\}\}=\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\}, then Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)returns a policyπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}satisfying0≤β⋆−βlow≤ϵ0\\leq\\beta^\{\\star\}\-\\beta\_\{\\mathrm\{low\}\}\\leq\\epsilonwith probability at least1−δ​log2⁡\(σ22​\|μref\|​ϵ\)1\-\\delta\\log\_\{2\}\\left\(\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\rm ref\}\|\\epsilon\}\\right\)inlog2⁡\(σ22​\|μref\|​ϵ\)\\log\_\{2\}\\left\(\\frac\{\\sigma^\{2\}\}\{2\|\\mu\_\{\\mathrm\{ref\}\}\|\\epsilon\}\\right\)bisection iterations\.

###### Proof\.

We have that

log⁡πβ​\(y\|x\)πref​\(y\|x\)=r​\(x,y\)β−log⁡Zβ​\(x\)\.\\log\\frac\{\\pi\_\{\\beta\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}=\\frac\{r\(x,y\)\}\{\\beta\}\-\\log Z\_\{\\beta\}\(x\)\.and so for each fixedxx,

KL\(πβ\(⋅\|x\),πref\(⋅\|x\)\)=𝔼y∼πβ\(⋅\|x\)\[r​\(x,y\)β−logZβ\(x\)\],\\mathrm\{KL\}\(\\pi\_\{\\beta\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\}\(\\cdot\|x\)\}\\left\[\\frac\{r\(x,y\)\}\{\\beta\}\-\\log Z\_\{\\beta\}\(x\)\\right\],which implies thatM​\(β\)=𝔼​\[β​log⁡Zβ​\(x\)\]M\(\\beta\)=\\mathbb\{E\}\[\\beta\\log Z\_\{\\beta\}\(x\)\]\. By Jensen’s inequality and Assumption[B\.6](https://arxiv.org/html/2607.26358#A2.Thmtheorem6), we have that

0≤β​log⁡Zβ​\(x\)−𝔼y∼πref\(⋅\|x\)​\[r​\(x,y\)\]≤σ22​β0\\leq\\beta\\log Z\_\{\\beta\}\(x\)\-\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\\leq\\frac\{\\sigma^\{2\}\}\{2\\beta\}and𝔼y∼πref\(⋅\|x\)​\[r​\(x,y\)\]−μref\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\mu\_\{\\mathrm\{ref\}\}isσ\\sigma\-sub\-Gaussian overx∼𝒟x\\sim\\mathcal\{D\}\. Hence by Cauchy–Schwarz and Hoeffding’s lemma,

𝔼​exp⁡\(λ​\(β​log⁡Zβ​\(x\)−M​\(β\)\)\)≤exp⁡\(λ22​\(2​σ2\+σ48​β2\)\)\.\\mathbb\{E\}\\exp\\left\(\\lambda\\left\(\\beta\\log Z\_\{\\beta\}\(x\)\-M\(\\beta\)\\right\)\\right\)\\leq\\exp\\left\(\\frac\{\\lambda^\{2\}\}\{2\}\\left\(2\\sigma^\{2\}\+\\frac\{\\sigma^\{4\}\}\{8\\beta^\{2\}\}\\right\)\\right\)\.Applying Lemma[B\.2](https://arxiv.org/html/2607.26358#A2.Thmtheorem2)with sub\-Gaussian parameter2​σ2\+σ48​β2\\sqrt\{2\\sigma^\{2\}\+\\frac\{\\sigma^\{4\}\}\{8\\beta^\{2\}\}\}gives the desired confidence sequence\. The result now follows from Theorem[4\.1](https://arxiv.org/html/2607.26358#S4.Thmtheorem1)\. ∎

### B\.1Approximate Equilibria

###### Theorem B\.8\.

The following guarantees hold when the monitor and the agent use the tilted policyπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}returned by Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1):

1. 1\.If the monitor is deploying an SPRT betweenπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}andπref\\pi\_\{\\mathrm\{ref\}\}, the agent can guarantee themselves utility at leastβlow​log⁡\(1/α\)\\beta\_\{\\mathrm\{low\}\}\\log\(1/\\alpha\)by playing any policyπ\\pisuch that𝔼y∼π\(⋅\|x\)​\[log⁡πβlow​\(y\|x\)πref​\(y\|x\)\]\>0\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\>0\(includingπ=πβlow\\pi=\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\)\.666Recall that0≤β⋆−βlow≤ϵ0\\leq\\beta^\{\\star\}\-\\beta\_\{\\mathrm\{low\}\}\\leq\\epsilonwith high probability, by Theorem[4\.1](https://arxiv.org/html/2607.26358#S4.Thmtheorem1)\.
2. 2\.If the agent is playing policyπβlow\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}and the monitor is playing a power\-one simple\-vs\-simple SPRT, then their best response is to testℋ0:π=πref\\mathcal\{H\}\_\{0\}\\;:\\;\\pi=\\pi\_\{\\mathrm\{ref\}\}versusℋ1:π=πβlow\\mathcal\{H\}\_\{1\}\\;:\\;\\pi=\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\. Moreover, asα↓0\\alpha\\downarrow 0, this is the optimal such test out of all power\-one sequential tests for the monitor to play\[wald1948optimum\]\.

###### Proof\.

maxπ⁡𝔼yt∼π\(⋅\|xt\)​\[∑t=1ταr​\(xt,yt\)\]\\displaystyle\\max\_\{\\pi\}\\mathbb\{E\}\_\{y\_\{t\}\\sim\\pi\(\\cdot\|x\_\{t\}\)\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\]=maxπ⁡𝔼yt∼π\(⋅\|x\)​\[τα\]⋅𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\displaystyle=\\max\_\{\\pi\}\\mathbb\{E\}\_\{y\_\{t\}\\sim\\pi\(\\cdot\|x\)\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]≥maxπ⁡Ey∼π\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)𝔼y∼π\(⋅\|x\)​\[log⁡πβlow​\(y\|x\)πref​\(y\|x\)\]\\displaystyle\\geq\\max\_\{\\pi\}\\frac\{E\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}where the inequality follows from Lemma[A\.2](https://arxiv.org/html/2607.26358#A1.Thmtheorem2)and the fact thatβlow\>0\\beta\_\{\\mathrm\{low\}\}\>0with high probability\. We can rewrite

𝔼y∼π\(⋅\|x\)​\[log⁡πβlow​\(y\|x\)πref​\(y\|x\)\]=1βlow​𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]−1βlow​M​\(βlow\)\.\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[\\log\\frac\{\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\]=\\frac\{1\}\{\\beta\_\{\\mathrm\{low\}\}\}\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\frac\{1\}\{\\beta\_\{\\mathrm\{low\}\}\}M\(\\beta\_\{\\mathrm\{low\}\}\)\.Sinceβlow≤β⋆\\beta\_\{\\mathrm\{low\}\}\\leq\\beta^\{\\star\}, we know thatM​\(βlow\)\>0M\(\\beta\_\{\\mathrm\{low\}\}\)\>0, and so

maxπ⁡Ey∼π\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)𝔼y∼π\(⋅\|x\)​\[log⁡πβlow​\(y\|x\)πref​\(y\|x\)\]≥βlow​log⁡\(1/α\)\\max\_\{\\pi\}\\frac\{E\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[\\log\\frac\{\\pi\_\{\\beta\_\{\\mathrm\{low\}\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\]\}\\geq\\beta\_\{\\mathrm\{low\}\}\\log\(1/\\alpha\)∎

Observe that even if each agent runs Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)separately, if they stop after the same number of iterations they will arrive at the same value forβlow\\beta\_\{\\mathrm\{low\}\}with high probability\. With that being said, it is still possible to say something about the quality of the equilibrium whenever the monitor and the agent run Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)for a different number of iterations\.

###### Theorem B\.9\.

LetβA\\beta\_\{A\}\(resp\.βM\\beta\_\{M\}\) be the agent’s \(resp\. monitor’s\) computation ofβlow\\beta\_\{\\mathrm\{low\}\}and suppose thatβA≤βM\\beta\_\{A\}\\leq\\beta\_\{M\}, i\.e\., the clean event holds and the monitor runs Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)for at least as long as the agent\. Then:

1. 1\.If the monitor is deploying a SPRT betweenπβM\\pi\_\{\\beta\_\{M\}\}andπref\\pi\_\{\\mathrm\{ref\}\}, the agent can guarantee themselves utility at least\(β⋆−ϵ\)​log⁡\(1/α\)\(\\beta^\{\\star\}\-\\epsilon\)\\log\(1/\\alpha\)by playing policyπβA\\pi\_\{\\beta\_\{A\}\}\.
2. 2\.If the agent is playing policyπβA\\pi\_\{\\beta\_\{A\}\}and the monitor plays a power\-one simple SPRT betweenℋ0:π=πref\\mathcal\{H\}\_\{0\}:\\pi=\\pi\_\{\\mathrm\{ref\}\}andℋ1:π=πβM\\mathcal\{H\}\_\{1\}:\\pi=\\pi\_\{\\beta\_\{M\}\}, then asα↓0\\alpha\\downarrow 0 τ∗≤𝔼y∼πβA​\[τα\]≤βMβA⋅τ∗,\\tau^\{\*\}\\leq\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\}\[\\tau\_\{\\alpha\}\]\\leq\\frac\{\\beta\_\{M\}\}\{\\beta\_\{A\}\}\\cdot\\tau^\{\*\},whereτ∗\\tau^\{\*\}is the best possible expected stopping time\.

###### Proof\.

Part 1: The key step is to show that𝔼y∼πA\(⋅\|x\)​\[log⁡πM​\(y\|x\)πref​\(y\|x\)\]\>0\\mathbb\{E\}\_\{y\\sim\\pi\_\{A\}\(\\cdot\|x\)\}\[\\log\\frac\{\\pi\_\{M\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\]\>0\. To see this, observe that

𝔼y∼πβA\(⋅\|x\)​\[log⁡πβM​\(y\|x\)πref​\(y\|x\)\]=1βM​\(𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]−M​\(βM\)\)\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{M\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]=\\frac\{1\}\{\\beta\_\{M\}\}\(\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-M\(\\beta\_\{M\}\)\)and

𝔼y∼πβA\(⋅\|x\)\[r\(x,y\)\]=M\(βA\)\+βA𝔼\[KL\(πβA\(⋅\|x\),πref\(⋅\|x\)\)\]\>M\(βA\)≥M\(βM\)\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]=M\(\\beta\_\{A\}\)\+\\beta\_\{A\}\\mathbb\{E\}\[\\mathrm\{KL\}\(\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\),\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\)\]\>M\(\\beta\_\{A\}\)\\geq M\(\\beta\_\{M\}\)where the last inequality follows from the fact thatβA≤βM\\beta\_\{A\}\\leq\\beta\_\{M\}\. Given this, we can bound

𝔼yt∼πβA\(⋅\|xt\)​\[∑t=1ταr​\(xt,yt\)\]\\displaystyle\\mathbb\{E\}\_\{y\_\{t\}\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\_\{t\}\)\}\\left\[\\sum\_\{t=1\}^\{\\tau\_\{\\alpha\}\}r\(x\_\{t\},y\_\{t\}\)\\right\]=𝔼yt∼πβA\(⋅\|x\)​\[τα\]⋅𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]\\displaystyle=\\mathbb\{E\}\_\{y\_\{t\}\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[\\tau\_\{\\alpha\}\]\\cdot\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]≥Ey∼πβA\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)𝔼y∼πβA\(⋅\|x\)​\[log⁡πβM​\(y\|x\)πref​\(y\|x\)\]\\displaystyle\\geq\\frac\{E\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{M\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}=Ey∼πβA\(⋅\|x\)​\[r​\(x,y\)\]​log⁡\(1/α\)1βM​𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]−1βM​M​\(βM\)\\displaystyle=\\frac\{E\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\\log\(1/\\alpha\)\}\{\\frac\{1\}\{\\beta\_\{M\}\}\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\frac\{1\}\{\\beta\_\{M\}\}M\(\\beta\_\{M\}\)\}≥βM​log⁡\(1/α\)≥\(β⋆−ϵ\)​log⁡\(1/α\)\\displaystyle\\geq\\beta\_\{M\}\\log\(1/\\alpha\)\\geq\(\\beta^\{\\star\}\-\\epsilon\)\\log\(1/\\alpha\)
Part 2: Letτα​\(βM\)\\tau\_\{\\alpha\}\(\\beta\_\{M\}\)be the stopping time when the monitor is using a simple versus simple SPRT with alternative hypothesisπ=πβM\\pi=\\pi\_\{\\beta\_\{M\}\}, andτα​\(βA\)\\tau\_\{\\alpha\}\(\\beta\_\{A\}\)be the counterfactual stopping time when using alternative hypothesisπ=πβA\\pi=\\pi\_\{\\beta\_\{A\}\}\. Consider the limit whereα↓0\\alpha\\downarrow 0\. We have that

𝔼y∼πβA\(⋅\|x\)​\[log⁡πβM\(⋅\|x\)πref\(⋅\|x\)\]\\displaystyle\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{M\}\}\(\\cdot\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\}\\right\]=1βM​𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]−1βM​M​\(βM\)\\displaystyle=\\frac\{1\}\{\\beta\_\{M\}\}\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\frac\{1\}\{\\beta\_\{M\}\}M\(\\beta\_\{M\}\)≥1βM​𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]−1βM​M​\(βA\)\\displaystyle\\geq\\frac\{1\}\{\\beta\_\{M\}\}\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\frac\{1\}\{\\beta\_\{M\}\}M\(\\beta\_\{A\}\)=βAβM​\(1βA​𝔼y∼πβA\(⋅\|x\)​\[r​\(x,y\)\]−1βA​M​\(βA\)\)\\displaystyle=\\frac\{\\beta\_\{A\}\}\{\\beta\_\{M\}\}\\left\(\\frac\{1\}\{\\beta\_\{A\}\}\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\frac\{1\}\{\\beta\_\{A\}\}M\(\\beta\_\{A\}\)\\right\)Therefore,

𝔼y∼πβA​\[τα\]τ∗\\displaystyle\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\}\[\\tau\_\{\\alpha\}\]\}\{\\tau^\{\*\}\}=𝔼y∼πβA​\[τα​\(βM\)\]𝔼y∼πβA​\[τα​\(βA\)\]\\displaystyle=\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\}\[\\tau\_\{\\alpha\}\(\\beta\_\{M\}\)\]\}\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\}\[\\tau\_\{\\alpha\}\(\\beta\_\{A\}\)\]\}=𝔼y∼πβA\(⋅\|x\)​\[log⁡πβA​\(y\|x\)πref​\(y\|x\)\]𝔼y∼πβA\(⋅\|x\)​\[log⁡πβM​\(y\|x\)πref​\(y\|x\)\]≤βMβA\.\\displaystyle=\\frac\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{A\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}\{\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\beta\_\{A\}\}\(\\cdot\|x\)\}\\left\[\\log\\frac\{\\pi\_\{\\beta\_\{M\}\}\(y\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\|x\)\}\\right\]\}\\leq\\frac\{\\beta\_\{M\}\}\{\\beta\_\{A\}\}\.∎

## Appendix CAppendix for Section[5](https://arxiv.org/html/2607.26358#S5): Experiments

Policies are fine\-tuned with rank 16 LoRA adapters\[hu2022lora\]using GRPO\[shao2024deepseekmath\]as the RL oracle and Adam\[kingma2014adam\]as the optimizer with learning rate 1e\-4, temperature1\.01\.0, top\-pp1\.01\.0, and a completion cap of 2048 tokens\. Each call to the RL oracle consists of 15 optimization steps with 400 rollouts each\. All models were trained using the Tinker API\[thinkingmachines2025tinker\]\.

### C\.1Continual Learning

#### Judge prompts\.

For each trained policy, we evaluated generated completions using two separate LLM\-judge prompts: one for narrative coherence and one for grammatical coherence\. The same prompts were used for bothGPT\-5\-miniandGPT\-5\-nano\.

Narrative coherence system prompt:

> You are an expert literary critic evaluating short stories written by a small language model\. Your task is to rate the NARRATIVE COHERENCE of a story on an integer scale from 0 to 10\. Narrative coherence means: The story has a recognizable beginning, middle, and end\. Events follow each other in a sensible causal or temporal order\. Characters and settings remain consistent throughout\. The story arrives at some kind of resolution or conclusion\. Ignore grammar and spelling mistakes – those are evaluated separately\. Focus only on the structure and coherence of the narrative\. Respond with ONLY a single integer between 0 and 10, with no other text\. 0 = no discernible narrative; 10 = a complete, well\-structured short story\.

Grammatical coherence system prompt:

> You are an expert linguist evaluating short stories written by a small language model\. Your task is to rate the GRAMMATICAL COHERENCE of a story on an integer scale from 0 to 10\. Grammatical coherence means: Sentences are syntactically well\-formed\. Subject\-verb agreement, tense, pronouns, and articles are used correctly\. Punctuation and capitalization roughly follow standard English conventions\. Words are spelled correctly and used in plausible contexts\. Ignore plot quality and narrative structure – those are evaluated separately\. Focus only on syntactic and morphological correctness\. Respond with ONLY a single integer between 0 and 10, with no other text\. 0 = essentially ungrammatical; 10 = fully grammatical, idiomatic English\.

![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p01_mini.png)Figure 3:Reward–retention trade\-offs forLlama\-3\.2\-1Bwith shiftρ=0\.01\\rho=0\.01\. Plotting conventions are the same as in Figure[1](https://arxiv.org/html/2607.26358#S5.F1)\. In both settings, Algorithm[1](https://arxiv.org/html/2607.26358#algorithm1)selects policies near the elbow of the empirical trade\-off curve traced out by the compute\-matchedβ\\betagrid\.Figure[3](https://arxiv.org/html/2607.26358#A3.F3)reports the additionalLlama\-3\.2\-1Bexperiment with calibration marginρ=0\.01\\rho=0\.01, usingGPT\-5\-minias the judge\.

![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/qwen_nano.png)Figure 4:Reward–retention trade\-offs forQwen3\-8Bwith shiftρ=0\.1\\rho=0\.1andGPT\-5\-nano\-as\-a\-judge\. All other plotting conventions are the same as in Figure[1](https://arxiv.org/html/2607.26358#S5.F1)\.![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p01_nano.png)Figure 5:Reward–retention trade\-offs forLlama\-3\.2\-1Bwith shiftρ=0\.01\\rho=0\.01andGPT\-5\-nano\-as\-a\-judge\.![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p1_nano.png)Figure 6:Reward–retention trade\-offs forLlama\-3\.2\-1Bwith shiftρ=0\.1\\rho=0\.1andGPT\-5\-nano\-as\-a\-judge\.![Refer to caption](https://arxiv.org/html/2607.26358v1/figures/llama_0p2_nano.png)Figure 7:Reward–retention trade\-offs forLlama\-3\.2\-1Bwith shiftρ=0\.2\\rho=0\.2andGPT\-5\-nano\-as\-a\-judge\.Figures[4](https://arxiv.org/html/2607.26358#A3.F4)–[7](https://arxiv.org/html/2607.26358#A3.F7)repeat the continual\-learning experiments usingGPT\-5\-nanoas the judge\. The absolute coherence scores differ slightly from theGPT\-5\-miniscores, but the qualitative pattern is unchanged: AcrossQwen3\-8BandLlama\-3\.2\-1B, and across the tested calibration margins, the stochastic\-bisection policies fall in the transition region of the Pareto frontier rather than at either extreme of the compute\-matchedβ\\betagrid\.

#### Plot details\.

This subsection contains theβ\\betavalues that are used in all figures, and the order in which they are plotted\. ForQwen3\-8B, allβ\\betavalues are listed in clockwise order\. ForLlama\-3\.2\-1B, allβ\\betavalues are listed from left to right\.

Qwen3\-8Bwith shiftρ=0\.1\\rho=0\.1details:

- •Figure[1](https://arxiv.org/html/2607.26358#S5.F1), left:0\.000580\.00058,2\.693​e−052\.693e\-05,0\.0001250\.000125,0\.002690\.00269,0\.00\.0,5\.802​e−065\.802e\-06,0\.0085450\.008545\(monitor\),0\.0085450\.008545\(agent\),0\.01250\.0125,0\.05800\.0580,0\.26930\.2693,1\.251\.25, base model\.
- •Figure[1](https://arxiv.org/html/2607.26358#S5.F1), right:0\.000580\.00058,5\.802​e−065\.802e\-06,0\.002690\.00269,2\.693​e−052\.693e\-05,0\.00\.0,0\.0001250\.000125,0\.0085450\.008545\(monitor\),0\.0085450\.008545\(agent\),0\.01250\.0125,0\.05800\.0580,0\.26930\.2693,1\.251\.25, base model\.
- •Figure[4](https://arxiv.org/html/2607.26358#A3.F4), left:0\.000580\.00058,0\.00\.0,2\.693​e−052\.693e\-05,0\.002690\.00269,5\.802​e−065\.802e\-06,0\.0001250\.000125,0\.0085450\.008545\(monitor\),0\.0085450\.008545\(agent\),0\.01250\.0125,0\.05800\.0580,1\.251\.25,0\.26930\.2693, base model\.
- •Figure[4](https://arxiv.org/html/2607.26358#A3.F4), right:0\.00\.0,2\.693​e−052\.693e\-05,0\.000580\.00058,0\.0001250\.000125,0\.002690\.00269,5\.802​e−065\.802e\-06,0\.0085450\.008545\(monitor\),0\.0085450\.008545\(agent\),0\.01250\.0125,0\.05800\.0580,1\.251\.25,0\.26930\.2693, base model\.

Llama\-3\.2\-1Bwith shiftρ=0\.01\\rho=0\.01details:

- •Figure[3](https://arxiv.org/html/2607.26358#A3.F3), left:0\.00040\.0004,0\.008840\.00884,0\.00\.0,0\.00190\.0019,8\.8437​e−058\.8437e\-05,0\.0410\.041,0\.0930\.093\(agent\),0\.0930\.093\(monitor\),0\.190\.19, base model,4\.104\.10,0\.8840\.884,19\.0519\.05
- •Figure[3](https://arxiv.org/html/2607.26358#A3.F3), right:0\.00040\.0004,0\.00\.0,0\.008840\.00884,8\.8437​e−058\.8437e\-05,0\.00190\.0019,0\.0410\.041,0\.0930\.093\(agent\),0\.0930\.093\(monitor\),0\.190\.19, base model,0\.8840\.884,19\.0519\.05,4\.104\.10
- •Figure[5](https://arxiv.org/html/2607.26358#A3.F5), left:0\.00040\.0004,0\.00190\.0019,0\.00\.0,0\.008840\.00884,8\.8437​e−058\.8437e\-05,0\.0410\.041,0\.0930\.093\(monitor\),0\.0930\.093\(agent\),0\.190\.19, base model,0\.8840\.884,19\.0519\.05,4\.104\.10
- •Figure[5](https://arxiv.org/html/2607.26358#A3.F5), right:0\.00040\.0004,0\.00\.0,0\.00190\.0019,0\.008840\.00884,8\.8437​e−058\.8437e\-05,0\.0410\.041,0\.0930\.093\(agent\),0\.0930\.093\(monitor\),0\.190\.19, base model,0\.8840\.884,19\.0519\.05,4\.104\.10

Llama\-3\.2\-1Bwith shiftρ=0\.1\\rho=0\.1details:

- •Figure[2\(a\)](https://arxiv.org/html/2607.26358#S5.F2.sf1), left:0\.00\.0,0\.00420\.0042,4\.202​e−054\.202e\-05,0\.0001950\.000195,0\.0009050\.000905,9\.0536​e−069\.0536e\-06,0\.01950\.0195,0\.083810\.08381\(monitor\),0\.083810\.08381\(agent\), base model,0\.09050\.0905,0\.42020\.4202,1\.951\.95
- •Figure[2\(a\)](https://arxiv.org/html/2607.26358#S5.F2.sf1), right:9\.0536​e−069\.0536e\-06,0\.00420\.0042,4\.202​e−054\.202e\-05,0\.0001950\.000195,0\.00\.0,0\.0009050\.000905,0\.01950\.0195,0\.083810\.08381\(monitor\),0\.083810\.08381\(agent\),0\.09050\.0905, base model,0\.42020\.4202,1\.951\.95
- •Figure[6](https://arxiv.org/html/2607.26358#A3.F6), left:0\.00420\.0042,0\.00\.0,4\.202​e−054\.202e\-05,0\.0001950\.000195,9\.0536​e−069\.0536e\-06,0\.0009050\.000905,0\.01950\.0195,0\.083810\.08381\(agent\),0\.083810\.08381\(monitor\),0\.09050\.0905, base model,0\.42020\.4202,1\.951\.95
- •Figure[6](https://arxiv.org/html/2607.26358#A3.F6), right:0\.00420\.0042,4\.202​e−054\.202e\-05,9\.0536​e−069\.0536e\-06,0\.0001950\.000195,0\.00\.0,0\.0009050\.000905,0\.01950\.0195,0\.083810\.08381\(monitor\),0\.083810\.08381\(agent\),0\.09050\.0905,0\.42020\.4202, base model,1\.951\.95

Llama\-3\.2\-1Bwith shiftρ=0\.2\\rho=0\.2details:

- •Figure[2\(b\)](https://arxiv.org/html/2607.26358#S5.F2.sf2), left:9\.352​e−059\.352e\-05,0\.0020\.002,0\.0004340\.000434,4\.34​e−064\.34e\-06,0\.00\.0,2\.0148​e−052\.0148e\-05,0\.009350\.00935,0\.04340\.0434,0\.06030\.0603\(monitor\),0\.06030\.0603\(agent\), base model,0\.2010\.201,0\.9350\.935
- •Figure[2\(b\)](https://arxiv.org/html/2607.26358#S5.F2.sf2), right:4\.34​e−064\.34e\-06,0\.0020\.002,9\.352​e−059\.352e\-05,0\.00\.0,0\.0004340\.000434,2\.0148​e−052\.0148e\-05,0\.009350\.00935,0\.04340\.0434,0\.06030\.0603\(agent\),0\.06030\.0603\(monitor\),0\.2010\.201, base model,0\.9350\.935
- •Figure[7](https://arxiv.org/html/2607.26358#A3.F7), left:9\.352​e−059\.352e\-05,0\.0020\.002,4\.34​e−064\.34e\-06,0\.0004340\.000434,0\.00\.0,2\.0148​e−052\.0148e\-05,0\.009350\.00935,0\.06030\.0603\(monitor\),0\.04340\.0434,0\.06030\.0603\(agent\), base model,0\.2010\.201,0\.9350\.935
- •Figure[7](https://arxiv.org/html/2607.26358#A3.F7), right:4\.34​e−064\.34e\-06,9\.352​e−059\.352e\-05,0\.0020\.002,0\.00\.0,0\.0004340\.000434,2\.0148​e−052\.0148e\-05,0\.009350\.00935,0\.06030\.0603\(agent\),0\.04340\.0434,0\.06030\.0603\(monitor\),0\.2010\.201, base model,0\.9350\.935

### C\.2Model Auditing

As a baseline, we fine\-tune surrogate policies on an exponentialβ\\beta\-grid \(the same one as in Appendix[C\.1](https://arxiv.org/html/2607.26358#A3.SS1)\) and, at each timestep, compute per\-arm log\-likelihood ratios against the reference on the same observed pairs\. The mixture likelihood\-ratio test aggregates evidence through a uniform mixture over grid points, stopping when the mixture statisticΛt=∑kwk​exp⁡\(Lk,t\)\\Lambda\_\{t\}=\\sum\_\{k\}w\_\{k\}\\exp\(L\_\{k,t\}\)exceeds1/α1/\\alpha, whereLk,tL\_\{k,t\}is the cumulative log\-LR for armkkandwk=1/Kw\_\{k\}=1/K\. Under strategic sampling we compare the stopping times of both composites to those of the monitor and the bisection\-optimal agent; under honest sampling we report the fraction of trials on which each composite crosses its threshold\.