Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias

# Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias
Source: [https://arxiv.org/html/2607.22837](https://arxiv.org/html/2607.22837)
Lior Shani Google Research liorshani@google\.comAviv Rosenberg Google Research avivros@google\.comAnkur Samanta Columbia University as7416@columbia\.eduTal Wagner Tel Aviv University talwag@tauex\.tau\.ac\.ilYonathan Efroni Tel Aviv University yefroni@tauex\.tau\.ac\.il

###### Abstract

Many organizations aim to adapt language models for internal use, both to improve performance on domain\-specific tasks and to address privacy concerns around sensitive data\. However, such adaptation remains non\-trivial: it often requires operationally challenging fine\-tuning of open\-source models or ad hoc prompt optimization\. We study a minimal alternative based on a simple API\-level control: allowing users to bias the model’s logits with a user\-defined vector\. We develop a black\-box method for learning a single context\-independent logit\-bias vector, added at every decoding step, without modifying model weights or requiring gradients\. Starting from a KL\-regularized reinforcement learning \(RL\) objective, we characterize when such a fixed logit\-bias vector can approximate the optimal prefix\-dependent correction and derive a closed\-form inverse\-propensity estimator from rollouts, rewards, and token probabilities\. Empirically, this simple decoding\-time intervention improves over base models on mathematical and reasoning benchmarks while using far fewer trainable parameters than conventional fine\-tuning\. Our results suggest that learned logit bias is a lightweight mechanism for adapting language models under minimal access requirements\.

Input\(x,y1:t−1\)\(x,y\_\{1:t\-1\}\)Frozen LMLogitsℓt\\ell\_\{t\}\+\+Biasδ\\deltaSoftmaxSampleyty\_\{t\}Figure 1:Overview of logit\-bias steering\. The fixed logit\-bias vectorδ\\deltais added to the base model’s logits at every decoding step before sampling\. Onlyδ\\deltais optimized; model weights remain unchanged\.## 1Introduction

Deploying frontier language models often requires adapting them to narrow operational needs: shorter reasoning traces, stricter output formatting, or improved accuracy on small, confidential in\-house distributions\[[48](https://arxiv.org/html/2607.22837#bib.bib48),[47](https://arxiv.org/html/2607.22837#bib.bib47),[21](https://arxiv.org/html/2607.22837#bib.bib21),[58](https://arxiv.org/html/2607.22837#bib.bib58),[2](https://arxiv.org/html/2607.22837#bib.bib2)\]\. In many practical settings, however, the model is available only through an inference API: the deployer cannot inspect or modify weights, cannot backpropagate through the model, and must work with the controls exposed at decoding time\[[55](https://arxiv.org/html/2607.22837#bib.bib55),[54](https://arxiv.org/html/2607.22837#bib.bib54),[8](https://arxiv.org/html/2607.22837#bib.bib8),[16](https://arxiv.org/html/2607.22837#bib.bib16)\]\. This makes standard adaptation—fine\-tuning, PEFT, prompt embeddings, RLHF\-style optimization\[[18](https://arxiv.org/html/2607.22837#bib.bib18),[25](https://arxiv.org/html/2607.22837#bib.bib25),[24](https://arxiv.org/html/2607.22837#bib.bib24),[19](https://arxiv.org/html/2607.22837#bib.bib19),[1](https://arxiv.org/html/2607.22837#bib.bib1),[28](https://arxiv.org/html/2607.22837#bib.bib28),[63](https://arxiv.org/html/2607.22837#bib.bib63),[40](https://arxiv.org/html/2607.22837#bib.bib40)\]—unavailable or mismatched to the setting\.

Can a single learned logit\-bias vector provide useful black\-box adaptation?

We study logit bias as a minimal adaptation primitive\. Token\-level logit\-bias controls are documented in major inference stacks and hosted APIs\[[57](https://arxiv.org/html/2607.22837#bib.bib57),[50](https://arxiv.org/html/2607.22837#bib.bib50),[31](https://arxiv.org/html/2607.22837#bib.bib31),[30](https://arxiv.org/html/2607.22837#bib.bib30),[37](https://arxiv.org/html/2607.22837#bib.bib37),[33](https://arxiv.org/html/2607.22837#bib.bib33),[11](https://arxiv.org/html/2607.22837#bib.bib11)\]\. We consider the simplest possible intervention: starting from a frozen base model with no learned logit bias, add a single context\-independent logit\-bias vectorδ\\deltaover the vocabulary to the model’s logits at every decoding step\. The base model’s weights are never modified; onlyδ\\deltais learned, using black\-box rollouts rather than gradients or access to model internals\.

This intervention is deliberately weak\. The fixed logit\-bias vectorδ\\deltacannot encode arbitrary prompt\- or prefix\-dependent behavior, unlike prompts that can express explicit conditional instructions\[[38](https://arxiv.org/html/2607.22837#bib.bib38),[34](https://arxiv.org/html/2607.22837#bib.bib34)\]\. But this weakness is also what makes the primitive attractive: when the desired correction is approximately token\-specific and stable across contexts, a single vector may capture a useful slice of adaptation with little serving overhead\. A particularly relevant case is verbosity control: althoughδ\\deltacannot decide which reasoning steps are mathematically necessary, it may still learn reusable finalization, formatting, and stopping cues that shorten completions without changing the model weights\. Unlike prompt steering it consumes no context length and acts directly on the output distribution, and unlike weight\-space adaptation it requires no gradients, hidden states, or provider\-side training access\[[36](https://arxiv.org/html/2607.22837#bib.bib36),[29](https://arxiv.org/html/2607.22837#bib.bib29),[18](https://arxiv.org/html/2607.22837#bib.bib18),[19](https://arxiv.org/html/2607.22837#bib.bib19)\]\.

We formalize this via a KL\-regularized RL objective\[[63](https://arxiv.org/html/2607.22837#bib.bib63),[53](https://arxiv.org/html/2607.22837#bib.bib53),[40](https://arxiv.org/html/2607.22837#bib.bib40),[22](https://arxiv.org/html/2607.22837#bib.bib22),[44](https://arxiv.org/html/2607.22837#bib.bib44),[62](https://arxiv.org/html/2607.22837#bib.bib62)\]; the unconstrained optimum induces a prefix\-dependent correction that we approximate with the fixed logit\-bias vectorδ\\delta, yielding a reward\-gap bound and a closed\-form IPS estimator from rollouts, rewards, and token probabilities\[[17](https://arxiv.org/html/2607.22837#bib.bib17)\]\.

##### Contributions\.

We make three contributions\.111Code available at[https://github\.com/Ofek\-Israeli/logit\_bias\_lm\_adaptation](https://github.com/Ofek-Israeli/logit_bias_lm_adaptation)\.First, we introduce*black\-box logit\-bias adaptation*, a setting for personalizing frozen language models with no weight updates and no gradient access\. Second, we derive a closed\-form estimator and a theoretical analysis connecting logit\-bias steering to the KL\-regularized optimum\. Third, we show empirically that full\-vocabulary logit bias improves base models on mathematical and reasoning benchmarks while using far fewer trainable parameters and far less compute than conventional fine\-tuning\.

## 2Related work

##### Black\-box or API\-only adaptation\.

Existing black\-box methods adapt through the input channel, using derivative\-free continuous prompts or reward\-learned discrete prompts\[[55](https://arxiv.org/html/2607.22837#bib.bib55),[54](https://arxiv.org/html/2607.22837#bib.bib54),[8](https://arxiv.org/html/2607.22837#bib.bib8)\]; such prompt\-based control consumes context and can interfere with task content or learned prompting behavior\[[25](https://arxiv.org/html/2607.22837#bib.bib25),[24](https://arxiv.org/html/2607.22837#bib.bib24),[29](https://arxiv.org/html/2607.22837#bib.bib29),[8](https://arxiv.org/html/2607.22837#bib.bib8)\]\. We instead adapt through the output channel: a single context\-independent logit\-bias vector learned from rollouts and applied at decoding\[[16](https://arxiv.org/html/2607.22837#bib.bib16)\], which is less expressive but easy to store, audit, and deploy via standard decoding controls, with no gradients or provider\-side fine\-tuning\[[37](https://arxiv.org/html/2607.22837#bib.bib37),[33](https://arxiv.org/html/2607.22837#bib.bib33),[57](https://arxiv.org/html/2607.22837#bib.bib57)\]\.

##### Decoding\-time control and logit guidance\.

Decoding\-time methods steer generation by perturbing hidden states, reweighting next\-token probabilities with auxiliary models, enforcing hard constraints, or applying generic logits processors\[[7](https://arxiv.org/html/2607.22837#bib.bib7),[61](https://arxiv.org/html/2607.22837#bib.bib61),[23](https://arxiv.org/html/2607.22837#bib.bib23),[27](https://arxiv.org/html/2607.22837#bib.bib27),[32](https://arxiv.org/html/2607.22837#bib.bib32),[13](https://arxiv.org/html/2607.22837#bib.bib13),[59](https://arxiv.org/html/2607.22837#bib.bib59)\]; our method is a simpler member of this family, using no hidden\-state gradients, auxiliary model, hard grammar, or per\-prefix optimization—only one learned logit\-bias vector reused at every step\.

##### RLHF, KL\-regularized objectives, and DPO\.

The alignment literature provides our theoretical backdrop\.Ziegler et al\. \[[63](https://arxiv.org/html/2607.22837#bib.bib63)\], Stiennon et al\. \[[53](https://arxiv.org/html/2607.22837#bib.bib53)\], Ouyang et al\. \[[40](https://arxiv.org/html/2607.22837#bib.bib40)\]established the standard pipeline of preference data, reward modeling, and policy optimization with a proximity penalty to the base model\.Korbak et al\. \[[22](https://arxiv.org/html/2607.22837#bib.bib22)\]cast KL\-regularized RL for language models as variational/Bayesian inference, and DPO gives a closed\-form preference\-to\-policy relation under a KL constraint\[[44](https://arxiv.org/html/2607.22837#bib.bib44)\]\. Our theory is closest to this KL\-tilted view: we start from a reward\-tilted target close to the base model, but instead of training a new policy or reward model, ask what improvement survives after projecting that target onto the severely restricted family of a single context\-independent logit\-bias vector\.

##### Parameter\-efficient and prompt\-based adaptation\.

Parameter\-efficient adaptation reduces cost by updating a small subset of parameters or learned prompts\. Representative methods include adapters\[[18](https://arxiv.org/html/2607.22837#bib.bib18)\], prefix\-tuning\[[25](https://arxiv.org/html/2607.22837#bib.bib25)\], prompt tuning\[[24](https://arxiv.org/html/2607.22837#bib.bib24)\], LoRA\[[19](https://arxiv.org/html/2607.22837#bib.bib19)\], BitFit\[[1](https://arxiv.org/html/2607.22837#bib.bib1)\], and\(I​A\)3\(IA\)^\{3\}\[[28](https://arxiv.org/html/2607.22837#bib.bib28)\]\. They show how far one can go with limited trainable state, but still need training\-time access to weights, gradients, hidden states, or prompt embeddings\. Our setting forbids this access: the base model stays frozen and adaptation is restricted to the fixed logit\-bias vectorδ\\deltaapplied at decoding\.

## 3Preliminaries

We consider autoregressive generation from a language model with a finite vocabulary𝕍\\mathbb\{V\}\. Letρ\\rhodenote a distribution over a prompt space𝒳\\mathcal\{X\}and letx∼ρx\\sim\\rhobe a prompt drawn from it\. For a given promptxx, generation proceeds for a fixed horizonT∈ℕT\\in\\mathbb\{N\}tokens\.222In practice the horizon varies per sequence due to early stopping at an EOS token\. We use a fixedTTto simplify the theoretical framework; all results extend to variable\-length generation by treating EOS as an absorbing state\.At each steptt, the base language model receives the prompt and the tokens generated so far,y1:t−1=\(y1,…,yt−1\)y\_\{1:t\-1\}=\(y\_\{1\},\\dots,y\_\{t\-1\}\), and produces logitsℓ​\(x,y1:t−1\)∈ℝ\|𝕍\|\\ell\(x,y\_\{1:t\-1\}\)\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}\. The base next\-token distribution isπ0\(yt∣x,y1:t−1\):=softmax\(ℓ\(x,y1:t−1\)\)yt\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\):=\\operatorname\{softmax\}\(\\ell\(x,y\_\{1:t\-1\}\)\)\_\{y\_\{t\}\}withsoftmax\(v\)y:=exp⁡\(vy\)∑y′∈𝕍exp⁡\(vy′\)\\operatorname\{softmax\}\(v\)\_\{y\}:=\\frac\{\\exp\(v\_\{y\}\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}\\exp\(v\_\{y^\{\\prime\}\}\)\},y∈𝕍y\\in\\mathbb\{V\}, and the trajectory law isp0​\(y1:T∣x\):=∏t=1Tπ0​\(yt∣x,y1:t−1\)p\_\{0\}\(y\_\{1:T\}\\mid x\):=\\prod\_\{t=1\}^\{T\}\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\. For a logit\-bias vectorδ∈ℝ\|𝕍\|\\delta\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}, the*fixed\-bias policy*isπδ\(yt∣x,y1:t−1\):=softmax\(ℓ\(x,y1:t−1\)\+δ\)yt\\pi\_\{\\delta\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\):=\\operatorname\{softmax\}\(\\ell\(x,y\_\{1:t\-1\}\)\+\\delta\)\_\{y\_\{t\}\}, with trajectory lawpδ​\(y1:T∣x\):=∏t=1Tπδ​\(yt∣x,y1:t−1\)p\_\{\\delta\}\(y\_\{1:T\}\\mid x\):=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\delta\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\. We assume full support:π0​\(yt∣x,y1:t−1\)\>0\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\>0for every reachable prefixy1:t−1y\_\{1:t\-1\}and every tokenyt∈𝕍y\_\{t\}\\in\\mathbb\{V\}\. This holds automatically for any softmax\-based model, since softmax outputs are strictly positive\. We use “trajectory” for the token sequencey1:Ty\_\{1:T\}in the theoretical development and “completion” for the same object in experimental and evaluation contexts; both names refer to the same generated token sequence\. Inside expectations and variances we writeYYfor the random trajectory drawn from the relevant law andy1:Ty\_\{1:T\}for a realized token sequence\.

##### Intervention notation\.

We writeY∼p0\(⋅∣x,do\(yt=y\)\)Y\\sim p\_\{0\}\\bigl\(\\cdot\\mid x,\\,\\mathrm\{do\}\(y\_\{t\}=y\)\\bigr\)to denote sampling a trajectory underp0\(⋅∣x\)p\_\{0\}\(\\cdot\\mid x\)with thett\-th token set toyyby*intervention*rather than conditioning\[[41](https://arxiv.org/html/2607.22837#bib.bib41)\]:y1:t−1∼p0\(⋅∣x\)y\_\{1:t\-1\}\\sim p\_\{0\}\(\\cdot\\mid x\),yt=yy\_\{t\}=yis fixed, andyt\+1:T∼p0\(⋅∣x,y1:t−1,y\)y\_\{t\+1:T\}\\sim p\_\{0\}\(\\cdot\\mid x,y\_\{1:t\-1\},y\)\. This differs from the conditional distributionp0\(⋅∣x,yt=y\)p\_\{0\}\(\\cdot\\mid x,y\_\{t\}=y\), which Bayes\-reweights the prefix byπ0​\(y∣x,y1:t−1\)\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)\.

Given a terminal reward functionr:𝒳×𝕍T→ℝr:\\mathcal\{X\}\\times\\mathbb\{V\}^\{T\}\\to\\mathbb\{R\}and a prompt\-conditional trajectory lawq\(⋅∣x\)q\(\\cdot\\mid x\)on𝕍T\\mathbb\{V\}^\{T\}, we define the reward functional

J​\(q\):=𝔼x∼ρ​\[𝔼Y∼q\(⋅∣x\)​\[r​\(x,Y\)\]\],J\(q\)\\;:=\\;\\mathbb\{E\}\_\{x\\sim\\rho\}\\\!\\left\[\\mathbb\{E\}\_\{Y\\sim q\(\\cdot\\mid x\)\}\\\!\\bigl\[r\(x,Y\)\\bigr\]\\right\],Throughout,𝔼^\\widehat\{\\mathbb\{E\}\}denotes an empirical average over samples\.

##### RLVR setting and accuracy reward\.

We primarily consider RL with verifiable rewards \(RLVR\): each promptxxhas a canonical correct answerm⋆m\_\{\\star\}, and a generated completiony1:Ty\_\{1:T\}can be automatically checked by extracting and canonicalizing its final answer\. Concretely, letmx​\(y1:T\)m\_\{x\}\(y\_\{1:T\}\)denote the answer extracted from the completiony1:Ty\_\{1:T\}for promptxx, represented in a canonical form so that symbolically equivalent answers are identified; if no valid answer can be extracted, we setmx​\(y1:T\)=⊥m\_\{x\}\(y\_\{1:T\}\)=\\bot\. A standard reward may be the correctness indicatorraccuracy:𝒳×𝕍T→\{0,1\}r\_\{\\mathrm\{accuracy\}\}:\\mathcal\{X\}\\times\\mathbb\{V\}^\{T\}\\to\\\{0,1\\\},

raccuracy​\(x,y1:T\):=𝟏​\{mx​\(y1:T\)=m⋆\},r\_\{\\mathrm\{accuracy\}\}\(x,y\_\{1:T\}\):=\\mathbf\{1\}\\\!\\left\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\right\\\},

##### KL\-regularized optimum\.

For regularization strengthτ\>0\\tau\>0, consider the problem

maxq𝔼x∼ρ\[𝔼Y∼q\(⋅∣x\)\[r\(x,Y\)\]−τKL\(q\(⋅∣x\)∥p0\(⋅∣x\)\)\],\\max\_\{q\}\\;\\mathbb\{E\}\_\{x\\sim\\rho\}\\\!\\left\[\\mathbb\{E\}\_\{Y\\sim q\(\\cdot\\mid x\)\}\[r\(x,Y\)\]\\;\-\\;\\tau\\,\\operatorname\{KL\}\\\!\\bigl\(q\(\\cdot\\mid x\)\\,\\\|\\,p\_\{0\}\(\\cdot\\mid x\)\\bigr\)\\right\],\(1\)where the maximum is over all families of prompt\-conditional trajectory lawsq\(⋅∣x\)q\(\\cdot\\mid x\)on𝕍T\\mathbb\{V\}^\{T\}\. Because the integrand decomposes into independent per\-prompt terms, the optimum is attained by solving each prompt separately\. By the Gibbs variational principle\[[6](https://arxiv.org/html/2607.22837#bib.bib6)\], the unique per\-prompt maximizer is \(see Appendix[G\.3](https://arxiv.org/html/2607.22837#A7.SS3)for a self\-contained proof; see alsoZiegler et al\. \[[63](https://arxiv.org/html/2607.22837#bib.bib63)\], Korbak et al\. \[[22](https://arxiv.org/html/2607.22837#bib.bib22)\], Rafailov et al\. \[[44](https://arxiv.org/html/2607.22837#bib.bib44)\], Zhou et al\. \[[62](https://arxiv.org/html/2607.22837#bib.bib62)\]for related derivations in the RLHF setting\)

pτ⋆​\(y1:T∣x\)=1Z​\(x\)​p0​\(y1:T∣x\)​er​\(x,y1:T\)/τ,Z​\(x\):=𝔼Y∼p0\(⋅∣x\)​\[er​\(x,Y\)/τ\]\.p^\{\\star\}\_\{\\tau\}\(y\_\{1:T\}\\mid x\)=\\frac\{1\}\{Z\(x\)\}\\,p\_\{0\}\(y\_\{1:T\}\\mid x\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\},\\qquad Z\(x\):=\\mathbb\{E\}\_\{Y\\sim p\_\{0\}\(\\cdot\\mid x\)\}\\\!\\left\[e^\{r\(x,Y\)/\\tau\}\\right\]\.\(2\)

##### Per\-step decomposition\.

The trajectory\-level KL decomposes into a sum of per\-step divergences: for any two autoregressive policiesπ,π′\\pi,\\pi^\{\\prime\},

KL\(pπ\(⋅∣x\)∥pπ′\(⋅∣x\)\)=𝔼Y∼pπ\[∑t=1TKL\(π\(⋅∣x,y1:t−1\)∥π′\(⋅∣x,y1:t−1\)\)\],\\operatorname\{KL\}\\\!\\bigl\(p\_\{\\pi\}\(\\cdot\\mid x\)\\,\\\|\\,p\_\{\\pi^\{\\prime\}\}\(\\cdot\\mid x\)\\bigr\)=\\mathbb\{E\}\_\{Y\\sim p\_\{\\pi\}\}\\\!\\left\[\\sum\_\{t=1\}^\{T\}\\operatorname\{KL\}\\\!\\bigl\(\\pi\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\,\\\|\\,\\pi^\{\\prime\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\bigr\)\\right\],by the chain rule for relative entropy\[[6](https://arxiv.org/html/2607.22837#bib.bib6)\]\. Applying this to the per\-prompt integrand in Equation \([1](https://arxiv.org/html/2607.22837#S3.E1)\), for each promptxxthe objective decomposes as

𝔼Y∼q\(⋅∣x\)\[r\(x,Y\)−τ∑t=1TKL\(q\(⋅∣x,y1:t−1\)∥π0\(⋅∣x,y1:t−1\)\)\],\\mathbb\{E\}\_\{Y\\sim q\(\\cdot\\mid x\)\}\\\!\\left\[r\(x,Y\)\-\\tau\\sum\_\{t=1\}^\{T\}\\operatorname\{KL\}\\\!\\bigl\(q\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\,\\\|\\,\\pi\_\{0\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\bigr\)\\right\],so optimizing over prompt\-conditional trajectory laws is equivalent to optimizing over autoregressive policies\. Define the*soft value*

Z​\(x,y1:t−1,y\):=𝔼yt\+1:T∼p0\(⋅∣x,y1:t−1,y\)​\[er​\(x,y1:T\(t,y\)\)/τ\],Z\(x,y\_\{1:t\-1\},y\):=\\mathbb\{E\}\_\{y\_\{t\+1:T\}\\sim p\_\{0\}\(\\cdot\\mid x,y\_\{1:t\-1\},y\)\}\\\!\\left\[e^\{r\(x,y\_\{1:T\}^\{\(t,y\)\}\)/\\tau\}\\right\],\(3\)wherey1:T\(t,y\):=\(y1:t−1,y,yt\+1:T\)y\_\{1:T\}^\{\(t,y\)\}:=\(y\_\{1:t\-1\},y,y\_\{t\+1:T\}\)\.

This is the expected exponentiated reward when tokenyyis chosen at stepttgiven promptxxand prefixy1:t−1y\_\{1:t\-1\}, and the remaining tokens are rolled out underp0p\_\{0\}\. Marginalizing Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\) over future tokensyt\+1:Ty\_\{t\+1:T\}and dividing by the prefix marginal yields the per\-step conditionals ofpτ⋆p^\{\\star\}\_\{\\tau\}\(see Appendix[G\.4](https://arxiv.org/html/2607.22837#A7.SS4)\):

πτ⋆\(yt∣x,y1:t−1\)=π0​\(yt∣x,y1:t−1\)​Z​\(x,y1:t−1,yt\)∑y′∈𝕍π0​\(y′∣x,y1:t−1\)​Z​\(x,y1:t−1,y′\)=softmax\(ℓ\(x,y1:t−1\)\+logZ\(x,y1:t−1,⋅\)\)yt\.\\pi^\{\\star\}\_\{\\tau\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)=\\frac\{\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\\,Z\(x,y\_\{1:t\-1\},y\_\{t\}\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}\\pi\_\{0\}\(y^\{\\prime\}\\mid x,y\_\{1:t\-1\}\)\\,Z\(x,y\_\{1:t\-1\},y^\{\\prime\}\)\}=\\operatorname\{softmax\}\\\!\\bigl\(\\ell\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\\bigr\)\_\{y\_\{t\}\}\.\(4\)Thus the optimal logit correction at stepttislog⁡Z​\(x,y1:t−1,⋅\)\\log Z\(x,y\_\{1:t\-1\},\\cdot\), up to an additive constant absorbed by the softmax\.

###### Proposition 1\.

Sampling a trajectoryy1:Ty\_\{1:T\}frompτ⋆p^\{\\star\}\_\{\\tau\}in Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\) is equivalent to sampling autoregressively withyt∼πτ⋆\(⋅∣x,y1:t−1\)y\_\{t\}\\sim\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)in Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\) fort=1,…,Tt=1,\\dots,T\. That is,pτ⋆​\(y1:T∣x\)=∏t=1Tπτ⋆​\(yt∣x,y1:t−1\)p^\{\\star\}\_\{\\tau\}\(y\_\{1:T\}\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi^\{\\star\}\_\{\\tau\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\.

###### Proof\.

Appendix[G\.4](https://arxiv.org/html/2607.22837#A7.SS4)shows that marginalizing Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\) yields the conditionals in Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\)\. The KL chain rule ensures that the trajectory\-level objective decomposes into per\-step terms, each uniquely maximized by Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\)\. The product of these per\-step conditionals recovers Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\)\. ∎

###### Corollary 2\(Exact factorization implies fixed\-bias optimality\)\.

Iflog⁡Z​\(x,y1:t−1,⋅\)=δ\+ct​\(x,y1:t−1\)​1\\log Z\(x,y\_\{1:t\-1\},\\cdot\)=\\delta\+c\_\{t\}\(x,y\_\{1:t\-1\}\)\\,\\mathbf\{1\}for allt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, promptsxxin the support ofρ\\rho, and reachable prefixesy1:t−1y\_\{1:t\-1\}\(with fixedδ∈ℝ\|𝕍\|\\delta\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}\), thenpδ\(⋅∣x\)=pτ⋆\(⋅∣x\)p\_\{\\delta\}\(\\cdot\\mid x\)=p^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x\)for every suchxx\.

###### Proof\.

Immediate from Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\), softmax shift invariance, and Proposition[1](https://arxiv.org/html/2607.22837#Thmtheorem1); see Appendix[G\.5](https://arxiv.org/html/2607.22837#A7.SS5)\. ∎

## 4Method

We keep the base language model entirely frozen and steer its outputs by adding a single, context\-independent logit\-bias vectorδ∈ℝ\|𝕍\|\\delta\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}to the logits at every decoding step\.

We seek a logit\-bias vectorδ\\deltathat approximates the unconstrained KL\-regularized optimum \([1](https://arxiv.org/html/2607.22837#S3.E1)\)\. By Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\), the optimal logit correction at stepttislog⁡Z​\(x,y1:t−1,⋅\)\\log Z\(x,y\_\{1:t\-1\},\\cdot\), up to an additive constant absorbed by the softmax\. This correction depends on the prompt and prefix, whereas a fixed logit\-bias vector applies the same vector everywhere\. A singleδ\\deltacan recover the optimum exactly whenlog⁡Z​\(x,y1:t−1,y\)\\log Z\(x,y\_\{1:t\-1\},y\)factorizes asδ​\(y\)\+ct​\(x,y1:t−1\)\\delta\(y\)\+c\_\{t\}\(x,y\_\{1:t\-1\}\)for all reachable prefixes—that is, when the token\-to\-token pattern of the ideal correction is the same everywhere, up to a prefix\-dependent baseline\. Section[5](https://arxiv.org/html/2607.22837#S5)formalizes this asε\\varepsilon\-approximate factorization and bounds the resulting reward gap\.

### 4\.1Population target and sampling procedure

Under the factorization assumption, the population\-level bias isδ​\(y\)=log⁡Zavg​\(y\)\+c\\delta\(y\)=\\log Z\_\{\\mathrm\{avg\}\}\(y\)\+c, whereccis a centering constant and

Zavg​\(y\):=𝔼t∼Unif​\{1,…,T\},x∼ρY∼p0\(⋅∣x,do\(yt=y\)\)​\[er​\(x,Y\)/τ\]=𝔼t∼Unif​\{1,…,T\},x∼ρy1:t−1∼p0\(⋅∣x\)​\[Z​\(x,y1:t−1,y\)\],Z\_\{\\mathrm\{avg\}\}\(y\)\\;:=\\;\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}t\\sim\\mathrm\{Unif\}\\\{1,\\dots,T\\\},\\;x\\sim\\rho\\\\ Y\\sim p\_\{0\}\(\\cdot\\mid x,\\,\\mathrm\{do\}\(y\_\{t\}=y\)\)\\end\{subarray\}\}\\\!\\left\[e^\{r\(x,Y\)/\\tau\}\\right\]\\;=\\;\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}t\\sim\\mathrm\{Unif\}\\\{1,\\dots,T\\\},\\;x\\sim\\rho\\\\ y\_\{1:t\-1\}\\sim p\_\{0\}\(\\cdot\\mid x\)\\end\{subarray\}\}\\\!\\left\[Z\(x,y\_\{1:t\-1\},y\)\\right\],\(5\)wheredo​\(yt=y\)\\mathrm\{do\}\(y\_\{t\}=y\)denotes Pearl\-style intervention at positiontt:y1:t−1∼p0\(⋅∣x\)y\_\{1:t\-1\}\\sim p\_\{0\}\(\\cdot\\mid x\),yt=yy\_\{t\}=yis fixed, andyt\+1:T∼p0\(⋅∣x,y1:t−1,y\)y\_\{t\+1:T\}\\sim p\_\{0\}\(\\cdot\\mid x,y\_\{1:t\-1\},y\);ZZis the soft value from Equation \([3](https://arxiv.org/html/2607.22837#S3.E3)\)\. ThusZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)is the expected exponentiated reward of a trajectory in which tokenyyis inserted at a uniformly random position and the prefix/suffix are drawn from the base model\.

The data we use to estimateZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)comes from on\-policy rollouts: for each prompt, sample a full trajectoryy1:T∼p0\(⋅∣x\)y\_\{1:T\}\\sim p\_\{0\}\(\\cdot\\mid x\)and drawP∈\{1,…,T\}P\\in\\\{1,\\dots,T\\\}time\-steps uniformly without replacement from\{1,…,T\}\\\{1,\\dots,T\\\}\(Algorithm[1](https://arxiv.org/html/2607.22837#alg1)\)\. Concrete estimator forZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)is given in Section[4\.2](https://arxiv.org/html/2607.22837#S4.SS2)\.

##### On the choice ofPP\.

Samples within a trajectory share the prefix, suffix, and reward, soP\>1P\>1makes them non\-i\.i\.d\. across positions\. Empirically, this dependence is negligible, while thePP\-fold data reuse—at fixed rollout budget, the dominant compute cost—materially cuts per\-token variance\.

Algorithm 1On\-policy rollout data generation1:prompts

\{x\(n\)\}n=1N\\\{x^\{\(n\)\}\\\}\_\{n=1\}^\{N\}, base model

π0\\pi\_\{0\}, horizon

TT, reward oracle

r​\(⋅,⋅\)r\(\\cdot,\\cdot\)
2:number of sampled positions per trajectory

P∈\{1,…,T\}P\\in\\\{1,\\dots,T\\\}
3:for

n=1,…,Nn=1,\\dots,Ndo

4:Generate trajectory

y1:T\(n\)∼p0\(⋅∣x\(n\)\)y\_\{1:T\}^\{\(n\)\}\\sim p\_\{0\}\(\\cdot\\mid x^\{\(n\)\}\)and query reward

r\(n\):=r​\(x\(n\),y1:T\(n\)\)r^\{\(n\)\}:=r\\\!\\left\(x^\{\(n\)\},y\_\{1:T\}^\{\(n\)\}\\right\)
5:Sample positions

\{t\(n,p\)\}p=1P\\\{t^\{\(n,p\)\}\\\}\_\{p=1\}^\{P\}uniformly*without replacement*from

\{1,…,T\}\\\{1,\\ldots,T\\\}
6:For

p=1,…,Pp=1,\\dots,P, set

x\(n,p\):=x\(n\)x^\{\(n,p\)\}:=x^\{\(n\)\},

y1:T\(n,p\):=y1:T\(n\)y\_\{1:T\}^\{\(n,p\)\}:=y^\{\(n\)\}\_\{1:T\}, and

r\(n,p\):=r\(n\)r^\{\(n,p\)\}:=r^\{\(n\)\}
7:endfor

8:return

𝒟=\{\(x\(n,p\),y1:T\(n,p\),t\(n,p\),r\(n,p\)\)\}n,p=1N,P\\mathcal\{D\}=\\bigl\\\{\\bigl\(x^\{\(n,p\)\},\\,y\_\{1:T\}^\{\(n,p\)\},\\,t^\{\(n,p\)\},\\,r^\{\(n,p\)\}\\bigr\)\\bigr\\\}\_\{n,p=1\}^\{N,P\}

### 4\.2Logit\-bias estimation

Given the on\-policy dataset𝒟\\mathcal\{D\}generated by Algorithm[1](https://arxiv.org/html/2607.22837#alg1), we estimateZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)using the inverse propensity scoring \(IPS\) estimator

Z^​\(y\):=1N​P​∑\(x,y1:T,t,r\)∈𝒟𝟏​\{yt=y\}​exp⁡\(r/τ\)π0​\(y∣x,y1:t−1\)\.\\widehat\{Z\}\(y\):=\\frac\{1\}\{NP\}\\sum\_\{\(x,\\,y\_\{1:T\},\\,t,\\,r\)\\,\\in\\,\\mathcal\{D\}\}\\frac\{\\mathbf\{1\}\\\!\\left\\\{y\_\{t\}=y\\right\\\}\\,\\exp\\\!\\left\(r/\\tau\\right\)\}\{\\pi\_\{0\}\\\!\\left\(y\\mid x,\\,y\_\{1:t\-1\}\\right\)\}\.\(6\)
This estimator is unbiased:𝔼​\[Z^​\(y\)\]=Zavg​\(y\)\\mathbb\{E\}\[\\widehat\{Z\}\(y\)\]=Z\_\{\\mathrm\{avg\}\}\(y\)\(Proposition[6](https://arxiv.org/html/2607.22837#Thmtheorem6), proved in Appendix[E](https://arxiv.org/html/2607.22837#A5)\)\.

In practice, we convertZ^\\widehat\{Z\}into a logit\-bias vector using additive smoothing:δ^​\(y\)=log⁡\(α\+Z^​\(y\)\)−c​\(α\),c​\(α\):=1\|𝕍\|​∑y′∈𝕍log⁡\(α\+Z^​\(y′\)\)\.\\widehat\{\\delta\}\(y\)=\\log\\\!\\bigl\(\\alpha\+\\widehat\{Z\}\(y\)\\bigr\)\-c\(\\alpha\),\\qquad c\(\\alpha\):=\\frac\{1\}\{\|\\mathbb\{V\}\|\}\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}\\log\\\!\\bigl\(\\alpha\+\\widehat\{Z\}\(y^\{\\prime\}\)\\bigr\)\.The smoothing constantα\>0\\alpha\>0acts as a pseudocount\[[26](https://arxiv.org/html/2607.22837#bib.bib26),[4](https://arxiv.org/html/2607.22837#bib.bib4)\]: it shrinks weak evidence toward a common baseline, preventslog⁡0\\log 0, and bounds the local sensitivity of the log transform by1/α1/\\alpha\. We chooseα\\alphaby validation performance\.

## 5Theoretical Analysis

We establish that the closed\-form estimator from Section[4](https://arxiv.org/html/2607.22837#S4)is near\-optimal when the soft valuelog⁡Z​\(x,y1:t−1,y\)\\log Z\(x,y\_\{1:t\-1\},y\)approximately factorizes\. Full proofs appear in Appendix[G](https://arxiv.org/html/2607.22837#A7)\.

### 5\.1Approximate factorization

As in Section[4](https://arxiv.org/html/2607.22837#S4), the ideal logit correction at stepttislog⁡Z​\(x,y1:t−1,⋅\)\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\(up to a constant\), which a fixed logit\-bias vectorδ\\deltarepresents exactly under the factorization assumptionlog⁡Z​\(x,y1:t−1,y\)=δ​\(y\)\+ct​\(x,y1:t−1\)\\log Z\(x,y\_\{1:t\-1\},y\)=\\delta\(y\)\+c\_\{t\}\(x,y\_\{1:t\-1\}\)for all reachable prefixes; we now relax this to an approximate version\.

###### Definition 3\(ε\\varepsilon\-Approximate Factorization\)\.

A logit\-bias vectorδ^∈ℝ\|𝕍\|\\widehat\{\\delta\}\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}satisfiesε\\varepsilon\-approximate factorization if, for everyt≤Tt\\leq T, every promptxxin the support ofρ\\rho, and every prefixy1:t−1y\_\{1:t\-1\}reachable underpτ⋆p^\{\\star\}\_\{\\tau\}at timett, there existsct​\(x,y1:t−1\)∈ℝc\_\{t\}\(x,y\_\{1:t\-1\}\)\\in\\mathbb\{R\}with

‖log⁡Z​\(x,y1:t−1,⋅\)−\(δ^\+ct​\(x,y1:t−1\)​𝟏\)‖2≤ε\.\\bigl\\\|\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\-\(\\widehat\{\\delta\}\+c\_\{t\}\(x,y\_\{1:t\-1\}\)\\mathbf\{1\}\)\\bigr\\\|\_\{2\}\\leq\\varepsilon\.

Whenε=0\\varepsilon=0this is exact factorization; the closed\-form estimator in Section[4\.1](https://arxiv.org/html/2607.22837#S4.SS1)targets exactly this regime\.

### 5\.2Reward gap bound

###### Theorem 4\(Reward gap\)\.

Assumer:𝒳×𝕍T→\[0,R\]r:\\mathcal\{X\}\\times\\mathbb\{V\}^\{T\}\\to\[0,R\]withR\>0R\>0andτ\>0\\tau\>0\. Let

V0:=𝔼x∼ρ​\[VarY∼p0\(⋅∣x\)⁡\(r​\(x,Y\)\)\]V\_\{0\}\\;:=\\;\\mathbb\{E\}\_\{x\\sim\\rho\}\\\!\\left\[\\operatorname\{Var\}\_\{Y\\sim p\_\{0\}\(\\cdot\\mid x\)\}\\\!\\bigl\(r\(x,Y\)\\bigr\)\\right\]denote the expected reward variance under the base model\. Ifδ^∈ℝ\|𝕍\|\\widehat\{\\delta\}\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}satisfiesε\\varepsilon\-approximate factorization assumption \(Definition[3](https://arxiv.org/html/2607.22837#Thmtheorem3)\) for every prompt in the support ofρ\\rho, then

J​\(pδ^\)−J​\(p0\)≥V0R​\(1−e−R/τ\)⏟improvement of​pτ⋆​over​p0−R​ε​T8⏟cost of fixed\-bias approximation\.J\(p\_\{\\widehat\{\\delta\}\}\)\-J\(p\_\{0\}\)\\;\\geq\\;\\underbrace\{\\frac\{V\_\{0\}\}\{R\}\\Bigl\(1\-e^\{\-R/\\tau\}\\Bigr\)\}\_\{\\text\{improvement of \}p^\{\\star\}\_\{\\tau\}\\text\{ over \}p\_\{0\}\}\\;\-\\;\\underbrace\{R\\,\\varepsilon\\sqrt\{\\frac\{T\}\{8\}\}\}\_\{\\text\{cost of fixed\-bias approximation\}\}\.\(7\)

###### Proof\.

See Appendix[G\.6](https://arxiv.org/html/2607.22837#A7.SS6)\. ∎

The first term is a lower bound onJ​\(pτ⋆\)−J​\(p0\)J\(p^\{\\star\}\_\{\\tau\}\)\-J\(p\_\{0\}\), the reward gain of the unconstrained KL\-regularized optimum over the base model; it grows with the expected reward varianceV0V\_\{0\}and is strictly positive wheneverV0\>0V\_\{0\}\>0, sinceR\>0R\>0impliese−R/τ<1e^\{\-R/\\tau\}<1\. The second term is the price of restricting to a context\-independent logit\-bias vector; it grows with the approximation errorε\\varepsilonand the horizonTT\. Logit\-bias steering strictly improves over the base model whenever the expected reward variance is large enough to dominate the approximation error\. The bound applies to*any*δ^\\widehat\{\\delta\}satisfying Definition[3](https://arxiv.org/html/2607.22837#Thmtheorem3)\.

##### Reward normalization and the role ofRR\.

Both terms of the bound in Theorem[4](https://arxiv.org/html/2607.22837#Thmtheorem4)depend on the reward rangeRR: the gain scales asV0/RV\_\{0\}/Rwhile the error scales asR​ε​T/8R\\,\\varepsilon\\sqrt\{T/8\}\. KeepingRRsmall is therefore essential for a tight guarantee\. Adding a constant torrshifts neitherpτ⋆p^\{\\star\}\_\{\\tau\}nor the reward gap, so the binding quantity is the*range*ofrr, not its location\. For pure binary accuracy,R=1R=1\. Adding an unnormalized length penalty−L​\(y\)\-L\(y\)would inflate the range toLmax\+1=2,049L\_\{\\max\}\+1=2\{,\}049, degrading the bound by a factor exceeding10310^\{3\}\. Normalizing byLmaxL\_\{\\max\}yieldsr​\(x,y1:T\)=𝟏​\{mx​\(y1:T\)=m⋆\}−L​\(y1:T\)/Lmaxr\(x,y\_\{1:T\}\)=\\mathbf\{1\}\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\\}\-L\(y\_\{1:T\}\)/L\_\{\\max\}with range22, preserving a bound of the same order as the binary case\. This motivates the length normalization used in the experiments\.

## 6Experiments

### 6\.1Experimental setup

##### Models\.

Accuracy uses Meta\-Llama\-3\.1\-8B\-Instruct\[[14](https://arxiv.org/html/2607.22837#bib.bib14)\]as the base model; the compression objective adds Gemma\-2\-9B\-IT\[[12](https://arxiv.org/html/2607.22837#bib.bib12)\]and Qwen3\-4B\[[60](https://arxiv.org/html/2607.22837#bib.bib60)\]\.

##### Tasks and data\.

We evaluate on MATH, GSM8K, and GPQA\-main\[[15](https://arxiv.org/html/2607.22837#bib.bib15),[9](https://arxiv.org/html/2607.22837#bib.bib9),[5](https://arxiv.org/html/2607.22837#bib.bib5),[39](https://arxiv.org/html/2607.22837#bib.bib39),[46](https://arxiv.org/html/2607.22837#bib.bib46),[20](https://arxiv.org/html/2607.22837#bib.bib20)\]with the default lm\-evaluation\-harness prompts\[[3](https://arxiv.org/html/2607.22837#bib.bib3),[10](https://arxiv.org/html/2607.22837#bib.bib10)\]\. MATH and GSM8K use the standard HuggingFace train/test splits with validation sampled from train; GPQA\-main is split 80/10/10\. For GPQA\-main we additionally shuffle answer choices and require explanation\-first, answer\-last formatting, so sampled positions for bias estimation are not tied to a fixed choice index \(Appendix[D](https://arxiv.org/html/2607.22837#A4)\)\.

##### Rewards\.

For accuracy, the logit\-bias method is learned from the log\-indicator variant of the RLVR accuracy rewardraccuracyr\_\{\\mathrm\{accuracy\}\}\(Section[3](https://arxiv.org/html/2607.22837#S3)\),

rlog\-accuracy​\(x,y1:T\):=log⁡𝟏​\{mx​\(y1:T\)=m⋆\}=\{0,mx​\(y1:T\)=m⋆,−∞,otherwise\.r\_\{\\text\{log\-accuracy\}\}\(x,y\_\{1:T\}\):=\\log\\mathbf\{1\}\\\!\\left\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\right\\\}=\\begin\{cases\}0,&m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\},\\\\ \-\\infty,&\\text\{otherwise\}\.\\end\{cases\}With the convention thatlog⁡0=−∞\\log 0=\-\\infty, this encodes the same correctness check asraccuracyr\_\{\\mathrm\{accuracy\}\}in the log\-reward form expected by the KL\-tilted objective \([1](https://arxiv.org/html/2607.22837#S3.E1)\): sinceerlog\-accuracy/τ=𝟏​\{mx​\(y1:T\)=m⋆\}e^\{r\_\{\\text\{log\-accuracy\}\}/\\tau\}=\\mathbf\{1\}\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\\}for everyτ\>0\\tau\>0, the estimatorZ^\\widehat\{Z\}reduces to an accuracy\-weighted count andτ\\taubecomes inert, eliminating the need to tune it\. Answers are extracted and canonicalized by the lm\-evaluation\-harness verifier; the reward is binary with no partial credit, and an unextractable answer \(mx​\(y1:T\)=⊥m\_\{x\}\(y\_\{1:T\}\)=\\bot\) counts as incorrect\.

We also evaluate the compression objective as a second reward\-driven adaptation target, useful both practically and diagnostically\. Practically, shorter completions reduce serving cost and latency, and are often desirable when users need concise answers rather than full reasoning traces\. Diagnostically, it stress\-tests whether the fixed logit\-bias vectorδ\\deltacan learn reusable, context\-independent answer\-finalization and stopping patterns rather than new problem\-specific reasoning\. For the compression reward, the logit\-bias method uses

rlength​\(y\)=ln⁡\(T/length⁡\(y\)\)\.r\_\{\\mathrm\{length\}\}\(y\)=\\ln\\\!\\left\(T/\\operatorname\{length\}\(y\)\\right\)\.\(8\)Rollout\-generation and LoRA settings are reported in Appendix[D](https://arxiv.org/html/2607.22837#A4)\.

##### Baselines\.

We compare against three baselines: \(i\) the base model, \(ii\) a 32\-cluster tied logit\-bias baseline that groups tokens with similar surface strings viakk\-means and shares one bias per cluster \(Appendix[D](https://arxiv.org/html/2607.22837#A4)\), and \(iii\) a LoRA\-GRPO baseline\[[19](https://arxiv.org/html/2607.22837#bib.bib19),[51](https://arxiv.org/html/2607.22837#bib.bib51)\]that fine\-tunes the same base model with checkpoint selection on validation\. LoRA\-GRPO uses the binary accuracy rewardraccuracyGRPO​\(x,y1:T\)=𝟏​\{mx​\(y1:T\)=m⋆\}r^\{\\mathrm\{GRPO\}\}\_\{\\mathrm\{accuracy\}\}\(x,y\_\{1:T\}\)=\\mathbf\{1\}\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\\}in the accuracy setting, and the accuracy\-gated compression rewardrlengthGRPO​\(x,y1:T\)=𝟏​\{mx​\(y1:T\)=m⋆\}⋅T/length⁡\(y1:T\)r^\{\\mathrm\{GRPO\}\}\_\{\\mathrm\{length\}\}\(x,y\_\{1:T\}\)=\\mathbf\{1\}\\\{m\_\{x\}\(y\_\{1:T\}\)=m\_\{\\star\}\\\}\\cdot T/\\operatorname\{length\}\(y\_\{1:T\}\)in the compression experiment\. Construction and training details are in Appendix[D](https://arxiv.org/html/2607.22837#A4)\.

##### Hyperparameters and evaluation\.

For accuracy runs,τ\\tauis inert underrlog\-accuracyr\_\{\\text\{log\-accuracy\}\}, so we tune onlyα∈\[0\.005,0\.11\]\\alpha\\in\[0\.005,0\.11\]on validation\. In the compression experiment, we tune bothα∈\[0\.005,0\.11\]\\alpha\\in\[0\.005,0\.11\]andτ∈\[0\.5,1\.5\]\\tau\\in\[0\.5,1\.5\]\. Accuracy runs select the configuration maximizing validation exact\-match accuracy; compression experiment runs treat the two as a Pareto tradeoff and pick the shortest completions that preserve validation accuracy relative to the base model\. We apply learned logit\-bias vectors via the vLLM logits\-processor API under greedy decoding, reporting exact\-match accuracy and completion length in tokens\. Confidence intervals are computed using percentile bootstrap resampling; see Appendix[D\.8](https://arxiv.org/html/2607.22837#A4.SS8)\.

### 6\.2Main results

Table 1:Exact\-match accuracy \(%\) on MATH \(n=5000n\{=\}5000\), GSM8K \(n=1319n\{=\}1319\), and GPQA \(n=45n\{=\}45\) under the accuracy\-reward setup\. Each cell is empirical accuracy with a percentile bootstrap 95% confidence interval \(10,000 resamples\), shown as±\\pmpercentage\-point offsets\. Higher is better\.Table 2:Mean completion length in tokens on MATH for Gemma\-9B, Llama\-8B, and Qwen3\-4B under Base, full\-vocabulary logit bias, and LoRA\-GRPO\. Each cell reports the empirical mean with a percentile bootstrap 95% confidence interval \(10,000 resamples\) fromn=5000n=5000examples, shown as±\\pmoffsets from the mean\. Lower is better\.9509508508507507506506500\.450\.450\.50\.50\.550\.55BLτ=1\.5\\tau=1\.5τ=1\.2\\tau=1\.2τ=1\.0\\tau=1\.0τ=0\.9\\tau=0\.9τ=0\.8\\tau=0\.8τ=0\.7\\tau=0\.7τ=0\.6\\tau=0\.6τ=0\.5\\tau=0\.5Completion length in tokensaccuracyFigure 2:Validation accuracy–completion\-length Pareto frontier acrossτ\\tauon MATH \(Llama\-3\.1\-8B\-Instruct\)\. Higher accuracy and shorter completions are preferred; horizontal and vertical lines mark the base model\. Lowerτ\\tauyields stronger compression under the compression reward, exposing the model\-selection tradeoff\.##### Logit\-bias statistics\.

Table[3](https://arxiv.org/html/2607.22837#S6.T3)summarizes the learned logit\-bias vectorδ^\\widehat\{\\delta\}after centering to zero mean\. As a sparsity descriptor we report the*selected\-coordinate fraction*: the fraction of vocabulary coordinates whose absolute bias exceeds the*effect\-size gate*\|δ^​\(a\)\|≥5​median⁡\|δ^\|\|\\widehat\{\\delta\}\(a\)\|\\geq 5\\,\\operatorname\{median\}\|\\widehat\{\\delta\}\|\. The qualitative analysis \(Section[7](https://arxiv.org/html/2607.22837#S7), Appendices[B](https://arxiv.org/html/2607.22837#A2)and[C](https://arxiv.org/html/2607.22837#A3)\) instead reads each vector’s realized decode\-time effect through the per\-token*intervention score*SS\(Appendix[A\.1](https://arxiv.org/html/2607.22837#A1.SS1)\), splitting the visited vocabulary by the sign ofSSrather than thresholding the raw magnitude ofδ^\\widehat\{\\delta\}\.

Table 3:Summary statistics of the learned logit\-bias vectorδ^\\widehat\{\\delta\}after centering to zero mean\. The selected\-coordinate fraction counts coordinates satisfying the effect\-size gate\|δ^​\(a\)\|≥5​median⁡\|δ^\|\|\\widehat\{\\delta\}\(a\)\|\\geq 5\\,\\operatorname\{median\}\|\\widehat\{\\delta\}\|\.Table[3](https://arxiv.org/html/2607.22837#S6.T3)shows that only a small selected\-coordinate fraction passes the effect\-size gate, suggesting sparse token\-level nudges toward answer boundaries, mathematical formatting, and compact finalization rather than broad retuning\.

##### Accuracy results\.

Table[1](https://arxiv.org/html/2607.22837#S6.T1)evaluates logit\-bias adaptation when the reward is instantiated as answer accuracy\. Full\-vocabulary logit bias gives small but consistent gains over the base model on all benchmarks, most clearly on MATH \(30\.30→33\.1430\.30\\\!\\to\\\!33\.14\) and GPQA \(22\.20→28\.9022\.20\\\!\\to\\\!28\.90\)\. In contrast, the 32\-cluster baseline stays close to the base model, suggesting that the useful signal is token\-specific rather than a coarse frequency shift\. LoRA\-GRPO remains stronger by adapting context\-dependent representations across the network, whereas logit bias is limited to a fixed vocabulary\-level shift\.

##### Compression results\.

Table[2](https://arxiv.org/html/2607.22837#S6.T2)evaluates logit\-bias adaptation under the compression reward, constrained to preserve answer quality: we sweepτ\\tauon validation and keep short completions only among settings that preserve base\-model accuracy\. The effect is modest but useful—the fixed logit\-bias vectorδ\\deltashortens Gemma and Llama outputs, with little effect on Qwen3\-4B—and Figure[2](https://arxiv.org/html/2607.22837#S6.F2)shows the accuracy–completion\-length frontier, where loweringτ\\taufirst shortens completions without hurting accuracy and eventually causes early stopping and accuracy loss\. LoRA\-GRPO compresses substantially more by changing context\-dependent reasoning and stopping behavior\.

## 7Discussion

##### Summary of findings\.

Across both adaptation targets, the fixed logit\-bias vectorδ\\deltaprovides lightweight black\-box steering of the base model: under the accuracy reward it yields small but consistent gains \(Table[1](https://arxiv.org/html/2607.22837#S6.T1)\), and under the compression reward it traces an accuracy–completion\-length tradeoff rather than unconstrained reduction of completion length in tokens \(Table[2](https://arxiv.org/html/2607.22837#S6.T2), Figure[2](https://arxiv.org/html/2607.22837#S6.F2)\)\. Because the same token\-level correction applies at every prefix, the bias steers reusable formatting, finalization, and stopping patterns but cannot adapt context\-dependent reasoning—the gap where LoRA\-GRPO remains stronger \(Section[8](https://arxiv.org/html/2607.22837#S8)\)\.

##### Structure of the learned biases\.

To read each vector’s structure we summarize its decode\-time effect by the per\-token*intervention score*S​\(y\)S\(y\): the average log\-probability shift the bias induces on tokenyyat the positions where the fixed\-bias policy generates it, soS​\(y\)\>0S\(y\)\>0\(<0<0\) marks tokens it makes more \(less\) likely \(Appendix[A\.1](https://arxiv.org/html/2607.22837#A1.SS1)\)\. Splitting the visited vocabulary by the sign ofSSand grouping each side into semantic families gives the analyses in Appendices[B](https://arxiv.org/html/2607.22837#A2)and[C](https://arxiv.org/html/2607.22837#A3), with complete suppressed\-side listings provided in the accompanying code repository\. At a high level, the accuracy bias promotes the surface and discourse regularities of correct solutions \(layout and math delimiters, solution openers, digits, and a decisive stop string\), while its suppressed side acts as an anti\-drift prior against question\-restatement, lesson/tutorial mode, scraped page furniture, and invented problem\-specific detail—favoring a clean, staged path to the answer rather than steering the mathematics itself\. The compression bias is markedly*two\-sided*: a small, sharp promoted set of stop, boundary, and compact\-format tokens against a broad suppressed set of explanation, derivation, and continuation tokens, so its main risk is*over\-compression*\(Section[8](https://arxiv.org/html/2607.22837#S8)\)\.

##### Decode\-time mechanism and fixed\-bias projection\.

These patterns reflect a single decode\-time mechanism: the method projects a reward\-tilted target policy onto a highly restricted class—policies that add the same vector to the vocabulary logits at every prefix—which is effective when the ideal correctionlog⁡Z​\(x,y<t,⋅\)\\log Z\(x,y\_\{<t\},\\cdot\)is approximately token\-specific and stable across contexts, as it is for formatting, termination, answer\-surface regularities, and verbosity control\. Applied at every step, these local nudges accumulate into trajectory\-level changes that extract useful task signal from black\-box rollouts, explaining why the bias improves over the base model \(Table[1](https://arxiv.org/html/2607.22837#S6.T1)\) and reduces completion length in tokens \(Table[2](https://arxiv.org/html/2607.22837#S6.T2)\); Figure[2](https://arxiv.org/html/2607.22837#S6.F2)shows the same mechanism at the hyperparameter level, with stronger compression moving along the accuracy–completion\-length frontier rather than minimizing length unconditionally\.

## 8Limitations

##### Expressivity and access–performance tradeoff\.

This restriction is also the central limitation\. The fixed logit\-bias vectorδ\\deltacannot rewrite reasoning, encode conditional rules, or learn prompt\-specific corrections \(e\.g\., which theorem or calculation is needed, which value is correct, or which steps are safe to omit\); it can promote formatting, digits, and finalization markers and push down drift and continuation tokens, but it cannot supply the correct reasoning content\. Because the same correction applies at every prefix, this context\-blindness also makes compression prone to*over\-compression*, suppressing useful verification or necessary derivation along with genuine verbosity\. This is the gap LoRA\-GRPO closes in Tables[1](https://arxiv.org/html/2607.22837#S6.T1)and[2](https://arxiv.org/html/2607.22837#S6.T2): weight\-space adaptation changes internal, context\-dependent behavior, whereas the fixed logit\-bias vectorδ\\deltaapplies the same token\-level correction at every prefix\. Learned logit bias is thus best viewed not as a replacement for fine\-tuning, but as a lightweight black\-box primitive when gradients, weights, and provider\-side training are unavailable\.

##### Estimation limits\.

Learned logit bias is also constrained by estimation\. The IPS estimator can be high\-variance for rare tokens, additive smoothing introduces shrinkage, and finite rollout budgets leave many vocabulary coordinates weakly supported\. As a result, some promoted coordinates are spurious rare\-token or dataset artifacts rather than meaningful steering directions—our intervention\-score analysis flags several such tokens \(Appendices[B](https://arxiv.org/html/2607.22837#A2)and[C](https://arxiv.org/html/2607.22837#A3); full suppressed\-side listings in the accompanying code repository\)—though they are often inert in practice, since the base model assigns them negligible probability in mathematical contexts\. This highlights the need for better smoothing and finite\-sample analysis, especially for objectives where rollouts give weak evidence about which local token choices would have improved the final reward\.

## 9Conclusion

We adapted frozen language models under black\-box constraints by learning a single, context\-independent logit\-bias vectorδ\\deltafrom rollouts, rewards, and token probabilities, applied at decoding\. From a KL\-regularized objective, we derived the prefix\-dependent optimal correction, characterized when a fixed logit\-bias vector approximates it, and obtained a closed\-form IPS estimator with a reward\-gap bound\.

Empirically, full\-vocabulary logit bias improves over the base model on accuracy benchmarks without gradient access or weight updates, and supports the compression objective by learning reusable answer\-finalization and stop\-boundary patterns, though LoRA\-GRPO remains substantially stronger when weight\-space training is available\. Overall, learned logit bias is a lightweight adaptation primitive between prompt engineering and fine\-tuning; future work should strengthen finite\-sample theory, improve rare\-token smoothing, extend to context\-dependent biases, and broaden safety, privacy, fairness, and user\-facing evaluation\.

## Acknowledgments

This work is partially supported by the Israeli Science Foundation \(ISF\) grant no 4032/25\.

## References

- Ben Zaken et al\. \[2022\]Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel\.Bitfit: Simple parameter\-efficient fine\-tuning for transformer\-based masked language\-models\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*, 2022\.
- Beurer\-Kellner et al\. \[2024\]Luca Beurer\-Kellner, Marc Fischer, and Martin Vechev\.Guiding llms the right way: Fast, non\-invasive constrained generation\.*arXiv preprint arXiv:2403\.06988*, 2024\.
- Biderman et al\. \[2024\]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y\. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A\. Wang, Genta Indra Winata, Francois Yvon, and Andy Zou\.Lessons from the trenches on reproducible evaluation of language models\.*arXiv preprint arXiv:2405\.14782*, 2024\.doi:10\.48550/arXiv\.2405\.14782\.URL[https://arxiv\.org/abs/2405\.14782](https://arxiv.org/abs/2405.14782)\.
- Chen and Goodman \[1999\]Stanley F\. Chen and Joshua Goodman\.An empirical study of smoothing techniques for language modeling\.*Computer Speech & Language*, 13\(4\):359–394, 1999\.doi:10\.1006/csla\.1999\.0128\.
- Cobbe et al\. \[2021\]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.doi:10\.48550/arXiv\.2110\.14168\.URL[https://arxiv\.org/abs/2110\.14168](https://arxiv.org/abs/2110.14168)\.
- Cover and Thomas \[2006\]Thomas M\. Cover and Joy A\. Thomas\.*Elements of Information Theory*\.Wiley, 2nd edition, 2006\.
- Dathathri et al\. \[2020\]Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu\.Plug and play language models: A simple approach to controlled text generation\.In*International Conference on Learning Representations*, 2020\.
- Deng et al\. \[2022\]Mingkai Deng, Jianyu Wang, Cheng\-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu\.RLPrompt: Optimizing discrete text prompts with reinforcement learning\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 3369–3391, Abu Dhabi, United Arab Emirates, December 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.emnlp\-main\.222\.URL[https://aclanthology\.org/2022\.emnlp\-main\.222/](https://aclanthology.org/2022.emnlp-main.222/)\.
- EleutherAI \[2026a\]EleutherAI\.Eleutherai/hendrycks\_math\.[https://huggingface\.co/datasets/EleutherAI/hendrycks\_math](https://huggingface.co/datasets/EleutherAI/hendrycks_math), 2026a\.Accessed: 2026\-05\-04\.
- EleutherAI \[2026b\]EleutherAI\.Eleutherai/lm\-evaluation\-harness\.[https://github\.com/EleutherAI/lm\-evaluation\-harness](https://github.com/EleutherAI/lm-evaluation-harness), 2026b\.Accessed: 2026\-05\-07\.
- Fireworks AI \[2026\]Fireworks AI\.Fireworks ai chat completions api \(logit\_bias\)\.[https://docs\.fireworks\.ai/api\-reference/post\-chatcompletions](https://docs.fireworks.ai/api-reference/post-chatcompletions), 2026\.Accessed: 2026\-05\-07\.
- Gemma Team et al\. \[2024\]Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, et al\.Gemma 2: Improving open language models at a practical size\.*arXiv preprint arXiv:2408\.00118*, 2024\.doi:10\.48550/arXiv\.2408\.00118\.URL[https://arxiv\.org/abs/2408\.00118](https://arxiv.org/abs/2408.00118)\.
- Geng et al\. \[2023\]Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West\.Grammar\-constrained decoding for structured NLP tasks without finetuning\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023\.
- Grattafiori et al\. \[2024\]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the MATH dataset\.*Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, 1, 2021\.URL[https://arxiv\.org/abs/2103\.03874](https://arxiv.org/abs/2103.03874)\.
- Hiranandani et al\. \[2025\]Gaurush Hiranandani, Haolun Wu, Subhojyoti Mukherjee, and Sanmi Koyejo\.Logits are all we need to adapt closed models\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 23261–23289\. PMLR, 13–19 Jul 2025\.URL[https://proceedings\.mlr\.press/v267/hiranandani25a\.html](https://proceedings.mlr.press/v267/hiranandani25a.html)\.
- Horvitz and Thompson \[1952\]Daniel G\. Horvitz and Donovan J\. Thompson\.A generalization of sampling without replacement from a finite universe\.*Journal of the American Statistical Association*, 47\(260\):663–685, 1952\.doi:10\.2307/2280784\.
- Houlsby et al\. \[2019\]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Maziar Attariyan, and Sylvain Gelly\.Parameter\-efficient transfer learning for NLP\.In*Proceedings of the 36th International Conference on Machine Learning*, 2019\.
- Hu et al\. \[2021\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*, 2021\.
- Hugging Face \[2026\]Hugging Face\.Idavidrein/gpqa\.[https://huggingface\.co/datasets/Idavidrein/gpqa](https://huggingface.co/datasets/Idavidrein/gpqa), 2026\.Accessed: 2026\-05\-04\.
- Koo et al\. \[2024\]Terry Koo, Frederick Liu, and Luheng He\.Automata\-based constraints for language model decoding\.COLM 2024, 2024\.URL[https://openreview\.net/forum?id=BDBdblmyzY](https://openreview.net/forum?id=BDBdblmyzY)\.Camera\-ready version; arXiv:2407\.08103\.
- Korbak et al\. \[2022\]Tomasz Korbak, Ethan Perez, and Christopher L\. Buckley\.RL with KL penalties is better viewed as Bayesian inference\.*Findings of EMNLP*, 2022\.
- Krause et al\. \[2021\]Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani\.GeDi: Generative discriminator guided sequence generation\.In*Findings of the Association for Computational Linguistics: EMNLP*, 2021\.
- Lester et al\. \[2021\]Brian Lester, Rami Al\-Rfou, and Noah Constant\.The power of scale for parameter\-efficient prompt tuning\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, 2021\.
- Li and Liang \[2021\]Xiang Lisa Li and Percy Liang\.Prefix\-tuning: Optimizing continuous prompts for generation\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics*, 2021\.
- Lidstone \[1920\]George James Lidstone\.Note on the general case of the bayes\-laplace formula for inductive or a posteriori probabilities\.*Transactions of the Faculty of Actuaries*, 8\(1\):182–192, 1920\.doi:10\.1017/S0071368600003490\.
- Liu et al\. \[2021\]Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A\. Smith, and Yejin Choi\.DExperts: Decoding\-time controlled text generation with experts and anti\-experts\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics*, 2021\.
- Liu et al\. \[2022\]Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel\.Few\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.In*Advances in Neural Information Processing Systems*, volume 35, 2022\.URL[https://openreview\.net/forum?id=rBCvMG\-JsPd](https://openreview.net/forum?id=rBCvMG-JsPd)\.
- Liu et al\. \[2024\]Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024\.
- llama\.cpp Contributors \[2026\]llama\.cpp Contributors\.llama\.cpp server documentation \(logit\_bias\)\.[https://github\.com/ggml\-org/llama\.cpp/blob/master/tools/server/README\.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), 2026\.Accessed: 2026\-05\-07\.
- LM Studio Team \[2026\]LM Studio Team\.LM Studio openai\-compatible chat completions documentation \(logit\_bias\)\.[https://lmstudio\.ai/docs/developer/openai\-compat/chat\-completions](https://lmstudio.ai/docs/developer/openai-compat/chat-completions), 2026\.Accessed: 2026\-05\-07\.
- Lu et al\. \[2021\]Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi\.Neurologic decoding: \(un\)supervised neural text generation with predicate logic constraints\.In*Proceedings of NAACL\-HLT*, 2021\.
- Microsoft Azure \[2025a\]Microsoft Azure\.Azure openai chat completions reference \(logit\_bias\)\.[https://learn\.microsoft\.com/azure/ai\-services/openai/reference](https://learn.microsoft.com/azure/ai-services/openai/reference), 2025a\.Accessed: 2026\-05\-03\.
- Microsoft Azure \[2025b\]Microsoft Azure\.Prompt engineering techniques for azure openai\.[https://learn\.microsoft\.com/en\-us/azure/foundry/openai/concepts/prompt\-engineering](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering), 2025b\.Accessed: 2026\-05\-03\.
- Nesterov \[2004\]Yurii Nesterov\.*Introductory Lectures on Convex Optimization: A Basic Course*\.Kluwer Academic Publishers, 2004\.
- OpenAI \[2025a\]OpenAI\.Openai api pricing\.[https://openai\.com/api/pricing](https://openai.com/api/pricing), 2025a\.Accessed: 2026\-05\-03\.
- OpenAI \[2025b\]OpenAI\.Openai api reference \(logit\_bias\)\.[https://platform\.openai\.com/docs/api\-reference/chat/create](https://platform.openai.com/docs/api-reference/chat/create), 2025b\.Accessed: 2026\-05\-03\.
- OpenAI \[2025c\]OpenAI\.Prompt engineering\.[https://platform\.openai\.com/docs/guides/prompt\-engineering](https://platform.openai.com/docs/guides/prompt-engineering), 2025c\.Accessed: 2026\-05\-03\.
- OpenAI and Hugging Face \[2026\]OpenAI and Hugging Face\.openai/gsm8k\.[https://huggingface\.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k), 2026\.Accessed: 2026\-05\-04\.
- Ouyang et al\. \[2022\]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe\.Training language models to follow instructions with human feedback\.*Advances in Neural Information Processing Systems*, 35, 2022\.
- Pearl \[2009\]Judea Pearl\.*Causality: Models, Reasoning, and Inference*\.Cambridge University Press, 2nd edition, 2009\.
- Penedo et al\. \[2024\]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf\.The FineWeb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track*, 2024\.URL[https://arxiv\.org/abs/2406\.17557](https://arxiv.org/abs/2406.17557)\.
- Pinsker \[1964\]Mark S\. Pinsker\.*Information and Information Stability of Random Variables and Processes*\.Holden\-Day, 1964\.
- Rafailov et al\. \[2023\]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.*Advances in Neural Information Processing Systems*, 36, 2023\.
- Reimers and Gurevych \[2019\]Nils Reimers and Iryna Gurevych\.Sentence\-bert: Sentence embeddings using siamese bert\-networks\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing*, 2019\.
- Rein et al\. \[2023\]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.GPQA: A graduate\-level google\-proof q&a benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.doi:10\.48550/arXiv\.2311\.12022\.URL[https://arxiv\.org/abs/2311\.12022](https://arxiv.org/abs/2311.12022)\.
- Salemi and Zamani \[2024\]Alireza Salemi and Hamed Zamani\.Comparing retrieval\-augmentation and parameter\-efficient fine\-tuning for privacy\-preserving personalization of large language models\.*arXiv preprint arXiv:2409\.09510*, 2024\.doi:10\.48550/arXiv\.2409\.09510\.URL[https://arxiv\.org/abs/2409\.09510](https://arxiv.org/abs/2409.09510)\.
- Salemi et al\. \[2024\]Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani\.LaMP: When large language models meet personalization\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 7370–7392, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.399\.URL[https://aclanthology\.org/2024\.acl\-long\.399/](https://aclanthology.org/2024.acl-long.399/)\.
- Sentence Transformers \[2026\]Sentence Transformers\.sentence\-transformers/all\-mpnet\-base\-v2\.[https://huggingface\.co/sentence\-transformers/all\-mpnet\-base\-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), 2026\.Accessed: 2026\-05\-07\.
- SGLang Team \[2025\]SGLang Team\.Sglang openai\-compatible api documentation \(logit\_bias\)\.[https://docs\.sglang\.ai/backend/openai\_api\_completions\.html](https://docs.sglang.ai/backend/openai_api_completions.html), 2025\.Accessed: 2026\-05\-03\.
- Shao et al\. \[2024\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.doi:10\.48550/arXiv\.2402\.03300\.URL[https://arxiv\.org/abs/2402\.03300](https://arxiv.org/abs/2402.03300)\.
- Song et al\. \[2020\]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie\-Yan Liu\.MPNet: Masked and permuted pre\-training for language understanding\.In*Advances in Neural Information Processing Systems*, 2020\.
- Stiennon et al\. \[2020\]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M\. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano\.Learning to summarize from human feedback\.*Advances in Neural Information Processing Systems*, 33, 2020\.
- Sun et al\. \[2022a\]Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu\.BBTv2: Towards a gradient\-free future with large language models\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 3916–3930, Abu Dhabi, United Arab Emirates, December 2022a\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.emnlp\-main\.259\.URL[https://aclanthology\.org/2022\.emnlp\-main\.259/](https://aclanthology.org/2022.emnlp-main.259/)\.
- Sun et al\. \[2022b\]Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu\.Black\-box tuning for language\-model\-as\-a\-service\.In*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pages 20841–20855\. PMLR, 17–23 Jul 2022b\.URL[https://proceedings\.mlr\.press/v162/sun22e\.html](https://proceedings.mlr.press/v162/sun22e.html)\.
- Tsybakov \[2009\]Alexandre B\. Tsybakov\.*Introduction to Nonparametric Estimation*\.Springer, 2009\.
- vLLM Team \[2026\]vLLM Team\.vLLM sampling parameters documentation \(logit\_bias\)\.[https://docs\.vllm\.ai/en/stable/api/vllm/sampling\_params/](https://docs.vllm.ai/en/stable/api/vllm/sampling_params/), 2026\.Accessed: 2026\-05\-07\.
- Willard and Louf \[2023\]Brandon T\. Willard and Rémi Louf\.Efficient guided generation for large language models\.*arXiv preprint arXiv:2307\.09702*, 2023\.
- Wolf et al\. \[2020\]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush\.Transformers: State\-of\-the\-art natural language processing\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online, October 2020\. Association for Computational Linguistics\.doi:10\.18653/v1/2020\.emnlp\-demos\.6\.URL[https://aclanthology\.org/2020\.emnlp\-demos\.6/](https://aclanthology.org/2020.emnlp-demos.6/)\.
- Yang et al\. \[2025\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.doi:10\.48550/arXiv\.2505\.09388\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Yang and Klein \[2021\]Kevin Yang and Dan Klein\.FUDGE: Controlled text generation with future discriminators\.In*Proceedings of NAACL\-HLT*, 2021\.
- Zhou et al\. \[2025\]Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q\. Weinberger, Kianté Brantley, and Wen Sun\.Q♯Q^\{\\sharp\}: Provably Optimal Distributional RL for LLM Post\-Training\.In*Advances in Neural Information Processing Systems*, 2025\.doi:10\.48550/arXiv\.2502\.20548\.URL[https://arxiv\.org/abs/2502\.20548](https://arxiv.org/abs/2502.20548)\.NeurIPS 2025\.
- Ziegler et al\. \[2019\]Daniel M\. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B\. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving\.Fine\-tuning language models from human preferences\.*arXiv preprint arXiv:1909\.08593*, 2019\.

## Appendix Contents

## Appendix AIntervention score, token selection, and grouping

Appendices[B](https://arxiv.org/html/2607.22837#A2)and[C](https://arxiv.org/html/2607.22837#A3)interpret a learned logit\-bias vectorδ^∈ℝ\|𝕍\|\\widehat\{\\delta\}\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\}by \(i\) summarizing its decode\-time effect as a per\-token*intervention score*SS, \(ii\) splitting the visited tokens into promoted and suppressed sets by the sign ofSS, and \(iii\) grouping those into semantic families\. Complete suppressed\-side listings are provided in the code repository underresults/\.

### A\.1Intervention score as realized log\-probability shift

A rawδ^​\(y\)\\widehat\{\\delta\}\(y\)lives in logit space, not directly in probability space: after softmax normalization, the actual change in a token’s probability depends on the prefix at which it is decoded\. We therefore summarize the decode\-time effect of the learned bias by measuring, at each generated position, the log\-probability shift it induces on the realized next token\. For a tokeny∈𝕍y\\in\\mathbb\{V\}, averaging this shift over the positions where the fixed\-bias policy generatesyygives the occurrence\-weighted*intervention score*

S\(y\):=𝔼x∼ρ,t∼Unif​\{1,…,T\}y1:T∼pδ^\(⋅∣x\)\[logπδ^​\(y∣x,y1:t−1\)π0​\(y∣x,y1:t−1\)\|yt=y\]\.S\(y\)\\;:=\\;\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}x\\sim\\rho,\\;t\\sim\\mathrm\{Unif\}\\\{1,\\dots,T\\\}\\\\ y\_\{1:T\}\\sim p\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x\)\\end\{subarray\}\}\\\!\\left\[\\,\\log\\frac\{\\pi\_\{\\widehat\{\\delta\}\}\(y\\mid x,y\_\{1:t\-1\}\)\}\{\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)\}\\;\\middle\|\\;y\_\{t\}=y\\,\\right\]\.\(9\)The expectation is occurrence\-weighted: it conditions on the positions where the fixed\-bias policy realizesyy, with the prefixy1:t−1y\_\{1:t\-1\}and the positionttsampled uniformly\. Since the log\-ratio depends only onxxand the prefixy1:t−1y\_\{1:t\-1\}, the suffixyt\+1:Ty\_\{t\+1:T\}is irrelevant\. Because both probabilities are post\-softmax,S​\(y\)S\(y\)reflects the actual probability change induced by the bias at decode time, not the raw coordinateδ^​\(y\)\\widehat\{\\delta\}\(y\):S​\(y\)\>0S\(y\)\>0promotes the realized token andS​\(y\)<0S\(y\)<0suppresses it\. This log\-ratio is the same local quantity that appears in the KL\-regularized derivation, but here we use it only as an interpretability score\.

##### Estimator\.

We estimateSSon held\-out*on\-policy*rollouts generated by the fixed\-bias policypδ^p\_\{\\widehat\{\\delta\}\}\. Given such a rollout set with recorded token ids, we use every position in each trajectory\. At each position\(n,t\)\(n,t\)we run the base model on the prompt and the fixed\-bias\-policy rollout prefix to obtain its logits, formπ0\\pi\_\{0\}andπδ^\\pi\_\{\\widehat\{\\delta\}\}from them, and record the shift on the realized token,

Δt\(n\):=log⁡πδ^​\(yt\(n\)∣x\(n\),y1:t−1\(n\)\)−log⁡π0​\(yt\(n\)∣x\(n\),y1:t−1\(n\)\)\.\\Delta\_\{t\}^\{\(n\)\}\\;:=\\;\\log\\pi\_\{\\widehat\{\\delta\}\}\\\!\\left\(y\_\{t\}^\{\(n\)\}\\mid x^\{\(n\)\},y\_\{1:t\-1\}^\{\(n\)\}\\right\)\-\\log\\pi\_\{0\}\\\!\\left\(y\_\{t\}^\{\(n\)\}\\mid x^\{\(n\)\},y\_\{1:t\-1\}^\{\(n\)\}\\right\)\.The score of a token is the mean of these shifts over the positions at which it is realized,

S^​\(y\)=1\|ℐy\|​∑\(n,t\)∈ℐyΔt\(n\),ℐy:=\{\(n,t\):yt\(n\)=y\},\\widehat\{S\}\(y\)\\;=\\;\\frac\{1\}\{\|\\mathcal\{I\}\_\{y\}\|\}\\sum\_\{\(n,t\)\\in\\mathcal\{I\}\_\{y\}\}\\Delta\_\{t\}^\{\(n\)\},\\qquad\\mathcal\{I\}\_\{y\}:=\\bigl\\\{\(n,t\):y\_\{t\}^\{\(n\)\}=y\\bigr\\\},with the conventionS^​\(y\):=0\\widehat\{S\}\(y\):=0for tokens that never appear at any position in the rollouts \(ℐy=∅\\mathcal\{I\}\_\{y\}=\\emptyset\)\.

### A\.2Splitting tokens by sign

We apply no magnitude threshold: every visited token is assigned to one side by the sign of its score,

ℛ\+=\{y∈𝕍:S​\(y\)\>0\},ℛ−=\{y∈𝕍:S​\(y\)<0\}\.\\mathcal\{R\}^\{\+\}=\\bigl\\\{\\,y\\in\\mathbb\{V\}:\\ S\(y\)\>0\\,\\bigr\\\},\\qquad\\mathcal\{R\}^\{\-\}=\\bigl\\\{\\,y\\in\\mathbb\{V\}:\\ S\(y\)<0\\,\\bigr\\\}\.The*promoted*tokensℛ\+\\mathcal\{R\}^\{\+\}are those the bias makes more likely at decode time and the*suppressed*tokensℛ−\\mathcal\{R\}^\{\-\}those it makes less likely; the remaining tokens, never visited underpδ^p\_\{\\widehat\{\\delta\}\}and hence carryingS​\(y\)=0S\(y\)=0, are left ungrouped\.

### A\.3Grouping selected tokens into families

To summarize each side \(ℛ\+\\mathcal\{R\}^\{\+\}andℛ−\\mathcal\{R\}^\{\-\}\) we group its tokens into semantic families\. This grouping is a purely interpretive aid, not part of the method, and the families carry no weight of their own; we refer to them by index in the reward\-specific appendices below\. The promoted set is small and dominated by a few recurring surface patterns—formatting, connectives, digits, and stop tokens—so we group the promoted tokens manually by inspection\. The suppressed set is substantially larger and noisier, so we group it by AI\-assisted manual curation, with every suppressed token listed in the code repository underresults/\.

## Appendix BQualitative analysis of the accuracy logit\-bias vector

This appendix examines what the accuracy intervention scoreSS\(Appendix[A\.1](https://arxiv.org/html/2607.22837#A1.SS1)\) encodes under the*accuracy reward*\. Of the Llama vocabulary \(128,256128\{,\}256tokens\),5,7395\{,\}739are visited in the rollouts—137137*promoted*\(S\>0S\>0\) and5,6025\{,\}602*suppressed*\(S<0S<0\)—and we group each side into semantic families as in Appendix[A](https://arxiv.org/html/2607.22837#A1); all5,6025\{,\}602suppressed tokens with their scores are listed in the code repository underresults/accuracy\_suppressed\_all\_families\.tex\.

−3\-3−2\.5\-2\.5−2\-2−1\.5\-1\.5−1\-1−0\.5\-0\.5010010^\{0\}10110^\{1\}10210^\{2\}10310^\{3\}S​\(y\)S\(y\)Number of tokens in bin0112210010^\{0\}10110^\{1\}S​\(y\)\>0S\(y\)\>0promoted side \(S​\(y\)\>0S\(y\)\>0\)Figure B\.3:Histogram of the intervention scoreS^​\(y\)\\widehat\{S\}\(y\)for the accuracy reward over the Llama vocabulary \(visited tokens only;122,517122\{,\}517unvisited tokens withS^​\(y\)=0\\widehat\{S\}\(y\)=0are omitted\)\. Theyy\-axis is logarithmic\. The main panel shows the*suppressed*side \(S^​\(y\)<0\\widehat\{S\}\(y\)<0,5,6025\{,\}602tokens\), which carries the bulk of the mass and a long tail reachingS^​\(y\)≈−3\.36\\widehat\{S\}\(y\)\\approx\-3\.36; the inset zooms into the*promoted*side \(S^​\(y\)\>0\\widehat\{S\}\(y\)\>0,137137tokens, up toS^​\(y\)≈2\.84\\widehat\{S\}\(y\)\\approx 2\.84\)\.The accuracy score is a*two\-sided correctness prior*: it promotes tokens that help format, scaffold, and terminate correct solutions, and it suppresses tokens that usually start explanations, page drift, or problem\-specific detours\. The positive side is small and sharp while the negative side is broad \(Table[B\.4](https://arxiv.org/html/2607.22837#A2.T4)\), so the bias acts mainly by discouraging failure\-mode continuations while also boosting a few answer\-surface and stop tokens\.

Table B\.4:Score bands of the accuracy intervention scoreS​\(y\)S\(y\)\.### B\.1Promoted token families

The promoted side \(S​\(y\)\>0S\(y\)\>0\) contains the137137tokens with a positive log\-probability shift; we list every one, grouped into families with a short gloss noting why each token plausibly co\-occurs with correct solutions\. Interpretation is collected in Appendix[B\.3](https://arxiv.org/html/2607.22837#A2.SS3)\.

Families are ordered by their mean intervention scoreS¯\\bar\{S\}\(the average ofS​\(y\)S\(y\)over the family’s tokens\)\.

- •Stop strings \(2,S¯=1\.66\\bar\{S\}=1\.66\)\.`<\|end\_of\_text\|\>`and`␣Problem`, which terminate the current completion\.
- •Instruction / educational metadata \(6,S¯=1\.06\\bar\{S\}=1\.06\)\.`␣Keywords`\(label such as “Problem Keywords”\),`Basic`\(tags like “Basic Algebra”\),`User`\(AoPS/wiki`/User:`fragments\),`␣Vocabulary`\(“Key Vocabulary”\),`Personal`\(page/persona drift\), and`Skill`\(“Reasoning Skill”\)\.
- •Text / document / page\-drift concepts \(2,S¯=0\.51\\bar\{S\}=0\.51\)\.`Ann`\(name prefix, page/person drift\) and`␣topics`\(generic page language such as “other topics”\)\.
- •Actions / commands / ordering / problem\-solving context \(34,S¯=0\.078\\bar\{S\}=0\.078\)\.`Problem`,`␣Solutions`,`␣Since`,`␣number`,`␣logically`,`␣Answer`,`␣answer`,`␣Compute`,`␣find`,`␣write`,`␣use`,`use`,`align`,`edit`,`istinguish`\(suffix of*distinguish*\),`prehensive`\(suffix of*comprehensive*\),`ifying`\(as in*simplifying/identifying/verifying*\),`ifies`\(as in*simplifies/verifies*\),`Alternate`,`Other`,`task`,`Check`,`confirmation`,`␣central`,`␣Next`,`␣best`,`␣final`,`Final`,`Last`,`␣first`,`␣ways`,`Long`,`big`, and`Com`\(start of*Compute/Compare/Combine/Complex*\)\.
- •Function words / grammatical glue \(14,S¯=0\.066\\bar\{S\}=0\.066\)\.`␣We`,`The`,`␣the`,`␣The`,`␣This`,`␣is`,`␣have`,`␣can`,`␣and`,`␣to`,`not`,`one`,`new`, and`all`—ordinary grammar tokens\.
- •Math /LaTeX/ symbols / delimiters / formatting \(27,S¯=0\.066\\bar\{S\}=0\.066\)\.`␣$`\(inline math after a space\),`$`\(math delimiter\),`␣$\\`\(space\+\+`$`\+\+command start\),`␣\\`\(space\+\+backslash\),`\\`\(command escape\),`=`,`cal`\(as in`\\mathcal`or “calculate”\),`begin`\(as in`\\begin\{align\}`\),`Math`,`frac`\(`\\frac`\),`cdot`\(`\\cdot`\),`\.sqrt`\(square\-root fragment\),`x`,`\}`,`\)`,`\{`,`\]\(`\(markdown link boundary\),`\{\|`\(table/wiki delimiter\),`\{$`\(brace\+\+math delimiter\),`\\n`\(newline\),`␣\#\#`\(markdown heading\),`\.`, the space token,`,`,`␣\*\*`\(markdown bold\),`‘\\n`\(backtick then newline\), and the middle\-dot token·\.
- •Math structure / math context \(16,S¯=0\.012\\bar\{S\}=0\.012\)\.`let`\(“letx=…x=\\dots”\),`␣given`\(problem\-condition language\),`␣quadr`\(*quadratic/quadrilateral/quadrant*\),`␣hex`\(*hexagon/hexadecimal*\),`ponents`\(suffix of*exponents*\),`ultiply`\(suffix of*multiply*\),`aring`\(as in*squaring*\),`triangle`,`positive`,`line`,`␣foot`\(unit/geometry\),`ector`\(as in*vector/bisector/sector*\),`␣Extended`\(Extended Euclidean Algorithm\),`AC`\(geometry notation, e\.g\. sideA​CAC\),`Im`\(imaginary part, or start of “Important”\), and`AY`\(geometry label, or URL/name noise\)\.
- •Proper names / acronyms / geography / source artifacts \(8,S¯=0\.005\\bar\{S\}=0\.005\)\.`L`\(from Ł\),`␣America`\(“Mathematical Association of America”\),`J`,`Ace`,`AL`,`TH`,`external`, and`␣external`\.
- •Numbers \(13,S¯=0\.004\\bar\{S\}=0\.004\)\.`2`,`21`,`1`,`3`,`453`,`228`,`669`,`734`,`813`,`355`,`415`,`424`, and`406`—mostly literal numerals; the three\-digit ones are likely dataset\-specific artifacts, examples, ids, or intermediate values\.
- •Subword fragments \(15,S¯<0\.001\\bar\{S\}<0\.001\)\.`qu`\(*quadratic/quantity/equation/sequence*\),`ens`\(*tens/dimensions*\),`ast`\(*last/least/past*\),`ics`\(*mathematics/statistics*\),`ct`\(*factor/function/product*\),`ze`\(*zero/analyze/finalize*\),`ant`\(*constant/discriminant/quadrant*\),`ub`\(*subtract/cube/subset*\),`ja`\(as in*adjacent*\),`istrib`\(*distribute/distribution*\),`ner`\(*inner/corner*\),`ee`\(*three/degree/between*\), and`il`\(*similar/probability*\)\. Two further fragments are notable:`oted`\(most math\-relevant as*denoted*\) and`ric`\(*geometric/metric/symmetric/trigonometric*\)\.

### B\.2Suppressed token families

The suppressed side \(S​\(y\)<0S\(y\)<0\) contains5,6025\{,\}602tokens\. These tokens are not “wrong” in isolation\. Under the log\-indicator accuracy reward, they receive little positive support from correct rollouts relative to the promoted answer\-surface tokens and the smoothing/centering baseline\. After softmax normalization this appears as a broad negative side of the realized intervention score\. The lower tail is therefore best read as an anti\-drift pattern learned by the accuracy\-trained bias, rather than as token\-level proof that every suppressed token causes incorrectness\. The strongest negative tail includes`␣Question`,`␣Examples`,`␣Explanation`,`␣Approach`,`␣Lesson`, markdown headings, and page/template fragments\. A useful first cut is by strength \(Table[B\.4](https://arxiv.org/html/2607.22837#A2.T4)\); we therefore avoid over\-interpreting tiny negatives near zero, as the meaningful story comes from the lower tail\.

Following the grouping procedure in Appendix[A\.3](https://arxiv.org/html/2607.22837#A1.SS3), we group the suppressed tokens into twelve interpretive families\. The summaries below describe each family, ordered by mean intervention scoreS¯\\bar\{S\}\(most strongly suppressed first\); the complete per\-token listings with intervention scores are provided in the code repository underresults/accuracy\_suppressed\_all\_families\.tex\.

- •Verbose reasoning wrappers / discourse markers \(44,S¯=−0\.73\\bar\{S\}=\-0\.73\)\.Tokens such as`␣Moreover`,`␣Clearly`,`␣Furthermore`,`␣However`, and`␣Therefore`often introduce extended explanatory scaffolding or discourse transitions rather than the final answer itself\. Their suppression suggests that the terminal reward favors reaching the answer directly over producing lecture\-style reasoning\.
- •Question\-restatement / prompt\-template tokens \(40,S¯=−0\.73\\bar\{S\}=\-0\.73\)\.Strongly suppressed tokens such as`␣Question`usually start a new prompt or open a Q/A template instead of solving the current problem, so the shift discourages worksheet, course\-page, and new\-problem mode\. \(Some metadata tokens—`␣Keywords`,`Basic`,`␣Vocabulary`—were instead*promoted*: those correlated with scraped correct\-solution pages, whereas`␣Question`/`␣Examples`/`␣Lesson`correlate with drifting into extra page content\.\)
- •Names, initials, usernames, source/person artifacts \(294,S¯=−0\.30\\bar\{S\}=\-0\.30\)\.First names and initials such as`␣Alice`,`␣Brian`,`␣Julie`, and`␣Kevin`are problem\-specific rather than generally useful; suppressing them reduces hallucinated story details and invented diagrams\.
- •Educational\-site / AoPS / forum / wiki artifacts \(44,S¯=−0\.19\\bar\{S\}=\-0\.19\)\.Unlike the literal web markup of the HTML / URL / markdown / code family, these are the*natural\-language*page furniture of the math sites in pretraining—contest wikis, Art\-of\-Problem\-Solving threads, and Q&A forums—the kind of boilerplate and page scaffolding that pervades web\-scraped pretraining corpora\[[42](https://arxiv.org/html/2607.22837#bib.bib42)\]\. Since worked solutions there sit inside such pages, the base model tends to continue past the answer into the surrounding scaffolding \(“Posted by…”, “Reply”, “edited”, “References”, user/wiki links\)\. Suppressing these tokens keeps the completion on the current problem instead of rolling over into copied page tails\.
- •Math\-topic labels suppressed despite being mathematical \(73,S¯=−0\.14\\bar\{S\}=\-0\.14\)\.The most subtle group: tokens such as`␣Coordinate`,`␣Distance`,`␣Equation`, and`␣calculus`are meaningful, but often appear as topic labels or wrong\-route triggers and, averaged over the dataset, were less predictive of success than direct answer\-oriented tokens\. General math vocabulary can be negatively correlated with correctness when it signals overcomplication or wrong method selection\.
- •Function words and ordinary prose \(2,536,S¯=−0\.13\\bar\{S\}=\-0\.13\)\.Tokens such as`␣while`,`␣there`,`␣because`, and`␣their`are not a blanket anti\-English bias \(some function words were promoted\); these reflect the longer prose style of rambling completions\.
- •Subword shards / tokenizer fragments \(1,525,S¯=−0\.12\\bar\{S\}=\-0\.12\)\.Fragments such as`ime`\(as in*time/prime*\),`reet`\(*street*\),`ting`, and`ul`inherit the statistics of the words containing them; they are suppressed because those words correlated with lower\-quality continuations, not because the fragment itself is meaningful\.
- •Geometry labels and diagram\-letter clutter \(214,S¯=−0\.091\\bar\{S\}=\-0\.091\)\.Arbitrary point/segment labels such as`␣AE`,`␣OB`,`␣II`, and`␣III`are easy to hallucinate, so this reads as a bias against invented diagrams\. \(Note`AC`was*promoted*: token scores are empirical correlations, not semantic rules\.\)
- •HTML / URL / markdown / code artifacts \(189,S¯=−0\.077\\bar\{S\}=\-0\.077\)\.Tokens such as`<a`,`␣https`, and`˜˜`usually mean the model has left the solution and is emitting web/markdown/HTML/source residue, which is almost always bad in a final\-answer setting\.
- •Generic assistant / politeness / meta\-answer style \(28,S¯=−0\.068\\bar\{S\}=\-0\.068\)\.Conversational filler such as`Thanks`,`Please`,`␣sorry`, and`␣appreciate`\(“I hope this helps”\) rather than answer extraction\.
- •Arbitrary numbers and numeric constants \(516,S¯=−0\.052\\bar\{S\}=\-0\.052\)\.Globally promoting arbitrary numerals such as`846`,`523`, and`123`would invite hallucinated answers, so suppression here is healthy: “do not prefer this particular number unless the context demands it\.”
- •Word\-problem scenery / concrete objects \(99,S¯=−0\.049\\bar\{S\}=\-0\.049\)\.Surface nouns from problem statements such as`␣plane`,`␣coins`,`␣pigs`, and`␣night`, not solution moves; these should be driven by the actual prompt\.

### B\.3Interpretation

Read together, the two sides describe a steering vector that favors a clean, staged path to the answer\. The promoted side raises the surface and discourse regularities of correct, well\-formatted solutions: layout and delimiters; the headers, connectives, and solution verbs that scaffold explicit step\-by\-step working; digits and symbolic notation; and decisive termination\. It thus gently favors solutions that lay out their reasoning rather than emit a bare value, without choosing the problem\-specific reasoning path; the promoted digits only supply the surface for calculations and do not imply the bias knows which number is correct\.

The suppressed side acts as an anti\-drift prior against the recurring failure modes in Appendix[B\.2](https://arxiv.org/html/2607.22837#A2.SS2): restating or generating a new question and slipping into lesson/tutorial mode; padding with verbose discourse markers; emitting scraped web/page residue; lapsing into generic assistant politeness; inventing problem\-specific details; and overcomplicating with topic labels\. The surprising suppression of genuine math vocabulary should not be read as “avoid math”: averaged over the dataset, those tokens more often started verbose, topic\-heavy, website\-like, or problem\-specific continuations than the direct, answer\-oriented tokens the shift prefers\. The code repository lists all5,6025\{,\}602suppressed tokens with their intervention scores underresults/accuracy\_suppressed\_all\_families\.tex\.

## Appendix CQualitative analysis of the compression logit\-bias vector

This appendix examines what the compression intervention scoreSS\(Appendix[A\.1](https://arxiv.org/html/2607.22837#A1.SS1)\) encodes under the*compression reward*\. Of the Llama vocabulary \(128,256128\{,\}256tokens\),4,1644\{,\}164are visited—141141*promoted*\(S\>0S\>0\) and4,0234\{,\}023*suppressed*\(S<0S<0\); leading spaces are preserved exactly \(`␣use`and`use`are distinct tokens\), and each side is grouped into semantic families as in Appendix[A](https://arxiv.org/html/2607.22837#A1), with all4,0234\{,\}023suppressed tokens listed in the code repository underresults/compression\_suppressed\_all\_families\.tex\.

The compression score is a*two\-sided length\-control prior*: it promotes tokens that help stop, shift to a boundary, package a short answer, or express mathematics compactly, and it suppresses tokens that usually start explanations, checks, examples, long formatting, page tails, or multi\-step derivations\. The positive side is small and sharp while the negative side is broad \(Table[C\.5](https://arxiv.org/html/2607.22837#A3.T5)\), so the bias acts mainly by discouraging length\-expanding continuations while also boosting a few stop, answer, and compact\-format tokens\.

−2\.5\-2\.5−2\-2−1\.5\-1\.5−1\-1−0\.5\-0\.5010010^\{0\}10110^\{1\}10210^\{2\}10310^\{3\}S​\(y\)S\(y\)Number of tokens in bin0112210010^\{0\}10110^\{1\}S​\(y\)\>0S\(y\)\>0promoted side \(S​\(y\)\>0S\(y\)\>0\)Figure C\.4:Histogram of the intervention scoreS^​\(y\)\\widehat\{S\}\(y\)for the compression reward over the Llama vocabulary \(visited tokens only;124,092124\{,\}092unvisited tokens withS^​\(y\)=0\\widehat\{S\}\(y\)=0are omitted\)\. Theyy\-axis is logarithmic\. The main panel shows the*suppressed*side \(S^​\(y\)<0\\widehat\{S\}\(y\)<0,4,0234\{,\}023tokens\), which carries the bulk of the mass and a long tail reachingS^​\(y\)≈−2\.86\\widehat\{S\}\(y\)\\approx\-2\.86; the inset zooms into the*promoted*side \(S^​\(y\)\>0\\widehat\{S\}\(y\)\>0,141141tokens, up toS^​\(y\)≈2\.80\\widehat\{S\}\(y\)\\approx 2\.80\)\.Table C\.5:Score bands of the compression intervention scoreS​\(y\)S\(y\)\.### C\.1Promoted token families

The promoted side \(S\>0S\>0\) contains all141141tokens with a positive log\-probability shift; we list every one, grouped into families with a short gloss\. Interpretation is collected in Appendix[C\.3](https://arxiv.org/html/2607.22837#A3.SS3)\.

Families are ordered by their mean intervention scoreS¯\\bar\{S\}\(the average ofS​\(y\)S\(y\)over the family’s tokens\)\.

- •Stop / boundary tokens \(6,S¯=0\.61\\bar\{S\}=0\.61\)\.`<\|end\_of\_text\|\>`,`␣Problem`,`Problem`,`\\n`,`:\\n`, and`␣\#\#`\.`<\|end\_of\_text\|\>`and`␣Problem`are configured stop strings:`<\|end\_of\_text\|\>`is the direct end\-of\-text token, and`␣Problem`terminates the completion at the next problem header \(reaching a new problem effectively ends the current answer\)\. The remaining`Problem`,`\\n`,`:\\n`, and`␣\#\#`are not stop strings but compact structural boundaries that let the model finish or segment without opening a long paragraph\. This is the clearest compression signal: end now, or move to a boundary that makes ending natural\.
- •Educational / page / UI / source artifacts \(19,S¯=0\.46\\bar\{S\}=0\.46\)\.`Personal`,`Recommended`,`␣video`,`Basic`,`Win`,`Ac`,`User`,`Skill`,`Special`,`J`,`␣Ch`,`Credit`,`␣Display`,`black`,`␣America`,`␣engineering`,`\-person`,`␣Day`, and`␣Mathematics`\. These are scraped\-page, UI, source, or metadata artifacts rather than compression mechanisms—e\.g\.`Basic`\(“Basic Algebra”\),`User`\(forum/wiki metadata\),`␣Ch`\(“Chapter”\),`␣America`\(“Mathematical Association of America”\)\. They are most likely correlational shortcuts that appeared near short page snippets or early\-stop regions; the strongest promoted token being`Personal`is a warning that the intervention also picked up dataset artifacts\.
- •Ordering / option / concise navigation \(12,S¯=0\.22\\bar\{S\}=0\.22\)\.`Alt`,`Alternate`,`Alternative`,`␣central`,`␣Next`,`Next`,`␣first`,`␣Ways`,`␣best`,`Long`,`␣systematic`, and`␣changing`\. A mixed group:`␣Next`/`Next`/`␣first`are compact sequencing tokens and`␣best`supports answer selection, while`Alt`/`Alternate`/`Alternative`may be option labels or page artifacts \(`Long`,`␣systematic`,`␣changing`are weak positives\)\.
- •Minimal action / solution hooks \(9,S¯=0\.16\\bar\{S\}=0\.16\)\.`show`,`␣Get`,`␣find`,`␣use`,`␣Use`,`␣given`,`␣Since`,`␣Define`, and`align`\. These are short solution hooks \(e\.g\.`␣Get`“Getx=3x=3”,`␣given`a compact reference to the conditions,`␣Since`a short justification starter\)\. The bias prefers skeletal action hooks over derivational prose \(the imperative\-vs\-prose minimal pairs are collected in Appendix[C\.3](https://arxiv.org/html/2607.22837#A3.SS3)\)\.
- •Minimal grammar / punctuation scaffold \(22,S¯=0\.077\\bar\{S\}=0\.077\)\.`␣We`,`The`,`␣the`,`␣The`,`␣A`,`␣This`,`␣`,`␣can`,`␣have`,`,`,`␣is`,`␣and`,`␣to`,`␣in`,`␣of`,`␣a`,`␣my`,`␣All`,`\.`,`␣"`,`\!\)`, and`␣\.\.\.`\. Compression does not strip all prose: it keeps just enough grammar and punctuation to emit a compact, well\-formed response \(`␣`is a bare space token,`␣\.\.\.`an ellipsis/short\-tail marker\)\.
- •Short final\-answer markers \(11,S¯=0\.055\\bar\{S\}=0\.055\)\.`␣Final`,`Final`,`␣final`,`␣Answer`,`␣answer`,`Choice`,`␣lowest`,`␣number`,`Number`,`␣useful`, and`␣important`\. These let the model package a response as a final answer \(“Final answer…”, “The answer is…”, compact multiple\-choice output\) instead of producing a full worked solution\. \(Leading\-space forms occur inside compact sentences; see Appendix[C\.3](https://arxiv.org/html/2607.22837#A3.SS3)for the promoted/suppressed minimal pairs\.\)
- •Compact math /LaTeX/ symbolic / geometry \(33,S¯=0\.038\\bar\{S\}=0\.038\)\.`\{$`,`␣$`,`␣$\\`,`␣quadr`,`\{\|`,`$`,`begin`,`\\`,`\}`,`␣hex`,`␣\\`,`tan`,`\)`,`␣\(`,`frac`,`triangle`,`AC`,`\{`,`␣Square`,`=`,`x`,`␣\-`,`␣=`,`␣Extended`,`EF`,`cdot`,`\_C`,`QR`,`cs`,`cal`,`\("$`,`\("`, and`·`\(token 115, a raw\-byte artifact\)\. These support compact mathematical expression: inline math delimiters \(`␣$`,`$`\), command escapes \(`\\`\), subword notation \(`frac`,`cdot`,`tan`\), geometry labels \(`AC`,`EF`,`QR`\), and compact algebra \(`x`,`=`,`␣\-`\): the bias likes inline math, not display\-math derivations \(see the minimal pairs in Appendix[C\.3](https://arxiv.org/html/2607.22837#A3.SS3)\)\.
- •Numbers \(9,S¯=0\.011\\bar\{S\}=0\.011\)\.`2`,`21`,`1`,`3`,`996`,`163`,`926`,`222`, and`354`\. Small digits \(`1`,`2`,`3`\) are useful in concise answers and enumeration; the larger numbers are weak positives and probably dataset\-specific rather than generally meaningful\.
- •Subword fragments / morphemes \(20,S¯<0\.001\\bar\{S\}<0\.001\)\.`stit`,`cept`,`ector`,`oid`,`\-negative`,`ilateral`,`istinguish`,`ate`,`om`,`ifying`,`ifies`,`ues`,`ithmetic`,`ee`,`\_r`,`ultiply`,`itions`,`aring`,`oted`, and`anging`\. Tokenizer fragments that inherit meaning from the words they appear in \(e\.g\.`ector`as*vector/bisector*,`ilateral`as*quadrilateral/equilateral*,`ithmetic`as*arithmetic*\); treat them as low\-confidence weak positives\.

### C\.2Suppressed token families

The suppressed side \(S<0S<0\) contains4,0234\{,\}023tokens and is best read as an anti\-continuation prior: these tokens are not “wrong” in isolation, but they tend to predict that the answer is about to get longer\. The summaries below describe each family, ordered by mean intervention scoreS¯\\bar\{S\}\(most strongly suppressed first\); the complete per\-token listings with intervention scores are provided in the code repository underresults/compression\_suppressed\_all\_families\.tex\.

- •Multiline formatting / display math / markdown expansion \(15,S¯=−0\.55\\bar\{S\}=\-0\.55\)\.A very clean compression signal: multi\-line formatting \(`␣\\n\\n`\), display math \(`␣$$`\), deeper headings \(`␣\#\#\#`\), and long spacing all predict a longer answer, whereas a single boundary or inline formula is compact\.
- •Course\-page / website / practice\-mode tokens \(29,S¯=−0\.54\\bar\{S\}=\-0\.54\)\.Among the strongest negatives \(`␣Latest`,`␣Practice`,`␣Community`\): page navigation, educational\-site residue, and scraped tails that predict the model has moved away from the answer into website content\.
- •Explanation / solution\-section headings \(18,S¯=−0\.47\\bar\{S\}=\-0\.47\)\.Tokens such as`␣Explanation`,`␣Solution`,`␣Approach`, and`Definition`are useful for tutoring but they tend to open a new section or paragraph, so compression penalizes entering explanation mode—a clear difference from the accuracy reward, which may keep such scaffolding\.
- •Step\-by\-step derivation starters \(27,S¯=−0\.36\\bar\{S\}=\-0\.36\)\.These begin visible reasoning \(`Using`,`Given`,`␣Finding`\), so the model is pushed away from “show the work” mode\.
- •Checking / verification / self\-correction \(26,S¯=−0\.32\\bar\{S\}=\-0\.32\)\.Verification aids reliability but costs tokens \(it usually follows the answer\), so the bias suppresses it—at the risk of removing useful sanity checks\.`␣Check`is especially suppressed\.
- •Names, people, and story nouns \(170,S¯=−0\.18\\bar\{S\}=\-0\.18\)\.Narrative detail from problem statements such as`␣Lisa`,`␣Natalie`, and`␣Point`; a short answer should not invent people, holidays, objects, or scenery\.
- •Programming / code / system\-ish artifacts \(23,S¯=−0\.13\\bar\{S\}=\-0\.13\)\.Code\-like continuations such as`␣Java`,`␣Python`,`␣program`, and`␣Test`usually lengthen the output and drift from a compact final answer\.
- •Math\-topic tokens that invite derivation \(85,S¯=−0\.12\\bar\{S\}=\-0\.12\)\.These are real math tokens; suppression does not mean “avoid math” but that they tend to open a topic\-specific derivation or calculation, which is costly under a shorter\-answer reward \(`␣integration`is the clearest case\)\.
- •Discourse connectives and paragraph extenders \(1,669,S¯=−0\.098\\bar\{S\}=\-0\.098\)\.Tokens such as`␣But`,`␣Also`,`␣And`, and`␣Specifically`imply the answer is continuing—another sentence, case, or justification—an almost anti\-chain\-of\-thought surface signature\.
- •Subword fragments \(1,218,S¯=−0\.067\\bar\{S\}=\-0\.067\)\.Fragments whose larger words tend to start longer explanations or theorem references \(`␣Squ`*Square*,`␣Con`*Consider*,`␣Ferm`*Fermat*\)\.
- •Geometry labels / arbitrary point names / initials \(203,S¯=−0\.061\\bar\{S\}=\-0\.061\)\.Some labels are promoted \(`AC`,`EF`,`QR`\) while others are suppressed \(`␣OA`,`␣AB`,`␣OB`\)—an empirical, not semantic, split; suppression helps avoid invented geometry scaffolding\.
- •Arbitrary numbers \(540,S¯=−0\.023\\bar\{S\}=\-0\.023\)\.A compression prior should not globally promote random constants such as`388`,`522`, and`540`; suppressing them reduces hallucinated numeric detail \(the robust positive numeric signal is only for small digits\)\.

### C\.3Interpretation

A handful of contrasts are more informative than individual tokens, because they isolate the length\-control axis from token identity:

- •`<\|end\_of\_text\|\>`,`␣Problem`,`Problem`promoted: stop / next\-problem boundary behavior\.
- •`\\n`promoted but`␣\\n\\n`suppressed: one boundary is compact, paragraph expansion is not\.
- •`␣\#\#`promoted but`␣\#\#\#`suppressed: a shallow boundary can help stop, deeper sectioning extends\.
- •`$`/`␣$`promoted but`␣$$`suppressed: inline math is compact, display math invites derivation\.
- •`␣Answer`promoted but`Answer`suppressed: a compact answer phrase beats heading/template mode\.
- •`␣Use`promoted but`Using`suppressed: an imperative hook beats a prose derivation starter\.
- •`Alternate`promoted but`␣Alternate`suppressed: a compact label beats an “alternate solution” continuation\.
- •`␣Check`suppressed: verification is useful but length\-expanding\.
- •`␣integration`suppressed: math\-topic words can be anti\-compression when they open long derivations\.
- •`␣Latest`,`␣Practice`,`␣Community`strongly suppressed: page/navigation/practice tails are anti\-answer\.

Overall, the compression intervention learns a*terminal, compact\-answer style*: unlike the accuracy bias it is not merely promotional but actively pushes down tokens that predict continuation\. Its main risk is therefore*over\-compression*, since useful verification, necessary derivation, and explanatory clarity can be suppressed along with genuine verbosity \(Section[8](https://arxiv.org/html/2607.22837#S8)\)\. The code repository lists all4,0234\{,\}023suppressed tokens with their intervention scores underresults/compression\_suppressed\_all\_families\.tex\.

## Appendix DAdditional experimental details

### D\.1Prompt templates and GPQA formatting

We use the default lm\-evaluation\-harness prompt templates for all tasks\. In particular, the core format follows the standard task prompt pattern \(e\.g\., “Problem: …Answer: …”\)\.

For GPQA\-main, we format prompts so the model first provides an explanation and then outputs a final answer choice\. We also shuffle the answer choices per example to avoid systematic bias toward any fixed label\. For bias estimation, sampled positions are taken from the explanation\-first completion before the final answer token sequence\.

### D\.2Rollout\-generation settings

For bias estimation, we use dataset\-specific rollout budgets and context lengths: MATH uses 64 rollouts per problem with train/test caps of 1024/1024 tokens and 128 sampled positions per rollout; GSM8K uses 128 rollouts with 512/1024 caps and 256 positions; GPQA uses 128 rollouts with 512/4096 caps and 128 positions\.

### D\.3Tilting parameterτ\\tauand additive smoothingα\\alpha

We tune the additive\-smoothing constantα\\alpha\(defined in Section[4\.2](https://arxiv.org/html/2607.22837#S4.SS2)\) on validation\.

For accuracy experiments, the log\-indicator rewardrlog\-accuracy∈\{−∞,0\}r\_\{\\text\{log\-accuracy\}\}\\in\\\{\-\\infty,0\\\}makesτ\\tauinert:er/τ∈\{0,1\}e^\{r/\\tau\}\\in\\\{0,1\\\}for anyτ\>0\\tau\>0, so the IPS estimator in Equation \([6](https://arxiv.org/html/2607.22837#S4.E6)\) reduces to an accuracy\-weighted count and the tilt parameter drops out\. We therefore only reportα\\alpha: MATH usesα=0\.052\\alpha=0\.052, GSM8K usesα=0\.110\\alpha=0\.110, and GPQA usesα=0\.007\\alpha=0\.007\. For the compression experiment, Gemma usesα=0\.738\\alpha=0\.738withτ=1\.1\\tau=1\.1, Llama usesα=0\.052\\alpha=0\.052withτ=0\.7\\tau=0\.7, and Qwen usesα=0\.465\\alpha=0\.465withτ=0\.8\\tau=0\.8\.

Sweeping\(τ,α\)\(\\tau,\\alpha\)during model selection is inexpensive because the estimator factorizes as𝐙^=𝐯1/τ​𝐌\\widehat\{\\mathbf\{Z\}\}=\\mathbf\{v\}^\{1/\\tau\}\\mathbf\{M\}\(Proposition[5](https://arxiv.org/html/2607.22837#Thmtheorem5)\): the rollout statistics𝐌\\mathbf\{M\}are computed once, and each\(τ,α\)\(\\tau,\\alpha\)configuration is then obtained by a single reward\-weighted matrix–vector product followed by the additive\-smoothing step, without rescanning the rollouts\.

### D\.4Cluster\-tied baseline

The cluster\-tied baseline buildsK=32K\{=\}32clusters from whitespace\-split answer tokens and ties one bias value per cluster\. Concretely, we runkk\-means on token/word representations from the answer corpus, then assign each vocabulary token to its nearest centroid and share the same learned bias within each cluster\. This yields a low\-parameter tied\-bias baseline used for fairness comparisons against full\-vocabulary logit bias and LoRA\-GRPO\.

### D\.5LoRA\-GRPO baseline

LoRA\-GRPO uses rankr=16r=16,α=32\\alpha=32, dropout0\.050\.05, target modulesq\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj,down\_proj, bf16, and gradient checkpointing, trained for one epoch\. For accuracy experiments, the best checkpoint is selected among 3 validation evaluations\.

Dataset\-specific settings are:

- •GPQA: learning rate5×10−55\\times 10^\{\-5\}, gradient accumulation 16, max completion 1024, eval max new tokens 4096\.
- •MATH/GSM8K: learning rate2×10−42\\times 10^\{\-4\}, gradient accumulation 8, max completion 1024 for MATH and 512 for GSM8K, eval max new tokens 1024\.

### D\.6Compression experiment

The compression experiment evaluates three model identifiers:google/gemma\-2\-9b\-it, meta\-llama/Meta\-Llama\-3\.1\-8B\-Instruct, andQwen/Qwen3\-4B\.

For length\-aware LoRA\-GRPO, we train one epoch on MATH with objective modecorr\*T/length,T=1024T\{=\}1024, learning rate2×10−42\\times 10^\{\-4\}, batch size 2, gradient accumulation 8, and LoRA\(r,α,dropout\)=\(16,32,0\.05\)\(r,\\alpha,\\text\{dropout\}\)=\(16,32,0\.05\)\. We keep the best checkpoint among 32 evaluations\.

For both logit bias and LoRA\-GRPO in the compression experiment, model selection chooses the checkpoint orτ\\tau\-α\\alphapair that minimizes mean completion length in tokens among configurations whose accuracy is the closest to the base model\.

The compression reward for the logit\-bias method isrlengthr\_\{\\mathrm\{length\}\}, first defined in Equation \([8](https://arxiv.org/html/2607.22837#S6.E8)\) \(Section[6\.1](https://arxiv.org/html/2607.22837#S6.SS1)\):

rlength​\(y\)=ln⁡\(Tlength⁡\(y\)\)\.r\_\{\\mathrm\{length\}\}\(y\)=\\ln\\\!\\left\(\\frac\{T\}\{\\operatorname\{length\}\(y\)\}\\right\)\.

### D\.7Compute resources and minor evaluation variance from batch chunking

Experiments were run on multiple GPU setups, including RTX 4090, RTX 5090, A100, H100, and H200\. The code supports multi\-GPU execution\. During evaluation, prompts are batched withpadding=Trueand left padding, so tensor shapes depend on which examples are grouped together\. ChangingCONFIG\_EVAL\_CF\_WORKER\_CHUNK\_SIZEchanges how validation problems are partitioned across workers, which in turn changes batch composition and padding shapes\.

Because evaluation uses fp16 on GPU, different batch shapes can induce small differences in floating\-point execution order inside GPU kernels\. These tiny numerical changes can slightly perturb logits\. In borderline cases where two next\-token logits are nearly tied, this can flip the greedyargmax, changing the generated completion and, for a small number of examples, accuracy\. Subject to the caveat above, the reported results are reproducible on a 4×\\timesH200 machine withCONFIG\_EVAL\_CF\_BATCH\_SIZE=128andCONFIG\_EVAL\_CF\_WORKER\_CHUNK\_SIZE=0\.

### D\.8Bootstrap confidence intervals

We report 95% percentile bootstrap confidence intervals for all evaluation metrics\. The procedure is the same in every experiment; only the resampling unit and the per\-replicate statistic differ between the accuracy \(non\-paired bootstrap\) and compression \(paired bootstrap\) settings\. In each case we drawB=10,000B=10\{,\}000bootstrap replicates with fixed random seeds for reproducibility\. Each replicate resamplesnnevaluation examples with replacement, wherennis the number of evaluated examples in the corresponding evaluation set \(Table[D\.6](https://arxiv.org/html/2607.22837#A4.T6)\), computes a replicate statistic, and the reported interval is the pair of empirical2\.52\.5th and97\.597\.5th percentiles of that statistic across theBBreplicates\. HereBBcontrols only the Monte Carlo precision of the percentile estimates; it is not treated as additional data and does not change the statistical sample size, which is the evaluation\-set sizenn\. Consequently, smaller evaluation sets, such as GPQA\-main withn=45n=45, yield correspondingly wider intervals\.

Table D\.6:Evaluation sample sizes used for bootstrap confidence intervals\. Each bootstrap replicate resamplesnnexamples with replacement from the corresponding evaluation set\.#### D\.8\.1Accuracy

For accuracy we resample over examples\. Let

ci=𝟏​\{mxi​\(y1:T\(i\)\)=m⋆,i\},i=1,…,n,c\_\{i\}=\\mathbf\{1\}\\\{m\_\{x\_\{i\}\}\(y^\{\(i\)\}\_\{1:T\}\)=m\_\{\\star,i\}\\\},\\qquad i=1,\\ldots,n,denote the correctness indicator for exampleii\. For replicateb∈\{1,…,B\}b\\in\\\{1,\\ldots,B\\\}, letci∗\(b\)c\_\{i\}^\{\*\(b\)\}be theii\-th indicator drawn with replacement from\{ci\}i=1n\\\{c\_\{i\}\\\}\_\{i=1\}^\{n\}, and form the replicate mean

c¯n∗\(b\)=1n​∑i=1nci∗\(b\)\.\\bar\{c\}\_\{n\}^\{\*\(b\)\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}c\_\{i\}^\{\*\(b\)\}\.The reported 95% confidence interval is

\[Q0\.025​\(\{c¯n∗\(b\)\}b=1B\),Q0\.975​\(\{c¯n∗\(b\)\}b=1B\)\],\\left\[Q\_\{0\.025\}\\left\(\\\{\\bar\{c\}\_\{n\}^\{\*\(b\)\}\\\}\_\{b=1\}^\{B\}\\right\),Q\_\{0\.975\}\\left\(\\\{\\bar\{c\}\_\{n\}^\{\*\(b\)\}\\\}\_\{b=1\}^\{B\}\\right\)\\right\],whereQpQ\_\{p\}denotes the empiricalpp\-quantile\.

#### D\.8\.2Compression

For comparisons in the compression experiment, the baseline and candidate methods are evaluated on the same examples\. Let

di=ℓicandidate−ℓibaselined\_\{i\}=\\ell\_\{i\}^\{\\mathrm\{candidate\}\}\-\\ell\_\{i\}^\{\\mathrm\{baseline\}\}be the per\-example paired difference in completion length in tokens, or in the relevant numeric metric\. For replicateb∈\{1,…,B\}b\\in\\\{1,\\ldots,B\\\}, letdi∗\(b\)d\_\{i\}^\{\*\(b\)\}be theii\-th paired difference drawn with replacement from\{di\}i=1n\\\{d\_\{i\}\\\}\_\{i=1\}^\{n\}, and form the replicate mean

d¯n∗\(b\)=1n​∑i=1ndi∗\(b\)\.\\bar\{d\}\_\{n\}^\{\*\(b\)\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}d\_\{i\}^\{\*\(b\)\}\.We report the2\.52\.5th and97\.597\.5th empirical percentiles of\{d¯n∗\(b\)\}b=1B\\\{\\bar\{d\}\_\{n\}^\{\*\(b\)\}\\\}\_\{b=1\}^\{B\}as the 95% confidence interval for the mean difference\.

## Appendix EUnbiasedness of the IPS estimator

### E\.1Matrix form of the IPS estimator

###### Proposition 5\.

Define

vn:=exp⁡\(r\(n\)\),Mn,y:=1N​P​∑p=1P𝟏​\{yt\(n,p\)\(n,p\)=y\}π0​\(y∣x\(n\),y1:t\(n,p\)−1\(n,p\)\)\.v\_\{n\}:=\\exp\\\!\\left\(r^\{\(n\)\}\\right\),\\qquad M\_\{n,y\}:=\\frac\{1\}\{NP\}\\sum\_\{p=1\}^\{P\}\\frac\{\\mathbf\{1\}\\\!\\left\\\{y\_\{t^\{\(n,p\)\}\}^\{\(n,p\)\}=y\\right\\\}\}\{\\pi\_\{0\}\\\!\\left\(y\\mid x^\{\(n\)\},y\_\{1:t^\{\(n,p\)\}\-1\}^\{\(n,p\)\}\\right\)\}\.Then the IPS estimator from Equation \([6](https://arxiv.org/html/2607.22837#S4.E6)\) satisfies

Z^​\(y\)=∑n=1Nvn1/τ​Mn,y,\\widehat\{Z\}\(y\)=\\sum\_\{n=1\}^\{N\}v\_\{n\}^\{1/\\tau\}M\_\{n,y\},equivalently,

𝐙^=𝐯1/τ​𝐌\.\\widehat\{\\mathbf\{Z\}\}=\\mathbf\{v\}^\{\\,1/\\tau\}\\mathbf\{M\}\.

###### Proof\.

Indexing each entry of𝒟\\mathcal\{D\}by its origin\(n,p\)\(n,p\)in Algorithm[1](https://arxiv.org/html/2607.22837#alg1), Equation \([6](https://arxiv.org/html/2607.22837#S4.E6)\) can be written as

Z^​\(y\):=1N​P​∑n=1N∑p=1P𝟏​\{yt\(n,p\)\(n,p\)=y\}​er\(n,p\)/τπ0​\(y\|x\(n,p\),y1:t\(n,p\)−1\(n,p\)\)\.\\widehat\{Z\}\(y\):=\\frac\{1\}\{NP\}\\sum\_\{n=1\}^\{N\}\\sum\_\{p=1\}^\{P\}\\frac\{\\mathbf\{1\}\\\!\\left\\\{y\_\{t^\{\(n,p\)\}\}^\{\(n,p\)\}=y\\right\\\}\\,e^\{r^\{\(n,p\)\}/\\tau\}\}\{\\pi\_\{0\}\\\!\\left\(y\\,\\bigm\|\\,x^\{\(n,p\)\},\\,y\_\{1:t^\{\(n,p\)\}\-1\}^\{\(n,p\)\}\\right\)\}\.In Algorithm[1](https://arxiv.org/html/2607.22837#alg1), each replicated sample satisfiesx\(n,p\)=x\(n\)x^\{\(n,p\)\}=x^\{\(n\)\}andr\(n,p\)=r\(n\)r^\{\(n,p\)\}=r^\{\(n\)\}, soer\(n,p\)/τ=vn1/τe^\{r^\{\(n,p\)\}/\\tau\}=v\_\{n\}^\{1/\\tau\}is independent ofpp\. Factoring this term outside the inner sum gives

Z^​\(y\)=∑n=1Nvn1/τ​\(1N​P​∑p=1P𝟏​\{yt\(n,p\)\(n,p\)=y\}π0​\(y∣x\(n\),y1:t\(n,p\)−1\(n,p\)\)\):=∑n=1Nvn1/τ​Mn,y\.\\widehat\{Z\}\(y\)=\\sum\_\{n=1\}^\{N\}v\_\{n\}^\{1/\\tau\}\\left\(\\frac\{1\}\{NP\}\\sum\_\{p=1\}^\{P\}\\frac\{\\mathbf\{1\}\\\!\\left\\\{y\_\{t^\{\(n,p\)\}\}^\{\(n,p\)\}=y\\right\\\}\}\{\\pi\_\{0\}\\\!\\left\(y\\mid x^\{\(n\)\},y\_\{1:t^\{\(n,p\)\}\-1\}^\{\(n,p\)\}\\right\)\}\\right\):=\\sum\_\{n=1\}^\{N\}v\_\{n\}^\{1/\\tau\}M\_\{n,y\}\.Stacking overy∈𝕍y\\in\\mathbb\{V\}yields𝐙^=𝐯1/τ​𝐌\\widehat\{\\mathbf\{Z\}\}=\\mathbf\{v\}^\{\\,1/\\tau\}\\mathbf\{M\}\. ∎

Factoring𝐙^=𝐯1/τ​𝐌\\widehat\{\\mathbf\{Z\}\}=\\mathbf\{v\}^\{1/\\tau\}\\mathbf\{M\}separates rollout statistics from reward weights\. The matrix𝐌\\mathbf\{M\}depends only on sampled tokens and base\-model probabilities, so it can be computed once; changing the reward transform or temperature only changes𝐯1/τ\\mathbf\{v\}^\{1/\\tau\}, making recomputation a single matrix\-vector product\.

### E\.2Unbiasedness

###### Proposition 6\.

LetZ^​\(y\)\\widehat\{Z\}\(y\)be the IPS estimator in Equation \([6](https://arxiv.org/html/2607.22837#S4.E6)\), computed on the dataset𝒟\\mathcal\{D\}produced by Algorithm[1](https://arxiv.org/html/2607.22837#alg1)\. Assumeπ0​\(y∣x,y1:t−1\)\>0\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)\>0for every promptxxin the support ofρ\\rho, everyt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, and every prefixy1:t−1y\_\{1:t\-1\}reachable underp0\(⋅∣x\)p\_\{0\}\(\\cdot\\mid x\)\. Then

𝔼​\[Z^​\(y\)\]=Zavg​\(y\),\\mathbb\{E\}\[\\widehat\{Z\}\(y\)\]=Z\_\{\\mathrm\{avg\}\}\(y\),where the expectation is over the randomness of the estimator\.

###### Proof\.

By linearity of expectation, it suffices to show that each summand has meanZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)\. Fix\(n,p\)\(n,p\)and condition onx\(n,p\)=xx^\{\(n,p\)\}=x,t\(n,p\)=tt^\{\(n,p\)\}=t\. The autoregressive factorizationp0​\(y1:T∣x\)=p0​\(y1:t−1∣x\)​π0​\(yt∣x,y1:t−1\)​p0​\(yt\+1:T∣x,y1:t\)p\_\{0\}\(y\_\{1:T\}\\mid x\)=p\_\{0\}\(y\_\{1:t\-1\}\\mid x\)\\,\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\\,p\_\{0\}\(y\_\{t\+1:T\}\\mid x,y\_\{1:t\}\), together with the indicator𝟏​\{yt=y\}\\mathbf\{1\}\\\{y\_\{t\}=y\\\}, cancels theπ0​\(y∣x,y1:t−1\)\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)factor in the denominator, so

𝔼​\[𝟏​\{yt=y\}​er​\(x,Y\)/τπ0​\(y∣x,y1:t−1\)\|x,t\]=𝔼y1:t−1∼p0\(⋅∣x\)yt\+1:T∼p0\(⋅∣x,y1:t−1,y\)​\[er​\(x,y1:t−1,y,yt\+1:T\)/τ\]\.\\mathbb\{E\}\\\!\\left\[\\frac\{\\mathbf\{1\}\\\{y\_\{t\}=y\\\}\\,e^\{r\(x,Y\)/\\tau\}\}\{\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)\}\\,\\bigg\|\\,x,t\\right\]=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}y\_\{1:t\-1\}\\sim p\_\{0\}\(\\cdot\\mid x\)\\\\ y\_\{t\+1:T\}\\sim p\_\{0\}\(\\cdot\\mid x,y\_\{1:t\-1\},y\)\\end\{subarray\}\}\\\!\\left\[e^\{r\(x,\\,y\_\{1:t\-1\},\\,y,\\,y\_\{t\+1:T\}\)/\\tau\}\\right\]\.The right\-hand side is exactly the interventionaldo​\(yt=y\)\\mathrm\{do\}\(y\_\{t\}=y\)expectation that definesZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)in Equation \([5](https://arxiv.org/html/2607.22837#S4.E5)\): the importance weighting realizes the distributionp0\(⋅∣x,do\(yt=y\)\)p\_\{0\}\(\\cdot\\mid x,\\mathrm\{do\}\(y\_\{t\}=y\)\), drawing the prefix and suffix from the base model withyyfixed at positiontt\. Taking the outer expectation overx∼ρx\\sim\\rhoandt∼Unif​\{1,…,T\}t\\sim\\mathrm\{Unif\}\\\{1,\\dots,T\\\}via the tower rule matches Equation \([5](https://arxiv.org/html/2607.22837#S4.E5)\), so each summand has meanZavg​\(y\)Z\_\{\\mathrm\{avg\}\}\(y\)\. ∎

## Appendix FBias Parameterization Algorithms

This appendix provides the full algorithmic details for the two bias parameterizations introduced in Section[4](https://arxiv.org/html/2607.22837#S4)\. All take as input the dataset𝒟\\mathcal\{D\}produced by Algorithm[1](https://arxiv.org/html/2607.22837#alg1)\.

### F\.1Per\-token bias

Algorithm[2](https://arxiv.org/html/2607.22837#alg2)computes the per\-token IPS estimatorZ^​\(y\)\\widehat\{Z\}\(y\)from Equation \([6](https://arxiv.org/html/2607.22837#S4.E6)\) by accumulating, for each sampled position, the weighter/τ/π0​\(y∣x,y1:t−1\)e^\{r/\\tau\}/\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)into the bin of the observed token identity\. The per\-token biasδ^​\(y\)=log⁡Z^​\(y\)\+c0\\widehat\{\\delta\}\(y\)=\\log\\widehat\{Z\}\(y\)\+c\_\{0\}is then obtained by the step in Section[4\.2](https://arxiv.org/html/2607.22837#S4.SS2)\.

Algorithm 2PerTokenIPS: per\-token IPS estimatorZ^​\(y\)\\widehat\{Z\}\(y\)1:dataset

𝒟=\{\(x\(n,p\),y1:T\(n,p\),t\(n,p\),r\(n,p\)\)\}n,p=1N,P\\mathcal\{D\}=\\\{\(x^\{\(n,p\)\},\\,y\_\{1:T\}^\{\(n,p\)\},\\,t^\{\(n,p\)\},\\,r^\{\(n,p\)\}\)\\\}\_\{n,p=1\}^\{N,P\}, base model

π0\\pi\_\{0\}, temperature

τ\\tau
2:

S​\(⋅\)←0S\(\\cdot\)\\leftarrow 0
3:for

\(x,y1:T,t,r\)∈𝒟\(x,\\,y\_\{1:T\},\\,t,\\,r\)\\in\\mathcal\{D\}do

4:

π←π0​\(yt\|x,y1:t−1\)\\pi\\leftarrow\\pi\_\{0\}\\bigl\(y\_\{t\}\\,\\bigm\|\\,x,\\,y\_\{1:t\-1\}\\bigr\)⊳\\trianglerightbase\-model probability of the on\-policy token at positiontt

5:

S​\(yt\)←S​\(yt\)\+er/τ/πS\(y\_\{t\}\)\\leftarrow S\(y\_\{t\}\)\+e^\{r/\\tau\}\\,/\\,\\pi
6:endfor

7:for alltoken

yywith

S​\(y\)\>0S\(y\)\>0do

8:

Z^​\(y\)←S​\(y\)/\(N​P\)\\widehat\{Z\}\(y\)\\leftarrow S\(y\)/\(NP\)
9:endfor

10:return

\{Z^​\(y\)\}y∈𝕍\+\\\{\\widehat\{Z\}\(y\)\\\}\_\{y\\in\\mathbb\{V\}\_\{\+\}\}

### F\.2Cluster\-tied bias

The cluster\-tied parameterization reduces the number of learned bias values by tying together vocabulary tokens with similar string embeddings\. It first builds a vocabulary partition\{C​\(1\),…,C​\(K\)\}\\\{C\(1\),\\dots,C\(K\)\\\}, then replaces the per\-token IPS estimate by the average estimate within the token’s cluster\.

Let

detok:𝕍∗→Σ∗\\operatorname\{detok\}:\\mathbb\{V\}^\{\*\}\\to\\Sigma^\{\*\}denote the tokenizer detokenization map, which takes a token*sequence*as input\. For a single tokeny∈𝕍y\\in\\mathbb\{V\}, write\(y\)\(y\)for the length\-one sequence containing it and define its string form as the detokenization of that singleton sequence,

str⁡\(y\):=detok⁡\(\(y\)\)\.\\operatorname\{str\}\(y\):=\\operatorname\{detok\}\\bigl\(\(y\)\\bigr\)\.Thusstr⁡\(y\)\\operatorname\{str\}\(y\)is the tokenizer’s surface string for the one\-token sequence\(y\)\(y\)\. We cluster the vocabulary token by token, so each token is embedded through its own surface string rather than through any surrounding sentence\.

##### Cluster construction\.

Clusters are built from an auxiliary prompt set

𝒳c=\{xc\(m\)\}m=1Nc,\\mathcal\{X\}\_\{c\}=\\\{x\_\{c\}^\{\(m\)\}\\\}\_\{m=1\}^\{N\_\{c\}\},chosen independently of the rollout prompts used to form𝒟\\mathcal\{D\}\. For each auxiliary prompt, we sample a completion

y1:Tc,\(m\)∼p0\(⋅∣xc\(m\)\)y\_\{1:T\}^\{c,\(m\)\}\\sim p\_\{0\}\(\\cdot\\mid x\_\{c\}^\{\(m\)\}\)and detokenize it to a string

u\(m\):=detok⁡\(y1:Tc,\(m\)\)\.u^\{\(m\)\}:=\\operatorname\{detok\}\\\!\\left\(y\_\{1:T\}^\{c,\(m\)\}\\right\)\.We then form a string corpus

𝒮:=\(⋃m=1Ncwords⁡\(u\(m\)\)\)∪𝒮special,\\mathcal\{S\}:=\\left\(\\bigcup\_\{m=1\}^\{N\_\{c\}\}\\operatorname\{words\}\(u^\{\(m\)\}\)\\right\)\\cup\\mathcal\{S\}\_\{\\mathrm\{special\}\},wherewords⁡\(u\)\\operatorname\{words\}\(u\)denotes the set of whitespace\-delimited words inuu, and𝒮special\\mathcal\{S\}\_\{\\mathrm\{special\}\}contains tokenizer\-specific vocabulary strings that should be clustered explicitly, such as EOS, control, whitespace\-only, newline, and byte\-fallback tokens\.

We embed eachs∈𝒮s\\in\\mathcal\{S\}withsentence\-transformers/all\-mpnet\-base\-v2\(Sentence\-BERT on MPNet;[45](https://arxiv.org/html/2607.22837#bib.bib45),[52](https://arxiv.org/html/2607.22837#bib.bib52),[49](https://arxiv.org/html/2607.22837#bib.bib49)\), i\.e\.,Encoder​\(s\)∈ℝd\\mathrm\{Encoder\}\(s\)\\in\\mathbb\{R\}^\{d\}, runKK\-means, and obtain centroids\{μk\}k=1K\\\{\\mu\_\{k\}\\\}\_\{k=1\}^\{K\}\. Each vocabulary token is assigned to its nearest centroid by

c​\(y\):=arg⁡mink∈\{1,…,K\}⁡‖Encoder​\(str⁡\(y\)\)−μk‖22,y∈𝕍\.c\(y\):=\\arg\\min\_\{k\\in\\\{1,\\dots,K\\\}\}\\left\\\|\\mathrm\{Encoder\}\(\\operatorname\{str\}\(y\)\)\-\\mu\_\{k\}\\right\\\|\_\{2\}^\{2\},\\qquad y\\in\\mathbb\{V\}\.This induces the partition

C​\(k\):=\{y∈𝕍:c​\(y\)=k\},k=1,…,K\.C\(k\):=\\\{y\\in\\mathbb\{V\}:c\(y\)=k\\\},\\qquad k=1,\\dots,K\.

##### Cluster\-tied estimation\.

Given the partition, we first compute the per\-token IPS estimates\{Z^​\(y\)\}y∈𝕍\+\\\{\\widehat\{Z\}\(y\)\\\}\_\{y\\in\\mathbb\{V\}\_\{\+\}\}using Algorithm[2](https://arxiv.org/html/2607.22837#alg2)\. Then each cluster receives the average of its observed token estimates:

Z¯​\(k\):=1\|C​\(k\)∩𝕍\+\|​∑y∈C​\(k\)∩𝕍\+Z^​\(y\)\.\\bar\{Z\}\(k\):=\\frac\{1\}\{\|C\(k\)\\cap\\mathbb\{V\}\_\{\+\}\|\}\\sum\_\{y\\in C\(k\)\\cap\\mathbb\{V\}\_\{\+\}\}\\widehat\{Z\}\(y\)\.The tied estimate for a token is

Ztied​\(y\):=Z¯​\(c​\(y\)\)\.Z\_\{\\mathrm\{tied\}\}\(y\):=\\bar\{Z\}\(c\(y\)\)\.
Algorithm 3Cluster\-tiedZ¯\\bar\{Z\}estimation1:rollout dataset

𝒟\\mathcal\{D\}, base model

π0\\pi\_\{0\}, temperature

τ\\tau
2:auxiliary prompts

𝒳c\\mathcal\{X\}\_\{c\}, embedding map

Encoder\\mathrm\{Encoder\}, number of clusters

KK
3:

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset
4:for all

xc∈𝒳cx\_\{c\}\\in\\mathcal\{X\}\_\{c\}do

5:Sample

y1:Tc∼p0\(⋅∣xc\)y\_\{1:T\}^\{c\}\\sim p\_\{0\}\(\\cdot\\mid x\_\{c\}\)
6:

u←detok⁡\(y1:Tc\)u\\leftarrow\\operatorname\{detok\}\(y\_\{1:T\}^\{c\}\)⊳\\trianglerightdecode the rollout to a text string

7:

𝒮←𝒮∪words⁡\(u\)\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\operatorname\{words\}\(u\)⊳\\trianglerightwords⁡\(⋅\)\\operatorname\{words\}\(\\cdot\): split into whitespace\-delimited words

8:endfor

9:

𝒮←𝒮∪𝒮special\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\mathcal\{S\}\_\{\\mathrm\{special\}\}
10:Fit

KK\-means to

\{Encoder​\(s\):s∈𝒮\}\\\{\\mathrm\{Encoder\}\(s\):s\\in\\mathcal\{S\}\\\}, obtaining centroids

\{μk\}k=1K\\\{\\mu\_\{k\}\\\}\_\{k=1\}^\{K\}
11:for all

y∈𝕍y\\in\\mathbb\{V\}do

12:

c​\(y\)←arg⁡mink⁡‖Encoder​\(str⁡\(y\)\)−μk‖22c\(y\)\\leftarrow\\arg\\min\_\{k\}\\left\\\|\\mathrm\{Encoder\}\(\\operatorname\{str\}\(y\)\)\-\\mu\_\{k\}\\right\\\|\_\{2\}^\{2\}
13:endfor

14:

\{Z^​\(y\)\}y∈𝕍\+←PerTokenIPS​\(𝒟,π0,τ\)\\\{\\widehat\{Z\}\(y\)\\\}\_\{y\\in\\mathbb\{V\}\_\{\+\}\}\\leftarrow\\textsc\{PerTokenIPS\}\(\\mathcal\{D\},\\pi\_\{0\},\\tau\)⊳\\trianglerightAlgorithm[2](https://arxiv.org/html/2607.22837#alg2)

15:for

k=1,…,Kk=1,\\dots,Kdo

16:

C​\(k\)←\{y∈𝕍:c​\(y\)=k\}C\(k\)\\leftarrow\\\{y\\in\\mathbb\{V\}:c\(y\)=k\\\}
17:

Z¯​\(k\)←1\|C​\(k\)∩𝕍\+\|​∑y∈C​\(k\)∩𝕍\+Z^​\(y\)\\bar\{Z\}\(k\)\\leftarrow\\frac\{1\}\{\|C\(k\)\\cap\\mathbb\{V\}\_\{\+\}\|\}\\sum\_\{y\\in C\(k\)\\cap\\mathbb\{V\}\_\{\+\}\}\\widehat\{Z\}\(y\)
18:endfor

19:for all

y∈𝕍y\\in\\mathbb\{V\}do

20:

Ztied​\(y\)←Z¯​\(c​\(y\)\)Z\_\{\\mathrm\{tied\}\}\(y\)\\leftarrow\\bar\{Z\}\(c\(y\)\)
21:endfor

22:return

\{Ztied​\(y\)\}y∈𝕍\\\{Z\_\{\\mathrm\{tied\}\}\(y\)\\\}\_\{y\\in\\mathbb\{V\}\}

## Appendix GProofs

This appendix collects all deferred proofs\. We first establish two technical lemmas \(Appendices[G\.1](https://arxiv.org/html/2607.22837#A7.SS1)–[G\.2](https://arxiv.org/html/2607.22837#A7.SS2)\) that are used throughout, then prove the results from the main text in the order they appear\.

### G\.1Softmax–KL bound

###### Lemma 8\.

For allu,z,z′∈ℝ\|𝕍\|u,z,z^\{\\prime\}\\in\\mathbb\{R\}^\{\|\\mathbb\{V\}\|\},

KL⁡\(softmax⁡\(u\+z\)∥softmax⁡\(u\+z′\)\)≤14​‖z−z′‖22\.\\operatorname\{KL\}\\\!\\bigl\(\\operatorname\{softmax\}\(u\+z\)\\,\\\|\\,\\operatorname\{softmax\}\(u\+z^\{\\prime\}\)\\bigr\)\\leq\\frac\{1\}\{4\}\\\|z\-z^\{\\prime\}\\\|\_\{2\}^\{2\}\.

###### Proof\.

Define the log\-partition functionψ​\(s\):=log⁡\(∑a∈𝕍eua\+sa\)\\psi\(s\):=\\log\\\!\\bigl\(\\sum\_\{a\\in\\mathbb\{V\}\}e^\{u\_\{a\}\+s\_\{a\}\}\\bigr\)\. Its gradient is∇ψ\(s\)=softmax\(u\+s\)=:p\(s\)\\nabla\\psi\(s\)=\\operatorname\{softmax\}\(u\+s\)=:p\(s\), and its Hessian is∇2ψ​\(s\)=Diag⁡\(p​\(s\)\)−p​\(s\)​p​\(s\)⊤\\nabla^\{2\}\\psi\(s\)=\\operatorname\{Diag\}\(p\(s\)\)\-p\(s\)\\,p\(s\)^\{\\top\}\. By Gershgorin’s theorem, every eigenvalue of∇2ψ​\(s\)\\nabla^\{2\}\\psi\(s\)lies in\[0,12\]\[0,\\tfrac\{1\}\{2\}\]: theii\-th diagonal entry ispi​\(1−pi\)p\_\{i\}\(1\-p\_\{i\}\)and theii\-th absolute row sum off the diagonal is∑j≠ipi​pj=pi​\(1−pi\)\\sum\_\{j\\neq i\}p\_\{i\}p\_\{j\}=p\_\{i\}\(1\-p\_\{i\}\), so theii\-th Gershgorin disc is\[0,2​pi​\(1−pi\)\]⊆\[0,12\]\[0,2p\_\{i\}\(1\-p\_\{i\}\)\]\\subseteq\[0,\\tfrac\{1\}\{2\}\]\. Henceψ\\psihas12\\tfrac\{1\}\{2\}\-Lipschitz gradient\.

By Theorem 2\.1\.5 ofNesterov \[[35](https://arxiv.org/html/2607.22837#bib.bib35)\], the Bregman divergence of a convex function withLL\-Lipschitz gradient satisfiesDψ​\(z′,z\)≤L2​‖z′−z‖22D\_\{\\psi\}\(z^\{\\prime\},z\)\\leq\\frac\{L\}\{2\}\\\|z^\{\\prime\}\-z\\\|\_\{2\}^\{2\}\. WithL=12L=\\tfrac\{1\}\{2\}this givesDψ​\(z′,z\)≤14​‖z′−z‖22D\_\{\\psi\}\(z^\{\\prime\},z\)\\leq\\frac\{1\}\{4\}\\\|z^\{\\prime\}\-z\\\|\_\{2\}^\{2\}\. It remains to identifyDψD\_\{\\psi\}with the KL divergence\. Letp=softmax⁡\(u\+z\)=∇ψ​\(z\)p=\\operatorname\{softmax\}\(u\+z\)=\\nabla\\psi\(z\)\. Then

Dψ​\(z′,z\)\\displaystyle D\_\{\\psi\}\(z^\{\\prime\},z\)=ψ​\(z′\)−ψ​\(z\)−⟨∇ψ​\(z\),z′−z⟩\\displaystyle=\\psi\(z^\{\\prime\}\)\-\\psi\(z\)\-\\langle\\nabla\\psi\(z\),z^\{\\prime\}\-z\\rangle=ψ​\(z′\)−ψ​\(z\)−∑apa​\(za′−za\)\\displaystyle=\\psi\(z^\{\\prime\}\)\-\\psi\(z\)\-\\sum\_\{a\}p\_\{a\}\(z^\{\\prime\}\_\{a\}\-z\_\{a\}\)=∑apa​\[log⁡pa−\(ua\+za′−ψ​\(z′\)\)\]\\displaystyle=\\sum\_\{a\}p\_\{a\}\\bigl\[\\log p\_\{a\}\-\(u\_\{a\}\+z^\{\\prime\}\_\{a\}\-\\psi\(z^\{\\prime\}\)\)\\bigr\]=∑apa​log⁡pasoftmax\(u\+z′\)a=KL⁡\(softmax⁡\(u\+z\)∥softmax⁡\(u\+z′\)\)\.∎\\displaystyle=\\sum\_\{a\}p\_\{a\}\\log\\frac\{p\_\{a\}\}\{\\operatorname\{softmax\}\(u\+z^\{\\prime\}\)\_\{a\}\}=\\operatorname\{KL\}\\\!\\bigl\(\\operatorname\{softmax\}\(u\+z\)\\,\\\|\\,\\operatorname\{softmax\}\(u\+z^\{\\prime\}\)\\bigr\)\.\\qed

### G\.2KL lower bound via exponential tilting

###### Lemma 9\.

Fix a promptxx\. Letr:𝒳×𝕍T→\[0,R\]r:\\mathcal\{X\}\\times\\mathbb\{V\}^\{T\}\\to\[0,R\],σ02:=VarY∼p0\(⋅∣x\)⁡\(r​\(x,Y\)\)\\sigma^\{2\}\_\{0\}:=\\operatorname\{Var\}\_\{Y\\sim p\_\{0\}\(\\cdot\\mid x\)\}\(r\(x,Y\)\), andqs​\(y1:T∣x\):=p0​\(y1:T∣x\)​es​r​\(x,y1:T\)/Zsq\_\{s\}\(y\_\{1:T\}\\mid x\):=p\_\{0\}\(y\_\{1:T\}\\mid x\)\\,e^\{s\\,r\(x,y\_\{1:T\}\)\}/Z\_\{s\}fors≥0s\\geq 0\. Then

KL⁡\(qs∥p0\)=∫0st​Varqt⁡\(r\)​𝑑t≥σ02R2​\(1−e−R​s​\(1\+R​s\)\)\.\\operatorname\{KL\}\(q\_\{s\}\\\|p\_\{0\}\)=\\int\_\{0\}^\{s\}t\\,\\operatorname\{Var\}\_\{q\_\{t\}\}\(r\)\\,dt\\;\\geq\\;\\frac\{\\sigma^\{2\}\_\{0\}\}\{R^\{2\}\}\\bigl\(1\-e^\{\-Rs\}\(1\+Rs\)\\bigr\)\.

###### Proof\.

LetΛ​\(s\):=log⁡𝔼p0​\[es​r​\(x,Y\)\]\\Lambda\(s\):=\\log\\mathbb\{E\}\_\{p\_\{0\}\}\[e^\{s\\,r\(x,Y\)\}\]\. ThenΛ′​\(s\)=𝔼qs​\[r\]\\Lambda^\{\\prime\}\(s\)=\\mathbb\{E\}\_\{q\_\{s\}\}\[r\]andΛ′′​\(s\)=Varqs⁡\(r\)\\Lambda^\{\\prime\\prime\}\(s\)=\\operatorname\{Var\}\_\{q\_\{s\}\}\(r\)\. Sincelog⁡\(qs/p0\)=s​r−Λ​\(s\)\\log\(q\_\{s\}/p\_\{0\}\)=sr\-\\Lambda\(s\), we haveKL⁡\(qs∥p0\)=s​Λ′​\(s\)−Λ​\(s\)\\operatorname\{KL\}\(q\_\{s\}\\\|p\_\{0\}\)=s\\Lambda^\{\\prime\}\(s\)\-\\Lambda\(s\)\. WritingF​\(s\):=s​Λ′​\(s\)−Λ​\(s\)F\(s\):=s\\Lambda^\{\\prime\}\(s\)\-\\Lambda\(s\), noteF′​\(s\)=s​Λ′′​\(s\)=s​Varqs⁡\(r\)F^\{\\prime\}\(s\)=s\\Lambda^\{\\prime\\prime\}\(s\)=s\\,\\operatorname\{Var\}\_\{q\_\{s\}\}\(r\)andF​\(0\)=0F\(0\)=0, soKL⁡\(qs∥p0\)=∫0st​Varqt⁡\(r\)​𝑑t\\operatorname\{KL\}\(q\_\{s\}\\\|p\_\{0\}\)=\\int\_\{0\}^\{s\}t\\,\\operatorname\{Var\}\_\{q\_\{t\}\}\(r\)\\,dt\.

For the lower bound, letV​\(t\):=Varqt⁡\(r\)V\(t\):=\\operatorname\{Var\}\_\{q\_\{t\}\}\(r\)\. ThenV′​\(t\)=𝔼qt​\[\(r−𝔼qt​r\)3\]V^\{\\prime\}\(t\)=\\mathbb\{E\}\_\{q\_\{t\}\}\[\(r\-\\mathbb\{E\}\_\{q\_\{t\}\}r\)^\{3\}\]\. Sincer∈\[0,R\]r\\in\[0,R\], we have\|r−𝔼qt​r\|≤R\|r\-\\mathbb\{E\}\_\{q\_\{t\}\}r\|\\leq R, so\(r−𝔼qt​r\)3≥−R​\(r−𝔼qt​r\)2\(r\-\\mathbb\{E\}\_\{q\_\{t\}\}r\)^\{3\}\\geq\-R\\,\(r\-\\mathbb\{E\}\_\{q\_\{t\}\}r\)^\{2\}, givingV′​\(t\)≥−R​V​\(t\)V^\{\\prime\}\(t\)\\geq\-R\\,V\(t\)and henceV​\(t\)≥σ02​e−R​tV\(t\)\\geq\\sigma^\{2\}\_\{0\}\\,e^\{\-Rt\}by Grönwall’s inequality\. ThereforeKL⁡\(qs∥p0\)≥σ02​∫0st​e−R​t​𝑑t=σ02R2​\(1−e−R​s​\(1\+R​s\)\)\\operatorname\{KL\}\(q\_\{s\}\\\|p\_\{0\}\)\\geq\\sigma^\{2\}\_\{0\}\\int\_\{0\}^\{s\}t\\,e^\{\-Rt\}\\,dt=\\frac\{\\sigma^\{2\}\_\{0\}\}\{R^\{2\}\}\(1\-e^\{\-Rs\}\(1\+Rs\)\)\. ∎

### G\.3KL\-regularized optimum \(Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\)\)

###### Proof\.

Writeμ:=p0\(⋅∣x\)\\mu:=p\_\{0\}\(\\cdot\\mid x\)andΩ:=𝕍T\\Omega:=\\mathbb\{V\}^\{T\}\. Introducing a Lagrange multiplierλ\\lambdafor the constraint∑y1:T∈Ωq​\(y1:T\)=1\\sum\_\{y\_\{1:T\}\\in\\Omega\}q\(y\_\{1:T\}\)=1:

ℒλ​\(q\)=∑y1:T∈Ωq​\(y1:T\)​\(r​\(x,y1:T\)τ−log⁡q​\(y1:T\)μ​\(y1:T\)\)\+λ​\(∑y1:T∈Ωq​\(y1:T\)−1\)\.\\mathcal\{L\}\_\{\\lambda\}\(q\)=\\sum\_\{y\_\{1:T\}\\in\\Omega\}q\(y\_\{1:T\}\)\\\!\\left\(\\frac\{r\(x,y\_\{1:T\}\)\}\{\\tau\}\-\\log\\frac\{q\(y\_\{1:T\}\)\}\{\\mu\(y\_\{1:T\}\)\}\\right\)\+\\lambda\\\!\\left\(\\sum\_\{y\_\{1:T\}\\in\\Omega\}q\(y\_\{1:T\}\)\-1\\right\)\.Setting∂ℒλ/∂q​\(y1:T\)=0\\partial\\mathcal\{L\}\_\{\\lambda\}/\\partial q\(y\_\{1:T\}\)=0for eachy1:T∈Ωy\_\{1:T\}\\in\\Omega:

r​\(x,y1:T\)τ−log⁡q​\(y1:T\)μ​\(y1:T\)−1\+λ=0⟹q​\(y1:T\)=μ​\(y1:T\)​er​\(x,y1:T\)/τ\+λ−1\.\\frac\{r\(x,y\_\{1:T\}\)\}\{\\tau\}\-\\log\\frac\{q\(y\_\{1:T\}\)\}\{\\mu\(y\_\{1:T\}\)\}\-1\+\\lambda=0\\qquad\\Longrightarrow\\qquad q\(y\_\{1:T\}\)=\\mu\(y\_\{1:T\}\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\+\\lambda\-1\}\.Normalizing:∑y1:T∈Ωq​\(y1:T\)=eλ−1​∑y1:T∈Ωμ​\(y1:T\)​er​\(x,y1:T\)/τ=eλ−1​Z​\(x\)=1\\sum\_\{y\_\{1:T\}\\in\\Omega\}q\(y\_\{1:T\}\)=e^\{\\lambda\-1\}\\sum\_\{y\_\{1:T\}\\in\\Omega\}\\mu\(y\_\{1:T\}\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\}=e^\{\\lambda\-1\}\\,Z\(x\)=1, soeλ−1=1/Z​\(x\)e^\{\\lambda\-1\}=1/Z\(x\)\. Substituting back givespτ⋆​\(y1:T∣x\)=p0​\(y1:T∣x\)​er​\(x,y1:T\)/τ/Z​\(x\)p^\{\\star\}\_\{\\tau\}\(y\_\{1:T\}\\mid x\)=p\_\{0\}\(y\_\{1:T\}\\mid x\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\}/Z\(x\)\. Uniqueness follows from strict concavity:−KL⁡\(q∥μ\)\-\\operatorname\{KL\}\(q\\\|\\mu\)is strictly concave inqq\(sinceq↦q​log⁡qq\\mapsto q\\log qis strictly convex\) and𝔼q​\[r\]\\mathbb\{E\}\_\{q\}\[r\]is linear\. ∎

### G\.4Optimal per\-step policy \(Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\)\)

###### Proof\.

Fixt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, a promptxx, and a prefixy1:t−1y\_\{1:t\-1\}\. From Equation \([2](https://arxiv.org/html/2607.22837#S3.E2)\), the joint probability of prefixy1:t−1y\_\{1:t\-1\}and next tokenyty\_\{t\}underpτ⋆p^\{\\star\}\_\{\\tau\}is

pτ⋆​\(y1:t−1,yt∣x\)=∑yt\+1:Tpτ⋆​\(y1:T∣x\)=1Z​\(x\)​∑yt\+1:Tp0​\(y1:T∣x\)​er​\(x,y1:T\)/τ\.p^\{\\star\}\_\{\\tau\}\(y\_\{1:t\-1\},y\_\{t\}\\mid x\)=\\sum\_\{y\_\{t\+1:T\}\}p^\{\\star\}\_\{\\tau\}\(y\_\{1:T\}\\mid x\)=\\frac\{1\}\{Z\(x\)\}\\sum\_\{y\_\{t\+1:T\}\}p\_\{0\}\(y\_\{1:T\}\\mid x\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\}\.Factorp0p\_\{0\}autoregressively asp0​\(y1:T∣x\)=p0​\(y1:t−1∣x\)​π0​\(yt∣x,y1:t−1\)​p0​\(yt\+1:T∣x,y1:t\)p\_\{0\}\(y\_\{1:T\}\\mid x\)=p\_\{0\}\(y\_\{1:t\-1\}\\mid x\)\\,\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\\,p\_\{0\}\(y\_\{t\+1:T\}\\mid x,y\_\{1:t\}\)\. The prefix term andπ0​\(yt∣x,y1:t−1\)\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)do not depend onyt\+1:Ty\_\{t\+1:T\}, so

pτ⋆​\(y1:t−1,yt∣x\)=p0​\(y1:t−1∣x\)​π0​\(yt∣x,y1:t−1\)Z​\(x\)​∑yt\+1:Tp0​\(yt\+1:T∣x,y1:t\)​er​\(x,y1:T\)/τ⏟=Z​\(x,y1:t−1,yt\),p^\{\\star\}\_\{\\tau\}\(y\_\{1:t\-1\},y\_\{t\}\\mid x\)=\\frac\{p\_\{0\}\(y\_\{1:t\-1\}\\mid x\)\\,\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\}\{Z\(x\)\}\\underbrace\{\\sum\_\{y\_\{t\+1:T\}\}p\_\{0\}\(y\_\{t\+1:T\}\\mid x,y\_\{1:t\}\)\\,e^\{r\(x,y\_\{1:T\}\)/\\tau\}\}\_\{=\\,Z\(x,\\,y\_\{1:t\-1\},\\,y\_\{t\}\)\},whereZ​\(x,y1:t−1,yt\)Z\(x,y\_\{1:t\-1\},y\_\{t\}\)is the soft value in Equation \([3](https://arxiv.org/html/2607.22837#S3.E3)\)\. Summing overyty\_\{t\}gives the prefix marginal:

pτ⋆​\(y1:t−1∣x\)=p0​\(y1:t−1∣x\)Z​\(x\)​∑y′∈𝕍π0​\(y′∣x,y1:t−1\)​Z​\(x,y1:t−1,y′\)\.p^\{\\star\}\_\{\\tau\}\(y\_\{1:t\-1\}\\mid x\)=\\frac\{p\_\{0\}\(y\_\{1:t\-1\}\\mid x\)\}\{Z\(x\)\}\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}\\pi\_\{0\}\(y^\{\\prime\}\\mid x,y\_\{1:t\-1\}\)\\,Z\(x,y\_\{1:t\-1\},y^\{\\prime\}\)\.Dividing the joint by the prefix marginal:

πτ⋆​\(yt∣x,y1:t−1\)=π0​\(yt∣x,y1:t−1\)​Z​\(x,y1:t−1,yt\)∑y′∈𝕍π0​\(y′∣x,y1:t−1\)​Z​\(x,y1:t−1,y′\)\.\\pi^\{\\star\}\_\{\\tau\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)=\\frac\{\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)\\,Z\(x,y\_\{1:t\-1\},y\_\{t\}\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}\\pi\_\{0\}\(y^\{\\prime\}\\mid x,y\_\{1:t\-1\}\)\\,Z\(x,y\_\{1:t\-1\},y^\{\\prime\}\)\}\.Sinceπ0​\(y∣x,y1:t−1\)=eℓy​\(x,y1:t−1\)/∑y′eℓy′​\(x,y1:t−1\)\\pi\_\{0\}\(y\\mid x,y\_\{1:t\-1\}\)=e^\{\\ell\_\{y\}\(x,y\_\{1:t\-1\}\)\}/\\sum\_\{y^\{\\prime\}\}e^\{\\ell\_\{y^\{\\prime\}\}\(x,y\_\{1:t\-1\}\)\}, the base\-model normalizer cancels and

πτ⋆\(yt∣x,y1:t−1\)=eℓyt​\(x,y1:t−1\)\+log⁡Z​\(x,y1:t−1,yt\)∑y′∈𝕍eℓy′​\(x,y1:t−1\)\+log⁡Z​\(x,y1:t−1,y′\)=softmax\(ℓ\(x,y1:t−1\)\+logZ\(x,y1:t−1,⋅\)\)yt\.∎\\pi^\{\\star\}\_\{\\tau\}\(y\_\{t\}\\mid x,y\_\{1:t\-1\}\)=\\frac\{e^\{\\ell\_\{y\_\{t\}\}\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,\\,y\_\{1:t\-1\},\\,y\_\{t\}\)\}\}\{\\sum\_\{y^\{\\prime\}\\in\\mathbb\{V\}\}e^\{\\ell\_\{y^\{\\prime\}\}\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,\\,y\_\{1:t\-1\},\\,y^\{\\prime\}\)\}\}=\\operatorname\{softmax\}\\\!\\bigl\(\\ell\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\\bigr\)\_\{y\_\{t\}\}\.\\qed

### G\.5Exact factorization implies fixed\-bias optimality \(Corollary[2](https://arxiv.org/html/2607.22837#Thmtheorem2)\)

###### Proof\.

Fixxx,tt, and a reachable prefixy1:t−1y\_\{1:t\-1\}\. By Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\),

πτ⋆\(⋅∣x,y1:t−1\)=softmax\(ℓ\(x,y1:t−1\)\+logZ\(x,y1:t−1,⋅\)\)\.\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)=\\operatorname\{softmax\}\\\!\\bigl\(\\ell\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\\bigr\)\.Under the exact\-factorization assumption,

log⁡Z​\(x,y1:t−1,⋅\)=δ\+ct​\(x,y1:t−1\)​1,\\log Z\(x,y\_\{1:t\-1\},\\cdot\)=\\delta\+c\_\{t\}\(x,y\_\{1:t\-1\}\)\\,\\mathbf\{1\},so

πτ⋆\(⋅∣x,y1:t−1\)=softmax\(ℓ\(x,y1:t−1\)\+δ\+ct\(x,y1:t−1\)1\)\.\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)=\\operatorname\{softmax\}\\\!\\bigl\(\\ell\(x,y\_\{1:t\-1\}\)\+\\delta\+c\_\{t\}\(x,y\_\{1:t\-1\}\)\\,\\mathbf\{1\}\\bigr\)\.Sincesoftmax⁡\(u\+c​1\)=softmax⁡\(u\)\\operatorname\{softmax\}\(u\+c\\,\\mathbf\{1\}\)=\\operatorname\{softmax\}\(u\)for anyuuand scalarcc,

πτ⋆\(⋅∣x,y1:t−1\)=softmax\(ℓ\(x,y1:t−1\)\+δ\)=πδ\(⋅∣x,y1:t−1\)\.\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)=\\operatorname\{softmax\}\\\!\\bigl\(\\ell\(x,y\_\{1:t\-1\}\)\+\\delta\\bigr\)=\\pi\_\{\\delta\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\.Thus the fixed\-bias policy matches the optimal per\-step policy at every reachable prefix\. By Proposition[1](https://arxiv.org/html/2607.22837#Thmtheorem1), equality of all per\-step conditionals implies equality of the induced trajectory laws:

pδ\(⋅∣x\)=pτ⋆\(⋅∣x\)\.∎p\_\{\\delta\}\(\\cdot\\mid x\)=p^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x\)\.\\qed

### G\.6Reward gap \(Theorem[4](https://arxiv.org/html/2607.22837#Thmtheorem4)\)

###### Proof\.

We first establish the bound for a fixed promptxx, then average overρ\\rho\. WriteJx​\(q\):=𝔼Y∼q\(⋅∣x\)​\[r​\(x,Y\)\]J\_\{x\}\(q\):=\\mathbb\{E\}\_\{Y\\sim q\(\\cdot\\mid x\)\}\[r\(x,Y\)\]for the per\-prompt reward andσ02​\(x\):=VarY∼p0\(⋅∣x\)⁡\(r​\(x,Y\)\)\\sigma^\{2\}\_\{0\}\(x\):=\\operatorname\{Var\}\_\{Y\\sim p\_\{0\}\(\\cdot\\mid x\)\}\(r\(x,Y\)\)\. The proof combines three ingredients: a KL bound betweenpδ^p\_\{\\widehat\{\\delta\}\}andpτ⋆p^\{\\star\}\_\{\\tau\}, Pinsker’s inequality, and a lower bound on the improvement ofpτ⋆p^\{\\star\}\_\{\\tau\}overp0p\_\{0\}\.

*Step 1: KL bound \(per\-prompt\)\.*By Equation \([4](https://arxiv.org/html/2607.22837#S3.E4)\),πτ⋆\(⋅∣x,y1:t−1\)=softmax\(ℓ\(x,y1:t−1\)\+logZ\(x,y1:t−1,⋅\)\)\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)=\\operatorname\{softmax\}\(\\ell\(x,y\_\{1:t\-1\}\)\+\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\)andπδ^\(⋅∣x,y1:t−1\)=softmax\(ℓ\(x,y1:t−1\)\+δ^\)\\pi\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)=\\operatorname\{softmax\}\(\\ell\(x,y\_\{1:t\-1\}\)\+\\widehat\{\\delta\}\)\. Definition[3](https://arxiv.org/html/2607.22837#Thmtheorem3)givesinfc‖log⁡Z​\(x,y1:t−1,⋅\)−\(δ^\+c​1\)‖2≤ε\\inf\_\{c\}\\\|\\log Z\(x,y\_\{1:t\-1\},\\cdot\)\-\(\\widehat\{\\delta\}\+c\\,\\mathbf\{1\}\)\\\|\_\{2\}\\leq\\varepsilonfor every reachable prefix\. Since addingc​1c\\,\\mathbf\{1\}to a softmax argument does not change the distribution, Lemma[8](https://arxiv.org/html/2607.22837#Thmtheorem8)yields

KL\(πδ^\(⋅∣x,y1:t−1\)∥πτ⋆\(⋅∣x,y1:t−1\)\)≤ε24\\operatorname\{KL\}\\\!\\bigl\(\\pi\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\,\\\|\\,\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\bigr\)\\leq\\frac\{\\varepsilon^\{2\}\}\{4\}at every reachable prefix\. The chain rule for autoregressive KL\[[6](https://arxiv.org/html/2607.22837#bib.bib6)\]gives

KL\(pδ^\(⋅∣x\)∥pτ⋆\(⋅∣x\)\)=𝔼Y∼pδ^\(⋅∣x\)\[∑t=1TKL\(πδ^\(⋅∣x,y1:t−1\)∥πτ⋆\(⋅∣x,y1:t−1\)\)\]≤T​ε24\.\\operatorname\{KL\}\\\!\\bigl\(p\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x\)\\,\\\|\\,p^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x\)\\bigr\)=\\mathbb\{E\}\_\{Y\\sim p\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x\)\}\\\!\\left\[\\sum\_\{t=1\}^\{T\}\\operatorname\{KL\}\\\!\\bigl\(\\pi\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\,\\\|\\,\\pi^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x,y\_\{1:t\-1\}\)\\bigr\)\\right\]\\leq\\frac\{T\\varepsilon^\{2\}\}\{4\}\.
*Step 2: Per\-prompt reward gap via Pinsker\.*By Pinsker’s inequality\[[43](https://arxiv.org/html/2607.22837#bib.bib43),[56](https://arxiv.org/html/2607.22837#bib.bib56)\],TV\(pδ^\(⋅∣x\),pτ⋆\(⋅∣x\)\)≤εT/8\\operatorname\{TV\}\\\!\\bigl\(p\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x\),p^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x\)\\bigr\)\\leq\\varepsilon\\sqrt\{T/8\}\. Sincer∈\[0,R\]r\\in\[0,R\]:

\|Jx\(pτ⋆\)−Jx\(pδ^\)\|≤R⋅TV\(pδ^\(⋅∣x\),pτ⋆\(⋅∣x\)\)≤RεT8\.\\bigl\|J\_\{x\}\(p^\{\\star\}\_\{\\tau\}\)\-J\_\{x\}\(p\_\{\\widehat\{\\delta\}\}\)\\bigr\|\\leq R\\cdot\\operatorname\{TV\}\\\!\\bigl\(p\_\{\\widehat\{\\delta\}\}\(\\cdot\\mid x\),\\,p^\{\\star\}\_\{\\tau\}\(\\cdot\\mid x\)\\bigr\)\\leq R\\,\\varepsilon\\sqrt\{\\frac\{T\}\{8\}\}\.
*Step 3: Per\-prompt improvement ofpτ⋆p^\{\\star\}\_\{\\tau\}overp0p\_\{0\}\.*Define the exponential tilting pathqs​\(y1:T∣x\):=p0​\(y1:T∣x\)​es​r​\(x,y1:T\)/Zsq\_\{s\}\(y\_\{1:T\}\\mid x\):=p\_\{0\}\(y\_\{1:T\}\\mid x\)\\,e^\{s\\,r\(x,y\_\{1:T\}\)\}/Z\_\{s\}fors≥0s\\geq 0, and the log\-partition functionΛx​\(s\):=log⁡𝔼Y∼p0\(⋅∣x\)​\[es​r​\(x,Y\)\]\\Lambda\_\{x\}\(s\):=\\log\\mathbb\{E\}\_\{Y\\sim p\_\{0\}\(\\cdot\\mid x\)\}\[e^\{s\\,r\(x,Y\)\}\]\. ThenΛx′​\(s\)=𝔼qs\(⋅∣x\)​\[r\]=Jx​\(qs\)\\Lambda\_\{x\}^\{\\prime\}\(s\)=\\mathbb\{E\}\_\{q\_\{s\}\(\\cdot\\mid x\)\}\[r\]=J\_\{x\}\(q\_\{s\}\)andΛx′′​\(s\)=Varqs\(⋅∣x\)⁡\(r\)\\Lambda\_\{x\}^\{\\prime\\prime\}\(s\)=\\operatorname\{Var\}\_\{q\_\{s\}\(\\cdot\\mid x\)\}\(r\), so

Jx​\(qs\)−Jx​\(p0\)=∫0sVarqt\(⋅∣x\)⁡\(r\)​𝑑t\.J\_\{x\}\(q\_\{s\}\)\-J\_\{x\}\(p\_\{0\}\)=\\int\_\{0\}^\{s\}\\operatorname\{Var\}\_\{q\_\{t\}\(\\cdot\\mid x\)\}\(r\)\\,dt\.LetVx​\(t\):=Varqt\(⋅∣x\)⁡\(r\)V\_\{x\}\(t\):=\\operatorname\{Var\}\_\{q\_\{t\}\(\\cdot\\mid x\)\}\(r\)\. Sincer∈\[0,R\]r\\in\[0,R\], Grönwall’s inequality givesVx​\(t\)≥σ02​\(x\)​e−R​tV\_\{x\}\(t\)\\geq\\sigma^\{2\}\_\{0\}\(x\)\\,e^\{\-Rt\}\(cf\. Lemma[9](https://arxiv.org/html/2607.22837#Thmtheorem9)\)\. Therefore, settings=1/τs=1/\\tau:

Jx​\(pτ⋆\)−Jx​\(p0\)≥σ02​\(x\)R​\(1−e−R/τ\)\.J\_\{x\}\(p^\{\\star\}\_\{\\tau\}\)\-J\_\{x\}\(p\_\{0\}\)\\geq\\frac\{\\sigma^\{2\}\_\{0\}\(x\)\}\{R\}\\bigl\(1\-e^\{\-R/\\tau\}\\bigr\)\.
*Per\-prompt bound\.*Combining Steps 2 and 3:

Jx​\(pδ^\)−Jx​\(p0\)≥σ02​\(x\)R​\(1−e−R/τ\)−R​ε​T/8\.J\_\{x\}\(p\_\{\\widehat\{\\delta\}\}\)\-J\_\{x\}\(p\_\{0\}\)\\geq\\frac\{\\sigma^\{2\}\_\{0\}\(x\)\}\{R\}\\bigl\(1\-e^\{\-R/\\tau\}\\bigr\)\-R\\,\\varepsilon\\sqrt\{T/8\}\.
*Averaging over prompts\.*Taking𝔼x∼ρ\\mathbb\{E\}\_\{x\\sim\\rho\}of both sides and using linearity:

J​\(pδ^\)−J​\(p0\)≥V0R​\(1−e−R/τ\)−R​ε​T/8,V0:=𝔼x∼ρ​\[σ02​\(x\)\]\.∎J\(p\_\{\\widehat\{\\delta\}\}\)\-J\(p\_\{0\}\)\\geq\\frac\{V\_\{0\}\}\{R\}\\bigl\(1\-e^\{\-R/\\tau\}\\bigr\)\-R\\,\\varepsilon\\sqrt\{T/8\},\\qquad V\_\{0\}:=\\mathbb\{E\}\_\{x\\sim\\rho\}\\\!\\bigl\[\\sigma^\{2\}\_\{0\}\(x\)\\bigr\]\.\\qed