面向大语言模型的摊销式离策略评估

# Amortized Off-Policy Evaluation for LLMs
Source: [https://arxiv.org/html/2610.10848](https://arxiv.org/html/2610.10848)
Younwoo ChoiAffiliation:RBC BorealisAffiliation:University of TorontoAffiliation:Vector InstituteEmail:[ywchoi@cs\.toronto\.edu](mailto:)Vincent LiuAffiliation:RBC BorealisEmail:[vincent\.liu@rbc\.com](mailto:)Haanvid LeeAffiliation:RBC BorealisEmail:[haanvid\.lee@rbc\.com](mailto:)\*Work done during an internship at RBC Borealis\. Corresponding author

###### Abstract

Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model\. Teams therefore evaluate candidates offline, on data produced by already\-deployed models\. This is off\-policy evaluation \(OPE\), and it faces two distribution shifts: as a model is updated in post\-training, its responses diverge from the logged ones \(policy shift\), and the reward definition under which it is judged changes with business requirements \(reward shift\)\. Classical OPE methods are ill\-suited to this continual\-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition\. To address this, we propose PFN\-OPE, a prior\-data fitted network that amortizes OPE across a distribution of contextual\-bandit tasks\. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur\. At test\-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per\-task fitting\. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN\-OPE achieves 2\.0 to 9\.3 times lower error than the best baselines across all tested configurations in the reward\-shifted settings\.

## 1Introduction

Choosing which large language model \(LLM\) to deploy hinges on accurate evaluation\([Zheng et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib33)\), yet testing a candidate on live traffic exposes users to an unvetted model\. As such, candidates are often evaluated offline with logged data from previously deployed LLMs\. A company has human\-rated chat logs and wants to estimate the reward a candidate model would receive on the same conversations\. The logs measure the deployed assistant, not the candidate\. The candidate’s responses differ from the logged ones, which we call*policy shift*, and the rubric the human annotators apply changes with the company’s requirements, which we call*reward shift*\. Answering the company’s question from these logs is off\-policy evaluation\.

Off\-policy evaluation \(OPE\) for LLM evaluation can be viewed as OPE on a contextual bandit, a one\-step interaction in which a context arrives, a policy samples an action, and a reward is observed\([Kallus et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib14);[Wang et al\., 2017](https://arxiv.org/html/2610.10848#bib.bib27)\)\. For an LLM, the context is the prompt, including any conversation history, the action is the response, and the reward is assigned to the response by an annotator or a reward model\([Ouyang et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib19)\)\. The deployed model that produced the logs is the behavior policy, the candidate is the target policy, and its value is the expected reward it earns on the prompts\. OPE estimates that value from the logs, which hold a reward for every behavior response and none for the response the target policy would give to the same prompt\.

Three classical estimators exist — inverse propensity scoring \(IPS\), doubly robust \(DR\), and the direct method \(DM\)\([Jiang & Li, 2016](https://arxiv.org/html/2610.10848#bib.bib13);[Strehl et al\., 2010](https://arxiv.org/html/2610.10848#bib.bib25);[Dudík et al\., 2011](https://arxiv.org/html/2610.10848#bib.bib5)\)— and the first two are out of reach for many popular LLMs\. IPS and DR estimators need the probability of each logged response under both policies, which a black\-box model does not expose, and the target policy’s responses almost never recur in the logs, so the weights would have little to no support\([Saito & Joachims, 2022](https://arxiv.org/html/2610.10848#bib.bib23)\)\. LLM evaluation therefore uses the DM: train a reward model on the logs and apply it to the target policy’s responses\([Ouyang et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib19);[Gao et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib7)\)\. This has two costs\. First, every new logged dataset or reward definition means training a new reward model\. Each checkpoint of a post\-training run is judged under a rubric that shifts with requirements, so each is a fit from scratch\. Second, no reward model can be checked on the target policy’s responses, whose rewards are unobserved — exactly where it must extrapolate\. Both costs come from fitting one model to one logged dataset at a time\.

Prior\-data fitted networks \(PFNs\) remove the per\-dataset fit\([Müller et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib17);[Hollmann et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib11);[Hollmann et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib12)\)\. A PFN is a transformer pretrained on datasets sampled from a prior over tasks; given a new dataset and a query in context, it returns a prediction in one forward pass, so a new dataset is a new input rather than new training data\. For OPE this offers two things the direct method lacks\. The first is amortization\([Cranmer et al\., 2020](https://arxiv.org/html/2610.10848#bib.bib3)\): each new logged dataset is a new input to the same network\. The second is generalization across the shifts: a reward model fitted to the logs must extrapolate to responses it has never seen, whereas a network pretrained on many tasks in which the target policy differs from the behavior policy has seen such shifts and learned how far the target value departs from the logged rewards\. Neither comes for free, because a PFN is only as good as its prior, and no prior for OPE exists\. The question is whether one can build a prior whose tasks capture the reward and policy shifts LLM evaluation faces\.

We propose*prior\-data fitted network for off\-policy evaluationPFN\-OPE*that maps a logged dataset and one sampled response per prompt from the target policy to an estimate of the target policy’s value, in one forward pass and with no per\-task fitting\. Our key insight is that the label a PFN needs, the target policy’s value, is unobservable on real logs but computable on tasks built from a pool of stored responses with rewards: when every policy is a distribution over the pool, every value is a weighted sum of stored rewards\. We build the pool once, from several LLMs and several reward functions, and sample tasks from it in which both shifts occur: affine combinations of the reward functions play the role of new reward definitions, and reward\-tilted reweightings of an LLM’s responses play the role of post\-trained policies\. Pretrained on these tasks, the network learns to read the shift off the observation itself, how the target responses differ from the logged ones and what the logged rewards say under the current definition, and by the PFN argument it approximates the minimum\-MSE estimator for tasks drawn from the prior\.

Our contributions are:

- •Amortized OPE via prior\-data fitted in\-context learning\.One network maps a logged dataset and sampled target responses to a value estimate in one forward pass, with no per\-task fitting\. This is a new problem setting for PFNs: the input is a set of paired responses, one with a reward and one without, and the label is a functional of the whole set rather than the label of one query \([Section3\.1](https://arxiv.org/html/2610.10848#S3.SS1)\)\.
- •A task prior for LLM off\-policy evaluation\.Tasks are built from a pool of LLM responses with rewards from several reward functions: affine combinations of the reward functions as reward definitions, reward\-tilted and routed policies as policy shifts, and target values computed from the pool \([Section3\.2](https://arxiv.org/html/2610.10848#S3.SS2)\)\.
- •Evaluation with LLM policies\.On HelpSteer2 and UltraFeedback, PFN\-OPE has the lowest MSE under all five reward definitions, four of them absent from pretraining, by2\.02\.0to9\.39\.3times over the best of the baselines\. It has the lowest MSE in 11 of 12 behavior–target pairs when the target LLM and the reward model are both absent from pretraining \([Section4](https://arxiv.org/html/2610.10848#S4)\)\.

## 2Background

#### Off\-policy evaluation\.

We consider off\-policy evaluation \(OPE\) in a contextual\-bandit setting\. In a contextual\-bandit interaction, the environment provides a contextx∈𝒳x\\in\\mathcal\{X\}, a policy samples an actiona∈𝒜a\\in\\mathcal\{A\}, and the environment returns a rewardrrwith a reward functionR:𝒳×𝒜→ℝR:\\mathcal\{X\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}\. The behavior policyμ:𝒳→Δ⁡\(𝒜\)\\mu:\\mathcal\{X\}\\rightarrow\\Delta\(\\mathcal\{A\}\)samples the logged data𝒟μ≔\{\(xi,ai,ri\)\}i=1n\\mathcal\{D\}\_\{\\mu\}\\coloneqq\\\{\(x\_\{i\},a\_\{i\},r\_\{i\}\)\\\}\_\{i=1\}^\{n\}from an environment, while the target policyπ\\piis the policy whose value we want to estimate\. OPE estimates the target policy valueV⁡\(π\)V\(\\pi\)as the expected reward ofπ\\pi:V\(π\)≔𝔼x∼p\(⋅\),a∼π\(⋅∣x\)\[R\(x,a\)\]V\(\\pi\)\\coloneqq\\mathbb\{E\}\_\{x\\sim p\(\\cdot\),a\\sim\\pi\(\\cdot\\mid x\)\}\\left\[R\(x,a\)\\right\]\.

The main challenge in OPE is that the behavior and target policies may induce different action distributions, resulting in distribution shift\([Saito & Joachims, 2022](https://arxiv.org/html/2610.10848#bib.bib23)\)\. The logged data contain rewards only for actions sampled by the behavior policy, leaving the rewards of actions sampled from the target policy unobserved\. The distribution shift becomes more severe as the behavior and target policies diverge\.

In our application,xxis a prompt,aais a generated response, andrris the reward assigned to that response by a human annotator\. The behavior policyμ\\muis the LLM that produced the logged responses, andπ\\piis the LLM being evaluated\.

#### Off\-policy evaluation estimators\.

OPE estimators fall into three methods: inverse propensity scoring \(IPS\), direct method \(DM\), and doubly robust \(DR\) estimation\([Jiang & Li, 2016](https://arxiv.org/html/2610.10848#bib.bib13)\)\. IPS estimates the target policy value by correcting the behavior policy induced action distribution to that of the target policy by reweighting logged rewards with IPS weights\([Saito & Joachims, 2022](https://arxiv.org/html/2610.10848#bib.bib23)\)\. IPS is unbiased given behavior policy\. However, its variance grows as the action space increases\. The direct method approximates the reward function from logged data and therefore has low variance but high bias due to estimation error from distribution shift\. DR combines the advantages of both IPS and DM\([Farajtabar et al\., 2018](https://arxiv.org/html/2610.10848#bib.bib6)\), doubly robust estimator is unbiased when either IPS or DM is unbiased and has lower variance compared to IPS\.

Conventional OPE estimators do not amortize inference across evaluation datasets\. For IPS and DR, they must recompute importance weights for each behavior–target policy pair and require policy probability information that may be unavailable for black\-box LLMs\. DM also requires re\-training when the ground\-truth reward function changes\.

Our work focuses on improving the DM’s ability to predict rewards for target policy responses with amortized off\-policy evaluation capability\. This focus is motivated by the high variance of IPS in the large response spaces induced by LLM policies\. The resulting reward model can also serve as the DM component of a DR estimator when combined with an IPS method\.

#### Prior\-data fitted networks\.

A prior\-data fitted network \(PFN\) is a single transformerqθq\_\{\\theta\}pretrained across many datasets: given a dataset and a query in context, it outputs a predictive distribution for the query’s label in one forward pass, so a new dataset is a new input rather than new training data\([Müller et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib17)\)\. The datasets come from a synthetic prior over tasksp⁡\(τ\)p\(\\tau\)\. In the standard PFN a taskτ\\tauis a relationship between inputszzand labelsyy, so samplingτ∼p⁡\(τ\)\\tau\\sim p\(\\tau\)samples a function; fromτ\\tauwe draw a dataset𝒟≔\{\(zi,yi\)\}i=1n\\mathcal\{D\}\\coloneqq\\\{\(z\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}and a queryz∗z\_\{\*\}with hidden labely∗y\_\{\*\}\. Pretraining minimizes the negative log\-likelihood of the hidden label,

ℒPFN\(θ\)≔𝔼τ∼p⁡\(τ\)𝔼\(𝒟,z∗,y∗\)∼p\(⋅∣τ\)\[−logqθ\(y∗∣𝒟,z∗\)\],\\mathcal\{L\}\_\{\\mathrm\{PFN\}\}\(\\theta\)\\coloneqq\\mathbb\{E\}\_\{\\tau\\sim p\(\\tau\)\}\\,\\mathbb\{E\}\_\{\(\\mathcal\{D\},z\_\{\*\},y\_\{\*\}\)\\sim p\(\\cdot\\mid\\tau\)\}\\bigl\[\-\\log q\_\{\\theta\}\(y\_\{\*\}\\mid\\mathcal\{D\},z\_\{\*\}\)\\bigr\],\(1\)an expectation over tasks and, within each task, over the dataset and query drawn from it\.

The property PFN\-OPE inherits is that this loss is minimized by the posterior predictive,q⋆​\(y∗∣𝒟,z∗\)=p⁡\(y∗∣𝒟,z∗\)=∫p⁡\(y∗∣z∗,τ\)​p​\(τ∣𝒟\)​𝑑τ\{q^\{\\star\}\(y\_\{\*\}\\mid\\mathcal\{D\},z\_\{\*\}\)=p\(y\_\{\*\}\\mid\\mathcal\{D\},z\_\{\*\}\)\}=\{\\int p\(y\_\{\*\}\\mid z\_\{\*\},\\tau\)\\,p\(\\tau\\mid\\mathcal\{D\}\)\\,d\\tau\}, whose mean is the minimum\-MSE estimate ofy∗y\_\{\*\}for tasks drawn from the prior\([Müller et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib17)\)\. A trained PFN thus performs Bayesian inference underp⁡\(τ\)p\(\\tau\)without representing the posterior, and the prior is its entire inductive bias: which tasks the prior contains decides what the network infers from a small dataset and how far it transfers\. For a real\-valued label, a PFN’s output is a categorical distribution over fixed intervals of the label’s range\([Müller et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib17)\)\. PFN\-OPE keeps this pretraining recipe and the Bayes argument\.[Section3](https://arxiv.org/html/2610.10848#S3)changes what a task, a dataset, and a prediction target are\.

## 3PFN\-OPE: Prior\-Data Fitted Network for Off\-Policy Evaluation

We propose PFN\-OPE for efficient off\-policy evaluation of LLM policies under policy and reward shifts\. Unlike conventional OPE methods, which fit an estimator separately for each task, PFN\-OPE amortizes estimation across a prior of OPE tasks, allowing a single pretrained network to estimate target policy values without per\-task fitting\. We first formulate OPE as in\-context prediction in[Section3\.1](https://arxiv.org/html/2610.10848#S3.SS1), then define the task prior used to generate pretraining tasks in[Section3\.2](https://arxiv.org/html/2610.10848#S3.SS2), and finally describe the transformer architecture in[Section3\.3](https://arxiv.org/html/2610.10848#S3.SS3)\.

### 3\.1OPE as in\-context prediction

PFN\-OPE formulates OPE as an in\-context prediction problem in which the target policy value is estimated from a behavior policy sampled logged data and target policy sampled target responses\.

#### Input\.

We assume access to a logged dataset𝒟μ≔\{\(xi,aiμ,riμ\)\}i=1n\\mathcal\{D\}\_\{\\mu\}\\coloneqq\\\{\(x\_\{i\},a\_\{i\}^\{\\mu\},r\_\{i\}^\{\\mu\}\)\\\}\_\{i=1\}^\{n\}, wherexix\_\{i\}is a prompt,aiμa\_\{i\}^\{\\mu\}is the response sampled from the behavior policyμ\\mu, andriμ=R⁡\(xi,aiμ\)r\_\{i\}^\{\\mu\}=R\(x\_\{i\},a\_\{i\}^\{\\mu\}\)is its observed reward\. We also assume black\-box sampling access to the target policyπ\\pi\. For each promptxix\_\{i\}, we sample one target responseaiπ∼π\(⋅∣xi\)a\_\{i\}^\{\\pi\}\\sim\\pi\(\\cdot\\mid x\_\{i\}\)and form the target prompt–response dataset𝒟π≔\{\(xi,aiπ\)\}i=1n\\mathcal\{D\}\_\{\\pi\}\\coloneqq\\\{\(x\_\{i\},a\_\{i\}^\{\\pi\}\)\\\}\_\{i=1\}^\{n\}\. Together,𝒟μ\\mathcal\{D\}\_\{\\mu\}and𝒟π\\mathcal\{D\}\_\{\\pi\}constitute the OPE observationO≔\(𝒟μ,𝒟π\)O\\coloneqq\(\\mathcal\{D\}\_\{\\mu\},\\mathcal\{D\}\_\{\\pi\}\), which serves as the input to the PFN\-OPE transformer\. Equivalently, the input can be represented asO=\{\(xi,aiμ,riμ,aiπ\)\}i=1nO=\\\{\(x\_\{i\},a\_\{i\}^\{\\mu\},r\_\{i\}^\{\\mu\},a\_\{i\}^\{\\pi\}\)\\\}\_\{i=1\}^\{n\}\. The target\-response rewards are not included inOOand must be inferred by the model\.

#### Prediction target\.

PFN\-OPE is trained to predict the correction valueCCin[Equation2](https://arxiv.org/html/2610.10848#S3.E2)givenOOrather than predicting target\-policy value directly\. Subtracting the average logged reward fromV⁡\(π\)V\(\\pi\)centers the target on the observed behavior reward and focuses prediction on the target\-policy correction\.

C≔V⁡\(π\)−r¯,r¯≔1n​∑i=1nri\.C\\coloneqq V\(\\pi\)\-\\bar\{r\},\\qquad\\bar\{r\}\\coloneqq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}r\_\{i\}\.\(2\)

#### Pretraining\.

During pretraining, PFN\-OPE learns an inference rule across various OPE tasks\. Tasksτ\\tauare sampled from the task prior described in[Section3\.2](https://arxiv.org/html/2610.10848#S3.SS2)\. A sampled taskτ\\tauspecifies a reward function, and behavior and target policies\. Givenτ\\tau, we generate the observation datasetO∼p⁡\(O∣τ\)O\\sim p\(O\\mid\\tau\)for pretraining\. Because reward functions and policies are known, PFN\-OPE can be supervised to predict the correction target and consequently the target policy value\.

We train the network via quantile regression\. A quantile head returnsQ^θκ​\(O\)\\widehat\{Q\}\_\{\\theta\}^\{\\kappa\}\(O\), an estimate of theκ\\kappa\-quantile ofCCgivenOO, for every levelκ\\kappain𝒦\\mathcal\{K\}, and pretraining minimizes the pinball loss in[Equation3](https://arxiv.org/html/2610.10848#S3.E3)\([Koenker & Bassett, 1978](https://arxiv.org/html/2610.10848#bib.bib15)\)\. We follow TabICLv2\([Qu et al\., 2026](https://arxiv.org/html/2610.10848#bib.bib20)\), which found quantile regression more accurate than TabPFN’s discretized output for regression targets\. Another advantage is that a categorical head over fixed intervals[Section2](https://arxiv.org/html/2610.10848#S2)fixes the range ofCCbefore pretraining, whereas the quantile levels adapt to each task’s predictive distribution\.

For each levelκ\\kappa, the expected pinball loss is minimized by theκ\\kappa\-quantile of the posterior predictivep⁡\(C∣O\)p\(C\\mid O\)induced by the prior\([Koenker & Bassett, 1978](https://arxiv.org/html/2610.10848#bib.bib15)\), by the argument behind[Equation1](https://arxiv.org/html/2610.10848#S2.E1)with\(O,C\)\(O,C\)in place of\(𝒟,z∗,y∗\)\(\\mathcal\{D\},z\_\{\*\},y\_\{\*\}\)\. A network that attains the minimum is therefore the posterior\-quantile estimator for the prior, and the prior is its inductive bias, as in[Section2](https://arxiv.org/html/2610.10848#S2)\.

ℒ\(θ\)≔𝔼τ∼p⁡\(τ\)𝔼O∼p⁡\(O∣τ\)\[1\|𝒦\|∑κ∈𝒦ℓκ\(C−Q^θκ\(O\)\)\],ℓκ\(u\)≔u\(κ−𝟏\{u<0\}\)\.\\mathcal\{L\}\(\\theta\)\\coloneqq\\mathbb\{E\}\_\{\\tau\\sim p\(\\tau\)\}\\,\\mathbb\{E\}\_\{O\\sim p\(O\\mid\\tau\)\}\\Bigl\[\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{\\kappa\\in\\mathcal\{K\}\}\\ell\_\{\\kappa\}\\bigl\(C\-\\widehat\{Q\}\_\{\\theta\}^\{\\kappa\}\(O\)\\bigr\)\\Bigr\],\\qquad\\ell\_\{\\kappa\}\(u\)\\coloneqq u\\,\\bigl\(\\kappa\-\\mathbf\{1\}\\\{u<0\\\}\\bigr\)\.\(3\)

#### Inference\.

At test\-time, PFN\-OPE receives an observationOOfrom a test\-time OPE task\. The target\-policy rewards and value are unknown\. The pretrained network usesOOto predict the correction\. In a single forward pass, it produces the quantile estimates\{Q^θκ​\(O\)\}κ∈𝒦\\\{\\widehat\{Q\}\_\{\\theta\}^\{\\kappa\}\(O\)\\\}\_\{\\kappa\\in\\mathcal\{K\}\}\. We aggregate these quantiles asC^θ​\(O\)\\widehat\{C\}\_\{\\theta\}\(O\)and estimate the target policy valueV^θ​\(π,O\)\\widehat\{V\}\_\{\\theta\}\(\\pi;O\):

C^θ​\(O\)≔1\|𝒦\|​∑κ∈𝒦Q^θκ​\(O\),V^θ​\(π,O\)≔r¯\+C^θ​\(O\)\.\\widehat\{C\}\_\{\\theta\}\(O\)\\coloneqq\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{\\kappa\\in\\mathcal\{K\}\}\\widehat\{Q\}\_\{\\theta\}^\{\\kappa\}\(O\),\\qquad\\widehat\{V\}\_\{\\theta\}\(\\pi;O\)\\coloneqq\\bar\{r\}\+\\widehat\{C\}\_\{\\theta\}\(O\)\.\(4\)
Averaging the predicted quantiles approximates the mean of the predictive distribution of the correction\. Hence,V^θ​\(π,O\)\\widehat\{V\}\_\{\\theta\}\(\\pi;O\)approximates the posterior\-mean estimate of the target\-policy value under the task prior, which is the minimum\-MSE estimator for tasks drawn from that prior\.

PFN\-OPE handles both policy and reward shifts through the observation, without parameter updates\. A policy shift changes the target\-response component𝒟π\\mathcal\{D\}\_\{\\pi\}inOO, because the new target policy produces different responses for the logged prompts\. A reward shift changes the rewards assigned to the logged responses when they are rescored under a new reward function, while the target responses remain unscored\. Both shifts are reflected inOOand processed by the same pretrained inference rule\.

### 3\.2Constructing Task Prior from a Response Pool

The task prior determines the reward and policy shifts during pretraining, and therefore serves as the estimator’s inductive bias\. Broader coverage of reward and policy shifts during pretraining generally improves PFN\-OPE’s ability to generalize to related shifts at test\-time\. Each pretraining task must also provide a target\-policy valueVτ​\(π\)V\_\{\\tau\}\(\\pi\)for supervision\.

Generating and scoring new LLM responses for every pretraining task would be computationally expensive\. We instead construct a finite response pool before pretraining and reuse it across tasks\. Each task is obtained by selecting a prompt distributionp⁡\(x∣τ\)p\(x\\mid\\tau\), a reward functionRτR\_\{\\tau\}, a behavior policyμ\\mu, and a target policyπ\\pi\. The target\-policy valueVτ​\(π\)V\_\{\\tau\}\(\\pi\)can then be evaluated as a finite sum over the pool, allowing us to sample pretraining tasks without additional LLM generation or reward\-model evaluation\.

#### Response pool\.

We assume that a set of LLMs𝒢\\mathcal\{G\}is available for pretraining\. An LLMg∈𝒢g\\in\\mathcal\{G\}is a model’s parameters together with its system prompt and decoding settings, andg⁡\(a∣x\)g\(a\\mid x\)is its response distribution\. For every promptxxin a finite set𝒳0⊂𝒳\\mathcal\{X\}\_\{0\}\\subset\\mathcal\{X\}and everyg∈𝒢g\\in\\mathcal\{G\}, we sampleKKresponses fromggonce and store them as𝒜g​\(x\)≔\{ag\(1\)​\(x\),…,ag\(K\)​\(x\)\}\\mathcal\{A\}\_\{g\}\(x\)\\coloneqq\\\{a\_\{g\}^\{\(1\)\}\(x\),\\ldots,a\_\{g\}^\{\(K\)\}\(x\)\\\}\. Reward functionsR1,…,RJR\_\{1\},\\ldots,R\_\{J\}, the heads of one or more reward models, assign a reward to every stored response, so the pool is the set of tuples\(x,g,a,R1​\(x,a\),…,RJ​\(x,a\)\)\(x,g,a,R\_\{1\}\(x,a\),\\ldots,R\_\{J\}\(x,a\)\)\.

#### Reward functions\.

A task’s reward function is an affine combination of the stored rewards, each standardized over the pool,R~j≔\(Rj−mj\)/sj\\tilde\{R\}\_\{j\}\\coloneqq\(R\_\{j\}\-m\_\{j\}\)/s\_\{j\}withmjm\_\{j\}andsjs\_\{j\}the mean and standard deviation ofRjR\_\{j\}on the pool:

Rτ​\(x,a\)≔∑j=1Jλτ​j​R~j​\(x,a\),λτ∼𝒩⁡\(w,diag⁡\(w\)−w​w⊤\),R\_\{\\tau\}\(x,a\)\\coloneqq\\sum\_\{j=1\}^\{J\}\\lambda\_\{\\tau j\}\\,\\tilde\{R\}\_\{j\}\(x,a\),\\qquad\\lambda\_\{\\tau\}\\sim\\mathcal\{N\}\\bigl\(w,\\ \\operatorname\{diag\}\(w\)\-ww^\{\\top\}\\bigr\),\(5\)with evaluation weightsλτ\\lambda\_\{\\tau\}drawn once per task from a Gaussian centered on base weightsw∈ΔJ−1w\\in\\Delta^\{J\-1\}\. The covariance is singular, so every draw satisfies∑jλτ​j=1\\sum\_\{j\}\\lambda\_\{\\tau j\}=1\. We standardize the rewards so that the weights compare criteria on one scale, and we allow negative weights because an evaluation criterion can penalize an aspect, as helpfulness minus verbosity does\. The same weights apply to every prompt and response, so the logged rewards of a task describe one consistent evaluation criterion, and each new draw ofλτ\\lambda\_\{\\tau\}is a new reward definition obtained from the stored rewards alone\.

#### Behavior and target policies\.

A task’s policies are distributions over stored responses\. For LLMggand promptxx, the pool policy is the uniform distribution on𝒜g​\(x\)\\mathcal\{A\}\_\{g\}\(x\),

g^\(a∣x\)≔1K∑k=1K𝟏\{a=ag\(k\)\(x\)\},\\widehat\{g\}\(a\\mid x\)\\coloneqq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbf\{1\}\\bigl\\\{a=a\_\{g\}^\{\(k\)\}\(x\)\\bigr\\\},\(6\)theKK\-sample Monte Carlo approximation ofg⁡\(a∣x\)g\(a\\mid x\)\. A task picks a behavior LLMg∈𝒢g\\in\\mathcal\{G\}and a target LLMg′∈𝒢g^\{\\prime\}\\in\\mathcal\{G\}and setsμ≔g^\\mu\\coloneqq\\widehat\{g\}\. The target policy starts fromg^′\\widehat\{g\}^\{\\prime\}and is shifted in the way reward\-guided post\-training shifts a policy, toward the responses a reward model prefers\([Ouyang et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib19)\)\. The preference is a selection rewardSτ​\(x,a\)≔∑j=1Jωτ​j​R~j​\(x,a\)S\_\{\\tau\}\(x,a\)\\coloneqq\\sum\_\{j=1\}^\{J\}\\omega\_\{\\tau j\}\\tilde\{R\}\_\{j\}\(x,a\)with selection weightsωτ∈ΔJ−1\\omega\_\{\\tau\}\\in\\Delta^\{J\-1\}, and it acts through a selection strengthη≥0\\eta\\geq 0and a routing probabilityα∈\[0,1\]\\alpha\\in\[0,1\]:

νη​\(a∣x\)≔g^′​\(a∣x\)​eη​Sτ​\(x,a\)∑a′g^′​\(a′∣x\)​eη​Sτ​\(x,a′\),π≔\(1−α\)​μ\+α​νη\.\\nu\_\{\\eta\}\(a\\mid x\)\\coloneqq\\frac\{\\widehat\{g\}^\{\\prime\}\(a\\mid x\)\\,e^\{\\eta S\_\{\\tau\}\(x,a\)\}\}\{\\sum\_\{a^\{\\prime\}\}\\widehat\{g\}^\{\\prime\}\(a^\{\\prime\}\\mid x\)\\,e^\{\\eta S\_\{\\tau\}\(x,a^\{\\prime\}\)\}\},\\qquad\\pi\\coloneqq\(1\-\\alpha\)\\,\\mu\+\\alpha\\,\\nu\_\{\\eta\}\.\(7\)The tiltνη\\nu\_\{\\eta\}has the form of the optimal policy of a KL\-regularized reward objective\([Rafailov et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib21)\), which is how post\-training moves a policy: largerη\\etafavors responses with higherSτS\_\{\\tau\}, andη=0\\eta=0withα=1\\alpha=1returnsg^′\\widehat\{g\}^\{\\prime\}itself, a change of the LLM with no reward guidance\. Routing mixesνη\\nu\_\{\\eta\}withμ\\mubecause the value gapVτ​\(π\)−Vτ​\(μ\)V\_\{\\tau\}\(\\pi\)\-V\_\{\\tau\}\(\\mu\)is then linear inα\\alpha, so the prior covers shifts from none \(α=0\\alpha=0\) to the full gap \(α=1\\alpha=1\); a target response routed toμ\\muis a fresh draw fromg^\\widehat\{g\}, not a copy of the logged response\. The selection weightsωτ\\omega\_\{\\tau\}are drawn independently ofλτ\\lambda\_\{\\tau\}because the reward that steered a policy need not be the reward under which it is judged\.

#### Training labels\.

A task fixes its prompt distributionp⁡\(x∣τ\)p\(x\\mid\\tau\)on𝒳0\\mathcal\{X\}\_\{0\}before any observation is drawn\. The value of the target policyπ\\piat a particular taskτ\\tauis computed as

Vτ​\(π\)=∑x∈𝒳0p⁡\(x∣τ\)​∑aπ⁡\(a∣x\)​Rτ​\(x,a\)\.V\_\{\\tau\}\(\\pi\)=\\sum\_\{x\\in\\mathcal\{X\}\_\{0\}\}p\(x\\mid\\tau\)\\sum\_\{a\}\\pi\(a\\mid x\)\\,R\_\{\\tau\}\(x,a\)\.\(8\)
The labelC=Vτ​\(π\)−r¯C=V\_\{\\tau\}\(\\pi\)\-\\bar\{r\}of[Equation2](https://arxiv.org/html/2610.10848#S3.E2)is defined relative to the observed logged mean, so two observations of one task carry different labels, and the network learns to predict the correction for the dataset in hand rather than the task’s average gap\.[Algorithm1](https://arxiv.org/html/2610.10848#alg1)summarizes pretraining\.

### 3\.3PFN\-OPE Architecture

PFN\-OPE follows the PFN framework, but adapts its transformer architecture to the structure of the OPE observationOO\. In particular, we modify three components of the standard PFN architecture: the input representation, the attention mask, and the readout\.

#### Input representation\.

The input is the OPE observationOO, which is represented by two tokens:

Li=fμ​\(ϕx​\(xi\),ϕa​\(aiμ\),riμ\),Qi=fπ​\(ϕx​\(xi\),ϕa​\(aiπ\),MASK\),L\_\{i\}=f\_\{\\mu\}\\bigl\(\\phi\_\{x\}\(x\_\{i\}\),\\phi\_\{a\}\(a\_\{i\}^\{\\mu\}\),r\_\{i\}^\{\\mu\}\\bigr\),\\qquad Q\_\{i\}=f\_\{\\pi\}\\bigl\(\\phi\_\{x\}\(x\_\{i\}\),\\phi\_\{a\}\(a\_\{i\}^\{\\pi\}\),\\mathrm\{MASK\}\\bigr\),\(9\)whereLiL\_\{i\}represents the logged response and its observed reward, whereasQiQ\_\{i\}represents the target response whose reward is unobserved\. Prompts and responses are mapped to frozen text embeddingsϕx​\(x\)\\phi\_\{x\}\(x\)andϕa​\(a\)\\phi\_\{a\}\(a\)using Qwen3\-Embedding\-4B\([Zhang et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib32)\)\. The symbolMASK\\mathrm\{MASK\}denotes a learned vector that replaces the unobserved target reward\.

#### Attention mask\.

For the input structure, we use a directional attention mask\. Each logged tokenLiL\_\{i\}attends only to the logged tokens, whereas each target tokenQiQ\_\{i\}attends to all logged and target tokens\. Target tokens attend to the logged tokens because the reward of a target response must be inferred from the logged responses and their observed rewards\. They also attend to each other because PFN\-OPE predicts the correctionCC, which depends jointly on the entire set of sampled target responses\.

Readout\.The transformer includes a learned summary token whose final representation is passed to an output head that predicts the quantiles\{Q^θκ​\(O\)\}κ∈𝒦\\\{\\widehat\{Q\}\_\{\\theta\}^\{\\kappa\}\(O\)\\\}\_\{\\kappa\\in\\mathcal\{K\}\}of the correctionCC\. We obtainC^θ​\(O\)\\widehat\{C\}\_\{\\theta\}\(O\)by averaging these predicted quantiles, as defined in Equation equation[4](https://arxiv.org/html/2610.10848#S3.E4)\.

Figure 1:One checkpoint transfers to four rewards absent from pretraining\.MSE atn=512n=512over the six pairs of seen LLMs, with 95% intervals\. The ArmoRM aggregate is the seen reward; the other four are unseen\. PFN\-OPE’s MSE is2\.02\.0to9\.39\.3times lower than the best baseline’s\.

## 4Experiments

The experiments ask whether PFN\-OPE replaces per\-task fitting under the*reward shift*and*policy shift*\. The metric is the MSE betweenV^​\(π\)\\widehat\{V\}\(\\pi\)andVτ​\(π\)V\_\{\\tau\}\(\\pi\)\([Saito et al\., 2021](https://arxiv.org/html/2610.10848#bib.bib24)\)\. Five questions organize the section:

1. Q1Does PFN\-OPE generalize to rewards absent from pretraining? \([Section4\.2](https://arxiv.org/html/2610.10848#S4.SS2)\)
2. Q2Does it generalize to unseen target LLMs under an unseen reward model? \([Section4\.3](https://arxiv.org/html/2610.10848#S4.SS3)\)
3. Q3Mechanism: why does PFN\-OPE win, and how much logged data does it need? \([Section4\.4](https://arxiv.org/html/2610.10848#S4.SS4)\)
4. Q4Prior ablation: which axis of the prior carries the gain? \([Section4\.5](https://arxiv.org/html/2610.10848#S4.SS5)\)
5. Q5Prior coverage: what do the shifted policies add? \([Section4\.6](https://arxiv.org/html/2610.10848#S4.SS6)\)

### 4\.1Common experimental setup

#### Pool, prompts, and LLMs\.

Prompts come from HelpSteer2\([Wang et al\., 2024b](https://arxiv.org/html/2610.10848#bib.bib28)\)and UltraFeedback\([Cui et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib4)\), reported separately\. Each dataset holds out1,0241\{,\}024evaluation prompts that never enter pretraining\. Each LLM contributesK=8K=8responses per prompt\. Three seen LLMs form the pool𝒢\\mathcal\{G\}of[Section3\.2](https://arxiv.org/html/2610.10848#S3.SS2): Qwen2\.5\-1\.5B\-Instruct, Qwen2\.5\-7B\-Instruct\([Yang et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib30)\), and Llama\-3\.2\-1B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib9)\)\. Two unseen LLMs serve only as evaluation targets: Llama\-3\.2\-3B\-Instruct, a larger checkpoint of a seen family, and Gemma\-3\-1B\-IT\([Gemma Team et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib8)\), an unseen family\. Every evaluation policy is the pool policy of one LLM \([Equation6](https://arxiv.org/html/2610.10848#S3.E6)\), so policy shift in evaluation is a change of LLM\.

#### Rewards\.

ArmoRM\([Wang et al\., 2024a](https://arxiv.org/html/2610.10848#bib.bib26)\)outputs 18 sub\-rewards and the aggregated value\. The prior’s reward functionsR1,…,RJR\_\{1\},\\ldots,R\_\{J\}in[Equation5](https://arxiv.org/html/2610.10848#S3.E5)are this aggregate and 15 of ArmoRM’s objectives\. Four rewards are unseen\. We hold out three from ArmoRM: helpfulness, honesty, and helpfulness minus0\.50\.5times verbosity\. The fourth is GRM\-Gemma2\-2B\([Yang et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib31)\), written as GRM, a different reward model\. All rewards are standardized with pretraining statistics\.

#### Evaluation tasks and intervals\.

For each setting of dataset, a behavior LLM, a target LLM, and a reward, there are 100 tasks\. A task drawsnnevaluation prompts\. At each prompt, it takes one stored behavior response with its reward and one stored target response without\. This is the observationOO, and every estimator receives the same one\. The true target value is the mean reward of the target LLM’s1,024×81\{,\}024\\times 8stored responses\. We report the MSE over the 100 tasks with a 95% interval\.

#### Baseline estimators\.

PFN\-OPE is one checkpoint, never updated for any reward, policy, ornn\. We compare it with six estimators on the same frozen embeddings\. Three direct methods refit a reward regressor on every logged dataset, the per\-task fitting we replace: DM\-ridge, DM\-kernel, and DM\-MLP\([Kallus et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib14);[Wang et al\., 2017](https://arxiv.org/html/2610.10848#bib.bib27)\)\. Two direct methods, DM\-TabPFN\([Hollmann et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib11)\)and DM\-TabICLv2\([Qu et al\., 2026](https://arxiv.org/html/2610.10848#bib.bib20)\), predict the target rewards in context with frozen tabular foundation models\. They are the closest baselines: they amortize the regressor, whereas PFN\-OPE amortizes the OPE\. MIPS\([Saito & Joachims, 2022](https://arxiv.org/html/2610.10848#bib.bib23)\)is marginalized importance sampling with a density ratio estimated from action embeddings\.

Table 1:PFN\-OPE achieves the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are absent from pretraining\.MSE \(×10−2\\times 10^\{\-2\}\) atn=512n=512under GRM; the small grey number is the larger half\-width of the 95% interval\. Bold marks the lowest MSE within each dataset and column, determined before rounding\.Target: Llama\-3\.2\-3BTarget: Gemma\-3\-1BBehavior LLMQwen2\.57BQwen2\.51\.5BLlama\-3\.21BQwen2\.57BQwen2\.51\.5BLlama\-3\.21BHelpSteer2PFN\-OPE \(ours\)0\.041±\\pm0\.0060\.87±\\pm0\.160\.97±\\pm0\.231\.54±\\pm0\.302\.01±\\pm0\.212\.75±\\pm0\.42DM\-ridge0\.130±\\pm0\.04717\.78±\\pm1\.108\.45±\\pm0\.665\.94±\\pm0\.5630\.35±\\pm1\.9019\.37±\\pm1\.80DM\-kernel0\.050±\\pm0\.05516\.57±\\pm0\.917\.72±\\pm0\.874\.91±\\pm0\.7330\.32±\\pm1\.7018\.98±\\pm1\.90DM\-MLP1\.26±\\pm1\.6026\.40±\\pm6\.607\.99±\\pm2\.007\.44±\\pm3\.2039\.57±\\pm8\.1020\.52±\\pm5\.40DM\-TabPFN0\.055±\\pm0\.06618\.17±\\pm0\.936\.99±\\pm0\.654\.72±\\pm0\.5232\.46±\\pm2\.2018\.84±\\pm1\.70DM\-TabICLv21\.05±\\pm0\.2525\.28±\\pm1\.3013\.35±\\pm0\.8110\.12±\\pm0\.7644\.05±\\pm2\.1030\.10±\\pm2\.10MIPS0\.27±\\pm0\.1232\.56±\\pm35\.0010\.31±\\pm0\.768\.26±\\pm4\.0027\.45±\\pm4\.2018\.86±\\pm4\.10UltraFeedbackPFN\-OPE \(ours\)1\.120±\\pm0\.0415\.01±\\pm1\.009\.86±\\pm0\.420\.038±\\pm0\.0046\.22±\\pm0\.586\.52±\\pm0\.39DM\-ridge1\.55±\\pm0\.159\.50±\\pm0\.9212\.11±\\pm0\.560\.043±\\pm0\.04313\.76±\\pm1\.4019\.55±\\pm1\.10DM\-kernel1\.56±\\pm0\.158\.71±\\pm0\.8912\.24±\\pm0\.510\.046±\\pm0\.04614\.42±\\pm1\.3020\.18±\\pm1\.10DM\-MLP1\.59±\\pm0\.6712\.95±\\pm2\.2013\.86±\\pm2\.701\.66±\\pm1\.2018\.88±\\pm2\.9024\.08±\\pm4\.40DM\-TabPFN2\.49±\\pm0\.248\.65±\\pm0\.799\.99±\\pm0\.680\.062±\\pm0\.04014\.60±\\pm1\.6016\.80±\\pm1\.10DM\-TabICLv20\.230±\\pm0\.06613\.05±\\pm1\.1017\.80±\\pm0\.691\.11±\\pm0\.2320\.07±\\pm1\.8030\.16±\\pm1\.10MIPS1\.31±\\pm0\.4110\.30±\\pm1\.5014\.47±\\pm0\.484\.25±\\pm3\.8015\.80±\\pm4\.1024\.62±\\pm4\.60

### 4\.2PFN\-OPE generalizes to four unseen rewards

#### Setup\.

Reward shift is the case in which the direct method must refit\. We test whether PFN\-OPE, which never saw the new reward, can replace that refit\. The policies are the six ordered pairs of seen LLMs, and only the reward changes: the seen aggregate and the four unseen rewards, atn=512n=512\. The held\-out ArmoRM rewards are new definitions from a reward model the prior knows\. GRM is a reward model the prior never saw\.

#### Results\.

PFN\-OPE has the lowest MSE under all five rewards on both datasets, by2\.02\.0to9\.39\.3times over the best baseline \([Figure1](https://arxiv.org/html/2610.10848#S3.F1)\)\. The gain shrinks with distance from the prior but does not vanish\. It is3\.73\.7to9\.39\.3times for the rewards built from ArmoRM’s objectives, which the prior’s reward combinations span, and2\.02\.0to4\.14\.1times under GRM, which lies outside them\. The baselines see the new reward through the same logged rewards and refit to it\. The gain is 3\.7–9\.3× for rewards spanned by the prior and 2\.0–4\.1× for out\-of\-prior GRM\.

Figure 2:The advantage grows with the correction and survives 32 times more logged data than the baselines\.Legend as in Figure[1](https://arxiv.org/html/2610.10848#S3.F1)\. \(a\) MSE against the standardized correction\|Cstd\|\|C\_\{\\mathrm\{std\}\}\|atn=512n=512: above0\.20\.2, every baseline’s MSE rises with the correction, PFN\-OPE’s stays lower\. \(b, c\) MSE against logged rowsnn, for seen LLMs under ArmoRM \(b\) and unseen LLMs under GRM \(c\)\.

### 4\.3PFN\-OPE generalizes to unseen target LLMs under an unseen reward model

#### Setup\.

This is the deployment case of[Section1](https://arxiv.org/html/2610.10848#S1): a candidate LLM the prior never saw, scored by a reward model the prior never saw\. The target policies are Llama\-3\.2\-3B and Gemma\-3\-1B, and the reward is GRM, on both datasets: 12 behavior–target pairs atn=512n=512\.

#### Results\.

PFN\-OPE has the lowest MSE in 11 of 12 pairs \([Table1](https://arxiv.org/html/2610.10848#S4.T1)\)\. The gain is3\.13\.1to1919times in five pairs on HelpSteer2 and1\.71\.7to2\.62\.6times in three pairs on UltraFeedback\. In the other four pairs, the estimators are close\. Two of these have an MSE below0\.0010\.001for PFN\-OPE and the closest DM baselines: the logged mean is already near the target value, so there is little to correct\. In the remaining two, both on UltraFeedback, PFN\-OPE ties DM\-TabPFN from Llama\-3\.2\-1B to Llama\-3\.2\-3B and loses to DM\-TabICLv2 by4\.94\.9times from Qwen2\.5\-7B to Llama\-3\.2\-3B\.

Figure 3:Ablation of the prior\.MSE of four PFNs pretrained in a2×22\\times 2design over the reward and the policy prior\.

### 4\.4PFN\-OPE is robust to large correction and less logged data

#### Setup\.

[Figure2](https://arxiv.org/html/2610.10848#S4.F2)a tests whether PFN\-OPE performs in large correctionCCsettings\. We pool alln=512n=512tasks of Q1 and Q2 and bin them by the correction in units of the logged reward’s standard deviation,\|Cstd\|\|C\_\{\\mathrm\{std\}\}\|\. For[Figure2](https://arxiv.org/html/2610.10848#S4.F2)b,c, we take the high\-correction third of the logged datasets and grownnfrom 64 to 2,048, for seen targets under the ArmoRM aggregate \(b\) and unseen targets under GRM \(c\)\. PFN\-OPE is evaluated up ton=512n=512\.

#### Results\.

Below\|Cstd\|=0\.2\|C\_\{\\mathrm\{std\}\}\|=0\.2every estimator is accurate, with MSE at or below0\.0120\.012for every direct method, because the logged mean is already close to the target value\. Above0\.20\.2the baselines’ MSE grows with the correction and reaches0\.210\.21to0\.320\.32in the largest bin, while PFN\-OPE’s stays between0\.040\.04and0\.070\.07\([Figure2](https://arxiv.org/html/2610.10848#S4.F2)a\)\. The ratio to the best baseline widens from1\.61\.6to3\.13\.1\. A regressor fitted to the logged rewards predicts target rewards near them, so it inherits the logged mean’s errorC2C^\{2\}\. An estimator trained to predictCCdoes not\. More data does not close the gap\. With 64 rows, PFN\-OPE is more accurate than every baseline with 2,048, by3\.93\.9times for seen targets and1\.61\.6times for unseen targets \([Figure2](https://arxiv.org/html/2610.10848#S4.F2)b,c\)\. The baselines’ curves flatten above PFN\-OPE’s, so their remaining error is due to extrapolation\. In deployment, those rows are annotations: PFN\-OPE with 64 rows is more accurate than the baselines with 32 times as many\.

### 4\.5Prior ablation: the policy axis of the prior carries the gain

#### Setup\.

The prior is the estimator’s entire inductive bias \([Section2](https://arxiv.org/html/2610.10848#S2)\), and[Section3\.2](https://arxiv.org/html/2610.10848#S3.SS2)widened it along two axes: reward definitions and policy shifts\. To see which axis carries the gain, we pretrained four PFNs in a2×22\\times 2design: one reward function per task or the affine combination of[Equation5](https://arxiv.org/html/2610.10848#S3.E5), crossed with the direct policies \(η=0\\eta=0,α=1\\alpha=1\) or the full policy prior with tilted and routed policies\.

#### Results\.

The full policy prior lowers MSE by 28% to 36% at everynn, whether the reward prior is a single function or a combination \([Figure3](https://arxiv.org/html/2610.10848#S4.F3)\)\. Reward combinations help little on their own, improving by 2% to 3% with pool policies and by 10% to 12% on top of the full policy prior\.

### 4\.6Prior coverage is important in pretraining

#### Setup\.

We place every task in a plane whose two axes measure how hard the task is for a linear reward model on the frozen embeddings \(definitions in[AppendixD](https://arxiv.org/html/2610.10848#A4)\)\. The horizontal axis,LΔL\_\{\\Delta\}, is how far the target responses move away from the logged ones in embedding space, in directions the logged responses do not cover\. The vertical axis, the probe correction errorBΔ2B\_\{\\Delta\}^\{2\}, is how wrong a linear reward model fitted to the logs is about the value gap between the two policies\. We plot 1,000 pretraining tasks from each of the two arms of the ablation and the 1,000 evaluation tasks with an unseen target LLM under GRM, atn=128n=128\. Each evaluation task is colored by the error of the PFN pretrained on that panel’s prior\.

#### Results\.

The two priors cover the same region except the left corner and upper region, and that corner is where the two PFNs differ \([Figure4](https://arxiv.org/html/2610.10848#S4.F4.fig1)\)\. The pool\-only prior has no pretraining task there, and its PFN has its highest error on the evaluation tasks there\. The full prior has tasks there, and its PFN’s error there is low\. Elsewhere, both PFNs are accurate where their priors are dense\. Every evaluation task in the corner is a Llama\-3\.2\-1B→\\toLlama\-3\.2\-3B task: a target from the family of the logged LLM, whose responses differ in reward more than in form\. This shift is closest to a post\-trained checkpoint of the logged model, and the shifted policies are what put it in the prior\.

![Refer to caption](https://arxiv.org/html/2610.10848v1/fig_prior_coverage.png)

Figure 4:The PFN fails where its prior has no tasks, and the shifted policies fill the gap\.Each panel shows the pretraining tasks\. Triangles are the evaluation tasks with an unseen target LLM under GRM\. Left: one reward per task, direct policies\. Right: affine reward combinations, full policy prior\.

## 5Conclusion

We introduced PFN\-OPE, a prior\-data fitted network that amortizes off\-policy evaluation of LLMs\. We pretrained it once on tasks built from a response pool, where every target value is known by construction\. At test\-time, it maps a logged dataset and sampled target responses to a value estimate in one forward pass, with no per\-task fitting\. On HelpSteer2 and UltraFeedback, PFN\-OPE has2\.02\.0to9\.39\.3times lower MSE than the best baseline\. It has the lowest MSE in 11 of 12 behavior–target pairs when the target LLM and the reward model are both unseen\. As post\-training objectives and feedback signals continue to shift, we see PFN\-OPE as a step toward evaluation that keeps pace by reusing one pretraining run across the stream of target policies that deployment produces\.

### AI use statement

In this work, we used generative AI tools to assist with the design of our research methodology and experiments \(providing feedback on experiments\), to implement methods, to reformat datasets, and to support qualitative data analysis\. Beyond the LLMs and reward models studied in this work, whose responses and scores form the response pool, we did not use generative AI tools to generate dataset develop theoretical models or conceptual frameworks, propose or refine hypotheses, or interpret results, and formulating or proving mathematical claims, writing proofs, and translation are not applicable to this work\. Additionally, we used generative AI tools to suggest experimental parameters, create and edit software code, draft parts of the paper, and edit the paper to improve readability\. We have reviewed all AI\-assisted work\. AI\-generated code was reviewed and tested for correctness before being used in any experiment\. AI\-assisted qualitative analyses were independently verified by 2 authors\. All AI\-drafted or AI\-edited text was checked by the authors for accuracy and revised as needed\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

## References

- Balazadeh et al\. \(2025\)Vahid Balazadeh, Hamidreza Kamkari, Valentin Thomas, Junwei Ma, Bingru Li, Jesse C\. Cresswell, and Rahul Krishnan\.CausalPFN: Amortized causal effect estimation via in\-context learning\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.URL[https://openreview\.net/forum?id=RblaNJGx8C](https://openreview.net/forum?id=RblaNJGx8C)\.
- Cief et al\. \(2024\)Matej Cief, Jacek Golebiowski, Philipp Schmidt, Ziawasch Abedjan, and Artur Bekasov\.Learning action embeddings for off\-policy evaluation\.In*Advances in Information Retrieval*, 2024\.
- Cranmer et al\. \(2020\)Kyle Cranmer, Johann Brehmer, and Gilles Louppe\.The frontier of simulation\-based inference\.*Proceedings of the National Academy of Sciences*, 117\(48\):30055–30062, 2020\.doi:10\.1073/pnas\.1912789117\.URL[https://www\.pnas\.org/doi/abs/10\.1073/pnas\.1912789117](https://www.pnas.org/doi/abs/10.1073/pnas.1912789117)\.
- Cui et al\. \(2024\)Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun\.ULTRAFEEDBACK: Boosting language models with scaled AI feedback\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp \(eds\.\),*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 9722–9744\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/cui24f\.html](https://proceedings.mlr.press/v235/cui24f.html)\.
- Dudík et al\. \(2011\)Miroslav Dudík, John Langford, and Lihong Li\.Doubly robust policy evaluation and learning\.In*Proceedings of the 28th International Conference on International Conference on Machine Learning*, ICML’11, pp\. 1097–1104, Madison, WI, USA, 2011\. Omnipress\.ISBN 9781450306195\.
- Farajtabar et al\. \(2018\)Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh\.More robust doubly robust off\-policy evaluation\.In*International Conference on Machine Learning*, pp\. 1447–1456\. PMLR, 2018\.
- Gao et al\. \(2023\)Leo Gao, John Schulman, and Jacob Hilton\.Scaling laws for reward model overoptimization\.In*Proceedings of the 40th International Conference on Machine Learning*, ICML’23\. JMLR\.org, 2023\.
- Gemma Team et al\. \(2025\)Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, et al\.Gemma 3 technical report\.*arXiv preprint arXiv:2503\.19786*, 2025\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Helli et al\. \(2024\)Kai Helli, David Schnurr, Noah Hollmann, Samuel Müller, and Frank Hutter\.Drift\-resilient tabPFN: In\-context learning temporal distribution shifts on tabular data\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*, 2024\.URL[https://openreview\.net/forum?id=p3tSEFMwpG](https://openreview.net/forum?id=p3tSEFMwpG)\.
- Hollmann et al\. \(2023\)Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter\.TabPFN: A transformer that solves small tabular classification problems in a second\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=cp5PvcI6w8\_](https://openreview.net/forum?id=cp5PvcI6w8_)\.
- Hollmann et al\. \(2025\)Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter\.Accurate predictions on small data with a tabular foundation model\.*Nature*, 637\(8045\):319–326, 2025\.doi:10\.1038/s41586\-024\-08328\-6\.URL[https://doi\.org/10\.1038/s41586\-024\-08328\-6](https://doi.org/10.1038/s41586-024-08328-6)\.
- Jiang & Li \(2016\)Nan Jiang and Lihong Li\.Doubly robust off\-policy value evaluation for reinforcement learning\.In*International conference on machine learning*, pp\. 652–661\. PMLR, 2016\.
- Kallus et al\. \(2022\)Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou\.Doubly robust distributionally robust off\-policy evaluation and learning\.In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato \(eds\.\),*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pp\. 10598–10632\. PMLR, 17–23 Jul 2022\.URL[https://proceedings\.mlr\.press/v162/kallus22a\.html](https://proceedings.mlr.press/v162/kallus22a.html)\.
- Koenker & Bassett \(1978\)Roger Koenker and Gilbert Bassett\.Regression quantiles\.*Econometrica*, 46\(1\):33–50, 1978\.
- Lee et al\. \(2024\)Haanvid Lee, Tri Wahyu Guntara, Jongmin Lee, Yung\-Kyun Noh, and Kee\-Eung Kim\.Kernel metric learning for in\-sample off\-policy evaluation of deterministic RL policies\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=plebgsdiiV](https://openreview.net/forum?id=plebgsdiiV)\.
- Müller et al\. \(2022\)Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter\.Transformers can do bayesian inference\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=KSugKcbNf9](https://openreview.net/forum?id=KSugKcbNf9)\.
- Müller et al\. \(2023\)Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter\.PFNs4BO: In\-context learning for Bayesian optimization\.In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett \(eds\.\),*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pp\. 25444–25470\. PMLR, 23–29 Jul 2023\.URL[https://proceedings\.mlr\.press/v202/muller23a\.html](https://proceedings.mlr.press/v202/muller23a.html)\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe\.Training language models to follow instructions with human feedback\.In S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(eds\.\),*Advances in Neural Information Processing Systems*, volume 35, pp\. 27730–27744\. Curran Associates, Inc\., 2022\.doi:10\.52202/068431\-2011\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/file/b1efde53be364a73914f58805a001731\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)\.
- Qu et al\. \(2026\)Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan\.TabICLv2: A better, faster, scalable, and open tabular foundation model\.2026\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=HPuSIXJaa9](https://openreview.net/forum?id=HPuSIXJaa9)\.
- Sachdeva et al\. \(2024\)Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley\.Off\-policy evaluation for large action spaces via policy convolution\.In*Proceedings of the ACM Web Conference 2024*, WWW ’24, pp\. 3576–3585, New York, NY, USA, 2024\. Association for Computing Machinery\.ISBN 9798400701719\.doi:10\.1145/3589334\.3645501\.URL[https://doi\.org/10\.1145/3589334\.3645501](https://doi.org/10.1145/3589334.3645501)\.
- Saito & Joachims \(2022\)Yuta Saito and Thorsten Joachims\.Off\-policy evaluation for large action spaces via embeddings\.In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato \(eds\.\),*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pp\. 19089–19122\. PMLR, 17–23 Jul 2022\.URL[https://proceedings\.mlr\.press/v162/saito22a\.html](https://proceedings.mlr.press/v162/saito22a.html)\.
- Saito et al\. \(2021\)Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita\.Open bandit dataset and pipeline: Towards realistic and reproducible off\-policy evaluation, 2021\.URL[https://openreview\.net/forum?id=GfWRFRuO51B](https://openreview.net/forum?id=GfWRFRuO51B)\.
- Strehl et al\. \(2010\)Alex Strehl, John Langford, Lihong Li, and Sham M Kakade\.Learning from logged implicit exploration data\.In J\. Lafferty, C\. Williams, J\. Shawe\-Taylor, R\. Zemel, and A\. Culotta \(eds\.\),*Advances in Neural Information Processing Systems*, volume 23\. Curran Associates, Inc\., 2010\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2010/file/c0f168ce8900fa56e57789e2a2f2c9d0\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/2010/file/c0f168ce8900fa56e57789e2a2f2c9d0-Paper.pdf)\.
- Wang et al\. \(2024a\)Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang\.Interpretable preferences via multi\-objective reward modeling and mixture\-of\-experts\.In*Findings of Empirical Methods in Natural Language Processing*, 2024a\.
- Wang et al\. \(2017\)Yu\-Xiang Wang, Alekh Agarwal, and Miroslav Dudík\.Optimal and adaptive off\-policy evaluation in contextual bandits\.In Doina Precup and Yee Whye Teh \(eds\.\),*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pp\. 3589–3597\. PMLR, 06–11 Aug 2017\.URL[https://proceedings\.mlr\.press/v70/wang17a\.html](https://proceedings.mlr.press/v70/wang17a.html)\.
- Wang et al\. \(2024b\)Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J\. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev\.Helpsteer 2: Open\-source dataset for training top\-performing reward models\.In*The Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024b\.URL[https://openreview\.net/forum?id=PvVKUFhaNy](https://openreview.net/forum?id=PvVKUFhaNy)\.
- Xu et al\. \(2022\)Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan\.Prompting decision transformer for few\-shot policy generalization\.In*International Conference on Machine Learning*, pp\. 24631–24645\. PMLR, 2022\.
- Yang et al\. \(2025\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu\.Qwen2\.5 technical report, 2025\.URL[https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)\.
- Yang et al\. \(2024\)Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang\.Regularizing hidden states enables learning generalizable reward model for llms\.In A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(eds\.\),*Advances in Neural Information Processing Systems*, volume 37, pp\. 62279–62309\. Curran Associates, Inc\., 2024\.doi:10\.52202/079017\-1990\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/71f7154547c748c8041505521ca433ab\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/71f7154547c748c8041505521ca433ab-Paper-Conference.pdf)\.
- Zhang et al\. \(2025\)Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou\.Qwen3 embedding: Advancing text embedding and reranking through foundation models\.*arXiv preprint arXiv:2506\.05176*, 2025\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.In A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(eds\.\),*Advances in Neural Information Processing Systems*, volume 36, pp\. 46595–46623\. Curran Associates, Inc\., 2023\.doi:10\.52202/075280\-2020\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/91f18a1287b398d378ef22505bf41832\-Paper\-Datasets\_and\_Benchmarks\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)\.

## Appendix Contents

## Appendix ARelated Work

#### Prior\-fitted networks\.

Prior\-fitted networks \(PFN\) train a transformer on datasets sampled from a synthetic prior so that a single forward pass approximates posterior\-predictive inference in context\([Müller et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib17)\)\. PFNs have been successfully applied for tabular prediction\([Hollmann et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib11);[Hollmann et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib12);[Helli et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib10)\), Bayesian optimization\([Müller et al\., 2023](https://arxiv.org/html/2610.10848#bib.bib18)\)and causal effect estimation\([Balazadeh et al\., 2025](https://arxiv.org/html/2610.10848#bib.bib1)\)\. In our work, we extend the idea to off\-policy evaluation of LLMs\.

#### Off\-policy evaluation from text embeddings\.

Classical OPE estimators such as IPS are unable to handle continuous or large action space\. Kernel\-based IPS methods relax target policy distributions in IPS computation to a kernel\([Lee et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib16);[Sachdeva et al\., 2024](https://arxiv.org/html/2610.10848#bib.bib22)\)\. However, it is non\-trivial to extend these kernel\-based IPS to high dimensional text embedding\. MIPS\([Saito & Joachims, 2022](https://arxiv.org/html/2610.10848#bib.bib23)\)applies IPS in a given embedding space, marginalizing over action variations that share the same embedding\.[Cief et al\. \(2024\)](https://arxiv.org/html/2610.10848#bib.bib2)proposed using reward prediction loss to train the embedding for MIPS\. However, these MIPS\-based methods still need the per\-dataset fit\.

#### In\-context bandits and reinforcement learning\.

In bandits and reinforcement learning settings, in\-context learning has demonstrated efficient policy generalization\([Xu et al\., 2022](https://arxiv.org/html/2610.10848#bib.bib29)\)\. These methods condition transformer\-based policy models on trajectory data or interaction history to generalize to new tasks\. Our work applies this in\-context paradigm to off\-policy evaluation: we train PFN\-OPE to perform in\-context value estimation, eliminating per\-task fitting while enabling generalization to unseen policies and reward functions\.

## Appendix BAlgorithms

[Algorithm2](https://arxiv.org/html/2610.10848#alg2)gives every step of the procedure summarized in[Algorithm1](https://arxiv.org/html/2610.10848#alg1)\.

Algorithm 1PFN\-OPE pretraining1:Response pool: prompts

𝒳0\\mathcal\{X\}\_\{0\}, pool policies

\{g^\}g∈𝒢\\\{\\widehat\{g\}\\\}\_\{g\\in\\mathcal\{G\}\}, stored rewards

R1,…,RJR\_\{1\},\\ldots,R\_\{J\}\.

2:foreach pretraining exampledo

3:Sample a task

τ\\tau: prompt distribution

p⁡\(x∣τ\)p\(x\\mid\\tau\), weights

λτ,ωτ\\lambda\_\{\\tau\},\\omega\_\{\\tau\}, LLMs

g,g′g,g^\{\\prime\}, and

\(η,α\)\(\\eta,\\alpha\); form

RτR\_\{\\tau\}\([Equation5](https://arxiv.org/html/2610.10848#S3.E5)\),

μ=g^\\mu=\\widehat\{g\}, and

π\\pi\([Equation7](https://arxiv.org/html/2610.10848#S3.E7)\)\.

4:Compute

Vτ​\(π\)V\_\{\\tau\}\(\\pi\)over the pool \([Equation8](https://arxiv.org/html/2610.10848#S3.E8)\)\.

5:Draw

nnprompts

xi∼p\(⋅∣τ\)x\_\{i\}\\sim p\(\\cdot\\mid\\tau\)and, for each,

aiμ∼μ\(⋅∣xi\)a\_\{i\}^\{\\mu\}\\sim\\mu\(\\cdot\\mid x\_\{i\}\),

aiπ∼π\(⋅∣xi\)a\_\{i\}^\{\\pi\}\\sim\\pi\(\\cdot\\mid x\_\{i\}\), and

riμ=Rτ​\(xi,aiμ\)r\_\{i\}^\{\\mu\}=R\_\{\\tau\}\(x\_\{i\},a\_\{i\}^\{\\mu\}\); form

O=\(𝒟μ,𝒟π\)O=\(\\mathcal\{D\}\_\{\\mu\},\\mathcal\{D\}\_\{\\pi\}\)\.

6:Set

C=Vτ​\(π\)−r¯C=V\_\{\\tau\}\(\\pi\)\-\\bar\{r\}\([Equation2](https://arxiv.org/html/2610.10848#S3.E2)\)\.

7:Update

θ\\thetaon the loss of[Equation3](https://arxiv.org/html/2610.10848#S3.E3)at

\(O,C\)\(O,C\)\.

8:endfor

Algorithm 2Generating pretraining data for PFN\-OPE1:Prompts

𝒳0\\mathcal\{X\}\_\{0\}; LLMs

𝒢\\mathcal\{G\}; reward functions

R1,…,RJR\_\{1\},\\ldots,R\_\{J\}; responses per prompt

KK\.

2:Pretraining examples

\(O,C\)\(O,C\), with the rewards of the target responses hidden from

OO\.

3:Stage 1: build the response pool once

4:foreach prompt

x∈𝒳0x\\in\\mathcal\{X\}\_\{0\}and LLM

g∈𝒢g\\in\\mathcal\{G\}do

5:Sample

KKresponses

ag\(1\)\(x\),…,ag\(K\)\(x\)∼g\(⋅∣x\)a\_\{g\}^\{\(1\)\}\(x\),\\ldots,a\_\{g\}^\{\(K\)\}\(x\)\\sim g\(\\cdot\\mid x\)and store them as

𝒜g​\(x\)\\mathcal\{A\}\_\{g\}\(x\)\.

6:Compute

Rj​\(x,a\)R\_\{j\}\(x,a\)for every stored response

a∈𝒜g​\(x\)a\\in\\mathcal\{A\}\_\{g\}\(x\)and every

j=1,…,Jj=1,\\ldots,J\.

7:endfor

8:Standardize each reward over the pool,

R~j=\(Rj−mj\)/sj\\tilde\{R\}\_\{j\}=\(R\_\{j\}\-m\_\{j\}\)/s\_\{j\}\.

9:Define the pool policy

g^\(⋅∣x\)\\widehat\{g\}\(\\cdot\\mid x\)of each LLM as the uniform distribution on

𝒜g​\(x\)\\mathcal\{A\}\_\{g\}\(x\)\([Equation6](https://arxiv.org/html/2610.10848#S3.E6)\)\.

10:Stage 2: sample one task and one observation per pretraining example

11:foreach pretraining exampledo

12:Draw a prompt distribution

p\(⋅∣τ\)p\(\\cdot\\mid\\tau\)on

𝒳0\\mathcal\{X\}\_\{0\}and the number of logged rows

nn\.

13:Draw evaluation weights

λτ∼𝒩⁡\(w,diag⁡\(w\)−w​w⊤\)\\lambda\_\{\\tau\}\\sim\\mathcal\{N\}\\bigl\(w,\\operatorname\{diag\}\(w\)\-ww^\{\\top\}\\bigr\)and set

Rτ​\(x,a\)=∑j=1Jλτ​j​R~j​\(x,a\)R\_\{\\tau\}\(x,a\)=\\sum\_\{j=1\}^\{J\}\\lambda\_\{\\tau j\}\\tilde\{R\}\_\{j\}\(x,a\)\([Equation5](https://arxiv.org/html/2610.10848#S3.E5)\)\.

14:Draw selection weights

ωτ∈ΔJ−1\\omega\_\{\\tau\}\\in\\Delta^\{J\-1\}, independently of

λτ\\lambda\_\{\\tau\}, and set

Sτ​\(x,a\)=∑j=1Jωτ​j​R~j​\(x,a\)S\_\{\\tau\}\(x,a\)=\\sum\_\{j=1\}^\{J\}\\omega\_\{\\tau j\}\\tilde\{R\}\_\{j\}\(x,a\)\.

15:Draw a behavior LLM

g∈𝒢g\\in\\mathcal\{G\}and a target LLM

g′∈𝒢g^\{\\prime\}\\in\\mathcal\{G\}, and set

μ=g^\\mu=\\widehat\{g\}\.

16:Draw the policy mechanism: pool policy \(

η=0\\eta=0,

α=1\\alpha=1\) with probability

0\.40\.4, selected \(

η\>0\\eta\>0,

α=1\\alpha=1\) with probability

0\.40\.4, routed \(

α<1\\alpha<1\) with probability

0\.20\.2; draw

η\\etaand

α\\alphaaccordingly\.

17:Form the target policy \([Equation7](https://arxiv.org/html/2610.10848#S3.E7)\),

18:

νη​\(a∣x\)=g^′​\(a∣x\)​eη​Sτ​\(x,a\)∑a′g^′​\(a′∣x\)​eη​Sτ​\(x,a′\),π=\(1−α\)​μ\+α​νη\\displaystyle\\nu\_\{\\eta\}\(a\\mid x\)=\\frac\{\\widehat\{g\}^\{\\prime\}\(a\\mid x\)\\,e^\{\\eta S\_\{\\tau\}\(x,a\)\}\}\{\\sum\_\{a^\{\\prime\}\}\\widehat\{g\}^\{\\prime\}\(a^\{\\prime\}\\mid x\)\\,e^\{\\eta S\_\{\\tau\}\(x,a^\{\\prime\}\)\}\},\\qquad\\pi=\(1\-\\alpha\)\\,\\mu\+\\alpha\\,\\nu\_\{\\eta\}\.

19:Compute the target value over the pool \([Equation8](https://arxiv.org/html/2610.10848#S3.E8)\),

20:

Vτ​\(π\)=∑x∈𝒳0p⁡\(x∣τ\)​∑aπ⁡\(a∣x\)​Rτ​\(x,a\)\\displaystyle V\_\{\\tau\}\(\\pi\)=\\sum\_\{x\\in\\mathcal\{X\}\_\{0\}\}p\(x\\mid\\tau\)\\sum\_\{a\}\\pi\(a\\mid x\)\\,R\_\{\\tau\}\(x,a\)\.

21:Draw

nnprompts

x1,…,xn∼p\(⋅∣τ\)x\_\{1\},\\ldots,x\_\{n\}\\sim p\(\\cdot\\mid\\tau\)\.

22:for

i=1,…,ni=1,\\ldots,ndo

23:Draw

aiμ∼μ\(⋅∣xi\)a\_\{i\}^\{\\mu\}\\sim\\mu\(\\cdot\\mid x\_\{i\}\)and

aiπ∼π\(⋅∣xi\)a\_\{i\}^\{\\pi\}\\sim\\pi\(\\cdot\\mid x\_\{i\}\)independently\.

24:Set the logged reward

riμ=Rτ​\(xi,aiμ\)r\_\{i\}^\{\\mu\}=R\_\{\\tau\}\(x\_\{i\},a\_\{i\}^\{\\mu\}\)\.

25:endfor

26:Form

𝒟μ=\{\(xi,aiμ,riμ\)\}i=1n\\mathcal\{D\}\_\{\\mu\}=\\\{\(x\_\{i\},a\_\{i\}^\{\\mu\},r\_\{i\}^\{\\mu\}\)\\\}\_\{i=1\}^\{n\},

𝒟π=\{\(xi,aiπ\)\}i=1n\\mathcal\{D\}\_\{\\pi\}=\\\{\(x\_\{i\},a\_\{i\}^\{\\pi\}\)\\\}\_\{i=1\}^\{n\}, and

O=\(𝒟μ,𝒟π\)O=\(\\mathcal\{D\}\_\{\\mu\},\\mathcal\{D\}\_\{\\pi\}\)\.

27:Set

r¯=1n​∑i=1nriμ\\bar\{r\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}r\_\{i\}^\{\\mu\}and

C=Vτ​\(π\)−r¯C=V\_\{\\tau\}\(\\pi\)\-\\bar\{r\}\([Equation2](https://arxiv.org/html/2610.10848#S3.E2)\)\.

28:Emit

\(O,C\)\(O,C\);

OOholds prompts, responses, and logged rewards only, and no task parameter\.

29:endfor

## Appendix CExperimental details

This appendix gives the evaluation protocol and the estimator settings behind[Section4](https://arxiv.org/html/2610.10848#S4)\.

### C\.1Pool

#### Prompts and splits\.

Each dataset’s prompts are split at random into a pretraining set and1,0241\{,\}024evaluation prompts\. Every evaluation task in[Section4](https://arxiv.org/html/2610.10848#S4)draws its prompts from the evaluation set, so no evaluation prompt, behavior response, or target response appears in pretraining\.

#### Response generation and scoring\.

For each prompt and each LLM, seen or unseen, we sampleK=8K=8responses by nucleus sampling with temperature0\.70\.7and top\-pp0\.90\.9\. Every response is scored against its prompt by ArmoRM and GRM\.

### C\.2Evaluation protocol

#### Correction and rank groups\.

For a task with logged rewardsr1,…,rnr\_\{1\},\\ldots,r\_\{n\}, the correction is the prediction targetC=Vτ​\(π\)−r¯C=V\_\{\\tau\}\(\\pi\)\-\\bar\{r\}of[Equation2](https://arxiv.org/html/2610.10848#S3.E2)\. The standardized correction of[Section4\.4](https://arxiv.org/html/2610.10848#S4.SS4)isCstd=C/sμC\_\{\\mathrm\{std\}\}=C/s\_\{\\mu\}, wheresμs\_\{\\mu\}is the population standard deviation ofr1,…,rnr\_\{1\},\\ldots,r\_\{n\}, floored at10−610^\{\-6\}\.CCis not the value gapVτ​\(π\)−Vτ​\(μ\)V\_\{\\tau\}\(\\pi\)\-V\_\{\\tau\}\(\\mu\): it also contains the sampling deviation ofr¯\\bar\{r\}, so it varies across tasks of one setting whileVτ​\(π\)V\_\{\\tau\}\(\\pi\)stays fixed\. Within each setting \(dataset, reward, behavior LLM, target LLM, andnn\), 100 candidate tasks were generated, ranked by\|Cstd\|\|C\_\{\\mathrm\{std\}\}\|\.

### C\.3Estimators

All estimators receive the same observationO=\(𝒟μ,𝒟π\)O=\(\\mathcal\{D\}\_\{\\mu\},\\mathcal\{D\}\_\{\\pi\}\); target rewards are used only to computeVτ​\(π\)V\_\{\\tau\}\(\\pi\)\. Hyperparameters are fixed across all settings\. DM\-ridge, DM\-kernel, and DM\-MLP are refitted on every logged dataset\. DM\-TabPFN and DM\-TabICLv2 use frozen pretrained weights\. The TabPFN adapter projects the response embeddingsϕa\\phi\_\{a\}of[Section3\.3](https://arxiv.org/html/2610.10848#S3.SS3)to 128 dimensions with a fixed projection fitted on the response pool\. MIPS estimates the density ratio between target and logged responses from their embeddings with prompt\-grouped cross\-fitting, and never sees a target reward\. PFN\-OPE’s estimate isV^θ​\(π,O\)\\widehat\{V\}\_\{\\theta\}\(\\pi;O\)of[Equation4](https://arxiv.org/html/2610.10848#S3.E4): the logged mean plus the average of the 99 predicted quantiles ofCC\.

## Appendix DPrior\-Coverage Diagnostics

This appendix defines the two axes of[Figure4](https://arxiv.org/html/2610.10848#S4.F4.fig1)\. Both describe one fixed linear\-ridge reward probe on the frozen embeddings of[Section3\.3](https://arxiv.org/html/2610.10848#S3.SS3); neither is a PFN\-OPE prediction error\.

#### Probe and features\.

For a task with logged data𝒟μ\\mathcal\{D\}\_\{\\mu\}of sizen=128n=128, the feature of a prompt–response pair is the concatenation of the unit\-normalized prompt and response embeddings,ϕ⁡\(x,a\)≔12​\[ϕx​\(x\)/‖ϕx​\(x\)‖;ϕa​\(a\)/‖ϕa​\(a\)‖\]\\phi\(x,a\)\\coloneqq\\tfrac\{1\}\{\\sqrt\{2\}\}\\,\\bigl\[\\phi\_\{x\}\(x\)/\\\|\\phi\_\{x\}\(x\)\\\|;\\ \\phi\_\{a\}\(a\)/\\\|\\phi\_\{a\}\(a\)\\\|\\bigr\]\. A ridge regressor is fitted to the logged rewards inF=4F=4prompt\-grouped folds, foldbbusing the9696logged rows outside it, givingf^b​\(x,a\)=β^b⊤​ϕ​\(x,a\)\+γ^b\\widehat\{f\}\_\{b\}\(x,a\)=\\widehat\{\\beta\}\_\{b\}^\{\\top\}\\phi\(x,a\)\+\\widehat\{\\gamma\}\_\{b\}; the probe is the fold averagef¯≔1F​∑b=1Ff^b\\bar\{f\}\\coloneqq\\tfrac\{1\}\{F\}\\sum\_\{b=1\}^\{F\}\\widehat\{f\}\_\{b\}\. The expectations𝔼μ​\[⋅\]\\mathbb\{E\}\_\{\\mu\}\[\\cdot\]and𝔼π​\[⋅\]\\mathbb\{E\}\_\{\\pi\}\[\\cdot\]below are over the task’s prompt distribution and its pool policies, computed exactly on the stored responses\.

#### Probe correction error \(vertical axis\)\.

The probe’s estimate of the value gap isΔ^≔𝔼π​\[f¯\]−𝔼μ​\[f¯\]\\widehat\{\\Delta\}\\coloneqq\\mathbb\{E\}\_\{\\pi\}\[\\bar\{f\}\]\-\\mathbb\{E\}\_\{\\mu\}\[\\bar\{f\}\]and the true gap isΔ≔Vτ​\(π\)−Vτ​\(μ\)\\Delta\\coloneqq V\_\{\\tau\}\(\\pi\)\-V\_\{\\tau\}\(\\mu\)\. The diagnostic is the squared error of the estimate in units of the logged reward varianceσμ2≔Varμ​\(Rτ\)\\sigma\_\{\\mu\}^\{2\}\\coloneqq\\mathrm\{Var\}\_\{\\mu\}\(R\_\{\\tau\}\),

BΔ2≔\(Δ^−Δ\)2σμ2\.B\_\{\\Delta\}^\{2\}\\coloneqq\\frac\{\(\\widehat\{\\Delta\}\-\\Delta\)^\{2\}\}\{\\sigma\_\{\\mu\}^\{2\}\}\.\(10\)Writinge≔f¯−Rτe\\coloneqq\\bar\{f\}\-R\_\{\\tau\}for the probe’s pointwise error,BΔ2=\(𝔼π​\[e\]−𝔼μ​\[e\]\)2/σμ2B\_\{\\Delta\}^\{2\}=\(\\mathbb\{E\}\_\{\\pi\}\[e\]\-\\mathbb\{E\}\_\{\\mu\}\[e\]\)^\{2\}/\\sigma\_\{\\mu\}^\{2\}: an error common to logged and target responses cancels, soBΔ2B\_\{\\Delta\}^\{2\}is zero when the probe misjudges both values by the same amount and large only when its error differs between the two response sets\. It mixes finite\-sample error, regularization, and misspecification of the linear probe, and it is not the probe’s error on the target value itself\.

#### Value\-relevant extrapolation \(horizontal axis\)\.

Letd≔𝔼π​\[ϕ\]−𝔼μ​\[ϕ\]d\\coloneqq\\mathbb\{E\}\_\{\\pi\}\[\\phi\]\-\\mathbb\{E\}\_\{\\mu\}\[\\phi\]be the mean feature shift andΣ^b\\widehat\{\\Sigma\}\_\{b\}the covariance of the features of foldbb’s9696logged rows\. The diagnostic is the ridge\-regularized leverage of the shift, averaged over folds,

LΔ≔1F​∑b=1Fd⊤​\(Σ^b\+λb​I\)−1​d,L\_\{\\Delta\}\\coloneqq\\frac\{1\}\{F\}\\sum\_\{b=1\}^\{F\}d^\{\\top\}\\bigl\(\\widehat\{\\Sigma\}\_\{b\}\+\\lambda\_\{b\}I\\bigr\)^\{\-1\}d,\(11\)withλb\\lambda\_\{b\}the ridge penalty of foldbb\. In the eigenbasis\{\(sk,uk\)\}\\\{\(s\_\{k\},u\_\{k\}\)\\\}ofΣ^b\\widehat\{\\Sigma\}\_\{b\},d⊤​\(Σ^b\+λb​I\)−1​d=∑k\(uk⊤​d\)2/\(sk\+λb\)d^\{\\top\}\(\\widehat\{\\Sigma\}\_\{b\}\+\\lambda\_\{b\}I\)^\{\-1\}d=\\sum\_\{k\}\(u\_\{k\}^\{\\top\}d\)^\{2\}/\(s\_\{k\}\+\\lambda\_\{b\}\), so a shift along a direction in which the logged features barely vary \(smallsks\_\{k\}\) counts more than the same shift along a direction they cover\. The shiftddis what a linear probe’s value correction depends on,Δ^=\(1F​∑bβ^b\)⊤​d\\widehat\{\\Delta\}=\\bigl\(\\tfrac\{1\}\{F\}\\sum\_\{b\}\\widehat\{\\beta\}\_\{b\}\\bigr\)^\{\\top\}d, which is why we call the leverage value\-relevant\.

## Appendix ELimitations

#### Evaluation targets are other LLMs\.

Every evaluation target is the pool policy of one LLM, so the policy shifts we test are changes of LLM\. The prior\-coverage analysis \([Section4\.6](https://arxiv.org/html/2610.10848#S4.SS6)\) shows that the tilted policies reach the kind of task a post\-trained checkpoint would pose\.

#### Rewards come from reward models\.

Every reward in pretraining and evaluation is the output of ArmoRM or GRM, so the setting is semi\-synthetic: reward definitions are heads and combinations of reward models rather than human rubrics, and any bias of those models enters the pool\. Human\-rated logs would test the reward shift the introduction describes; we did not have them at the scale the pool requires\.

#### The prior is built from three LLMs\.

Transfer to an unseen checkpoint of a seen family and to one unseen family \([Section4\.3](https://arxiv.org/html/2610.10848#S4.SS3)\) is the evidence that the estimator generalizes beyond the pool’s LLMs, and one unseen family is limited evidence\. Widening the pool to more families and to actual post\-training checkpoints is the natural extension, at the one\-time cost of generating and scoring their responses\.