预算约束下的多源反事实标注用于离策略评估

# Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
Source: [https://arxiv.org/html/2610.10974](https://arxiv.org/html/2610.10974)
Ali Eshragh††thanks:Equal supervision\.Affiliation:Johns Hopkins Carey Business School, Washington, DCAffiliation:International Computer Science Institute, Berkeley, CAYuexing LiKai Wang\[0\.1em\] Pennsylvania State UniversityState CollegePAAffiliation:Georgia Institute of Technology, Atlanta, GA

###### Abstract

Off\-policy evaluation \(OPE\) estimates the value of a target policy from logged data, but limited behavior\-policy coverage can force high\-variance reweighting or reward\-model extrapolation\. Counterfactual annotations can add evidence about unobserved actions, yet practical sources, including domain experts and large language models \(LLMs\), may be costly, biased, or noisy\. We study budgeted acquisition of such annotations for contextual\-bandit OPE\. Given source\-specific costs and error profiles, we formulate an integer allocation problem over context\-action pairs and annotation sources to minimize the component of estimator variance that depends on the annotation plan\. We characterize when annotations are valuable through a first\-annotation threshold and local annotation\-value regimes\. For the coupled multi\-source problem, we develop a majorization\-minimization algorithm with dynamic\-programming subroutines that monotonically improves the objective\. Experiments in synthetic clinical and LLM\-annotated education bandits show that our allocation method reduces fixed\-profile mean squared error \(MSE\) by20\.58%20\.58\\%and10\.77%10\.77\\%, respectively, relative to no annotation\.

## 1Introduction

Off\-policy evaluation \(OPE\) estimates the value of a target policy from data collected by a different behavior policy\. It is especially important when online evaluation is expensive, disruptive, or risky, as in healthcare and education\. In contextual bandits, logged data reveal rewards only for actions selected by the behavior policy\. When the target policy assigns probability to actions that are poorly covered in the logged data, evaluation can depend on high\-variance reweighting, reward\-model extrapolation, or both\.

Counterfactual annotations offer a practical way to add evidence about unobserved actions without deploying the target policy\. For example, a clinician may assess what would likely have happened under an alternative treatment, or an automated tutor may estimate a student’s response under a different instructional activity\. Recent work has shown how such annotations can be incorporated into OPE estimators\[[1](https://arxiv.org/html/2610.10974#bib.bib1),[2](https://arxiv.org/html/2610.10974#bib.bib2)\]\. However, a key practical question remains: when annotations are costly and imperfect, which ones should be collected?

This question matters because annotation sources vary in cost and reliability\. Domain experts may be accurate but expensive, while large language models \(LLMs\) may scale cheaply yet make systematic errors on particular context\-action pairs\. In the counterfactual\-annotated estimator we study, such errors do not introduce bias into the policy\-value estimate but affect variance through the reward model\. Annotation value therefore depends jointly on source quality, cost, factual coverage, and how strongly the target policy weights the annotated action\.

We formulate counterfactual\-annotation acquisition as a budget\-constrained allocation problem: given estimates of each source’s cost, bias, and variability, choose which context\-action pairs to annotate and which sources to query so as to minimize the part of the estimation error that depends on the annotation plan\. The problem is discrete because annotations are acquired one at a time, and coupled because annotations for different actions in the same context jointly shape the reward model used for evaluation\. These challenges require both a statistical understanding of when annotations are valuable and an optimization method for allocating them under budget\. To address them, we make three contributions:

1. \(I\)Annotation\-value regimes\.We derive a first\-annotation threshold and a four\-regime characterization of when annotations help immediately, help only up to a finite amount, help only after acquiring a sufficiently large batch, or do not help locally\.
2. \(II\)Budgeted allocation algorithm\.We show that the allocation problem is coupled, nonconvex, and not generally submodular, and develop a majorization\-minimization \(MM\) algorithm with dynamic\-programming subroutines that monotonically improves the original objective\.
3. \(III\)Empirical validation\.Across synthetic and LLM\-annotated bandit environments, we compare against greedy allocation and certified solver benchmarks, quantify profile\-estimation sensitivity, and validate the final OPE estimator using finite pilots and independent evaluation samples\.

## 2Related Work

Off\-policy evaluation \(OPE\) estimates the value of a target policy from data generated by a behavior policy\. Core estimators include importance sampling and per\-decision variants\[[19](https://arxiv.org/html/2610.10974#bib.bib19)\], direct\-method estimators that learn reward, value, or transition models\[[25](https://arxiv.org/html/2610.10974#bib.bib25),[26](https://arxiv.org/html/2610.10974#bib.bib26)\], and doubly robust \(DR\) estimators that combine model estimates with importance\-weighted corrections\[[11](https://arxiv.org/html/2610.10974#bib.bib11),[12](https://arxiv.org/html/2610.10974#bib.bib12),[7](https://arxiv.org/html/2610.10974#bib.bib7)\]\. Variance reduction and reliability have been further studied through data\-efficient DR estimators\[[24](https://arxiv.org/html/2610.10974#bib.bib24),[13](https://arxiv.org/html/2610.10974#bib.bib13)\], representation\-based OPE\[[27](https://arxiv.org/html/2610.10974#bib.bib27)\], empirical benchmarking\[[28](https://arxiv.org/html/2610.10974#bib.bib28)\], and confidence or estimator\-selection methods\[[23](https://arxiv.org/html/2610.10974#bib.bib23),[22](https://arxiv.org/html/2610.10974#bib.bib22),[14](https://arxiv.org/html/2610.10974#bib.bib14),[17](https://arxiv.org/html/2610.10974#bib.bib17)\]\. Our work builds on this OPE foundation, but addresses the acquisition stage: which counterfactual annotations to collect beforeapplying the estimator\.

A related stream augments logged data with counterfactual or auxiliary information\. Structural\-causal methods construct counterfactual trajectories\[[10](https://arxiv.org/html/2610.10974#bib.bib10),[18](https://arxiv.org/html/2610.10974#bib.bib18)\]; semi\-offline OPE reweights human counterfactual annotations to preserve the logged context distribution\[[1](https://arxiv.org/html/2610.10974#bib.bib1)\]; and CANDOR incorporates imperfect annotations into DR contextual\-bandit estimators, particularly through the reward model\[[2](https://arxiv.org/html/2610.10974#bib.bib2)\]\. Auxiliary\-data methods such as trajectory augmentation, PERRY, and APRIL use generated trajectories or LLM\-derived predictions for inference and policy evaluation\[[9](https://arxiv.org/html/2610.10974#bib.bib9),[15](https://arxiv.org/html/2610.10974#bib.bib15),[16](https://arxiv.org/html/2610.10974#bib.bib16),[4](https://arxiv.org/html/2610.10974#bib.bib4)\]\. These works study how to use auxiliary information once available; we study how to allocate a finite annotation budget across heterogeneous sources\.

Our problem is also related to budgeted data acquisition: active learning and optimal design choose measurements under resource constraints\[[21](https://arxiv.org/html/2610.10974#bib.bib21),[20](https://arxiv.org/html/2610.10974#bib.bib20)\], while multiple\-logger OPE combines logged data from different behavior policies\[[8](https://arxiv.org/html/2610.10974#bib.bib8)\]\. Our setting differs because the acquired data are counterfactual rewards used in a DR reward model, rather than supervised labels or additional logged actions\. Thus, annotation value is not standalone informativeness; it is defined by how the annotation changes the resulting OPE estimator’s variance under the target and behavior policies\.

## 3Problem Formulation

We study off\-policy evaluation with counterfactual annotations in a contextual bandit with finite context space𝒮\\mathcal\{S\}and action space𝒜\\mathcal\{A\}\. We use separate factual datasets for reward\-model fitting and OPE:𝒟fit=\{\(s~j,a~j,r~j\)\}j=1M\\mathcal\{D\}\_\{\\mathrm\{fit\}\}=\\\{\(\\tilde\{s\}\_\{j\},\\tilde\{a\}\_\{j\},\\tilde\{r\}\_\{j\}\)\\\}\_\{j=1\}^\{M\}and𝒟ope=\{\(si,ai,ri\)\}i=1N\\mathcal\{D\}\_\{\\mathrm\{ope\}\}=\\\{\(s\_\{i\},a\_\{i\},r\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. For each record in either dataset, the contexts∈𝒮s\\in\\mathcal\{S\}is drawn from a distributiond0d\_\{0\}, the actiona∈𝒜a\\in\\mathcal\{A\}is chosen by a behavior policyπb\(⋅\|s\)\\pi\_\{b\}\(\\cdot\|s\), and the rewardrris drawn from the reward distributionR⁡\(s,a\)R\(s,a\)\. We writeμr\(s,a\)=𝔼\[r\|s,a\]\\mu\_\{r\}\(s,a\)=\\mathbb\{E\}\[r\|s,a\]andσr2​\(s,a\)=Var⁡\(r\|s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)=\\operatorname\{Var\}\(r\|s,a\)for the true mean and variance of rewards for context\-action pair\(s,a\)\(s,a\)\.

In addition to the logged data, annotations may be obtained fromKKindependent sources\. For each pair\(s,a\)\(s,a\)and sourcek∈𝒦=\{1,…,K\}k\\in\\mathcal\{K\}=\\\{1,\\ldots,K\\\}, letnk​\(s,a\)n\_\{k\}\(s,a\)be the number of collected annotations, and letgk,j∼Gk​\(s,a\)g\_\{k,j\}\\sim G\_\{k\}\(s,a\),j=1,…,nk​\(s,a\)j=1,\\ldots,n\_\{k\}\(s,a\), denote independent annotation draws from that source\. Denote𝔼\[gk,j\|s,a\]=μr\(s,a\)\+εgk\(s,a\)\\mathbb\{E\}\[g\_\{k,j\}\|s,a\]=\\mu\_\{r\}\(s,a\)\+\\varepsilon\_\{g\_\{k\}\}\(s,a\)andVar⁡\(gk,j\|s,a\)=σr2​\(s,a\)\+Δgk​\(s,a\)\\operatorname\{Var\}\(g\_\{k,j\}\|s,a\)=\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{g\_\{k\}\}\(s,a\), whereεgk\\varepsilon\_\{g\_\{k\}\}denotes annotation bias andΔgk\\Delta\_\{g\_\{k\}\}denotes excess annotation variance for sourcekk\. The fitting data and acquired annotations are independent of the OPE evaluation sample\.

Because annotations may be biased, we leverage the idea of theDM\+​\-​IS\\mathrm\{DM\}^\{\+\}\\text\{\-\}\\mathrm\{IS\}estimator\[[2](https://arxiv.org/html/2610.10974#bib.bib2)\]by incorporating annotations only in the reward estimate, rather than treating them as factual rewards in the importance\-sampling correction\. For each pair\(s,a\)\(s,a\), letnf​\(s,a\)=∑j=1M𝟙​\{s~j=s,a~j=a\}n\_\{f\}\(s,a\)=\\sum\_\{j=1\}^\{M\}\\mathds\{1\}\\\{\\tilde\{s\}\_\{j\}=s,\\tilde\{a\}\_\{j\}=a\\\}denote the number of fitting observations for that pair in𝒟fit\\mathcal\{D\}\_\{\\mathrm\{fit\}\}, and letN⁡\(s,a\)=nf​\(s,a\)\+∑k=1Knk​\(s,a\)N\(s,a\)=n\_\{f\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)denote the total number of factual and annotated reward observations available for that pair\. We define the annotation\-augmented reward estimate, forN⁡\(s,a\)\>0N\(s,a\)\>0, as

r^\+​\(s,a\)=∑j=1M𝟙​\{s~j=s,a~j=a\}​r~j\+∑k=1K∑j=1nk​\(s,a\)gk,jN⁡\(s,a\)\.\\hat\{r\}^\{\+\}\(s,a\)=\\frac\{\\sum\_\{j=1\}^\{M\}\\mathds\{1\}\\\{\\tilde\{s\}\_\{j\}=s,\\tilde\{a\}\_\{j\}=a\\\}\\tilde\{r\}\_\{j\}\+\\sum\_\{k=1\}^\{K\}\\sum\_\{j=1\}^\{n\_\{k\}\(s,a\)\}g\_\{k,j\}\}\{N\(s,a\)\}\.
ForN⁡\(s,a\)=0N\(s,a\)=0, use a fixed defaultr0​\(s,a\)r\_\{0\}\(s,a\), this case is treated separately in Appendix[B\.8](https://arxiv.org/html/2610.10974#A2.SS8)\.

Thus, for a target policyπe\\pi\_\{e\}with standard common support, i\.e\.,πb​\(a\|s\)\>0\\pi\_\{b\}\(a\|s\)\>0wheneverπe​\(a\|s\)\>0\\pi\_\{e\}\(a\|s\)\>0for alls∈𝒮,a∈𝒜s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}, theDM\+​\-​IS\\mathrm\{DM\}^\{\+\}\\text\{\-\}\\mathrm\{IS\}estimator is:

v^DM\+​\-​IS​\(πe\)=1N​∑i=1N\[r^\+​\(si,πe\)\+πe​\(ai\|si\)πb​\(ai\|si\)​\(ri−r^\+​\(si,ai\)\)\],\\displaystyle\\hat\{v\}^\{\\mathrm\{DM\}^\{\+\}\\text\{\-\}\\mathrm\{IS\}\}\(\\pi\_\{e\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[\\hat\{r\}^\{\+\}\(s\_\{i\},\\pi\_\{e\}\)\+\\frac\{\\pi\_\{e\}\(a\_\{i\}\|s\_\{i\}\)\}\{\\pi\_\{b\}\(a\_\{i\}\|s\_\{i\}\)\}\\bigl\(r\_\{i\}\-\\hat\{r\}^\{\+\}\(s\_\{i\},a\_\{i\}\)\\bigr\)\\right\],\(1\)wherer^\+​\(s,πe\)=∑a∈𝒜πe​\(a\|s\)​r^\+​\(s,a\)\\hat\{r\}^\{\+\}\(s,\\pi\_\{e\}\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)\\hat\{r\}^\{\+\}\(s,a\)is the estimated value ofπe\\pi\_\{e\}at contextss\. Our goal is to minimize the mean squared error \(MSE\) ofv^DM\+​\-​IS​\(πe\)\\hat\{v\}^\{\\mathrm\{DM\}^\{\+\}\\text\{\-\}\\mathrm\{IS\}\}\(\\pi\_\{e\}\)by choosing𝐧\\mathbf\{n\}, i\.e\., how many annotations should be queried from each source for each context\-action pair\.

## 4Objective Function and Budget\-Constrained Allocation

To express how allocation affects MSE, we define reward\-model bias and variance forN⁡\(s,a\)\>0N\(s,a\)\>0as

B⁡\(s,a,𝐧\)\\displaystyle B\(s,a;\\mathbf\{n\}\)=∑knk​\(s,a\)​εgk​\(s,a\)N⁡\(s,a\),\\displaystyle=\\frac\{\\sum\_\{k\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\}\{N\(s,a\)\},W⁡\(s,a,𝐧\)\\displaystyle W\(s,a;\\mathbf\{n\}\)=nf​\(s,a\)​σr2​\(s,a\)\+∑knk​\(s,a\)​\[σr2​\(s,a\)\+Δgk​\(s,a\)\]N​\(s,a\)2\.\\displaystyle=\\frac\{n\_\{f\}\(s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k\}n\_\{k\}\(s,a\)\\bigl\[\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{g\_\{k\}\}\(s,a\)\\bigr\]\}\{N\(s,a\)^\{2\}\}\.ForN⁡\(s,a\)=0N\(s,a\)=0, setB⁡\(s,a,𝐧\)=r0​\(s,a\)−μr​\(s,a\)B\(s,a;\\mathbf\{n\}\)=r\_\{0\}\(s,a\)\-\\mu\_\{r\}\(s,a\)andW⁡\(s,a,𝐧\)=0W\(s,a;\\mathbf\{n\}\)=0\. Writeℬs​\(𝐧\)=∑aπe​\(a∣s\)​B​\(s,a,𝐧\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)=\\sum\_\{a\}\\pi\_\{e\}\(a\\mid s\)B\(s,a;\\mathbf\{n\}\),η⁡\(s,a\)=1−πb​\(a∣s\)\\eta\(s,a\)=1\-\\pi\_\{b\}\(a\\mid s\),α1​\(s,a\)=d0​\(s\)​πe​\(a∣s\)2/πb​\(a∣s\)\\alpha\_\{1\}\(s,a\)=d\_\{0\}\(s\)\\pi\_\{e\}\(a\\mid s\)^\{2\}/\\pi\_\{b\}\(a\\mid s\), andα3​\(s,a\)=η⁡\(s,a\)​α1​\(s,a\)\\alpha\_\{3\}\(s,a\)=\\eta\(s,a\)\\alpha\_\{1\}\(s,a\), withα1=α3=0\\alpha\_\{1\}=\\alpha\_\{3\}=0whenπe​\(a∣s\)=0\\pi\_\{e\}\(a\\mid s\)=0\. The following result isolates the part of OPE error controlled by the annotation plan\.

###### Theorem 1\(Unbiasedness and allocation\-dependent MSE\)\.

Under the independent\-sample design above, known behavior propensities, common support, and finite second moments, fix the factual counts and a feasible allocation𝐧\\mathbf\{n\}\. The estimator in \([1](https://arxiv.org/html/2610.10974#S3.E1)\) is unbiased, and

MSE⁡\(𝐧\)=C0\+J⁡\(𝐧\)N,\\operatorname\{MSE\}\(\\mathbf\{n\}\)=C\_\{0\}\+\\frac\{J\(\\mathbf\{n\}\)\}\{N\},whereC0C\_\{0\}is allocation\-invariant and

J⁡\(𝐧\)\\displaystyle J\(\\mathbf\{n\}\)=J1​\(𝐧\)−J2​\(𝐧\)\+J3​\(𝐧\),\\displaystyle=J\_\{1\}\(\\mathbf\{n\}\)\-J\_\{2\}\(\\mathbf\{n\}\)\+J\_\{3\}\(\\mathbf\{n\}\),\(2\)J1​\(𝐧\)\\displaystyle J\_\{1\}\(\\mathbf\{n\}\)=∑s,aα1​\(s,a\)​B​\(s,a,𝐧\)2,\\displaystyle=\\sum\_\{s,a\}\\alpha\_\{1\}\(s,a\)B\(s,a;\\mathbf\{n\}\)^\{2\},J2​\(𝐧\)\\displaystyle J\_\{2\}\(\\mathbf\{n\}\)=∑sd0​\(s\)​ℬs​\(𝐧\)2,\\displaystyle=\\sum\_\{s\}d\_\{0\}\(s\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\},J3​\(𝐧\)\\displaystyle J\_\{3\}\(\\mathbf\{n\}\)=∑s,aα3​\(s,a\)​W​\(s,a,𝐧\)\.\\displaystyle=\\sum\_\{s,a\}\\alpha\_\{3\}\(s,a\)W\(s,a;\\mathbf\{n\}\)\.

Although the OPE estimator remains unbiased, annotation bias affects its variance throughJ1−J2J\_\{1\}\-J\_\{2\}, alongside reward\-model varianceJ3J\_\{3\}\. BecauseC0C\_\{0\}is allocation\-invariant, minimizingJJexactly minimizes fixed\-profile MSE\. For reporting, we distinguishJJfrom scaled total MSE,𝒱⁡\(𝐧\):=N​C0\+J⁡\(𝐧\)=N​MSE⁡\(𝐧\)\\mathcal\{V\}\(\\mathbf\{n\}\):=NC\_\{0\}\+J\(\\mathbf\{n\}\)=N\\operatorname\{MSE\}\(\\mathbf\{n\}\)\. The unbiasedness result extends[Mandyam et al\. \[2\]](https://arxiv.org/html/2610.10974#bib.bib2)to multiple sources; Appendix[A](https://arxiv.org/html/2610.10974#A1)gives the proof andC0C\_\{0\}\.

Letck\>0c\_\{k\}\>0be the cost of obtaining one annotation from sourcekk, and letκ\>0\\kappa\>0be the total annotation budget\. To focus on how the budget should be distributed across sources and context\-action pairs, we consider the setting where each factual sample receives at most one counterfactual annotation\. LetNs:=∑a∈𝒜nf​\(s,a\)N\_\{s\}:=\\sum\_\{a\\in\\mathcal\{A\}\}n\_\{f\}\(s,a\)denote the number of factual samples with contextss\. The resulting budget\-constrained allocation problem is

min𝐧\\displaystyle\\min\_\{\\mathbf\{n\}\}J⁡\(𝐧\)\\displaystyle J\(\\mathbf\{n\}\)\(3\)s\.t\.\\displaystyle\\text\{s\.t\.\}∑s∈𝒮∑a∈𝒜∑k∈𝒦ck​nk​\(s,a\)≤κ,\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\sum\_\{k\\in\\mathcal\{K\}\}c\_\{k\}n\_\{k\}\(s,a\)\\leq\\kappa,∑a∈𝒜∑k∈𝒦nk\(s,a\)≤Ns,∀s∈𝒮,\\displaystyle\\sum\_\{a\\in\\mathcal\{A\}\}\\sum\_\{k\\in\\mathcal\{K\}\}n\_\{k\}\(s,a\)\\leq N\_\{s\},\\quad\\forall s\\in\\mathcal\{S\},∑k∈𝒦nk\(s,a\)≤Ns−nf\(s,a\),∀s∈𝒮,a∈𝒜,\\displaystyle\\sum\_\{k\\in\\mathcal\{K\}\}n\_\{k\}\(s,a\)\\leq N\_\{s\}\-n\_\{f\}\(s,a\),\\quad\\forall s\\in\\mathcal\{S\},\\;a\\in\\mathcal\{A\},nk\(s,a\)∈ℤ≥0,∀s∈𝒮,a∈𝒜,k∈𝒦\.\\displaystyle n\_\{k\}\(s,a\)\\in\\mathbb\{Z\}\_\{\\geq 0\},\\quad\\forall s\\in\\mathcal\{S\},\\;a\\in\\mathcal\{A\},\\;k\\in\\mathcal\{K\}\.
Writeℱ\\mathcal\{F\}for the feasible set in \([3](https://arxiv.org/html/2610.10974#S4.E3)\)\. The first constraint enforces the budget; the second and third encode the one\-annotation protocol and can be relaxed if repeated annotations are allowed\.

In practice, source profiles are estimated from an independent pilot, so we compare allocations selected under estimated and true profiles\. Letθ\\thetaandθ^\\hat\{\\theta\}denote the true and estimated source profiles, and let𝐧⋆∈arg⁡min𝐧∈ℱ⁡J⁡\(𝐧,θ\)\\mathbf\{n\}^\{\\star\}\\in\\arg\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\};\\theta\)\. For a feasible allocation𝐧^\\widehat\{\\mathbf\{n\}\}computed usingθ^\\hat\{\\theta\}, writeJ⁡\(𝐧^,θ^\)≤min𝐧∈ℱ⁡J⁡\(𝐧,θ^\)\+ξalgJ\(\\widehat\{\\mathbf\{n\}\};\\hat\{\\theta\}\)\\leq\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\};\\hat\{\\theta\}\)\+\\xi\_\{\\mathrm\{alg\}\}, whereξalg≥0\\xi\_\{\\mathrm\{alg\}\}\\geq 0is its optimization error\. Letδε,δΔ\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}denote the uniform errors in estimated biases and excess variances,βmax\\beta\_\{\\max\}bound annotation and default\-model bias magnitudes, andA1,A2A\_\{1\},A\_\{2\}be the sensitivity constants defined in Appendix[A\.4](https://arxiv.org/html/2610.10974#A1.SS4)\. The following result translates profile\-estimation and optimization errors into excess OPE MSE\.

###### Theorem 2\(Effect of source\-profile estimation on MSE\)\.

Under Theorem[1](https://arxiv.org/html/2610.10974#Thmtheorem1)’s assumptions, conditional on the independent pilot and with fixed factual reward moments and policy probabilities,

MSE⁡\(𝐧^\)−MSE⁡\(𝐧⋆\)≤2​A1​δε​\(2​βmax\+δε\)\+2​A2​δΔ\+ξalgN\.\\operatorname\{MSE\}\(\\widehat\{\\mathbf\{n\}\}\)\-\\operatorname\{MSE\}\(\\mathbf\{n\}^\{\\star\}\)\\leq\\frac\{2A\_\{1\}\\delta\_\{\\varepsilon\}\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\)\+2A\_\{2\}\\delta\_\{\\Delta\}\+\\xi\_\{\\mathrm\{alg\}\}\}\{N\}\.\(4\)

The guarantee allows estimated profiles to change which sources or pairs receive annotations while controlling the resulting MSE loss\. As profile errors vanish, the bound reduces toξalg/N\\xi\_\{\\mathrm\{alg\}\}/N, distinguishing the gains from better source\-profile estimates from those obtained by improving the allocation solver\. We examine these two effects in §[7](https://arxiv.org/html/2610.10974#S7); Appendix[A\.4](https://arxiv.org/html/2610.10974#A1.SS4)gives the proof\.

## 5Annotation Value: Thresholds, Regimes, and Multi\-Source Benefits

Before solving problem \([3](https://arxiv.org/html/2610.10974#S4.E3)\), we examine when counterfactual annotations are statistically valuable\. We first study a single context\-action pair and source to derive the first\-annotation threshold \(§[5\.1](https://arxiv.org/html/2610.10974#S5.SS1)\) and four regimes of annotation value \(§[5\.2](https://arxiv.org/html/2610.10974#S5.SS2)\)\. These results motivate source selection and bias cancellation \(§[5\.3](https://arxiv.org/html/2610.10974#S5.SS3)\), followed by an analysis of cross\-action coupling in the full allocation problem \(§[5\.4](https://arxiv.org/html/2610.10974#S5.SS4)\)\.

### 5\.1Local Single\-Source Objective and First\-Annotation Threshold

To gain actionable insight, we first focus on a single context\-action pair\(s,a\)\(s,a\)withnf​\(s,a\)\>0n\_\{f\}\(s,a\)\>0andα1​\(s,a\)\>0\\alpha\_\{1\}\(s,a\)\>0, and a single annotation sourcegg\. The casenf​\(s,a\)=0n\_\{f\}\(s,a\)=0is discussed in Appendix[B\.8](https://arxiv.org/html/2610.10974#A2.SS8)\. For ease of notation, we suppress the dependence on\(s,a\)\(s,a\)throughout and writenf:=nf​\(s,a\)n\_\{f\}:=n\_\{f\}\(s,a\),ε:=εg​\(s,a\)\\varepsilon:=\\varepsilon\_\{g\}\(s,a\),Δ:=Δg​\(s,a\)\\Delta:=\\Delta\_\{g\}\(s,a\),σ2:=σr2​\(s,a\)\\sigma^\{2\}:=\\sigma\_\{r\}^\{2\}\(s,a\),σg2:=σ2\+Δ\\sigma\_\{g\}^\{2\}:=\\sigma^\{2\}\+\\Delta,η:=η⁡\(s,a\)\\eta:=\\eta\(s,a\), andαi:=αi​\(s,a\)\\alpha\_\{i\}:=\\alpha\_\{i\}\(s,a\)fori=1,3i=1,3\. Fornnannotations, the\(s,a\)\(s,a\)terms ofJ1\+J3J\_\{1\}\+J\_\{3\}give the local objective

ϕ⁡\(n\):=α1​n2​ε2\+α3​\(nf​σ2\+n​σg2\)\(nf\+n\)2\.\\phi\(n\):=\\frac\{\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}\\bigl\(n\_\{f\}\\sigma^\{2\}\+n\\sigma\_\{g\}^\{2\}\\bigr\)\}\{\(n\_\{f\}\+n\)^\{2\}\}\.\(5\)We omitJ2J\_\{2\}here because it couples actions within the same context, and revisit its effect in §[5\.4](https://arxiv.org/html/2610.10974#S5.SS4)\. By isolating this local scenario, we derive a threshold condition under which adding the first annotation does not increase the local objective \([5](https://arxiv.org/html/2610.10974#S5.E5)\)\.

###### Theorem 3\(First Annotation Threshold\)\.

The first annotation, corresponding to the transitionn=0n=0ton=1n=1, does not increase the local objective \([5](https://arxiv.org/html/2610.10974#S5.E5)\) if and only if

ε2​\(s,a\)≤η⁡\(s,a\)​\(\(nf​\(s,a\)\+1\)​σ2​\(s,a\)nf​\(s,a\)−Δ⁡\(s,a\)\),\\varepsilon^\{2\}\(s,a\)\\leq\\eta\(s,a\)\\\!\\left\(\\frac\{\(n\_\{f\}\(s,a\)\+1\)\\sigma^\{2\}\(s,a\)\}\{n\_\{f\}\(s,a\)\}\-\\Delta\(s,a\)\\right\),\(6\)whereη⁡\(s,a\)=1−πb​\(a∣s\)\\eta\(s,a\)=1\-\\pi\_\{b\}\(a\\mid s\)\. In particular, ifΔ⁡\(s,a\)≥\(nf​\(s,a\)\+1\)​σ2/nf​\(s,a\)\\Delta\(s,a\)\\geq\(n\_\{f\}\(s,a\)\+1\)\\sigma^\{2\}/n\_\{f\}\(s,a\), thenδ​ϕ​\(0\)\>0\\delta\\phi\(0\)\>0for every \(s,a\) withε⁡\(s,a\)≠0\\varepsilon\(s,a\)\\neq 0\.

The threshold explains why the same source can be useful at one pair but not another: its bias is judged against the variance reduction available at that pair\. Greater factual coverage lowers the factor\(nf\+1\)/nf\(n\_\{f\}\+1\)/n\_\{f\}, while greater excess variance reduces the remaining variance benefit; both make the first annotation harder to justify\. Appendix[B\.3](https://arxiv.org/html/2610.10974#A2.SS3)gives the proof\.

### 5\.2Four Regimes of Annotation Value

The first\-annotation threshold compares onlyϕ⁡\(1\)\\phi\(1\)withϕ⁡\(0\)\\phi\(0\)\. To understand how annotation value evolves as more annotations are added, we now characterize the behavior of the local objectiveϕ⁡\(n\)\\phi\(n\)over all annotation counts\. Althoughnnis integer\-valued in the allocation problem, treating it as continuous reveals the marginal structure ofϕ⁡\(n\)\\phi\(n\)\. Differentiating \([5](https://arxiv.org/html/2610.10974#S5.E5)\) givesϕ′​\(n\)=\(𝔞​n\+𝔟\)/\(nf\+n\)3\\phi^\{\\prime\}\(n\)=\(\\mathfrak\{a\}\\,n\+\\mathfrak\{b\}\)/\(n\_\{f\}\+n\)^\{3\}, where𝔞:=2​α1​ε2​nf−α3​σg2\\mathfrak\{a\}:=2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}and𝔟:=α3​nf​\(Δ−σ2\)\\mathfrak\{b\}:=\\alpha\_\{3\}n\_\{f\}\(\\Delta\-\\sigma^\{2\}\)\. Thus, the sign ofϕ′​\(n\)\\phi^\{\\prime\}\(n\)is determined by the affine term𝔞​n\+𝔟\\mathfrak\{a\}n\+\\mathfrak\{b\}, yielding four possible regimes\.

###### Theorem 4\(Four Regimes of Annotation Value\)\.

In all cases,ϕ⁡\(n\)→α1​ε2\\phi\(n\)\\to\\alpha\_\{1\}\\varepsilon^\{2\}asn→∞n\\to\\infty\. When𝔞=𝔟=0\\mathfrak\{a\}=\\mathfrak\{b\}=0,ϕ\\phiis constant; otherwise, it falls into four regimes:

1. \(I\)If𝔞≥0\\mathfrak\{a\}\\geq 0and𝔟≥0\\mathfrak\{b\}\\geq 0, thenϕ\\phiis increasing, so annotations are locally harmful\.
2. \(II\)If𝔞\>0\\mathfrak\{a\}\>0and𝔟<0\\mathfrak\{b\}<0, thenϕ\\phidecreases and then increases, with unique continuous minimizern∗=η​nf​\(σ2−Δ\)/\(2​ε2​nf−η​σg2\)n^\{\*\}=\\eta n\_\{f\}\(\\sigma^\{2\}\-\\Delta\)/\(2\\varepsilon^\{2\}n\_\{f\}\-\\eta\\sigma\_\{g\}^\{2\}\)\.
3. \(III\)If𝔞≤0\\mathfrak\{a\}\\leq 0and𝔟≤0\\mathfrak\{b\}\\leq 0, thenϕ\\phiis decreasing, so each annotation locally reduces the objective\.
4. \(IV\)If𝔞<0\\mathfrak\{a\}<0and𝔟\>0\\mathfrak\{b\}\>0, thenϕ\\phifirst increases and then decreases, with unique continuous maximizern¯=η​nf​\(Δ−σ2\)/\(η​σg2−2​ε2​nf\)\\bar\{n\}=\\eta n\_\{f\}\(\\Delta\-\\sigma^\{2\}\)/\(\\eta\\sigma\_\{g\}^\{2\}\-2\\varepsilon^\{2\}n\_\{f\}\)\. In this regime, a strict improvement over no annotation is possible if and only ifε2<η​σ2/nf\\varepsilon^\{2\}<\\eta\\sigma^\{2\}/n\_\{f\}\.

Regime \(II\) favors a finite count, so spending the entire available budget on one pair can be counterproductive\. In Regime \(IV\), a sufficiently large batch improves on no annotation only if the limiting bias contribution satisfiesα1​ε2<ϕ⁡\(0\)\\alpha\_\{1\}\\varepsilon^\{2\}<\\phi\(0\)\(IVa\); otherwise,n=0n=0remains best \(IVb\)\. Table[1](https://arxiv.org/html/2610.10974#S5.T1)translates these distinctions into local allocation decisions, subject to the available budget and capacities; Appendix[B\.4](https://arxiv.org/html/2610.10974#A2.SS4)gives the proof\.

Table 1:Conditions and local allocation decisions forη\>0\\eta\>0, excluding the constant\-objective case\.
### 5\.3Source Selection and Bias Cancellation

The local regimes reveal how access to multiple sources can improve allocation\. Because every single\-source plan remains feasible under the same budget and capacities, the multi\-source optimum is no worse than the best single\-source optimum\. To identify where source selection can help, we apply the first\-annotation threshold \([6](https://arxiv.org/html/2610.10974#S5.E6)\) to sourcekk:

εgk​\(s,a\)2≤η⁡\(s,a\)​\(\(nf​\(s,a\)\+1\)​σr2​\(s,a\)nf​\(s,a\)−Δgk​\(s,a\)\)\.\\varepsilon\_\{g\_\{k\}\}\(s,a\)^\{2\}\\;\\leq\\;\\eta\(s,a\)\\\!\\left\(\\frac\{\(n\_\{f\}\(s,a\)\+1\)\\,\\sigma\_\{r\}^\{2\}\(s,a\)\}\{n\_\{f\}\(s,a\)\}\-\\Delta\_\{g\_\{k\}\}\(s,a\)\\right\)\.\(7\)Because factual coverage and source quality vary across pairs, condition \([7](https://arxiv.org/html/2610.10974#S5.E7)\) can hold for different sources at different pairs\.

Sources can also complement one another within a pair\. At a fixed total annotation count, mixing two sources with opposite\-signed biases can reduce the magnitude of the reward\-model biasB⁡\(s,a,𝐧\)=∑knk​\(s,a\)​εgk​\(s,a\)/N⁡\(s,a\)B\(s,a;\\mathbf\{n\}\)=\\sum\_\{k\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)/N\(s,a\)below that obtained from either source alone\. The allocation must balance this reduction against source variances and costs\. When one source performs well across pairs and mixing offers little bias cancellation, gains over the best single source can be modest\.

### 5\.4From Local Thresholds to the Full Allocation Problem

The benefits of source selection and mixing must be evaluated through the full objective, which includes cross\-action coupling throughJ2J\_\{2\}\. Forα3\>0\\alpha\_\{3\}\>0and zero aggregate reward\-model bias in contextss, includingJ2J\_\{2\}changes the first\-annotation threshold toε2≤\(nf\+1\)​σ2/nf−Δ\\varepsilon^\{2\}\\leq\(n\_\{f\}\+1\)\\sigma^\{2\}/n\_\{f\}\-\\Delta, removing the factorη\\etafrom Theorem[3](https://arxiv.org/html/2610.10974#Thmtheorem3)and permitting larger admissible bias when the right\-hand side is positive\. More generally, the marginal effect ofJ2J\_\{2\}depends on the effective biases of all actions in the same context, while source mixing can reduceB⁡\(s,a,𝐧\)B\(s,a;\\mathbf\{n\}\)via bias cancellation but reshapesℬs​\(𝐧\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\. Together with the shared budget, this coupling motivates the joint allocation method in §[6](https://arxiv.org/html/2610.10974#S6)\.

## 6Budget\-Constrained Multi\-Source Allocation

We now solve the multi\-source allocation problem \([3](https://arxiv.org/html/2610.10974#S4.E3)\)\. Proposition[1](https://arxiv.org/html/2610.10974#Thmproposition1)formalizes why the local rules from §[5](https://arxiv.org/html/2610.10974#S5)cannot be applied independently: the problem has cross\-action coupling, nonconvexity, and integer constraints\.

###### Proposition 1\(Structure of the allocation objective\)\.

The objectiveJ⁡\(𝐧\)J\(\\mathbf\{n\}\)in problem \([3](https://arxiv.org/html/2610.10974#S4.E3)\) is nonnegative and bounded over all feasible allocations\. Moreover, there exist valid instances for which:

1. \(I\)JJis non\-separable across actions within the same context\.
2. \(II\)The continuous relaxation ofJJis neither convex nor concave\.
3. \(III\)JJis neither submodular nor supermodular over annotation allocations\.

An annotation’s full value depends on the other allocations in its context, so independent pairwise decisions need not produce a good joint allocation\. We use majorization\-minimization \(MM\)\[[29](https://arxiv.org/html/2610.10974#bib.bib29)\]to replace the cross\-action coupling by a locally tight upper surrogate while retaining the exact per\-pair bias\-variance structure\. Feasible surrogate\-improving updates then guarantee monotone descent\. Appendices[C](https://arxiv.org/html/2610.10974#A3)and[D\.2](https://arxiv.org/html/2610.10974#A4.SS2)give the structural proofs and descent analysis, respectively\.

### 6\.1Majorization\-Minimization Surrogate

We optimize the objectiveJJusing an iterative MM scheme\. Starting from a feasible allocation𝐧\(0\)\\mathbf\{n\}^\{\(0\)\}\(zero or a greedy allocation\), each MM iteration constructs an upper bound around𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}and seeks a feasible update that improves this surrogate\. The term−J2\-J\_\{2\}is the only part ofJJthat couples actions within the same context throughℬs​\(𝐧\)2\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\}\. We majorize this coupling term, turning it into an affine function ofℬs​\(𝐧\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)while leavingJ1J\_\{1\}andJ3J\_\{3\}unchanged\. The following proposition shows that this replacement yields an upper bound that is exact at the current allocation\.

###### Proposition 2\(Tight majorization ofJJ\)\.

Fix a feasible allocation𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}at iterationttand letℬs\(t\):=ℬs​\(𝐧\(t\)\)\\mathcal\{B\}\_\{s\}^\{\(t\)\}:=\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\. DefineJ~full​\(𝐧,𝐧\(t\)\)=J1​\(𝐧\)\+J3​\(𝐧\)\+∑s∈𝒮d0​\(s\)​\[\(ℬs\(t\)\)2−2​ℬs\(t\)​ℬs​\(𝐧\)\]\.\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)=J\_\{1\}\(\\mathbf\{n\}\)\+J\_\{3\}\(\\mathbf\{n\}\)\+\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\bigl\[\(\\mathcal\{B\}\_\{s\}^\{\(t\)\}\)^\{2\}\-2\\mathcal\{B\}\_\{s\}^\{\(t\)\}\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\\bigr\]\.ThenJ~full\\tilde\{J\}\_\{\\mathrm\{full\}\}is a tight upper bound onJJ:J~full​\(𝐧,𝐧\(t\)\)−J⁡\(𝐧\)=∑s∈𝒮d0​\(s\)​\(ℬs​\(𝐧\)−ℬs\(t\)\)2≥0\.\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\-J\(\\mathbf\{n\}\)=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\-\\mathcal\{B\}\_\{s\}^\{\(t\)\}\\bigr\)^\{2\}\\geq 0\.In particular,J~full​\(𝐧\(t\),𝐧\(t\)\)=J⁡\(𝐧\(t\)\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)=J\(\\mathbf\{n\}^\{\(t\)\}\)\.

Tightness at the current allocation ensures that a feasible update improving the surrogate cannot increaseJJ\. For computation, we drop the allocation\-independent constant∑sd0​\(s\)​\(ℬs\(t\)\)2\\sum\_\{s\}d\_\{0\}\(s\)\(\\mathcal\{B\}\_\{s\}^\{\(t\)\}\)^\{2\}, obtaining the equivalent surrogate

J~​\(𝐧,𝐧\(t\)\)=J1​\(𝐧\)\+J3​\(𝐧\)−2​∑s∈𝒮d0​\(s\)​ℬs\(t\)​ℬs​\(𝐧\)\.\\displaystyle\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)=J\_\{1\}\(\\mathbf\{n\}\)\+J\_\{3\}\(\\mathbf\{n\}\)\-2\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\mathcal\{B\}\_\{s\}^\{\(t\)\}\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\.\(8\)Substitutingℬs​\(𝐧\)=∑aπe​\(a\|s\)​B​\(s,a,𝐧\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)=\\sum\_\{a\}\\pi\_\{e\}\(a\|s\)B\(s,a;\\mathbf\{n\}\)into \([8](https://arxiv.org/html/2610.10974#S6.E8)\) shows thatJ~\\tilde\{J\}is additively separable in\(s,a\)\(s,a\)\. Each iteration therefore searches for an improving allocation using these separable costs under the original constraints, preserving descent inJJ\. Appendix[D\.2](https://arxiv.org/html/2610.10974#A4.SS2)gives the majorization proof\.

### 6\.2Solving the Surrogate Allocation Problem

We now describe the computational step at a fixed MM iterationtt\. We writeJ~​\(𝐧,𝐧\(t\)\)=∑s,aJ~s,a\(t\)​\(𝐧s,a\)\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)=\\sum\_\{s,a\}\\tilde\{J\}\_\{s,a\}^\{\(t\)\}\(\\mathbf\{n\}\_\{s,a\}\), whereJ~s,a\(t\)\\tilde\{J\}\_\{s,a\}^\{\(t\)\}denotes the summand ofJ~\\tilde\{J\}specific to the context\-action pair\(s,a\)\(s,a\), and𝐧s,a:=\(n1​\(s,a\),…,nK​\(s,a\)\)\\mathbf\{n\}\_\{s,a\}:=\(n\_\{1\}\(s,a\),\\ldots,n\_\{K\}\(s,a\)\)is the source\-count vector\. The remaining challenge is to handle the shared budget and capacity constraints\.

LetC⁡\(𝐧\):=∑s,a,kck​nk​\(s,a\)C\(\\mathbf\{n\}\):=\\sum\_\{s,a,k\}c\_\{k\}n\_\{k\}\(s,a\)denote the total annotation cost\. For a fixed multiplierλ≥0\\lambda\\geq 0, we minimize the penalized surrogateJ~​\(𝐧,𝐧\(t\)\)\+λ​C​\(𝐧\)\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\+\\lambda C\(\\mathbf\{n\}\), while retaining the per\-context and context\-action capacity constraints\. Thus, the penalized problem decomposes by context\. For each context\-action pair\(s,a\)\(s,a\)and each feasible annotation countm∈\{0,…,Ns−nf​\(s,a\)\}m\\in\\\{0,\\ldots,N\_\{s\}\-n\_\{f\}\(s,a\)\\\}, defineHs,a\(t,λ\)\(m\):=min𝐳∈ℤ\+K:1⊤​𝐳=m\{J~s,a\(t\)\(𝐳\)\+λ∑k=1Kckzk\}\.H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\):=\\min\_\{\\mathbf\{z\}\\in\\mathbb\{Z\}\_\{\+\}^\{K\}:\\,\\mathbf\{1\}^\{\\top\}\\mathbf\{z\}=m\}\\bigl\\\{\\tilde\{J\}\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\}\)\+\\lambda\\sum\_\{k=1\}^\{K\}c\_\{k\}z\_\{k\}\\bigr\\\}\.Here,𝐳=\(z1,…,zK\)\\mathbf\{z\}=\(z\_\{1\},\\ldots,z\_\{K\}\)specifies how themmannotations for\(s,a\)\(s,a\)are split across sources, and the local surrogate is evaluated withN⁡\(s,a\)=nf​\(s,a\)\+mN\(s,a\)=n\_\{f\}\(s,a\)\+m\. Thus,Hs,a\(t,λ\)​\(m\)H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\)is the best penalized surrogate value among all source mixtures that assign exactlymmannotations to\(s,a\)\(s,a\)\. Exact enumeration over all source\-count vectors𝐳\\mathbf\{z\}yields a certified solution forHs,a\(t,λ\)​\(m\)H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\), but the cost grows combinatorially withKK\. We instead obtain approximate valuesHs,a♯​\(m,λ\)H^\{\\sharp\}\_\{s,a\}\(m,\\lambda\)by pairwise\-exchange local search, which reallocates one annotation between sources at a time to reduce the penalized surrogate while keepingmmfixed\. Proposition[9](https://arxiv.org/html/2610.10974#Thmproposition9)in Appendix[D\.3](https://arxiv.org/html/2610.10974#A4.SS3)bounds the resulting approximation error\.

GivenHs,a\(t,λ\)​\(m\)H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\), it remains to choose how many annotations each context\-action pair receives within contextss\. Letms,a:=∑k=1Knk​\(s,a\)m\_\{s,a\}:=\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)denote the total number of annotations assigned to\(s,a\)\(s,a\), aggregated over sources\. For each context, the remaining problem is

min⁡∑a∈𝒜\{ms,a\}a∈𝒜⁡Hs,a\(t,λ\)​\(ms,a\)s\.t\.∑a∈𝒜ms,a≤Ns,ms,a∈\{0,…,Ns−nf​\(s,a\)\},∀a∈𝒜\.\\displaystyle\\min\_\{\\\{m\_\{s,a\}\\\}\_\{a\\in\\mathcal\{A\}\}\}\\sum\_\{a\\in\\mathcal\{A\}\}H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\_\{s,a\}\)\\quad\\text\{s\.t\.\}\\quad\\sum\_\{a\\in\\mathcal\{A\}\}m\_\{s,a\}\\leq N\_\{s\},\\quad m\_\{s,a\}\\in\\\{0,\\ldots,N\_\{s\}\-n\_\{f\}\(s,a\)\\\},~\\forall a\\in\\mathcal\{A\}\.We solve the aggregation by dynamic programming \(DP\), usingH♯H^\{\\sharp\}in place ofHHand allowing unused context capacity\. DP is exact for these tables; the source mixtures remain approximate\.

We search overλ\\lambdaby bisection\. Each trial value ofλ\\lambdareturns a candidate allocation from the penalized problem above, computed by local source search followed by DP across actions within each context\. Among the candidates satisfyingC⁡\(𝐧\)≤κC\(\\mathbf\{n\}\)\\leq\\kappa, together with the current allocation𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}, we select the one with the smallest surrogate valueJ~​\(⋅,𝐧\(t\)\)\\tilde\{J\}\(\\cdot;\\mathbf\{n\}^\{\(t\)\}\)\. Because𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}is included as a fallback, the selected update𝐧\(t\+1\)\\mathbf\{n\}^\{\(t\+1\)\}satisfiesJ~​\(𝐧\(t\+1\),𝐧\(t\)\)≤J~​\(𝐧\(t\),𝐧\(t\)\)\\tilde\{J\}\(\\mathbf\{n\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)\\leq\\tilde\{J\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)and henceJ⁡\(𝐧\(t\+1\)\)≤J⁡\(𝐧\(t\)\)J\(\\mathbf\{n\}^\{\(t\+1\)\}\)\\leq J\(\\mathbf\{n\}^\{\(t\)\}\)\. We assess allocation quality by benchmarking against exact solvers\. The evaluation of the empirical optimality gaps is detailed in §[7](https://arxiv.org/html/2610.10974#S7)\. We next bound the computational cost of an MM update\.

###### Proposition 3\(Runtime per MM update\)\.

LetTBisT\_\{\\mathrm\{Bis\}\}denote the total number of multiplier trials and letTswap≥1T\_\{\\mathrm\{swap\}\}\\geq 1bound the number of annotation reassignments per local search\. One MM update costs𝒪⁡\(TBis​\|𝒮\|​\|𝒜\|​\(K2​Nmax​Tswap\+Nmax2\)\)\\mathcal\{O\}\\\!\\bigl\(T\_\{\\mathrm\{Bis\}\}\\,\|\\mathcal\{S\}\|\\,\|\\mathcal\{A\}\|\\,\(K^\{2\}N\_\{\\max\}\\,T\_\{\\mathrm\{swap\}\}\+N\_\{\\max\}^\{2\}\)\\bigr\), whereNmax:=maxs⁡Ns≥1N\_\{\\max\}:=\\max\_\{s\}N\_\{s\}\\geq 1\.

The termsK2​Nmax​TswapK^\{2\}N\_\{\\max\}T\_\{\\mathrm\{swap\}\}andNmax2N\_\{\\max\}^\{2\}correspond to local source search and DP aggregation, respectively\. With the other parameters fixed, the runtime bound grows linearly with the number of context\-action pairs\. Appendices[D\.7](https://arxiv.org/html/2610.10974#A4.SS7)and[D\.1](https://arxiv.org/html/2610.10974#A4.SS1)give the proof and pseudocode\.

## 7Numerical Experiments

We evaluate1\)how source quality affects annotation value,2\)how MM compares with greedy and solver benchmarks, and3\)how profile\-estimation error affects final OPE performance\. For comparisons with fixed source profiles, we report scaled total MSE of the estimator \([1](https://arxiv.org/html/2610.10974#S3.E1)\),𝒱⁡\(𝐧\)=N​MSE​\(𝐧\)=N​C0\+J⁡\(𝐧\)\\mathcal\{V\}\(\\mathbf\{n\}\)=N\\,\\mathrm\{MSE\}\(\\mathbf\{n\}\)=NC\_\{0\}\+J\(\\mathbf\{n\}\)\. Full experimental details are deferred to Appendix[E](https://arxiv.org/html/2610.10974#A5)\.

### 7\.1Annotation Value and Synthetic Validation

We construct a synthetic clinical bandit with100100contexts,55actions,M=1,000M=1\{,\}000fitting records, and budgetκ=10,000\\kappa=10\{,\}000\. Detailed environment settings are in Appendix[E\.1](https://arxiv.org/html/2610.10974#A5.SS1)\.

To illustrate the four regimes in Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4), we construct four sources combining low or high bias with low or high excess variance \(Table[2](https://arxiv.org/html/2610.10974#S7.T2)\)\. For each sourcekkand pair\(s,a\)\(s,a\), we independently draw bias and excess variance from normal distributions parameterized byεk\\varepsilon\_\{k\}andΔk\\Delta\_\{k\}, respectively\.

Table 2:Configuration of annotation sources\.We activate one source at a time and randomly allocate increasing numbers of its annotations to examine how scaled MSE changes\. We average over100100independently sampled environments\.

Figure[1](https://arxiv.org/html/2610.10974#S7.F1)matches the predicted regimes: Source 1 decreases monotonically \(Regime III\), Source 2 first increases and then decreases \(Regime IV\), Source 3 decreases and then increases \(Regime II\), and Source 4 increases monotonically \(Regime I\)\. The decomposition shows how variance reduction \(J3J\_\{3\}\) competes with bias accumulation \(J1J\_\{1\}\) and correction \(J2J\_\{2\}\)\.

Figure 1:Scaled mean squared error \(MSE\)𝒱\\mathcal\{V\}and the allocation\-dependent componentsJ1J\_\{1\},J2J\_\{2\}, andJ3J\_\{3\}versus the number of annotations for the four annotation sources in Table[2](https://arxiv.org/html/2610.10974#S7.T2)\. Shaded bands show±1\\pm 1standard deviation over100100environments\.We next examine how the set of available annotation sources affects allocation quality\. We evaluate all2K−1=152^\{K\}\-1=15nonempty source subsets𝒦sub⊆\{1,2,3,4\}\\mathcal\{K\}\_\{\\mathrm\{sub\}\}\\subseteq\\\{1,2,3,4\\\}from Table[2](https://arxiv.org/html/2610.10974#S7.T2)\. For multi\-source subsets, we run the proposed MM algorithm in §[6](https://arxiv.org/html/2610.10974#S6)restricted to the active sources\. For single\-source subsets, we use a uniformly random allocation \(assigning annotations uniformly across eligible logged samples and counterfactual actions until the budget or per\-sample capacity is reached\) as a non\-optimized baseline; the corresponding optimized single\-source results are reported in Appendix[E\.1\.2](https://arxiv.org/html/2610.10974#A5.SS1.SSS2)\. All runs use the same environment, factual data, and budgetκ=10,000\\kappa=10\{,\}000\.

Figure[2](https://arxiv.org/html/2610.10974#S7.F2)shows that the four\-source setting achieves the lowest scaled MSE among the evaluated subsets, illustrating the potential benefit of source selection and mixing discussed in §[5\.3](https://arxiv.org/html/2610.10974#S5.SS3)\. With fixed source profiles, the four\-source MM allocation reduces scaled MSE from𝒱⁡\(𝟎\)=84\.23\\mathcal\{V\}\(\\mathbf\{0\}\)=84\.23to66\.9066\.90, corresponding to a20\.58%20\.58\\%MSE reduction relative to no annotation\. Full convergence traces are provided in Appendix[E\.1\.3](https://arxiv.org/html/2610.10974#A5.SS1.SSS3)\.

Figure 2:Scaled mean squared error \(MSE\)𝒱⁡\(𝐧\)\\mathcal\{V\}\(\\mathbf\{n\}\)for each active source subset\. Bars are grouped by the number of available sources; the dashed line is the no\-annotation baseline𝒱⁡\(𝟎\)=84\.23\\mathcal\{V\}\(\\mathbf\{0\}\)=84\.23\.
### 7\.2Baselines and Solver Benchmarks

To compare allocation methods with realistic annotation sources, we also construct a semi\-synthetic education bandit from6\.16\.1million ASSISTments student–problem interactions\[[5](https://arxiv.org/html/2610.10974#bib.bib5)\]\. Contexts are\|𝒮\|=100\|\\mathcal\{S\}\|=100K\-means clusters of student answer history and affect features, including past correct rate, hint usage, and detector confidences for frustration, confusion, concentration, and boredom\. Actions are the\|𝒜\|=10\|\\mathcal\{A\}\|=10problem\_type×\\timesoriginalcombinations\.

Following the feature\-prediction approach of APRIL\[[4](https://arxiv.org/html/2610.10974#bib.bib4)\], each LLM predicts the student’s first\-response timett, which is converted to reward throughR⁡\(s,a,t\)=Rmax⋅PIRT​\(s,a\)⋅TimeEff⁡\(t,t~a\)R\(s,a,t\)=R\_\{\\max\}\\cdot P\_\{\\mathrm\{IRT\}\}\(s,a\)\\cdot\\mathrm\{TimeEff\}\(t,\\tilde\{t\}\_\{a\}\)\. Here,Rmax=30R\_\{\\max\}=30andPIRT​\(s,a\)=σ⁡\(θs−ba\)P\_\{\\mathrm\{IRT\}\}\(s,a\)=\\sigma\(\\theta\_\{s\}\-b\_\{a\}\)is a one\-parameter logistic item response theory \(IRT\) model\[[3](https://arxiv.org/html/2610.10974#bib.bib3)\], with context\-level abilityθs\\theta\_\{s\}and action difficultybab\_\{a\}\. The factorTimeEff⁡\(t,t~a\)=max⁡\{0\.3,1−0\.4​\|log10⁡\(t/t~a\)\|\}\\mathrm\{TimeEff\}\(t,\\tilde\{t\}\_\{a\}\)=\\max\\\{0\.3,\\,1\-0\.4\|\\log\_\{10\}\(t/\\tilde\{t\}\_\{a\}\)\|\\\}discounts responses that are much faster or slower than actionaa’s empirical median response timet~a\\tilde\{t\}\_\{a\}\.

We use Gemini\-2\.5\-Flash, GPT\-4o\-mini, and Claude\-Haiku\-4\.5 as annotation sources, collecting2020predictions per source and\(s,a\)\(s,a\)to estimateεgk​\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)andΔgk​\(s,a\)\\Delta\_\{g\_\{k\}\}\(s,a\)\. Appendix[E\.2](https://arxiv.org/html/2610.10974#A5.SS2)details the data processing, reward construction, LLM prompts, and MM algorithm results\.

For the baselines, Greedy \(max\-gain\) repeatedly selects the feasible annotation with largest positive marginal gain, while Greedy \(cost\-aware\) uses gain per unit cost\. We compare these methods with zero\-initialized MM and MM initialized from the better greedy allocation on the ASSISTments, original synthetic, and I/IV\-focused synthetic environments\. The I/IV\-focused environment tests allocation with locally harmful or initially harmful sources, for which single\-annotation gains can be uninformative\. Within each environment, all methods use the same budget, capacities, source costs, and source profiles\.

Table 3:Scaled MSE of allocations from MM and greedy baselines\.Table[3](https://arxiv.org/html/2610.10974#S7.T3)shows that MM or greedy\-initialized MM achieves the lowest MSE in each environment, with reductions of10\.77%10\.77\\%,20\.68%20\.68\\%, and16\.11%16\.11\\%relative to no annotation\. On the I/IV\-focused instance, MM improves on both greedy baselines, illustrating the value of jointly allocating annotations beyond single\-step gains\.

To assess how close these allocations are to a global optimum, we construct3030small instances with55contexts,33actions, and2525logged samples\. For each instance, we enumerate the feasible context\-level choices and use Gurobi to solve the resulting multiple\-choice knapsack mixed\-integer linear program to certified global optimality\. Table[4](https://arxiv.org/html/2610.10974#S7.T4)shows that greedy\-initialized MM has the lowest mean optimality gap\. On the full Synthetic and ASSISTments instances, it reaches within0\.2807%0\.2807\\%and0\.0050%0\.0050\\%of the Gurobi lower bounds, respectively \(Appendix[E\.3](https://arxiv.org/html/2610.10974#A5.SS3)\)\.

Table 4:Relative MSE gap to the certified small\-instance optima\.
### 7\.3Robustness to Finite\-Pilot Profile Estimation

The preceding comparisons use fixed source profiles; we now test whether the gains persist when profiles are estimated from finite pilots\. Across3030pilot replications, factual moments are estimated from2020pilot rewards per pair and annotation moments from2020pilot draws per source–pair\. The pilot, reward\-model fitting data, acquired annotations, and OPE evaluation data are mutually independent\. After selecting an allocation, we evaluate DM\+\-IS using5050independent final estimates per pilot replication\. Greedy\-initialized MM performs best, reducing empirical OPE MSE by19\.13%19\.13\\%on Synthetic and8\.47%8\.47\\%on ASSISTments relative to no annotation\.

As a complementary sensitivity check, we perform3030within\-pair bootstraps of the2020observed LLM predictions and rerun MM after re\-estimating every source profile\. On ASSISTments, the resulting mean reference\-profile scaled\-MSE reduction is10\.765%10\.765\\%, compared with10\.774%10\.774\\%using the original profiles, with a standard deviation of only0\.00380\.0038percentage points\. The bootstrap indicates stable allocation quality under resampling of the observed LLM outputs, complementing the gains in final OPE MSE measured with independent evaluation data\. Full protocols and baseline results are reported in Appendices[E\.4](https://arxiv.org/html/2610.10974#A5.SS4)and[E\.5](https://arxiv.org/html/2610.10974#A5.SS5)\.

## 8Conclusion

We formulated budgeted counterfactual annotation for contextual\-bandit off\-policy evaluation as an integer allocation problem minimizing the MSE of theDM\+​\-​IS\\mathrm\{DM\}^\{\+\}\\text\{\-\}\\mathrm\{IS\}estimator\. The objective separates effective annotation bias, cross\-action bias correction, and reward/annotation variance, explaining why annotation sources are not interchangeable\. For a single context\-action pair and source, we derived a first\-annotation threshold and four annotation\-value regimes\. For multiple sources, we established cross\-action coupling, nonconvexity, and lack of submodularity, and developed a majorization\-minimization algorithm with dynamic\-programming subroutines that monotonically improves the objective\. These results guide the purchase and allocation of counterfactual annotations under a limited budget to improve policy evaluation\.

Limitations and future work\.Our framework is limited to contextual bandits and uses known or estimated source profiles\(εgk,Δgk\)\(\\varepsilon\_\{g\_\{k\}\},\\Delta\_\{g\_\{k\}\}\)\. The ASSISTments experiment uses real LLM annotations in a semi\-synthetic environment\. Future work includes sequential Markov decision processes, online estimation of source profiles, and human annotations in fully real\-world experiments\.

## Acknowledgments and Disclosure of Funding

The authors thank Aishwarya Mandyam for introducing the original problem that motivated this line of work\. This work was supported in part by the Johns Hopkins Carey Business School\. This work was supported by NSF IIS\-2403240, NSF IIS\-2552007, NIH R01HL184139, and Schmidt Sciences AI2050 Fellowship\. The authors declare no competing interests\.

## References

- \[1\]Shengpu Tang and Jenna Wiens\.Counterfactual\-augmented importance sampling for semi\-offline policy evaluation\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- \[2\]Aishwarya Mandyam, Shengpu Tang, Jiayu Yao, Jenna Wiens, and Barbara E\. Engelhardt\.CANDOR: Counterfactual annotated doubly robust off\-policy evaluation\.*arXiv preprint arXiv:2412\.08052*, 2024\.
- \[3\]Maria Orlando Edelen and Bryce B\. Reeve\.Applying item response theory \(IRT\) modeling to questionnaire development, evaluation, and refinement\.*Quality of Life Research*, 16\(Suppl\. 1\):5–18, 2007\.
- \[4\]Aishwarya Mandyam, Kalyani Limaye, Barbara E\. Engelhardt, and Emily Alsentzer\.APRIL: Annotations for policy evaluation with reliable inference from LLMs\.In*Machine Learning for Health*, volume 297 of*Proceedings of Machine Learning Research*, 2025\.
- \[5\]Mingyu Feng, Neil T\. Heffernan, and Kenneth R\. Koedinger\.Addressing the assessment challenge with an online system that tutors as it assesses\.*User Modeling and User\-Adapted Interaction*, 19\(3\):243–266, 2009\.
- \[6\]Yutao Wang, Neil T\. Heffernan, and Cristina Heffernan\.Towards better affect detectors: Effect of missing skills, class features and common wrong answers\.In*Proceedings of the Fifth International Conference on Learning Analytics and Knowledge*, pages 31–35\. ACM, 2015\.
- \[7\]Nan Jiang and Lihong Li\.Doubly robust off\-policy value evaluation for reinforcement learning\.In*Proceedings of the 33rd International Conference on Machine Learning*, volume 48 of*Proceedings of Machine Learning Research*, pages 652–661\. PMLR, 2016\.
- \[8\]Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims\.Effective evaluation using logged bandit feedback from multiple loggers\.In*Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 687–696, 2017\.
- \[9\]Anastasios N\. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I\. Jordan, and Tijana Zrnic\.Prediction\-powered inference\.*Science*, 382\(6671\):669–674, 2023\.
- \[10\]Lars Buesing, Théophane Weber, Yori Zwols, Sébastien Racanière, Arthur Guez, Jean\-Baptiste Lespiau, and Nicolas Heess\.Woulda, coulda, shoulda: Counterfactually\-guided policy search\.In*International Conference on Learning Representations*, 2019\.
- \[11\]Miroslav Dudík, John Langford, and Lihong Li\.Doubly robust policy evaluation and learning\.In*Proceedings of the 28th International Conference on Machine Learning*, pages 1097–1104, 2011\.
- \[12\]Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li\.Doubly robust policy evaluation and optimization\.*Statistical Science*, 29\(4\):485–511, 2014\.
- \[13\]Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh\.More robust doubly robust off\-policy evaluation\.In*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pages 1447–1456\. PMLR, 2018\.
- \[14\]Daniele Foffano, Alessio Russo, and Alexandre Proutière\.Conformal off\-policy evaluation in Markov decision processes\.In*Proceedings of the 62nd IEEE Conference on Decision and Control*, pages 3087–3094\. IEEE, 2023\.
- \[15\]Ge Gao, Qitong Gao, Xi Yang, Song Ju, Miroslav Pajić, and Min Chi\.On trajectory augmentations for off\-policy evaluation\.In*International Conference on Learning Representations*, 2024\.
- \[16\]Aishwarya Mandyam, Jason Meng, Ge Gao, Jiankai Sun, Mac Schwager, Barbara E\. Engelhardt, and Emma Brunskill\.PERRY: Policy evaluation with confidence intervals using auxiliary data\.*arXiv preprint arXiv:2507\.20068*, 2025\.
- \[17\]Allen Nie, Yash Chandak, Christina J\. Yuan, Anirudhan Badrinath, Yannis Flet\-Berliac, and Emma Brunskill\.OPERA: Automatic offline policy evaluation with re\-weighted aggregates of multiple estimators\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- \[18\]Michael Oberst and David Sontag\.Counterfactual off\-policy evaluation with Gumbel\-Max structural causal models\.In*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 4881–4890\. PMLR, 2019\.
- \[19\]Doina Precup, Richard S\. Sutton, and Satinder Singh\.Eligibility traces for off\-policy policy evaluation\.In*Proceedings of the 17th International Conference on Machine Learning*, pages 759–766, 2000\.
- \[20\]Friedrich Pukelsheim\.*Optimal Design of Experiments*\.Society for Industrial and Applied Mathematics, 2006\.
- \[21\]Burr Settles\.Active learning literature survey\.Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009\.
- \[22\]Muhammad Faaiz Taufiq, Jean\-François Ton, Rob Cornish, Yee Whye Teh, and Arnaud Doucet\.Conformal off\-policy prediction in contextual bandits\.In*Advances in Neural Information Processing Systems*, volume 35, pages 31512–31524, 2022\.
- \[23\]Philip S\. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh\.High\-confidence off\-policy evaluation\.In*Proceedings of the Twenty\-Ninth AAAI Conference on Artificial Intelligence*, pages 3000–3006, 2015\.
- \[24\]Philip S\. Thomas and Emma Brunskill\.Data\-efficient off\-policy policy evaluation for reinforcement learning\.In*Proceedings of the 33rd International Conference on Machine Learning*, volume 48 of*Proceedings of Machine Learning Research*, pages 2139–2148\. PMLR, 2016\.
- \[25\]Alina Beygelzimer and John Langford\.The offset tree for learning with partial labels\.In*Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 129–138, 2009\.
- \[26\]Hoang M\. Le, Cameron Voloshin, and Yisong Yue\.Batch policy learning under constraints\.In*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 3703–3712\. PMLR, 2019\.
- \[27\]Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo A\. Faisal, Finale Doshi\-Velez, and Emma Brunskill\.Representation balancing MDPs for off\-policy policy evaluation\.In*Advances in Neural Information Processing Systems*, volume 31, pages 2649–2658, 2018\.
- \[28\]Cameron Voloshin, Hoang M\. Le, Nan Jiang, and Yisong Yue\.Empirical study of off\-policy policy evaluation for reinforcement learning\.In*Advances in Neural Information Processing Systems, Datasets and Benchmarks Track*, 2021\.
- \[29\]David R\. Hunter and Kenneth Lange\.A tutorial on MM algorithms\.*The American Statistician*, 58\(1\):30–37, 2004\.

## Appendix ADerivations for the Objective Function

This appendix provides the full, step\-by\-step derivations for the explicit forms of the variance decomposition described in the main text\. Throughout the appendices, we interpretπe​\(a\|s\)/πb​\(a\|s\)\\pi\_\{e\}\(a\|s\)/\\pi\_\{b\}\(a\|s\)andπe​\(a\|s\)2/πb​\(a\|s\)\\pi\_\{e\}\(a\|s\)^\{2\}/\\pi\_\{b\}\(a\|s\)as zero wheneverπe​\(a\|s\)=0\\pi\_\{e\}\(a\|s\)=0, consistent with the coefficient convention preceding \([2](https://arxiv.org/html/2610.10974#S4.E2)\)\.

Recall that variance of theDM\+​\-IS\\text\{DM\}^\{\+\}\\text\{\-IS\}estimator is decomposed as:

N​Var⁡\(v^DM\+​\-IS​\(πe\)\)\\displaystyle N\\operatorname\{Var\}\\left\(\\hat\{v\}^\{\\text\{DM\}^\{\+\}\\text\{\-IS\}\}\(\\pi\_\{e\}\)\\right\)=Vars∼d0⁡\(v⁡\(πe\|s\)\)\\displaystyle=\\operatorname\{Var\}\_\{s\\sim d\_\{0\}\}\(v\(\\pi\_\{e\}\|s\)\)\+𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[ρ\(s,a\)2σr2\(s,a\)\]\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\left\[\\rho\(s,a\)^\{2\}\\sigma\_\{r\}^\{2\}\(s,a\)\\right\]\\right\]\+𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[ρ\(s,a\)2ε^r^\+\(s,a\)2\]−ε^r^\+πe\(s\)2\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\left\[\\rho\(s,a\)^\{2\}\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)^\{2\}\\right\]\-\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}^\{\\pi\_\{e\}\}\(s\)^\{2\}\\right\]\+𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[\(ρ\(s,a\)2−πe​\(a\|s\)2πb​\(a\|s\)\)×Var𝒟\(r^\+\(s,a\)\)\]\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\Bigg\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\Bigg\[\\left\(\\rho\(s,a\)^\{2\}\-\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\\right\)\\times\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\\Bigg\]\\Bigg\]The total variance is decomposed as follows:

N​Var⁡\(v^DM\+​\-IS​\(πe\)\)=T1\+T2\+T3\+T4N\\operatorname\{Var\}\\left\(\\hat\{v\}^\{\\text\{DM\}^\{\+\}\\text\{\-IS\}\}\(\\pi\_\{e\}\)\\right\)=T\_\{1\}\+T\_\{2\}\+T\_\{3\}\+T\_\{4\}whereT1T\_\{1\}andT2T\_\{2\}can be seen as constant with respect to𝐧\\mathbf\{n\}\. We focus on deriving the explicit forms ofT3T\_\{3\}andT4T\_\{4\}\.

In addition, as proved in\[[2](https://arxiv.org/html/2610.10974#bib.bib2)\],v^DM\+​\-IS​\(πe\)\\hat\{v\}^\{\\text\{DM\}^\{\+\}\\text\{\-IS\}\}\(\\pi\_\{e\}\)remain unbiased even the annotations are biased\. Thus,

MSE⁡\(𝐧\)=0\+Var⁡\(v^DM\+​\-IS​\(πe\)\)=C0\+J⁡\(𝐧\)N,\\operatorname\{MSE\}\(\\mathbf\{n\}\)=0\+\\operatorname\{Var\}\\left\(\\hat\{v\}^\{\\text\{DM\}^\{\+\}\\text\{\-IS\}\}\(\\pi\_\{e\}\)\\right\)=C\_\{0\}\+\\frac\{J\(\\mathbf\{n\}\)\}\{N\},
whereC0=Vars∼d0\(v\(πe\|s\)\)\+𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[ρ\(s,a\)2σr2\(s,a\)\]\]NC\_\{0\}=\\frac\{\\operatorname\{Var\}\_\{s\\sim d\_\{0\}\}\(v\(\\pi\_\{e\}\|s\)\)\+\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\left\[\\rho\(s,a\)^\{2\}\\sigma\_\{r\}^\{2\}\(s,a\)\\right\]\\right\]\}\{N\}\.

### A\.1Term 3

The third term is given by:

T3=𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[ρ\(s,a\)2ε^r^\+\(s,a\)2\]−ε^r^\+πe\(s\)2\]T\_\{3\}=\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\left\[\\rho\(s,a\)^\{2\}\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)^\{2\}\\right\]\-\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}^\{\\pi\_\{e\}\}\(s\)^\{2\}\\right\]\(9\)
First, we derive the biasε^r^\+​\(s,a\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)for a pair withN⁡\(s,a\)\>0N\(s,a\)\>0\. We begin by calculating the expectation𝔼𝒟​\[r^\+​\(s,a\)\]\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\hat\{r\}^\{\+\}\(s,a\)\]:

𝔼𝒟​\[r^\+​\(s,a\)\]\\displaystyle\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\hat\{r\}^\{\+\}\(s,a\)\]=1N⁡\(s,a\)​\(∑i=1nf​\(s,a\)𝔼⁡\[ri\]\+∑k=1K∑j=1nk​\(s,a\)𝔼⁡\[gk,j\]\)\\displaystyle=\\frac\{1\}\{N\(s,a\)\}\\left\(\\sum\_\{i=1\}^\{n\_\{f\}\(s,a\)\}\\mathbb\{E\}\[r\_\{i\}\]\+\\sum\_\{k=1\}^\{K\}\\sum\_\{j=1\}^\{n\_\{k\}\(s,a\)\}\\mathbb\{E\}\[g\_\{k,j\}\]\\right\)=1N⁡\(s,a\)​\(nf​\(s,a\)​μr​\(s,a\)\+∑k=1Knk​\(s,a\)​\(μr​\(s,a\)\+εgk​\(s,a\)\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)\}\\left\(n\_\{f\}\(s,a\)\\mu\_\{r\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\(\\mu\_\{r\}\(s,a\)\+\\varepsilon\_\{g\_\{k\}\}\(s,a\)\)\\right\)=1N⁡\(s,a\)​\(nf​\(s,a\)​μr​\(s,a\)\+∑k=1Knk​\(s,a\)​μr​\(s,a\)\+∑k=1Knk​\(s,a\)​εgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)\}\\left\(n\_\{f\}\(s,a\)\\mu\_\{r\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\mu\_\{r\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\\right\)=1N⁡\(s,a\)​\(\(nf​\(s,a\)\+∑k=1Knk​\(s,a\)\)​μr​\(s,a\)\+∑k=1Knk​\(s,a\)​εgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)\}\\left\(\\left\(n\_\{f\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\right\)\\mu\_\{r\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\\right\)=1N⁡\(s,a\)​\(N⁡\(s,a\)​μr​\(s,a\)\+∑k=1Knk​\(s,a\)​εgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)\}\\left\(N\(s,a\)\\mu\_\{r\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\\right\)=μr​\(s,a\)\+∑k=1Knk​\(s,a\)​εgk​\(s,a\)N⁡\(s,a\)\.\\displaystyle=\\mu\_\{r\}\(s,a\)\+\\frac\{\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\}\{N\(s,a\)\}\.The bias termε^r^\+​\(s,a\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)is defined as:

ε^r^\+​\(s,a\)\\displaystyle\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\):=𝔼𝒟​\[r^\+​\(s,a\)\]−μr​\(s,a\)\\displaystyle:=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\hat\{r\}^\{\+\}\(s,a\)\]\-\\mu\_\{r\}\(s,a\)=\(μr​\(s,a\)\+∑k=1Knk​\(s,a\)​εgk​\(s,a\)N⁡\(s,a\)\)−μr​\(s,a\)\\displaystyle=\\left\(\\mu\_\{r\}\(s,a\)\+\\frac\{\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\}\{N\(s,a\)\}\\right\)\-\\mu\_\{r\}\(s,a\)=∑k=1Knk​\(s,a\)​εgk​\(s,a\)N⁡\(s,a\)\\displaystyle=\\frac\{\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\varepsilon\_\{g\_\{k\}\}\(s,a\)\}\{N\(s,a\)\}\(10\)IfN⁡\(s,a\)=0N\(s,a\)=0, the fixed default instead givesε^r^\+​\(s,a\)=r0​\(s,a\)−μr​\(s,a\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)=r\_\{0\}\(s,a\)\-\\mu\_\{r\}\(s,a\)\. Thusε^r^\+​\(s,a\)=B⁡\(s,a,𝐧\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)=B\(s,a;\\mathbf\{n\}\)in both cases, using the definition preceding \([2](https://arxiv.org/html/2610.10974#S4.E2)\)\. Next, we deriveε^r^\+πe​\(s\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}^\{\\pi\_\{e\}\}\(s\)by expanding the expectation over the target policyπe\\pi\_\{e\}:

ε^r^\+πe​\(s\)\\displaystyle\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}^\{\\pi\_\{e\}\}\(s\):=𝔼a∼πe\(⋅\|s\)\[ε^r^\+\(s,a\)\]\\displaystyle:=\\mathbb\{E\}\_\{a\\sim\\pi\_\{e\}\(\\cdot\|s\)\}\[\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)\]=∑a∈𝒜πe​\(a\|s\)​ε^r^\+​\(s,a\)\\displaystyle=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)=∑a∈𝒜πe​\(a\|s\)​B​\(s,a,𝐧\)\\displaystyle=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)B\(s,a;\\mathbf\{n\}\)\(11\)Substituting these bias expressions into \([9](https://arxiv.org/html/2610.10974#A1.E9)\) gives

T3\\displaystyle T\_\{3\}=∑s∈𝒮d0\(s\)\(𝔼a∼πb\(⋅\|s\)\[ρ\(s,a\)2ε^r^\+\(s,a\)2\]−\(ε^r^\+πe\(s\)\)2\)\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\left\(\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\left\[\\rho\(s,a\)^\{2\}\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)^\{2\}\\right\]\-\\left\(\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}^\{\\pi\_\{e\}\}\(s\)\\right\)^\{2\}\\right\)=∑s∈𝒮d0​\(s\)​\(∑a∈𝒜πb​\(a\|s\)​ρ​\(s,a\)2​B​\(s,a,𝐧\)2−\(∑a∈𝒜πe​\(a\|s\)​B​\(s,a,𝐧\)\)2\)\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\left\(\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{b\}\(a\|s\)\\rho\(s,a\)^\{2\}B\(s,a;\\mathbf\{n\}\)^\{2\}\-\\left\(\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)B\(s,a;\\mathbf\{n\}\)\\right\)^\{2\}\\right\)\(12\)We simplify the importance sampling weight termπb​\(a\|s\)​ρ​\(s,a\)2\\pi\_\{b\}\(a\|s\)\\rho\(s,a\)^\{2\}:

πb​\(a\|s\)​ρ​\(s,a\)2\\displaystyle\\pi\_\{b\}\(a\|s\)\\rho\(s,a\)^\{2\}=πb​\(a\|s\)​\(πe​\(a\|s\)πb​\(a\|s\)\)2=πe​\(a\|s\)2πb​\(a\|s\)\\displaystyle=\\pi\_\{b\}\(a\|s\)\\left\(\\frac\{\\pi\_\{e\}\(a\|s\)\}\{\\pi\_\{b\}\(a\|s\)\}\\right\)^\{2\}=\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}Substituting this back into \([12](https://arxiv.org/html/2610.10974#A1.E12)\):

T3\\displaystyle T\_\{3\}=∑s∈𝒮d0\(s\)\[∑a∈𝒜πe​\(a\|s\)2πb​\(a\|s\)B\(s,a;𝐧\)2\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\Bigg\[\\sum\_\{a\\in\\mathcal\{A\}\}\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}B\(s,a;\\mathbf\{n\}\)^\{2\}−\(∑a∈𝒜πe\(a\|s\)B\(s,a;𝐧\)\)2\]\\displaystyle\\quad\-\\left\(\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)B\(s,a;\\mathbf\{n\}\)\\right\)^\{2\}\\Bigg\]\(13\)

### A\.2Term 4

The fourth term is given by:

T4=𝔼s∼d0\[𝔼a∼πb\(⋅\|s\)\[\(ρ\(s,a\)2−πe​\(a\|s\)2πb​\(a\|s\)\)Var𝒟\(r^\+\(s,a\)\)\]\]T\_\{4\}=\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\Bigg\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\|s\)\}\\Bigg\[\\left\(\\rho\(s,a\)^\{2\}\-\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\\right\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\\Bigg\]\\Bigg\]\(14\)First, forN⁡\(s,a\)\>0N\(s,a\)\>0, we calculate the reward\-model varianceVar𝒟⁡\(r^\+​\(s,a\)\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\. Using the independence of samples:

Var𝒟⁡\(r^\+​\(s,a\)\)\\displaystyle\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)=Var𝒟⁡\(1N⁡\(s,a\)​\(∑i=1nf​\(s,a\)ri\+∑k=1K∑j=1nk​\(s,a\)gk,j\)\)\\displaystyle=\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\frac\{1\}\{N\(s,a\)\}\\left\(\\sum\_\{i=1\}^\{n\_\{f\}\(s,a\)\}r\_\{i\}\+\\sum\_\{k=1\}^\{K\}\\sum\_\{j=1\}^\{n\_\{k\}\(s,a\)\}g\_\{k,j\}\\right\)\\right\)=1N​\(s,a\)2​\(∑i=1nf​\(s,a\)Var⁡\(ri\)\+∑k=1K∑j=1nk​\(s,a\)Var⁡\(gk,j\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)^\{2\}\}\\left\(\\sum\_\{i=1\}^\{n\_\{f\}\(s,a\)\}\\operatorname\{Var\}\(r\_\{i\}\)\+\\sum\_\{k=1\}^\{K\}\\sum\_\{j=1\}^\{n\_\{k\}\(s,a\)\}\\operatorname\{Var\}\(g\_\{k,j\}\)\\right\)=1N​\(s,a\)2​\(nf​\(s,a\)​σr2​\(s,a\)\+∑k=1Knk​\(s,a\)​\(σr2​\(s,a\)\+Δgk​\(s,a\)\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)^\{2\}\}\\left\(n\_\{f\}\(s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\(\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{g\_\{k\}\}\(s,a\)\)\\right\)=1N​\(s,a\)2​\(nf​\(s,a\)​σr2​\(s,a\)\+∑k=1Knk​\(s,a\)​σr2​\(s,a\)\+∑k=1Knk​\(s,a\)​Δgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)^\{2\}\}\\left\(n\_\{f\}\(s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\Delta\_\{g\_\{k\}\}\(s,a\)\\right\)=1N​\(s,a\)2​\(\(nf​\(s,a\)\+∑k=1Knk​\(s,a\)\)​σr2​\(s,a\)\+∑k=1Knk​\(s,a\)​Δgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)^\{2\}\}\\left\(\\left\(n\_\{f\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\right\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\Delta\_\{g\_\{k\}\}\(s,a\)\\right\)=1N​\(s,a\)2​\(N⁡\(s,a\)​σr2​\(s,a\)\+∑k=1Knk​\(s,a\)​Δgk​\(s,a\)\)\\displaystyle=\\frac\{1\}\{N\(s,a\)^\{2\}\}\\left\(N\(s,a\)\\sigma\_\{r\}^\{2\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\Delta\_\{g\_\{k\}\}\(s,a\)\\right\)=σr2​\(s,a\)N⁡\(s,a\)\+∑k=1Knk​\(s,a\)​Δgk​\(s,a\)N​\(s,a\)2\\displaystyle=\\frac\{\\sigma\_\{r\}^\{2\}\(s,a\)\}\{N\(s,a\)\}\+\\frac\{\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\Delta\_\{g\_\{k\}\}\(s,a\)\}\{N\(s,a\)^\{2\}\}\(15\)
ForN⁡\(s,a\)=0N\(s,a\)=0, the fixed default has variance zero, soVar𝒟⁡\(r^\+​\(s,a\)\)=W⁡\(s,a,𝐧\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\(\\hat\{r\}^\{\+\}\(s,a\)\)=W\(s,a;\\mathbf\{n\}\)in both cases\. Next, we substitute this variance into \([14](https://arxiv.org/html/2610.10974#A1.E14)\) and expand the expectations, evaluating divisions byπb​\(a\|s\)\\pi\_\{b\}\(a\|s\)only on its positive support:

T4\\displaystyle T\_\{4\}=∑s∈𝒮d0\(s\)∑a:πb​\(a\|s\)\>0πb\(a\|s\)\[\(ρ\(s,a\)2−πe​\(a\|s\)2πb​\(a\|s\)\)Var𝒟\(r^\+\(s,a\)\)\]\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\sum\_\{a:\\pi\_\{b\}\(a\|s\)\>0\}\\pi\_\{b\}\(a\|s\)\\left\[\\left\(\\rho\(s,a\)^\{2\}\-\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\\right\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\\right\]=∑s∈𝒮d0\(s\)∑a:πb​\(a\|s\)\>0\[\(πb\(a\|s\)ρ\(s,a\)2−πb​\(a\|s\)πb​\(a\|s\)πe\(a\|s\)2\)Var𝒟\(r^\+\(s,a\)\)\]\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\sum\_\{a:\\pi\_\{b\}\(a\|s\)\>0\}\\left\[\\left\(\\pi\_\{b\}\(a\|s\)\\rho\(s,a\)^\{2\}\-\\frac\{\\pi\_\{b\}\(a\|s\)\}\{\\pi\_\{b\}\(a\|s\)\}\\pi\_\{e\}\(a\|s\)^\{2\}\\right\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\\right\]=∑s∈𝒮d0\(s\)∑a:πb​\(a\|s\)\>0\[\(πb\(a\|s\)πe​\(a\|s\)2πb​\(a\|s\)2−πe\(a\|s\)2\)Var𝒟\(r^\+\(s,a\)\)\]\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\sum\_\{a:\\pi\_\{b\}\(a\|s\)\>0\}\\left\[\\left\(\\pi\_\{b\}\(a\|s\)\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)^\{2\}\}\-\\pi\_\{e\}\(a\|s\)^\{2\}\\right\)\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\\left\(\\hat\{r\}^\{\+\}\(s,a\)\\right\)\\right\]=∑s∈𝒮d0​\(s\)​∑a∈𝒜\(πe​\(a\|s\)2πb​\(a\|s\)−πe​\(a\|s\)2\)​W​\(s,a,𝐧\)\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\(\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\-\\pi\_\{e\}\(a\|s\)^\{2\}\\right\)W\(s,a;\\mathbf\{n\}\)\(16\)

### A\.3Objective Function

Let𝐧=\{nj​\(s,a\)\}s∈𝒮,a∈𝒜,j∈𝒦\\mathbf\{n\}=\\\{n\_\{j\}\(s,a\)\\\}\_\{s\\in\\mathcal\{S\},a\\in\\mathcal\{A\},j\\in\\mathcal\{K\}\}denotes the set of decision variables\. Combining \([13](https://arxiv.org/html/2610.10974#A1.E13)\) and \([16](https://arxiv.org/html/2610.10974#A1.E16)\), the objective function is:

J⁡\(𝐧\)\\displaystyle J\(\\mathbf\{n\}\)=∑s∈𝒮d0\(s\)\[∑a∈𝒜πe​\(a\|s\)2πb​\(a\|s\)B\(s,a;𝐧\)2\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\Bigg\[\\sum\_\{a\\in\\mathcal\{A\}\}\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}B\(s,a;\\mathbf\{n\}\)^\{2\}−\(∑a∈𝒜πe​\(a\|s\)​B​\(s,a,𝐧\)\)2\\displaystyle\\quad\-\\left\(\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)B\(s,a;\\mathbf\{n\}\)\\right\)^\{2\}\+∑a∈𝒜\(πe​\(a\|s\)2πb​\(a\|s\)−πe\(a\|s\)2\)W\(s,a;𝐧\)\]\.\\displaystyle\\quad\+\\sum\_\{a\\in\\mathcal\{A\}\}\\left\(\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\-\\pi\_\{e\}\(a\|s\)^\{2\}\\right\)W\(s,a;\\mathbf\{n\}\)\\Bigg\]\.

### A\.4Theoretical bound for source\-profile estimation error

The MSE decomposition above depends on source biasesεgk\\varepsilon\_\{g\_\{k\}\}and excess variancesΔgk\\Delta\_\{g\_\{k\}\}, which must often be estimated from pilot annotations\. We first bound the resulting perturbation ofJJuniformly over feasible allocations, then translate this bound into the MSE guarantee in Theorem[2](https://arxiv.org/html/2610.10974#Thmtheorem2)\. Appendices[E\.4](https://arxiv.org/html/2610.10974#A5.SS4)and[E\.5](https://arxiv.org/html/2610.10974#A5.SS5)complement this analysis with finite\-pilot evaluation and within\-pair resampling\.

Writex=\(s,a\)∈𝒳:=𝒮×𝒜x=\(s,a\)\\in\\mathcal\{X\}:=\\mathcal\{S\}\\times\\mathcal\{A\}, and define

fx:=nf​\(s,a\),mx​\(𝐧\):=∑k=1Knk​\(s,a\),Nx​\(𝐧\):=fx\+mx​\(𝐧\)\.f\_\{x\}:=n\_\{f\}\(s,a\),\\qquad m\_\{x\}\(\\mathbf\{n\}\):=\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\),\\qquad N\_\{x\}\(\\mathbf\{n\}\):=f\_\{x\}\+m\_\{x\}\(\\mathbf\{n\}\)\.Let

θ:=\{\(εgk​\(x\),Δgk​\(x\)\)\}x,k\\theta:=\\left\\\{\\bigl\(\\varepsilon\_\{g\_\{k\}\}\(x\),\\Delta\_\{g\_\{k\}\}\(x\)\\bigr\)\\right\\\}\_\{x,k\}and

θ^:=\{\(ε^gk​\(x\),Δ^gk​\(x\)\)\}x,k\\hat\{\\theta\}:=\\left\\\{\\bigl\(\\widehat\{\\varepsilon\}\_\{g\_\{k\}\}\(x\),\\widehat\{\\Delta\}\_\{g\_\{k\}\}\(x\)\\bigr\)\\right\\\}\_\{x,k\}denote the true and estimated source profiles, respectively\. Define the uniform estimation errors

δε:=maxx,k⁡\|ε^gk​\(x\)−εgk​\(x\)\|,δΔ:=maxx,k⁡\|Δ^gk​\(x\)−Δgk​\(x\)\|\.\\delta\_\{\\varepsilon\}:=\\max\_\{x,k\}\\left\|\\widehat\{\\varepsilon\}\_\{g\_\{k\}\}\(x\)\-\\varepsilon\_\{g\_\{k\}\}\(x\)\\right\|,\\qquad\\delta\_\{\\Delta\}:=\\max\_\{x,k\}\\left\|\\widehat\{\\Delta\}\_\{g\_\{k\}\}\(x\)\-\\Delta\_\{g\_\{k\}\}\(x\)\\right\|\.
To include the zero\-factual\-coverage case of Appendix[B\.8](https://arxiv.org/html/2610.10974#A2.SS8), define

Bx​\(𝐧,θ\):=\{∑k=1Knk​\(x\)​εgk​\(x\)Nx​\(𝐧\),Nx​\(𝐧\)\>0,r0​\(x\)−μr​\(x\),Nx​\(𝐧\)=0\.B\_\{x\}\(\\mathbf\{n\};\\theta\):=\\begin\{cases\}\\displaystyle\\frac\{\\sum\_\{k=1\}^\{K\}n\_\{k\}\(x\)\\varepsilon\_\{g\_\{k\}\}\(x\)\}\{N\_\{x\}\(\\mathbf\{n\}\)\},&N\_\{x\}\(\\mathbf\{n\}\)\>0,\\\\\[12\.0pt\] r\_\{0\}\(x\)\-\\mu\_\{r\}\(x\),&N\_\{x\}\(\\mathbf\{n\}\)=0\.\\end\{cases\}The same defaultr0​\(x\)r\_\{0\}\(x\)is used underθ\\thetaandθ^\\hat\{\\theta\}\. Let

βmax:=max\{maxx,k\|εgk\(x\)\|,maxx:fx=0\|r0\(x\)−μr\(x\)\|\}\.\\beta\_\{\\max\}:=\\max\\left\\\{\\max\_\{x,k\}\|\\varepsilon\_\{g\_\{k\}\}\(x\)\|,\\,\\max\_\{x:f\_\{x\}=0\}\|r\_\{0\}\(x\)\-\\mu\_\{r\}\(x\)\|\\right\\\}\.If factual rewards, source annotations, and the default value all lie in\[0,R\]\[0,R\], thenβmax≤R\\beta\_\{\\max\}\\leq R\.

We next define two sensitivity constants\. First,

A1:=∑s∈𝒮∑a∈𝒜α1​\(s,a\)\.A\_\{1\}:=\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\alpha\_\{1\}\(s,a\)\.
For the excess\-variance term, letMxM\_\{x\}denote the maximum number of annotations that can feasibly be assigned to pairxx\. Define

γx:=max0≤m≤Mxm∈ℤ⁡\{m\(fx\+m\)2,fx\+m\>0,0,fx=m=0\.\\gamma\_\{x\}:=\\max\_\{\\begin\{subarray\}\{c\}0\\leq m\\leq M\_\{x\}\\\\ m\\in\\mathbb\{Z\}\\end\{subarray\}\}\\begin\{cases\}\\displaystyle\\frac\{m\}\{\(f\_\{x\}\+m\)^\{2\}\},&f\_\{x\}\+m\>0,\\\\\[8\.0pt\] 0,&f\_\{x\}=m=0\.\\end\{cases\}\(17\)Define

A2:=∑x∈𝒳α3​\(x\)​γx\.A\_\{2\}:=\\sum\_\{x\\in\\mathcal\{X\}\}\\alpha\_\{3\}\(x\)\\gamma\_\{x\}\.
###### Lemma 1\(Uniform perturbation of the allocation objective\)\.

For every feasible allocation𝐧∈ℱ\\mathbf\{n\}\\in\\mathcal\{F\},

\|J⁡\(𝐧,θ^\)−J⁡\(𝐧,θ\)\|≤Φ⁡\(δε,δΔ\),\\left\|J\(\\mathbf\{n\};\\hat\{\\theta\}\)\-J\(\\mathbf\{n\};\\theta\)\\right\|\\leq\\Phi\(\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}\),\(18\)where

Φ⁡\(δε,δΔ\):=A1​δε​\(2​βmax\+δε\)\+A2​δΔ\.\\Phi\(\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}\):=A\_\{1\}\\delta\_\{\\varepsilon\}\\left\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\\right\)\+A\_\{2\}\\delta\_\{\\Delta\}\.

###### Proof\.

Fix an allocation𝐧\\mathbf\{n\}, and define

Ux​\(𝐧\):=Bx​\(𝐧,θ^\)−Bx​\(𝐧,θ\)\.U\_\{x\}\(\\mathbf\{n\}\):=B\_\{x\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\-B\_\{x\}\(\\mathbf\{n\};\\theta\)\.IfNx​\(𝐧\)\>0N\_\{x\}\(\\mathbf\{n\}\)\>0, then

Ux​\(𝐧\)=∑knk​\(x\)​\(ε^gk​\(x\)−εgk​\(x\)\)Nx​\(𝐧\),U\_\{x\}\(\\mathbf\{n\}\)=\\frac\{\\sum\_\{k\}n\_\{k\}\(x\)\\left\(\\widehat\{\\varepsilon\}\_\{g\_\{k\}\}\(x\)\-\\varepsilon\_\{g\_\{k\}\}\(x\)\\right\)\}\{N\_\{x\}\(\\mathbf\{n\}\)\},and hence

\|Ux​\(𝐧\)\|≤mx​\(𝐧\)Nx​\(𝐧\)​δε≤δε\.\|U\_\{x\}\(\\mathbf\{n\}\)\|\\leq\\frac\{m\_\{x\}\(\\mathbf\{n\}\)\}\{N\_\{x\}\(\\mathbf\{n\}\)\}\\delta\_\{\\varepsilon\}\\leq\\delta\_\{\\varepsilon\}\.\(19\)IfNx​\(𝐧\)=0N\_\{x\}\(\\mathbf\{n\}\)=0, the same default is used under both parameter sets, soUx​\(𝐧\)=0U\_\{x\}\(\\mathbf\{n\}\)=0\. Moreover,

\|Bx​\(𝐧,θ\)\|≤βmax\.\|B\_\{x\}\(\\mathbf\{n\};\\theta\)\|\\leq\\beta\_\{\\max\}\.
For each contextss, define

Ms:=d0​\(s\)​\[diag⁡\(πe​\(a∣s\)2πb​\(a∣s\)\)a∈𝒜−𝝅e​\(s\)​𝝅e​\(s\)⊤\],M\_\{s\}:=d\_\{0\}\(s\)\\left\[\\operatorname\{diag\}\\left\(\\frac\{\\pi\_\{e\}\(a\\mid s\)^\{2\}\}\{\\pi\_\{b\}\(a\\mid s\)\}\\right\)\_\{a\\in\\mathcal\{A\}\}\-\\boldsymbol\{\\pi\}\_\{e\}\(s\)\\boldsymbol\{\\pi\}\_\{e\}\(s\)^\{\\top\}\\right\],where𝝅e​\(s\):=\(πe​\(a∣s\)\)a∈𝒜\\boldsymbol\{\\pi\}\_\{e\}\(s\):=\(\\pi\_\{e\}\(a\\mid s\)\)\_\{a\\in\\mathcal\{A\}\}\. For anyz∈ℝ\|𝒜\|z\\in\\mathbb\{R\}^\{\|\\mathcal\{A\}\|\},

z⊤​Ms​z\\displaystyle z^\{\\top\}M\_\{s\}z=d0​\(s\)​\[∑aπe​\(a∣s\)2πb​\(a∣s\)​za2−\(∑aπe​\(a∣s\)​za\)2\]\\displaystyle=d\_\{0\}\(s\)\\left\[\\sum\_\{a\}\\frac\{\\pi\_\{e\}\(a\\mid s\)^\{2\}\}\{\\pi\_\{b\}\(a\\mid s\)\}z\_\{a\}^\{2\}\-\\left\(\\sum\_\{a\}\\pi\_\{e\}\(a\\mid s\)z\_\{a\}\\right\)^\{2\}\\right\]=d0\(s\)Vara∼πb\(⋅∣s\)\[πe​\(a∣s\)πb​\(a∣s\)za\]≥0\.\\displaystyle=d\_\{0\}\(s\)\\operatorname\{Var\}\_\{a\\sim\\pi\_\{b\}\(\\cdot\\mid s\)\}\\left\[\\frac\{\\pi\_\{e\}\(a\\mid s\)\}\{\\pi\_\{b\}\(a\\mid s\)\}z\_\{a\}\\right\]\\geq 0\.Therefore,Ms⪰0M\_\{s\}\\succeq 0\.

Let

𝑩s:=\(Bs,a​\(𝐧,θ\)\)a∈𝒜,𝑼s:=\(Us,a​\(𝐧\)\)a∈𝒜\.\\boldsymbol\{B\}\_\{s\}:=\\bigl\(B\_\{s,a\}\(\\mathbf\{n\};\\theta\)\\bigr\)\_\{a\\in\\mathcal\{A\}\},\\qquad\\boldsymbol\{U\}\_\{s\}:=\\bigl\(U\_\{s,a\}\(\\mathbf\{n\}\)\\bigr\)\_\{a\\in\\mathcal\{A\}\}\.The combined bias term satisfies

J1​\(𝐧,θ\)−J2​\(𝐧,θ\)=∑s𝑩s⊤​Ms​𝑩s\.J\_\{1\}\(\\mathbf\{n\};\\theta\)\-J\_\{2\}\(\\mathbf\{n\};\\theta\)=\\sum\_\{s\}\\boldsymbol\{B\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{B\}\_\{s\}\.Using the seminorm induced by\{Ms\}s\\\{M\_\{s\}\\\}\_\{s\},

\|\[J1​\(𝐧,θ^\)−J2​\(𝐧,θ^\)\]−\[J1​\(𝐧,θ\)−J2​\(𝐧,θ\)\]\|\\displaystyle\\left\|\\bigl\[J\_\{1\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\-J\_\{2\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\\bigr\]\-\\bigl\[J\_\{1\}\(\\mathbf\{n\};\\theta\)\-J\_\{2\}\(\\mathbf\{n\};\\theta\)\\bigr\]\\right\|\(20\)≤2​\(∑s𝑩s⊤​Ms​𝑩s\)1/2​\(∑s𝑼s⊤​Ms​𝑼s\)1/2\+∑s𝑼s⊤​Ms​𝑼s\.\\displaystyle\\leq 2\\left\(\\sum\_\{s\}\\boldsymbol\{B\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{B\}\_\{s\}\\right\)^\{1/2\}\\left\(\\sum\_\{s\}\\boldsymbol\{U\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{U\}\_\{s\}\\right\)^\{1/2\}\+\\sum\_\{s\}\\boldsymbol\{U\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{U\}\_\{s\}\.Because the rank\-one term subtracted inMsM\_\{s\}is nonnegative,

∑s𝑩s⊤​Ms​𝑩s≤∑s,aα1​\(s,a\)​Bs,a2≤A1​βmax2,\\sum\_\{s\}\\boldsymbol\{B\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{B\}\_\{s\}\\leq\\sum\_\{s,a\}\\alpha\_\{1\}\(s,a\)B\_\{s,a\}^\{2\}\\leq A\_\{1\}\\beta\_\{\\max\}^\{2\},\(21\)and, using \([19](https://arxiv.org/html/2610.10974#A1.E19)\),

∑s𝑼s⊤​Ms​𝑼s≤∑s,aα1​\(s,a\)​Us,a2≤A1​δε2\.\\sum\_\{s\}\\boldsymbol\{U\}\_\{s\}^\{\\top\}M\_\{s\}\\boldsymbol\{U\}\_\{s\}\\leq\\sum\_\{s,a\}\\alpha\_\{1\}\(s,a\)U\_\{s,a\}^\{2\}\\leq A\_\{1\}\\delta\_\{\\varepsilon\}^\{2\}\.\(22\)Substituting \([21](https://arxiv.org/html/2610.10974#A1.E21)\)–\([22](https://arxiv.org/html/2610.10974#A1.E22)\) into \([20](https://arxiv.org/html/2610.10974#A1.E20)\) gives

\|\[J1​\(𝐧,θ^\)−J2​\(𝐧,θ^\)\]−\[J1​\(𝐧,θ\)−J2​\(𝐧,θ\)\]\|≤A1​δε​\(2​βmax\+δε\)\.\\left\|\\bigl\[J\_\{1\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\-J\_\{2\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\\bigr\]\-\\bigl\[J\_\{1\}\(\\mathbf\{n\};\\theta\)\-J\_\{2\}\(\\mathbf\{n\};\\theta\)\\bigr\]\\right\|\\leq A\_\{1\}\\delta\_\{\\varepsilon\}\\left\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\\right\)\.\(23\)
For the excess\-variance term,

\|J3​\(𝐧,θ^\)−J3​\(𝐧,θ\)\|\\displaystyle\\left\|J\_\{3\}\(\\mathbf\{n\};\\hat\{\\theta\}\)\-J\_\{3\}\(\\mathbf\{n\};\\theta\)\\right\|≤∑xα3​\(x\)​∑knk​\(x\)​\|Δ^gk​\(x\)−Δgk​\(x\)\|Nx​\(𝐧\)2\\displaystyle\\leq\\sum\_\{x\}\\alpha\_\{3\}\(x\)\\frac\{\\sum\_\{k\}n\_\{k\}\(x\)\\left\|\\widehat\{\\Delta\}\_\{g\_\{k\}\}\(x\)\-\\Delta\_\{g\_\{k\}\}\(x\)\\right\|\}\{N\_\{x\}\(\\mathbf\{n\}\)^\{2\}\}\(24\)≤δΔ​∑xα3​\(x\)​mx​\(𝐧\)\(fx\+mx​\(𝐧\)\)2\\displaystyle\\leq\\delta\_\{\\Delta\}\\sum\_\{x\}\\alpha\_\{3\}\(x\)\\frac\{m\_\{x\}\(\\mathbf\{n\}\)\}\{\(f\_\{x\}\+m\_\{x\}\(\\mathbf\{n\}\)\)^\{2\}\}≤A2​δΔ,\\displaystyle\\leq A\_\{2\}\\delta\_\{\\Delta\},where the ratio is defined as zero whenfx=mx=0f\_\{x\}=m\_\{x\}=0, and the last inequality follows from \([17](https://arxiv.org/html/2610.10974#A1.E17)\)\. Combining \([23](https://arxiv.org/html/2610.10974#A1.E23)\) and \([24](https://arxiv.org/html/2610.10974#A1.E24)\) proves \([18](https://arxiv.org/html/2610.10974#A1.E18)\)\. ∎

Let

𝐧⋆∈arg⁡min𝐧∈ℱ⁡J⁡\(𝐧,θ\)\\mathbf\{n\}^\{\\star\}\\in\\arg\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\};\\theta\)be an optimal allocation under the population profiles\. Let𝐧^\\widehat\{\\mathbf\{n\}\}be the allocation returned by Algorithm[1](https://arxiv.org/html/2610.10974#alg1)usingθ^\\hat\{\\theta\}, and suppose that

J⁡\(𝐧^,θ^\)≤min𝐧∈ℱ⁡J⁡\(𝐧,θ^\)\+ξalg,J\(\\widehat\{\\mathbf\{n\}\};\\hat\{\\theta\}\)\\leq\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\};\\hat\{\\theta\}\)\+\\xi\_\{\\mathrm\{alg\}\},\(25\)whereξalg≥0\\xi\_\{\\mathrm\{alg\}\}\\geq 0denotes the algorithmic suboptimality on the estimated objective\.

###### Theorem[2](https://arxiv.org/html/2610.10974#Thmtheorem2)\(Effect of source\-profile estimation on MSE\)\.

Under the assumptions of Theorem[1](https://arxiv.org/html/2610.10974#Thmtheorem1), assume that the pilot/allocation sample, the reward\-model and annotation sample, and the OPE evaluation sample are independent\. Conditional on the pilot, with factual counts, factual reward moments, and policy probabilities fixed, source\-profile estimation introduces no additional OPE bias, and

MSE⁡\(𝐧^\)−MSE⁡\(𝐧⋆\)≤2​A1​δε​\(2​βmax\+δε\)\+2​A2​δΔ\+ξalgN\.\\operatorname\{MSE\}\(\\widehat\{\\mathbf\{n\}\}\)\-\\operatorname\{MSE\}\(\\mathbf\{n\}^\{\\star\}\)\\leq\\frac\{2A\_\{1\}\\delta\_\{\\varepsilon\}\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\)\+2A\_\{2\}\\delta\_\{\\Delta\}\+\\xi\_\{\\mathrm\{alg\}\}\}\{N\}\.

###### Proof\.

Let

𝐧^⋆∈arg⁡min𝐧∈ℱ⁡J⁡\(𝐧,θ^\)\.\\widehat\{\\mathbf\{n\}\}^\{\\,\\star\}\\in\\arg\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\};\\hat\{\\theta\}\)\.By adding and subtracting the estimated\-objective values,

J⁡\(𝐧^,θ\)−J⁡\(𝐧⋆,θ\)\\displaystyle J\(\\widehat\{\\mathbf\{n\}\};\\theta\)\-J\(\\mathbf\{n\}^\{\\star\};\\theta\)=\[J⁡\(𝐧^,θ\)−J⁡\(𝐧^,θ^\)\]\+\[J⁡\(𝐧^,θ^\)−J⁡\(𝐧^⋆,θ^\)\]\\displaystyle=\\left\[J\(\\widehat\{\\mathbf\{n\}\};\\theta\)\-J\(\\widehat\{\\mathbf\{n\}\};\\hat\{\\theta\}\)\\right\]\+\\left\[J\(\\widehat\{\\mathbf\{n\}\};\\hat\{\\theta\}\)\-J\(\\widehat\{\\mathbf\{n\}\}^\{\\,\\star\};\\hat\{\\theta\}\)\\right\]\+\[J⁡\(𝐧^⋆,θ^\)−J⁡\(𝐧⋆,θ^\)\]\+\[J⁡\(𝐧⋆,θ^\)−J⁡\(𝐧⋆,θ\)\]\.\\displaystyle\+\\left\[J\(\\widehat\{\\mathbf\{n\}\}^\{\\,\\star\};\\hat\{\\theta\}\)\-J\(\\mathbf\{n\}^\{\\star\};\\hat\{\\theta\}\)\\right\]\+\\left\[J\(\\mathbf\{n\}^\{\\star\};\\hat\{\\theta\}\)\-J\(\\mathbf\{n\}^\{\\star\};\\theta\)\\right\]\.By Lemma[1](https://arxiv.org/html/2610.10974#Thmlemma1), the first and fourth terms are each at mostΦ⁡\(δε,δΔ\)\\Phi\(\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}\)\. By \([25](https://arxiv.org/html/2610.10974#A1.E25)\), the second term is at mostξalg\\xi\_\{\\mathrm\{alg\}\}, and the third term is nonpositive by the optimality of𝐧^⋆\\widehat\{\\mathbf\{n\}\}^\{\\,\\star\}for the estimated objective\. Hence,

J⁡\(𝐧^,θ\)−J⁡\(𝐧⋆,θ\)≤2​Φ​\(δε,δΔ\)\+ξalg\.J\(\\widehat\{\\mathbf\{n\}\};\\theta\)\-J\(\\mathbf\{n\}^\{\\star\};\\theta\)\\leq 2\\Phi\(\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}\)\+\\xi\_\{\\mathrm\{alg\}\}\.\(26\)Using

Φ⁡\(δε,δΔ\)=A1​δε​\(2​βmax\+δε\)\+A2​δΔ,\\Phi\(\\delta\_\{\\varepsilon\},\\delta\_\{\\Delta\}\)=A\_\{1\}\\delta\_\{\\varepsilon\}\\left\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\\right\)\+A\_\{2\}\\delta\_\{\\Delta\},this becomes

J⁡\(𝐧^,θ\)−J⁡\(𝐧⋆,θ\)≤2​A1​δε​\(2​βmax\+δε\)\+2​A2​δΔ\+ξalg\.J\(\\widehat\{\\mathbf\{n\}\};\\theta\)\-J\(\\mathbf\{n\}^\{\\star\};\\theta\)\\leq 2A\_\{1\}\\delta\_\{\\varepsilon\}\\left\(2\\beta\_\{\\max\}\+\\delta\_\{\\varepsilon\}\\right\)\+2A\_\{2\}\\delta\_\{\\Delta\}\+\\xi\_\{\\mathrm\{alg\}\}\.\(27\)
Under the stated data separation, DM\+\-IS is unbiased conditional on the pilot\-selected allocation\. Theorem[1](https://arxiv.org/html/2610.10974#Thmtheorem1)therefore gives

MSE⁡\(𝐧\)=C0\+J⁡\(𝐧,θ\)N\.\\operatorname\{MSE\}\(\\mathbf\{n\}\)=C\_\{0\}\+\\frac\{J\(\\mathbf\{n\};\\theta\)\}\{N\}\.Subtracting the MSE at𝐧⋆\\mathbf\{n\}^\{\\star\}and applying \([27](https://arxiv.org/html/2610.10974#A1.E27)\) proves \([4](https://arxiv.org/html/2610.10974#S4.E4)\)\. ∎

## Appendix BThreshold for Valuable Annotation for a single annotation source

In this appendix, we analyze, for a single annotation source, under what conditions adding counterfactual annotations improves or degrades the estimator variance\. We begin with the simplest case \(within a single context\-action pair\), then extend to actions within a single context, and finally consider arbitrary context\-action pairs\.

### B\.1Setup and Notation

We consider a single annotation source \(K=1K=1\) and suppress the source index\. For a fixed context\-action pair\(s,a\)\(s,a\), letnf:=nf​\(s,a\)n\_\{f\}:=n\_\{f\}\(s,a\),ε:=εg​\(s,a\)\\varepsilon:=\\varepsilon\_\{g\}\(s,a\),Δ:=Δg​\(s,a\)\\Delta:=\\Delta\_\{g\}\(s,a\),σ2:=σr2​\(s,a\)\\sigma^\{2\}:=\\sigma\_\{r\}^\{2\}\(s,a\), andnndenote the number of annotations\. We assumenf\>0n\_\{f\}\>0andα1\>0\\alpha\_\{1\}\>0for the analysis below; the casenf=0n\_\{f\}=0is treated in Appendix[B\.8](https://arxiv.org/html/2610.10974#A2.SS8)\. The total sample size isN:=nf\+nN:=n\_\{f\}\+n\. Define the total annotation variance asσg2:=σ2\+Δ\\sigma\_\{g\}^\{2\}:=\\sigma^\{2\}\+\\Delta, and the policy\-dependent ratio:

η⁡\(s,a\):=α3​\(s,a\)α1​\(s,a\)=πe​\(a\|s\)2πb​\(a\|s\)−πe​\(a\|s\)2πe​\(a\|s\)2πb​\(a\|s\)=1−πb​\(a\|s\)\.\\eta\(s,a\):=\\frac\{\\alpha\_\{3\}\(s,a\)\}\{\\alpha\_\{1\}\(s,a\)\}=\\frac\{\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\-\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\}\}=1\-\\pi\_\{b\}\(a\|s\)\.\(28\)This ratio satisfies0≤η⁡\(s,a\)<10\\leq\\eta\(s,a\)<1under the common support assumptionπb​\(a\|s\)\>0\\pi\_\{b\}\(a\|s\)\>0, and measures how much the behavior policy under\-covers actionaa: smallπb​\(a\|s\)\\pi\_\{b\}\(a\|s\)yieldsη\\etaclose to 1\.

### B\.2Bias Threshold for a Single Context\-Action Pair

Ignoring the cross\-action bias correction term \(J2J\_\{2\}\) \(sinceJ2J\_\{2\}couples all actions together, we will analysis it later in[B\.6](https://arxiv.org/html/2610.10974#A2.SS6)\), the contribution of a single\(s,a\)\(s,a\)to the objective is:

ϕ⁡\(n\):=α1​n2​ε2N2\+α3​\(σ2N\+n​ΔN2\)=α1​n2​ε2\+α3​\(nf​σ2\+n​σg2\)N2\.\\phi\(n\):=\\alpha\_\{1\}\\frac\{n^\{2\}\\varepsilon^\{2\}\}\{N^\{2\}\}\+\\alpha\_\{3\}\\left\(\\frac\{\\sigma^\{2\}\}\{N\}\+\\frac\{n\\Delta\}\{N^\{2\}\}\\right\)=\\frac\{\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}\\left\(n\_\{f\}\\sigma^\{2\}\+n\\sigma\_\{g\}^\{2\}\\right\)\}\{N^\{2\}\}\.\(29\)
###### Lemma 2\(Marginal Change for a single \(s,a\)\)\.

The marginal change in the local objective when increasing the annotation count fromnnton\+1n\+1is:

δ​ϕ​\(n\)\\displaystyle\\delta\\phi\(n\):=ϕ⁡\(n\+1\)−ϕ⁡\(n\)\\displaystyle:=\\phi\(n\+1\)\-\\phi\(n\)\(30\)=α1​ε2​nf​\(\(2​n\+1\)​N\+n\)\+α3​σg2​\(nf2−n⁡\(n\+1\)\)−α3​nf​σ2​\(2​N\+1\)N2​\(N\+1\)2,\\displaystyle=\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\\bigl\(\(2n\+1\)N\+n\\bigr\)\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigl\(n\_\{f\}^\{2\}\-n\(n\+1\)\\bigr\)\-\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\(2N\+1\)\}\{N^\{2\}\(N\+1\)^\{2\}\},whereN=nf\+nN=n\_\{f\}\+n\.

###### Proof\.

Writeϕ⁡\(n\)=P⁡\(n\)N2\\phi\(n\)=\\frac\{P\(n\)\}\{N^\{2\}\}whereP⁡\(n\)=α1​n2​ε2\+α3​nf​σ2\+α3​n​σg2P\(n\)=\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\+\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}\. Then:

δ​ϕ​\(n\)=P⁡\(n\+1\)⋅N2−P⁡\(n\)⋅\(N\+1\)2N2​\(N\+1\)2\.\\delta\\phi\(n\)=\\frac\{P\(n\+1\)\\cdot N^\{2\}\-P\(n\)\\cdot\(N\+1\)^\{2\}\}\{N^\{2\}\(N\+1\)^\{2\}\}\.\(31\)We decomposeP⁡\(n\+1\)=P⁡\(n\)\+α1​\(2​n\+1\)​ε2\+α3​σg2P\(n\+1\)=P\(n\)\+\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}and expand the numerator:

P⁡\(n\+1\)​N2−P⁡\(n\)​\(N\+1\)2\\displaystyle P\(n\+1\)N^\{2\}\-P\(n\)\(N\+1\)^\{2\}=\[P⁡\(n\)\+α1​\(2​n\+1\)​ε2\+α3​σg2\]​N2−P⁡\(n\)​\(N\+1\)2\\displaystyle=\\bigl\[P\(n\)\+\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigr\]N^\{2\}\-P\(n\)\(N\+1\)^\{2\}=P⁡\(n\)​\[N2−\(N\+1\)2\]\+\[α1​\(2​n\+1\)​ε2\+α3​σg2\]​N2\\displaystyle=P\(n\)\\bigl\[N^\{2\}\-\(N\+1\)^\{2\}\\bigr\]\+\\bigl\[\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigr\]N^\{2\}=−P⁡\(n\)​\(2​N\+1\)\+\[α1​\(2​n\+1\)​ε2\+α3​σg2\]​N2\.\\displaystyle=\-P\(n\)\(2N\+1\)\+\\bigl\[\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigr\]N^\{2\}\.\(32\)We evaluate each component in \([32](https://arxiv.org/html/2610.10974#A2.E32)\) separately\. For the first term:

−P⁡\(n\)​\(2​N\+1\)=−α1​n2​ε2​\(2​N\+1\)−α3​nf​σ2​\(2​N\+1\)−α3​n​σg2​\(2​N\+1\)\.\-P\(n\)\(2N\+1\)=\-\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\(2N\+1\)\-\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\(2N\+1\)\-\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}\(2N\+1\)\.\(33\)For the second term:

\[α1​\(2​n\+1\)​ε2\+α3​σg2\]​N2=α1​\(2​n\+1\)​ε2​N2\+α3​σg2​N2\.\\bigl\[\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigr\]N^\{2\}=\\alpha\_\{1\}\(2n\+1\)\\varepsilon^\{2\}N^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}N^\{2\}\.\(34\)Collectingα1​ε2\\alpha\_\{1\}\\varepsilon^\{2\}terms we getα1​ε2​\[\(2​n\+1\)​N2−n2​\(2​N\+1\)\]\\alpha\_\{1\}\\varepsilon^\{2\}\\bigl\[\(2n\+1\)N^\{2\}\-n^\{2\}\(2N\+1\)\\bigr\]\.

We use the factorization\(2​n\+1\)​N2−n2​\(2​N\+1\)=\[\(2​n\+1\)​N\+n\]​\[N−n\]=nf​\[\(2​n\+1\)​N\+n\]\(2n\+1\)N^\{2\}\-n^\{2\}\(2N\+1\)=\\bigl\[\(2n\+1\)N\+n\\bigr\]\\bigl\[N\-n\\bigr\]=n\_\{f\}\\bigl\[\(2n\+1\)N\+n\\bigr\], which follows from the difference\-of\-products identity withN−n=nfN\-n=n\_\{f\}\. Collectingα3​σg2\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}terms:

α3​σg2​\[N2−n⁡\(2​N\+1\)\]=α3​σg2​\[nf2−n⁡\(n\+1\)\],\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigl\[N^\{2\}\-n\(2N\+1\)\\bigr\]=\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigl\[n\_\{f\}^\{2\}\-n\(n\+1\)\\bigr\],\(35\)where we expandedN2−n⁡\(2​N\+1\)=\(nf\+n\)2−2​n​\(nf\+n\)−n=nf2\+2​n​nf\+n2−2​n​nf−2​n2−n=nf2−n2−nN^\{2\}\-n\(2N\+1\)=\(n\_\{f\}\+n\)^\{2\}\-2n\(n\_\{f\}\+n\)\-n=n\_\{f\}^\{2\}\+2nn\_\{f\}\+n^\{2\}\-2nn\_\{f\}\-2n^\{2\}\-n=n\_\{f\}^\{2\}\-n^\{2\}\-n\. The remaining term is−α3​nf​σ2​\(2​N\+1\)\-\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\(2N\+1\)\. Substitute \([33](https://arxiv.org/html/2610.10974#A2.E33)\) – \([35](https://arxiv.org/html/2610.10974#A2.E35)\) back into \([31](https://arxiv.org/html/2610.10974#A2.E31)\) yields Eq\. \([30](https://arxiv.org/html/2610.10974#A2.E30)\)\. ∎

### B\.3Proof of Theorem[3](https://arxiv.org/html/2610.10974#Thmtheorem3)\(First Annotation Threshold\)

###### Theorem[3](https://arxiv.org/html/2610.10974#Thmtheorem3)\.

Withη⁡\(s,a\)\\eta\(s,a\)in \([28](https://arxiv.org/html/2610.10974#A2.E28)\), the first annotation \(n=0→1n=0\\to 1\) does not increase the local objective \([29](https://arxiv.org/html/2610.10974#A2.E29)\), i\.e\.,δ​ϕ​\(0\)≤0\\delta\\phi\(0\)\\leq 0, if and only if

ε2​\(s,a\)≤η⁡\(s,a\)​\(\(nf​\(s,a\)\+1\)​σ2​\(s,a\)nf​\(s,a\)−Δ⁡\(s,a\)\)\.\\varepsilon^\{2\}\(s,a\)\\leq\\eta\(s,a\)\\\!\\left\(\\frac\{\(n\_\{f\}\(s,a\)\+1\)\\sigma^\{2\}\(s,a\)\}\{n\_\{f\}\(s,a\)\}\-\\Delta\(s,a\)\\right\)\.\(36\)Equality in \([36](https://arxiv.org/html/2610.10974#A2.E36)\) leaves the local objective unchanged, whereas strict inequality gives a strict decrease\. In particular, ifΔ⁡\(s,a\)≥\(nf​\(s,a\)\+1\)​σ2/nf​\(s,a\)\\Delta\(s,a\)\\geq\(n\_\{f\}\(s,a\)\+1\)\\sigma^\{2\}/n\_\{f\}\(s,a\), thenδ​ϕ​\(0\)\>0\\delta\\phi\(0\)\>0for every \(s,a\) withε⁡\(s,a\)≠0\\varepsilon\(s,a\)\\neq 0\.

###### Proof\.

Substitutingn=0n=0andN=nfN=n\_\{f\}into Eq\. \([30](https://arxiv.org/html/2610.10974#A2.E30)\):

δ​ϕ​\(0\)\\displaystyle\\delta\\phi\(0\)=α1​ε2​nf⋅nf\+α3​σg2⋅nf2−α3​nf​σ2​\(2​nf\+1\)nf2​\(nf\+1\)2\\displaystyle=\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\\cdot n\_\{f\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\cdot n\_\{f\}^\{2\}\-\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\(2n\_\{f\}\+1\)\}\{n\_\{f\}^\{2\}\(n\_\{f\}\+1\)^\{2\}\}=nf​\[α1​ε2​nf\+α3​nf​σg2−α3​\(2​nf\+1\)​σ2\]nf2​\(nf\+1\)2\\displaystyle=\\frac\{n\_\{f\}\\bigl\[\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\+\\alpha\_\{3\}n\_\{f\}\\sigma\_\{g\}^\{2\}\-\\alpha\_\{3\}\(2n\_\{f\}\+1\)\\sigma^\{2\}\\bigr\]\}\{n\_\{f\}^\{2\}\(n\_\{f\}\+1\)^\{2\}\}=α1​ε2​nf\+α3​\[nf​\(σ2\+Δ\)−\(2​nf\+1\)​σ2\]nf​\(nf\+1\)2\\displaystyle=\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\+\\alpha\_\{3\}\\bigl\[n\_\{f\}\(\\sigma^\{2\}\+\\Delta\)\-\(2n\_\{f\}\+1\)\\sigma^\{2\}\\bigr\]\}\{n\_\{f\}\(n\_\{f\}\+1\)^\{2\}\}=α1​ε2​nf\+α3​\[nf​Δ−\(nf\+1\)​σ2\]nf​\(nf\+1\)2\.\\displaystyle=\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\+\\alpha\_\{3\}\\bigl\[n\_\{f\}\\Delta\-\(n\_\{f\}\+1\)\\sigma^\{2\}\\bigr\]\}\{n\_\{f\}\(n\_\{f\}\+1\)^\{2\}\}\.\(37\)Becausenf\>0n\_\{f\}\>0andα1\>0\\alpha\_\{1\}\>0, the conditionδ​ϕ​\(0\)≤0\\delta\\phi\(0\)\\leq 0is equivalent to

α1​ε2​nf\\displaystyle\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}≤α3​\[\(nf\+1\)​σ2−nf​Δ\],\\displaystyle\\leq\\alpha\_\{3\}\\bigl\[\(n\_\{f\}\+1\)\\sigma^\{2\}\-n\_\{f\}\\Delta\\bigr\],ε2\\displaystyle\\varepsilon^\{2\}≤α3α1​\(\(nf\+1\)​σ2nf−Δ\)=η⁡\(s,a\)​\(\(nf\+1\)​σ2nf−Δ\)\.\\displaystyle\\leq\\frac\{\\alpha\_\{3\}\}\{\\alpha\_\{1\}\}\\left\(\\frac\{\(n\_\{f\}\+1\)\\sigma^\{2\}\}\{n\_\{f\}\}\-\\Delta\\right\)=\\eta\(s,a\)\\left\(\\frac\{\(n\_\{f\}\+1\)\\sigma^\{2\}\}\{n\_\{f\}\}\-\\Delta\\right\)\.The same equivalence holds with either equality or strict inequality throughout\. WhenΔ≥\(nf\+1\)​σ2/nf\\Delta\\geq\(n\_\{f\}\+1\)\\sigma^\{2\}/n\_\{f\}andε≠0\\varepsilon\\neq 0, the right\-hand side is nonpositive whileε2\>0\\varepsilon^\{2\}\>0, so \([37](https://arxiv.org/html/2610.10974#A2.E37)\) givesδ​ϕ​\(0\)\>0\\delta\\phi\(0\)\>0\. ∎

### B\.4Proof of Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4)\(Four Regimes of Annotation Value\)

###### Lemma 3\(Continuous Critical Pointn∗n^\{\*\}\)\.

Treatingnnas a continuous variable, the derivative ofϕ\\phiis:

ϕ′​\(n\)=\(2​α1​ε2​nf−α3​σg2\)​n\+α3​nf​\(Δ−σ2\)\(nf\+n\)3\.\\phi^\{\\prime\}\(n\)=\\frac\{\(2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\)\\,n\+\\alpha\_\{3\}n\_\{f\}\(\\Delta\-\\sigma^\{2\}\)\}\{\(n\_\{f\}\+n\)^\{3\}\}\.\(38\)If2​α1​ε2​nf≠α3​σg22\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\\neq\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}, the only possible critical point is

n∗=α3​nf​\(σ2−Δ\)2​α1​ε2​nf−α3​σg2=η​nf​\(σ2−Δ\)2​ε2​nf−η​σg2\.n^\{\*\}=\\frac\{\\alpha\_\{3\}n\_\{f\}\(\\sigma^\{2\}\-\\Delta\)\}\{2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\}=\\frac\{\\eta\\,n\_\{f\}\(\\sigma^\{2\}\-\\Delta\)\}\{2\\varepsilon^\{2\}n\_\{f\}\-\\eta\\,\\sigma\_\{g\}^\{2\}\}\.\(39\)It is an interior critical point exactly whenn∗\>0n^\{\*\}\>0\. If2​α1​ε2​nf=α3​σg22\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}=\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}, there is no interior critical point unlessα3​\(Δ−σ2\)=0\\alpha\_\{3\}\(\\Delta\-\\sigma^\{2\}\)=0, in which caseϕ\\phiis constant\. Furthermore,ϕ⁡\(n\)→α1​ε2\\phi\(n\)\\to\\alpha\_\{1\}\\varepsilon^\{2\}asn→∞n\\to\\infty\.

###### Proof\.

From Eq\. \([29](https://arxiv.org/html/2610.10974#A2.E29)\),ϕ⁡\(n\)=P⁡\(n\)/N2\\phi\(n\)=P\(n\)/N^\{2\}whereP⁡\(n\)=α1​n2​ε2\+α3​nf​σ2\+α3​n​σg2P\(n\)=\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\+\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}andN=nf\+nN=n\_\{f\}\+n\. By the quotient rule:

ϕ′​\(n\)\\displaystyle\\phi^\{\\prime\}\(n\)=P′​\(n\)​N2−P⁡\(n\)⋅2​NN4=P′​\(n\)​N−2​P​\(n\)N3,\\displaystyle=\\frac\{P^\{\\prime\}\(n\)N^\{2\}\-P\(n\)\\cdot 2N\}\{N^\{4\}\}=\\frac\{P^\{\\prime\}\(n\)N\-2P\(n\)\}\{N^\{3\}\},\(40\)whereP′​\(n\)=2​α1​n​ε2\+α3​σg2P^\{\\prime\}\(n\)=2\\alpha\_\{1\}n\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\. Expanding the numerator:

\(2​α1​n​ε2\+α3​σg2\)​\(nf\+n\)−2​\(α1​n2​ε2\+α3​nf​σ2\+α3​n​σg2\)\\displaystyle\(2\\alpha\_\{1\}n\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\)\(n\_\{f\}\+n\)\-2\(\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\+\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}\)=2​α1​n​ε2​nf\+2​α1​n2​ε2\+α3​σg2​nf\+α3​σg2​n−2​α1​n2​ε2−2​α3​nf​σ2−2​α3​n​σg2\\displaystyle=2\\alpha\_\{1\}n\\varepsilon^\{2\}n\_\{f\}\+2\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}n\_\{f\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}n\-2\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\-2\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}\-2\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}=2​α1​ε2​nf​n\+α3​nf​\(σg2−2​σ2\)−α3​σg2​n\\displaystyle=2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\\,n\+\\alpha\_\{3\}n\_\{f\}\(\\sigma\_\{g\}^\{2\}\-2\\sigma^\{2\}\)\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}n=\(2​α1​ε2​nf−α3​σg2\)​n\+α3​nf​\(Δ−σ2\),\\displaystyle=\(2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\)\\,n\+\\alpha\_\{3\}n\_\{f\}\(\\Delta\-\\sigma^\{2\}\),\(41\)where we usedσg2−2​σ2=Δ−σ2\\sigma\_\{g\}^\{2\}\-2\\sigma^\{2\}=\\Delta\-\\sigma^\{2\}\. When its slope is nonzero, setting the numerator to zero yields Eq\. \([39](https://arxiv.org/html/2610.10974#A2.E39)\), with an interior solution exactly whenn∗\>0n^\{\*\}\>0\. When the slope is zero, the numerator is the constantα3​nf​\(Δ−σ2\)\\alpha\_\{3\}n\_\{f\}\(\\Delta\-\\sigma^\{2\}\), so the derivative either never vanishes or is identically zero\.

For the limit, divide numerator and denominator ofϕ⁡\(n\)\\phi\(n\)byn2n^\{2\}:

ϕ⁡\(n\)=α1​ε2\+α3​nf​σ2/n2\+α3​σg2/n\(nf/n\+1\)2→n→∞α1​ε2\.∎\\phi\(n\)=\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}\+\\alpha\_\{3\}n\_\{f\}\\sigma^\{2\}/n^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}/n\}\{\(n\_\{f\}/n\+1\)^\{2\}\}\\xrightarrow\{n\\to\\infty\}\\alpha\_\{1\}\\varepsilon^\{2\}\.\\qed

Define the slope and intercept of the numerator in Eq\. \([38](https://arxiv.org/html/2610.10974#A2.E38)\) as:

𝔞:=2​α1​ε2​nf−α3​σg2,𝔟:=α3​nf​\(Δ−σ2\)\.\\mathfrak\{a\}:=2\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\},\\qquad\\mathfrak\{b\}:=\\alpha\_\{3\}n\_\{f\}\(\\Delta\-\\sigma^\{2\}\)\.\(42\)
###### Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4)\(Four Regimes of Annotation Value\)\.

In all cases,ϕ⁡\(n\)→α1​ε2\\phi\(n\)\\to\\alpha\_\{1\}\\varepsilon^\{2\}asn→∞n\\to\\infty\. When𝔞=𝔟=0\\mathfrak\{a\}=\\mathfrak\{b\}=0,ϕ\\phiis constant; otherwise, it falls into four regimes:

1. \(I\)If𝔞≥0\\mathfrak\{a\}\\geq 0and𝔟≥0\\mathfrak\{b\}\\geq 0, thenϕ\\phiis increasing, so annotations are locally harmful\.
2. \(II\)If𝔞\>0\\mathfrak\{a\}\>0and𝔟<0\\mathfrak\{b\}<0, thenϕ\\phidecreases and then increases, with unique continuous minimizern∗=η​nf​\(σ2−Δ\)/\(2​ε2​nf−η​σg2\)n^\{\*\}=\\eta n\_\{f\}\(\\sigma^\{2\}\-\\Delta\)/\(2\\varepsilon^\{2\}n\_\{f\}\-\\eta\\sigma\_\{g\}^\{2\}\)\.
3. \(III\)If𝔞≤0\\mathfrak\{a\}\\leq 0and𝔟≤0\\mathfrak\{b\}\\leq 0, thenϕ\\phiis decreasing, so each additional annotation locally reduces the objective\.
4. \(IV\)If𝔞<0\\mathfrak\{a\}<0and𝔟\>0\\mathfrak\{b\}\>0, thenϕ\\phifirst increases and then decreases, with unique continuous maximizern¯=η​nf​\(Δ−σ2\)/\(η​σg2−2​ε2​nf\)\\bar\{n\}=\\eta n\_\{f\}\(\\Delta\-\\sigma^\{2\}\)/\(\\eta\\sigma\_\{g\}^\{2\}\-2\\varepsilon^\{2\}n\_\{f\}\)\. In this regime, a strict improvement over no annotation is possible if and only ifε2<η​σ2/nf\\varepsilon^\{2\}<\\eta\\sigma^\{2\}/n\_\{f\}\.

###### Proof\.

The numerator ofϕ′​\(n\)\\phi^\{\\prime\}\(n\)is𝔞​n\+𝔟\\mathfrak\{a\}n\+\\mathfrak\{b\}, and the denominator\(nf\+n\)3\>0\(n\_\{f\}\+n\)^\{3\}\>0for alln≥0n\\geq 0\. Hencesign⁡\(ϕ′​\(n\)\)=sign⁡\(𝔞​n\+𝔟\)\\operatorname\{sign\}\(\\phi^\{\\prime\}\(n\)\)=\\operatorname\{sign\}\(\\mathfrak\{a\}n\+\\mathfrak\{b\}\)forn\>0n\>0\. If𝔞=𝔟=0\\mathfrak\{a\}=\\mathfrak\{b\}=0, thenϕ′​\(n\)=0\\phi^\{\\prime\}\(n\)=0for everyn≥0n\\geq 0, soϕ⁡\(n\)=ϕ⁡\(0\)=α1​ε2\\phi\(n\)=\\phi\(0\)=\\alpha\_\{1\}\\varepsilon^\{2\}throughout\. Whenη=0\\eta=0, we haveα3=0\\alpha\_\{3\}=0andϕ⁡\(n\)=α1​ε2​n2/\(nf\+n\)2\\phi\(n\)=\\alpha\_\{1\}\\varepsilon^\{2\}n^\{2\}/\(n\_\{f\}\+n\)^\{2\}; thusε≠0\\varepsilon\\neq 0gives Regime \(I\), whileε=0\\varepsilon=0gives the constant zero objective, regardless ofΔ\\Delta\. Forη\>0\\eta\>0, equivalentlyα3\>0\\alpha\_\{3\}\>0, we translate the four sign patterns into the source conditions in Table[1](https://arxiv.org/html/2610.10974#S5.T1), excluding the constant case\.

Regime \(I\):𝔞≥0\\mathfrak\{a\}\\geq 0and𝔟≥0\\mathfrak\{b\}\\geq 0\(not both zero\)\. Then𝔞​n\+𝔟\>0\\mathfrak\{a\}n\+\\mathfrak\{b\}\>0for alln\>0n\>0, soϕ′​\(n\)\>0\\phi^\{\\prime\}\(n\)\>0\. The condition𝔟≥0\\mathfrak\{b\}\\geq 0isΔ≥σ2\\Delta\\geq\\sigma^\{2\}\. The condition𝔞≥0\\mathfrak\{a\}\\geq 0is2​α1​ε2​nf≥α3​σg22\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\\geq\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}, which givesε2≥α3​σg22​α1​nf=η​σg22​nf\\varepsilon^\{2\}\\geq\\frac\{\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\}\{2\\alpha\_\{1\}n\_\{f\}\}=\\frac\{\\eta\\sigma\_\{g\}^\{2\}\}\{2n\_\{f\}\}\.

Regime \(II\):𝔞\>0\\mathfrak\{a\}\>0and𝔟<0\\mathfrak\{b\}<0\. The affine function𝔞​n\+𝔟\\mathfrak\{a\}n\+\\mathfrak\{b\}starts negative atn=0n=0and increases, crossing zero atn∗=−𝔟/𝔞\>0n^\{\*\}=\-\\mathfrak\{b\}/\\mathfrak\{a\}\>0\. Forn<n∗n<n^\{\*\},ϕ′​\(n\)<0\\phi^\{\\prime\}\(n\)<0\(decreasing\); forn\>n∗n\>n^\{\*\},ϕ′​\(n\)\>0\\phi^\{\\prime\}\(n\)\>0\(increasing\)\. The condition𝔟<0\\mathfrak\{b\}<0isΔ<σ2\\Delta<\\sigma^\{2\}\. The condition𝔞\>0\\mathfrak\{a\}\>0givesε2\>η​σg22​nf\\varepsilon^\{2\}\>\\frac\{\\eta\\sigma\_\{g\}^\{2\}\}\{2n\_\{f\}\}\.

Regime \(III\):𝔞≤0,𝔟≤0\\mathfrak\{a\}\\leq 0,\\mathfrak\{b\}\\leq 0\(not both zero\) gives𝔞​n\+𝔟≤0\\mathfrak\{a\}n\+\\mathfrak\{b\}\\leq 0for alln\>0n\>0, with strict inequality whenever at least one of𝔞,𝔟\\mathfrak\{a\},\\mathfrak\{b\}is strictly negative\. The conditions areΔ≤σ2\\Delta\\leq\\sigma^\{2\}andε2≤η​σg2/\(2​nf\)\\varepsilon^\{2\}\\leq\\eta\\sigma\_\{g\}^\{2\}/\(2n\_\{f\}\), with at least one inequality strict\. In particular,Δ=σ2\\Delta=\\sigma^\{2\}is included whenε2<η​σ2/nf\\varepsilon^\{2\}<\\eta\\sigma^\{2\}/n\_\{f\}; equality in both conditions gives the constant case already treated\.

Regime \(IV\):The sign transition is reversed:𝔟\>0\\mathfrak\{b\}\>0givesϕ′​\(0\+\)\>0\\phi^\{\\prime\}\(0^\{\+\}\)\>0, and𝔞<0\\mathfrak\{a\}<0forces a sign change at the maximizern¯=−𝔟/𝔞\>0\\bar\{n\}=\-\\mathfrak\{b\}/\\mathfrak\{a\}\>0stated in the theorem\. For the sub\-case analysis, we examineϕ⁡\(n\)−ϕ⁡\(0\)\\phi\(n\)\-\\phi\(0\)\. The function increases up ton¯\\bar\{n\}, then decreases towardα1​ε2\\alpha\_\{1\}\\varepsilon^\{2\}\.

Sub\-case \(IVa\):α1​ε2<ϕ⁡\(0\)\\alpha\_\{1\}\\varepsilon^\{2\}<\\phi\(0\)\. Becauselimn→∞ϕ⁡\(n\)<ϕ⁡\(0\)\\lim\_\{n\\to\\infty\}\\phi\(n\)<\\phi\(0\), there is a finiteT\>n¯T\>\\bar\{n\}withϕ⁡\(T\)<ϕ⁡\(0\)\\phi\(T\)<\\phi\(0\)\. The intermediate value theorem on\[n¯,T\]\[\\bar\{n\},T\], together withϕ⁡\(n¯\)\>ϕ⁡\(0\)\\phi\(\\bar\{n\}\)\>\\phi\(0\)and strict decrease aftern¯\\bar\{n\}, gives a unique break\-even countnbe\>n¯n\_\{\\mathrm\{be\}\}\>\\bar\{n\}satisfyingϕ⁡\(nbe\)=ϕ⁡\(0\)\\phi\(n\_\{\\mathrm\{be\}\}\)=\\phi\(0\)\. Consequently,ϕ⁡\(n\)<ϕ⁡\(0\)\\phi\(n\)<\\phi\(0\)exactly whenn\>nben\>n\_\{\\mathrm\{be\}\}; under a finite capacity, this improvement is attainable only if such a count is feasible\.

Sub\-case \(IVb\):α1​ε2≥ϕ⁡\(0\)\\alpha\_\{1\}\\varepsilon^\{2\}\\geq\\phi\(0\)\. We showϕ⁡\(n\)≥ϕ⁡\(0\)\\phi\(n\)\\geq\\phi\(0\)for alln≥0n\\geq 0\. Sinceϕ\\phiis increasing on\(0,n¯\)\(0,\\bar\{n\}\), we haveϕ⁡\(n\)\>ϕ⁡\(0\)\\phi\(n\)\>\\phi\(0\)forn∈\(0,n¯\]n\\in\(0,\\bar\{n\}\]\. On\(n¯,∞\)\(\\bar\{n\},\\infty\),ϕ\\phiis strictly decreasing towardα1​ε2≥ϕ⁡\(0\)\\alpha\_\{1\}\\varepsilon^\{2\}\\geq\\phi\(0\)\. Thusϕ⁡\(n\)\>α1​ε2≥ϕ⁡\(0\)\\phi\(n\)\>\\alpha\_\{1\}\\varepsilon^\{2\}\\geq\\phi\(0\)for everyn\>n¯n\>\\bar\{n\}, proving the claim, including the boundary caseα1​ε2=ϕ⁡\(0\)\\alpha\_\{1\}\\varepsilon^\{2\}=\\phi\(0\)\.

The sub\-case conditionα1​ε2≶ϕ⁡\(0\)=α3​σ2/nf\\alpha\_\{1\}\\varepsilon^\{2\}\\lessgtr\\phi\(0\)=\\alpha\_\{3\}\\sigma^\{2\}/n\_\{f\}translates toε2≶η​σ2/nf\\varepsilon^\{2\}\\lessgtr\\eta\\sigma^\{2\}/n\_\{f\}\. ∎

##### Integer counts in Regime \(II\)\.

For the local problem with feasible countsn∈\{0,…,U\}n\\in\\\{0,\\ldots,U\\\}, whereUUis a nonnegative integer, it suffices to compareϕ\\phiatmin⁡\{⌊n∗⌋,U\}\\min\\\{\\lfloor n^\{\*\}\\rfloor,U\\\}andmin⁡\{⌈n∗⌉,U\}\\min\\\{\\lceil n^\{\*\}\\rceil,U\\\}\. Ifn∗≤Un^\{\*\}\\leq U, monotonicity on each side ofn∗n^\{\*\}implies that every feasible integer has an objective value at least as large as that at⌊n∗⌋\\lfloor n^\{\*\}\\rflooror⌈n∗⌉\\lceil n^\{\*\}\\rceil\. Ifn∗\>Un^\{\*\}\>U,ϕ\\phidecreases throughout\[0,U\]\[0,U\], so both capped candidates equal the optimumUU\. When the candidates coincide, includingU=0U=0or integern∗n^\{\*\}, only one evaluation is needed; if their objective values tie, either is optimal\.

### B\.5Asymptotic Behavior and Bias Floor

###### Corollary 1\(Diminishing Marginal Returns\)\.

In all regimes, the marginal effect of annotation vanishes asymptotically:

\|ϕ′​\(n\)\|=\|𝔞​n\+𝔟\|\(nf\+n\)3=O⁡\(1n2\)as​n→∞\.\|\\phi^\{\\prime\}\(n\)\|=\\frac\{\|\\mathfrak\{a\}\\,n\+\\mathfrak\{b\}\|\}\{\(n\_\{f\}\+n\)^\{3\}\}=O\\\!\\left\(\\frac\{1\}\{n^\{2\}\}\\right\)\\quad\\text\{as \}n\\to\\infty\.\(43\)In Regime \(II\), annotations beyondn∗n^\{\*\}increaseϕ\\phi, with marginal cost of magnitudeO⁡\(1/n2\)O\(1/n^\{2\}\)\. In Regime \(III\), and in Regime \(IV\) beyond the maximizern¯\\bar\{n\}, annotations decreaseϕ\\phi, with marginal benefit of magnitudeO⁡\(1/n2\)O\(1/n^\{2\}\)\. The discrete marginal changeδ​ϕ​\(n\)=ϕ⁡\(n\+1\)−ϕ⁡\(n\)\\delta\\phi\(n\)=\\phi\(n\+1\)\-\\phi\(n\)also has magnitudeO⁡\(1/n2\)O\(1/n^\{2\}\)\.

###### Proof\.

Forn≥1n\\geq 1,\|𝔞​n\+𝔟\|≤\(\|𝔞\|\+\|𝔟\|\)​n\|\\mathfrak\{a\}n\+\\mathfrak\{b\}\|\\leq\(\|\\mathfrak\{a\}\|\+\|\\mathfrak\{b\}\|\)nand\(nf\+n\)3≥n3\(n\_\{f\}\+n\)^\{3\}\\geq n^\{3\}, so\|ϕ′​\(n\)\|≤\(\|𝔞\|\+\|𝔟\|\)/n2\|\\phi^\{\\prime\}\(n\)\|\\leq\(\|\\mathfrak\{a\}\|\+\|\\mathfrak\{b\}\|\)/n^\{2\}\. The signs follow from Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4):ϕ′\>0\\phi^\{\\prime\}\>0after the minimum in Regime \(II\), whereasϕ′<0\\phi^\{\\prime\}<0throughout Regime \(III\) and after the maximum in Regime \(IV\)\. For integern≥1n\\geq 1, the mean value theorem givesδ​ϕ​\(n\)=ϕ′​\(ξn\)\\delta\\phi\(n\)=\\phi^\{\\prime\}\(\\xi\_\{n\}\)for someξn∈\(n,n\+1\)\\xi\_\{n\}\\in\(n,n\+1\), yielding the same bound on its magnitude\. ∎

###### Corollary 2\(Irreducible Bias Floor\)\.

Regardless of regime,limn→∞ϕ⁡\(n\)=α1​ε2\\lim\_\{n\\to\\infty\}\\phi\(n\)=\\alpha\_\{1\}\\varepsilon^\{2\}\. The asymptotic reduction relative to no annotation is

ϕ⁡\(0\)−α1​ε2=α1nf​\[η​σ2−nf​ε2\]\.\\phi\(0\)\-\\alpha\_\{1\}\\varepsilon^\{2\}=\\frac\{\\alpha\_\{1\}\}\{n\_\{f\}\}\\bigl\[\\eta\\,\\sigma^\{2\}\-n\_\{f\}\\varepsilon^\{2\}\\bigr\]\.\(44\)This quantity is positive if and only ifε2<η​σ2nf\\varepsilon^\{2\}<\\frac\{\\eta\\,\\sigma^\{2\}\}\{n\_\{f\}\}, which is exactly the condition separating sub\-cases \(IVa\) from \(IVb\)\. The condition holds throughout the nonconstant Regime \(III\), including the boundaryΔ=σ2\\Delta=\\sigma^\{2\}\. In Regime \(II\), the asymptotic floor may exceedϕ⁡\(0\)\\phi\(0\)whenε\\varepsilonis large; however, the local optimumϕ⁡\(n∗\)\\phi\(n^\{\*\}\)always satisfiesϕ⁡\(n∗\)<ϕ⁡\(0\)\\phi\(n^\{\*\}\)<\\phi\(0\)sinceϕ′​\(0\+\)<0\\phi^\{\\prime\}\(0^\{\+\}\)<0guarantees initial descent\.

###### Proof\.

The limit and the formula follow from Lemma[3](https://arxiv.org/html/2610.10974#Thmlemma3)\. Regime \(III\) requiresη\>0\\eta\>0, becauseη=0\\eta=0gives either Regime \(I\) or the constant case\. IfΔ<σ2\\Delta<\\sigma^\{2\}, thenε2≤η​σg2/\(2​nf\)<η​σ2/nf\\varepsilon^\{2\}\\leq\\eta\\sigma\_\{g\}^\{2\}/\(2n\_\{f\}\)<\\eta\\sigma^\{2\}/n\_\{f\}\. IfΔ=σ2\\Delta=\\sigma^\{2\}, then𝔟=0\\mathfrak\{b\}=0and the nonconstant Regime \(III\) requires𝔞<0\\mathfrak\{a\}<0, which givesε2<η​σ2/nf\\varepsilon^\{2\}<\\eta\\sigma^\{2\}/n\_\{f\}directly\. In the excluded constant case,ϕ⁡\(0\)=α1​ε2\\phi\(0\)=\\alpha\_\{1\}\\varepsilon^\{2\}, so the asymptotic reduction is zero\. For Regime \(II\):𝔟<0\\mathfrak\{b\}<0impliesϕ′​\(0\+\)=𝔟/nf3<0\\phi^\{\\prime\}\(0^\{\+\}\)=\\mathfrak\{b\}/n\_\{f\}^\{3\}<0, soϕ\\phiis initially decreasing\. Sincen∗n^\{\*\}is the unique minimum ofϕ\\phion\(0,∞\)\(0,\\infty\), continuity givesϕ⁡\(n∗\)<ϕ⁡\(ϵ\)<ϕ⁡\(0\)\\phi\(n^\{\*\}\)<\\phi\(\\epsilon\)<\\phi\(0\)for sufficiently smallϵ\>0\\epsilon\>0\. ∎

### B\.6Cross\-Action Coupling within a Context

When multiple actions are annotated within the same context, the marginal value of an annotation also depends on how it changes the aggregate bias inJ2J\_\{2\}\. We compute this correction to identify when it strengthens or weakens the local benefit derived above\.

###### Lemma 4\(Marginal change includingJ2J\_\{2\}\)\.

Consider adding one annotation \(fromnnton\+1n\+1\) for actionaaat contextss\. The total marginal change in the objective, including the bias correction termJ2J\_\{2\}, is:

δ​J​\(n\)=δ​ϕ​\(n\)−δ​J2​\(n\),\\delta J\(n\)=\\delta\\phi\(n\)\-\\delta J\_\{2\}\(n\),\(45\)whereδ​ϕ​\(n\)\\delta\\phi\(n\)is the local change from Lemma[2](https://arxiv.org/html/2610.10974#Thmlemma2)and:

δ​J2​\(n\)=d0​\(s\)​\[2​ℬs⋅πe​\(a\|s\)​nf​εN⁡\(N\+1\)\+\(πe​\(a\|s\)​nf​εN⁡\(N\+1\)\)2\],\\delta J\_\{2\}\(n\)=d\_\{0\}\(s\)\\left\[2\\mathcal\{B\}\_\{s\}\\cdot\\pi\_\{e\}\(a\|s\)\\frac\{n\_\{f\}\\varepsilon\}\{N\(N\+1\)\}\+\\left\(\\pi\_\{e\}\(a\|s\)\\frac\{n\_\{f\}\\varepsilon\}\{N\(N\+1\)\}\\right\)^\{2\}\\right\],\(46\)withℬs:=∑a′∈𝒜πe​\(a′\|s\)​B​\(s,a′,𝐧\)\\mathcal\{B\}\_\{s\}:=\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a^\{\\prime\}\|s\)B\(s,a^\{\\prime\};\\mathbf\{n\}\)being the current weighted bias accumulation, including default\-model biases at pairs withN⁡\(s,a′\)=0N\(s,a^\{\\prime\}\)=0\.

###### Proof\.

The bias accumulation for contextsschanges fromℬs\\mathcal\{B\}\_\{s\}toℬs′=ℬs\+πe​\(a\|s\)​δ​B​\(s,a\)\\mathcal\{B\}\_\{s\}^\{\\prime\}=\\mathcal\{B\}\_\{s\}\+\\pi\_\{e\}\(a\|s\)\\,\\delta B\(s,a\), where:

δ​B​\(s,a\)\\displaystyle\\delta B\(s,a\):=\(n\+1\)​εN\+1−n​εN=ε​\(n\+1\)​N−n⁡\(N\+1\)N⁡\(N\+1\)=nf​εN⁡\(N\+1\)\.\\displaystyle:=\\frac\{\(n\+1\)\\varepsilon\}\{N\+1\}\-\\frac\{n\\varepsilon\}\{N\}=\\varepsilon\\frac\{\(n\+1\)N\-n\(N\+1\)\}\{N\(N\+1\)\}=\\frac\{n\_\{f\}\\varepsilon\}\{N\(N\+1\)\}\.\(47\)Therefore:

δ​J2​\(n\)\\displaystyle\\delta J\_\{2\}\(n\)=d0​\(s\)​\[\(ℬs′\)2−ℬs2\]=d0​\(s\)​\[2​ℬs⋅πe​\(a\|s\)​δ​B\+\(πe​\(a\|s\)​δ​B\)2\]\.\\displaystyle=d\_\{0\}\(s\)\\bigl\[\(\\mathcal\{B\}\_\{s\}^\{\\prime\}\)^\{2\}\-\\mathcal\{B\}\_\{s\}^\{2\}\\bigr\]=d\_\{0\}\(s\)\\bigl\[2\\mathcal\{B\}\_\{s\}\\cdot\\pi\_\{e\}\(a\|s\)\\,\\delta B\+\\bigl\(\\pi\_\{e\}\(a\|s\)\\,\\delta B\\bigr\)^\{2\}\\bigr\]\.\(48\)Since−J2\-J\_\{2\}appears in the objective, the total change isδ​J=δ​ϕ−δ​J2\\delta J=\\delta\\phi\-\\delta J\_\{2\}\. ∎

###### Theorem 5\(Threshold including bias correction\)\.

Assumeα3\>0\\alpha\_\{3\}\>0\. If the aggregate bias before the first annotation for actionaaisℬs\(0\)=0\\mathcal\{B\}\_\{s\}^\{\(0\)\}=0, then this annotation does not increase the objective \(δ​J​\(0\)≤0\\delta J\(0\)\\leq 0\) if and only if

ε2≤\(nf\+1\)​σ2nf−Δ\.\\varepsilon^\{2\}\\;\\leq\\;\\frac\{\(n\_\{f\}\+1\)\\sigma^\{2\}\}\{n\_\{f\}\}\-\\Delta\.\(49\)Equality in \([49](https://arxiv.org/html/2610.10974#A2.E49)\) givesδ​J​\(0\)=0\\delta J\(0\)=0, whereas strict inequality givesδ​J​\(0\)<0\\delta J\(0\)<0\. When the right\-hand side of \([49](https://arxiv.org/html/2610.10974#A2.E49)\) is positive, includingJ2J\_\{2\}multiplies the allowable squared bias from Theorem[3](https://arxiv.org/html/2610.10974#Thmtheorem3)by1/η\>11/\\eta\>1\. For generalℬs\(0\)≠0\\mathcal\{B\}\_\{s\}^\{\(0\)\}\\neq 0, the threshold becomes

α3​ε2≤α3​\[\(nf\+1\)​σ2−nf​Δ\]nf\+2​d0​\(s\)​ℬs\(0\)​πe​\(a\|s\)​\(nf\+1\)​ε,\\alpha\_\{3\}\\,\\varepsilon^\{2\}\\;\\leq\\;\\frac\{\\alpha\_\{3\}\\bigl\[\(n\_\{f\}\+1\)\\sigma^\{2\}\-n\_\{f\}\\Delta\\bigr\]\}\{n\_\{f\}\}\+2d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}^\{\(0\)\}\\,\\pi\_\{e\}\(a\|s\)\\,\(n\_\{f\}\+1\)\\,\\varepsilon,\(50\)which couples the allowable bias of actionaato the existing aggregate bias of contextss\.

###### Proof\.

The conditionδ​J​\(0\)≤0\\delta J\(0\)\\leq 0is equivalent toδ​ϕ​\(0\)≤δ​J2​\(0\)\\delta\\phi\(0\)\\leq\\delta J\_\{2\}\(0\)\. Atn=0n=0,N=nfN=n\_\{f\}and the incrementδ​B​\(s,a\)=ε/\(nf\+1\)\\delta B\(s,a\)=\\varepsilon/\(n\_\{f\}\+1\)\(Eq\.[47](https://arxiv.org/html/2610.10974#A2.E47)\)\. Substitutingδ​ϕ​\(0\)\\delta\\phi\(0\)from Eq\. \([37](https://arxiv.org/html/2610.10974#A2.E37)\) andδ​J2​\(0\)\\delta J\_\{2\}\(0\)from Eq\. \([48](https://arxiv.org/html/2610.10974#A2.E48)\):

α1​ε2​nf\+α3​\(nf​Δ−\(nf\+1\)​σ2\)nf​\(nf\+1\)2≤d0​\(s\)​2​ℬs\(0\)​πe​\(a\|s\)​εnf\+1\+d0​\(s\)​πe​\(a\|s\)2​ε2\(nf\+1\)2\.\\frac\{\\alpha\_\{1\}\\varepsilon^\{2\}n\_\{f\}\+\\alpha\_\{3\}\(n\_\{f\}\\Delta\-\(n\_\{f\}\+1\)\\sigma^\{2\}\)\}\{n\_\{f\}\(n\_\{f\}\+1\)^\{2\}\}\\leq\\frac\{d\_\{0\}\(s\)\\,2\\mathcal\{B\}\_\{s\}^\{\(0\)\}\\pi\_\{e\}\(a\|s\)\\,\\varepsilon\}\{n\_\{f\}\+1\}\+\\frac\{d\_\{0\}\(s\)\\,\\pi\_\{e\}\(a\|s\)^\{2\}\\,\\varepsilon^\{2\}\}\{\(n\_\{f\}\+1\)^\{2\}\}\.\(51\)Multiplying both sides by\(nf\+1\)2\(n\_\{f\}\+1\)^\{2\}and rearranging, we collectε2\\varepsilon^\{2\}terms:

\(α1−d0​\(s\)​πe​\(a\|s\)2\)​ε2≤α3​\(\(nf\+1\)​σ2−nf​Δ\)nf\+2​d0​\(s\)​ℬs\(0\)​πe​\(a\|s\)​\(nf\+1\)​ε\.\\bigl\(\\alpha\_\{1\}\-d\_\{0\}\(s\)\\pi\_\{e\}\(a\|s\)^\{2\}\\bigr\)\\varepsilon^\{2\}\\leq\\frac\{\\alpha\_\{3\}\\bigl\(\(n\_\{f\}\+1\)\\sigma^\{2\}\-n\_\{f\}\\Delta\\bigr\)\}\{n\_\{f\}\}\+2d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}^\{\(0\)\}\\,\\pi\_\{e\}\(a\|s\)\\,\(n\_\{f\}\+1\)\\,\\varepsilon\.\(52\)The key identity is:

α1−d0​\(s\)​πe​\(a\|s\)2=d0​\(s\)​πe​\(a\|s\)2​\(1πb​\(a\|s\)−1\)=α3\.\\alpha\_\{1\}\-d\_\{0\}\(s\)\\pi\_\{e\}\(a\|s\)^\{2\}=d\_\{0\}\(s\)\\pi\_\{e\}\(a\|s\)^\{2\}\\\!\\left\(\\frac\{1\}\{\\pi\_\{b\}\(a\|s\)\}\-1\\right\)=\\alpha\_\{3\}\.\(53\)Substitute Eq\. \([53](https://arxiv.org/html/2610.10974#A2.E53)\) into LHS of Eq\. \([52](https://arxiv.org/html/2610.10974#A2.E52)\) yields Eq\. \([50](https://arxiv.org/html/2610.10974#A2.E50)\)\. Settingℬs\(0\)=0\\mathcal\{B\}\_\{s\}^\{\(0\)\}=0and dividing byα3\>0\\alpha\_\{3\}\>0gives Eq\. \([49](https://arxiv.org/html/2610.10974#A2.E49)\)\. All transformations preserve equality and strict inequality, proving the corresponding statements forδ​J​\(0\)=0\\delta J\(0\)=0andδ​J​\(0\)<0\\delta J\(0\)<0\. ∎

### B\.7Extension to Arbitrary Context\-Action Pairs

In \([2](https://arxiv.org/html/2610.10974#S4.E2)\), each aggregate biasℬs\\mathcal\{B\}\_\{s\}depends only on allocations within contextss\. Thus the objective is additive across contexts:J⁡\(𝐧\)=∑s∈𝒮Js​\(𝐧s\)J\(\\mathbf\{n\}\)=\\sum\_\{s\\in\\mathcal\{S\}\}J\_\{s\}\(\\mathbf\{n\}\_\{s\}\), where𝐧s:=\{nk\(s,a\):a∈𝒜,k∈𝒦\}\\mathbf\{n\}\_\{s\}:=\\\{n\_\{k\}\(s,a\):a\\in\\mathcal\{A\},\\ k\\in\\mathcal\{K\}\\\}andJsJ\_\{s\}contains that context’s terms\. The shared budget in \([3](https://arxiv.org/html/2610.10974#S4.E3)\) nevertheless couples the feasible choices across contexts\.

The four regimes in Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4)and the diminishing marginal effects in Corollary[1](https://arxiv.org/html/2610.10974#Thmcorollary1)describe the local objectiveϕ\\phi, which omitsJ2J\_\{2\}\. They do not directly characterize the coupled context objectiveJsJ\_\{s\}\. For a single annotation with other allocations held fixed, Theorem[5](https://arxiv.org/html/2610.10974#Thmtheorem5c)gives the full first\-annotation condition through the current aggregate bias\. Joint allocation must therefore account for both within\-context bias coupling and the shared budget, as in §[6](https://arxiv.org/html/2610.10974#S6)\.

### B\.8The Casenf​\(s,a\)=0n\_\{f\}\(s,a\)=0

Whennf​\(s,a\)=0n\_\{f\}\(s,a\)=0, the fitting data contain no observations for the pair\(s,a\)\(s,a\)\. This can occur even whenπb​\(a\|s\)\>0\\pi\_\{b\}\(a\|s\)\>0, so an empty fitting cell does not by itself violate common support\. The formulas in Theorem[4](https://arxiv.org/html/2610.10974#Thmtheorem4)requirenf\>0n\_\{f\}\>0\. We treat this case separately\.

#### B\.8\.1πb​\(a\|s\)\>0\\pi\_\{b\}\(a\|s\)\>0butnf​\(s,a\)=0n\_\{f\}\(s,a\)=0

We retain the single\-source setting and consider pairs withα1\>0\\alpha\_\{1\}\>0; ifα1=0\\alpha\_\{1\}=0, thenα3=0\\alpha\_\{3\}=0and the local objective is identically zero\. When no annotations have been acquired,N⁡\(s,a\)=0N\(s,a\)=0and we use the fixed defaultr^\+​\(s,a\)=r0​\(s,a\)\\hat\{r\}^\{\+\}\(s,a\)=r\_\{0\}\(s,a\)\. Then

ε^r^\+​\(s,a\)\|N=0=r0​\(s,a\)−μr​\(s,a\),Var𝒟⁡\(r^\+​\(s,a\)\)\|N=0=0\.\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)\\big\|\_\{N=0\}=r\_\{0\}\(s,a\)\-\\mu\_\{r\}\(s,a\),\\qquad\\operatorname\{Var\}\_\{\\mathcal\{D\}\}\(\\hat\{r\}^\{\+\}\(s,a\)\)\\big\|\_\{N=0\}=0\.\(54\)The baseline local objective is:

ϕ0:=α1​\(μr​\(s,a\)−r0​\(s,a\)\)2\.\\phi\_\{0\}:=\\alpha\_\{1\}\\bigl\(\\mu\_\{r\}\(s,a\)\-r\_\{0\}\(s,a\)\\bigr\)^\{2\}\.\(55\)Forn≥1n\\geq 1,N=nN=nand the local objective \(Eq\.[29](https://arxiv.org/html/2610.10974#A2.E29)\) becomes:

ϕ⁡\(n\)=α1​n2​ε2\+α3​n​σg2n2=α1​ε2\+α3​σg2n,n≥1\.\\phi\(n\)=\\frac\{\\alpha\_\{1\}n^\{2\}\\varepsilon^\{2\}\+\\alpha\_\{3\}n\\sigma\_\{g\}^\{2\}\}\{n^\{2\}\}=\\alpha\_\{1\}\\varepsilon^\{2\}\+\\frac\{\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\}\{n\},\\qquad n\\geq 1\.\(56\)Substitutingnf=0n\_\{f\}=0into Eq\. \([42](https://arxiv.org/html/2610.10974#A2.E42)\) gives𝔞=−α3​σg2≤0\\mathfrak\{a\}=\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\leq 0and𝔟=0\\mathfrak\{b\}=0\. The derivativeϕ′\(n\)=−α3σg2/n2≤0\\phi^\{\\prime\}\(n\)=\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}/n^\{2\}\\leq 0shows that the local objective is nonincreasing forn≥1n\\geq 1, with strict decrease exactly whenα3​σg2\>0\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\>0\.

###### Proposition 4\(Properties Whennf=0n\_\{f\}=0andπb​\(a\|s\)\>0\\pi\_\{b\}\(a\|s\)\>0\)\.

Fornf​\(s,a\)=0n\_\{f\}\(s,a\)=0andα1\>0\\alpha\_\{1\}\>0, the local objective has the following properties:

1. 1\.First Annotation ThresholdThe transition fromn=0n=0ton=1n=1satisfies: δ​ϕ​\(0→1\):=ϕ⁡\(1\)−ϕ0=α1​\[ε2−\(μr−r0\)2\]\+α3​σg2\.\\delta\\phi\(0\\to 1\):=\\phi\(1\)\-\\phi\_\{0\}=\\alpha\_\{1\}\\bigl\[\\varepsilon^\{2\}\-\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}\\bigr\]\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\.\(57\)This is positive \(i\.e\., the first annotation is locally harmful\) if and only if ε2\>\(μr−r0\)2−η​σg2\.\\varepsilon^\{2\}\>\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}\-\\eta\\,\\sigma\_\{g\}^\{2\}\.\(58\)Equality in \([58](https://arxiv.org/html/2610.10974#A2.E58)\) leaves the local objective unchanged, and reversing the strict inequality gives a strict decrease\. For an exact defaultr0=μrr\_\{0\}=\\mu\_\{r\}, the first annotation is locally harmful if and only ifε2\+η​σg2\>0\\varepsilon^\{2\}\+\\eta\\sigma\_\{g\}^\{2\}\>0\. Conversely, if the default is poor \(\(μr−r0\)2\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}large\), the first annotation can be beneficial even with high bias\.
2. 2\.Nonincreasing objective forn≥1n\\geq 1:The marginal benefit of the\(n\+1\)\(n\+1\)\-th annotation is −δ​ϕ​\(n\)=α3​σg2n⁡\(n\+1\)=O⁡\(1n2\)\.\-\\delta\\phi\(n\)=\\frac\{\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\}\{n\(n\+1\)\}=O\\\!\\left\(\\frac\{1\}\{n^\{2\}\}\\right\)\.\(59\)The decrease is strict ifα3​σg2\>0\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\>0; otherwise,ϕ⁡\(n\)=α1​ε2\\phi\(n\)=\\alpha\_\{1\}\\varepsilon^\{2\}for everyn≥1n\\geq 1\.
3. 3\.Bias is never diluted:The bias of the reward estimator isε^r^\+​\(s,a\)=ε\\hat\{\\varepsilon\}\_\{\\hat\{r\}^\{\+\}\}\(s,a\)=\\varepsilonfor alln≥1n\\geq 1, independent of the annotation count\. Without factual data, there is no unbiased component to attenuate the annotation bias\.
4. 4\.Irreducible floor:Forn≥1n\\geq 1, the local objective is bounded below byα1​ε2\\alpha\_\{1\}\\varepsilon^\{2\}, which cannot be reduced by increasingnn\. A net improvement over the baseline \(ϕ⁡\(n\)<ϕ0\\phi\(n\)<\\phi\_\{0\}for somen≥1n\\geq 1\) requires: ε2<\(μr−r0\)2\.\\varepsilon^\{2\}<\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}\.\(60\)That is, the annotation bias must be strictly smaller in magnitude than the default model’s bias\.

###### Proof\.

\(1\)From Eqs\. \([56](https://arxiv.org/html/2610.10974#A2.E56)\) and \([55](https://arxiv.org/html/2610.10974#A2.E55)\):

δ​ϕ​\(0→1\)\\displaystyle\\delta\\phi\(0\\to 1\)=ϕ⁡\(1\)−ϕ0\\displaystyle=\\phi\(1\)\-\\phi\_\{0\}=\[α1​ε2\+α3​σg2\]−α1​\(μr−r0\)2\\displaystyle=\\bigl\[\\alpha\_\{1\}\\varepsilon^\{2\}\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\\bigr\]\-\\alpha\_\{1\}\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}=α1​\[ε2−\(μr−r0\)2\]\+α3​σg2\.\\displaystyle=\\alpha\_\{1\}\\bigl\[\\varepsilon^\{2\}\-\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}\\bigr\]\+\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\.Becauseα1\>0\\alpha\_\{1\}\>0, the conditionδ​ϕ​\(0→1\)\>0\\delta\\phi\(0\\to 1\)\>0is equivalent to Eq\. \([58](https://arxiv.org/html/2610.10974#A2.E58)\), with equality or the reversed strict inequality corresponding to zero or negative change, respectively\. Substitutingr0=μrr\_\{0\}=\\mu\_\{r\}gives the exact\-default statement\.

\(2\)From Eq\. \([56](https://arxiv.org/html/2610.10974#A2.E56)\),δϕ\(n\)=α3σg2\(1/\(n\+1\)−1/n\)=−α3σg2/\(n\(n\+1\)\)≤0\\delta\\phi\(n\)=\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\(1/\(n\+1\)\-1/n\)=\-\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}/\(n\(n\+1\)\)\\leq 0\. The denominator is positive forn≥1n\\geq 1, so the decrease is strict exactly whenα3​σg2\>0\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}\>0, and its magnitude is at mostα3​σg2/n2\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}/n^\{2\}\. Ifα3​σg2=0\\alpha\_\{3\}\\sigma\_\{g\}^\{2\}=0, Eq\. \([56](https://arxiv.org/html/2610.10974#A2.E56)\) gives the stated constant value\.

\(3\)Whennf=0n\_\{f\}=0,B⁡\(s,a\)=n​εn=εB\(s,a\)=\\frac\{n\\varepsilon\}\{n\}=\\varepsilonfor alln≥1n\\geq 1\.

\(4\)Equation \([56](https://arxiv.org/html/2610.10974#A2.E56)\) givesϕ⁡\(n\)≥α1​ε2\\phi\(n\)\\geq\\alpha\_\{1\}\\varepsilon^\{2\}for everyn≥1n\\geq 1, withlimn→∞ϕ⁡\(n\)=α1​ε2\\lim\_\{n\\to\\infty\}\\phi\(n\)=\\alpha\_\{1\}\\varepsilon^\{2\}\. Ifϕ⁡\(n\)<ϕ0\\phi\(n\)<\\phi\_\{0\}for a feasiblen≥1n\\geq 1, thenα1​ε2<ϕ0=α1​\(μr−r0\)2\\alpha\_\{1\}\\varepsilon^\{2\}<\\phi\_\{0\}=\\alpha\_\{1\}\(\\mu\_\{r\}\-r\_\{0\}\)^\{2\}; dividing byα1\>0\\alpha\_\{1\}\>0proves the necessary condition \([60](https://arxiv.org/html/2610.10974#A2.E60)\)\.

∎

#### B\.8\.2πb​\(a\|s\)=0\\pi\_\{b\}\(a\|s\)=0andnf​\(s,a\)=0n\_\{f\}\(s,a\)=0

Whenπb​\(a\|s\)=0\\pi\_\{b\}\(a\|s\)=0, neither the fitting sample nor the evaluation sample contains observations of actionaaat contextss\. If alsoπe​\(a\|s\)=0\\pi\_\{e\}\(a\|s\)=0, the pair contributes nothing to the estimator orJJ, and the zero\-weight convention givesα1​\(s,a\)=α3​\(s,a\)=0\\alpha\_\{1\}\(s,a\)=\\alpha\_\{3\}\(s,a\)=0\. Ifπe​\(a\|s\)\>0\\pi\_\{e\}\(a\|s\)\>0, common support fails; at a context withd0​\(s\)\>0d\_\{0\}\(s\)\>0, the importance\-weighted correction cannot remove the reward\-model error for that action\. The unbiasedness guarantee and MSE decomposition in Theorem[1](https://arxiv.org/html/2610.10974#Thmtheorem1)therefore do not generally apply\. This does not imply infinite variance: with one context,πb​\(a1\|s\)=1\\pi\_\{b\}\(a\_\{1\}\|s\)=1, andπe​\(a2\|s\)=1\\pi\_\{e\}\(a\_\{2\}\|s\)=1fora1≠a2a\_\{1\}\\neq a\_\{2\}, the estimator in \([1](https://arxiv.org/html/2610.10974#S3.E1)\) reduces tor^\+​\(s,a2\)\\hat\{r\}^\{\+\}\(s,a\_\{2\}\), which has finite variance under the stated moment assumptions but may be biased\.

## Appendix CProperties of the Objective Function

This appendix establishes the four structural properties ofJ⁡\(𝐧\)J\(\\mathbf\{n\}\)claimed in Proposition[1](https://arxiv.org/html/2610.10974#Thmproposition1)of the main text\. Nonnegativity and boundedness hold for every valid instance; the remaining properties are established by counterexamples\. We organize the proof into four parts, summarized below:

1. \(I\)JJis non\-negative and bounded overℱ\\mathcal\{F\}\(Proposition[5](https://arxiv.org/html/2610.10974#Thmproposition5)\);
2. \(II\)JJis non\-separable across actions within the same context \(Proposition[6](https://arxiv.org/html/2610.10974#Thmproposition6)\);
3. \(III\)the continuous relaxation ofJJis neither convex nor concave \(Proposition[7](https://arxiv.org/html/2610.10974#Thmproposition7)\);
4. \(IV\)JJis neither submodular nor supermodular over annotation allocations \(Corollary[3](https://arxiv.org/html/2610.10974#Thmcorollary3)\)\.

### C\.1Decomposition and Notation

We useB⁡\(s,a\):=B⁡\(s,a,𝐧\)B\(s,a\):=B\(s,a;\\mathbf\{n\}\)andW⁡\(s,a\):=W⁡\(s,a,𝐧\)W\(s,a\):=W\(s,a;\\mathbf\{n\}\)as defined before \([2](https://arxiv.org/html/2610.10974#S4.E2)\), including the default\-model caseN⁡\(s,a\)=0N\(s,a\)=0\. The objective decomposes as

J⁡\(𝐧\)=J1​\(𝐧\)−J2​\(𝐧\)\+J3​\(𝐧\),J\(\\mathbf\{n\}\)\\;=\\;J\_\{1\}\(\\mathbf\{n\}\)\\;\-\\;J\_\{2\}\(\\mathbf\{n\}\)\\;\+\\;J\_\{3\}\(\\mathbf\{n\}\),where the three components are:

J1​\(𝐧\)\\displaystyle J\_\{1\}\(\\mathbf\{n\}\)=∑s∑aα1​\(s,a\)​B​\(s,a\)2,\\displaystyle=\\sum\_\{s\}\\sum\_\{a\}\\alpha\_\{1\}\(s,a\)\\,B\(s,a\)^\{2\},J2​\(𝐧\)\\displaystyle J\_\{2\}\(\\mathbf\{n\}\)=∑sd0​\(s\)​ℬs​\(𝐧\)2,\\displaystyle=\\sum\_\{s\}d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\},J3​\(𝐧\)\\displaystyle J\_\{3\}\(\\mathbf\{n\}\)=∑s∑aα3​\(s,a\)​W​\(s,a\)\.\\displaystyle=\\sum\_\{s\}\\sum\_\{a\}\\alpha\_\{3\}\(s,a\)W\(s,a\)\.
For a fixed context\-action pair\(s,a\)\(s,a\), we define:

N:=N⁡\(s,a\)=nf\+∑knk,\\displaystyle N\\;:=\\;N\(s,a\)=n\_\{f\}\+\\textstyle\\sum\_\{k\}n\_\{k\},nf:=nf​\(s,a\),\\displaystyle n\_\{f\}\\;:=\\;n\_\{f\}\(s,a\),P:=∑k=1Knk​εgk​\(s,a\),\\displaystyle P\\;:=\\;\\textstyle\\sum\_\{k=1\}^\{K\}n\_\{k\}\\,\\varepsilon\_\{g\_\{k\}\}\(s,a\),QD:=∑k=1Knk​Δgk​\(s,a\),\\displaystyle Q\_\{D\}\\;:=\\;\\textstyle\\sum\_\{k=1\}^\{K\}n\_\{k\}\\,\\Delta\_\{g\_\{k\}\}\(s,a\),B⁡\(s,a\)=P/N\(N\>0\),\\displaystyle B\(s,a\)=P/N\\quad\(N\>0\),σ2:=σr2​\(s,a\),\\displaystyle\\sigma^\{2\}\\;:=\\;\\sigma\_\{r\}^\{2\}\(s,a\),α1​\(s,a\):=d0​\(s\)​πe​\(a\|s\)2πb​\(a\|s\),\\displaystyle\\alpha\_\{1\}\(s,a\)\\;:=\\;d\_\{0\}\(s\)\\,\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\}\{\\pi\_\{b\}\(a\|s\)\},α3​\(s,a\)=\(1−πb​\(a\|s\)\)​α1​\(s,a\)\.\\displaystyle\\alpha\_\{3\}\(s,a\)=\\bigl\(1\-\\pi\_\{b\}\(a\|s\)\\bigr\)\\alpha\_\{1\}\(s,a\)\.ForN\>0N\>0,W⁡\(s,a\)=σ2/N\+QD/N2W\(s,a\)=\\sigma^\{2\}/N\+Q\_\{D\}/N^\{2\}; forN=0N=0, we useB⁡\(s,a\)=r0​\(s,a\)−μr​\(s,a\)B\(s,a\)=r\_\{0\}\(s,a\)\-\\mu\_\{r\}\(s,a\)andW⁡\(s,a\)=0W\(s,a\)=0\. The coefficients use the zero\-weight conventionα1=α3=0\\alpha\_\{1\}=\\alpha\_\{3\}=0whenπe​\(a\|s\)=0\\pi\_\{e\}\(a\|s\)=0\. The context\-level bias accumulation is

ℬs​\(𝐧\):=∑a∈𝒜πe​\(a\|s\)​B​\(s,a\)\.\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\\;:=\\;\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)\\,B\(s,a\)\.We writeEk:=εgk​\(s,a\)E\_\{k\}:=\\varepsilon\_\{g\_\{k\}\}\(s,a\)andDk:=Δgk​\(s,a\)D\_\{k\}:=\\Delta\_\{g\_\{k\}\}\(s,a\)when\(s,a\)\(s,a\)is clear from context\. The shorthandnjn\_\{j\}denotes the annotation countnj​\(s,a\)n\_\{j\}\(s,a\)from sourcejjfor the pair under consideration\.

We note the following elementary sign properties:

- •α1​\(s,a\)≥0\\alpha\_\{1\}\(s,a\)\\geq 0, with strict positivity exactly whend0​\(s\)\>0d\_\{0\}\(s\)\>0andπe​\(a\|s\)\>0\\pi\_\{e\}\(a\|s\)\>0under common support;
- •α3​\(s,a\)=α1​\(s,a\)−d0​\(s\)​πe​\(a\|s\)2=\(1−πb​\(a\|s\)\)​α1​\(s,a\)≥0\\alpha\_\{3\}\(s,a\)=\\alpha\_\{1\}\(s,a\)\-d\_\{0\}\(s\)\\pi\_\{e\}\(a\|s\)^\{2\}=\\bigl\(1\-\\pi\_\{b\}\(a\|s\)\\bigr\)\\alpha\_\{1\}\(s,a\)\\geq 0, with equality exactly whenα1​\(s,a\)=0\\alpha\_\{1\}\(s,a\)=0orπb​\(a\|s\)=1\\pi\_\{b\}\(a\|s\)=1\.

### C\.2First and Second\-Order Derivatives

We compute all partial derivatives with respect to the annotation countsnk:=nk​\(s,a\)n\_\{k\}:=n\_\{k\}\(s,a\)\. SinceJ1J\_\{1\}andJ3J\_\{3\}decompose over individual\(s,a\)\(s,a\)pairs whileJ2J\_\{2\}decomposes over contexts, the analysis is organized accordingly\. AtN⁡\(s,a\)=0N\(s,a\)=0, finite annotation changes are evaluated using the default definitions rather than the rational derivative formulas below\.

###### Lemma 5\(First\-order partial derivatives\)\.

For a fixed\(s,a\)\(s,a\)pair withN⁡\(s,a\)\>0N\(s,a\)\>0and annotation sourcek∈𝒦k\\in\\mathcal\{K\}:

∂J1\(s,a\)∂nk\\displaystyle\\frac\{\\partial J\_\{1\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\}=2​α1​P​\(Ek​N−P\)N3,\\displaystyle=\\frac\{2\\alpha\_\{1\}\\,P\\,\(E\_\{k\}N\-P\)\}\{N^\{3\}\},\(61\)∂J2\(s\)∂nk​\(s,a\)\\displaystyle\\frac\{\\partial J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\}=2​d0​\(s\)​ℬs​πe​\(a\|s\)​Ek​N−PN2,\\displaystyle=2\\,d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\\,\\pi\_\{e\}\(a\|s\)\\,\\frac\{E\_\{k\}N\-P\}\{N^\{2\}\},\(62\)∂J3\(s,a\)∂nk\\displaystyle\\frac\{\\partial J\_\{3\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\}=α3N3​\(\(Dk−σ2\)​N−2​QD\)\.\\displaystyle=\\frac\{\\alpha\_\{3\}\}\{N^\{3\}\}\\Big\(\(D\_\{k\}\-\\sigma^\{2\}\)\\,N\-2\\,Q\_\{D\}\\Big\)\.\(63\)

###### Proof\.

Eq\. \([61](https://arxiv.org/html/2610.10974#A3.E61)\)\.WriteJ1\(s,a\)=α1​\(P/N\)2J\_\{1\}^\{\(s,a\)\}=\\alpha\_\{1\}\\,\(P/N\)^\{2\}\. Noting∂P∂nk=Ek\\frac\{\\partial P\}\{\\partial n\_\{k\}\}=E\_\{k\}and∂N∂nk=1\\frac\{\\partial N\}\{\\partial n\_\{k\}\}=1, the chain rule gives

∂∂nk​\(PN\)=Ek​N−PN2,\\frac\{\\partial\}\{\\partial n\_\{k\}\}\\\!\\left\(\\frac\{P\}\{N\}\\right\)=\\frac\{E\_\{k\}\\,N\-P\}\{N^\{2\}\},hence∂J1\(s,a\)∂nk=2​α1⋅PN⋅Ek​N−PN2\\frac\{\\partial J\_\{1\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\}=2\\alpha\_\{1\}\\cdot\\frac\{P\}\{N\}\\cdot\\frac\{E\_\{k\}N\-P\}\{N^\{2\}\}, yielding \([61](https://arxiv.org/html/2610.10974#A3.E61)\)\.

Eq\. \([62](https://arxiv.org/html/2610.10974#A3.E62)\)\.Sinceℬs=∑a′πe​\(a′\|s\)​B​\(s,a′\)\\mathcal\{B\}\_\{s\}=\\sum\_\{a^\{\\prime\}\}\\pi\_\{e\}\(a^\{\\prime\}\|s\)\\,B\(s,a^\{\\prime\}\)and onlyB⁡\(s,a\)B\(s,a\)depends onnk​\(s,a\)n\_\{k\}\(s,a\):

∂ℬs∂nk​\(s,a\)=πe​\(a\|s\)​Ek​N−PN2\.\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{k\}\(s,a\)\}=\\pi\_\{e\}\(a\|s\)\\,\\frac\{E\_\{k\}N\-P\}\{N^\{2\}\}\.Then∂J2\(s\)∂nk​\(s,a\)=2​d0​\(s\)​ℬs⋅∂ℬs∂nk​\(s,a\)\\frac\{\\partial J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\}=2\\,d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\\cdot\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{k\}\(s,a\)\}gives \([62](https://arxiv.org/html/2610.10974#A3.E62)\)\.

Eq\. \([63](https://arxiv.org/html/2610.10974#A3.E63)\)\.WritingJ3\(s,a\)=α3​σ2​N−1\+α3​QD​N−2J\_\{3\}^\{\(s,a\)\}=\\alpha\_\{3\}\\,\\sigma^\{2\}N^\{\-1\}\+\\alpha\_\{3\}\\,Q\_\{D\}\\,N^\{\-2\}and differentiating:

∂J3\(s,a\)∂nk=α3​\(−σ2N2\+Dk​N−2​QDN3\)=α3N3​\(\(Dk−σ2\)​N−2​QD\)\.∎\\frac\{\\partial J\_\{3\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\}=\\alpha\_\{3\}\\\!\\left\(\-\\frac\{\\sigma^\{2\}\}\{N^\{2\}\}\+\\frac\{D\_\{k\}\\,N\-2\\,Q\_\{D\}\}\{N^\{3\}\}\\right\)=\\frac\{\\alpha\_\{3\}\}\{N^\{3\}\}\\Big\(\(D\_\{k\}\-\\sigma^\{2\}\)N\-2Q\_\{D\}\\Big\)\.\\qed

###### Lemma 6\(Hessian entries\)\.

For a fixed contextss, the following second\-order partial derivatives hold when every pair being differentiated has positive total count\.

##### \(i\)J1\(s,a\)J\_\{1\}^\{\(s,a\)\}: same pair, sourcesk,lk,l\.

∂2J1\(s,a\)∂nk​∂nl=2​α1N4​\(Ek​El​N2−2​\(Ek\+El\)​P​N\+3​P2\)\.\\frac\{\\partial^\{2\}J\_\{1\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}=\\frac\{2\\alpha\_\{1\}\}\{N^\{4\}\}\\Big\(E\_\{k\}E\_\{l\}\\,N^\{2\}\-2\(E\_\{k\}\+E\_\{l\}\)\\,P\\,N\+3P^\{2\}\\Big\)\.\(64\)

##### \(ii\)J2\(s\)J\_\{2\}^\{\(s\)\}: same actionaa, sourcesk,lk,l\.

∂2J2\(s\)∂nk​\(s,a\)​∂nl​\(s,a\)=2​d0​\(s\)​\[πe​\(a\|s\)2​\(Ek​N−P\)​\(El​N−P\)N4\+ℬs​πe​\(a\|s\)​\(2​P−\(Ek\+El\)​N\)N3\]\.\\frac\{\\partial^\{2\}J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a\)\}=2\\,d\_\{0\}\(s\)\\\!\\left\[\\frac\{\\pi\_\{e\}\(a\|s\)^\{2\}\\,\(E\_\{k\}N\{\-\}P\)\(E\_\{l\}N\{\-\}P\)\}\{N^\{4\}\}\+\\frac\{\\mathcal\{B\}\_\{s\}\\,\\pi\_\{e\}\(a\|s\)\\,\(2P\{\-\}\(E\_\{k\}\{\+\}E\_\{l\}\)N\)\}\{N^\{3\}\}\\right\]\\\!\.\(65\)

##### \(iii\)J2\(s\)J\_\{2\}^\{\(s\)\}: different actionsa≠a′a\\neq a^\{\\prime\}, sourcesk,lk,l\.

∂2J2\(s\)∂nk​\(s,a\)​∂nl​\(s,a′\)=2​d0​\(s\)​πe​\(a\|s\)​πe​\(a′\|s\)​\(Ek​Na−Pa\)​\(El​Na′−Pa′\)Na2​Na′2,\\frac\{\\partial^\{2\}J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a^\{\\prime\}\)\}=\\frac\{2\\,d\_\{0\}\(s\)\\,\\pi\_\{e\}\(a\|s\)\\,\\pi\_\{e\}\(a^\{\\prime\}\|s\)\\,\(E\_\{k\}N\_\{a\}\-P\_\{a\}\)\(E\_\{l\}N\_\{a^\{\\prime\}\}\-P\_\{a^\{\\prime\}\}\)\}\{N\_\{a\}^\{2\}\\,N\_\{a^\{\\prime\}\}^\{2\}\},\(66\)whereNa:=N⁡\(s,a\)N\_\{a\}:=N\(s,a\),Pa:=P⁡\(s,a\)P\_\{a\}:=P\(s,a\), etc\.

##### \(iv\)J3\(s,a\)J\_\{3\}^\{\(s,a\)\}: same pair, sourcesk,lk,l\.

∂2J3\(s,a\)∂nk​∂nl=2​α3N4​\(\(σ2−Dk−Dl\)​N\+3​QD\)\.\\frac\{\\partial^\{2\}J\_\{3\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}=\\frac\{2\\alpha\_\{3\}\}\{N^\{4\}\}\\Big\(\(\\sigma^\{2\}\-D\_\{k\}\-D\_\{l\}\)\\,N\+3\\,Q\_\{D\}\\Big\)\.\(67\)
###### Proof\.

All four identities follow from applying the quotient rule to the first\-order expressions in Lemma[5](https://arxiv.org/html/2610.10974#Thmlemma5)\. We detail each computation below\.

Part \(i\)\.Letu:=P/Nu:=P/Nso thatJ1\(s,a\)=α1​u2J\_\{1\}^\{\(s,a\)\}=\\alpha\_\{1\}u^\{2\}\. From Lemma[5](https://arxiv.org/html/2610.10974#Thmlemma5),∂u∂nk=\(Ek−u\)/N\\frac\{\\partial u\}\{\\partial n\_\{k\}\}=\(E\_\{k\}\-u\)/N\. Differentiating:

∂2\(α1​u2\)∂nk​∂nl\\displaystyle\\frac\{\\partial^\{2\}\(\\alpha\_\{1\}u^\{2\}\)\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}=2​α1​\(∂u∂nk​∂u∂nl\+u​∂2u∂nk​∂nl\)\.\\displaystyle=2\\alpha\_\{1\}\\\!\\left\(\\frac\{\\partial u\}\{\\partial n\_\{k\}\}\\,\\frac\{\\partial u\}\{\\partial n\_\{l\}\}\+u\\,\\frac\{\\partial^\{2\}u\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}\\right\)\.We compute

∂2u∂nk​∂nl=∂∂nl​\(Ek−uN\)=−∂u∂nl​N−\(Ek−u\)N2=2​u−Ek−ElN2\.\\frac\{\\partial^\{2\}u\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}=\\frac\{\\partial\}\{\\partial n\_\{l\}\}\\\!\\left\(\\frac\{E\_\{k\}\-u\}\{N\}\\right\)=\\frac\{\-\\frac\{\\partial u\}\{\\partial n\_\{l\}\}\\,N\-\(E\_\{k\}\-u\)\}\{N^\{2\}\}=\\frac\{2u\-E\_\{k\}\-E\_\{l\}\}\{N^\{2\}\}\.Substituting and collecting terms withu=P/Nu=P/N:

∂2J1\(s,a\)∂nk​∂nl\\displaystyle\\frac\{\\partial^\{2\}J\_\{1\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\\,\\partial n\_\{l\}\}=2​α1N2​\[\(Ek−u\)​\(El−u\)\+u⁡\(2​u−Ek−El\)\]\\displaystyle=\\frac\{2\\alpha\_\{1\}\}\{N^\{2\}\}\\\!\\left\[\(E\_\{k\}\-u\)\(E\_\{l\}\-u\)\+u\(2u\-E\_\{k\}\-E\_\{l\}\)\\right\]=2​α1N2​\[Ek​El−2​\(Ek\+El\)​u\+3​u2\],\\displaystyle=\\frac\{2\\alpha\_\{1\}\}\{N^\{2\}\}\\\!\\left\[E\_\{k\}E\_\{l\}\-2\(E\_\{k\}\+E\_\{l\}\)u\+3u^\{2\}\\right\],which yields \([64](https://arxiv.org/html/2610.10974#A3.E64)\) after substitutingu=P/Nu=P/N\.

Part \(ii\)\.Differentiating \([62](https://arxiv.org/html/2610.10974#A3.E62)\) with respect tonl​\(s,a\)n\_\{l\}\(s,a\):

∂2J2\(s\)∂nk​\(s,a\)​∂nl​\(s,a\)=2​d0​\(s\)​\[∂ℬs∂nl⋅πe​\(a\|s\)​Ek​N−PN2\+ℬs⋅πe​\(a\|s\)​∂∂nl​\(Ek​N−PN2\)\]\.\\frac\{\\partial^\{2\}J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a\)\}=2\\,d\_\{0\}\(s\)\\\!\\left\[\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{l\}\}\\\!\\cdot\\\!\\pi\_\{e\}\(a\|s\)\\frac\{E\_\{k\}N\{\-\}P\}\{N^\{2\}\}\+\\mathcal\{B\}\_\{s\}\\\!\\cdot\\\!\\pi\_\{e\}\(a\|s\)\\frac\{\\partial\}\{\\partial n\_\{l\}\}\\\!\\left\(\\frac\{E\_\{k\}N\{\-\}P\}\{N^\{2\}\}\\right\)\\right\]\.Using∂ℬs∂nl​\(s,a\)=πe​\(a\|s\)​El​N−PN2\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{l\}\(s,a\)\}=\\pi\_\{e\}\(a\|s\)\\frac\{E\_\{l\}N\-P\}\{N^\{2\}\}from Lemma[5](https://arxiv.org/html/2610.10974#Thmlemma5)and

∂∂nl​\(Ek​N−PN2\)=2​P−\(Ek\+El\)​NN3,\\frac\{\\partial\}\{\\partial n\_\{l\}\}\\\!\\left\(\\frac\{E\_\{k\}N\-P\}\{N^\{2\}\}\\right\)=\\frac\{2P\-\(E\_\{k\}\+E\_\{l\}\)N\}\{N^\{3\}\},\(computed via the quotient rule\) yields \([65](https://arxiv.org/html/2610.10974#A3.E65)\)\.

Part \(iii\)\.Fora≠a′a\\neq a^\{\\prime\}, sinceB⁡\(s,a\)B\(s,a\)depends only on\{nk​\(s,a\)\}k\\\{n\_\{k\}\(s,a\)\\\}\_\{k\}andB⁡\(s,a′\)B\(s,a^\{\\prime\}\)depends only on\{nk​\(s,a′\)\}k\\\{n\_\{k\}\(s,a^\{\\prime\}\)\\\}\_\{k\}, the termℬs⋅∂∂nl​\(s,a′\)​\(⋯\)\\mathcal\{B\}\_\{s\}\\cdot\\frac\{\\partial\}\{\\partial n\_\{l\}\(s,a^\{\\prime\}\)\}\(\\cdots\)does not contribute a cross\-action second derivative from the same factor\. Only the product of first derivatives remains:

∂2J2\(s\)∂nk​\(s,a\)​∂nl​\(s,a′\)=2​d0​\(s\)​∂ℬs∂nk​\(s,a\)​∂ℬs∂nl​\(s,a′\),\\frac\{\\partial^\{2\}J\_\{2\}^\{\(s\)\}\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a^\{\\prime\}\)\}=2\\,d\_\{0\}\(s\)\\,\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{k\}\(s,a\)\}\\,\\frac\{\\partial\\mathcal\{B\}\_\{s\}\}\{\\partial n\_\{l\}\(s,a^\{\\prime\}\)\},giving \([66](https://arxiv.org/html/2610.10974#A3.E66)\)\.

Part \(iv\)\.LetRk:=\(Dk−σ2\)​N−2​QDR\_\{k\}:=\(D\_\{k\}\-\\sigma^\{2\}\)N\-2Q\_\{D\}so that∂J3\(s,a\)∂nk=α3​Rk/N3\\frac\{\\partial J\_\{3\}^\{\(s,a\)\}\}\{\\partial n\_\{k\}\}=\\alpha\_\{3\}R\_\{k\}/N^\{3\}\. Then∂Rk∂nl=\(Dk−σ2\)−2​Dl\\frac\{\\partial R\_\{k\}\}\{\\partial n\_\{l\}\}=\(D\_\{k\}\-\\sigma^\{2\}\)\-2D\_\{l\}, and the quotient rule gives:

∂∂nl​\(RkN3\)\\displaystyle\\frac\{\\partial\}\{\\partial n\_\{l\}\}\\\!\\left\(\\frac\{R\_\{k\}\}\{N^\{3\}\}\\right\)=\(Dk−σ2−2​Dl\)​N−3​RkN4\.\\displaystyle=\\frac\{\(D\_\{k\}\-\\sigma^\{2\}\-2D\_\{l\}\)\\,N\-3R\_\{k\}\}\{N^\{4\}\}\.Expanding3​Rk=3​\(Dk−σ2\)​N−6​QD3R\_\{k\}=3\(D\_\{k\}\-\\sigma^\{2\}\)N\-6Q\_\{D\}and simplifying:

\(Dk−σ2−2​Dl\)​N−3​\(Dk−σ2\)​N\+6​QD=2​\(σ2−Dk−Dl\)​N\+6​QD\.∎\(D\_\{k\}\-\\sigma^\{2\}\-2D\_\{l\}\)N\-3\(D\_\{k\}\-\\sigma^\{2\}\)N\+6Q\_\{D\}=2\(\\sigma^\{2\}\-D\_\{k\}\-D\_\{l\}\)N\+6Q\_\{D\}\.\\qed

##### Hessian matrices in compact form\.

For a fixed\(s,a\)\(s,a\), define the vectors𝐞=\(E1,…,EK\)⊤\\mathbf\{e\}=\(E\_\{1\},\\ldots,E\_\{K\}\)^\{\\top\},𝐝=\(D1,…,DK\)⊤\\mathbf\{d\}=\(D\_\{1\},\\ldots,D\_\{K\}\)^\{\\top\}, and𝟏=\(1,…,1\)⊤∈ℝK\\mathbf\{1\}=\(1,\\ldots,1\)^\{\\top\}\\in\\mathbb\{R\}^\{K\}\. The Hessian matrices𝐇1,𝐇3∈ℝK×K\\mathbf\{H\}\_\{1\},\\mathbf\{H\}\_\{3\}\\in\\mathbb\{R\}^\{K\\times K\}with respect to\(n1,…,nK\)\(n\_\{1\},\\ldots,n\_\{K\}\)for the given\(s,a\)\(s,a\)pair are:

𝐇1\\displaystyle\\mathbf\{H\}\_\{1\}=2​α1N4​\(N2​𝐞𝐞⊤−2​N​P​\(𝐞𝟏⊤\+𝟏​𝐞⊤\)\+3​P2​11⊤\),\\displaystyle=\\frac\{2\\alpha\_\{1\}\}\{N^\{4\}\}\\\!\\left\(N^\{2\}\\,\\mathbf\{e\}\\mathbf\{e\}^\{\\top\}\-2NP\\,\(\\mathbf\{e\}\\mathbf\{1\}^\{\\top\}\+\\mathbf\{1\}\\mathbf\{e\}^\{\\top\}\)\+3P^\{2\}\\,\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\\right\),𝐇3\\displaystyle\\mathbf\{H\}\_\{3\}=2​α3N4​\(\(σ2​N\+3​QD\)​11⊤−N⁡\(𝐝𝟏⊤\+𝟏​𝐝⊤\)\)\.\\displaystyle=\\frac\{2\\alpha\_\{3\}\}\{N^\{4\}\}\\\!\\left\(\(\\sigma^\{2\}N\+3Q\_\{D\}\)\\,\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\-N\\,\(\\mathbf\{d\}\\mathbf\{1\}^\{\\top\}\+\\mathbf\{1\}\\mathbf\{d\}^\{\\top\}\)\\right\)\.Both are sums of rank\-one and rank\-two matrices; their spectral properties are parameter\-dependent and analyzed in §[C\.3](https://arxiv.org/html/2610.10974#A3.SS3)\.

### C\.3Structural Properties

We now prove the structural claims in Proposition[1](https://arxiv.org/html/2610.10974#Thmproposition1)\.

###### Proposition 5\(Non\-negativity and boundedness\)\.

For every feasible allocation𝐧∈ℱ\\mathbf\{n\}\\in\\mathcal\{F\},

0≤J⁡\(𝐧\)≤Jmax<∞\.0\\;\\leq\\;J\(\\mathbf\{n\}\)\\;\\leq\\;J\_\{\\max\}<\\infty\.

###### Proof\.

Lower bound\.ForJ1−J2J\_\{1\}\-J\_\{2\}: fix any contextssand definexa:=B⁡\(s,a\)x\_\{a\}:=B\(s,a\),wa:=πe​\(a\|s\)w\_\{a\}:=\\pi\_\{e\}\(a\|s\), andca:=πb​\(a\|s\)c\_\{a\}:=\\pi\_\{b\}\(a\|s\)\. In the following display, action sums run overaawithca\>0c\_\{a\}\>0; common support giveswa=0w\_\{a\}=0on the omitted actions, so their contributions are zero\. By the Cauchy–Schwarz inequality applied with weights\(ca\)\(\\sqrt\{c\_\{a\}\}\)and\(waca​xa\)\(\\frac\{w\_\{a\}\}\{\\sqrt\{c\_\{a\}\}\}\\,x\_\{a\}\):

\(∑awa​xa\)2⏟=ℬs2=\(∑aca⋅wa​xaca\)2≤\(∑aca\)​\(∑awa2ca​xa2\)=∑awa2ca​xa2,\\underbrace\{\\left\(\\sum\_\{a\}w\_\{a\}\\,x\_\{a\}\\right\)^\{2\}\}\_\{\\textstyle=\\mathcal\{B\}\_\{s\}^\{2\}\}=\\left\(\\sum\_\{a\}\\sqrt\{c\_\{a\}\}\\cdot\\frac\{w\_\{a\}\\,x\_\{a\}\}\{\\sqrt\{c\_\{a\}\}\}\\right\)^\{2\}\\leq\\left\(\\sum\_\{a\}c\_\{a\}\\right\)\\\!\\left\(\\sum\_\{a\}\\frac\{w\_\{a\}^\{2\}\}\{c\_\{a\}\}\\,x\_\{a\}^\{2\}\\right\)=\\sum\_\{a\}\\frac\{w\_\{a\}^\{2\}\}\{c\_\{a\}\}\\,x\_\{a\}^\{2\},where we used∑aca=∑aπb​\(a\|s\)=1\\sum\_\{a\}c\_\{a\}=\\sum\_\{a\}\\pi\_\{b\}\(a\|s\)=1\. Multiplying both sides byd0​\(s\)d\_\{0\}\(s\)and summing overssyieldsJ2​\(𝐧\)≤J1​\(𝐧\)J\_\{2\}\(\\mathbf\{n\}\)\\leq J\_\{1\}\(\\mathbf\{n\}\), henceJ1−J2≥0J\_\{1\}\-J\_\{2\}\\geq 0\.

ForJ3J\_\{3\}, pairs withN⁡\(s,a\)=0N\(s,a\)=0haveW⁡\(s,a\)=0W\(s,a\)=0and contribute zero\. For a pair withN⁡\(s,a\)\>0N\(s,a\)\>0, combining the two terms inW⁡\(s,a\)W\(s,a\)gives

W⁡\(s,a\)=σr2​\(s,a\)N⁡\(s,a\)\+QD​\(s,a\)N​\(s,a\)2=σr2​\(s,a\)​N​\(s,a\)\+QD​\(s,a\)N​\(s,a\)2,W\(s,a\)=\\frac\{\\sigma\_\{r\}^\{2\}\(s,a\)\}\{N\(s,a\)\}\+\\frac\{Q\_\{D\}\(s,a\)\}\{N\(s,a\)^\{2\}\}=\\frac\{\\sigma\_\{r\}^\{2\}\(s,a\)\\,N\(s,a\)\+Q\_\{D\}\(s,a\)\}\{N\(s,a\)^\{2\}\},and the numerator expands as

σr2​\(s,a\)​nf​\(s,a\)\+∑k=1Knk​\(s,a\)​\[σr2​\(s,a\)\+Δgk​\(s,a\)\]≥0,\\sigma\_\{r\}^\{2\}\(s,a\)\\,n\_\{f\}\(s,a\)\+\\sum\_\{k=1\}^\{K\}n\_\{k\}\(s,a\)\\,\\bigl\[\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{g\_\{k\}\}\(s,a\)\\bigr\]\\geq 0,sinceσr2​\(s,a\)≥0\\sigma\_\{r\}^\{2\}\(s,a\)\\geq 0,nf​\(s,a\),nk​\(s,a\)≥0n\_\{f\}\(s,a\),\\,n\_\{k\}\(s,a\)\\geq 0, andσr2​\(s,a\)\+Δgk​\(s,a\)=Var⁡\(gk,j\|s,a\)≥0\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{g\_\{k\}\}\(s,a\)=\\operatorname\{Var\}\(g\_\{k,j\}\|s,a\)\\geq 0for everykk\. The denominator is positive, soW⁡\(s,a\)≥0W\(s,a\)\\geq 0; together withα3​\(s,a\)≥0\\alpha\_\{3\}\(s,a\)\\geq 0, this shows that everyJ3J\_\{3\}summand is nonnegative and henceJ3≥0J\_\{3\}\\geq 0\. ThereforeJ⁡\(𝐧\)=\(J1−J2\)\+J3≥0J\(\\mathbf\{n\}\)=\(J\_\{1\}\-J\_\{2\}\)\+J\_\{3\}\\geq 0\.

Upper bound\.Each countnk​\(s,a\)n\_\{k\}\(s,a\)is a nonnegative integer bounded byNsN\_\{s\}, soℱ\\mathcal\{F\}is finite and contains the zero allocation\. Common support and the zero\-weight convention make all coefficientsα1\\alpha\_\{1\}andα3\\alpha\_\{3\}finite\. For positive total counts,N⁡\(s,a\)≥1N\(s,a\)\\geq 1, so finite reward and annotation moments give finiteB⁡\(s,a\)B\(s,a\)andW⁡\(s,a\)W\(s,a\); for zero counts, the fixed finite defaults give finiteB⁡\(s,a\)B\(s,a\)andW⁡\(s,a\)=0W\(s,a\)=0\. ThusJ⁡\(𝐧\)J\(\\mathbf\{n\}\)is finite for every𝐧∈ℱ\\mathbf\{n\}\\in\\mathcal\{F\}, andJmax:=max𝐧∈ℱ⁡J⁡\(𝐧\)<∞J\_\{\\max\}:=\\max\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}J\(\\mathbf\{n\}\)<\\infty\. ∎

###### Proposition 6\(Non\-separability across actions\)\.

There exist valid instances with a single contextssand\|𝒜\|≥2\|\\mathcal\{A\}\|\\geq 2for which no functions\{fa\}a∈𝒜\\\{f\_\{a\}\\\}\_\{a\\in\\mathcal\{A\}\}satisfy

J⁡\(𝐧\)=∑a∈𝒜fa​\(\{nk​\(s,a\)\}k∈𝒦\)\.J\(\\mathbf\{n\}\)=\\sum\_\{a\\in\\mathcal\{A\}\}f\_\{a\}\\\!\\big\(\\\{n\_\{k\}\(s,a\)\\\}\_\{k\\in\\mathcal\{K\}\}\\big\)\.

###### Proof\.

We first identify the cross\-action coupling in the continuous relaxation, then verify non\-separability on feasible integer allocations\.

SinceJ1J\_\{1\}andJ3J\_\{3\}decompose over individual\(s,a\)\(s,a\)pairs, they contribute zero to the cross\-action derivative\. The only contribution comes from−J2\-J\_\{2\}\. By Lemma[6](https://arxiv.org/html/2610.10974#Thmlemma6)\(iii\):

∂2J∂nk​\(s,a\)​∂nl​\(s,a′\)=−2​d0​\(s\)​πe​\(a\|s\)​πe​\(a′\|s\)​\(Ek​Na−Pa\)​\(El​Na′−Pa′\)Na2​Na′2\.\\frac\{\\partial^\{2\}J\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a^\{\\prime\}\)\}=\-\\frac\{2\\,d\_\{0\}\(s\)\\,\\pi\_\{e\}\(a\|s\)\\,\\pi\_\{e\}\(a^\{\\prime\}\|s\)\\,\(E\_\{k\}\\,N\_\{a\}\-P\_\{a\}\)\(E\_\{l\}\\,N\_\{a^\{\\prime\}\}\-P\_\{a^\{\\prime\}\}\)\}\{N\_\{a\}^\{2\}\\,N\_\{a^\{\\prime\}\}^\{2\}\}\.\(68\)For pairs withnf​\(s,a\),nf​\(s,a′\)\>0n\_\{f\}\(s,a\),n\_\{f\}\(s,a^\{\\prime\}\)\>0, at𝐧=𝟎\\mathbf\{n\}=\\mathbf\{0\}we havePa=Pa′=0P\_\{a\}=P\_\{a^\{\\prime\}\}=0, giving

∂2J∂nk​\(s,a\)​∂nl​\(s,a′\)\|𝐧=𝟎=−2​d0​\(s\)​πe​\(a\|s\)​πe​\(a′\|s\)​Ek​Elnf​\(s,a\)​nf​\(s,a′\),\\frac\{\\partial^\{2\}J\}\{\\partial n\_\{k\}\(s,a\)\\,\\partial n\_\{l\}\(s,a^\{\\prime\}\)\}\\bigg\|\_\{\\mathbf\{n\}=\\mathbf\{0\}\}=\-\\frac\{2\\,d\_\{0\}\(s\)\\,\\pi\_\{e\}\(a\|s\)\\,\\pi\_\{e\}\(a^\{\\prime\}\|s\)\\,E\_\{k\}\\,E\_\{l\}\}\{n\_\{f\}\(s,a\)\\,n\_\{f\}\(s,a^\{\\prime\}\)\},which is non\-zero wheneverd0​\(s\)\>0d\_\{0\}\(s\)\>0,πe​\(a\|s\),πe​\(a′\|s\)\>0\\pi\_\{e\}\(a\|s\),\\pi\_\{e\}\(a^\{\\prime\}\|s\)\>0, andEk,El≠0E\_\{k\},E\_\{l\}\\neq 0\.

For integer allocations, we choose one context, two actions, one source, andπb​\(ai\|s\)=πe​\(ai\|s\)=1/2\\pi\_\{b\}\(a\_\{i\}\|s\)=\\pi\_\{e\}\(a\_\{i\}\|s\)=1/2fori=1,2i=1,2\. We setnf​\(s,ai\)=1n\_\{f\}\(s,a\_\{i\}\)=1,εg1​\(s,ai\)=1\\varepsilon\_\{g\_\{1\}\}\(s,a\_\{i\}\)=1,σr2​\(s,ai\)=1\\sigma\_\{r\}^\{2\}\(s,a\_\{i\}\)=1,Δg1​\(s,ai\)=0\\Delta\_\{g\_\{1\}\}\(s,a\_\{i\}\)=0,c1=1c\_\{1\}=1, andκ=2\\kappa=2\. ThenNs=2N\_\{s\}=2and each action has annotation capacity one, so\(0,0\)\(0,0\),\(1,0\)\(1,0\),\(0,1\)\(0,1\), and\(1,1\)\(1,1\)are feasible annotation\-count vectors\. WritingJ⁡\(x,y\)J\(x,y\)for the objective at countsxxandyyon the two actions, substitution into \([2](https://arxiv.org/html/2610.10974#S4.E2)\) gives

J⁡\(0,0\)=12,J⁡\(1,0\)=J⁡\(0,1\)=716,J⁡\(1,1\)=14\.J\(0,0\)=\\frac\{1\}\{2\},\\qquad J\(1,0\)=J\(0,1\)=\\frac\{7\}\{16\},\\qquad J\(1,1\)=\\frac\{1\}\{4\}\.Consequently,J\(1,1\)−J\(1,0\)−J\(0,1\)\+J\(0,0\)=−1/8J\(1,1\)\-J\(1,0\)\-J\(0,1\)\+J\(0,0\)=\-1/8\. Any additive per\-action decomposition would make this difference zero, proving the claim\. ∎

###### Proposition 7\(Non\-convexity and non\-concavity\)\.

There exist valid instances for which the continuous relaxation ofJ⁡\(𝐧\)J\(\\mathbf\{n\}\)is neither convex nor concave over feasible allocations\.

###### Proof\.

We show non\-convexity and non\-concavity by exhibiting a feasible direction along which a diagonal entry of the Hessian of the continuous relaxation takes opposite signs at two configurations\.

We construct an instance with one contextss, two actions𝒜=\{a1,a2\}\\mathcal\{A\}=\\\{a\_\{1\},a\_\{2\}\\\}, a single annotation source \(K=1K=1\), andπb​\(a1\|s\)=πb​\(a2\|s\)=1/2\\pi\_\{b\}\(a\_\{1\}\|s\)=\\pi\_\{b\}\(a\_\{2\}\|s\)=1/2\. Setπe​\(a1\|s\)=1\\pi\_\{e\}\(a\_\{1\}\|s\)=1andπe​\(a2\|s\)=0\\pi\_\{e\}\(a\_\{2\}\|s\)=0, and fixn1​\(s,a2\)=0n\_\{1\}\(s,a\_\{2\}\)=0, so the only decision variable isn:=n1​\(s,a1\)n:=n\_\{1\}\(s,a\_\{1\}\)\. Write

E:=εg1​\(s,a1\),D:=Δg1​\(s,a1\),σ2:=σr2​\(s,a1\),nf:=nf​\(s,a1\),N:=nf\+n\.E:=\\varepsilon\_\{g\_\{1\}\}\(s,a\_\{1\}\),\\quad D:=\\Delta\_\{g\_\{1\}\}\(s,a\_\{1\}\),\\quad\\sigma^\{2\}:=\\sigma\_\{r\}^\{2\}\(s,a\_\{1\}\),\\quad n\_\{f\}:=n\_\{f\}\(s,a\_\{1\}\),\\quad N:=n\_\{f\}\+n\.We choose a positive integernfn\_\{f\}, setnf​\(s,a2\)=2​nfn\_\{f\}\(s,a\_\{2\}\)=2n\_\{f\},c1=1c\_\{1\}=1, andκ=2​nf\\kappa=2n\_\{f\}; thenNs=3​nfN\_\{s\}=3n\_\{f\}and the feasible interval along this direction is0≤n≤2​nf0\\leq n\\leq 2n\_\{f\}\. Under this configuration, only\(s,a1\)\(s,a\_\{1\}\)contributes to the bias\-related terms, andℬs​\(𝐧\)=πe​\(a1\|s\)​B​\(s,a1\)=B⁡\(s,a1\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)=\\pi\_\{e\}\(a\_\{1\}\|s\)\\,B\(s,a\_\{1\}\)=B\(s,a\_\{1\}\)\. Using the identityα1​\(s,a1\)−d0​\(s\)​πe​\(a1\|s\)2=α3​\(s,a1\)\\alpha\_\{1\}\(s,a\_\{1\}\)\-d\_\{0\}\(s\)\\pi\_\{e\}\(a\_\{1\}\|s\)^\{2\}=\\alpha\_\{3\}\(s,a\_\{1\}\)\(Eq\. \([53](https://arxiv.org/html/2610.10974#A2.E53)\)\), the contribution of\(s,a1\)\(s,a\_\{1\}\)toJ1−J2J\_\{1\}\-J\_\{2\}collapses intoα3​\(s,a1\)​B​\(s,a1\)2\\alpha\_\{3\}\(s,a\_\{1\}\)\\,B\(s,a\_\{1\}\)^\{2\}\. Thus, along this feasible direction,

J⁡\(n\)=α3​\[n2​E2N2\+σ2N\+n​DN2\],α3:=α3​\(s,a1\)\>0\.J\(n\)=\\alpha\_\{3\}\\\!\\left\[\\frac\{n^\{2\}E^\{2\}\}\{N^\{2\}\}\+\\frac\{\\sigma^\{2\}\}\{N\}\+\\frac\{nD\}\{N^\{2\}\}\\right\],\\qquad\\alpha\_\{3\}:=\\alpha\_\{3\}\(s,a\_\{1\}\)\>0\.Specializing the Hessian formulas of Lemma[6](https://arxiv.org/html/2610.10974#Thmlemma6)\(i\) and \(iv\) toK=1K=1\(so that the mixed partial reduces to the second derivative\), the diagonal entry of the Hessian is

J′′​\(n\)=2​α3N4​\[E2​nf​\(nf−2​n\)\+\(σ2−2​D\)​N\+3​n​D\]\.J^\{\\prime\\prime\}\(n\)=\\frac\{2\\alpha\_\{3\}\}\{N^\{4\}\}\\\!\\left\[E^\{2\}\\,n\_\{f\}\(n\_\{f\}\-2n\)\+\(\\sigma^\{2\}\-2D\)\\,N\+3nD\\right\]\.\(69\)
Non\-concavity\.Evaluating \([69](https://arxiv.org/html/2610.10974#A3.E69)\) atn=0n=0\(soN=nfN=n\_\{f\}\):

J′′​\(0\)=2​α3nf4​\[E2​nf2\+\(σ2−2​D\)​nf\]=2​α3​\(E2​nf\+σ2−2​D\)nf3\.J^\{\\prime\\prime\}\(0\)=\\frac\{2\\alpha\_\{3\}\}\{n\_\{f\}^\{4\}\}\\\!\\left\[E^\{2\}n\_\{f\}^\{2\}\+\(\\sigma^\{2\}\-2D\)n\_\{f\}\\right\]=\\frac\{2\\alpha\_\{3\}\\bigl\(E^\{2\}n\_\{f\}\+\\sigma^\{2\}\-2D\\bigr\)\}\{n\_\{f\}^\{3\}\}\.Forα3\>0\\alpha\_\{3\}\>0and parameters withE2​nf\+σ2−2​D\>0E^\{2\}n\_\{f\}\+\\sigma^\{2\}\-2D\>0\(e\.g\.E≠0E\\neq 0and0≤D≤σ2/20\\leq D\\leq\\sigma^\{2\}/2\),J′′​\(0\)\>0J^\{\\prime\\prime\}\(0\)\>0\. By continuity,J′′​\(n\)\>0J^\{\\prime\\prime\}\(n\)\>0at interior points of the feasible interval sufficiently close to zero, soJJis not concave\.

Non\-convexity\.Atn=nfn=n\_\{f\}\(soN=2​nfN=2n\_\{f\}\):

J′′​\(nf\)=2​α3\(2​nf\)4​\[−E2​nf2\+2​\(σ2−2​D\)​nf\+3​nf​D\]=α3​\(−E2​nf\+2​σ2−D\)8​nf3\.J^\{\\prime\\prime\}\(n\_\{f\}\)=\\frac\{2\\alpha\_\{3\}\}\{\(2n\_\{f\}\)^\{4\}\}\\\!\\left\[\-E^\{2\}n\_\{f\}^\{2\}\+2\(\\sigma^\{2\}\-2D\)n\_\{f\}\+3n\_\{f\}D\\right\]=\\frac\{\\alpha\_\{3\}\\bigl\(\-E^\{2\}n\_\{f\}\+2\\sigma^\{2\}\-D\\bigr\)\}\{8\\,n\_\{f\}^\{3\}\}\.Choosing the bias withE2\>\(2​σ2−D\)/nfE^\{2\}\>\(2\\sigma^\{2\}\-D\)/n\_\{f\}makesJ′′​\(nf\)<0J^\{\\prime\\prime\}\(n\_\{f\}\)<0, so the Hessian is not positive semidefinite;JJis not convex\.

Both regimes coexist on a single instance: pick anynf≥1n\_\{f\}\\geq 1,σ2\>0\\sigma^\{2\}\>0,DDwith0≤D≤σ2/20\\leq D\\leq\\sigma^\{2\}/2, and anEEsatisfyingE2\>\(2​σ2−D\)/nfE^\{2\}\>\(2\\sigma^\{2\}\-D\)/n\_\{f\}\. ThenJ′′​\(0\)\>0J^\{\\prime\\prime\}\(0\)\>0andJ′′​\(nf\)<0J^\{\\prime\\prime\}\(n\_\{f\}\)<0simultaneously, certifying that the continuous relaxation ofJJis neither convex nor concave\. ∎

###### Corollary 3\(Neither submodular nor supermodular\)\.

There exist valid instances for whichJ⁡\(𝐧\)J\(\\mathbf\{n\}\), viewed as a function on the integer latticeℤ\+\|𝒮\|​\|𝒜\|​K\\mathbb\{Z\}\_\{\+\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|K\}, is neither submodular nor supermodular, with both properties violated by feasible allocations\.

###### Proof\.

We test the defining lattice inequality on feasible integer allocations\. Submodularity requiresJ⁡\(𝐱\)\+J⁡\(𝐲\)≥J⁡\(𝐱∧𝐲\)\+J⁡\(𝐱∨𝐲\)J\(\\mathbf\{x\}\)\+J\(\\mathbf\{y\}\)\\geq J\(\\mathbf\{x\}\\wedge\\mathbf\{y\}\)\+J\(\\mathbf\{x\}\\vee\\mathbf\{y\}\), where∧\\wedgeand∨\\veedenote coordinate\-wise minimum and maximum; supermodularity requires the reverse inequality\. For distinct coordinatesi,ji,jwith unit vectors𝐞i,𝐞j\\mathbf\{e\}\_\{i\},\\mathbf\{e\}\_\{j\}, we define the mixed finite difference

δi​j​J​\(𝐧\):=J⁡\(𝐧\+𝐞i\+𝐞j\)−J⁡\(𝐧\+𝐞i\)−J⁡\(𝐧\+𝐞j\)\+J⁡\(𝐧\)\.\\delta\_\{ij\}J\(\\mathbf\{n\}\):=J\(\\mathbf\{n\}\+\\mathbf\{e\}\_\{i\}\+\\mathbf\{e\}\_\{j\}\)\-J\(\\mathbf\{n\}\+\\mathbf\{e\}\_\{i\}\)\-J\(\\mathbf\{n\}\+\\mathbf\{e\}\_\{j\}\)\+J\(\\mathbf\{n\}\)\.Taking𝐱=𝐧\+𝐞i\\mathbf\{x\}=\\mathbf\{n\}\+\\mathbf\{e\}\_\{i\}and𝐲=𝐧\+𝐞j\\mathbf\{y\}=\\mathbf\{n\}\+\\mathbf\{e\}\_\{j\}shows that submodularity requiresδi​j​J​\(𝐧\)≤0\\delta\_\{ij\}J\(\\mathbf\{n\}\)\\leq 0, whereas supermodularity requiresδi​j​J​\(𝐧\)≥0\\delta\_\{ij\}J\(\\mathbf\{n\}\)\\geq 0\.

We construct one instance violating both requirements, with one contextss, three actionsa1,a2,a3a\_\{1\},a\_\{2\},a\_\{3\}, and one source\. For each action, we setπb​\(ai\|s\)=πe​\(ai\|s\)=1/3\\pi\_\{b\}\(a\_\{i\}\|s\)=\\pi\_\{e\}\(a\_\{i\}\|s\)=1/3,nf​\(s,ai\)=1n\_\{f\}\(s,a\_\{i\}\)=1,σr2​\(s,ai\)=1\\sigma\_\{r\}^\{2\}\(s,a\_\{i\}\)=1, andΔg1​\(s,ai\)=0\\Delta\_\{g\_\{1\}\}\(s,a\_\{i\}\)=0\. We writeεi:=εg1​\(s,ai\)\\varepsilon\_\{i\}:=\\varepsilon\_\{g\_\{1\}\}\(s,a\_\{i\}\)and choose\(ε1,ε2,ε3\)=\(1,1,−1\)\(\\varepsilon\_\{1\},\\varepsilon\_\{2\},\\varepsilon\_\{3\}\)=\(1,1,\-1\)\. These moments can be realized by independent rewards taking values11and33with equal probability, with each annotation distributed as an independent reward draw plus the corresponding bias\. We setc1=1c\_\{1\}=1andκ=2\\kappa=2, givingNs=3N\_\{s\}=3and per\-action capacitiesNs−nf​\(s,ai\)=2N\_\{s\}\-n\_\{f\}\(s,a\_\{i\}\)=2\. Thus𝟎\\mathbf\{0\},𝐞i\\mathbf\{e\}\_\{i\},𝐞j\\mathbf\{e\}\_\{j\}, and𝐞i\+𝐞j\\mathbf\{e\}\_\{i\}\+\\mathbf\{e\}\_\{j\}are feasible for everyi≠ji\\neq j, where𝐞i\\mathbf\{e\}\_\{i\}now denotes one annotation for actionaia\_\{i\}\.

The termsJ1J\_\{1\}andJ3J\_\{3\}are sums of per\-action functions, so their mixed finite differences vanish\. At the zero allocation, all reward\-model biases are zero; adding one annotation for actionaia\_\{i\}givesB⁡\(s,ai\)=εi/2B\(s,a\_\{i\}\)=\\varepsilon\_\{i\}/2\. Consequently,ℬs​\(𝟎\)=0\\mathcal\{B\}\_\{s\}\(\\mathbf\{0\}\)=0,ℬs​\(𝐞i\)=εi/6\\mathcal\{B\}\_\{s\}\(\\mathbf\{e\}\_\{i\}\)=\\varepsilon\_\{i\}/6, andℬs​\(𝐞i\+𝐞j\)=\(εi\+εj\)/6\\mathcal\{B\}\_\{s\}\(\\mathbf\{e\}\_\{i\}\+\\mathbf\{e\}\_\{j\}\)=\(\\varepsilon\_\{i\}\+\\varepsilon\_\{j\}\)/6\. Only−J2=−ℬs2\-J\_\{2\}=\-\\mathcal\{B\}\_\{s\}^\{2\}contributes to the mixed difference, yielding

δi​j​J​\(𝟎\)=−\(εi\+εj\)2−εi2−εj236=−εi​εj18\.\\delta\_\{ij\}J\(\\mathbf\{0\}\)=\-\\frac\{\(\\varepsilon\_\{i\}\+\\varepsilon\_\{j\}\)^\{2\}\-\\varepsilon\_\{i\}^\{2\}\-\\varepsilon\_\{j\}^\{2\}\}\{36\}=\-\\frac\{\\varepsilon\_\{i\}\\varepsilon\_\{j\}\}\{18\}\.Henceδ12J\(𝟎\)=−1/18<0\\delta\_\{12\}J\(\\mathbf\{0\}\)=\-1/18<0violates supermodularity, whileδ13​J​\(𝟎\)=1/18\>0\\delta\_\{13\}J\(\\mathbf\{0\}\)=1/18\>0violates submodularity on the same instance\. The same violations persist with additional sources by setting their annotation counts to zero\. ∎

## Appendix DAnalysis of the Majorization\-Minimization Algorithm

This appendix provides the theoretical analysis of the proposed MM framework\. Throughout this section, we use the notation established in §[C\.1](https://arxiv.org/html/2610.10974#A3.SS1)\. In particular,ℬs​\(𝐧\):=∑a∈𝒜πe​\(a\|s\)​B​\(s,a\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\):=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{e\}\(a\|s\)B\(s,a\)denotes the context\-level bias accumulation, withBBandWWdefined in \([2](https://arxiv.org/html/2610.10974#S4.E2)\) and its preceding text, including the default\-model caseN⁡\(s,a\)=0N\(s,a\)=0\. We writeℱ\\mathcal\{F\}for the full feasible set defined by all constraints in \([3](https://arxiv.org/html/2610.10974#S4.E3)\), andℱ0\\mathcal\{F\}\_\{0\}for the feasible set with the budget constraint removed\.

### D\.1Pseudocode of the Algorithms

Algorithm[1](https://arxiv.org/html/2610.10974#alg1)retains the best feasible allocation found at each majorization\-minimization \(MM\) iteration, while Algorithm[2](https://arxiv.org/html/2610.10974#alg2)generates candidates by local source search and dynamic programming \(DP\)\. Both algorithms use factual fitting counts, reward moments, source profiles, policy probabilities, default rewards, and costs from the allocation problem; these fixed inputs are suppressed in subroutine calls\. The MM coefficients areω1​\(s,a\)=α1​\(s,a\)\\omega\_\{1\}\(s,a\)=\\alpha\_\{1\}\(s,a\),ω3​\(s,a\)=α3​\(s,a\)\\omega\_\{3\}\(s,a\)=\\alpha\_\{3\}\(s,a\), andω2\(t\)​\(s,a\)=−2​d0​\(s\)​ℬs​\(𝐧\(t\)\)​πe​\(a\|s\)\\omega\_\{2\}^\{\(t\)\}\(s,a\)=\-2d\_\{0\}\(s\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\pi\_\{e\}\(a\|s\)\. The local search and DP tables use the complete penalized costψs,a\(t\)​\(𝐳,λ\):=J~s,a\(t\)​\(𝐳\)\+λ​∑kck​zk\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\};\\lambda\):=\\tilde\{J\}\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\}\)\+\\lambda\\sum\_\{k\}c\_\{k\}z\_\{k\}, whereJ~s,a\(t\)\\tilde\{J\}\_\{s,a\}^\{\(t\)\}is the summand defined in §[6\.2](https://arxiv.org/html/2610.10974#S6.SS2)\. For each total countmm, the subproblem solver records the largest remaining positive swap gainδs,a​\(m,λ\)\\delta\_\{s,a\}\(m,\\lambda\), with zero assigned when no improving swap exists\. These residuals certify the approximation error even when the swap cap is reached \(Proposition[9](https://arxiv.org/html/2610.10974#Thmproposition9)\)\. All ties in source selection, swaps, DP, and candidate ordering use a fixed index order, and the outer update retains the current allocation unless the surrogate strictly improves\.

Algorithm 1Majorization\-minimization \(MM\) allocation with feasible\-candidate retention1:Fixed problem inputs above, budget

κ\\kappa, feasible

𝐧\(0\)\\mathbf\{n\}^\{\(0\)\}\(default

𝟎\\mathbf\{0\}\), iteration limits

Tmax,Tswap≥1T\_\{\\max\},T\_\{\\mathrm\{swap\}\}\\geq 1, tolerances

ϵ≥0\\epsilon\\geq 0,

ϵλ\>0\\epsilon\_\{\\lambda\}\>0, initial multiplier

λ0\>0\\lambda\_\{0\}\>0\.

2:Feasible allocation with objective no larger than

J⁡\(𝐧\(0\)\)J\(\\mathbf\{n\}^\{\(0\)\}\)\.

3:for

t=0,1,…,Tmax−1t=0,1,\\dots,T\_\{\\max\}\-1do

4:Compute MM coefficients

𝝎\(t\)\\boldsymbol\{\\omega\}^\{\(t\)\}from

𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}\.

5:

𝐧best←𝐧\(t\)\\mathbf\{n\}\_\{\\mathrm\{best\}\}\\leftarrow\\mathbf\{n\}^\{\(t\)\};

𝒬t←∅\\mathcal\{Q\}\_\{t\}\\leftarrow\\varnothing\.

6:

\(𝐧0,𝜹0\)←DP​\(0,𝝎\(t\),Tswap\)\(\\mathbf\{n\}\_\{0\},\\boldsymbol\{\\delta\}\_\{0\}\)\\leftarrow\\textsc\{DP\}\(0,\\boldsymbol\{\\omega\}^\{\(t\)\},T\_\{\\mathrm\{swap\}\}\)\.

7:If

C⁡\(𝐧0\)≤κC\(\\mathbf\{n\}\_\{0\}\)\\leq\\kappa, add

\(0,𝐧0,𝜹0\)\(0,\\mathbf\{n\}\_\{0\},\\boldsymbol\{\\delta\}\_\{0\}\)to

𝒬t\\mathcal\{Q\}\_\{t\}\.

8:

λlow←0\\lambda\_\{\\mathrm\{low\}\}\\leftarrow 0;

λhigh←λ0\\lambda\_\{\\mathrm\{high\}\}\\leftarrow\\lambda\_\{0\}\.

9:

\(𝐧high,𝜹high\)←DP​\(λhigh,𝝎\(t\),Tswap\)\(\\mathbf\{n\}\_\{\\mathrm\{high\}\},\\boldsymbol\{\\delta\}\_\{\\mathrm\{high\}\}\)\\leftarrow\\textsc\{DP\}\(\\lambda\_\{\\mathrm\{high\}\},\\boldsymbol\{\\omega\}^\{\(t\)\},T\_\{\\mathrm\{swap\}\}\)\.

10:while

C⁡\(𝐧high\)\>κC\(\\mathbf\{n\}\_\{\\mathrm\{high\}\}\)\>\\kappado

11:

λlow←λhigh\\lambda\_\{\\mathrm\{low\}\}\\leftarrow\\lambda\_\{\\mathrm\{high\}\};

λhigh←2​λhigh\\lambda\_\{\\mathrm\{high\}\}\\leftarrow 2\\lambda\_\{\\mathrm\{high\}\}\.

12:

\(𝐧high,𝜹high\)←DP​\(λhigh,𝝎\(t\),Tswap\)\(\\mathbf\{n\}\_\{\\mathrm\{high\}\},\\boldsymbol\{\\delta\}\_\{\\mathrm\{high\}\}\)\\leftarrow\\textsc\{DP\}\(\\lambda\_\{\\mathrm\{high\}\},\\boldsymbol\{\\omega\}^\{\(t\)\},T\_\{\\mathrm\{swap\}\}\)\.

13:endwhile

14:Add

\(λhigh,𝐧high,𝜹high\)\(\\lambda\_\{\\mathrm\{high\}\},\\mathbf\{n\}\_\{\\mathrm\{high\}\},\\boldsymbol\{\\delta\}\_\{\\mathrm\{high\}\}\)to

𝒬t\\mathcal\{Q\}\_\{t\}\.

15:while

λhigh−λlow\>ϵλ\\lambda\_\{\\mathrm\{high\}\}\-\\lambda\_\{\\mathrm\{low\}\}\>\\epsilon\_\{\\lambda\}do

16:

λmid←\(λlow\+λhigh\)/2\\lambda\_\{\\mathrm\{mid\}\}\\leftarrow\(\\lambda\_\{\\mathrm\{low\}\}\+\\lambda\_\{\\mathrm\{high\}\}\)/2\.

17:

\(𝐧′,𝜹′\)←DP​\(λmid,𝝎\(t\),Tswap\)\(\\mathbf\{n\}^\{\\prime\},\\boldsymbol\{\\delta\}^\{\\prime\}\)\\leftarrow\\textsc\{DP\}\(\\lambda\_\{\\mathrm\{mid\}\},\\boldsymbol\{\\omega\}^\{\(t\)\},T\_\{\\mathrm\{swap\}\}\)\.

18:if

C⁡\(𝐧′\)\>κC\(\\mathbf\{n\}^\{\\prime\}\)\>\\kappathen

19:

λlow←λmid\\lambda\_\{\\mathrm\{low\}\}\\leftarrow\\lambda\_\{\\mathrm\{mid\}\}\.

20:else

21:

λhigh←λmid\\lambda\_\{\\mathrm\{high\}\}\\leftarrow\\lambda\_\{\\mathrm\{mid\}\}\.

22:Add

\(λmid,𝐧′,𝜹′\)\(\\lambda\_\{\\mathrm\{mid\}\},\\mathbf\{n\}^\{\\prime\},\\boldsymbol\{\\delta\}^\{\\prime\}\)to

𝒬t\\mathcal\{Q\}\_\{t\}\.

23:endif

24:endwhile

25:foreach

\(λ,𝐧′,𝜹′\)∈𝒬t\(\\lambda,\\mathbf\{n\}^\{\\prime\},\\boldsymbol\{\\delta\}^\{\\prime\}\)\\in\\mathcal\{Q\}\_\{t\}do

26:if

J~​\(𝐧′,𝐧\(t\)\)<J~​\(𝐧best,𝐧\(t\)\)\\tilde\{J\}\(\\mathbf\{n\}^\{\\prime\};\\mathbf\{n\}^\{\(t\)\}\)<\\tilde\{J\}\(\\mathbf\{n\}\_\{\\mathrm\{best\}\};\\mathbf\{n\}^\{\(t\)\}\)then

27:

𝐧best←𝐧′\\mathbf\{n\}\_\{\\mathrm\{best\}\}\\leftarrow\\mathbf\{n\}^\{\\prime\}\.

28:endif

29:endfor

30:

𝐧\(t\+1\)←𝐧best\\mathbf\{n\}^\{\(t\+1\)\}\\leftarrow\\mathbf\{n\}\_\{\\mathrm\{best\}\}\.

31:if

𝐧\(t\+1\)=𝐧\(t\)\\mathbf\{n\}^\{\(t\+1\)\}=\\mathbf\{n\}^\{\(t\)\}or

J⁡\(𝐧\(t\)\)−J⁡\(𝐧\(t\+1\)\)≤ϵJ\(\\mathbf\{n\}^\{\(t\)\}\)\-J\(\\mathbf\{n\}^\{\(t\+1\)\}\)\\leq\\epsilonthen

32:break

33:endif

34:endfor

35:return

𝐧\(t\+1\)\\mathbf\{n\}^\{\(t\+1\)\}

Algorithm 2Capped source search and dynamic programming \(DP\)1:Fixed problem inputs, multiplier

λ≥0\\lambda\\geq 0, MM coefficients

𝝎\(t\)\\boldsymbol\{\\omega\}^\{\(t\)\}, swap cap

Tswap≥1T\_\{\\mathrm\{swap\}\}\\geq 1\.

2:Capacity\-feasible allocation

𝐧∈ℱ0\\mathbf\{n\}\\in\\mathcal\{F\}\_\{0\}and table of residual swap gains

𝜹\\boldsymbol\{\\delta\}\.

3:foreach context

s∈𝒮s\\in\\mathcal\{S\}do

4:foreach action

a∈𝒜a\\in\\mathcal\{A\}do

5:

Ms,a←Ns−nf​\(s,a\)M\_\{s,a\}\\leftarrow N\_\{s\}\-n\_\{f\}\(s,a\)\.

6:for

m=0,…,Ms,am=0,\\ldots,M\_\{s,a\}do

7:Initialize

𝐳\\mathbf\{z\}with

mmannotations from the cheapest source\.

8:if

m\>0m\>0and

K\>1K\>1then

9:for

i=1,…,Tswapi=1,\\ldots,T\_\{\\mathrm\{swap\}\}do

10:Choose

\(u,v\)\(u,v\),

u≠vu\\neq v,

zu≥1z\_\{u\}\\geq 1, maximizing

g=ψs,a\(t\)​\(𝐳,λ\)−ψs,a\(t\)​\(𝐳\+𝐞v−𝐞u,λ\)g=\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\};\\lambda\)\-\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\}\+\\mathbf\{e\}\_\{v\}\-\\mathbf\{e\}\_\{u\};\\lambda\)\.

11:If

g≤0g\\leq 0,break\.

12:

𝐳←𝐳\+𝐞v−𝐞u\\mathbf\{z\}\\leftarrow\\mathbf\{z\}\+\\mathbf\{e\}\_\{v\}\-\\mathbf\{e\}\_\{u\}\.

13:endfor

14:endif

15:Store

𝐧s,a♯​\[m\]←𝐳\\mathbf\{n\}^\{\\sharp\}\_\{s,a\}\[m\]\\leftarrow\\mathbf\{z\}and

Hs,a♯​\[m\]←ψs,a\(t\)​\(𝐳,λ\)H^\{\\sharp\}\_\{s,a\}\[m\]\\leftarrow\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{z\};\\lambda\)\.

16:Store

δs,a​\(m,λ\)\\delta\_\{s,a\}\(m,\\lambda\)as the largest remaining positive swap gain \(zero if none\)\.

17:endfor

18:endfor

19:Order actions as

a1,…,a\|𝒜\|a\_\{1\},\\ldots,a\_\{\|\\mathcal\{A\}\|\}; set

D0​\(0\)=0D\_\{0\}\(0\)=0and

D0​\(w\)=∞D\_\{0\}\(w\)=\\inftyfor

w\>0w\>0\.

20:for

j=1,…,\|𝒜\|j=1,\\ldots,\|\\mathcal\{A\}\|do

21:for

w=0,…,Nsw=0,\\ldots,N\_\{s\}do

22:

Dj​\(w\)←min0≤m≤min⁡\{w,Ms,aj\}⁡\{Dj−1​\(w−m\)\+Hs,aj♯​\[m\]\}D\_\{j\}\(w\)\\leftarrow\\min\_\{0\\leq m\\leq\\min\\\{w,M\_\{s,a\_\{j\}\}\\\}\}\\\{D\_\{j\-1\}\(w\-m\)\+H^\{\\sharp\}\_\{s,a\_\{j\}\}\[m\]\\\}\.

23:Store a minimizing count

mj​\(w\)m\_\{j\}\(w\)\.

24:endfor

25:endfor

26:

w⋆∈arg⁡min0≤w≤Ns​D\|𝒜\|​\(w\)w^\{\\star\}\\in\\arg\\min\_\{0\\leq w\\leq N\_\{s\}\}D\_\{\|\\mathcal\{A\}\|\}\(w\); set

w←w⋆w\\leftarrow w^\{\\star\}\.

27:for

j=\|𝒜\|,…,1j=\|\\mathcal\{A\}\|,\\ldots,1do

28:

m←mj​\(w\)m\\leftarrow m\_\{j\}\(w\);

𝐧s,aj←𝐧s,aj♯​\[m\]\\mathbf\{n\}\_\{s,a\_\{j\}\}\\leftarrow\\mathbf\{n\}^\{\\sharp\}\_\{s,a\_\{j\}\}\[m\];

w←w−mw\\leftarrow w\-m\.

29:endfor

30:endfor

31:return

𝐧,𝜹\\mathbf\{n\},\\boldsymbol\{\\delta\}\.

### D\.2Surrogate Construction and Majorization Property

The MM framework replaces the non\-convex objectiveJ⁡\(𝐧\)J\(\\mathbf\{n\}\)with a locally tight upper bound\. To remove the cross\-action coupling, we majorize−J2\(𝐧\)=−∑sd0\(s\)ℬs\(𝐧\)2\-J\_\{2\}\(\\mathbf\{n\}\)=\-\\sum\_\{s\}d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\}while retainingJ1J\_\{1\}andJ3J\_\{3\}\. We apply the tangent majorant of the concave scalar functionh⁡\(x\)=−x2h\(x\)=\-x^\{2\}atx=ℬs​\(𝐧\(t\)\)x=\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\), giving an expression affine inℬs​\(𝐧\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\.

###### Lemma 7\(Valid Majorization\)\.

For any fixed feasible allocation𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}, the full surrogateJ~full​\(𝐧,𝐧\(t\)\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)defined in Proposition[2](https://arxiv.org/html/2610.10974#Thmproposition2)of §[6\.1](https://arxiv.org/html/2610.10974#S6.SS1)satisfies:

1. \(i\)\(Upper bound\)J~full​\(𝐧,𝐧\(t\)\)≥J⁡\(𝐧\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)\\geq J\(\\mathbf\{n\}\)for all𝐧∈ℱ\\mathbf\{n\}\\in\\mathcal\{F\}\.
2. \(ii\)\(Tangency\)J~full​\(𝐧\(t\),𝐧\(t\)\)=J⁡\(𝐧\(t\)\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)=J\(\\mathbf\{n\}^\{\(t\)\}\)\.

Moreover, the majorization gap admits the closed\-form expression:

J~full​\(𝐧,𝐧\(t\)\)−J⁡\(𝐧\)=∑s∈𝒮d0​\(s\)​\(ℬs​\(𝐧\)−ℬs​\(𝐧\(t\)\)\)2\.\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)\-J\(\\mathbf\{n\}\)=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\bigr\)^\{2\}\.\(70\)

###### Proof\.

We fix𝐧\(t\)∈ℱ\\mathbf\{n\}^\{\(t\)\}\\in\\mathcal\{F\}and any𝐧∈ℱ\\mathbf\{n\}\\in\\mathcal\{F\}\. Becauseh⁡\(x\)=−x2h\(x\)=\-x^\{2\}is concave, its tangent at anyx0∈ℝx\_\{0\}\\in\\mathbb\{R\}gives a global upper bound:

−x2≤−x02−2​x0​\(x−x0\)=x02−2​x0​x,∀x∈ℝ,\-x^\{2\}\\;\\leq\\;\-x\_\{0\}^\{2\}\-2x\_\{0\}\(x\-x\_\{0\}\)\\;=\\;x\_\{0\}^\{2\}\-2x\_\{0\}\\,x,\\qquad\\forall\\,x\\in\\mathbb\{R\},with equality if and only ifx=x0x=x\_\{0\}\. Substitutingx=ℬs​\(𝐧\)x=\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)andx0=ℬs​\(𝐧\(t\)\)x\_\{0\}=\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\), multiplying by the nonnegative weightsd0​\(s\)d\_\{0\}\(s\), and summing over contexts gives

−∑sd0\(s\)ℬs\(𝐧\)2≤∑sd0\(s\)\[ℬs\(𝐧\(t\)\)2−2ℬs\(𝐧\(t\)\)ℬs\(𝐧\)\]\.\-\\sum\_\{s\}d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\}\\;\\leq\\;\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\[\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)^\{2\}\-2\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\\bigr\]\.The full surrogate \(including the constant term\) is:

J~full​\(𝐧,𝐧\(t\)\)=J1​\(𝐧\)\+∑sd0​\(s\)​\[ℬs​\(𝐧\(t\)\)2−2​ℬs​\(𝐧\(t\)\)​ℬs​\(𝐧\)\]\+J3​\(𝐧\)\.\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)=J\_\{1\}\(\\mathbf\{n\}\)\+\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\[\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)^\{2\}\-2\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\\bigr\]\+J\_\{3\}\(\\mathbf\{n\}\)\.\(71\)*Property \(i\)*follows from−J2​\(𝐧\)≤J~full​\(𝐧,𝐧\(t\)\)−J1​\(𝐧\)−J3​\(𝐧\)\-J\_\{2\}\(\\mathbf\{n\}\)\\leq\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)\-J\_\{1\}\(\\mathbf\{n\}\)\-J\_\{3\}\(\\mathbf\{n\}\)\. At𝐧=𝐧\(t\)\\mathbf\{n\}=\\mathbf\{n\}^\{\(t\)\}, the summed linearized term in \([71](https://arxiv.org/html/2610.10974#A4.E71)\) equals

∑sd0\(s\)\[\(ℬs\(t\)\)2−2\(ℬs\(t\)\)2\]=−∑sd0\(s\)\(ℬs\(t\)\)2=−J2\(𝐧\(t\)\)\.\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\[\(\\mathcal\{B\}\_\{s\}^\{\(t\)\}\)^\{2\}\-2\(\\mathcal\{B\}\_\{s\}^\{\(t\)\}\)^\{2\}\\bigr\]=\-\\sum\_\{s\}d\_\{0\}\(s\)\(\\mathcal\{B\}\_\{s\}^\{\(t\)\}\)^\{2\}=\-J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)\.Substitution into \([71](https://arxiv.org/html/2610.10974#A4.E71)\) givesJ~full​\(𝐧\(t\),𝐧\(t\)\)=J1​\(𝐧\(t\)\)−J2​\(𝐧\(t\)\)\+J3​\(𝐧\(t\)\)=J⁡\(𝐧\(t\)\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)=J\_\{1\}\(\\mathbf\{n\}^\{\(t\)\}\)\-J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)\+J\_\{3\}\(\\mathbf\{n\}^\{\(t\)\}\)=J\(\\mathbf\{n\}^\{\(t\)\}\), proving*property \(ii\)*\.

For the gap formula, we compute:

J~full​\(𝐧,𝐧\(t\)\)−J⁡\(𝐧\)\\displaystyle\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)\-J\(\\mathbf\{n\}\)=∑sd0​\(s\)​\[ℬs​\(𝐧\(t\)\)2−2​ℬs​\(𝐧\(t\)\)​ℬs​\(𝐧\)\+ℬs​\(𝐧\)2\]\\displaystyle=\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\[\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)^\{2\}\-2\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\+\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)^\{2\}\\bigr\]=∑sd0​\(s\)​\(ℬs​\(𝐧\)−ℬs​\(𝐧\(t\)\)\)2\.\\displaystyle=\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\bigr\)^\{2\}\.The optimization surrogate in \([8](https://arxiv.org/html/2610.10974#S6.E8)\) satisfies

J~​\(𝐧,𝐧\(t\)\)=J~full​\(𝐧,𝐧\(t\)\)−J2​\(𝐧\(t\)\)\.\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)=\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\-J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)\.BecauseJ2​\(𝐧\(t\)\)J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)is independent of𝐧\\mathbf\{n\}, both surrogates have the same minimizers and give the same comparisons between allocations at fixedtt\. At the current allocation, however,J~​\(𝐧\(t\),𝐧\(t\)\)=J⁡\(𝐧\(t\)\)−J2​\(𝐧\(t\)\)\\tilde\{J\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)=J\(\\mathbf\{n\}^\{\(t\)\}\)\-J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)\. ∎

###### Proposition 8\(Monotone Descent under Surrogate\-Improving Updates\)\.

Suppose at every iterationt≥0t\\geq 0the update𝐧\(t\+1\)∈ℱ\\mathbf\{n\}^\{\(t\+1\)\}\\in\\mathcal\{F\}satisfies the*surrogate\-improvement*condition

J~​\(𝐧\(t\+1\),𝐧\(t\)\)≤J~​\(𝐧\(t\),𝐧\(t\)\)\.\\tilde\{J\}\\\!\\bigl\(\\mathbf\{n\}^\{\(t\+1\)\};\\,\\mathbf\{n\}^\{\(t\)\}\\bigr\)\\;\\leq\\;\\tilde\{J\}\\\!\\bigl\(\\mathbf\{n\}^\{\(t\)\};\\,\\mathbf\{n\}^\{\(t\)\}\\bigr\)\.\(72\)Then the original objective is monotonically non\-increasing,

J⁡\(𝐧\(t\+1\)\)≤J⁡\(𝐧\(t\)\),∀t≥0,J\\\!\\bigl\(\\mathbf\{n\}^\{\(t\+1\)\}\\bigr\)\\;\\leq\\;J\\\!\\bigl\(\\mathbf\{n\}^\{\(t\)\}\\bigr\),\\qquad\\forall\\,t\\geq 0,and the sequence\{J⁡\(𝐧\(t\)\)\}\\\{J\(\\mathbf\{n\}^\{\(t\)\}\)\\\}converges\. The fallback rule𝐧\(t\+1\)←𝐧\(t\)\\mathbf\{n\}^\{\(t\+1\)\}\\leftarrow\\mathbf\{n\}^\{\(t\)\}used in Algorithm[1](https://arxiv.org/html/2610.10974#alg1)ensures \([72](https://arxiv.org/html/2610.10974#A4.E72)\) regardless of how accurately the inner Lagrangian subproblem is solved; in particular, exact surrogate minimization is*not*required for descent ofJJ\.

###### Proof\.

At fixedtt, addingJ2​\(𝐧\(t\)\)J\_\{2\}\(\\mathbf\{n\}^\{\(t\)\}\)to both sides of \([72](https://arxiv.org/html/2610.10974#A4.E72)\) gives the same improvement inequality forJ~full\\tilde\{J\}\_\{\\mathrm\{full\}\}\. Combining this inequality with the upper bound and tangency in Lemma[7](https://arxiv.org/html/2610.10974#Thmlemma7)yields

J⁡\(𝐧\(t\+1\)\)\\displaystyle J\(\\mathbf\{n\}^\{\(t\+1\)\}\)≤J~full​\(𝐧\(t\+1\),𝐧\(t\)\)\\displaystyle\\leq\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)≤J~full​\(𝐧\(t\),𝐧\(t\)\)\\displaystyle\\leq\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)=J⁡\(𝐧\(t\)\)\.\\displaystyle=J\(\\mathbf\{n\}^\{\(t\)\}\)\.The fallback update satisfies \([72](https://arxiv.org/html/2610.10974#A4.E72)\) with equality\. Proposition[5](https://arxiv.org/html/2610.10974#Thmproposition5)givesJ≥0J\\geq 0, so the non\-increasing objective sequence converges\. ∎

### D\.3Local Source Search and Exact Count Aggregation

At a fixed multiplierλ\\lambda, separability allows us to search source mixtures independently and then aggregate their costs across actions\. We bound the local\-search error before propagating it through the exact DP aggregation\.

###### Lemma 8\(Separability of the Surrogate\)\.

The surrogate objectiveJ~​\(𝐧,𝐧\(t\)\)\\tilde\{J\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)decomposes as:

J~​\(𝐧,𝐧\(t\)\)=∑s∈𝒮∑a∈𝒜ϕs,a\(t\)​\(\{nk​\(s,a\)\}k=1K\)\.\\tilde\{J\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)=\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\phi\_\{s,a\}^\{\(t\)\}\\\!\\bigl\(\\\{n\_\{k\}\(s,a\)\\\}\_\{k=1\}^\{K\}\\bigr\)\.Hereϕs,a\(t\)=J~s,a\(t\)\\phi\_\{s,a\}^\{\(t\)\}=\\tilde\{J\}\_\{s,a\}^\{\(t\)\}is the local summand from §[6\.2](https://arxiv.org/html/2610.10974#S6.SS2)and depends only on the annotation counts for\(s,a\)\(s,a\)\.

###### Proof\.

The termsJ1J\_\{1\}andJ3J\_\{3\}are already sums over individual\(s,a\)\(s,a\)pairs\. The linearized−J2\-J\_\{2\}contribution to the surrogate is:

−2∑sd0\(s\)ℬs\(𝐧\(t\)\)∑aπe\(a\|s\)B\(s,a\)=∑s∑aω2\(t\)\(s,a\)B\(s,a\),\-2\\sum\_\{s\}d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\\!\\sum\_\{a\}\\pi\_\{e\}\(a\|s\)\\,B\(s,a\)=\\sum\_\{s\}\\sum\_\{a\}\\omega\_\{2\}^\{\(t\)\}\(s,a\)\\,B\(s,a\),whereω2\(t\)​\(s,a\)=−2​d0​\(s\)​ℬs​\(𝐧\(t\)\)​πe​\(a\|s\)\\omega\_\{2\}^\{\(t\)\}\(s,a\)=\-2\\,d\_\{0\}\(s\)\\,\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\,\\pi\_\{e\}\(a\|s\)is a constant with respect to𝐧\\mathbf\{n\}\. Because eachB⁡\(s,a\)B\(s,a\)depends only on\{nk​\(s,a\)\}k\\\{n\_\{k\}\(s,a\)\\\}\_\{k\}, the optimization surrogateJ~\\tilde\{J\}is separable\. ∎

###### Lemma 9\(Convexity of the Local Cost for Fixed Total Count\)\.

Fix an MM iterationtt, a pair\(s,a\)\(s,a\), a multiplierλ≥0\\lambda\\geq 0, and a total countmmwithN0:=nf​\(s,a\)\+m\>0N\_\{0\}:=n\_\{f\}\(s,a\)\+m\>0\. HoldingN0N\_\{0\}constant, the complete local cost is

ψs,a\(t\)​\(𝐧K,λ\)\\displaystyle\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{n\}\_\{K\};\\lambda\)=ω1N02​\(∑knk​Ek\)2\+ω2\(t\)N0​∑knk​Ek\\displaystyle=\\frac\{\\omega\_\{1\}\}\{N\_\{0\}^\{2\}\}\\left\(\\sum\_\{k\}n\_\{k\}E\_\{k\}\\right\)^\{2\}\+\\frac\{\\omega\_\{2\}^\{\(t\)\}\}\{N\_\{0\}\}\\sum\_\{k\}n\_\{k\}E\_\{k\}\(73\)\+ω3​\(σr2​\(s,a\)N0\+∑knk​DkN02\)\+λ​∑kck​nk,\\displaystyle\+\\omega\_\{3\}\\left\(\\frac\{\\sigma\_\{r\}^\{2\}\(s,a\)\}\{N\_\{0\}\}\+\\frac\{\\sum\_\{k\}n\_\{k\}D\_\{k\}\}\{N\_\{0\}^\{2\}\}\\right\)\+\\lambda\\sum\_\{k\}c\_\{k\}n\_\{k\},and is convex in𝐧K:=\(n1,…,nK\)⊤\\mathbf\{n\}\_\{K\}:=\(n\_\{1\},\\ldots,n\_\{K\}\)^\{\\top\}on the simplex\{𝐧K≥𝟎,1⊤​𝐧K=m\}\\\{\\mathbf\{n\}\_\{K\}\\geq\\mathbf\{0\},\\ \\mathbf\{1\}^\{\\top\}\\mathbf\{n\}\_\{K\}=m\\\}\. The coefficientsω1=α1​\(s,a\)\\omega\_\{1\}=\\alpha\_\{1\}\(s,a\),ω3=α3​\(s,a\)\\omega\_\{3\}=\\alpha\_\{3\}\(s,a\), andω2\(t\)=−2​d0​\(s\)​ℬs​\(𝐧\(t\)\)​πe​\(a\|s\)\\omega\_\{2\}^\{\(t\)\}=\-2d\_\{0\}\(s\)\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\pi\_\{e\}\(a\|s\)are fixed during this search\.

###### Proof\.

The Hessian ofψ\\psiwith respect to𝐧K\\mathbf\{n\}\_\{K\}is

𝐇ψ=2​ω1N02​𝐞​𝐞⊤,𝐞=\(E1,…,EK\)⊤\.\\mathbf\{H\}\_\{\\psi\}=\\frac\{2\\omega\_\{1\}\}\{N\_\{0\}^\{2\}\}\\,\\mathbf\{e\}\\,\\mathbf\{e\}^\{\\top\},\\qquad\\mathbf\{e\}=\(E\_\{1\},\\ldots,E\_\{K\}\)^\{\\top\}\.For every direction𝐯∈ℝK\\mathbf\{v\}\\in\\mathbb\{R\}^\{K\},𝐯⊤​𝐇ψ​𝐯=2​ω1​\(𝐞⊤​𝐯\)2/N02≥0\\mathbf\{v\}^\{\\top\}\\mathbf\{H\}\_\{\\psi\}\\mathbf\{v\}=2\\omega\_\{1\}\(\\mathbf\{e\}^\{\\top\}\\mathbf\{v\}\)^\{2\}/N\_\{0\}^\{2\}\\geq 0becauseω1=α1​\(s,a\)≥0\\omega\_\{1\}=\\alpha\_\{1\}\(s,a\)\\geq 0\. Thus the quadratic extension with fixed denominator is convex, as is its restriction to the simplex\. Strict convexity is not required and need not hold because the Hessian has rank at most one\. ∎

The termω3​σr2​\(s,a\)/N0\\omega\_\{3\}\\sigma\_\{r\}^\{2\}\(s,a\)/N\_\{0\}is constant across source mixtures at fixedmm, but varies across counts and must be retained inHs,a♯​\(m,λ\)H^\{\\sharp\}\_\{s,a\}\(m,\\lambda\)\. Form=0m=0, the only source vector is𝟎\\mathbf\{0\}, so Algorithm[2](https://arxiv.org/html/2610.10974#alg2)stores its complete local cost without a swap search\. If alsonf​\(s,a\)=0n\_\{f\}\(s,a\)=0, that cost isω1​\(r0−μr\)2\+ω2\(t\)​\(r0−μr\)\\omega\_\{1\}\(r\_\{0\}\-\\mu\_\{r\}\)^\{2\}\+\\omega\_\{2\}^\{\(t\)\}\(r\_\{0\}\-\\mu\_\{r\}\), using the default bias and zero reward\-model variance\.

###### Proposition 9\(Pairwise\-Exchange Termination and Approximation Error\)\.

Fixtt,\(s,a\)\(s,a\),m≥1m\\geq 1, andλ≥0\\lambda\\geq 0\. The source search in Algorithm[2](https://arxiv.org/html/2610.10974#alg2)accepts at mostmin⁡\{Tswap,\(m\+K−1K−1\)−1\}\\min\\\{T\_\{\\mathrm\{swap\}\},\\binom\{m\+K\-1\}\{K\-1\}\-1\\\}swaps\. For its output𝐧♯\\mathbf\{n\}^\{\\sharp\}, letHs,a♯​\(m,λ\):=ψs,a\(t\)​\(𝐧♯,λ\)H^\{\\sharp\}\_\{s,a\}\(m,\\lambda\):=\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{n\}^\{\\sharp\};\\lambda\)and define the residual gain

δs,a​\(m,λ\):=maxu,v∈𝒦nu♯≥1⁡\[ψs,a\(t\)​\(𝐧♯,λ\)−ψs,a\(t\)​\(𝐧♯\+𝐞v−𝐞u,λ\)\]\+,\\delta\_\{s,a\}\(m,\\lambda\):=\\max\_\{\\begin\{subarray\}\{c\}u,v\\in\\mathcal\{K\}\\\\ n\_\{u\}^\{\\sharp\}\\geq 1\\end\{subarray\}\}\\left\[\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{n\}^\{\\sharp\};\\lambda\)\-\\psi\_\{s,a\}^\{\(t\)\}\(\\mathbf\{n\}^\{\\sharp\}\+\\mathbf\{e\}\_\{v\}\-\\mathbf\{e\}\_\{u\};\\lambda\)\\right\]\_\{\+\},where\[x\]\+:=max⁡\{x,0\}\[x\]\_\{\+\}:=\\max\\\{x,0\\\}and allowingu=vu=vincludes a zero\-gain move whenK=1K=1\. Relative to the exact fixed\-count minimumHs,a\(t,λ\)​\(m\)H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\), the output satisfies

0\\displaystyle 0≤Hs,a♯​\(m,λ\)−Hs,a\(t,λ\)​\(m\)≤ηs,a​\(m\)\+m​δs,a​\(m,λ\),\\displaystyle\\leq H^\{\\sharp\}\_\{s,a\}\(m,\\lambda\)\-H\_\{s,a\}^\{\(t,\\lambda\)\}\(m\)\\leq\\eta\_\{s,a\}\(m\)\+m\\,\\delta\_\{s,a\}\(m,\\lambda\),\(74\)ηs,a​\(m\)\\displaystyle\\eta\_\{s,a\}\(m\):=α1​\(s,a\)​m​Δ​Es,a2\(nf​\(s,a\)\+m\)2,ΔEs,a:=maxkEk−minkEk\.\\displaystyle:=\\frac\{\\alpha\_\{1\}\(s,a\)m\\Delta E\_\{s,a\}^\{\\,2\}\}\{\(n\_\{f\}\(s,a\)\+m\)^\{2\}\},\\qquad\\Delta E\_\{s,a\}:=\\max\_\{k\}E\_\{k\}\-\\min\_\{k\}E\_\{k\}\.The output is swap\-local if and only ifδs,a​\(m,λ\)=0\\delta\_\{s,a\}\(m,\\lambda\)=0\. With no swap cap, strict descent reaches such an output after at most\(m\+K−1K−1\)−1\\binom\{m\+K\-1\}\{K\-1\}\-1accepted swaps\. Form=0m=0, the local gap is zero, and we setηs,a​\(0\)=δs,a​\(0,λ\)=0\\eta\_\{s,a\}\(0\)=\\delta\_\{s,a\}\(0,\\lambda\)=0\. Whennf​\(s,a\)\>0n\_\{f\}\(s,a\)\>0,ηs,a​\(m\)≤α1​\(s,a\)​Δ​Es,a2/\[4​nf​\(s,a\)\]\\eta\_\{s,a\}\(m\)\\leq\\alpha\_\{1\}\(s,a\)\\Delta E\_\{s,a\}^\{\\,2\}/\[4n\_\{f\}\(s,a\)\]for everymm\.

###### Proof\.

*Finite termination\.*The integer simplex contains\(m\+K−1K−1\)\\binom\{m\+K\-1\}\{K\-1\}configurations\. Each accepted swap strictly decreasesψs,a\(t\)​\(⋅,λ\)\\psi\_\{s,a\}^\{\(t\)\}\(\\cdot;\\lambda\), so it cannot revisit a configuration\. Including the initial configuration gives at most\(m\+K−1K−1\)−1\\binom\{m\+K\-1\}\{K\-1\}\-1accepted swaps, and the cap imposes the additional limitTswapT\_\{\\mathrm\{swap\}\}\. Without a cap, the search stops only when there is no positive swap gain; this is preciselyδs,a​\(m,λ\)=0\\delta\_\{s,a\}\(m,\\lambda\)=0\. ForK=1K=1there are no swaps, and the sole source mixture is optimal\.

*Approximation error\.*Withm≥1m\\geq 1, the denominatorN0=nf​\(s,a\)\+mN\_\{0\}=n\_\{f\}\(s,a\)\+mis positive even if the factual count is zero\. We writeψ⁡\(𝐧K\)=ω1​\(𝐞⊤​𝐧K\)2/N02\+𝐋⊤​𝐧K\+C\\psi\(\\mathbf\{n\}\_\{K\}\)=\\omega\_\{1\}\(\\mathbf\{e\}^\{\\top\}\\mathbf\{n\}\_\{K\}\)^\{2\}/N\_\{0\}^\{2\}\+\\mathbf\{L\}^\{\\top\}\\mathbf\{n\}\_\{K\}\+C, where𝐞=\(E1,…,EK\)⊤\\mathbf\{e\}=\(E\_\{1\},\\ldots,E\_\{K\}\)^\{\\top\}and𝐋,C\\mathbf\{L\},Ccollect the fixed\-count linear and constant terms in \([73](https://arxiv.org/html/2610.10974#A4.E73)\)\. Its Hessian is𝐇=2​ω1​𝐞𝐞⊤/N02⪰0\\mathbf\{H\}=2\\omega\_\{1\}\\mathbf\{e\}\\mathbf\{e\}^\{\\top\}/N\_\{0\}^\{2\}\\succeq 0\.

For any feasible swap𝐝u,v=𝐞v−𝐞u\\mathbf\{d\}\_\{u,v\}=\\mathbf\{e\}\_\{v\}\-\\mathbf\{e\}\_\{u\}, the residual definition and the exact quadratic expansion give

−∇ψ\(𝐧♯\)⊤𝐝u,v≤δs,a\(m,λ\)\+12𝐝u,v⊤𝐇𝐝u,v=δs,a\(m,λ\)\+ω1​\(Ev−Eu\)2N02\.\-\\nabla\\psi\(\\mathbf\{n\}^\{\\sharp\}\)^\{\\\!\\top\}\\mathbf\{d\}\_\{u,v\}\\leq\\delta\_\{s,a\}\(m,\\lambda\)\+\\tfrac\{1\}\{2\}\\mathbf\{d\}\_\{u,v\}^\{\\top\}\\mathbf\{H\}\\mathbf\{d\}\_\{u,v\}=\\delta\_\{s,a\}\(m,\\lambda\)\+\\frac\{\\omega\_\{1\}\(E\_\{v\}\-E\_\{u\}\)^\{2\}\}\{N\_\{0\}^\{2\}\}\.\(75\)Let𝐳∗\\mathbf\{z\}^\{\*\}be a global minimizer on the same integer simplex\. Moving units from coordinates withnu♯\>zu∗n\_\{u\}^\{\\sharp\}\>z\_\{u\}^\{\*\}to coordinates withnv♯<zv∗n\_\{v\}^\{\\sharp\}<z\_\{v\}^\{\*\}gives𝐳∗−𝐧♯=∑u,vqu​v​𝐝u,v\\mathbf\{z\}^\{\*\}\-\\mathbf\{n\}^\{\\sharp\}=\\sum\_\{u,v\}q\_\{uv\}\\mathbf\{d\}\_\{u,v\}, wherequ​vq\_\{uv\}are nonnegative integers and every donor hasnu♯≥1n\_\{u\}^\{\\sharp\}\\geq 1\. The total moved mass is∑u,vqu​v=12​‖𝐳∗−𝐧♯‖1≤m\\sum\_\{u,v\}q\_\{uv\}=\\tfrac\{1\}\{2\}\\\|\\mathbf\{z\}^\{\*\}\-\\mathbf\{n\}^\{\\sharp\}\\\|\_\{1\}\\leq m\. Convexity and \([75](https://arxiv.org/html/2610.10974#A4.E75)\) therefore yield

ψ⁡\(𝐧♯\)−ψ⁡\(𝐳∗\)\\displaystyle\\psi\(\\mathbf\{n\}^\{\\sharp\}\)\-\\psi\(\\mathbf\{z\}^\{\*\}\)≤−∇ψ\(𝐧♯\)⊤\(𝐳∗−𝐧♯\)\\displaystyle\\leq\-\\nabla\\psi\(\\mathbf\{n\}^\{\\sharp\}\)^\{\\top\}\(\\mathbf\{z\}^\{\*\}\-\\mathbf\{n\}^\{\\sharp\}\)≤∑u,vqu​v​\(δs,a​\(m,λ\)\+ω1​\(Ev−Eu\)2N02\)\\displaystyle\\leq\\sum\_\{u,v\}q\_\{uv\}\\left\(\\delta\_\{s,a\}\(m,\\lambda\)\+\\frac\{\\omega\_\{1\}\(E\_\{v\}\-E\_\{u\}\)^\{2\}\}\{N\_\{0\}^\{2\}\}\\right\)≤m​δs,a​\(m,λ\)\+ω1​m​Δ​Es,a2N02\.\\displaystyle\\leq m\\,\\delta\_\{s,a\}\(m,\\lambda\)\+\\frac\{\\omega\_\{1\}m\\Delta E\_\{s,a\}^\{\\,2\}\}\{N\_\{0\}^\{2\}\}\.The lower bound follows because𝐧♯\\mathbf\{n\}^\{\\sharp\}is a feasible source mixture\. Substitutingω1=α1​\(s,a\)\\omega\_\{1\}=\\alpha\_\{1\}\(s,a\)proves \([74](https://arxiv.org/html/2610.10974#A4.E74)\)\. Form=0m=0, the simplex is the singleton\{𝟎\}\\\{\\mathbf\{0\}\\\}, so the exact and stored costs coincide\. Finally,\(nf\+m\)2≥4​nf​m\(n\_\{f\}\+m\)^\{2\}\\geq 4n\_\{f\}mgives the uniform bound whennf\>0n\_\{f\}\>0\. The count\-specific bound remains valid fornf=0n\_\{f\}=0andm≥1m\\geq 1, whereηs,a​\(m\)=α1​\(s,a\)​Δ​Es,a2/m\\eta\_\{s,a\}\(m\)=\\alpha\_\{1\}\(s,a\)\\Delta E\_\{s,a\}^\{\\,2\}/m\. ∎

### D\.4Lagrangian Relaxation Analysis

The global budget constraint∑s,a,kck​nk​\(s,a\)≤κ\\sum\_\{s,a,k\}c\_\{k\}\\,n\_\{k\}\(s,a\)\\leq\\kappacouples the subproblems across contexts\. We relax it via a Lagrange multiplierλ≥0\\lambda\\geq 0, yielding the dual function:

q⁡\(λ\)=min𝐧∈ℱ0⁡\{J~​\(𝐧,𝐧\(t\)\)\+λ⁡\(∑s,a,kck​nk​\(s,a\)−κ\)\}\.q\(\\lambda\)=\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\_\{0\}\}\\Bigl\\\{\\tilde\{J\}\(\\mathbf\{n\};\\,\\mathbf\{n\}^\{\(t\)\}\)\+\\lambda\\bigl\(\\textstyle\\sum\_\{s,a,k\}c\_\{k\}\\,n\_\{k\}\(s,a\)\-\\kappa\\bigr\)\\Bigr\\\}\.By Lemma[8](https://arxiv.org/html/2610.10974#Thmlemma8), the Lagrangian problem decomposes over contexts\. Algorithm[2](https://arxiv.org/html/2610.10974#alg2)solves the count aggregation exactly for its suppliedH♯H^\{\\sharp\}tables; the local approximation errors determine its gap to the exact Lagrangian minimum\.

The multiplier search always produces a feasible candidate in exact arithmetic\. Indeed,ℱ0\\mathcal\{F\}\_\{0\}is finite,𝟎\\mathbf\{0\}is represented in every set of DP tables, andcmin:=mink⁡ck\>0c\_\{\\min\}:=\\min\_\{k\}c\_\{k\}\>0\. For fixedtt, every nonzero allocation has cost at leastcminc\_\{\\min\}, so any

λ\>J~​\(𝟎,𝐧\(t\)\)−min𝐧∈ℱ0⁡J~​\(𝐧,𝐧\(t\)\)cmin\\lambda\>\\frac\{\\tilde\{J\}\(\\mathbf\{0\};\\mathbf\{n\}^\{\(t\)\}\)\-\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\_\{0\}\}\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\}\{c\_\{\\min\}\}makes the zero allocation strictly better than every nonzero allocation in the penalized objective\. Thus doubling eventually returns𝟎\\mathbf\{0\}, or terminates earlier at another budget\-feasible candidate\. Subsequent bisection is a finite candidate search, without a monotonicity guarantee for the costs returned by approximate local solves\.

###### Proposition 10\(Inner\-loop suboptimality\)\.

Fixttand let𝐧^\(t\+1\)∈arg⁡min𝐧∈ℱ​J~​\(𝐧,𝐧\(t\)\)\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}\\in\\arg\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\}\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\. Choose any recorded feasible trial\(λ\(t\),𝐧cand\(t\),𝛅\(t\)\)∈𝒬t\(\\lambda^\{\(t\)\},\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\},\\boldsymbol\{\\delta\}^\{\(t\)\}\)\\in\\mathcal\{Q\}\_\{t\}from Algorithm[1](https://arxiv.org/html/2610.10974#alg1)\. To bound the aggregate local\-search error, define

EGreedy\(t\):=∑s,amax0≤m≤Ms,a⁡\{ηs,a​\(m\)\+m​δs,a​\(m,λ\(t\)\)\},E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}:=\\sum\_\{s,a\}\\max\_\{0\\leq m\\leq M\_\{s,a\}\}\\left\\\{\\eta\_\{s,a\}\(m\)\+m\\,\\delta\_\{s,a\}\(m,\\lambda^\{\(t\)\}\)\\right\\\},where the maxima are over integer counts andMs,a=Ns−nf​\(s,a\)M\_\{s,a\}=N\_\{s\}\-n\_\{f\}\(s,a\)\. The actual outer update, including the current\-allocation fallback, satisfies

0≤ΔLag\(t\)\\displaystyle 0\\leq\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}:=J~​\(𝐧\(t\+1\),𝐧\(t\)\)−J~​\(𝐧^\(t\+1\),𝐧\(t\)\)\\displaystyle:=\\tilde\{J\}\(\\mathbf\{n\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)\-\\tilde\{J\}\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)\(76\)≤λ\(t\)​\(κ−C⁡\(𝐧cand\(t\)\)\)\+EGreedy\(t\)\.\\displaystyle\\leq\\lambda^\{\(t\)\}\\bigl\(\\kappa\-C\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\}\)\\bigr\)\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\.If all local residuals in that trial vanish and all factual counts are positive, thenEGreedy\(t\)≤∑s,aα1​\(s,a\)​Δ​Es,a2/\[4​nf​\(s,a\)\]E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\\leq\\sum\_\{s,a\}\\alpha\_\{1\}\(s,a\)\\Delta E\_\{s,a\}^\{\\,2\}/\[4n\_\{f\}\(s,a\)\]\.

###### Proof\.

Feasibility of𝐧\(t\+1\)\\mathbf\{n\}^\{\(t\+1\)\}and optimality of𝐧^\(t\+1\)\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}giveΔLag\(t\)≥0\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}\\geq 0\. For the upper bound, fix the selected trial multiplierλ=λ\(t\)\\lambda=\\lambda^\{\(t\)\}and writeL⁡\(𝐧,λ\):=J~​\(𝐧,𝐧\(t\)\)\+λ​C​\(𝐧\)L\(\\mathbf\{n\};\\lambda\):=\\tilde\{J\}\(\\mathbf\{n\};\\mathbf\{n\}^\{\(t\)\}\)\+\\lambda C\(\\mathbf\{n\}\)\.

First,Dj​\(w\)D\_\{j\}\(w\)in Algorithm[2](https://arxiv.org/html/2610.10974#alg2)is the minimum stored cost for exactlywwannotations across the firstjjactions\. The initialization gives this interpretation forj=0j=0\. For stagejj, any feasible assignment uses somem≤min⁡\{w,Ms,aj\}m\\leq\\min\\\{w,M\_\{s,a\_\{j\}\}\\\}on actionaja\_\{j\}, leavingw−mw\-mannotations for the firstj−1j\-1actions\. Minimizing over these choices proves the interpretation inductively and supplies valid backtracking counts\. Selecting the bestw≤Nsw\\leq N\_\{s\}enforces the context capacity inequality\. Assembling the independent context solutions therefore minimizes the sum of stored costs overℱ0\\mathcal\{F\}\_\{0\}\.

Let𝐧~λ∈arg⁡min𝐧∈ℱ0⁡L⁡\(𝐧,λ\)\\tilde\{\\mathbf\{n\}\}\_\{\\lambda\}\\in\\arg\\min\_\{\\mathbf\{n\}\\in\\mathcal\{F\}\_\{0\}\}L\(\\mathbf\{n\};\\lambda\)and letm~s,a:=∑kn~λ,k​\(s,a\)\\tilde\{m\}\_\{s,a\}:=\\sum\_\{k\}\\tilde\{n\}\_\{\\lambda,k\}\(s,a\)be its counts\. By exact aggregation and Proposition[9](https://arxiv.org/html/2610.10974#Thmproposition9),

L⁡\(𝐧cand\(t\),λ\)\\displaystyle L\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\};\\lambda\)≤∑s,aHs,a♯​\(m~s,a,λ\)\\displaystyle\\leq\\sum\_\{s,a\}H^\{\\sharp\}\_\{s,a\}\(\\tilde\{m\}\_\{s,a\},\\lambda\)≤∑s,a\[Hs,a\(t,λ\)​\(m~s,a\)\+ηs,a​\(m~s,a\)\+m~s,a​δs,a​\(m~s,a,λ\)\]\\displaystyle\\leq\\sum\_\{s,a\}\\left\[H\_\{s,a\}^\{\(t,\\lambda\)\}\(\\tilde\{m\}\_\{s,a\}\)\+\\eta\_\{s,a\}\(\\tilde\{m\}\_\{s,a\}\)\+\\tilde\{m\}\_\{s,a\}\\delta\_\{s,a\}\(\\tilde\{m\}\_\{s,a\},\\lambda\)\\right\]≤L⁡\(𝐧~λ,λ\)\+EGreedy\(t\)\.\\displaystyle\\leq L\(\\tilde\{\\mathbf\{n\}\}\_\{\\lambda\};\\lambda\)\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\.The last inequality uses the fact that the exact fixed\-count costs are no greater than the costs of the corresponding mixtures in𝐧~λ\\tilde\{\\mathbf\{n\}\}\_\{\\lambda\}\. Because𝐧^\(t\+1\)∈ℱ⊆ℱ0\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}\\in\\mathcal\{F\}\\subseteq\\mathcal\{F\}\_\{0\}, this gives

L⁡\(𝐧cand\(t\),λ\)≤L⁡\(𝐧~λ,λ\)\+EGreedy\(t\)≤L⁡\(𝐧^\(t\+1\),λ\)\+EGreedy\(t\)\.L\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\};\\lambda\)\\leq L\(\\tilde\{\\mathbf\{n\}\}\_\{\\lambda\};\\lambda\)\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\\leq L\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\};\\lambda\)\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\.\(77\)The outer comparison ensuresJ~​\(𝐧\(t\+1\),𝐧\(t\)\)≤J~​\(𝐧cand\(t\),𝐧\(t\)\)\\tilde\{J\}\(\\mathbf\{n\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)\\leq\\tilde\{J\}\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\), even when the current allocation wins\. Rearranging \([77](https://arxiv.org/html/2610.10974#A4.E77)\) and usingC⁡\(𝐧^\(t\+1\)\)≤κC\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}\)\\leq\\kappayields

ΔLag\(t\)\\displaystyle\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}≤J~​\(𝐧cand\(t\),𝐧\(t\)\)−J~​\(𝐧^\(t\+1\),𝐧\(t\)\)\\displaystyle\\leq\\tilde\{J\}\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\)\-\\tilde\{J\}\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)≤λ⁡\[C⁡\(𝐧^\(t\+1\)\)−C⁡\(𝐧cand\(t\)\)\]\+EGreedy\(t\)\\displaystyle\\leq\\lambda\\bigl\[C\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}\)\-C\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\}\)\\bigr\]\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}≤λ⁡\[κ−C⁡\(𝐧cand\(t\)\)\]\+EGreedy\(t\)\.\\displaystyle\\leq\\lambda\\bigl\[\\kappa\-C\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\}\)\\bigr\]\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\.If every residual is zero and every factual count is positive, the uniform local bound in Proposition[9](https://arxiv.org/html/2610.10974#Thmproposition9)gives the stated bound onEGreedy\(t\)E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\. ∎

### D\.5Per\-Iteration Approximation Error

We now combine the majorization and Lagrangian analyses to bound the optimality gap\.

###### Theorem 6\(Per\-Iteration Approximation Error\)\.

Let𝐧∗\\mathbf\{n\}^\{\*\}denote any global minimizer ofJ⁡\(𝐧\)J\(\\mathbf\{n\}\)overℱ\\mathcal\{F\}, and use any recorded feasible trial and its error certificate from Proposition[10](https://arxiv.org/html/2610.10974#Thmproposition10)\. The solution𝐧\(t\+1\)\\mathbf\{n\}^\{\(t\+1\)\}returned by one MM iteration satisfies:

J\(𝐧\(t\+1\)\)−J\(𝐧∗\)≤∑s∈𝒮d0​\(s\)​\(ℬs​\(𝐧∗\)−ℬs​\(𝐧\(t\)\)\)2⏟𝒢MM\(t\):majorization gap\+λ\(t\)​\(κ−C⁡\(𝐧cand\(t\)\)\)⏟Lagrangian relaxation gap\+EGreedy\(t\)⏟inner greedy gap\.J\(\\mathbf\{n\}^\{\(t\+1\)\}\)\-J\(\\mathbf\{n\}^\{\*\}\)\\;\\leq\\;\\underbrace\{\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\*\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\bigr\)^\{2\}\}\_\{\\textstyle\\mathcal\{G\}\_\{\\mathrm\{MM\}\}^\{\(t\)\}\\;:\\;\\text\{majorization gap\}\}\\;\+\\;\\underbrace\{\\lambda^\{\(t\)\}\\bigl\(\\kappa\-C\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\)\}\)\\bigr\)\}\_\{\\text\{Lagrangian relaxation gap\}\}\\;\+\\;\\underbrace\{E\_\{\\mathrm\{Greedy\}\}^\{\(t\)\}\}\_\{\\text\{inner greedy gap\}\}\.\(78\)The majorization gap is bounded uniformly by𝒢MM\(t\)≤4​βmax2\\mathcal\{G\}\_\{\\mathrm\{MM\}\}^\{\(t\)\}\\leq 4\\beta\_\{\\max\}^\{2\}, whereβmax\\beta\_\{\\max\}from Appendix[A\.4](https://arxiv.org/html/2610.10974#A1.SS4)bounds both annotation biases and default\-model biases at zero factual counts\. The greedy error certificate includes remaining swap gains when the local search stops at its cap\.

###### Proof\.

At fixedtt, the constant difference betweenJ~full\\tilde\{J\}\_\{\\mathrm\{full\}\}andJ~\\tilde\{J\}leaves both the constrained minimizer𝐧^\(t\+1\)\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}and the value differenceΔLag\(t\)\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}unchanged\. Consequently, Lemma[7](https://arxiv.org/html/2610.10974#Thmlemma7)and the definition ofΔLag\(t\)\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}in Proposition[10](https://arxiv.org/html/2610.10974#Thmproposition10)give

J⁡\(𝐧\(t\+1\)\)\\displaystyle J\(\\mathbf\{n\}^\{\(t\+1\)\}\)≤J~full​\(𝐧\(t\+1\),𝐧\(t\)\)\\displaystyle\\leq\\tilde\{J\}\_\{\\mathrm\{full\}\}\\\!\\bigl\(\\mathbf\{n\}^\{\(t\+1\)\};\\,\\mathbf\{n\}^\{\(t\)\}\\bigr\)\(majorization\)=J~full​\(𝐧^\(t\+1\),𝐧\(t\)\)\+ΔLag\(t\)\\displaystyle=\\tilde\{J\}\_\{\\mathrm\{full\}\}\\\!\\bigl\(\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\};\\,\\mathbf\{n\}^\{\(t\)\}\\bigr\)\+\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}\(definition ofΔLag\(t\)\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}\)≤J~full​\(𝐧∗,𝐧\(t\)\)\+ΔLag\(t\)\\displaystyle\\leq\\tilde\{J\}\_\{\\mathrm\{full\}\}\\\!\\bigl\(\\mathbf\{n\}^\{\*\};\\,\\mathbf\{n\}^\{\(t\)\}\\bigr\)\+\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}\(optimality of𝐧^\(t\+1\)\)\.\\displaystyle\\text\{\(optimality of $\\hat\{\\mathbf\{n\}\}^\{\(t\+1\)\}$\)\}\.SubtractingJ⁡\(𝐧∗\)J\(\\mathbf\{n\}^\{\*\}\)and applying the gap formula \([70](https://arxiv.org/html/2610.10974#A4.E70)\) yields

J⁡\(𝐧\(t\+1\)\)−J⁡\(𝐧∗\)≤∑s∈𝒮d0​\(s\)​\(ℬs​\(𝐧∗\)−ℬs​\(𝐧\(t\)\)\)2\+ΔLag\(t\)\.J\(\\mathbf\{n\}^\{\(t\+1\)\}\)\-J\(\\mathbf\{n\}^\{\*\}\)\\leq\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\*\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\\bigr\)^\{2\}\+\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}\.The upper bound onΔLag\(t\)\\Delta\_\{\\mathrm\{Lag\}\}^\{\(t\)\}from Proposition[10](https://arxiv.org/html/2610.10974#Thmproposition10)then gives \([78](https://arxiv.org/html/2610.10974#A4.E78)\)\. ForN⁡\(s,a\)\>0N\(s,a\)\>0, the bias is a convex combination of zero factual bias and source biases, so\|B⁡\(s,a\)\|≤βmax\|B\(s,a\)\|\\leq\\beta\_\{\\max\}\. ForN⁡\(s,a\)=0N\(s,a\)=0, the definition ofβmax\\beta\_\{\\max\}also bounds the default bias\. Because the target\-policy probabilities sum to one,\|ℬs\|≤βmax\|\\mathcal\{B\}\_\{s\}\|\\leq\\beta\_\{\\max\}, and hence\|ℬs​\(𝐧∗\)−ℬs​\(𝐧\(t\)\)\|≤2​βmax\|\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\*\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(t\)\}\)\|\\leq 2\\beta\_\{\\max\}\. Summing the squared differences with weightsd0​\(s\)d\_\{0\}\(s\), which also sum to one, gives𝒢MM\(t\)≤4​βmax2\\mathcal\{G\}\_\{\\mathrm\{MM\}\}^\{\(t\)\}\\leq 4\\beta\_\{\\max\}^\{2\}\. ∎

### D\.6Convergence Guarantees

###### Corollary 4\(Finite Descent and Stopping Criteria\)\.

With fixed problem inputs and deterministic tie\-breaking, Algorithm[1](https://arxiv.org/html/2610.10974#alg1)has the following properties:

1. \(i\)Every changed allocation strictly decreasesJJ, and there are at most\|ℱ\|−1\|\\mathcal\{F\}\|\-1such updates\.
2. \(ii\)If the outer iteration limit is removed andϵ=0\\epsilon=0, the algorithm reaches an unchanged allocation in finitely many iterations\. This allocation is a fixed point of the deterministic update rule\. A positive tolerance or finite iteration limit may terminate before a fixed point is reached\.
3. \(iii\)If an unchanged allocation𝐧\(∞\)\\mathbf\{n\}^\{\(\\infty\)\}is reached at iterationt⋆t\_\{\\star\}, then J⁡\(𝐧\(∞\)\)−J⁡\(𝐧∗\)\\displaystyle J\(\\mathbf\{n\}^\{\(\\infty\)\}\)\-J\(\\mathbf\{n\}^\{\*\}\)≤∑sd0​\(s\)​\(ℬs​\(𝐧∗\)−ℬs​\(𝐧\(∞\)\)\)2\\displaystyle\\leq\\sum\_\{s\}d\_\{0\}\(s\)\\bigl\(\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\*\}\)\-\\mathcal\{B\}\_\{s\}\(\\mathbf\{n\}^\{\(\\infty\)\}\)\\bigr\)^\{2\}\(79\)\+λ\(t⋆\)​\(κ−C⁡\(𝐧cand\(t⋆\)\)\)\+EGreedy\(t⋆\),\\displaystyle\+\\lambda^\{\(t\_\{\\star\}\)\}\\bigl\(\\kappa\-C\(\\mathbf\{n\}\_\{\\mathrm\{cand\}\}^\{\(t\_\{\\star\}\)\}\)\\bigr\)\+E\_\{\\mathrm\{Greedy\}\}^\{\(t\_\{\\star\}\)\},using any feasible trial recorded att⋆t\_\{\\star\}\.

###### Proof\.

The best\-candidate variable starts at𝐧\(t\)\\mathbf\{n\}^\{\(t\)\}and changes only on a strict surrogate improvement\. Thus any changed allocation satisfiesJ~full​\(𝐧\(t\+1\),𝐧\(t\)\)<J~full​\(𝐧\(t\),𝐧\(t\)\)\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\+1\)\};\\mathbf\{n\}^\{\(t\)\}\)<\\tilde\{J\}\_\{\\mathrm\{full\}\}\(\\mathbf\{n\}^\{\(t\)\};\\mathbf\{n\}^\{\(t\)\}\), and the majorization chain in Proposition[8](https://arxiv.org/html/2610.10974#Thmproposition8)gives strict descent ofJJ\. No changed allocation can therefore revisit a previous allocation in the finite setℱ\\mathcal\{F\}, proving \(i\)\. After at most\|ℱ\|−1\|\\mathcal\{F\}\|\-1changes, an update must leave the allocation unchanged\. With fixed inputs, search parameters, and tie\-breaking, repeating the update at that allocation returns the same result, proving \(ii\)\. The practical stopping conditions can stop after a small strict improvement or at the iteration limit, neither of which implies an unchanged next update\. At an unchanged update,𝐧\(t⋆\+1\)=𝐧\(t⋆\)=𝐧\(∞\)\\mathbf\{n\}^\{\(t\_\{\\star\}\+1\)\}=\\mathbf\{n\}^\{\(t\_\{\\star\}\)\}=\\mathbf\{n\}^\{\(\\infty\)\}, so Theorem[6](https://arxiv.org/html/2610.10974#Thmtheorem6)gives \([79](https://arxiv.org/html/2610.10974#A4.E79)\), proving \(iii\)\. ∎

### D\.7Computational Complexity

###### Proposition 11\(Total Computational Complexity\)\.

LetS=\|𝒮\|S=\|\\mathcal\{S\}\|,A=\|𝒜\|A=\|\\mathcal\{A\}\|,K=\|𝒦\|K=\|\\mathcal\{K\}\|,Nmax=maxs⁡Ns≥1N\_\{\\max\}=\\max\_\{s\}N\_\{s\}\\geq 1, and letTswap≥1T\_\{\\mathrm\{swap\}\}\\geq 1cap accepted swaps per local call\. LetTMMT\_\{\\mathrm\{MM\}\}be the number of outer iterations andTBisT\_\{\\mathrm\{Bis\}\}an upper bound on all subproblem calls per outer iteration, including the zero\-multiplier trial, doubling, and bisection\. The total complexity of Algorithm[1](https://arxiv.org/html/2610.10974#alg1)is

𝒪⁡\(TMM⋅TBis⋅S⋅A⋅\(K2​Nmax​Tswap\+Nmax2\)\)\.\\mathcal\{O\}\\\!\\Bigl\(T\_\{\\mathrm\{MM\}\}\\cdot T\_\{\\mathrm\{Bis\}\}\\cdot S\\cdot A\\cdot\\bigl\(K^\{2\}\\,N\_\{\\max\}\\,T\_\{\\mathrm\{swap\}\}\+N\_\{\\max\}^\{2\}\\bigr\)\\Bigr\)\.\(80\)For an iteration withddmultiplier doublings and bracket widthW\>0W\>0before bisection, the number of subproblem calls is2\+d\+max⁡\{0,⌈log2⁡\(W/ϵλ\)⌉\}2\+d\+\\max\\\{0,\\lceil\\log\_\{2\}\(W/\\epsilon\_\{\\lambda\}\)\\rceil\\\}\. The finite\-state bound in Proposition[9](https://arxiv.org/html/2610.10974#Thmproposition9)also limits the actual number of accepted swaps, regardless of the chosen cap\.

###### Proof\.

For a fixed pair and total count, source search accepts at mostTswapT\_\{\\mathrm\{swap\}\}swaps and needs at most one additional scan to certify the final residual\. Each scan evaluatesK⁡\(K−1\)K\(K\-1\)swaps in𝒪⁡\(K2\)\\mathcal\{O\}\(K^\{2\}\)by maintaining the sums∑knk​Ek\\sum\_\{k\}n\_\{k\}E\_\{k\},∑knk​Dk\\sum\_\{k\}n\_\{k\}D\_\{k\}, and∑kck​nk\\sum\_\{k\}c\_\{k\}n\_\{k\}; each accepted swap updates these sums in constant time\. Initialization and storing the source vector cost𝒪⁡\(K\)\\mathcal\{O\}\(K\), so the full fixed\-count cost is𝒪⁡\(K2​Tswap\)\\mathcal\{O\}\(K^\{2\}T\_\{\\mathrm\{swap\}\}\)forTswap≥1T\_\{\\mathrm\{swap\}\}\\geq 1\. There are at mostNmax\+1N\_\{\\max\}\+1counts per pair, giving𝒪⁡\(A​Nmax​K2​Tswap\)\\mathcal\{O\}\(AN\_\{\\max\}K^\{2\}T\_\{\\mathrm\{swap\}\}\)per context whenNmax≥1N\_\{\\max\}\\geq 1\. The action\-indexed DP hasA⁡\(Ns\+1\)A\(N\_\{s\}\+1\)entries, each scanning at mostNs\+1N\_\{s\}\+1feasible counts, so aggregation costs𝒪⁡\(A​Nmax2\)\\mathcal\{O\}\(AN\_\{\\max\}^\{2\}\)per context\. Backtracking costs𝒪⁡\(A​K\)\\mathcal\{O\}\(AK\), and scanning the residual tables costs𝒪⁡\(A​Nmax\)\\mathcal\{O\}\(AN\_\{\\max\}\), both within the preceding bounds\. Candidate objective comparisons cost𝒪⁡\(S​A​K\)\\mathcal\{O\}\(SAK\)per trial and are also dominated by the subproblem cost\. Summing over contexts, all multiplier trials, and outer iterations proves \([80](https://arxiv.org/html/2610.10974#A4.E80)\)\. There are two initial trials, one at zero and one atλ0\\lambda\_\{0\}, followed bydddoubling trials\. Each bisection trial halves the interval width, giving the stated remaining trial count\. ∎

## Appendix EExperimental Details and Additional Results

##### Computational resources\.

Local preprocessing, allocation, and OPE calculations used only the central processing unit \(CPU\) of a MacBook with an Apple M3 Pro chip; the codebase is also compatible with the standard Google Colab CPU environment\. LLM annotations were obtained through hosted application programming interfaces \(APIs\) from Google, OpenAI, and Anthropic\. Table[7](https://arxiv.org/html/2610.10974#A5.T7)reports full\-instance solver runtimes, with a one\-hour limit for each Gurobi run\. Theoretical runtime bounds are given in Proposition[3](https://arxiv.org/html/2610.10974#Thmproposition3)and Appendix[D\.7](https://arxiv.org/html/2610.10974#A4.SS7)\.

### E\.1Synthetic Clinical Bandit

#### E\.1\.1Environment and source profiles

We use100100contexts and55actions\. For each context,πb\(⋅∣s\)∼Dirichlet\(0\.2𝟏\)\\pi\_\{b\}\(\\cdot\\mid s\)\\sim\\operatorname\{Dirichlet\}\(0\.2\\mathbf\{1\}\)\. The target policy retains a behavior\-selected action with probability0\.90\.9and otherwise replaces it by a uniformly sampled alternative\. The context distribution is sampled from a symmetric Dirichlet distribution\.

Pairwise reward means are independently drawn from

0\.2​Normal⁡\(−2,32\)\+0\.8​SkewNormal⁡\(ξ=5,ω=6,α=−2\),0\.2\\,\\operatorname\{Normal\}\(\-2,3^\{2\}\)\+0\.8\\,\\operatorname\{SkewNormal\}\(\\xi=5,\\omega=6,\\alpha=\-2\),and reward variances fromUniform⁡\(20,50\)\\operatorname\{Uniform\}\(20,50\)\. The fitting dataset containsM=1,000M=1\{,\}000records\. Sourcekkhas meanμr​\(s,a\)\+εk\\mu\_\{r\}\(s,a\)\+\\varepsilon\_\{k\}, varianceσr2​\(s,a\)\+Δk\\sigma\_\{r\}^\{2\}\(s,a\)\+\\Delta\_\{k\}, and costckc\_\{k\}\. The allocation budget isκ=10,000\\kappa=10\{,\}000\.

#### E\.1\.2Source Subset Comparison Under Consistent Optimization

To isolate the effect of source availability from allocation strategy, we evaluate all nonempty subsets𝒦sub⊆\{1,2,3,4\}\\mathcal\{K\}\_\{\\mathrm\{sub\}\}\\subseteq\\\{1,2,3,4\\\}with MM, including the single\-source subsets evaluated under random allocation in Figure[2](https://arxiv.org/html/2610.10974#S7.F2)\. Figure[3](https://arxiv.org/html/2610.10974#A5.F3)shows the resulting scaled MSE values\. Under consistent optimization, the qualitative conclusions of §[7](https://arxiv.org/html/2610.10974#S7)are unchanged\.

Figure 3:Scaled mean squared error \(MSE\)𝒱⁡\(𝐧\)\\mathcal\{V\}\(\\mathbf\{n\}\)for each active source subset*under the proposed majorization\-minimization \(MM\) algorithm*\. Compared with Figure[2](https://arxiv.org/html/2610.10974#S7.F2), the single\-source bars \(red\) are now obtained by optimization rather than uniformly random allocation; the multi\-source bars are unchanged\.
#### E\.1\.3Convergence Trace on the Synthetic Bandit

Figure[4](https://arxiv.org/html/2610.10974#A5.F4)reports the convergence trace of the MM algorithm on the synthetic clinical bandit, replayed across the source subsets that achieve successively lower scaled MSE values:\{1,3\}\\\{1,3\\\},\{1,2,3\}\\\{1,2,3\\\}, and finally\{1,2,3,4\}\\\{1,2,3,4\\\}\. Scaled MSE drops sharply within the first MM iteration of each subset and stabilizes within roughly55–66iterations, consistent with the monotone\-descent property in Proposition[8](https://arxiv.org/html/2610.10974#Thmproposition8)\. The annotation\-count tuples annotated on the trajectory show that the optimizer concentrates the budget on the lowest\-bias source \(Source 1\) at first and progressively allocates a small share to higher\-variance, lower\-cost sources as additional sources become available\.

Figure 4:Convergence trace of scaled mean squared error \(MSE\)𝒱⁡\(𝐧\(t\)\)\\mathcal\{V\}\(\\mathbf\{n\}^\{\(t\)\}\)across majorization\-minimization \(MM\) iterations on the synthetic clinical bandit\. Different colors mark the active source subset; the tuple annotated at selected iterations records the allocation\[n1,n2,n3,n4\]\[n\_\{1\},n\_\{2\},n\_\{3\},n\_\{4\}\]across Sources 1–4 in Table[2](https://arxiv.org/html/2610.10974#S7.T2)\.

### E\.2ASSISTments Semi\-Synthetic Environment

#### E\.2\.1Dataset, Context Space, and Action Space

The ASSISTments 2012–13 School Data with Affect\[[5](https://arxiv.org/html/2610.10974#bib.bib5)\]contains6,123,2706\{,\}123\{,\}270student–problem interactions from the ASSISTments online middle\-school mathematics tutoring platform\. Each row records: \(i\) student/problem identifiers, \(ii\) first\-response latencyms\_first\_response, \(iii\) correctness of first attempt, hint count, and attempt count, \(iv\) problem metadata \(problem\_type,originalindicator\), and \(v\) four affect\-detector confidences \(frustration, confusion, concentration, boredom\)\. We remove interactions withms\_first\_responseoutside\[1,000,600,000\]\[1\{,\}000,\\;600\{,\}000\]ms \(<1<1s likely auto\-fill,\>10\>10min likely idle\), retaining6,117,2796\{,\}117\{,\}279rows \(99\.9%99\.9\\%\), then sort by\(user\_id, start\_time\)so that past statistics use only strictly prior interactions\.

##### Context Space \(\|𝒮\|=100\|\\mathcal\{S\}\|=100\)

Each interaction is embedded as a 6\-dimensional vector

ϕcontext=\[PastCorrect,PastHints,Frust,Conf,Conc,Bore\],\\phi\_\{\\mathrm\{context\}\}=\\bigl\[\\mathrm\{PastCorrect\},\\;\\mathrm\{PastHints\},\\;\\mathrm\{Frust\},\\;\\mathrm\{Conf\},\\;\\mathrm\{Conc\},\\;\\mathrm\{Bore\}\\bigr\],wherePastCorrect\\mathrm\{PastCorrect\}andPastHints\\mathrm\{PastHints\}are expanding\-mean correct rate and hint count over the student’s strictly prior problems \(initialized to0\.50\.5and00\)\. Features are standardized; K\-means withk=100k\{=\}100,ninit=10n\_\{\\mathrm\{init\}\}\{=\}10produces100100clusters\. Centroids in the original scale are passed to the LLM as the context descriptor\.

##### Action Space \(\|𝒜\|=10\|\\mathcal\{A\}\|=10\)

We crossproblem\_type\(6 values\) withoriginal\(2 values\), then retain the1010combinations with at least100100observed interactions\. Table[5](https://arxiv.org/html/2610.10974#A5.T5)lists the resulting actions and their empirical statistics used both in the reward function and in the LLM prompt\.

Table 5:Action space of the ASSISTments environment\.

#### E\.2\.2Reward Function

Following the feature\-prediction spirit of APRIL\[[4](https://arxiv.org/html/2610.10974#bib.bib4)\], we decompose the reward into a data\-driven base and an LLM\-predictable multiplier\. WithRmax=30R\_\{\\max\}=30:

R⁡\(s,a,t\)=Rmax⋅PIRT​\(s,a\)⏟data\-driven⋅TimeEff⁡\(t,t~a\)⏟LLM\-predictable\.R\(s,a,t\)\\;=\\;R\_\{\\max\}\\cdot\\underbrace\{P\_\{\\mathrm\{IRT\}\}\(s,a\)\}\_\{\\text\{data\-driven\}\}\\cdot\\underbrace\{\\mathrm\{TimeEff\}\(t,\\tilde\{t\}\_\{a\}\)\}\_\{\\text\{LLM\-predictable\}\}\.\(81\)
##### IRT component\.

For contextsswith centroid correct ratepsp\_\{s\}and actionaawith overall correct rateqaq\_\{a\}, setθs=logit⁡\(ps\)\\theta\_\{s\}=\\mathrm\{logit\}\(p\_\{s\}\),ba=logit⁡\(1−qa\)b\_\{a\}=\\mathrm\{logit\}\(1\-q\_\{a\}\), andPIRT​\(s,a\)=σ⁡\(θs−ba\)P\_\{\\mathrm\{IRT\}\}\(s,a\)=\\sigma\(\\theta\_\{s\}\-b\_\{a\}\), withps,qap\_\{s\},q\_\{a\}clipped to\[0\.02,0\.98\]\[0\.02,0\.98\]\. This is a Rasch \(1\-PL\) model depending only on observed data\. Across the100×10100\\times 10grid,PIRTP\_\{\\mathrm\{IRT\}\}has range\[0\.03,0\.98\]\[0\.03,0\.98\], mean0\.680\.68, std0\.220\.22—providing the*primary*reward variation\.

##### Time\-efficiency component\.

Lett~a\\tilde\{t\}\_\{a\}be actionaa’s median response time in seconds\. Define

TimeEff⁡\(t,t~a\)=max⁡\{0\.3,1−α⋅\|log10⁡\(t/t~a\)\|\},α=0\.4\.\\mathrm\{TimeEff\}\(t,\\tilde\{t\}\_\{a\}\)\\;=\\;\\max\\\!\\left\\\{0\.3,\\;1\-\\alpha\\cdot\\bigl\|\\log\_\{10\}\(t/\\tilde\{t\}\_\{a\}\)\\bigr\|\\right\\\},\\quad\\alpha=0\.4\.This symmetric clamped\-linear function is1\.01\.0att=t~at=\\tilde\{t\}\_\{a\}, decays linearly in log\-space, and floors at0\.30\.3for extreme mispredictions\. This constructed discount encodes a preference for response times near the action\-specific median, penalizing both unusually short and unusually long responses\.

##### Ground\-truth reward\.

For each observed\(s,a\)\(s,a\)pair, the referenceμr​\(s,a\)\\mu\_\{r\}\(s,a\)is computed by plugging the empirical mediant~s,a\\tilde\{t\}\_\{s,a\}into Eq\. \([81](https://arxiv.org/html/2610.10974#A5.E81)\)\. For the54/100054/1000pairs with fewer than2020observations, we fit a random forest on context centroids\+\+action one\-hot features to predictlog10⁡t~s,a\\log\_\{10\}\\tilde\{t\}\_\{s,a\}\(55\-fold CVR2=0\.89R^\{2\}=0\.89\) and fill the gap\. The finalμr\\mu\_\{r\}grid has range\[0\.43,28\.7\]\[0\.43,28\.7\], mean20\.0820\.08, std10\.7710\.77\.

![Refer to caption](https://arxiv.org/html/2610.10974v1/assist_reward.png)Figure 5:Ground\-Truth rewardLeft:distribution of Ground\-Truth RewardsMiddle:heatmap of factual context\-action pair dataRight:influence of Time\-efficiency discount on the total reward

#### E\.2\.3LLM Prompt Design

Each prompt is a*batch*of1010\(s,a\)\(s,a\)pairs sharing a common task description, action\-reference table, and domain\-knowledge preamble, followed by pair\-specific student profiles and88bootstrap\-sampled historical response times per pair\. All per\-pair information \(ability\-level label, centroid features, action statistics, historical samples\) is precomputed once from the dataframe before any API call, keeping per\-batch overhead below11ms and the mean prompt length at∼3,500\\sim\\\!3\{,\}500characters\. Figure[6](https://arxiv.org/html/2610.10974#A5.F6)shows the template; curly\-braced fields are populated from precomputed lookups\.

LLM Prompt: ASSISTments Response\-Time PredictionYou are an expert in educational data mining and intelligent tutoring systems\. Your task: predict theresponse time in secondsfor students solving math problems on the ASSISTments platform\. “Response time” = seconds from when a problem appears until the student’s first action \(submitting an answerorrequesting a hint\)\.\[\[ \#\# Action Reference \(Problem Types\) \#\# \]\]\[\[ \#\# Key Factors Affecting Response Time \#\# \]\]1\. Student ability\(past correct rate\): higher performers respond faster; strugglers take longer or give up quickly\.2\. Hint\-seeking habit: students who frequently request hints pause longer before their first action\.3\. Emotional context: frustration/confusion→\\tolonger; boredom→\\torush or disengage; concentration→\\tomoderate, productive\.4\. Problem type: multiple choice fastest; open response and rank slowest; algebra and fill\-in in between\.5\. Original vs\. scaffolding: scaffolding problems typically3030–50%50\\%faster than the original\.6\. Interaction effects: struggling×\\timeshard→\\tomuch longer than able×\\timeseasy\.\[\[ \#\# Historical Response Time Samples \#\# \]\]For each pair below, we provide actual response times from similar students on similar problems\. Use them to calibrate your prediction\.\[\[ \#\# Student–Problem Pairs \#\# \]\]\-\-\- Pair 1 \(context\_id: \{ss\}, action\_id: \{aa\}\) \-\-\-
Student: \{ability\_level\}, correct\_rate=\{psp\_\{s\}\}, avg\_hints=\{hsh\_\{s\}\},
frustration=\{fsf\_\{s\}\}, confusion=\{csc\_\{s\}\}, concentration=\{ksk\_\{s\}\}, boredom=\{bsb\_\{s\}\}
Problem: \{problem\_type\}, original=\{orig\}, typical\_median=\{t~a\\tilde\{t\}\_\{a\}\}s
Historical response times from similar cases: \[\{t1t\_\{1\}\}s, \{t2t\_\{2\}\}s, \.\.\., \{t8t\_\{8\}\}s\]… \(9 more pairs in the same format\) …\[\[ \#\# Prediction Task \#\# \]\]For each pair, predict a single number: the response time in seconds\. Consider all factors and the historical samples carefully\. Output a JSON object conforming to the schema, with one entry per pair containingcontext\_id,action\_id, andpredicted\_response\_time\_seconds\.Figure 6:Prompt template used for all three large language models \(LLMs\) on the ASSISTments environment\. Bracketed headers\[\[ \#\# …\#\# \]\]follow the section\-delimiter style of[Mandyam et al\. \[4\]](https://arxiv.org/html/2610.10974#bib.bib4); \{curly\_braces\} denote per\-pair fields populated from precomputed pandas lookups\. Each batch contains1010pairs;2020independent queries at temperature0\.80\.8are issued per\(s,a\)\(s,a\)\.
#### E\.2\.4LLM Annotation Allocation by MM

We evaluate four LLM annotation sources: Gemini\-2\.5\-Flash, GPT\-4o\-mini, GPT\-o3\-mini, and Claude\-Haiku\-4\.5\. For each of the100×10100\\times 10context\-action pairs, each model produces2020first\-response\-time predictions at temperature0\.80\.8, yielding20,00020\{,\}000predictions per model\. We estimate each source’s bias asε^k​\(s,a\)=r¯k​\(s,a\)−μr​\(s,a\)\\hat\{\\varepsilon\}\_\{k\}\(s,a\)=\\bar\{r\}\_\{k\}\(s,a\)\-\\mu\_\{r\}\(s,a\), wherer¯k​\(s,a\)\\bar\{r\}\_\{k\}\(s,a\)is the mean of the2020predicted rewards andμr​\(s,a\)\\mu\_\{r\}\(s,a\)is the reference reward from the simulator\. We estimate prediction variability byσ^k​\(s,a\)\\hat\{\\sigma\}\_\{k\}\(s,a\), the standard deviation of the2020predicted rewards\. Table[6](https://arxiv.org/html/2610.10974#A5.T6)summarizes these estimates and the API costs\. GPT\-4o\-mini has the lowest mean absolute bias and cost, while GPT\-o3\-mini costs about78×78\\timesmore without improving the reported quality summaries, consistent with the feature\-prediction findings of APRIL\[[4](https://arxiv.org/html/2610.10974#bib.bib4)\]\.

Table 6:LLM annotation quality on ASSISTments\. Bias and variability are computed after converting predicted response times to rewards\. The cost column reports the total API cost for the20,00020\{,\}000predictions used to estimate each model’s empirical bias and variability; the allocation experiment uses the corresponding per\-prediction cost\.We generate6,0006\{,\}000logged samples from the ASSISTments simulator, with pair\-specific reward variancesσr2​\(s,a\)∼Uniform⁡\(30,60\)\\sigma\_\{r\}^\{2\}\(s,a\)\\sim\\mathrm\{Uniform\}\(30,60\)\. Based on the source diagnostics in Table[6](https://arxiv.org/html/2610.10974#A5.T6), we run the MM\-based allocation algorithm with the three lower\-cost sources, excluding GPT\-o3\-mini, under budgetκ=$0\.50\\kappa=\\$0\.50\. The algorithm decides to purchase6,0006\{,\}000annotations and reduces the variance objective fromJ⁡\(𝟎\)=88\.61J\(\\mathbf\{0\}\)=88\.61to79\.0679\.06, a10\.77%10\.77\\%reduction\.

Mirroring the synthetic experiment, we evaluate every non\-empty LLM source subset \(excluding the high\-cost GPT\-o3\-mini, which Table[6](https://arxiv.org/html/2610.10974#A5.T6)shows offers no quality advantage\) under two protocols for the single source baseline: a uniformly random allocation and the proposed MM algorithm\.

Figure[7](https://arxiv.org/html/2610.10974#A5.F7)reports the random\-allocation baseline\. Single\-source subsets yield variance objectives in the range80\.7980\.79–83\.7283\.72, while every multi\-source subset drops below8080\.

Figure 7:Variance objectiveJ⁡\(𝐧\)J\(\\mathbf\{n\}\)for each active LLM source subset under uniformly random allocation\. Bars are grouped by the number of available sources; the dashed line is the no\-annotation baselineJ⁡\(𝟎\)=88\.61J\(\\mathbf\{0\}\)=88\.61\.Figure[8](https://arxiv.org/html/2610.10974#A5.F8)repeats the comparison under consistent MM optimization\. The single\-source bars improve to79\.3279\.32–80\.8280\.82, narrowing the gap to multi\-source results\. The three\-source configuration\{Gemini,Claude,GPT\-4o\-mini\}\\\{\\text\{Gemini\},\\text\{Claude\},\\text\{GPT\-4o\-mini\}\\\}remains the global optimum atJ=79\.06J=79\.06, a10\.77%10\.77\\%reduction overJ⁡\(𝟎\)=88\.61J\(\\mathbf\{0\}\)=88\.61\.

Figure 8:Variance objectiveJ⁡\(𝐧\)J\(\\mathbf\{n\}\)for each active LLM source subset under the proposed MM algorithmTo illustrate the dynamics of the MM algorithm under realistic LLM annotation profiles, Figure[9](https://arxiv.org/html/2610.10974#A5.F9)shows the convergence trace on ASSISTments\. The objective decreases monotonically from88\.6188\.61to79\.0679\.06\.

Figure 9:Convergence trace ofJ⁡\(𝐧\(t\)\)J\(\\mathbf\{n\}^\{\(t\)\}\)across MM iterations on ASSISTments\. The tuple at each highlighted iteration denotes the optimal allocation across \[Gemini\-2\.5\-Flash,Claude\-Haiku\-4\.5,GPT\-4o\-mini\]\. The algorithm heavily favorsGPT\-4o\-minidue to its low empirical bias and high cost\-efficiency, while utilizingClaude\-Haiku\-4\.5more conservatively because of its higher API cost\.

### E\.3Exact\-Solver Benchmarks and Runtime

To complement the certified small\-instance optima in Table[4](https://arxiv.org/html/2610.10974#S7.T4), we warm\-started Gurobi on the full Synthetic and ASSISTments instances with the best heuristic solution and imposed a one\-hour time limit\. The runs did not certify global optimality within that limit, so we compare the methods against the valid lower bounds available at termination\. LetLLbe the scaled\-MSE lower bound available at termination andzA=𝒱⁡\(𝐧A\)z\_\{A\}=\\mathcal\{V\}\(\\mathbf\{n\}\_\{A\}\)the scaled MSE returned by methodAA\. The table reports100​\(zA−L\)/zA100\(z\_\{A\}\-L\)/z\_\{A\}\. The heuristic runtimes exclude the Gurobi computation used to obtainLL\.

Table 7:Solver benchmarks and runtimes for the Synthetic and ASSISTments environments\.Table[4](https://arxiv.org/html/2610.10974#S7.T4)shows greedy\-initialized MM has a0\.05%0\.05\\%mean gap and a0\.26%0\.26\\%maximum gap to the certified optima on the small instances\. On the full instances, Table[7](https://arxiv.org/html/2610.10974#A5.T7)shows that it reaches within0\.2807%0\.2807\\%and0\.0050%0\.0050\\%of the Gurobi lower bounds in5\.35\.3and522522seconds, respectively\. These comparisons show small empirical gaps to the available exact and lower\-bound benchmarks\.

### E\.4End\-to\-End Evaluation with Finite Pilots

The solver benchmarks assess optimization quality for fixed profiles; we next evaluate final OPE performance when allocations are selected using finite\-pilot estimates\. For each of3030pilot replications, factual moments are estimated from2020pilot rewards per pair and annotation moments from2020pilot draws per source–pair\. The pilot is separate from subsequent fitting rewards, acquired annotation realizations, and OPE outcomes\. Fitting\-data context–action counts determine the capacities\.

Each allocator receives the same pilot estimates and constraints\. For the optimized single\-source baseline, we optimize each source separately and use the pilot objective to select the predicted best source\. Thus, this baseline does not select its source using final OPE outcomes\.

After fixing an allocation, we generated5050independent reward and annotation realizations from the distributions used in each environment and evaluated DM\+\-IS on independent OPE datasets \(N=1,000N=1\{,\}000for Synthetic andN=6,000N=6\{,\}000for ASSISTments\)\. In ASSISTments, the annotation realizations are drawn from empirical source models fitted to the observed LLM predictions\. Within each pilot replication, empirical MSE is the average squared deviation of the5050final value estimates from the environment’s true policy value\. Table[8](https://arxiv.org/html/2610.10974#A5.T8)reports the mean across pilot replications of the relative MSE reduction versus the corresponding no\-annotation baseline\.

Table 8:Mean relative reduction in empirical mean squared error \(MSE\) over3030pilot replications\. For each pilot, MSE is estimated from5050independent final off\-policy value estimates\.Table[8](https://arxiv.org/html/2610.10974#A5.T8)shows greedy\-initialized MM reduces empirical MSE by19\.13%19\.13\\%on Synthetic and8\.47%8\.47\\%on ASSISTments relative to no annotation\. Thus, the allocation gains persist in these experiments after finite\-sample profile estimation and evaluation of the final OPE estimator\.

### E\.5Within\-pair bootstrap

As a complementary check, we assess sensitivity to resampling the LLM outputs used to estimate source profiles\. For each of3030bootstrap replications, we resample the2020observed predictions within each source–pair, re\-estimate its moments, and rerun MM\. We evaluate the resulting allocation using reference profiles estimated from all2020original predictions per source–pair\.

The average reference\-profile MSE reduction is10\.765%10\.765\\%, compared with10\.774%10\.774\\%using the original profiles\. The standard deviation is0\.00380\.0038percentage points, and the mean difference is0\.0090\.009percentage points\. This measures conditional sensitivity to resampling the observed outputs\. Because evaluation uses the same empirical source reference, it is not held\-out validation of the population LLM distribution and does not measure uncertainty in the simulator’s factual reward reference\.

## Appendix FBroader Impacts and Asset Use

### F\.1Broader Impacts

Budgeted counterfactual annotation can improve the efficiency of policy evaluation before deployment\. In healthcare and education, unreliable annotations or violations of the evaluation assumptions may nevertheless yield inaccurate policy comparisons with consequences for treatment or instruction\. Application\-specific validation, including assessment of subgroup outcomes, is therefore important before deployment\.

### F\.2Existing Data and Model Services

The ASSISTments 2012–13 School Data with Affect\[[5](https://arxiv.org/html/2610.10974#bib.bib5),[6](https://arxiv.org/html/2610.10974#bib.bib6)\]are provided by Neil T\. Heffernan and the ASSISTments team at Worcester Polytechnic Institute through the[official dataset release](https://sites.google.com/site/assistmentsdata/datasets/2012-13-school-data-with-affect)\. The[ASSISTments data\-use conditions](https://sites.google.com/site/assistmentsdata/)require users to report and delete any inadvertently discovered personally identifiable information and prohibit using that information\. Our experiments use existing records, with no participant recruitment, crowdsourcing, or interventions\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The claims in the abstract and introduction match the theoretical results proved in Appendices[A](https://arxiv.org/html/2610.10974#A1)–[D](https://arxiv.org/html/2610.10974#A4)and the experimental results in Section[7](https://arxiv.org/html/2610.10974#S7)\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: The Conclusion discusses the contextual\-bandit scope, reliance on known or estimated source profiles, and semi\-synthetic ASSISTments environment\. Sensitivity to estimated profiles is analyzed in Appendix[A\.4](https://arxiv.org/html/2610.10974#A1.SS4)and evaluated in Section[7\.3](https://arxiv.org/html/2610.10974#S7.SS3)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: Assumptions accompany the theoretical statements, and their derivations and proofs are provided in Appendices[A](https://arxiv.org/html/2610.10974#A1)–[D](https://arxiv.org/html/2610.10974#A4)\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: Section[7](https://arxiv.org/html/2610.10974#S7)and Appendix[E](https://arxiv.org/html/2610.10974#A5)describe the environments, reward functions, large language model \(LLM\) prompts, and evaluation protocols; code and reproduction instructions are available at the GitHub repository link in Appendix[E](https://arxiv.org/html/2610.10974#A5)\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification: We provide a GitHub repository containing all code, public data sources, data preprocessing scripts, and a detailed README to reproduce the experiments\. See details in Appendix[E](https://arxiv.org/html/2610.10974#A5)\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: Section[7](https://arxiv.org/html/2610.10974#S7)and Appendix[E](https://arxiv.org/html/2610.10974#A5)describe the synthetic and semi\-synthetic environments, preprocessing, source\-profile estimation, and separation of pilot, fitting, and evaluation samples\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: We report the mean and standard deviation across 100 independent environment initializations\(e\.g\., shaded bands in Figure[1](https://arxiv.org/html/2610.10974#S7.F1)\)\. We also provide bootstraps experiment to test the robustness of the algorithm in Section[7\.3](https://arxiv.org/html/2610.10974#S7.SS3)\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: Appendix[E](https://arxiv.org/html/2610.10974#A5)identifies the local hardware and hosted LLM services, and Table[7](https://arxiv.org/html/2610.10974#A5.T7)reports full\-instance solver runtimes\. Memory, runtimes for the remaining experiments, and total project compute are not reported\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The research conforms to the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: Appendix[F\.1](https://arxiv.org/html/2610.10974#A6.SS1)discusses the benefit of more efficient policy evaluation, potential consequences of inaccurate policy comparisons, and the need for application\-specific validation\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: The contribution is an allocation method using existing data and hosted LLMs; we do not release a pretrained generative model or a newly collected student dataset\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: Appendix[F\.2](https://arxiv.org/html/2610.10974#A6.SS2)identifies the ASSISTments release and creators, links its data\-use conditions, and identifies the LLM providers and applicable service terms\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: The code and detailed instructions are in our repository at \[repository URL\]\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: We use synthetic data and secondary analysis of existing ASSISTments records, with no participant recruitment, crowdsourcing, or interventions \(Appendix[F\.2](https://arxiv.org/html/2610.10974#A6.SS2)\)\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The experiments use synthetic data and secondary analysis of the public ASSISTments release, with no new participant recruitment or interventions \(Appendix[F\.2](https://arxiv.org/html/2610.10974#A6.SS2)\)\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification: Section[7\.2](https://arxiv.org/html/2610.10974#S7.SS2)and Appendix[E\.2](https://arxiv.org/html/2610.10974#A5.SS2)describe the LLM annotation sources, response\-time prediction task, prompts, and conversion of predictions to rewards\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.