When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide

# When Can You Trust Offline Evaluation of Equal-Cost Top- k Allocation?A Controlled, Reproducible Benchmark and Practitioner’s Guide
Source: [https://arxiv.org/html/2608.12489](https://arxiv.org/html/2608.12489)
###### Abstract

Organizations routinely decide*whom*to treat under a budget, and want to know what a targeting rule*would*have earned before deploying it\. Off\-policy evaluation promises this from logged data\. But the deployable rule is a*deterministic*top\-kkpolicy: it removes all averaging over actions, so weak overlap hits the estimate directly\. We benchmark six estimators across five datasets and two known\-effect sweeps, and validate the mechanisms against a non\-simulated paired reference\.

First, weak overlap is governed by logger–target*action*alignment, not by logging sharpness alone\. Temperature is not a valid overlap parameter: what governs support is the logger’s probability of the*target’s actions*\. Over the tested range, sharpening a logger built from the target’s own score barely moves overlap; disagreement at the action level collapses it\. Effective sample size, computed from logged actions and propensities, ranks this risk*across*logging environments — guidance for designing logs and choosing estimator families, since it is weak at ranking candidate policies*within*the single fixed log a practitioner holds, and its cut point does not transfer\.

Second, the optimizer’s curse is not fixed by cross\-fitting the outcome nuisance\. When the rule is fit on the data used to evaluate it, cross\-fitting the nuisance alone leaves the reuse bias in place and makes it worse\. Honest policy\-level splitting avoids the reuse by targeting the learning procedure’s value — a change of estimand, not a de\-biasing of the full\-sample policy\.

Third, propensity\-estimation error is the largest degradation we measure\. Replacing the exact propensity with an out\-of\-fold estimate hurtsIPSmore than any other stress we apply, leaves doubly\-robust estimation almost unchanged, and can invert the overlap diagnostic itself\.

Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)states what this benchmark cannot tell you\. Logging is synthesized throughout and propensities are floored at0\.020\.02, so every failure we document occurs with bounded weights; the floor also reduces the two tuned hybrids to their untuned parents in most cells, leaving four practically distinct estimators\. All exact\-value surfaces are synthetic or semi\-synthetic — hence the non\-simulated check\. Section[6](https://arxiv.org/html/2608.12489#S6)distills the operational guidance\. We release the benchmark; public data only\.

††footnotetext:Large language models assisted with writing and code; the authors take full responsibility for all content \(see the Use of Generative AI statement\)\.## 1Introduction

Consider a team allocating a fixed budget across a population: a retention offer to the20%20\\%of customers most likely to respond, enrollment for the patients a treatment will help most\. Each is a*budget\-constrained allocation*: score every unit by an estimated treatment effect and treat the highest\-scoring until the budget is exhausted\. Before deploying a new rule the team wants to know what value it*would*achieve, from data already logged under the current one — off\-policy evaluation \(OPE\)\([8](https://arxiv.org/html/2608.12489#bib.bib5);[23](https://arxiv.org/html/2608.12489#bib.bib8)\)\.

The difficulty is exposure to weak overlap, not determinism per se\.The deployable rule is a*deterministic*top\-kkpolicy: exactly one action per unit, soπe\\pi\_\{e\}puts probability11on it\. That does not by itself violate positivity — the relevant condition isπb​\(aπe​\(x\)∣x\)\>0\\pi\_\{b\}\(a\_\{\\pi\_\{e\}\}\(x\)\\mid x\)\>0for the actionsπe\\pi\_\{e\}selects\. What determinism removes is all averaging over actions: weights becomeπe​\(a∣x\)/πb​\(a∣x\)∈\{0,1/πb​\(a∣x\)\}\\pi\_\{e\}\(a\\mid x\)/\\pi\_\{b\}\(a\\mid x\)\\in\\\{0,\\,1/\\pi\_\{b\}\(a\\mid x\)\\\}, so wherever the logger rarely took the actionπe\\pi\_\{e\}demands the weight is extreme and the effective sample collapses; whereverπb\\pi\_\{b\}has structural zeros, support genuinely fails\. We exclude the second case by construction — propensities are floored at0\.020\.02, so every failure we document occurs with weights bounded by5050, arguably the more alarming finding\. Practitioners feel this as estimates wildly optimistic on some campaigns and fine on others, with little guidance on which case they are in\.

We treat this as an empirical, decision\-facing question\.Rather than propose a new estimator, we ask:*when*is offline evaluation of equal\-cost top\-kkallocation trustworthy,*which*estimator should a practitioner reach for, and*can the danger be detected from logged data alone*? Every estimate is paired with an explicit reference value —*exact*ground truth where potential outcomes are known, a noisy known\-propensity Horvitz–Thompson*reference estimate*on randomized trials — and the two categories are kept separate throughout\. Existing work separately establishes weak\-overlap instability, doubly\-robust estimation, ESS diagnostics, sample splitting and policy\-selection evaluation; our contribution is their integration under this estimand, plus new empirical evidence\.

We organize the results around four questions — RQ1 estimator accuracy, RQ2 overlap diagnostics, RQ3 the optimizer’s curse under policy–evaluation reuse, RQ4 policy selection — and take RQ2 and RQ3 as primary, with the estimated\-propensity boundary \(Contribution 3\) scoping the first\. Section[6](https://arxiv.org/html/2608.12489#S6)states the surviving claims at the strength the evidence supports; a reader who wants conclusions without the qualifications can read it first; Appendix[A](https://arxiv.org/html/2608.12489#A1)tabulates every claim against the evidence and scope supporting it, the fastest route to auditing claim–evidence alignment\.

#### Contributions\.

We make three\.

1. 1\.A design correction: overlap in top\-kkallocation is governed by logger–target misalignment, not logging sharpness alone \(RQ2\)\.The argument is analytic first, empirical second\. For a deterministic target the weight is1/πb​\(aπe​\(x\)∣x\)1/\\pi\_\{b\}\(a\_\{\\pi\_\{e\}\}\(x\)\\mid x\), so support is governed by the logger’s probability of the target’s*actions*, not by how sharply it logs\. Appendix[B](https://arxiv.org/html/2608.12489#A2)turns that into the exact sharpening limit of Proposition 1 \(Section[4](https://arxiv.org/html/2608.12489#S4)\): a score\-aligned logger’s flat overlap is a finite\-range plateau, while an action\-aligned one sharpens toward full support\. That is a prediction, and we tested it — re\-centring the logger at the budget cutoff produces the predicted reversal — sharpening now*improves*overlap, and the failure gradient sharpens to0\.0%0\.0\\%/1\.7%1\.7\\%/26\.7%26\.7\\%\(Appendix[E](https://arxiv.org/html/2608.12489#A5)\)\. We offer this as a correction to how such benchmarks are built, not as a discovery: our own first design swept temperature on a score\-aligned logger, never varying the quantity it meant to study\. The empirical content is magnitude\. Temperature alone moves median ESS only0\.562→0\.5220\.562\\to 0\.522; crossing alignment with temperature moves it0\.560\.56/0\.420\.42/0\.190\.19, withIPSfailure rates of8%8\\%/13%13\\%/32%32\\%at the sharpest temperature\. ESS ranks that error across logging environments, though only weakly within a single log \(ROC\-AUC0\.850\.85in\-sample;0\.830\.83and0\.910\.91on two held\-out families, the0\.910\.91at a2%2\\%error target chosen for that suite’s error scale, not selected for AUC; Table[7](https://arxiv.org/html/2608.12489#A7.T7)\)\. One caveat here: the design and ranking claims are not independent, since misalignment depresses coverage, a factor of ESS\. The estimated\-propensity boundary is Contribution 3\.
2. 2\.An estimand\-coherent comparison of nuisance\-only and policy\-level splitting under policy–evaluation reuse \(RQ3\)\.Cross\-fitting only the outcome nuisance does not remove reuse bias — frozen\-policy cross\-fitting is*more*optimistic than plain DR\. Honest policy\-level splitting evaluates the learning procedure on independent folds, with bias magnitude5858–92%92\\%smaller over eight known\-effect regimes, seven of which share theIHDPcovariate matrix \(response surfaces, not populations\)\. Honesty and sample splitting are long established\([2](https://arxiv.org/html/2608.12489#bib.bib13);[5](https://arxiv.org/html/2608.12489#bib.bib12);[3](https://arxiv.org/html/2608.12489#bib.bib14)\); the contribution is the estimand\-coherent demonstration in the allocation\-OPE setting\.
3. 3\.An empirical boundary for the diagnostic program: propensity estimation dominates, and can invert the screen \(RQ2, continued\)\.Replacing the exactπb\\pi\_\{b\}with an out\-of\-fold estimate is the largest degradation we measure —IPSfailure rises from6\.3%6\.3\\%to3737–63%63\\%of cells, dwarfing the2\.82\.8–11\.7%11\.7\\%from moving the logger regime — and a poor propensity model does not merely weaken the ESS screen but*inverts*it \(AUC0\.850\.85to a coin flip, then0\.050\.05\)\. The screen’s prerequisite is a credible propensity model; Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3)gives the mechanism\.

#### What this paper establishes\.

Stated affirmatively: overlap risk in top\-kkallocation is governed by logger–target misalignment and is rankable ex ante, across logging environments rather than within one log, from logged data,*conditional on a credible propensity model*; under policy–evaluation reuse, nuisance\-only cross\-fitting is counterproductive, and honest policy\-level splitting evaluates the learning procedure without reuse, at the price of targeting a different estimand; and doubly\-robust estimation is the most stable estimator across the*individual*stresses we apply, hence a defensible default when at least one nuisance model is credible\. Three further findings are logically independent\. The estimator rankings \(RQ1\), the negative model\-adequacy\-screen result \(Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\), and the policy\-selection caution \(RQ4\) — candidate\-aligned logs compare policy–logger pairs, not policies — all matter to anyone building an OPE selection benchmark\. Supporting all of it is a released benchmark: six fixed\-policy estimators over five datasets, plus two2,1602\{,\}160\-configuration known\-effect hardening sweeps\.

## 2Related Work

#### OPE estimators and weak overlap\.

The standard toolkit spans the direct method, inverse propensity scoring and its self\-normalized variant\([21](https://arxiv.org/html/2608.12489#bib.bib6)\), doubly\-robust estimation\([8](https://arxiv.org/html/2608.12489#bib.bib5)\), and variance\-controlled hybrids such as Switch\-DR\([24](https://arxiv.org/html/2608.12489#bib.bib7)\)and shrinkage\-based DR\([20](https://arxiv.org/html/2608.12489#bib.bib11)\); formal definitions are in Appendix[H](https://arxiv.org/html/2608.12489#A8)\. Their behaviour is well characterized for*stochastic*targets, and effective sample size is an established weak\-overlap diagnostic\([4](https://arxiv.org/html/2608.12489#bib.bib21)\)\. Deterministic and constrained targets have received less benchmark attention\([9](https://arxiv.org/html/2608.12489#bib.bib3);[16](https://arxiv.org/html/2608.12489#bib.bib4);[25](https://arxiv.org/html/2608.12489#bib.bib2)\), which is the gap this paper addresses\. The closest neighbour is[22](https://arxiv.org/html/2608.12489#bib.bib1), where determinism sits on the*opposite*side: their*logging*policy is deterministic, soπb\\pi\_\{b\}degenerates and weighting is unidentified rather than merely noisy, and they recover randomness from user\-click stochasticity for slate rankings\. Our determinism is in the*target*, withπb\\pi\_\{b\}stochastic and floored, so positivity holds and the difficulty is variance, not identification — complementary problems, and their*real*logged feedback is what our synthesized logging lacks \(Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)\)\. On the design side,[7](https://arxiv.org/html/2608.12489#bib.bib22)*choose*the logging policy, trading reward against coverage of the target’s actions — the design\-side mirror of our fixed\-log alignment axis\. Weight\-reliability constraints appear inside off\-policy*learning*too\([15](https://arxiv.org/html/2608.12489#bib.bib23)\); our contribution is validating ESS as an ex\-ante risk ranking, not the diagnostic itself\.

#### Cross\-fitting, honesty, and the optimizer’s curse\.

When a policy is fit and evaluated on the same sample, its in\-sample value is optimistically biased\([19](https://arxiv.org/html/2608.12489#bib.bib15)\)\. Cross\-fitting removes nuisance overfitting in DML\([5](https://arxiv.org/html/2608.12489#bib.bib12)\), and honesty — evaluating the learning*procedure*on data not used to fit it — is standard in the causal\-forest and policy\-learning literature\([2](https://arxiv.org/html/2608.12489#bib.bib13);[3](https://arxiv.org/html/2608.12489#bib.bib14)\)\. Our contribution is not that principle but an estimand\-coherent empirical demonstration in this setting, separating nuisance\-only cross\-fitting \(frozen policy\) from honest policy\-level splitting \(fold\-specific policies, each scored against its own fold\-matched reference\); the former does not address the observed reuse bias — indeed it is*more*optimistic than plain DR \(Section[5\.4](https://arxiv.org/html/2608.12489#S5.SS4)\)\.

#### Policy selection and benchmarks\.

OPE is used to*select*among candidates, and dedicated metrics quantify selection quality\([18](https://arxiv.org/html/2608.12489#bib.bib9);[13](https://arxiv.org/html/2608.12489#bib.bib16)\)\. Semi\-synthetic causal benchmarks supply known\-effect data\([11](https://arxiv.org/html/2608.12489#bib.bib19);[6](https://arxiv.org/html/2608.12489#bib.bib18)\), and open bandit datasets supply real logged feedback\([17](https://arxiv.org/html/2608.12489#bib.bib10)\)\. In uplift modelling,[14](https://arxiv.org/html/2608.12489#bib.bib24)shows within\-experiment ranking metrics can invert the ordering ground\-truth effect accuracy or policy risk would give — the selection question upstream of ours: there, which scoring rule to pick; here, whether the induced fixed policy’s value can be trusted from logged data\. We combine both traditions, pair every estimate with an explicit reference value, and — as Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)shows — find that giving each candidate its own aligned log confounds selection by comparing policy–logger*pairs*rather than policies\.

## 3Problem Setup

#### Allocation value\.

Each unitiihas contextxix\_\{i\}, a binary actiona∈\{0,1\}a\\in\\\{0,1\\\}\(withhold / treat\), potential outcomesYi​\(0\),Yi​\(1\)Y\_\{i\}\(0\),Y\_\{i\}\(1\), and unit costcic\_\{i\}\. A scoring rules⁡\(⋅\)s\(\\cdot\)and budgetk∈\[0,1\]k\\in\[0,1\]induce a deterministic allocationaπ​\(x\)a\_\{\\pi\}\(x\): treat units in descending score order while cumulative cost stays withink​∑jcjk\\sum\_\{j\}c\_\{j\}\(with unit costs, “treat the top\-kkfraction”\)\. The estimand is the gross expected outcome under the allocation,

V⁡\(πe\)=𝔼⁡\[Y⁡\(aπe​\(X\)\)\],V\(\\pi\_\{e\}\)\\;=\\;\\mathbb\{E\}\\big\[\\,Y\\big\(a\_\{\\pi\_\{e\}\}\(X\)\\big\)\\,\\big\],\(1\)a value level \(cost enters only through the budget constraint\)\. A top\-kkrule is a*batch*policy: whether unitiiis treated depends on the other units through the empirical cutoff\. We therefore target the*cohort\-conditional*valueV\(πe\)n\(z\)=1n∑i𝔼\[Yi\(zi\)∣X1:n\]V\(\\pi\_\{e\}\)\_\{n\}\(z\)=\\frac\{1\}\{n\}\\sum\_\{i\}\\mathbb\{E\}\[Y\_\{i\}\(z\_\{i\}\)\\mid X\_\{1:n\}\]for the fixed allocationz=aπe\(X1:n\)z=a\_\{\\pi\_\{e\}\}\(X\_\{1:n\}\)— what our estimators and references compute — not a population\-quantile policy with a predetermined cutoff\. Uncertainty is correspondingly conditional: the bootstrap holds allocation and cutoff fixed\.

Scope of the empirical study\.The formalism above admits heterogeneous costscic\_\{i\}, but*every experiment in this paper setsci=1c\_\{i\}\\\!=\\\!1*, so the allocation is always “treat the top\-kkfraction” under a single binary action\. We therefore study*equal\-cost top\-kkallocation*, not the general knapsack problem: we evaluate no heterogeneous\-cost or multi\-action setting, and all logged feedback is synthesized \(Section[4](https://arxiv.org/html/2608.12489#S4)\) rather than drawn from a naturally occurring observational logging policy\. Claims and practitioner guidance below should be read at that scope\.

#### Reference values\.

Two explicitly separate categories\. On data with known potential\-outcome means \(synthetic,IHDP, the ACIC\-style DGPs\) we compute the*exact*value of the allocation\. On randomized trials with known constant propensity \(Hillstrom,Lenta,Jobs\) no exact value exists, so we use an unbiased but noisy known\-propensity Horvitz–Thompson*reference estimate*on the randomized evaluation split shared with the estimators\. Error against the second is*not*estimator MSE: it combines OPE error with reference\-estimation variance\. We therefore never pool the two categories in a single claim, and report cross\-dataset averages separately for each\.

#### Why standard OPE strains\.

Becauseπe\\pi\_\{e\}is deterministic, the importance weightwi=πe​\(ai∣xi\)/πb​\(ai∣xi\)w\_\{i\}=\\pi\_\{e\}\(a\_\{i\}\\mid x\_\{i\}\)/\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\)is either00or1/πb​\(ai∣xi\)1/\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\), and the latter blows up whereverπb\\pi\_\{b\}seldom choseπe\\pi\_\{e\}’s action\. The strain is governed by the*overlap*betweenπe\\pi\_\{e\}andπb\\pi\_\{b\}, which we control explicitly and measure\.

## 4Benchmark Design

#### Datasets \(Table[10](https://arxiv.org/html/2608.12489#A8.T10), Appendix[H](https://arxiv.org/html/2608.12489#A8)\)\.

Five datasets span three regimes\.Synthetichas knownμ0,μ1\\mu\_\{0\},\\mu\_\{1\}and a tunable effect, giving an exact value and full control\.IHDP\(continuous outcome\) andJobs\(binary, randomized subset\) are the canonical semi\-synthetic causal benchmarks —IHDPsupplies exact effects,Jobsa randomized reference\.HillstromandLentaare real marketing RCTs with a known constant treatment probability;Hillstromis natively three\-arm and we collapse its two e\-mail arms into one “treat” action \(π1=2/3\\pi\_\{1\}=2/3\), so it evaluates a coarsened action space\. Table[10](https://arxiv.org/html/2608.12489#A8.T10)reports sizes*before*logging: rejection sampling retains a subset, and on the small datasets that matters\.Jobsfalls to132132–203203logged units \(Appendix[J](https://arxiv.org/html/2608.12489#A10)\), so it is noise\-dominated and the effective number of informative primary datasets is closer to four than five\. That caveat applies to every cross\-dataset mean and five\-cluster bootstrap here\. Large RCTs are capped at50,00050\{,\}000rows by uniform subsampling, preserving the constant propensity\. As a known\-effect*hardening*layer, we add six ACIC\-2017\-style data\-generating processes over the realIHDPcovariate matrix, following the competition’s “real covariates\+\+simulated response surfaces” design\([10](https://arxiv.org/html/2608.12489#bib.bib17)\): three surface families \(linear, nonlinear, step\-subgroup effects\) crossed with two noise levels\. Surface coefficients are fixed per setting \(only the factual treatment and outcome noise are redrawn per seed\), noise is calibrated to the effect scale, and the baseline surface carries a positive intercept so the gross\-outcome estimand is bounded away from zero at every budget — avoiding the small\-denominator artifact that inflates relative\-error metrics whenV⁡\(πe\)≈0V\(\\pi\_\{e\}\)\\\!\\approx\\\!0\.

#### Logging policies and the overlap knob\.

For a randomized dataset we synthesize logged feedback by*rejection sampling*: a logging policyπb​\(1∣x\)=σ⁡\(s~​\(x\)/τ\)\\pi\_\{b\}\(1\\mid x\)=\\sigma\\\!\\big\(\\tilde\{s\}\(x\)/\\tau\\big\)is a temperature\-τ\\tausoftmax over the*standardized*candidate scores~=\(s−s¯\)/sd⁡\(s\)\\tilde\{s\}=\(s\-\\bar\{s\}\)/\\mathrm\{sd\}\(s\), and we keep unitiiwith probability proportional toπb​\(ai∣xi\)/πrct​\(ai∣xi\)\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\)/\\pi\_\{\\mathrm\{rct\}\}\(a\_\{i\}\\mid x\_\{i\}\); retained units keep their real \(action, outcome\)\. The logging treat\-probability is*floored at0\.020\.02*for numerical stability — a design property with consequences worth stating up front: importance weights are bounded by5050by construction, structural zeros never occur, and the weak\-overlap failure documented in Section[5](https://arxiv.org/html/2608.12489#S5)is therefore a*bounded\-weight*phenomenon, not an unbounded\-tail one\. We sweepτ∈\{0\.5,2\.0,5\.0\}\\tau\\in\\\{0\.5,2\.0,5\.0\\\}\.

Temperature alone is not a valid overlap parameter, and the reason is specific to deterministic targets\.What matters isπb\\pi\_\{b\}on the*action the target selects*\. Our self\-aligned logger matches the candidate’s score*ranking*, not its action boundary: the logistic is centred at the score mean, while the top\-kkboundary sits at the budget cutoff\. And indeed, over the tested range, sharpening barely moves overlap\. That is a finite\-range plateau, not a guarantee — extreme sharpening collapses overlap through the low\-probability band between mean and cutoff, while a genuinely*action\-aligned*logger sharpens toward full support\. The limit is exact:

Proposition 1 \(sharpening limit\)\.*For a top\-kktarget under scoressand a logistic logger floored atε\\varepsilon, the normalized Kish ESS satisfiesESS/n→\{𝔼⁡\[1/qτ​\(X\)\]\}−1\\mathrm\{ESS\}/n\\to\\\{\\mathbb\{E\}\[1/q\_\{\\tau\}\(X\)\]\\\}^\{\-1\}in the large\-sample limit, whereqτ​\(X\)q\_\{\\tau\}\(X\)is the logging probability of the target’s action\. Consequently, for a score\-aligned logger, asτ→0\\tau\\to 0this population ESS fraction satisfies*

ESSn⟶\[c1−ε\+1−cε\]−1≈ε1−c,\\frac\{\\mathrm\{ESS\}\}\{n\}\\;\\longrightarrow\\;\\Big\[\\frac\{c\}\{1\-\\varepsilon\}\+\\frac\{1\-c\}\{\\varepsilon\}\\Big\]^\{\-1\}\\;\\approx\\;\\frac\{\\varepsilon\}\{1\-c\},*wherec=1−\|Fs~​\(q~\)−Fs~​\(0\)\|c=1\-\\lvert F\_\{\\tilde\{s\}\}\(\\tilde\{q\}\)\-F\_\{\\tilde\{s\}\}\(0\)\\rvertis the mass on which logger and target agree in the limit \(q~\\tilde\{q\}the standardized cutoff, reducing toc=k\+Fs~​\(0\)c=k\+F\_\{\\tilde\{s\}\}\(0\)forq~\>0\\tilde\{q\}\>0\); an action\-aligned logger \(the same logistic centred at the top\-kkcutoff\) has an empty mismatch band and instead sharpens toward full support,ESS/n→1−ε\\mathrm\{ESS\}/n\\to 1\-\\varepsilon\. Derivation in Appendix[B](https://arxiv.org/html/2608.12489#A2)\.*

For the benchmark’s score distributions atk=0\.1k\{=\}0\.1,c≈0\.6c\\approx 0\.6, so the score\-aligned limit is≈0\.05\\approx 0\.05\. Extending the grid downward confirms both branches on the benchmark’s own data: atk=0\.1k\{=\}0\.1,τ=0\.05\\tau\{=\}0\.05, median ESS is0\.0610\.061score\-aligned but0\.9600\.960action\-aligned \(Appendix[B](https://arxiv.org/html/2608.12489#A2)\)\. Overlap is therefore governed by logger–target*action*alignment, an explicit experimental axis crossed withτ\\tau\(Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\)\.IHDP, which has known potential\-outcome means but no recorded propensity, instead draws actions fromπb\\pi\_\{b\}and rewards from the known surface, so it retains every unit at every temperature — the sole constant\-nncontrol for separating overlap from sample\-size loss \(Appendix[J](https://arxiv.org/html/2608.12489#A10)\)\. Two properties documented in Appendix[M](https://arxiv.org/html/2608.12489#A13)matter downstream: the logistic is centred at the score*mean*, not the budget cutoff, soτ\\tauis comparable across learners but not across budgets; and rejection sampling retains fewer units at lowτ\\tau\. Appendix[E](https://arxiv.org/html/2608.12489#A5)re\-runs the full sweep with the cutoff\-centred logger\. We keep the mean\-centred sweep primary deliberately: it spans the wider overlap range that a risk\-*ranking*study needs\. Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)reports both failure gradients\. In the main sweepssis the*candidate’s own*score \(*self\-aligned*logging\): the logger shares the candidate’s score, not its top\-kkaction boundary, and is not the target policy\. This isolates the overlap axis; Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)makes alignment itself an axis\.

#### Estimators, metrics, and protocol\.

Through one interface we evaluate six estimators that all target the fixed deterministic policyV⁡\(πe\)V\(\\pi\_\{e\}\)and are therefore ranked on a common estimand:DM,IPS, self\-normalizedSNIPS\([21](https://arxiv.org/html/2608.12489#bib.bib6)\), doubly\-robustDR\([8](https://arxiv.org/html/2608.12489#bib.bib5)\),Switch\-DR\([24](https://arxiv.org/html/2608.12489#bib.bib7)\), andmIPS, an IPS variant whose propensity is mixed with a uniform\-exploration component \(retained for completeness; its median relative RMSE lands within0\.0010\.001ofIPSon every dataset; both tuned hybrids are retained because their data\-driven tuning collapsing to the untuned parents is itself a finding, Appendix[B](https://arxiv.org/html/2608.12489#A2)\)\. A perturbation\-smoothed DR targets a*different*, smoothed\-policy estimand; it is reported separately, scored against its own matched reference \(Appendix[I](https://arxiv.org/html/2608.12489#A9)\)\. All model\-based estimators share one LightGBM\([12](https://arxiv.org/html/2608.12489#bib.bib20)\)outcome model per cell, and the two hybrids’ hyper\-parameters \(Switch\-DR’s threshold, mIPS’s mixing weight\) are tuned on the logged sample by an estimated\-MSE proxy rather than hard\-coded, so no estimator gets a hand\-picked advantage\. All estimators are our own implementations behind one interface \(so nuisances are genuinely shared\);IPS,SNIPS,DMandDRagree with Open Bandit Pipeline’s reference implementations to floating\-point precision on identical inputs \(make reference\-check\)\. Because outcome scale spans∼30×\\sim\\\!30\\timesacross datasets we report*relative*RMSE, RMSE/\|Vref\|/\|V\_\{\\mathrm\{ref\}\}\|, per cell across seeds and aggregated by median \(robust to the heavy IPS tail\)\. Each dataset is split50/5050/50: scoring rules are fit on*train only*; logged feedback, the reference value and all estimators operate on the evaluation split\. Formal definitions, aggregation formulas, leakage guards and the anomaly validator are in Appendix[H](https://arxiv.org/html/2608.12489#A8)\.

We sweep ten seeds, six budgetsk∈\{0\.1,0\.2,0\.3,0\.5,0\.7,1\.0\}k\\in\\\{0\.1,0\.2,0\.3,0\.5,0\.7,1\.0\\\}, three temperatures and three candidate policies \(T\-learner,S\-learner, random baseline\)\. A*cell*is a \(dataset, candidate policy, budget, temperature\) tuple — the unit over which relative RMSE is computed, aggregating its ten seeds — giving270270cells across the five datasets \(2,7002\{,\}700configurations\); paired comparisons and cluster bootstraps resample the unit named at each use\. The full\-budget pointk=1k\\\!=\\\!1is degenerate for*selection*and excluded from RQ4; it is retained for RQ1–RQ2, where excluding it leaves the ordering unchanged\. The optimizer’s\-curse study uses a separate protocol with the policy fit*in\-sample*\.

#### Roadmap\.

Table[1](https://arxiv.org/html/2608.12489#S4.T1)maps each sweep to the axis it varies and the section that consumes it; the rows map onto themaketargets of Appendix[B\.1](https://arxiv.org/html/2608.12489#A2.SS1), runnable from the anonymized repository accompanying this submission; full reproduction takes roughly1717hours on a1010\-core laptop \(Appendix[B\.1](https://arxiv.org/html/2608.12489#A2.SS1)\)\.111[https://github\.com/binshuangli/allocation\-ope\-bench](https://github.com/binshuangli/allocation-ope-bench)\.

Table 1:The experimental surface at a glance\. Each sweep varies the named axes with everything else held fixed; Appendix[B\.1](https://arxiv.org/html/2608.12489#A2.SS1)lists the exact targets\.
### 4\.1What this benchmark cannot tell you

Five limits bound every claim below; we state them together so the results sections can report findings at their supported strength without relitigating scope\.

- •Which loggers are hard to estimate\.Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3)measures what estimated propensities costIPS— the largest degradation we observe — but our logger is a smooth function of a fitted uplift score, correspondingly hard to recover\. A simpler real\-world logger would be estimated far more accurately, so read the3737–63%63\\%failure rates as the hard case\. Propensities remain floored at0\.020\.02\.
- •Observational logs\.All logged feedback is synthesized, never observed from a deployed logger — which buys exact propensities and a controlled alignment axis at the cost of realism, and is what[22](https://arxiv.org/html/2608.12489#bib.bib1)have that we do not\. The control is not merely cheaper, though: the RQ2 claim requires*varying*logger–target alignment while target, data and truth stay fixed, and an observational log supplies one point on that axis, not the axis\. That one point would test whether the diagnostics, computed on a real log, rank its realized error consistently with the synthesized axis — a spot check — but could not test the alignment mechanism \(no counterfactual logger exists for the same population\) nor calibrate cut points \(a single environment\)\. Adding it is the concrete next step — a naturally logged bandit dataset with recorded propensities\([17](https://arxiv.org/html/2608.12489#bib.bib10)\)— but not a drop\-in: its multi\-action recommendation setting is not equal\-cost binary top\-kkallocation, so the adaptation changes the object of study and is work of its own\.
- •Populations, and whose surfaces they are\.Seven of eight known\-effect regimes share theIHDPcovariate matrix, so5858–92%92\\%is variation in response surfaces, not populations; both hardening suites reuse calibration covariates, so transfer is held out at the DGP level only\. The exact\-value evidence therefore remains synthetic or semi\-synthetic throughout; Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)removes that reliance on simulated response surfaces with a reference read off recorded paired outcomes, where the RQ1 ordering \(at a larger margin,5\.0×5\.0\\timesagainst2\.0×2\.0\\times\), the RQ2 alignment mechanism and the RQ3 cross\-fitting sign all replicate while the calibrated cut points do not\.
- •Inference\.Five primary datasets — one,Jobs, noise\-dominated at132132–203203units — so no cluster interval carries much information, and all intervals are*conditional*, understating uncertainty for model\-based estimators: measured coverage on exact\-value cells is0\.890\.89forDMagainst the nominal0\.950\.95, andIPSfalls to0\.890\.89in the flagged regime \(Appendix[L\.1](https://arxiv.org/html/2608.12489#A12.SS1)\)\.
- •Scope\.Binary action, unit costs, top\-kkfraction; gross valueV⁡\(πe\)V\(\\pi\_\{e\}\), whose relative RMSE partly rewards baseline predictability and so flattersDM— onTwinsit compresses anIPSerror of62%62\\%of the achievable policy spread to a relative RMSE of0\.0160\.016\(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\)\. Clipped IPS and shrinkage\-DR are both evaluated \(Appendix[B](https://arxiv.org/html/2608.12489#A2)\); perturbation\-DR is scored only against its own smoothed\-policy reference \(Appendix[I](https://arxiv.org/html/2608.12489#A9)\)\. The floor’s no\-op argument is specific to tail control \(clipping, switching, shrinkage\); the balancing\-weights family is outside the evaluated set — a stated boundary, not a finding\.

## 5Results

What survives everything below, in two sentences: overlap risk is governed by logger–target*action*alignment and is rankable*across*logging environments given credible propensities — not within one log; and under policy–evaluation reuse, only policy\-level honesty addresses the optimism — nuisance\-only cross\-fitting makes it*worse*\. The rest of this section states each finding at the strength its evidence supports, organized around four questions: accuracy \(RQ1\), overlap diagnostics \(RQ2\), the optimizer’s curse \(RQ3\), policy selection \(RQ4\) — with the outcome\-model\-quality axis between RQ2 and RQ3 and further robustness checks in Appendix[C](https://arxiv.org/html/2608.12489#A3)\.

### 5\.1RQ1 — Which estimators are accurate?

Table 2:RQ1 estimator accuracy: median relative RMSE \(RMSE/\|Vref\|/\|V\_\{\\mathrm\{ref\}\}\|, normalized by the configuration\-level reference — exact where known, else the HT estimate\); lower is better, best per column in bold\. The last two columns re\-run the benchmark with a five\-fold*out\-of\-fold*μ^\\hat\{\\mu\}\(make repro\-full\-oof\), all else fixed;IPS,SNIPSandmIPSuse no outcome model and are identical across the two, so the change isolates nuisance honesty: theDM/DRordering is unchanged, the model\-based advantage on exact\-value data persists, and on the RCT\-reference column all six fall within0\.0030\.003, which we do not read as a ranking\. Exact\-value and HT\-reference averages are reported*separately*, not pooled: they measure error against different references\. Column blocks, left to right: per\-dataset \(in\-sampleμ^\\hat\{\\mu\}\), then cross\-dataset means in two blocks — in\-sample and out\-of\-fold — each split exact\-value / HT\-reference\. Excluding the noise\-dominatedJobs, the in\-sample HT\-reference average separates by only0\.0020\.002\(DM0\.0240\.024,IPS0\.0260\.026; Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)\)\.Table 3:The same in\-sample errors under the*incremental\-value*renormalization,\|V⁡\(πe\)−V⁡\(∅\)\|\|V\(\\pi\_\{e\}\)\-V\(\\varnothing\)\|, on the exact\-value datasets \(Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)\)\. This is the metric that removes the baseline\-predictability component gross value rewards: the family ordering persists, and theDM/DRgap closes\.Table[2](https://arxiv.org/html/2608.12489#S5.T2)and Figure[3](https://arxiv.org/html/2608.12489#A7.F3)report median relative RMSE for the six estimators targeting the fixed deterministicV⁡\(πe\)V\(\\pi\_\{e\}\)\(perturbation\-DRtargets a smoothed\-policy estimand; Appendix[I](https://arxiv.org/html/2608.12489#A9)\)\. Model\-based estimators lead on four of five datasets and in aggregate; what matters is the*magnitude*of that lead and how much survives the checks below\.

The size of the gap depends on what we can compare against\.We report the cross\-dataset average separately by reference type, not pooled: the two measure error against different objects\. On the*exact\-value*datasets, where the comparison is against true potential outcomes, the gap is large:DM0\.0290\.029and the DR family0\.0300\.030versus0\.0570\.057for the IPS family, a factor of∼2\\sim\\\!2\. On the*HT\-reference*datasets the ordering compresses to near\-nothing: the DR family is nominally best \(0\.0370\.037\) againstIPS/mIPS0\.0400\.040andDM0\.0380\.038, a spread of0\.0030\.003we do not read as a ranking\. A noisy reference adds a common error floor that shrinks measurable differences; the next paragraph shows why\. The gap is widest onIHDP, where limited sample size \(n=672n\\\!=\\\!672\), treatment imbalance \(π1=0\.18\\pi\_\{1\}\\\!=\\\!0\.18\) and weak\-overlap weights let a handful of observations dominate the IPS average: relative RMSE≈0\.052\\approx\\\!0\.052, nearly4×4\\timesthat ofDM\(0\.0140\.014\)\.SNIPShelps but does not close the gap\.

RCT\-reference comparisons cannot reliably resolve estimator ordering\.Appendix[P](https://arxiv.org/html/2608.12489#A16)re\-runs all three randomized datasets under a matched design varying only whether the HT reference is computed on the*same*units the estimators see or on a*disjoint*held\-out third — so the contrast isolates reference dependence\. The ordering moves with the reference\. Under the shared one, a weighting or DR estimator is best on all three datasets andDMon none\. Under the disjoint one,DMis best on all three, andIPSfalls to last onJobs\(0\.16280\.1628\) though not worst everywhere \(DRonHillstrom,SNIPSonLenta\)\. Every estimator’s error also rises by4848–114%114\\%— several times the spread across estimators, so the reference supplies most of the measured error\. That reference is itself an inverse\-propensity construction, so on shared units it shares sampling noise withIPSand the two agree for reasons unrelated to accuracy — with three datasets and within\-column differences of0\.0010\.001–0\.0090\.009on the two large RCTs, evidence consistent with that mechanism rather than an isolation of it\. It is why we decline to read theIPScolumn of Table[2](https://arxiv.org/html/2608.12489#S5.T2)as a ranking\.

Jobsamplifies it: at132132–203203retained units all six estimators land within0\.0580\.058–0\.0680\.068, and excluding it the HT\-reference average separates by only0\.0020\.002\(DM0\.0240\.024,IPS0\.0260\.026\) — a direction, not a ranking\.

#### Which differences are real?

Configurations share data, so we test the ordering with a paired comparison: the per\-configuration signed difference in relative RMSE between two estimators, bootstrapped over dataset clusters\.IPSis worse thanDRby\+0\.015\+0\.015over all budgets \(95% CI\[\+0\.005,\+0\.027\]\[\+0\.005,\+0\.027\];\+0\.013\+0\.013on thek<1k<1grid\), and worse in72\.6%72\.6\\%of cells\.Switch\-DRis statistically indistinguishable fromDR\(−0\.0002\-0\.0002\)\.

Two cautions attach to that number\. The interval is a five\-cluster bootstrap, narrow because the within\-dataset gaps are similar rather than because five clusters carry much information; the same applies to every cluster interval here\. And\+0\.015\+0\.015is a*mean*paired difference, against a median of\+0\.007\+0\.007\(what Table[2](https://arxiv.org/html/2608.12489#S5.T2)reports\): the mean is larger because the right tail dominates it, and that tail is the practically important part\.

These are self\-aligned\-logger magnitudes; Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)shows theIPSpenalty is several times larger once the logger disagrees with the target\.

Table[3](https://arxiv.org/html/2608.12489#S5.T3)reinforces this by removing the baseline component of the gross\-outcome value: accuracy is renormalized by the*incremental*value magnitude\|V⁡\(πe\)−V⁡\(∅\)\|\|V\(\\pi\_\{e\}\)\-V\(\\varnothing\)\|\(treat\-none baseline\) on the exact\-value datasets\. The model\-based\-vs\-weighting ordering persists: theIPSfamily sits at0\.1420\.142–0\.2150\.215against0\.0350\.035–0\.1250\.125forDM/DR\. ButDMandDRbecome indistinguishable on synthetic \(0\.1240\.124vs\.0\.1250\.125\), confirming that part ofDM’s gross\-value edge reflects baseline predictability\. This removes a reason to preferDMon that renormalization; it is not itself an argument forDR, whose case rests on Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\.

The family comparison does not depend on in\-sample nuisance fitting\.Under a five\-fold*out\-of\-fold*μ^\\hat\{\\mu\}the model\-based family still beatsIPSin8989–100%100\\%of the nuisance\-cross\-fitting check’s7272configurations \(four exact\-value DGPs only;6262–68%68\\%over the whole benchmark’s225225, where the HT\-reference datasets compress every gap\), and re\-running the whole benchmark that way \(Table[2](https://arxiv.org/html/2608.12489#S5.T2); a controlled comparison, sinceIPS/SNIPS/mIPSare numerically identical across blocks\) leaves the family gap intact and the exact\-valueDM/DRordering unchanged, thoughDMbecomes worst of six onJobsandLenta, and worst of six on the HT\-reference mean itself \(0\.0410\.041; Table[11](https://arxiv.org/html/2608.12489#A14.T11)\)\.

The practical read is that model\-based estimators are more stable than pure importance\-weighting estimators*when outcome models are reasonably predictive*— a condition that should be assessed with treatment\-arm\-specific held\-out errors, calibration checks, and sensitivity across outcome\-model classes, not assumed; we stress it directly in Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1), where severe outcome\-model misspecification can invert even the within\-family ordering on the nonlinear synthetic DGP\. Doubly\-robust correction adds a safety margin; raw IPS is fragile precisely under the deterministic target this paper studies\.

The propensity floor shapes the estimator set itself\.Atε=0\.02\\varepsilon\{=\}0\.02importance weights are capped at5050, and that ceiling — not the tuning — is why the hybrids duplicate their parents:Switch\-DRis bit\-for\-bit identical toDRin96\.6%96\.6\\%of cells andmIPSreduces exactly toIPSin75\.0%75\.0\\%, while clipping is a no\-op \(Appendix[B](https://arxiv.org/html/2608.12489#A2)\)\. Sweeping the floor itself toε=0\.0002\\varepsilon\{=\}0\.0002locates where that regime ends: tail control begins to matter once weights exceed the5050ceiling \(medianIPS0\.1110\.111against clipped0\.0900\.090atε=0\.005\\varepsilon\{=\}0\.005\) whileDRstays flat \(Appendix[N](https://arxiv.org/html/2608.12489#A14)\)\. We keepε=0\.02\\varepsilon\{=\}0\.02primary deliberately: failures under bounded weights cannot be attributed to a few extreme weights \(the more alarming case, as Section[1](https://arxiv.org/html/2608.12489#S1)argues\), and the sweep bounds what changes below it\. “Clipping does not help” is a property of the bounded\-weight regime we chose, not of allocation OPE\.

### 5\.2RQ2 — Can you tell, in advance, when to trust OPE?

Accuracy is not constant: it is governed by overlap, overlap is governed by how much the logger disagrees with the policy being evaluated, and both are observable without ground truth\.

Figure 1:RQ2 — overlap risk is a property of logger–target*misalignment*, not of logging sharpness alone\. Three logger regimes crossed withτ\\tau, target fixed\. Sharpening a*self\-aligned*\(score\-aligned\) logger barely moves overlap over this range \(left\) — a finite\-range plateau, not a guarantee \(Appendix[B](https://arxiv.org/html/2608.12489#A2)\); once logger actions disagree with the target’s, lowτ\\taucollapses the effective sample, drives the maximum weight to the0\.020\.02\-floor ceiling \(centre\), and raises the IPS failure rate \(cells with median\-over\-seeds\|\|relative bias\|\>10%\|\>10\\%\) from8%8\\%to32%32\\%\(third panel\)\. The fourth panel shows the same failure rates under budget\-cutoff centring —0\.0%0\.0\\%,1\.7%1\.7\\%and26\.7%26\.7\\%atτ=0\.5\\tau\{=\}0\.5, with failure atτ≥2\\tau\\geq 2near zero \(Appendix[E](https://arxiv.org/html/2608.12489#A5)\): the self\-aligned failures follow from score\-mean centring, and the misalignment gradient survives the correction\.The overlap axis is alignment, not temperature\.We cross three logger regimes withτ∈\{0\.5,2\.0,5\.0\}\\tau\\in\\\{0\.5,2\.0,5\.0\\\}:*self\-aligned*, built from the evaluated candidate’s score but not its top\-kkboundary \(the main sweep’s design\);*misaligned*, built from the other learned candidate, a plausible incumbent; and*independent*, built from a candidate\-independent score\. Seeds, splits, fitted candidates and truth labels are held fixed, so differences are attributable to the logger alone \(540540cells\)\. Retainednnis not a confound — the regimes retain near\-identical medians \(1,7931\{,\}793/1,7931\{,\}793/1,8001\{,\}800atτ=0\.5\\tau\{=\}0\.5onsynthetic\), andIHDPholdsn=336n\{=\}336in all three\.

Figure[1](https://arxiv.org/html/2608.12489#S5.F1)reports the result\. Atτ=0\.5\\tau\{=\}0\.5the median maximum weight runs4\.0→16\.2→50\.04\.0\\to 16\.2\\to 50\.0across the regimes, median ESS runs0\.56→0\.42→0\.190\.56\\to 0\.42\\to 0\.19, andIPSfailure rates run8\.3%→13\.3%→31\.7%8\.3\\%\\to 13\.3\\%\\to 31\.7\\%\(2\.8%→4\.4%→11\.7%2\.8\\%\\to 4\.4\\%\\to 11\.7\\%pooled, Table[8](https://arxiv.org/html/2608.12489#A7.T8)\)\. Under the cutoff\-centred logger the same gradient is0\.0%0\.0\\%/1\.7%1\.7\\%/26\.7%26\.7\\%\(Appendix[E](https://arxiv.org/html/2608.12489#A5); Section[4](https://arxiv.org/html/2608.12489#S4)states why the mean\-centred sweep stays primary\)\. The screen does not depend on that choice: on the cutoff\-centred sweep the association replicates,ρ⁡\(ESS\)=−0\.32\\rho\(\\text\{ESS\}\)=\-0\.32pooled and−0\.58\-0\.58under the independent logger, against−0\.41\-0\.41and−0\.56\-0\.56on the primary sweep — the centring choice moves*which*cells fail, not whether ESS ranks failure\. Within the self\-aligned regime, temperature does almost nothing\. The practitioner’s question is therefore not “how smooth was my logger” but “how far is the policy I want to deploy from the one that collected my data”\.

Can a practitioner detect this without ground truth? Across the540540cells the logged\-data diagnostics do rank realizedIPSerror:ρ⁡\(ESS\)=−0\.41\\rho\(\\text\{ESS\}\)=\-0\.41,ρ⁡\(support deficiency\)=\+0\.42\\rho\(\\text\{support deficiency\}\)=\+0\.42,ρ⁡\(max weight\)=\+0\.35\\rho\(\\text\{max weight\}\)=\+0\.35— smaller than a temperature\-only design would report\.

Cells share datasets, seeds and policies, so a cell\-levelpp\-value would be anti\-conservative\. The trustworthy evidence is*consistency*: within each dataset the ESS correlation ranges−0\.35\-0\.35to−0\.71\-0\.71and support deficiency\+0\.46\+0\.46to\+0\.69\+0\.69, and leave\-one\-dataset\-out values stay at\|ρ\|=0\.37\|\\rho\|=0\.37–0\.450\.45\.

Appendix[G](https://arxiv.org/html/2608.12489#A7)works through the conditioning ladder in Table[8](https://arxiv.org/html/2608.12489#A7.T8): the ranking survives conditioning on dataset \(negative in5/55/5,−0\.35\-0\.35to−0\.71\-0\.71\) and on budget \(AUC0\.680\.68–0\.950\.95across strata\), and survives within the logger regime that actually produces failures \(ρ=−0\.56\\rho=\-0\.56under an independent logger,11\.7%11\.7\\%of cells unsafe\); but it*collapses*under joint regime\-and\-τ\\tauconditioning \(median−0\.01\-0\.01, negative in66of99strata\)\. We read that as expected rather than damaging — regime×\\timesτ\\tau*is*the mechanism generating overlap variation here, so fixing both removes what ESS exists to rank — but it does narrow the claim: ESS ranks risk*across*the logging situations a practitioner might face, not*within*a fixed one\.

The diagnostics also generalize beyond the data they were calibrated on, which is the practically useful test\. A simple*fragility screen*— best understood as a risk\-*ranking*heuristic, not a decision rule — is fit on this benchmark \(flag a cell when ESS fraction<0\.36<0\.36or support deficiency\>0\.02\>0\.02, targeting\|\|relative bias\|\>10%\|\>10\\%\)\. ROC\-AUCs are for the*continuous*ESS fraction, not the binary rule \(ESS ROC\-AUC0\.850\.85in\-sample\) and applied to the two held\-out hardening suites, run on the same alignment axis, whose DGPs were not used to calibrate it\. On the IHDP\-covariate suite it flags21%21\\%of cells, with error rate52%52\\%among flagged versus7%7\\%among unflagged \(base rate17%17\\%; ESS ROC\-AUC0\.830\.83; precision0\.520\.52, recall0\.670\.67\)\. These quantities need only the logged actions and propensities, so the indicator is computable before believing any estimate\.

The second suite reports a boundary rather than a failure\.On theHillstrom\-covariate suite*no*cell reaches10%10\\%relative bias, so the AUC is undefined there\. The cause is sample size, not a breakdown of the diagnostic: it retains a median5,0005\{,\}000logged units against336336and2,0722\{,\}072elsewhere, with median\|\|relative bias\|\|0\.0120\.012against0\.0560\.056and0\.0350\.035\. At a target matched to its own error scale the screen is at its strongest anywhere in the paper \(ROC\-AUC0\.910\.91atδ=2%\\delta\\\!=\\\!2\\%, precision0\.580\.58, recall0\.840\.84\); we report the fullδ\\deltagrid \(Table[7](https://arxiv.org/html/2608.12489#A7.T7), Appendix[G](https://arxiv.org/html/2608.12489#A7)\) rather than a per\-suite best, since choosingδ\\deltapost hoc would fit the held\-out data\. The*ranking*transfers while the*cut point*does not — recalibrate on your ownnn\. Appendix[E](https://arxiv.org/html/2608.12489#A5)re\-runs the benchmark under the cutoff\-centred logger\.

Within a single log the screen is much weaker\.Those correlations are*across*logging situations; a practitioner holds one\. Fixing dataset, regime, temperature and seed and ranking the1212candidate×\\timesbudget targets on that one log \(450450logs\) gives medianρ=−0\.11\\rho=\-0\.11, correct in sign in58%58\\%of logs and no better than chance under the candidate\-independent logger; it is negative only where overlap varies within the log \(−0\.30\-0\.30atτ=0\.5\\tau\{=\}0\.5,0\.000\.00atτ=5\\tau\{=\}5\)\. ESS ranks*which logging situation*carries risk, not which candidate to trust within one\.*Absolute*ESS ranks no better in practice \(0\.8580\.858against the fraction’s0\.8510\.851; identical on each suite, where constantnnmakes them monotone transforms\): what fails to transfer is the mapping from diagnostic value to error level, not the statistic\.

### 5\.3RQ2, continued — estimated propensities are the largest degradation, and can invert the diagnostic

Everything above hands the weighting estimators the exactπb\\pi\_\{b\}; in real observational logs propensity\-estimation error is first\-order\. We re\-run the alignment sweep withπb\\pi\_\{b\}replaced by an out\-of\-foldπ^b\\hat\{\\pi\}\_\{b\}\(LightGBMonxx, logistic onxx, or the sample treat rate\), floored at the same0\.020\.02\(Table[9](https://arxiv.org/html/2608.12489#A7.T9)\)\. Weights*and*diagnostics useπ^b\\hat\{\\pi\}\_\{b\}: a screen that works only on propensities you do not know would be useless\.

ForIPS, estimation error dominates everything else measured here\.The share of cells in whichIPSexceeds10%10\\%relative bias rises from6\.3%6\.3\\%to3737–63%63\\%— against2\.8%2\.8\\%to11\.7%11\.7\\%for moving the logger regime itself, on the same cell\-level unit\. Our logger is a smooth function of a fitted uplift score; the bestπ^b\\hat\{\\pi\}\_\{b\}correlates onlyρ=0\.30\\rho=0\.30with the truth \(mean absolute error0\.1560\.156; the logistic and marginal reach0\.1260\.126/0\.1290\.129, but only the marginal loses the rank information,ρ=−0\.02\\rho=\-0\.02against\+0\.27\+0\.27\)\. The degradation is*not*monotone in model quality\. The flexibleπ^b\\hat\{\\pi\}\_\{b\}damagesIPSmost \(median per\-cell max weight11\.811\.8against2\.12\.1for the marginal model\), trading bias for variance thatIPScannot absorb\. The*screen*, by contrast, degrades monotonically, because what it loses is dispersion information\. The table’s twoIPScolumns therefore point opposite ways, and the logistic model is least damaging by median error while still failing half its cells\.DMis unchanged by construction \(the pipeline check\),DRandSwitch\-DRare essentially immune \(0\.037→0\.0360\.037\\to 0\.036–0\.0420\.042\), andIPSdegrades1\.91\.9–4\.9×4\.9\\times: the outcome\-model term absorbs what the propensity model gets wrong\.

The screen is only as good as the propensity model, and can invert\.Under a flexibleπ^b\\hat\{\\pi\}\_\{b\}it is undamaged \(ESS ROC\-AUC0\.840\.84, matching the0\.850\.85it reaches with the exact propensity\); under a logistic model it is a coin flip \(0\.540\.54\); under the marginal model it is*actively misleading*\(AUC0\.050\.05, medianIPSerror rising from0\.0370\.037in the lowest estimated\-ESS quartile to0\.3490\.349in the highest\)\. The mechanism is specific: ESS factors as coverage×\\timesdispersion, and a near\-constantπ^b\\hat\{\\pi\}\_\{b\}carries no dispersion, soESS^\\widehat\{\\mathrm\{ESS\}\}collapses to coverage — which is highest exactly where the true propensity is most extreme\. The inversion is systematic, not noise \(Appendix[F](https://arxiv.org/html/2608.12489#A6)\)\. The propensity model is a*prerequisite*, not a caveat\.

#### The second axis: outcome\-model quality\.

The RQ1 lead of model\-based estimators is conditional on an adequate outcome model, so we degradeμ^\\hat\{\\mu\}along a ladder \(LightGBM→\\tostumps→\\toridge\)\. The effect is DGP\-specific and, on the nonlinearsyntheticsurface, decisive: degradingμ^\\hat\{\\mu\}makesDM*worst*of the four compared \(0\.036→0\.0990\.036\\to 0\.099, overtakingIPSat0\.0580\.058\) whileDRdegrades gracefully \(0\.037→0\.0490\.037\\to 0\.049\) and never becomes worst\. The crossing is beyond seed noise: seed\-bootstrap intervals put theDM−\-IPSgap at the stump rung at\[\+0\.022,\+0\.063\]\[\+0\.022,\\,\+0\.063\]—DMworse in100%100\\%of2,0002\{,\}000seed resamples — whileDR’s degradation interval,\[−0\.001,\+0\.019\]\[\-0\.001,\\,\+0\.019\], is consistent with graceful\. On the other two DGP familiesDMstays best or within a whisker at every rung, so the inversion is a property of one family out of three\. The practical map is therefore two\-dimensional — overlap bounds weighting estimators, outcome\-model quality bounds model\-based ones — andDRis the safer default when adequacy is uncertain\. Figure[2](https://arxiv.org/html/2608.12489#S5.F2)draws it\.

overlap with the target’s actionsoutcome\-model adequacymodel\-based \(DM/DR\)weighting fails \(RQ2\)both families usable;model\-based leads\(RQ1\)no estimator here is safeimprove the log or the modelbefore trusting OPEweighting \(IPS/SNIPS\)DMexposed;DRdegrades gracefully \(§[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\)DR: safer default underuncertainty on either axisFigure 2:Schematic: the two\-dimensional decision map of Sections[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)–[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1), drawn\. Positions are qualitative, not calibrated coordinates\. The overlap axis is rankable*across*logging environments byESS^\\widehat\{\\mathrm\{ESS\}\}given credible propensities \(Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\) — not within one log, and the cut points do not transfer\. The adequacy axis has*no*transferable screen \(the negative result of Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\); judge it by held\-out error per arm\.We also report a*negative*result: the obvious analogue of the ESS screen for model fragility, out\-of\-fold factual RMSE ofμ^\\hat\{\\mu\}, does*not*transfer \(ρ≈−0\.10\\rho\\\!\\approx\\\!\-0\.10, AUC0\.480\.48\), because factual predictive error conflates irreducible noise with structural misfit\. Full grid in Appendix[N](https://arxiv.org/html/2608.12489#A14)\.

### 5\.4RQ3 — The optimizer’s curse: what addresses policy–evaluation reuse?

When the candidate policy is*learned on the same data used to evaluate it*, the optimizer selects units whose scores are inflated by their own estimation noise, and plain DR overstates the deployed value\([19](https://arxiv.org/html/2608.12489#bib.bib15)\)\. Fair comparison requires care with*estimands*\. We compare two fixed\-policy estimators — plainDRand a*frozen\-policy*cross\-fitted DR \(in\-sample policy fixed; only the nuisance fit out\-of\-fold\) — each scored against the true value of that same in\-sample\-optimized policy\. Separately we evaluate the*honest fold\-policy pipeline*: the policy is re\-learned per fold on out\-of\-fold data and evaluated on the held\-out fold, so its estimand is the value of the learning*algorithm*, scored against its own fold\-matched exact reference\. Mixing the two would conflate curse removal with the estimand gap, so we do not; “true value” is*exact*on the datasets carrying the result\. \(Perturbation\-DRtargets a third estimand; Appendix[I](https://arxiv.org/html/2608.12489#A9)\.\)

Table 4:RQ3 optimizer’s curse, per dataset: plain\-DR\|\|bias\|\|\(in\-sample policy value\); the\|\|bias\|\|change from nuisance\-only cross\-fitting \(same fixed\-policy estimand\); and the bias reduction under honest*algorithm*evaluation \(a different estimand — the learning procedure’s value\)\. “Material optimistic bias” is an explicit criterion — plain DR’s mean*signed*bias exceeds2%2\\%of\|Vref\|\|V\_\{\\mathrm\{ref\}\}\|\(third column\) — and signed on purpose:Jobs\(−0\.2%\-0\.2\\%\) andLenta\(−3\.8%\-3\.8\\%\) carry non\-trivial\|\|bias\|\|but no optimism, so there is no upward reuse bias to remove;IHDP\(\+6\.9%\+6\.9\\%\) andsynthetic\(\+23\.6%\+23\.6\\%\) are flagged\. DR\|\|bias\|\|is shown raw and as a fraction of\|Vref\|\|V\_\{\\mathrm\{ref\}\}\|, since raw magnitudes are not comparable across outcome scales\. Italic entries are out of scope — on those datasets there is no material upward reuse bias to remove, so a “% removed” is not comparable to the flagged rows; they are shown for completeness and nothing in the text depends on them\.Jobs’\+8\.4\+8\.4is the only cell in the benchmark where nuisance\-only cross\-fitting helps, on the noise\-dominated dataset\.The central result: cross\-fitting the outcome nuisance — the reflexive fix — does not remove the reuse bias but makes it*worse*\.With the policy frozen, an out\-of\-foldμ^\\hat\{\\mu\}makes the DR correction*honest*, and that is the problem: the optimizer selected units whose scores were inflated by their own noise\.

*Observation \(informal\)\.*For a policy selected data\-adaptively on the evaluation sample, in the regimes we study the conditional residual is positive —𝔼⁡\[y−μ^∣selected\]\>0\\mathbb\{E\}\[y\-\\hat\{\\mu\}\\mid\\text\{selected\}\]\>0under an honestμ^\\hat\{\\mu\}— while an in\-sampleμ^\\hat\{\\mu\}shrinks exactly that conditional residual\. The DR correction averagesw⁡\(y−μ^\)w\\,\(y\-\\hat\{\\mu\}\), so honesty adds the optimism back\. The difference between the two pipelines is in fact exact algebra: by linearity of DR,

V^in−V^honest=1n∑i\(1−wi1\[ai=πe\(xi\)\]\)\(μ^in−μ^honest\)\(xi,πe\(xi\)\),\\widehat\{V\}\_\{\\text\{in\}\}\-\\widehat\{V\}\_\{\\text\{honest\}\}=\\tfrac\{1\}\{n\}\\textstyle\\sum\_\{i\}\\big\(1\-w\_\{i\}\\,\\mathbb\{1\}\[a\_\{i\}\{=\}\\pi\_\{e\}\(x\_\{i\}\)\]\\big\)\\,\\big\(\\hat\{\\mu\}\_\{\\text\{in\}\}\-\\hat\{\\mu\}\_\{\\text\{honest\}\}\\big\)\(x\_\{i\},\\pi\_\{e\}\(x\_\{i\}\)\),whose expectation is−Cov\(w1\[a=πe\],μ^in\)\-\\mathrm\{Cov\}\\big\(w\\,\\mathbb\{1\}\[a\{=\}\\pi\_\{e\}\],\\,\\hat\{\\mu\}\_\{\\text\{in\}\}\\big\)once cross\-fitting makesμ^honest\\hat\{\\mu\}\_\{\\text\{honest\}\}independent of the evaluation draw*and*the propensities are known, so𝔼\[w1\[a=πe\]∣x\]=1\\mathbb\{E\}\[w\\,\\mathbb\{1\}\[a\{=\}\\pi\_\{e\}\]\\mid x\]=1— as they are here by construction\. The sign condition is that the in\-sample nuisance be elevated exactly where the estimator’s weighted observation indicator is — which own\-observation fitting produces and cross\-fitting removes\. The identity is exact; whether the covariance is positive and*material*for a given learner and logging regime is empirical, and that is what the eight regimes and the ablation test\. The frozen\-policy cross\-fit is therefore*more*optimistic than plain DR \(\+0\.25\+0\.25/\+0\.28\+0\.28vs\.\+0\.18\+0\.18/\+0\.24\+0\.24; “% removed” negative,−16%\-16\\%to−36%\-36\\%across all eight known\-effect regimes\)\.

A decoupled ablation confirms the sign is not an artifact of the policy and nuisance sharing a model: with an in\-sampleLightGBMpolicy and a different\-classDRnuisance \(make repro\-optbias\-decoupled\), cross\-fitting stays more optimistic on four of five datasets \(pooled−32%\-32\\%\)\. The exception issyntheticat\+1%\+1\\%— essentially no effect either way — where the decoupled ridge nuisance is the class Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)shows is misspecified on that nonlinear surface, so model\-class bias plausibly swamps the selection\-noise channel there\. Scope:*for the data\-adaptive top\-kklearners studied here*, frozen\-policy nuisance cross\-fitting consistently increased optimism across the eight primary known\-effect regimes; in the decoupled ablation it increased optimism on four of five datasets and was essentially neutral on the fifth\. We do not establish when the covariance is*material*— and the obvious toy model illustrates why magnitude resists simple derivation: in a homogeneous normal\-means case with mild weights the covariance term is numerically negligible, leaving total DR optimism essentially invariant toμ^\\hat\{\\mu\}there\. The sign’s form is the identity above; its size in our regimes is measured, not derived\. Seven of eight regimes share one covariate population\. The bias lives in the policy–data dependence, not in nuisance reuse\.

What the practitioner gains from honest splitting\.Split at the*policy*level: re\-learn the rule per fold on out\-of\-fold data, evaluate on the held\-out fold\. Against its own fold\-matched reference that cuts\|\|bias\|\|by91\.7%91\.7\\%and68\.5%68\.5\\%— close to definitional\. The practitioner\-facing question is how much of the optimism they would otherwise carry is removed, measured against the true value of the*full\-sample*in\-sample policy — what plain DR was estimating\. On that target the honest pipeline removes89%89\\%and69%69\\%of the per\-cell error, andIHDP’s69%69\\%nearly matches the68\.5%68\.5\\%above: the estimand gap costs little\. Seed\-level bootstrap CIs\[88,95\]%\[88,95\]\\%and\[49,81\]%\[49,81\]\\%\.

And the regime in which any of this matters\.Material bias — plain DR’s mean signed bias above2%2\\%of\|Vref\|\|V\_\{\\mathrm\{ref\}\}\|— appears only on the two datasets with continuous, known effects where the in\-sample model can overfit \(synthetic\+23\.6%\+23\.6\\%,IHDP\+6\.9%\+6\.9\\%\), not on the three binary RCTs \(−0\.2%\-0\.2\\%,−3\.8%\-3\.8\\%,−0\.0%\-0\.0\\%\), whose non\-trivial\|\|bias\|\|is not*optimistic*\. “% removed” uses the conservative mean per\-cell absolute bias throughout \(Appendix[L](https://arxiv.org/html/2608.12489#A12)\)\.

The pattern is not an artifact of two generators\.We re\-run the same comparison on the six ACIC\-style exact\-value DGPs \(Section[C\.1](https://arxiv.org/html/2608.12489#A3.SS1)\), which also have known potential outcomes and an overfittable in\-sample model\. On all six, plain DR is again optimistically biased \(signed\+0\.14\+0\.14to\+0\.89\+0\.89\), nuisance\-only cross\-fitting again*worsens*it \(−16%\-16\\%to−27%\-27\\%\), and honest policy\-level splitting again reduces the bias, by5858–82%82\\%\(per\-setting values in Table[4](https://arxiv.org/html/2608.12489#S5.T4); every seed\-bootstrap interval excludes zero\)\. With the two original generators the reduction spans5858–92%92\\%over all eight continuous/known\-effect regimes with uniform direction\. That range is over*point*estimates — individual intervals are wider and extend below its floor \(IHDP\[49,81\]%\[49,81\]\\%\); Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)bounds its breadth\.

The estimand distinction, stated once\.The89%89\\%/69%69\\%figures above are an estimand\-gap diagnostic, not a bias measurement: against the full\-sample target the honest estimate sits at\+0\.017\+0\.017and−0\.019\-0\.019versus plain DR’s*signed*\+0\.181\+0\.181and\+0\.237\+0\.237\(Table[4](https://arxiv.org/html/2608.12489#S5.T4)reports mean per\-cell absolute bias,0\.1810\.181/0\.2420\.242\), but the honest pipeline is not an unbiased estimator of that target\. We are not de\-biasing a fixed data\-adaptive quantity; we are honestly evaluating the policy\-*learning procedure*, usually the deployment\-relevant object anyway \(Appendix[L](https://arxiv.org/html/2608.12489#A12)separates the signed and absolute readings\)\.

### 5\.5RQ4 — Do estimators pick the right policy?

Finally: does OPE pick the best candidate under a budget? “Best” is the value\-maximizing candidate on the exact\-value datasets and the HT\-reference\-best policy on the RCT ones\. We exclude the full\-budget point, where every candidate treats everyone and “best” is an arbitrary tie\.

The logging design is a first\-order confounder, so we vary it explicitly\.The main sweep logs from*each candidate’s own*score\. That is right for isolating the overlap axis and wrong for selection: comparingV^j\\widehat\{V\}\_\{j\}across candidates then compares policy–logger*pairs*rather than policies, and real selection has one historical log\. We therefore re\-run selection with a*common logger*— one logged dataset per cell, identical for every candidate — either aligned with theT\-learneror candidate\-*independent*\. Table[5](https://arxiv.org/html/2608.12489#S5.T5)reports all three designs, with seeds, splits, candidates and truth labels fixed, on the three\-candidate slate of Section[4](https://arxiv.org/html/2608.12489#S4)\(baseline1/31/3\) and a competitive seven\-policy slate \(six candidates on the two continuous\-outcome exact\-value datasets, seven on the three RCT ones; Appendix[O](https://arxiv.org/html/2608.12489#A15)\), baseline≈0\.15\\approx 0\.15rather than1/71/7\.

In correct\-selection accuracy the model\-based edge survives every design; regret is more qualified\.TheDM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}gap runs\+0\.083→\+0\.105→\+0\.056\+0\.083\\to\+0\.105\\to\+0\.056across the per\-candidate, common\-aligned and common\-independent designs on the three\-candidate slate, and\+0\.044→\+0\.020→\+0\.016\+0\.044\\to\+0\.020\\to\+0\.016on the seven\-policy slate\. The design therefore does*not*dominate the estimator comparison: the gap moves by about half its own size with no consistent direction, while a competitive slate shrinks it by1\.91\.9–5\.3×5\.3\\times— a null: no detectable logging\-design effect, and a slate effect larger than any we could resolve\. With five datasets no single interval is authoritative, so Table[5](https://arxiv.org/html/2608.12489#S5.T5)layers a per\-dataset sign count, a leave\-one\-dataset\-out range and two cluster bootstraps, all conditional on this benchmark\. On the three\-candidate slate the finer bootstrap excludes zero under the common\-aligned design, sits*at*zero to resampling resolution under the per\-candidate one \(its lower bound changes sign across bootstrap seeds\) and includes zero under the common\-independent one, while dataset\-level clustering excludes it nowhere; on the competitive slate support is absent at every level\.

Normalized regret agrees on the easy slate and dissents on the competitive one\.Regret asks how costly the mistake was, not merely whether the top pick was right\. On the three\-candidate slate theDM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}regret gap favoursDMunder every logging design \(−0\.026\-0\.026,−0\.052\-0\.052,−0\.022\-0\.022; lower is better\)\. On the seven\-policy slate the sign reverses \(\+0\.045\+0\.045,\+0\.012\+0\.012,\+0\.016\+0\.016\):IPSincurs slightly*lower*regret\. The summary is simple: on the easy slate both criteria favourDM; on the competitive slate they disagree, and the disagreement tracks reference type\. The\+0\.045\+0\.045regret gap is not noise — it is the size of the winner\-accuracy gap on that slate \(\+0\.044\+0\.044\), pointing the other way — and it splits exactly as winner accuracy does: the two exact\-value datasets favourDM\(−0\.206\-0\.206,−0\.018\-0\.018\), the RCT datasets favourIPS\(Appendix[O\.1](https://arxiv.org/html/2608.12489#A15.SS1)\)\. The separation we can defend is therefore the one on exact\-value data, and neither criterion shows the gap*closing*as the logger becomes candidate\-independent \(Figure[4](https://arxiv.org/html/2608.12489#A7.F4)\)\.

Table 5:RQ4 selection quality by logging design: pairedDM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}gap, per\-dataset sign count, LODO range, two cluster bootstraps — all*conditional on this benchmark*\(k=1k\{=\}1excluded throughout RQ4\)\. Dataset\-level clustering excludes zero nowhere; a competitive slate shrinks the gap and removes support at every level \(Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)\)\.Two further readings, developed in Appendices[D](https://arxiv.org/html/2608.12489#A4)and[O](https://arxiv.org/html/2608.12489#A15)\.Pooled rates hide a reference\-type split: on exact\-value data the model\-based advantage is large even under a shared independent log \(\+0\.263\+0\.263,\+0\.311\+0\.311\), while on HT\-reference data it reverses — the reference\-alignment mechanism of Appendix[P](https://arxiv.org/html/2608.12489#A16)\. AndIPSover\-selects the easiest\-to\-evaluate candidate \(∼1\.8×\\sim\\\!1\.8\\timesits true\-best rate on the easy slate, up to2\.8×2\.8\\timeson the competitive one,DMthe same at design\-dependent strength; Table[14](https://arxiv.org/html/2608.12489#A15.T14)\) — a pathology of degree\. \(Perturbation\-DRis excluded; Appendix[I](https://arxiv.org/html/2608.12489#A9)\.\)

### 5\.6An external check on a non\-simulated reference

One check stands apart from the robustness suite because its reference is not a surface anyone modelled\. On theTwinscohort —11,40011\{,\}400same\-sex twin pairs with both siblings’ outcomes recorded, taking the pair as the unit — the RQ1 ordering replicates at a*larger*margin \(5\.0×5\.0\\timesagainst2\.0×2\.0\\times\), the RQ2 alignment mechanism replicates, and the RQ3 cross\-fitting sign replicates\. The calibrated cut points do not transfer\. That is exactly the split Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)predicts: mechanisms travel, magnitudes do not\. Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)gives the design, the matched\-pair caveat, and the full account of what fails to transfer\.

## 6A Practitioner’s Guide

1. 1\.Ask how far your target is from your logger before trusting any importance\-weighted estimate\.A log whose policy broadly agrees with the rule you want to deploy supports weighting even under sharp logging; one that ignores it does not \(RQ2\)\. ESS fraction and support deficiency work at their demonstrated scope — ranking*logging situations*, not candidates within one log \(medianρ=−0\.11\\rho=\-0\.11\) — and our cut points \(0\.360\.36,0\.020\.02\) describe this benchmark: read a low ESS as a reason to seek a better\-aligned log or a model\-based estimator, not a go/no\-go test\. Use the screen ex ante: to choose which log to collect or buy, and to judge whether an evaluation*program*on a given log–target pair is viable at all\. It does not rank candidates within the log you already hold\.Fit and check a flexible propensity model first: a poor one leaves the screen uninformative or inverted without rescuingIPSaccuracy \(Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3)\)\.
2. 2\.Prefer DR under model uncertainty; DM is competitive when adequacy is demonstrable\.DMis best or tied\-best on four of five datasets under in\-sample nuisances but only three of five with out\-of\-fold ones, where it is the worst of six onJobsandLenta\(Table[11](https://arxiv.org/html/2608.12489#A14.T11)\); it leads the exact\-value cross\-dataset mean under both and neither HT\-reference mean\. The case forDRis*asymmetric risk*: under a weak outcome modelDMcan degrade to worst\-of\-four whileDRdegrades gracefully \(Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\) — though that rests on one DGP family of three, although the crossing itself survives seed resampling \(DM−\-IPSgap\[\+0\.022,\+0\.063\]\[\+0\.022,\\,\+0\.063\]at the stump rung\)\. Judge adequacy by held\-out error*per arm*— and trust the intervals less than the points: the reported CIs are conditional on the fitted nuisance and undercover forDM\(0\.890\.89against nominal0\.950\.95\), with weighting coverage degrading in exactly the flagged low\-overlap regime \(Appendix[L\.1](https://arxiv.org/html/2608.12489#A12.SS1)\)\.
3. 3\.Against the optimizer’s curse, decide by data provenance — do not try to detect the bias\.If the policy was learned from the outcomes you evaluate on, use honest policy\-level splitting; if trained on an independent sample, plain DR needs no policy\-level split for reuse bias\. Nuisance\-only cross\-fitting*worsened*it \(RQ3\)\.
4. 4\.When*selecting*among candidates, rank them all on one shared log\.Scoring each candidate on a log aligned with itself compares policy–logger pairs, not policies \(RQ4\)\. One robust pathology:IPSover\-selects the easiest\-to\-evaluate candidate — up to2\.8×2\.8\\timesthe decoy’s true\-best rate,DMmatching it under the common\-independent logger \(Table[14](https://arxiv.org/html/2608.12489#A15.T14)\)\.
5. 5\.Always show per\-dataset and per\-regime results alongside any aggregate\.Pooling exact\-value with noisy\-reference results hides mechanism\.

## 7Conclusion

Budget\-constrained allocation is where naive offline evaluation misleads most\. Weak overlap is governed by logger–target*action*alignment, not by sharpness, and is rankable in advance*across*logging environments — not within one log, and only with a good propensity model\. Under policy–evaluation reuse, cross\-fitting the nuisance alone makes the optimism*worse*; honest policy\-level splitting avoids the reuse by targeting the learning procedure’s value, not by de\-biasing the full\-sample policy’s\. The mechanisms — though not the calibrated cut points — replicate against a non\-simulated paired reference we did not construct \(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\)\. Doubly\-robust estimation is the most stable across our stresses, Section[6](https://arxiv.org/html/2608.12489#S6)is the operational form, and every number regenerates from the released repository’s documentedmaketargets \([https://github\.com/binshuangli/allocation\-ope\-bench](https://github.com/binshuangli/allocation-ope-bench)\)\.

## Broader Impact Statement

This is protective methodology: offline estimates of allocation policies can be confidently wrong, and we supply diagnostics for when not to trust them and when those diagnostics fail\. The main risk is reading the screen as a certificate; all data is public\.Use of generative AI\.Generative AI assisted with writing and code; the authors take full responsibility\.

## References

- Almondet al\.\(2005\)D\. Almond, K\. Y\. Chay, and D\. S\. LeeThe costs of low birth weight\.The Quarterly Journal of Economics120\(3\),pp\. 1031–1083\.Cited by:[§N\.3](https://arxiv.org/html/2608.12489#A14.SS3.SSS0.Px1.p1.1)\.
- Athey and Imbens \(2016\)S\. Athey and G\. ImbensRecursive partitioning for heterogeneous causal effects\.Proceedings of the National Academy of Sciences113\(27\),pp\. 7353–7360\.Cited by:[item 2](https://arxiv.org/html/2608.12489#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px2.p1.1)\.
- Athey and Wager \(2021\)S\. Athey and S\. WagerPolicy learning with observational data\.Econometrica89\(1\),pp\. 133–161\.Cited by:[item 2](https://arxiv.org/html/2608.12489#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px2.p1.1)\.
- Austin \(2011\)P\. C\. AustinAn introduction to propensity score methods for reducing the effects of confounding in observational studies\.Multivariate Behavioral Research46\(3\),pp\. 399–424\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Chernozhukovet al\.\(2018\)V\. Chernozhukov, D\. Chetverikov, M\. Demirer, E\. Duflo, C\. Hansen, W\. Newey, and J\. RobinsDouble/debiased machine learning for treatment and structural parameters\.The Econometrics Journal21\(1\),pp\. C1–C68\.Cited by:[item 2](https://arxiv.org/html/2608.12489#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px2.p1.1)\.
- Dorieet al\.\(2019\)V\. Dorie, J\. Hill, U\. Shalit, M\. Scott, and D\. CervoneAutomated versus do\-it\-yourself methods for causal inference: lessons learned from a data analysis competition\.Statistical Science34\(1\),pp\. 43–68\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1)\.
- Douglaset al\.\(2026\)C\. Douglas, J\. Persson, and F\. ProvostLogging policy design for off\-policy evaluation\.arXiv preprint arXiv:2605\.15108\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Dudíket al\.\(2011\)M\. Dudík, J\. Langford, and L\. LiDoubly robust policy evaluation and learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px5.p1.1),[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2608.12489#S1.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.12489#S4.SS0.SSS0.Px3.p1.1)\.
- Guoet al\.\(2022\)W\. Guo, M\. I\. Jordan, and A\. ZhouOff\-policy evaluation with policy\-dependent optimization response\.arXiv preprint arXiv:2202\.12958\.Cited by:[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px10.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Hahnet al\.\(2019\)P\. R\. Hahn, V\. Dorie, and J\. S\. MurrayAtlantic causal inference conference \(acic\) data analysis challenge 2017\.arXiv preprint arXiv:1905\.09515\.Cited by:[§4](https://arxiv.org/html/2608.12489#S4.SS0.SSS0.Px1.p1.1)\.
- Hill \(2011\)J\. L\. HillBayesian nonparametric modeling for causal inference\.Journal of Computational and Graphical Statistics20\(1\),pp\. 217–240\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1)\.
- Keet al\.\(2017\)G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. LiuLightGBM: a highly efficient gradient boosting decision tree\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4](https://arxiv.org/html/2608.12489#S4.SS0.SSS0.Px3.p1.1)\.
- Kiyoharaet al\.\(2024\)H\. Kiyohara, R\. Kishimoto, K\. Kawakami, K\. Kobayashi, K\. Nakata, and Y\. SaitoSCOPE\-rl: a python library for offline reinforcement learning and off\-policy evaluation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Table 15](https://arxiv.org/html/2608.12489#A15.T15),[Table 15](https://arxiv.org/html/2608.12489#A15.T15.5),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1)\.
- Li \(2026\)B\. LiUpliftBench: revealing outcome\-regime and objective mismatch in uplift evaluation\.arXiv preprint arXiv:2608\.00915\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2022\)Y\. Liu, Y\. Flet\-Berliac, and E\. BrunskillOffline policy optimization with eligible actions\.InUncertainty in Artificial Intelligence \(UAI\),Note:arXiv:2207\.00632Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Nishimuraet al\.\(2024\)N\. Nishimura, K\. Kobayashi, and K\. NakataBalancing immediate revenue and future off\-policy evaluation in coupon allocation\.arXiv preprint arXiv:2407\.11039\.Cited by:[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px9.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Saitoet al\.\(2021\)Y\. Saito, S\. Aihara, M\. Matsutani, and Y\. NaritaOpen bandit dataset and pipeline: towards realistic and reproducible off\-policy evaluation\.Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1),[2nd item](https://arxiv.org/html/2608.12489#S4.I1.i2.p1.1)\.
- Saito and Joachims \(2021\)Y\. Saito and T\. JoachimsCounterfactual learning and evaluation for recommender systems: foundations, implementations, and recent advances\.ACM Conference on Recommender Systems \(RecSys\) Tutorial\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px3.p1.1)\.
- Smith and Winkler \(2006\)J\. E\. Smith and R\. L\. WinklerThe optimizer’s curse: skepticism and postdecision surprise in decision analysis\.Management Science52\(3\),pp\. 311–322\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px2.p1.1),[§5\.4](https://arxiv.org/html/2608.12489#S5.SS4.p1.1)\.
- Suet al\.\(2020\)Y\. Su, M\. Dimakopoulou, A\. Krishnamurthy, and M\. DudíkDoubly robust off\-policy evaluation with shrinkage\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix B](https://arxiv.org/html/2608.12489#A2.SS0.SSS0.Px1.p7.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Swaminathan and Joachims \(2015\)A\. Swaminathan and T\. JoachimsThe self\-normalized estimator for counterfactual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.12489#S4.SS0.SSS0.Px3.p1.1)\.
- Tanakaet al\.\(2026\)K\. Tanaka, K\. Kawamura, T\. Muroi, Y\. Narita, Y\. Sasamoto, K\. Tateno, T\. Udagawa, W\. Du, and Y\. SaitoOff\-policy evaluation for ranking policies under deterministic logging policies\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2603\.21485Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2608.12489#S4.I1.i2.p1.1)\.
- Thomas and Brunskill \(2016\)P\. Thomas and E\. BrunskillData\-efficient off\-policy policy evaluation for reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.12489#S1.p1.1)\.
- Wanget al\.\(2017\)Y\. Wang, A\. Agarwal, and M\. DudíkOptimal and adaptive off\-policy evaluation in contextual bandits\.International Conference on Machine Learning \(ICML\)\.Cited by:[Appendix H](https://arxiv.org/html/2608.12489#A8.SS0.SSS0.Px8.p1.1),[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.12489#S4.SS0.SSS0.Px3.p1.1)\.
- Yeomet al\.\(2025\)H\. Yeom, J\. Shin, S\. Min, J\. Yoon, S\. Yu, and D\. KangBreaking determinism: stochastic modeling for reliable off\-policy evaluation in ad auctions\.arXiv preprint arXiv:2512\.03354\.Cited by:[§2](https://arxiv.org/html/2608.12489#S2.SS0.SSS0.Px1.p1.1)\.
- Yoonet al\.\(2018\)J\. Yoon, J\. Jordon, and M\. van der SchaarGANITE: estimation of individualized treatment effects using generative adversarial nets\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§N\.3](https://arxiv.org/html/2608.12489#A14.SS3.SSS0.Px1.p1.1)\.

## Appendix AClaims and supporting evidence

Table 6:Each headline claim and the section, table, or figure that supports it\.
## Appendix BDiagnostic mechanics, estimator addenda, and reproduction targets

#### Score alignment is not action alignment: the correct sharpening limit\.

Letπe\\pi\_\{e\}be top\-kkunder scoresswith standardized cutoffq~\>0\\tilde\{q\}\>0, and let the score\-aligned logger beπb​\(1∣x\)=clip⁡\(σ⁡\(s~​\(x\)/τ\),ε,1−ε\)\\pi\_\{b\}\(1\\mid x\)=\\mathrm\{clip\}\\big\(\\sigma\(\\tilde\{s\}\(x\)/\\tau\);\\,\\varepsilon,\\,1\-\\varepsilon\\big\),ε=0\.02\\varepsilon=0\.02\. Writingq⁡\(X\)=πb​\(aπe​\(X\)∣X\)q\(X\)=\\pi\_\{b\}\(a\_\{\\pi\_\{e\}\}\(X\)\\mid X\)for the logging probability of the*target’s*action, the weight at the logged action satisfies𝔼⁡\[W∣X\]=1\\mathbb\{E\}\[W\\mid X\]=1and𝔼⁡\[W2∣X\]=1/q⁡\(X\)\\mathbb\{E\}\[W^\{2\}\\mid X\]=1/q\(X\), soESS/n→1/𝔼⁡\[1/q⁡\(X\)\]\\mathrm\{ESS\}/n\\to 1/\\mathbb\{E\}\[1/q\(X\)\]\. Asτ→0\\tau\\to 0,q⁡\(X\)→1−εq\(X\)\\to 1\-\\varepsilonon the aligned massc=k\+Fs~​\(0\)c=k\+F\_\{\\tilde\{s\}\}\(0\)but→ε\\to\\varepsilonon the band0<s~<q~0<\\tilde\{s\}<\\tilde\{q\}between the score mean and the budget cutoff, where the logger treats and the target does not\. Hence

ESSn⟶\[c1−ε\+1−cε\]−1≈ε1−c,\\frac\{\\mathrm\{ESS\}\}\{n\}\\;\\longrightarrow\\;\\Big\[\\frac\{c\}\{1\-\\varepsilon\}\+\\frac\{1\-c\}\{\\varepsilon\}\\Big\]^\{\-1\}\\;\\approx\\;\\frac\{\\varepsilon\}\{1\-c\},which is small: withk=0\.1k=0\.1\(soc≈0\.6c\\approx 0\.6\) the limit is≈0\.05\\approx 0\.05\. The floor\-probability draws occur with probabilityO⁡\(ε\)O\(\\varepsilon\)but carry weight1/ε1/\\varepsilon, anO⁡\(1/ε\)O\(1/\\varepsilon\)second\-moment contribution that cannot be neglected\. The caseq~\>0\\tilde\{q\}\>0covers budgets smaller than the mass above the score mean; in general \(continuous scores\) the mismatch band lies between the mean and the cutoff on whichever side the cutoff falls,c=1−\|Fs~​\(q~\)−Fs~​\(0\)\|c=1\-\\lvert F\_\{\\tilde\{s\}\}\(\\tilde\{q\}\)\-F\_\{\\tilde\{s\}\}\(0\)\\rvert— reducing tok\+Fs~​\(0\)k\+F\_\{\\tilde\{s\}\}\(0\)forq~\>0\\tilde\{q\}\>0— so the same limit covers the benchmark’s full budget grid\.*Sharpening a score\-aligned logger therefore eventually collapses overlap*: the flatness of Figure[1](https://arxiv.org/html/2608.12489#S5.F1)is a finite\-range plateau, and the collapse begins just below the tested range\. Extending the temperature grid downward confirms this on the benchmark’s own data, and an*action\-aligned*logger — the same logistic centred at the target’s top\-kkcutoff, for which the mismatch band is empty — behaves oppositely, sharpening toward full support \(limit1−ε1\-\\varepsilon\):

median ESS atk=0\.1k\{=\}0\.1\(not budget\-pooled\)τ=5\\tau\{=\}5220\.50\.50\.250\.250\.10\.10\.050\.05score\-aligned0\.5120\.5190\.3980\.1710\.0760\.061action\-aligned0\.5600\.6320\.8360\.9070\.9460\.960median\|\|IPS rel\. bias\|\|\(pooledkk\)score\-aligned0\.0260\.0260\.0280\.0290\.0490\.065action\-aligned0\.0250\.0280\.0220\.0200\.0190\.017
\(make repro\-sharpening\-limit\.\) Temperature is therefore not a valid overlap parameter in either direction: what governs support is the logger’s probability of the*target’s actions*, and the main sweep’s temperature range happens to sit on the score\-aligned plateau — one more reason the alignment axis, not sharpness, is the right design variable for benchmarks of this estimand\.

What the three diagnostics measure\.Writewi=πe​\(ai∣xi\)/πb​\(ai∣xi\)w\_\{i\}=\\pi\_\{e\}\(a\_\{i\}\\mid x\_\{i\}\)/\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\)for the importance weight\. The*ESS fraction*is Kish’s effective sample size normalized by the logged sample size,\(∑iwi\)2/\(n​∑iwi2\)\\big\(\\sum\_\{i\}w\_\{i\}\\big\)^\{2\}\\big/\\big\(n\\sum\_\{i\}w\_\{i\}^\{2\}\\big\)\.*Support deficiency*is the fraction of evaluation units whose target\-selected action was taken by the logger with probability belowκ=0\.05\\kappa\\\!=\\\!0\.05— the share of the target policy’s decisions that the log barely covers\.*Max weight*ismaxi⁡wi\\max\_\{i\}w\_\{i\}, the largest realized importance weight — bounded by5050here by the propensity floor\. The quantity all three are asked to rank is IPS\|\|relative bias\|\|,\|V^−Vref\|/\|Vref\|\|\\widehat\{V\}\-V\_\{\\mathrm\{ref\}\}\|/\|V\_\{\\mathrm\{ref\}\}\|, taken per cell as the*median over its ten seeds*— a signed\-error magnitude, as opposed to the relative RMSE of Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1), which additionally averages over seed variance\. The correlations above are computed over all540540cells of the alignment sweep,*not*over the main accuracy sweep\.

ESS fraction is not a pure dispersion measure here, and it matters why\.Under a*deterministic*targetwi∈\{0,1/πb​\(ai∣xi\)\}w\_\{i\}\\in\\\{0,1/\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\)\\\}, so units whose logged action differs from the one the target selects contribute zeros that remain in the normalizer\. Writingmmfor the number of*matched*units, the statistic factors exactly as\(m/n\)×\[\(∑matchedw\)2/\(m​∑matchedw2\)\]\(m/n\)\\times\\big\[\(\\sum\_\{\\text\{matched\}\}w\)^\{2\}/\(m\\sum\_\{\\text\{matched\}\}w^\{2\}\)\\big\]— an action\-*coverage*term times a dispersion term on the matched set\. It therefore moves with the budget and the logger’s marginal treat rate, not with weight dispersion alone\. Two consequences we flag rather than paper over: the value11is unattainable by construction, and the numeric cut point used in Section[6](https://arxiv.org/html/2608.12489#S6)is not portable across budgets, loggers or sample sizes even within this benchmark\. What the diagnostic buys is a*ranking*of cells within a comparable design, which is what Table[8](https://arxiv.org/html/2608.12489#A7.T8)conditions on\.

Support deficiency rests on two nested constants—κ\\kappa, and then the screen’s threshold on the resulting fraction — so it deserves a sensitivity note\. Recording it atκ∈\{0\.01,0\.05,0\.10\}\\kappa\\in\\\{0\.01,0\.05,0\.10\\\}, the association is stable between0\.050\.05and0\.100\.10\(ρ=\+0\.42\\rho=\+0\.42and\+0\.43\+0\.43; all diagnostics are seed\-averaged per cell, as everywhere else in this section\), whileκ=0\.01\\kappa\\\!=\\\!0\.01is degenerate here: the0\.020\.02propensity floor \(Section[4](https://arxiv.org/html/2608.12489#S4)\) means no unit can fall below it — the same floor that gives weight clipping no purchase \(Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)\)\. The floor also makes support deficiency a floor\-*dependent*quantity \(mass in\[0\.02,κ\)\[0\.02,\\kappa\)\) where ESS fraction is floor\-robust, which is one reason we expect the latter to be the half of the screen that transfers\.

Does weight clipping rescue IPS?Truncating importance weights is the standard practitioner response to exactly this pathology, so we added clipped IPS \(min⁡\(wi,M\)\\min\(w\_\{i\},M\)forM∈\{5,10,50\}M\\in\\\{5,10,50\\\}\) on the exact\-value datasets \(synthetic,IHDP, and one ACIC nonlinear surface, on the reduced three\-budget grid of the robustness runs — absolute levels therefore differ from Table[2](https://arxiv.org/html/2608.12489#S5.T2), as in Appendix[P](https://arxiv.org/html/2608.12489#A16)\)\. Because our own Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)argument is that medians hide the tail where weighting fails, we score clipping on tail\-sensitive summaries, where it would show up if it worked\. It does not: the mean paired difference against rawIPSis\+0\.0000\+0\.0000atM=50M\{=\}50,−0\.0000\-0\.0000atM=10M\{=\}10, and\+0\.0005\+0\.0005atM=5M\{=\}5\(slightly*worse*— clipping bias appearing before any variance benefit\); the 90th and 99th percentiles of per\-cell error are unchanged atM≥10M\{\\geq\}10and the 99th*worsens*atM=5M\{=\}5\(0\.078→0\.0850\.078\\to 0\.085\); the single worst cell is untouched; and only11%11\\%of cells are affected at all even atM=5M\{=\}5\. The reason is structural\. Our logging propensities are floored at0\.020\.02\(Section[4](https://arxiv.org/html/2608.12489#S4)\), so weights are bounded by5050by construction, and the median per\-cell maximum weight is only5\.05\.0atτ=0\.5\\tau\{=\}0\.5on the main accuracy sweep, against a median ESS fraction of0\.560\.56in the same cells \(the worst single cell reaches0\.0840\.084, but that is a minimum, not the typical case\)\. The failure mode is*not*a handful of enormous weights that truncation would catch; it is the ordinary dispersion of*many*moderately inflated weights, which is bounded below by the maximum weight — atW=5\.0W\\\!=\\\!5\.0no reweighting scheme that only touches the top of the distribution can lift the dispersion term materially above where it already sits\. The natural objection — that this experiment, like the main sweep, uses a self\-aligned logger where weights stay modest — is now closed directly: rerunning clipping under all three logger regimes on the exact\-value datasets \(make repro\-tail\-control\), including the independent logger where maximum weights reach the5050ceiling, the paired median change against rawIPSis0\.00000\.0000at everyMM, withM=10M\{=\}10helping in at most25%25\\%of cells\. Clipping is ineffective even where the tail exists*under this floor*— and that qualifier carries the result\. Sweeping the floor itself \(make repro\-floor\-sensitivity;ε\\varepsilonto0\.00020\.0002, ceilings to5,0005\{,\}000, crossed withτ\\tauto0\.050\.05\), clipping is a no\-op atε=0\.02\\varepsilon\{=\}0\.02in100%100\\%of cells atM=50M\{=\}50— the floor has already clipped — but useful once weights exceed that ceiling \(medianIPS0\.1110\.111against0\.0900\.090atε=0\.005\\varepsilon\{=\}0\.005\), whileDRstays flat \(0\.038→0\.0370\.038\\to 0\.037\)\.*“Clipping does not help” is a property of the bounded\-weight regime we chose, not of allocation OPE\.*Large maximum weights do*co\-move*with realized error — max weight is a valid marker, positive in all five per\-dataset strata of the alignment sweep \(\+0\.29\+0\.29to\+0\.72\+0\.72\) — but the marker is a symptom of the same dispersion that collapses ESS, not a separable cause that clipping can remove\. Shrinkage\-DR\([20](https://arxiv.org/html/2608.12489#bib.bib11)\), which also targets the weight tail, is now evaluated in the same rerun: with itsλ\\lambdatuned on the logged sample by the same estimated\-MSE style as the other hybrids, it edges plainDRby a median0\.00030\.0003–0\.00080\.0008relative RMSE \(6767–74%74\\%of cells, largest under the independent logger\) — a real but small improvement, consistent with the expectation that tail control buys little when the pathology is ESS collapse\. Practitioners should not expect weight control to substitute for overlap*when weights are already bounded*; where they are not, clipping is worth using, and the floor sweep above locates that boundary\.

Why the two tuned hybrids duplicate their untuned parents\.In Table[2](https://arxiv.org/html/2608.12489#S5.T2)Switch\-DRsits within0\.00030\.0003ofDRandmIPSwithin0\.0010\.001ofIPSon every dataset, which invites the suspicion that the tuning is doing nothing\. Recording the*selected*hyper\-parameter in all2,7002\{,\}700configurations \(make repro\-tuner\-selection\) shows that is close to right, and identifies the mechanism as a property of the selection rules rather than a coincidence\.Switch\-DRnever selectsλsw=∞\\lambda\_\{\\mathrm\{sw\}\}=\\infty, but theλsw\\lambda\_\{\\mathrm\{sw\}\}it does select exceeds the cell’s*largest*importance weight in94\.9%94\.9\\%of configurations: no unit is switched, so the estimator is numerically identical toDRthere — close to the96\.6%96\.6\\%of cells in which the two estimates agree bit\-for\-bit\. The propensity floor is again the reason\. With weights bounded by5050and a median per\-cell maximum of2\.12\.1\(atτ=5\\tau\{=\}5\) to5\.05\.0\(atτ=0\.5\\tau\{=\}0\.5\), the grid’s*smallest*candidateλsw=5\\lambda\_\{\\mathrm\{sw\}\}\\\!=\\\!5— selected83%83\\%of the time — already sits above most of the weight distribution\.mIPSlikewise selectsα=0\\alpha=0, which reduces it toIPSexactly, in75\.0%75\.0\\%of configurations\. The residual25\.0%25\.0\\%is not negligible per cell \(median\|Δ​V^\|/\|V\|\|\\Delta\\widehat\{V\}\|/\|V\|of0\.0060\.006, 90th percentile0\.0800\.080\), somIPSis not a relabelling ofIPSconfiguration by configuration; those cells are simply too few to move any dataset’s median\. The general point is that both estimated\-MSE proxies collapse to the untuned estimator under precisely the conditions where tuning would have to earn its keep: neither can buy variance reduction when the weight tail is bounded by construction — the same reason clipping fails below\.

### B\.1Reproduction targets

- •make repro\-full— the2,7002\{,\}700\-configuration accuracy run \(18,90018\{,\}900estimator\-level results=16,200=16\{,\}200evaluated outputs from the six common\-estimand estimators plus2,7002\{,\}700perturbation\-DR rows excluded from the common\-estimand tables; their matched\-reference evaluation ismake repro\-perturbation\-matched; RQ1–RQ2, RQ4\);
- •make repro\-optbias— the optimization\-bias study \(RQ3\);
- •make repro\-acicandmake repro\-acic\-hillstrom— the two known\-effect*temperature\-design*hardening sweeps \(66DGPs×\\times33candidates×\\times44budgets×\\times33temperatures×\\times1010seeds=2,160=2\{,\}160configurations each\)\. Their*alignment\-axis*counterparts,make repro\-logger\-alignment\-acicandmake repro\-logger\-alignment\-acic\-hillstrom, add the three logger regimes but need two learned candidates rather than three \(each serves as the other’s misaligned logger\), so they come to4,3204\{,\}320configurations each rather than6,4806\{,\}480;
- •make repro\-twins— the non\-simulated paired\-reference validation of the RQ1–RQ3 mechanisms \(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\);
- •make reference\-check— numerical agreement ofIPS,SNIPS,DMandDRwith Open Bandit Pipeline on identical inputs;
- •make repro\-misspec— the outcome\-model degradation study;
- •make repro\-selection— the seven\-policy selection slate;
- •make repro\-commonlogandmake repro\-commonlog\-3cand— the common\-logger RQ4 experiment on the seven\- and three\-candidate slates \(Table[5](https://arxiv.org/html/2608.12489#S5.T5), Figure[4](https://arxiv.org/html/2608.12489#A7.F4)\), which is the primary selection evidence;
- •make repro\-full\-cutoff— the whole accuracy sweep and alignment sweep re\-run with the logistic centred at each budget’s top\-kkcutoff \(the cutoff\-centred design\);
- •make repro\-full\-oof— the whole accuracy sweep re\-run with out\-of\-fold outcome nuisances \(the last two columns of Table[2](https://arxiv.org/html/2608.12489#S5.T2)\);
- •make repro\-nuisance\-crossfit— the out\-of\-fold\-nuisance robustness check behind the RQ1 family claim \(Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)\);
- •make repro\-tuner\-selection— the selectedSwitch\-DRλsw\\lambda\_\{\\mathrm\{sw\}\}andmIPSα\\alphaover the main grid \(Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)\);
- •make repro\-tail\-control— clipping and shrinkage\-DR under all three logger regimes;
- •make repro\-sharpening\-limit— score\- vs action\-aligned logging down toτ=0\.05\\tau\{=\}0\.05\(the action\-aligned overlap limit\);
- •make repro\-floor\-sensitivity— the propensity\-floor sweep \(ε\\varepsilondown to0\.00020\.0002\) behind the scope of the tail\-control finding;
- •make repro\-optbias\-decoupled— the policy/nuisance independence ablation behind the RQ3 sign;
- •make repro\-refit\-intervals— refit\-aware intervals \(Appendix[L\.1](https://arxiv.org/html/2608.12489#A12.SS1)\);
- •make repro\-logger\-alignment— the alignment\-axis RQ2 sweep \(Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\);
- •make repro\-logger\-alignment\-acicandmake repro\-logger\-alignment\-acic\-hillstrom— the alignment\-axis held\-out suites that carry the screen AUCs;
- •make repro\-rct\-disjoint— the shared\-vs\-disjoint reference split on all three RCT datasets \(Appendix[P](https://arxiv.org/html/2608.12489#A16)\);
- •make analyze/make analyze\-acic— every figure and table in this paper, regenerated from the parquets\.

## Appendix CRobustness checks and the outcome\-model ladder: design and per\-DGP detail

### C\.1Further robustness checks

Three checks probe whether the RQ1–RQ2 conclusions depend on the benchmark’s construction; none suggests they do\. Full tables are in Appendix[N](https://arxiv.org/html/2608.12489#A14)\.

Known\-effect hardening on real covariates\.The exact\-value evidence so far rests on one synthetic generator and one semi\-synthetic dataset\. We add six known\-effect DGPs built on the*real*IHDPcovariates, and repeat the whole sweep on a second, independent covariate source \(a10,00010\{,\}000\-rowHillstromsubsample\) —2,1602\{,\}160further configurations, zero anomalies\. The RQ1 ordering replicates in full on both:DMmost accurate, the DR family next,IPS/mIPSworst*in every setting*\(2\.1×2\.1\\timesDMon theIHDP\-covariate sweep and3\.2×3\.2\\timeson theHillstrom\-covariate sweep\)\. The RQ2 diagnostics replicate independently too: on the*alignment\-axis*hardening suites — separate reruns of the same six DGPs \(4,3204\{,\}320configurations each, in addition to the two2,1602\{,\}160\-configuration temperature\-design sweeps; these reruns carry the held\-out screen AUCs of0\.830\.83and0\.910\.91\),ρ⁡\(ESS\)=−0\.45\\rho\(\\text\{ESS\}\)=\-0\.45and−0\.58\-0\.58, versus−0\.41\-0\.41on the primary sweep; the temperature\-design suites of Appendix[N](https://arxiv.org/html/2608.12489#A14)give−0\.37\-0\.37and−0\.52\-0\.52\. The separate alignment\-axis suites provide the held\-out validation results and AUCs reported in Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\.

A non\-simulated reference\.Both checks above still score against simulated response surfaces\. TheTwinscohort supplies an evaluation reference read directly off recorded paired outcomes instead \(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\)\. The RQ1 ordering holds with a*larger*model\-based margin \(5\.0×5\.0\\timesagainst2\.0×2\.0\\times\), the alignment mechanism and the screen’s association replicate, and the RQ3 signs replicate; the10%10\\%failure criterion and the absolute error levels do not transfer, which is the split Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)predicts\.

Misaligned logging\.The main sweep logs under a score\-aligned stochastic logger\. Crossing logger and target \(log under theT\-learner, evaluate theS\-learner, and vice versa\) raises every estimator’s error modestly — model\-based\+12\.3%\+12\.3\\%, IPS family\+15\.4%\+15\.4\\%— and preserves the family ordering\. This is a check on the*ordering*, not on the size of the gap: crossing two correlated meta\-learners is a weaker perturbation than the candidate\-independent logger of Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5), under which we did not re\-score accuracy, so the RQ1–RQ2 magnitudes remain self\-aligned\-logger quantities \(Appendix[N\.5](https://arxiv.org/html/2608.12489#A14.SS5)\)\.

#### The outcome\-model ladder\.

Design and per\-DGP detail are in Appendix[N\.4](https://arxiv.org/html/2608.12489#A14.SS4); the finding is stated in Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\.

## Appendix DRQ4: reference\-type split and decoy detail

Splitting by reference type shows the pooled null hides two opposing effects\.Our own rule — never pool exact\-value and HT\-reference results in a single claim — applies here too\. Under the common independent logger the split is reported in the last block of Table[5](https://arxiv.org/html/2608.12489#S5.T5)\. On the*exact\-value*datasets — where “correct” means recovering the genuinely value\-maximizing candidate — the model\-based advantage is large and intact even under a shared, candidate\-independent log \(\+0\.263\+0\.263and\+0\.311\+0\.311\)\. On the*HT\-reference*datasets it reverses:IPSmatches or beatsDM\. We do not read the reversal as evidence that weighting selects better\. The HT reference is itself an inverse\-propensity construction computed on the same randomized split, soIPSand the reference share an estimator family and much of their noise; agreement between them is partly structural rather than evidence of accuracy\. The pooled RQ4 null is therefore the average of a real model\-based advantage where truth is exact and what we suspect is a reference\-alignment artifact where it is not\. The defensible statement is narrower than either half: on a competitive slate scored against a shared independent log, we can separate the families only where exact ground truth is available, and we cannot rule out that the apparent parity on RCT data is an artifact of the reference construction\.

One dataset drives much of the instability\.Jobshas a*negative*DM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}gap in*all six*designs \(per\-dataset gaps are tabulated in Appendix[O](https://arxiv.org/html/2608.12489#A15);−0\.113\-0\.113to−0\.367\-0\.367\)\. That is coherent with the accuracy results:Jobsis the one dataset whereIPSis nominally best in Table[2](https://arxiv.org/html/2608.12489#S5.T2), and Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)attributes that to its small retained sample rather than to anyIPSadvantage\.

## Appendix EThe cutoff\-centred logger

The cutoff\-centred design confirms the analysis\.Our logistic is centred at the score*mean*, so the self\-aligned logger matches the candidate’s score ranking, not its action boundary\. Re\-running the benchmark and this sweep centred at each budget’s top\-kkcutoff \(make repro\-full\-cutoff\) confirms both predictions: sharpening an action\-aligned logger*improves*overlap \(median ESS0\.545→0\.7660\.545\\to 0\.766asτ\\taufalls, against0\.522→0\.5620\.522\\to 0\.562under mean\-centring\), and the gradient sharpens to0\.0%0\.0\\%/1\.7%1\.7\\%/26\.7%26\.7\\%failure atτ=0\.5\\tau\{=\}0\.5\(screen AUC0\.9540\.954\)\. That pooled AUC is*higher*than the primary sweep’s while the pooledρ\\rhoof Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)is lower, and the reconciliation is composition: failures here concentrate almost entirely in the independent regime \(1616of1717failing cells\), so a pooled AUC partly measures regime separation — within the independent regime it is0\.8690\.869— which is why the comparison in Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)leads withρ\\rho\. The drop in pooledρ\\rho\(−0\.41→−0\.32\-0\.41\\to\-0\.32\) is likewise what removing the self\-aligned stratum’s failures implies: fewer positives to rank, not a weaker screen\. The residual self\-aligned failures were an artifact of mean\-centring; the estimator ordering is unchanged and every error falls \(IPS0\.065→0\.0520\.065\\to 0\.052\)\. We keep the mean\-centred sweep primary for a substantive reason: it spans a*wider overlap range*, which is what a study of a risk\-*ranking*diagnostic needs\. Under the cutoff\-centred logger the self\-aligned cells essentially never fail, leaving the screen little to discriminate — a better logging design makes a worse testbed for a fragility diagnostic\.

## Appendix FWhy the screen inverts under a constant propensity model

Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3)reports that under the marginalπ^b\\hat\{\\pi\}\_\{b\}the fragility screen is not merely uninformative but*inverted*\(AUC0\.050\.05\)\. The stratum numbers show why this is systematic rather than noise\. Under that model the self\-alignedτ=0\.5\\tau\{=\}0\.5cells — the ones whose true propensities are most extreme, and therefore where a constant estimate is most wrong — carry simultaneously the*highest*estimated ESS fraction \(0\.6630\.663\) and the*highest*IPSerror \(0\.4670\.467\)\. A constantπ^b\\hat\{\\pi\}\_\{b\}contributes no dispersion, so the estimated ESS reduces to a coverage measure, and coverage is maximized exactly where the sharp, score\-aligned logger agrees most often with the target\. The diagnostic therefore ranks the worst cells as the safest\. This is the concrete sense in which fitting a flexible propensity model is a prerequisite for the screen rather than a refinement of it\.

## Appendix GRQ2 conditioning ladder: table and detail

Table 7:The fullδ\\deltagrid behind the held\-out screen validation: ROC\-AUC ofESS^\\widehat\{\\mathrm\{ESS\}\}for predicting medianIPS\|\|relative bias\|\>δ\|\>\\deltaon the two alignment\-axis hardening suites \(432432cells each\), with the share of cells above eachδ\\delta\. Dashes mark degenerate strata \(no cells, or essentially all cells, aboveδ\\delta\)\. The pre\-specified10%10\\%target is scoreable only on the IHDP\-covariate suite; on the Hillstrom\-covariate suite no cell reaches it, which is why that suite is scored at the2%2\\%target matched to its error scale — chosen for scale, not selected for AUC\.Table 8:RQ2 diagnostic association under progressively stricter conditioning \(IPS\), on the logger\-alignment sweep\. All quantities are computed over*cells*— the seed\-aggregated unit of Section[4](https://arxiv.org/html/2608.12489#S4)— so repeated seeds do not inflate the sample; AUC is the screen’s discrimination for the\|\|relative bias\|\>10%\|\>10\\%target and “unsafe” the share of cells exceeding it\. The ranking survives conditioning on dataset \(negative in5/55/5\) and on budget \(AUC0\.680\.68–0\.950\.95in every stratum\), and survives conditioning on the logger regime where failures actually occur \(independent logger,ρ=−0\.56\\rho=\-0\.56with11\.7%11\.7\\%unsafe\); it does*not*survive conditioning on regime andτ\\taujointly \(median−0\.01\-0\.01, negative in6/96/9\)\. We read the last row as expected rather than damaging: regime×\\timesτ\\tauis the mechanism that generates overlap variation here, so fixing both removes the variation the diagnostic exists to rank\. The defensible claim is between\-situation ranking — which logging situation am I in — not within\-situation discrimination\.How much of this is our own design?The variation the diagnostic ranks is*induced*by the two axes we control — the logger regime and the temperatureτ\\tau— so the association could in principle be recovering nothing more than “which experimental cell am I in”, which is useless to a practitioner holding one logged dataset and no dials\. Table[8](https://arxiv.org/html/2608.12489#A7.T8)works through the conditioning ladder, and the answer is partly yes\.

Conditioning on*dataset*leaves the association intact \(−0\.35\-0\.35to−0\.71\-0\.71, negative in5/55/5\), as does conditioning on*budget*\(ρ=−0\.22\\rho=\-0\.22to−0\.56\-0\.56, AUC0\.680\.68–0\.950\.95across strata\), so the ranking is not an artifact of pooling across datasets of very different size or across budget difficulty\. Conditioning on the*logger regime*it survives where it matters most —ρ=−0\.56\\rho=\-0\.56under an independent logger, the regime that actually produces failures \(11\.7%11\.7\\%of cells unsafe\) — and weakens where there is almost nothing left to rank \(−0\.27\-0\.27self\-aligned and−0\.17\-0\.17misaligned, with2\.8%2\.8\\%and4\.4%4\.4\\%unsafe cells\)\. But conditioning on regime*and*τ\\taujointly, the correlation collapses: median−0\.01\-0\.01, negative in only66of99strata\. We do not read this as the diagnostic failing\. Regime×\\timesτ\\tau*is*the mechanism that generates overlap variation in this design, so conditioning on both removes precisely the variation ESS exists to track, leaving residual within\-stratum noise\. It does mean the honest claim is narrower than a temperature\-only design would suggest: ESS fraction ranks importance\-weighting risk*across*the logging situations a practitioner might face, not*within*a single fixed one\. That is still the practitioner\-relevant statement — nobody knows in advance which regime their historical log came from — but it is a claim about between\-situation ranking, and we make no stronger one\.

Figure 3:RQ1 — median relative RMSE by estimator, per dataset \(lower is better\)\.DMand the DR family lead on four of five datasets;IPS/mIPSare worst, most dramatically on the continuous\-outcomeIHDP\.Figure 4:RQ4 — selection quality under three logging designs, on both slates \(top: three candidates; bottom: seven\)\. Left: rate of selecting the reference\-best policy \(higher better\)\. Right: mean normalized regret \(lower better\)\. Moving left to right — from a per\-candidate log, to one shared log aligned with a single candidate, to a shared candidate\-independent log — the bars move little, which is the point: the family ordering in correct\-selection accuracy is largely insensitive to the logging design, while it weakens sharply between the top and bottom rows as the slate becomes competitive\. These are*pooled*rates; Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)shows the pooled null averages a real model\-based advantage on exact\-value data with a reversal on HT\-reference data, so the figure should be read together with that split\. On the seven\-policy slate normalized regret \(right column\)*reverses*the pooledDM–IPSordering, placingIPSlower; both exact\-value datasets still favourDM\.Figure 5:RQ2 fragility map — IPS\|\|relative bias\|\|against two logged\-data diagnostics\. Lower effective sample size \(left\) and higher support deficiency \(right\) both track larger error, quantified by the Spearman correlations in the text\.Table 9:ForIPS, propensity estimation is the dominant error source\. Each row replaces the exactπb\\pi\_\{b\}with an out\-of\-fold estimate and re\-runs the whole alignment sweep, with weights*and*diagnostics computed from the estimate\.Aggregation:relative RMSE is computed per cell \(dataset×\\timescandidate×\\timeslogger regime×\\timesoverlap×\\timesbudget\) as RMSE over the1010seeds divided by\|\|mean true value\|\|, then*averaged*over the216216exact\-value cells \(IHDP,synthetic\) — a mean over cells, unlike Table[2](https://arxiv.org/html/2608.12489#S5.T2), which takes the median\. Under the median convention the levels are lower \(exact control:DM0\.0260\.026,DR0\.0270\.027,IPS0\.0570\.057,SNIPS0\.0400\.040\) but every comparison below is unchanged\. The failure rate and AUC are over all540540cells\.DMis constant by construction \(it uses no propensity\) and serves as the pipeline check\.DRis essentially immune whileIPSdegrades several\-fold, and the fragility screen degrades from useful to inverted asπ^b\\hat\{\\pi\}\_\{b\}worsens\.Figure 6:RQ3 — honest policy\-level splitting, not nuisance cross\-fitting, addresses optimization bias\. Mean signed bias under in\-sample policy fitting, each estimator against its own estimand’s reference\. On the datasets with material detected bias \(IHDP,synthetic\) plainDRis strongly optimistic; frozen\-policy nuisance cross\-fit \(same fixed\-policy estimand\) is*more*optimistic, not less; honest fold\-policy evaluation \(learning\-procedure estimand\) is near zero\. On the datasets with no material detected bias all three are near zero\.
## Appendix HFormal estimator definitions

Table 10:Benchmark datasets:*five primary*\(Synthetic,IHDP,Jobs,Hillstrom,Lenta\) plus*two known\-effect hardening suites*\(six ACIC\-style DGPs each, over theIHDPandHillstromcovariates; Appendix[N\.2](https://arxiv.org/html/2608.12489#A14.SS2)\)\.nn: sample size used \(large marketing RCTs capped at50,00050\{,\}000by uniform subsampling, which preserves the constant propensity; raw size in parentheses\)\. TheTwinsrow is an external validation \(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\), not a sixth primary dataset: no headline average includes it\.π1\\pi\_\{1\}: treatment fraction\.y¯\\bar\{y\}: outcome mean \(a base rate for binary outcomes; on the outcome scale for the continuousIHDP\)\. Reference: exact potential\-outcome means where known \(ground truth in the strict sense\), else an unbiased\-but\-noisy Horvitz–Thompson reference estimate on the randomized split\. For the hardening suites,π1\\pi\_\{1\}describes the factual confounded\-logistic assignment \(clipped to\[0\.15,0\.85\]\[0\.15,0\.85\]\); their logged feedback is drawn from the known surfaces, so factual overlap is incidental there\.DatasetRegimennπ1\\pi\_\{1\}y¯\\bar\{y\}Reference valueSyntheticSynthetic7,0000\.510\.51exactμ0,μ1\\mu\_\{0\},\\mu\_\{1\}IHDPSemi\-synth\. \(cont\.\)6720\.183\.17exactμ0,μ1\\mu\_\{0\},\\mu\_\{1\}JobsSemi\-synth\. \(RCT\)5780\.410\.73HT reference estimateHillstromMarketing RCT50,000 \(64K\)0\.670\.15HT reference estimateLentaMarketing RCT50,000 \(687K\)0\.750\.11HT reference estimate*Hardening suites \(Appendix[N\.2](https://arxiv.org/html/2608.12489#A14.SS2)\):*ACIC\-styleKnown\-effect \(×6\\times 6\)6720\.522\.0–2\.6exactμ0,μ1\\mu\_\{0\},\\mu\_\{1\}ACIC\-hsKnown\-effect \(×6\\times 6\)10,0000\.502\.1exactμ0,μ1\\mu\_\{0\},\\mu\_\{1\}*External validation, not a primary dataset \(Appendix[N\.3](https://arxiv.org/html/2608.12489#A14.SS3)\):*TwinsPaired real outcomes11,4000\.500\.83recordedμ0,μ1\\mu\_\{0\},\\mu\_\{1\}\(paired\)#### Symbols\.

Four symbol groups would otherwise be overloaded, so we fix them here\.τ\\tauis always the*logging temperature*; the Switch\-DR threshold isλsw\\lambda\_\{\\mathrm\{sw\}\}and the shrinkage\-DR parameterλ\\lambda\.ε=0\.02\\varepsilon=0\.02is always the*propensity floor*; the support\-deficiency threshold is writtenκ\\kappa\(default0\.050\.05;δ\\deltais reserved for the*error*target of Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\) and the perturbation\-DR noise scaleσp\\sigma\_\{p\}\.MMis always a*clipping cap*; the number of perturbation draws isBp=25B\_\{p\}=25\. The cohort\-conditional estimand of Section[3](https://arxiv.org/html/2608.12489#S3)is writtenV​\(πe\)nV\(\\pi\_\{e\}\)\_\{n\}, distinct from the populationV⁡\(πe\)V\(\\pi\_\{e\}\)\.

#### Compute and software\.

All experiments run on a single 10\-core Apple M\-series CPU; the full reproduction chain \(everymaketarget in Appendix[B\.1](https://arxiv.org/html/2608.12489#A2.SS1)\) takes roughly1717hours end to end, the longest single target being the propensity sweep at about3\.33\.3hours\. Python3\.123\.12,LightGBM, scikit\-learn and Hydra at the versions pinned in the releasedpyproject\.toml\. Seeds run42,…,5142,\\dots,51throughout, except the superseded misaligned\-logging check of Appendix[N\.5](https://arxiv.org/html/2608.12489#A14.SS5), which predates the ten\-seed rerun; every result row carries its git commit hash\. All six estimators are our own implementations against a common interface, so that they share one outcome model and one propensity input and differ only in the estimating equation; we do not wrapOBPorSCOPE\-RL, though we verify numerical agreement against the former on identical inputs \(make reference\-check; Section[4](https://arxiv.org/html/2608.12489#S4)\)\.

#### Two terms used throughout\.

A*cell*is a seed\-aggregated dataset×\\timespolicy×\\timesbudget×\\timestemperature tuple; a*configuration*is one seed\-specific execution of a cell\.*Normalized regret*of an estimator on a slate is\(V⁡\(π⋆\)−V⁡\(π^\)\)/\(V⁡\(π⋆\)−minj⁡V⁡\(πj\)\)\\big\(V\(\\pi^\{\\star\}\)\-V\(\\hat\{\\pi\}\)\\big\)/\\big\(V\(\\pi^\{\\star\}\)\-\\min\_\{j\}V\(\\pi\_\{j\}\)\\big\), whereπ^\\hat\{\\pi\}is the candidate the estimator ranks first,π⋆\\pi^\{\\star\}the reference\-best candidate and the denominator the spread of the slate; it is00for a correct pick and11for the worst, and is comparable across datasets whose value scales differ\. In the released parquets the three logger regimes carry the artifact identifiersself\_aligned,misalignedandindependent\.

Logged data are tuples\(xi,ai,yi\)\(x\_\{i\},a\_\{i\},y\_\{i\}\)with logging propensityei=πb​\(ai∣xi\)e\_\{i\}=\\pi\_\{b\}\(a\_\{i\}\\mid x\_\{i\}\); in this benchmarkeie\_\{i\}is*known by construction*\(the logging policies are synthesized; see Section[4](https://arxiv.org/html/2608.12489#S4)\), which favours importance\-weighted estimators relative to production settings where the propensity must be estimated\. The target policy contributesπie=πe​\(ai∣xi\)∈\{0,1\}\\pi\_\{i\}^\{e\}=\\pi\_\{e\}\(a\_\{i\}\\mid x\_\{i\}\)\\in\\\{0,1\\\}for the deterministic top\-kkallocation, giving the importance weightwi=πie/eiw\_\{i\}=\\pi\_\{i\}^\{e\}/e\_\{i\}\. The outcome modelμ^a​\(x\)\\hat\{\\mu\}\_\{a\}\(x\)is one LightGBM regressor per action, fit on the logged sample and shared by all model\-based estimators in a cell\.

#### DM\.

V^DM=1n​∑iμ^aπe​\(xi\)​\(xi\)\\widehat\{V\}\_\{\\mathrm\{DM\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\hat\{\\mu\}\_\{a\_\{\\pi\_\{e\}\}\(x\_\{i\}\)\}\(x\_\{i\}\)\.

#### IPS\.

V^IPS=1n​∑iwi​yi\\widehat\{V\}\_\{\\mathrm\{IPS\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}w\_\{i\}\\,y\_\{i\}\([8](https://arxiv.org/html/2608.12489#bib.bib5)\)\. Unbiased given correct propensities and support; for a deterministic target mostwiw\_\{i\}are00and the rest1/ei1/e\_\{i\}\.

#### SNIPS\.

V^SNIPS=∑iwi​yi/∑iwi\\widehat\{V\}\_\{\\mathrm\{SNIPS\}\}=\\sum\_\{i\}w\_\{i\}y\_\{i\}\\big/\\sum\_\{i\}w\_\{i\}\([21](https://arxiv.org/html/2608.12489#bib.bib6)\)\.

#### DR\.

V^DR=1n​∑i\[μ^aπe​\(xi\)​\(xi\)\+wi​\(yi−μ^ai​\(xi\)\)\]\\widehat\{V\}\_\{\\mathrm\{DR\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\big\[\\hat\{\\mu\}\_\{a\_\{\\pi\_\{e\}\}\(x\_\{i\}\)\}\(x\_\{i\}\)\+w\_\{i\}\\,\(y\_\{i\}\-\\hat\{\\mu\}\_\{a\_\{i\}\}\(x\_\{i\}\)\)\\big\]\([8](https://arxiv.org/html/2608.12489#bib.bib5)\)\.

#### Switch\-DR\.

DR with the correction term dropped wherewi\>λsww\_\{i\}\>\\lambda\_\{\\mathrm\{sw\}\}\(we write the Switch\-DR threshold asλsw\\lambda\_\{\\mathrm\{sw\}\}to keep it distinct from the logging temperatureτ\\tau\) \(falling back to DM there\)\([24](https://arxiv.org/html/2608.12489#bib.bib7)\):V^sw=1n∑i\[μ^aπe​\(xi\)\(xi\)\+𝟙\{wi≤λsw\}wi\(yi−μ^ai\(xi\)\)\]\\widehat\{V\}\_\{\\mathrm\{sw\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\big\[\\hat\{\\mu\}\_\{a\_\{\\pi\_\{e\}\}\(x\_\{i\}\)\}\(x\_\{i\}\)\+\\mathbb\{1\}\\\{w\_\{i\}\\leq\\lambda\_\{\\mathrm\{sw\}\}\\\}\\,w\_\{i\}\\,\(y\_\{i\}\-\\hat\{\\mu\}\_\{a\_\{i\}\}\(x\_\{i\}\)\)\\big\]\.λsw\\lambda\_\{\\mathrm\{sw\}\}is selected*on the logged sample*from the grid\{5,10,20,50,100,∞\}\\\{5,10,20,50,100,\\infty\\\}by minimizing the standard estimated\-MSE bound: \(dropped\-correction magnitude\)2\+\+sample variance of the retained contributions\.

#### Mixture\-propensity IPS \(mIPS\)\.

IPS with the propensity replaced by a mixture with uniform exploration,V^α=1n​∑iπie\(1−α\)​ei\+α/2​yi\\widehat\{V\}\_\{\\alpha\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\frac\{\\pi\_\{i\}^\{e\}\}\{\(1\-\\alpha\)\\,e\_\{i\}\+\\alpha/2\}\\,y\_\{i\}, keeping weights bounded \(the mixed\-logging device of[16](https://arxiv.org/html/2608.12489#bib.bib4)\)\.α\\alphais selected on the logged sample from\{0,0\.05,0\.1,0\.2,0\.5\}\\\{0,0\.05,0\.1,0\.2,0\.5\\\}by the same estimated\-MSE proxy \(squared deviation from the IPS point value as the bias proxy, plus sample variance\)\. Replacing the true propensity with the mixture denominator deliberately introduces bias in exchange for bounded weights, somIPSis a variance\-control heuristic, not an unbiased estimator of the behaviour policy’s IPS; and because the bias proxy is anchored to the raw\-IPS point value, the tuning target is itself noisy under poor overlap \(where IPS is unstable\) — we therefore treatα\\alpha\-selection as a heuristic, not a guarantee\.

#### Perturbation\-DR\.

A smoothing heuristic inspired by[9](https://arxiv.org/html/2608.12489#bib.bib3): with candidate scoresssandBp=25B\_\{p\}\{=\}25drawsξm∼𝒩⁡\(0,σp2​I\)\\xi\_\{m\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{p\}^\{2\}I\),σp=0\.5​sd​\(s\)\\sigma\_\{p\}=0\.5\\,\\mathrm\{sd\}\(s\), re\-solve the budgeted allocationzm=alloc⁡\(s\+ξm,k\)z\_\{m\}=\\mathrm\{alloc\}\(s\+\\xi\_\{m\},k\)and average the DR values:V^pert=1Bp​∑mV^DR​\(zm\)\\widehat\{V\}\_\{\\mathrm\{pert\}\}=\\frac\{1\}\{B\_\{p\}\}\\sum\_\{m\}\\widehat\{V\}\_\{\\mathrm\{DR\}\}\(z\_\{m\}\)\. Its estimand is therefore the expected value of a*perturbation\-smoothed policy distribution*around the deterministic allocation — not the deterministic policy itself — which is why it is excluded from the fixed\-policy RQ1, RQ3, and RQ4 comparisons and discussed separately in Appendix[I](https://arxiv.org/html/2608.12489#A9)\.

#### Cross\-fitted DR \(RQ3 only\)\.

Two modes with distinct estimands \(Section[5\.4](https://arxiv.org/html/2608.12489#S5.SS4)\):*frozen\-policy*\(the policy is fixed; onlyμ^\\hat\{\\mu\}is fit out\-of\-fold,55folds — same estimand as DR\) and*fold\-policy*\(score and allocation re\-derived per fold from the out\-of\-fold model — estimand is the value of the learning algorithm, scored against its own fold\-matched exact reference\)\.

#### Aggregation\.

Let an experimental*configuration*ccbe a \(dataset, candidate policy, budget, overlap temperature\) tuple, evaluated over seedss=1,…,Ss=1,\\dots,S; for estimatoreewrite its estimate asV^e,c,s\\widehat\{V\}\_\{e,c,s\}and the reference value asVc,sV\_\{c,s\}\(the reference does not depend onee\)\. Define

relRMSEe,c=1S​∑s\(V^e,c,s−Vc,s\)2\|1S​∑sVc,s\|,\\mathrm\{relRMSE\}\_\{e,c\}\\;=\\;\\frac\{\\sqrt\{\\tfrac\{1\}\{S\}\\sum\_\{s\}\(\\widehat\{V\}\_\{e,c,s\}\-V\_\{c,s\}\)^\{2\}\}\}\{\\big\|\\tfrac\{1\}\{S\}\\sum\_\{s\}V\_\{c,s\}\\big\|\},i\.e\. RMSE over seeds normalized by the configuration’s absolute mean reference value \(configurations with near\-zero mean reference would be dropped rather than inflated; none occur in the released runs\)\. The table entry for a \(dataset, estimator\) cell is thenmedianc​relRMSEe,c\\mathrm\{median\}\_\{c\}\\,\\mathrm\{relRMSE\}\_\{e,c\}over the configurations in that cell — the median, because IPS\-family errors are heavy\-tailed under poor overlap\. We spell this out because the order \(RMSE over seeds, then normalize, then median over configurations\) is not the only reading of “median relative RMSE\.” The reported bootstrap CIs are*conditional*contribution\-level intervals: they resample the completed per\-unit DR/IPS contributions on the evaluation split and do not refit the nuisance models, so they reflect sampling variation given the fittedμ^\\hat\{\\mu\}, not the uncertainty of fitting it\.

## Appendix IPolicy\-smoothing sensitivity \(perturbation\-DR\)

Perturbation\-DR changes the target from the fixed deterministic allocationV⁡\(πe\)V\(\\pi\_\{e\}\)to the expected value of a distribution of perturbed allocations,𝔼ε​\[V⁡\(πs\+ε\)\]\\mathbb\{E\}\_\{\\varepsilon\}\[V\(\\pi\_\{s\+\\varepsilon\}\)\]\(Appendix[H](https://arxiv.org/html/2608.12489#A8)\)\. It therefore cannot be ranked directly against the fixed\-policy estimators in RQ1, the fixed\-policy optimization\-bias estimators in RQ3, or across an RQ4 candidate slate \(it perturbs each candidate’s own scores, so no two candidates share an estimand\)\. The estimand\-coherent evaluation scores it against the averaged exact values of its own perturbed policies,Vpert,ref=1M​∑mVexact​\(zm\)V\_\{\\mathrm\{pert,ref\}\}=\\frac\{1\}\{M\}\\sum\_\{m\}V\_\{\\mathrm\{exact\}\}\(z\_\{m\}\), where thezmz\_\{m\}replicate the estimator’s own perturbation draws \(same seed and noise stream, so both sides average the sameM=25M\{=\}25allocations\)\. We ran this on the exact\-value datasets over the main grid \(SyntheticandIHDP, both candidate learners,τ∈\{0\.5,2,5\}\\tau\\in\\\{0\.5,2,5\\\}, the five non\-trivial budgets — atk=1k\{=\}1perturbation cannot change the allocation — ten seeds;6060cells\): median\|\|relative error\|\|is1\.6%1\.6\\%\(1\.9%/1\.4%/1\.4%1\.9\\%/1\.4\\%/1\.4\\%atτ=0\.5/2/5\\tau=0\.5/2/5\), and no cell exceeds10%10\\%\. Perturbation\-DR is therefore accurate*on its own estimand*; its exclusion from the fixed\-policy tables reflects an estimand gap, not an estimation failure\. Because it evaluates a different deployment policy \(a smoothed allocation, which practitioners may or may not wish to deploy\), it is still reported separately \(make repro\-perturbation\-matched\)\.

## Appendix JRetained sample size by overlap regime

Rejection sampling retains fewer units under sharper logging, so the overlap knob moves both propensity concentration and logged sample size\. Median retainednnper temperature:

Poor overlap therefore also costs∼25\\sim\\\!25–35%35\\%of the logged sample on the rejection\-sampled datasets\. The two effects can be separated usingIHDP, whose surface\-sampling construction retains*every*unit at every temperature \(nnconstant at336336\): there the temperature sweep changes only the propensity concentration, and the IPS\-family degradation under a misaligned logger persists \(Figure[1](https://arxiv.org/html/2608.12489#S5.F1)\) while the DR family stays flat\. Propensity concentration, not the incidental sample\-size reduction, is the dominant mechanism; a fully matched\-nncontrol on the rejection\-sampled datasets is left to future work\.

## Appendix KCluster\-bootstrap detail for the RQ2 diagnostics

Complementing the within\-dataset and leave\-one\-dataset\-out analysis in Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2), a percentile bootstrap of the Spearmanρ\\rhobetween each diagnostic and IPS\|\|relative bias\|\|, resampling whole clusters \(4,0004\{,\}000replicates\), gives the following 95% intervals\. The statistic is computed over*cells*\(Section[4](https://arxiv.org/html/2608.12489#S4)\), which already aggregate the ten seeds, so seeds are not an available resampling level; the two levels we can resample are the dataset and the \(dataset, candidate policy\) block, of which there are55and1010respectively on the alignment sweep \(two learned candidates per dataset\):

All intervals exclude zero, and the finer \(dataset×\\timespolicy\) clustering gives the narrower intervals, as one would expect from more resampling units\. We stress that the narrowness of the five\-cluster intervals is not evidence of a large effective sample: with only five top\-level clusters, it reflects how similar the within\-dataset correlations are \(Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\)\. We therefore treat the within\-dataset consistency and leave\-one\-dataset\-out stability, not these intervals, as the primary evidence\.

## Appendix LPaired comparisons and interval coverage

### L\.1Interval coverage against exact references

The title’s question is about trust, and trust in an interval is coverage\. Every estimator reports a95%95\\%percentile bootstrap CI over per\-unit contributions \(conditional on fitted nuisances\); on the exact\-value datasets of the alignment sweep we can score those intervals against the truth\. Coverage confirms the two warnings the paper states about its own intervals, and localizes them:

DMundercovers everywhere \(0\.890\.89pooled,0\.870\.87onsynthetic\): its conditional intervals ignore outcome\-model refit variance, exactly the deficiency Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)flags, and the reason we call the model\-based intervals optimistic\.IPScovers near\-nominally where weighting is safe but degrades to0\.890\.89under the independent logger atτ=0\.5\\tau\{=\}0\.5— coverage collapses in the same cells the fragility screen flags, so the screen also marks where the*intervals*stop being trustworthy\.SNIPSovercovers because its intervals are wide \(median width0\.20​\|V\|0\.20\|V\|against0\.12​\|V\|0\.12\|V\|forDM\)\. None of these are population\-coverage statements; they are conditional\-bootstrap coverage on this benchmark’s exact\-value cells \(12,96012\{,\}960estimator\-configuration intervals;2,1602\{,\}160per estimator\)\.

Mean versus median paired differences \(RQ1\)\.TheIPS−DR\\textsf\{IPS\}\-\\textsf\{DR\}gap of\+0\.015\+0\.015\(\+0\.013\+0\.013on thek<1k<1grid\) in Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)is a*mean*paired difference over cells; the corresponding*median*is\+0\.007\+0\.007on both grids\. The mean is larger precisely*because*it is a mean: the gap between the two summaries is the right tail ofIPSrelative RMSE under a misaligned logger — a minority of cells in which weighting fails badly, which a mean registers and a median does not\. Both agree in sign and ordering; we report the mean because that tail is the practically important part of the distribution, and flag that it is tail\-driven rather than typical\.

Within\-family comparisons \(RQ1\)\.Switch\-DRis statistically indistinguishable fromDR\(−0\.0002\-0\.0002, CI\[−0\.0007,\+0\.0001\]\[\-0\.0007,\+0\.0001\]\)\. TheDM\-over\-DRmargin is at most*tiny*:DR−DM=\+0\.0007\\textsf\{DR\}\-\\textsf\{DM\}=\+0\.0007\(CI\[−0\.0012,\+0\.0021\]\[\-0\.0012,\+0\.0021\], which includes zero; sign convention throughout is first\-minus\-second, so positive here meansDMis better;DMbetter in only61%61\\%of cells\) — and this interval is*conditional*\(it resamples completed contributions without refitting the outcome model; Appendix[H](https://arxiv.org/html/2608.12489#A8)\), so it understates uncertainty for the model\-based estimators\. Under a full nuisance\-refit resampling \(on the exact\-value datasets,55seeds×\\times8080cells withμ^\\hat\{\\mu\}refit inside every bootstrap resample \(make repro\-refit\-intervals\):DM−\-DR=−0\.0005=\-0\.0005, 95% CI\[−0\.0038,\+0\.0028\]\[\-0\.0038,\+0\.0028\]\), theDM–DRdifference is not distinguishable from zero\. We therefore do*not*claim a separation ofDMfromDR: the robust, practically meaningful claim is the model\-based*family*advantage over importance weighting\.

## Appendix MLogging construction: properties and retained sample size

The rejection sampler of Section[4](https://arxiv.org/html/2608.12489#S4)has two properties that bear on how the overlap sweep should be read\.

Centring\.Standardizingssbefore the logistic matters for comparability: raw score scales differ across learners, so without it a commonτ\\tauwould mean different degrees of logging sharpness for different candidates\. But the logistic is centred at the score*mean*, not at the budget\-dependent top\-kkcutoffq1−kq\_\{1\-k\}, so the logger’s marginal treatment rate is≈0\.5\\approx\\\!0\.5regardless ofkk\. A givenτ\\tautherefore does not correspond to the same overlap for a top\-10%10\\%and a top\-70%70\\%target:τ\\tauis comparable across learners but not across budgets\. A budget\-centred loggerσ⁡\(\(s−q1−k\)/τ\)\\sigma\(\(s\-q\_\{1\-k\}\)/\\tau\)removes this, and we ran it \(Appendix[E](https://arxiv.org/html/2608.12489#A5)\)\.

Retained sample size co\-varies with overlap\.Rejection sampling retains fewer units at lowτ\\tau, so the overlap sweep moves retainednnas well as propensity concentration \(Appendix[J](https://arxiv.org/html/2608.12489#A10)tabulates both\)\.IHDP, whose logged sample is drawn from known surfaces at constantnn, is the control: the IPS\-error inflation appears there too, so it is not purely a sample\-size effect\. We note that the differential pattern —IPSinflating while the DR family stays flat — is*not*itself evidence against the confound, sinceIPSerror is variance\-dominated \(∝𝔼⁡\[w2\]/n\\propto\\mathbb\{E\}\[w^\{2\}\]/n\) and would inflate under a sharednn\-loss while a bias\-dominatedDMwould not\. The constant\-nnIHDPcontrol is what carries the argument\. A matched\-nncontrol on the rejection\-sampled datasets is left to future work\.

## Appendix NRobustness checks: full results

Full tables and mechanism discussion for the outcome\-model\-quality ladder of Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)and the two robustness checks of Section[C\.1](https://arxiv.org/html/2608.12489#A3.SS1)\.

### N\.1Per\-dataset accuracy under out\-of\-fold nuisances

Table[2](https://arxiv.org/html/2608.12489#S5.T2)reports the out\-of\-fold\-nuisance rerun only as two cross\-dataset means\. Table[11](https://arxiv.org/html/2608.12489#A14.T11)gives the per\-dataset detail behind them, so that every per\-dataset claim about out\-of\-fold nuisances — including item 2 of Section[6](https://arxiv.org/html/2608.12489#S6)— can be checked against the paper itself\.

Table 11:Per\-dataset accuracy under*out\-of\-fold*nuisances \(make repro\-full\-oof\), the per\-dataset detail behind the two out\-of\-fold mean columns of Table[2](https://arxiv.org/html/2608.12489#S5.T2)\. Median relative RMSE, same convention as Table[2](https://arxiv.org/html/2608.12489#S5.T2); best per column in bold\. Four decimals because several columns separate only in the fourth\.IPS,SNIPSandmIPSuse no outcome model and are numerically identical to their in\-sample counterparts\.DMis best or tied\-best on three of the five datasets here \(Hillstromwithin0\.00010\.0001ofmIPS,IHDP,synthetic\) against four of five in\-sample, and is the*worst*of the six onJobsandLenta— both HT\-reference datasets, where Section[5\.1](https://arxiv.org/html/2608.12489#S5.SS1)argues the ordering is not resolvable\.
### N\.2Known\-effect hardening on real covariates

A skeptical reading of the evidence so far is that the exact\-value evidence rests on one synthetic generator and one semi\-synthetic dataset\. The hardening sweep addresses this with six additional known\-effect DGPs on the*real*IHDPcovariates \(Section[4](https://arxiv.org/html/2608.12489#S4)\), where the covariate joint distribution is real and the truth is exact by construction\.

Table 12:Known\-effect hardening: median relative RMSE on six ACIC\-2017\-style DGP settings over the realIHDPcovariates \(S1/S2 linear, S3/S4 nonlinear, S5/S6 step\-subgroup surfaces; odd==low noise, even==high\)\. Best per column in bold\.Table[12](https://arxiv.org/html/2608.12489#A14.T12)shows the RQ1 ordering replicates in full:DMis the most accurate \(mean relative RMSE0\.0360\.036, and lowest in*every one*of the six settings — its closest call is S1,0\.0190\.019againstSwitch\-DR’s0\.0230\.023\), the DR family follows \(0\.0420\.042\), andIPS/mIPSare worst*in every setting*\(0\.0750\.075and0\.0740\.074mean,2\.1×2\.1\\timesDM\)\. The margin is largest on the low\-noise settings, where the outcome model is easy to fit; high noise \(S2, S4\) compresses all estimators toward the same error, exactly as the outcome\-model\-quality caveat of Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)predicts\. The fragility map of Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)also replicates independently on this sweep: Spearmanρ\\rhobetween IPS\|\|relative bias\|\|and ESS fraction / support deficiency / max weight is−0\.37\-0\.37/\+0\.29\+0\.29/\+0\.23\+0\.23on this temperature\-design sweep — distinct from the alignment\-axis rerun of the same DGPs quoted in Section[C\.1](https://arxiv.org/html/2608.12489#A3.SS1), which gives−0\.45\-0\.45— against−0\.41\-0\.41/\+0\.42\+0\.42/\+0\.35\+0\.35on the primary alignment sweep; same signs, comparable magnitudes \(the fragility\-screen thresholds are validated on the alignment\-axis rerun of these DGPs, not on this sweep; cf\. Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)\)\.

Because the IHDP covariate matrix is small \(n=672n\{=\}672\), we additionally repeat the six\-DGP sweep on a*second, independent*real covariate set: a fixed10,00010\{,\}000\-row subsample of theHillstrommarketing covariates \(another2,1602\{,\}160configurations, zero anomalies\)\. All conclusions replicate:DMis most accurate \(mean relative RMSE0\.00550\.0055, and lowest in every one of the six settings\), the DR family follows \(0\.00620\.0062\),IPS/mIPSare worst in every setting \(0\.01760\.0176and0\.01770\.0177,3\.2×3\.2\\timesDM\), and the diagnostics correlate with error comparably \(ρ=−0\.52\\rho=\-0\.52/\+0\.34\+0\.34/\+0\.37\+0\.37on this sweep\)\. Absolute errors are roughly an order of magnitude smaller than on then=672n\{=\}672IHDP covariates at comparable overlap — a reminder that logged sample size and overlap both drive error, and the benchmark’s relative comparisons are the stable object, not the absolute error levels\. Exact ground truth built on real covariates therefore tells the same story as both the fully synthetic and the RCT\-reference evidence\.

### N\.3External validation on a non\-simulated paired reference

The hardening sweeps answer one version of the skeptical reading\. A sharper version survives them: every exact\-value surface in this paper is a*simulated*response surface — our own synthetic generator and ACIC\-style constructions, andIHDP’s canonical simulated surfaces, which we adopt rather than build — so an estimator is ultimately scored against outcomes somebody modelled\. This appendix removes that reliance using a dataset whose evaluation reference is built directly from recorded paired outcomes rather than from any modelled surface\.

#### Design\.

TheTwinscohort\([1](https://arxiv.org/html/2608.12489#bib.bib25)\), in the form distributed with[26](https://arxiv.org/html/2608.12489#bib.bib26), contains11,40011\{,\}400same\-sex twin pairs born under22kg with3030covariates\. Taking the*pair*as the unit, the treatment is being the heavier twin and the outcome is survival through the first year\. Both siblings’ outcomes are recorded, so the pair supplies two*recorded*outcome coordinates andμ0,μ1\\mu\_\{0\},\\mu\_\{1\}are read off the data\. Be precise about what this buys\. These are not two potential outcomes observed for one infant: the twins are different individuals, so this is a non\-simulated*paired*reference, and it proxies a causal effect of birth weight only under co\-twin exchangeability given the shared covariates — an assumption the design makes plausible but does not prove\. The narrower property is the one we need and it does hold: the reference value is fixed by the data and not by any surface we chose, so scoring against it is not circular\. Marginal survival is0\.8230\.823for the lighter twin against0\.8390\.839for the heavier — a paired heavier\-minus\-lighter survival difference of\+1\.61\+1\.61pp, reproducing the published figures for this cohort\. Because the two paired reference outcomes are recorded binary values and the factual outcome is set to the corresponding recorded one, there is no response\-surface sampling noise here \(the calibrated noise of Appendix[H](https://arxiv.org/html/2608.12489#A8)is exactly zero\): logged rewards are actual recorded outcomes and the retained comparison outcome is the actual co\-twin outcome\. The protocol is otherwise unchanged \(make repro\-twins\)\.

#### The mechanisms replicate; two of them more strongly\.

The RQ1 ordering holds exactly —DM0\.00310\.0031, the DR family0\.00350\.0035,SNIPS0\.00500\.0050,IPS/mIPS0\.01570\.0157— and the model\-based margin is*larger*than on simulated exact\-value data:5\.0×5\.0\\timesagainst the2\.0×2\.0\\timesof Table[2](https://arxiv.org/html/2608.12489#S5.T2)\. The RQ2 mechanism replicates in the form Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)states it: sharpening a self\-aligned logger leaves overlap flat \(median ESS fraction0\.6040\.604,0\.5560\.556,0\.5270\.527atτ=0\.5,2,5\\tau=0\.5,2,5\), while the misaligned and candidate\-independent loggers collapse under the same sharpening \(0\.3880\.388and0\.1730\.173atτ=0\.5\\tau\{=\}0\.5\)\. The screen’s association is comparable to the primary sweep: Spearmanρ\\rhobetween IPS\|\|relative bias\|\|and ESS fraction / support deficiency / max weight is−0\.51\-0\.51/\+0\.51\+0\.51/\+0\.36\+0\.36, against−0\.41\-0\.41/\+0\.42\+0\.42/\+0\.35\+0\.35, and its per\-regime ordering is preserved, weakest under the misaligned logger \(−0\.16\-0\.16\) and strongest under the independent one \(−0\.76\-0\.76\)\. RQ3 replicates as well: plain DR is optimistic by6\.0%6\.0\\%of\|Vref\|\|V\_\{\\mathrm\{ref\}\}\|— materially so by the criterion of Table[4](https://arxiv.org/html/2608.12489#S5.T4), and close toIHDP’s\+6\.9%\+6\.9\\%— nuisance\-only cross\-fitting makes it*worse*\(−25\.4%\-25\.4\\%\), inside the−16%\-16\\%to−36%\-36\\%band of Section[5\.4](https://arxiv.org/html/2608.12489#S5.SS4), and honest policy\-level splitting reduces fold\-matched bias by94\.7%94\.7\\%on the learning\-procedure estimand\. We report that last figure alongside the5858–92%92\\%range rather than inside it: those eight regimes come from two simulated generator families, andTwinsis evidence of a different kind\.

#### What does not transfer is the calibration\.

Two of the paper’s numbers are simply inoperative here, and both are ones Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)already flags as benchmark\-specific\. The10%10\\%relative\-bias failure criterion is never reached — the largest IPS\|\|relative bias\|\|anywhere in the run is0\.0950\.095— so all three logger regimes record a0\.0%0\.0\\%failure rate, and the headline triple of Section[5\.2](https://arxiv.org/html/2608.12489#S5.SS2)has no analogue\. Relative RMSE compresses by roughly an order of magnitude, because it normalizes by\|V\|≈0\.83\|V\|\\approx 0\.83while the spread between the best and worst candidate policy is about22pp; this is the gross\-value metric’s documented tendency to reward baseline predictability, in its most extreme form\. Read on the policy\-relevant scale the compression reverses the impression:IPS’s error is62%62\\%of the entire achievable spread it would need to resolve, against13%13\\%forDMand the DR family\. \(We did not re\-run the RQ4 selection protocol here, so this is a statement about error scale, not a measured selection result\.\) The lesson we draw is the one Section[4\.1](https://arxiv.org/html/2608.12489#S4.SS1)states in advance: the*mechanisms*transfer to a non\-simulated reference we did not construct, and the*cut points*do not\.

### N\.4Outcome\-model quality: the other axis of the decision map

Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)states the finding; this appendix gives the design and the per\-DGP detail\. We re\-run the protocol on the exact\-truth datasets with structured surfaces \(the nonlinear synthetic generator, ACIC S3 nonlinear, and ACIC S5 step\) while degrading the sharedμ^\\hat\{\\mu\}along a quality ladder: LightGBM \(well\-specified\), depth\-2 stumps \(weak\), and ridge regression \(misspecified: linear on non\-linear/step surfaces\); the candidate policies are unchanged, so only the*estimators’*model quality moves\. Table[13](https://arxiv.org/html/2608.12489#A14.T13)reports median relative RMSE*per DGP*— pooling across DGPs would average away the effect, since the ACIC surfaces are far more learnable than the synthetic one\.

Table 13:Outcome\-model misspecification,*per DGP*: median relative RMSE by DGP and outcome\-model quality \(pooled over overlap and budgets\)\. “best” ranges over the four shown estimators\. The inversion is DGP\-specific: on the nonlinearsyntheticsurface a degradedμ^\\hat\{\\mu\}makesDMthe worst of the four; on the smoother ACIC surfacesDMstays competitive\. On the ACIC step surface the misspecified linear model slightly*beats*LightGBM, because the arm means there are nearly linear, so the flexible learner buys variance without reducing bias\.Per\-DGP detail: on the nonlinearsyntheticsurfaceDM’s median relative RMSE goes0\.036→0\.0990\.036\\rightarrow 0\.099under stumps, overtaking evenIPS\(0\.0580\.058\), and sits at0\.0610\.061under ridge;DRdegrades far more gracefully \(0\.037→0\.0490\.037\\rightarrow 0\.049/0\.0430\.043\), its correction term absorbing much of the model error — though not quite best at every rung there:DRis the only model\-based estimator that never becomes worst\. The Table[13](https://arxiv.org/html/2608.12489#A14.T13)entries are per\-DGP medians over the sweep\. The headline crossing was subsequently seed\-resampled \(Section[5\.3](https://arxiv.org/html/2608.12489#S5.SS3.SSS0.Px1)\): at the stump rung onsynthetictheDM−\-IPSgap is\[\+0\.022,\+0\.063\]\[\+0\.022,\\,\+0\.063\],DMworse in100%100\\%of2,0002\{,\}000seed resamples, so the0\.0990\.099\-vs\-0\.0580\.058ordering is a resampling\-tested separation there\. The remaining rungs and the ACIC families carry point estimates only, and for those we still read the qualitative crossing rather than the precise gaps\. On the smoother ACIC nonlinear and step surfaces, the degradation is mild —DMremains best or within a whisker ofDReven under the misspecified linear model — because a linear/stump fit still approximates those surfaces adequately\.

A natural question is whether model\-based fragility admits a validated screen the way importance\-weighting fragility does \(RQ2\)\. We attempted the obvious candidate: an out\-of\-fold held\-out RMSE ofμ^\\hat\{\\mu\}on the logged sample, per arm and pooled, normalized by the logged\-reward standard deviation — computable without ground truth, recorded for every cell of this sweep\. The honest answer is that it does*not*transfer: pooled across the misspecification grid, its Spearman correlation with realizedDMerror is≈−0\.10\\approx\\\!\-0\.10\(flagging AUC0\.480\.48— uninformative\), even though it carries real signal*within*a DGP \(ρ\\rhoup to\+0\.42\+0\.42on the nonlinear synthetic ladder\)\. The mechanism is instructive: held\-out predictive error conflates irreducible outcome noise with structural misfit, and a high\-noise DGP scores “poor model” whileDM’s estimate of the*mean*remains accurate\. Weighting fragility admits a useful observable risk ranking because its failure mechanism is encoded in the importance weights themselves; in our study, model\-based fragility does*not*admit an analogous transferable screen from ordinary factual prediction error\. Constructing a validated, transferable model\-adequacy screen for OPE — theDM/DRanalogue of ESS — is therefore an open problem, and the qualitative checks below are prudent checks supported by our analysis, not a substitute for a known quantitative rule\. The lesson is not thatDMgenerically collapses, but that model\-based estimators inherit whatever error the outcome model makes, and that error is DGP\-dependent\. When overlap is good but the surface may defeat the model,DR’s doubly\-robust correction makes it the safer default than pureDM; when the model is trustworthy,DM/DRlead; when both overlap and model are poor, no estimator here is dependable\. Crucially, outcome\-model quality for*policy evaluation*is not fully certified by a single global held\-out residual \(a model can predict factual outcomes well yet rank treatment effects poorly, or lack support in one arm over part of the covariate space\); a practitioner should check held\-out error*per treatment arm*, calibration across the score distribution, and sensitivity across outcome\-model classes\.

### N\.5Misaligned logging

The main sweep uses*self\-aligned*logging \(Section[4](https://arxiv.org/html/2608.12489#S4)\): the logger is constructed from the same candidate score used by the evaluated policy — not from its top\-kkaction boundary — which cleanly isolates the overlap axis but does not probe the case where logging and target disagree\. Since OPE is often requested precisely when a new candidate differs from the deployed logger, we cross the two: for theT\-andS\-learnercandidates we log under one and evaluate the other \(*misaligned*\), and compare to the aligned case \(the logger built from the evaluated candidate’s own score\), holding everything else fixed\. Misaligned logging modestly raises every estimator’s error but preserves the ordering\. Pooled over the exact\-value and RCT datasets \(5,0405\{,\}040estimator rows=4=4datasets×5\\times 5seeds×3\\times 3budgets×3\\times 3overlaps×4\\times 4\(logging, target\) pairs×7\\times 7estimators, the seventh being perturbation\-DR, which is computed but excluded from the ordering below\) onsynthetic,IHDP,HillstromandLenta\(the four datasets on which both aT\-and anS\-learnerlogger can be built\), median cell\-level relative RMSE rises from0\.02140\.0214to0\.02400\.0240for the model\-based estimators \(DM,DR,Switch\-DR\) and from0\.03610\.0361to0\.04170\.0417for the IPS family \(IPS,mIPS\) —\+12\.3%\+12\.3\\%and\+15\.4%\+15\.4\\%respectively\. The ordering is preserved at the family level \(DR0\.02390\.0239,DM0\.02400\.0240,Switch\-DR0\.02430\.0243,SNIPS0\.03070\.0307,IPS0\.04080\.0408,mIPS0\.04240\.0424;DRandDMswap by0\.00010\.0001, well inside noise, consistent with our refusal to order them\), and the model\-based advantage is if anything slightly widened, though the difference in degradation rates is small enough that we do not claim weighting estimators are materially more misalignment\-sensitive\. The RQ1–RQ2 conclusions therefore survive*this*perturbation\. We are deliberate about its strength: logging under one meta\-learner and evaluating another perturbs the logger between two correlated scoring rules, which is weaker than the candidate\-*independent*logger that produces the RQ4 attenuation of Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)\. We did not re\-score accuracy under that logger, so the RQ1–RQ2*magnitudes*should be read as self\-aligned\-logger quantities; what this check supports is that the ordering is not an artifact of alignment, not that the gap size is logger\-invariant\.

## Appendix ORQ4: reference\-type split and the superseded per\-candidate design

A composition note first, since slate composition is one of this section’s own lessons\. The competitive slate holds seven candidates on the three RCT datasets but six onSyntheticandIHDP: the class\-transformation learner requires a binary outcome and both exact\-value datasets are continuous, so its failure is deterministic, and within every cell all estimators rank the identical surviving slate\. Because the six\-candidate cells are*exactly*the exact\-value datasets, slate size is perfectly confounded with reference type here — restricting to all\-seven cells just reproduces the HT\-reference subset, whose reversal of the pooled gap is the reference\-type split reported in Appendix[P](https://arxiv.org/html/2608.12489#A16), not a slate\-size effect\.

These are the full selection tables for the*per\-candidate*\(self\-aligned\) logging design, in which each candidate is scored on a log built from its own score\. As Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)shows, that design compares policy–logger pairs rather than policies; the common\-logger results of Table[5](https://arxiv.org/html/2608.12489#S5.T5)are the primary evidence\. We retain these for completeness and because the contrast between the two designs is itself the finding\. Note in particular that the “IPSselects at or below the uniform baseline” reading suggested by these numbers does*not*survive the shared\-log design\.

Correct\-selection rate under the common*independent*logger, split by reference type \(the numbers behind the reference\-split discussion in Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)\):

Table 14:RQ4 decoy over\-selection: how often each estimator selects the random\-score candidate, against the rate at which that candidate is genuinely reference\-best\. A random score induces smooth, well\-overlapped logging, so an estimator evaluates it unusually stably and may prefer it for reasons unrelated to its value — most strongly for weighting, butDMshows the same preference at roughly half the strength under the two aligned designs, matching or exceedingIPSunder the common\-independent one \(16\.3%16\.3\\%vs16\.8%16\.8\\%;8\.7%8\.7\\%vs8\.2%8\.2\\%\)\. The effect therefore varies with the logging design rather than being a fixed property of the estimators\.### O\.1Per\-datasetDM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}selection gaps

Paired per\-cellDM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}correct\-selection gap, per dataset, under all six designs \(regenerated fromfull\_run,selection\_run,commonlog\_3candandcommonlog\_run\)\. These are the numbers behind Table[5](https://arxiv.org/html/2608.12489#S5.T5)’s sign counts and behind the observation thatJobsis negative in*all six*designs\.

Cells are balanced within each column \(150150per dataset on the three\-candidate slate,9090on the seven\-policy slate\), so each column’s mean over the five datasets is exactly the pooled gap reported in Table[5](https://arxiv.org/html/2608.12489#S5.T5)\. As throughout RQ4, the full\-budget pointk=1k\{=\}1is excluded\.

The same paired per\-cell contrast for*normalized regret*\(DM−IPS\\textsf\{DM\}\-\\textsf\{IPS\}; negative favoursDM\), which is what Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)reads as the regret gap:

The two exact\-value datasets favourDMin every design; of the three HT\-reference datasets,JobsandLentafavourIPSin every design andHillstromswitches sign with the slate\. That split, not the logging design, is what drives the sign reversal of the pooled regret gap between the two slates\.

Table 15:RQ4 policy selection: rate of selecting the reference\-best policy \(the truly best on exact\-value datasets, the HT\-reference\-best on RCT datasets; higher better\), mean normalized regret \(lower better\), and SharpeRatio@k≥2k\{\\geq\}2\(higher better; the ratio of a policy’s estimated value to the standard deviation of that estimate across seeds, restricted to slates of at least two candidates, following[13](https://arxiv.org/html/2608.12489#bib.bib16)\)\.This table uses the per\-candidate \(self\-aligned\) logging design, in which each candidate is scored on a log built from its own score, so it compares policy–logger pairs rather than policies\. Section[5\.5](https://arxiv.org/html/2608.12489#S5.SS5)shows the model\-based\-over\-IPS margin is nonetheless comparable to what a shared log gives \(1\.3×1\.3\\timesunder the common\-aligned design,0\.7×0\.7\\timesunder the common\-independent one\); see Table[5](https://arxiv.org/html/2608.12489#S5.T5)for the common\-logger results, which we regard as the primary selection evidence\.Table 16:RQ4 selection quality split by reference type: rate of selecting the best candidate\. On exact\-value datasets \(synthetic,IHDP\) this is the*truly*best policy; on RCT\-reference datasets \(Jobs,Hillstrom,Lenta\) it is the policy ranked best by the noisy HT reference\. Three\-candidate slate; uniform\-random baseline=1/3≈0\.33=1/3\\approx 0\.33\.Per\-candidate \(self\-aligned\) logging design— see Table[5](https://arxiv.org/html/2608.12489#S5.T5)\.

## Appendix PShared vs disjoint RCT reference split

On the RCT datasets the reported error is relative to an HT*reference estimate*; in the main runs the logged data and that reference are built from the same randomized evaluation split\. To check that this shared\-split dependence does not drive the estimator ranking, we re\-ran all three RCT datasets —Hillstrom,Lenta, andJobs— under a disjoint40/30/3040/30/30split into policy\-training, logged\-feedback, and reference\-estimation units, computing each estimate on the logged split and scoring it against an HT reference on the*independent*reference split\. Both columns below are medians of the relative RMSE over1010seeds×\\times33budgets×\\times33overlaps×\\times33policies \(make repro\-rct\-disjoint;results/refdep\_run/\)\. We print four decimals because the differences*within*a column are of order0\.0010\.001:

Both columns come from this reduced design, not from the main run\.The “shared ref\.” column is the matched control*within*the40/30/3040/30/30experiment — same30%30\\%logged split and same three\-budget subset, with the reference computed on the shared logged units — so the two columns are comparable to each other but*not*to Table[2](https://arxiv.org/html/2608.12489#S5.T2), whoseHillstromcolumn uses the main50/5050/50split and all six budgets\. That is why the absolute levels here \(e\.g\.HillstromDM0\.03120\.0312\) sit above the main\-run values \(DM0\.0220\.022\): the logged split is smaller and the budget grid differs\. Only the within\-appendix comparison is meaningful\.

The reference, not the estimator, sets the level\.Moving to the disjoint reference raises relative RMSE for every estimator on every dataset — by48%48\\%\(Lenta\),65%65\\%\(Jobs\) and91%91\\%\(Hillstrom\) forDM, and by more forIPS— and that level shift is roughly22–6×6\\timesthe spread across estimators*within*either column\. The disjoint reference is a noisier HT estimate computed on fewer units, and most of the increase is reference variance rather than estimator error\.

Which estimator looks best depends on which reference is used\.DMis never best under the shared reference — a weighting or DR estimator wins on each dataset \(IPS0\.03050\.0305onHillstrom,DR0\.04050\.0405onLenta,SNIPS0\.07520\.0752onJobs\) — and is best on*all three*under the disjoint one\.IPSmoves the opposite way: first onHillstromand second onJobs\(within0\.00070\.0007ofSNIPS\) under the shared reference, third, third and last under the disjoint one\. Note thatIPSis worst of the four only onJobs\(0\.16280\.1628\); onHillstromthe worst isDR\(0\.06350\.0635\) and onLentaSNIPS\(0\.06810\.0681\), so this is a shift in the whole ordering rather than anIPS\-specific penalty\.

How much this experiment can carry\.The direction matches the mechanism we proposed: the HT reference is itself an inverse\-propensity construction, so on shared units it shares sampling noise withIPSand the two agree partly for reasons unrelated to accuracy; removing that sharing costs weighting estimators their apparent edge\. But the differences within a column are0\.0010\.001–0\.0090\.009onHillstromandLenta, this is three datasets, and the reference\-variance term dominates the level, so we report the result as*consistent with*that mechanism rather than as an isolation of it\. \(An earlier five\-seed version of this table circulated with different values and a sharper reading; the ten\-seed run reported here supersedes it\.\) What the experiment does establish is the negative conclusion we rely on: the choice of reference moves every estimator’s measured error by more than the estimators differ from each other, so RCT\-reference comparisons cannot resolve the estimator ordering\. That is why we decline to read theIPScolumn of Table[2](https://arxiv.org/html/2608.12489#S5.T2)as a ranking, and it is also why these RMSE values should be read as OPE error plus reference\-estimation variance, not as absolute estimator error\. On the model\-based side the check is reassuring in the direction that matters for the paper’s headline: the shared\-split construction did not manufacture the model\-based advantage — the cleaner disjoint design produces a stronger one\.