Design-Based Supervised Learning with Noisy Human Labels

# Design-Based Supervised Learning with Noisy Human Labels
Source: [https://arxiv.org/html/2607.15455](https://arxiv.org/html/2607.15455)
###### Abstract

Researchers increasingly use automated classifiers to label unstructured data for statistical analysis\. Existing rectification methods can correct errors in these automated labels using a probability\-sampled audit set, but they usually treat the audit labels as correct\. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator\. We propose Partially Adjudicated Design\-Based Supervised Learning \(PA\-DSL\), a method for this setting\. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels\. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known\. In synthetic and Wikipedia Detox semi\-synthetic experiments,PA\-DSLmaintains nominal coverage and reduces RMSE by 10–17% relative to using only adjudicated labels when noisy human labels contain recoverable signal\.

## 1Introduction

Empirical research in the social sciences, business, law, and related fields often depends on coding raw texts, images, or records into variables that measure substantive constructs\(Krippendorff,[2018](https://arxiv.org/html/2607.15455#bib.bib8); Grimmer and Stewart,[2013](https://arxiv.org/html/2607.15455#bib.bib9)\)\. Traditionally, this coding has relied on human or expert judgment, making it costly, time\-consuming, and prone to inconsistency\. Automated classifiers now make it possible to code data at scale, ranging from traditional supervised text classifiers to newer LLM\-based systems\(Gilardiet al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib10); Chewet al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib11); Törnberg,[2025](https://arxiv.org/html/2607.15455#bib.bib12)\)\. But classifier\-generated labels are noisy measurements, not observed outcomes\(Wanget al\.,[2020](https://arxiv.org/html/2607.15455#bib.bib13)\)\. Systematic prediction errors can bias downstream estimates and produce undercovered confidence intervals, even when classifiers achieve high overall accuracy\(Neuhaus,[1999](https://arxiv.org/html/2607.15455#bib.bib18)\)\.

This has motivated a growing set of rectification methods that use higher\-quality validation or audit data to correct estimates based on full\-population surrogate labels\(Hopkins and King,[2010](https://arxiv.org/html/2607.15455#bib.bib16); Wanget al\.,[2020](https://arxiv.org/html/2607.15455#bib.bib13); Angelopouloset al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib15); Egamiet al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib14); Fong and Tyler,[2021](https://arxiv.org/html/2607.15455#bib.bib17)\)\. The common structure is simple: inexpensive automated labels provide scale, while a smaller set of trusted labels anchors the correction\. Design\-Based Supervised Learning \(DSL\), for example, combines a full\-population surrogate score with a probability\-sampled audit to obtain valid inference for downstream estimands such as regression coefficients\(Egamiet al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib14)\)\.

In many real labeling pipelines, however, the audit labels used for correction are not themselves gold standard\. Human annotators disagree, and disagreement often reflects ambiguity, subjectivity, or guideline interpretation rather than only random coder error\(Aroyo and Welty,[2015](https://arxiv.org/html/2607.15455#bib.bib23); Umaet al\.,[2021](https://arxiv.org/html/2607.15455#bib.bib20); Chewet al\.,[2026](https://arxiv.org/html/2607.15455#bib.bib22)\)\. Because expert reconciliation is costly\(Snowet al\.,[2008](https://arxiv.org/html/2607.15455#bib.bib24); Chauet al\.,[2020](https://arxiv.org/html/2607.15455#bib.bib25)\), a common workflow is to have one or more human annotators label all items from an “audited” sample, with only selected items from it receiving expert adjudication\. This multi\-reviewer or adjudication design is widespread, reappearing across domains diverse as content analysis, systematic reviews, clinical endpoint assessment, and NLP annotation\(Lombardet al\.,[2002](https://arxiv.org/html/2607.15455#bib.bib29); Pageet al\.,[2021](https://arxiv.org/html/2607.15455#bib.bib27); Held,[2019](https://arxiv.org/html/2607.15455#bib.bib28); Finlayson and Erjavec,[2017](https://arxiv.org/html/2607.15455#bib.bib26)\)\. Existing rectification strategies do not fully address this setting because the audit used to correct the surrogate may itself be measured with error\.

This paper introduces*Partially Adjudicated Design\-Based Supervised Learning*\(PA\-DSL\), a design\-based estimator for downstream inference under hierarchical label noise\. Conceptually,PA\-DSLis not a new estimating principle from scratch, but a nested application of known\-probability AIPW/DSL corrections to a partially adjudicated annotation design\. It uses an inner correction to turn the adjudicated subset into audit\-level pseudo\-labels, and an outer correction to use those pseudo\-labels to debias inference based on the full\-population surrogate\. To our knowledge, this is the first work to extend rectification methods beyond the usual gold\-audit setting, allowing the human audit labels used for correction to be noisy and only partially adjudicated\.

### Contributions\.

The paper makes three contributions\. First, it formalizes a three\-tier nested measurement design with noisy audit labels and partial adjudication\. Second, it proves that the resulting pseudo\-outcome is design\-valid for any downstream estimating equation affine in the latent outcome, provided the audit and adjudication probabilities are known by design\. Third, it shows in synthetic and semi\-synthetic experiments thatPA\-DSLretains the coverage of adjudication\-only DSL while improving efficiency when the noisy audit labels contain recoverable information about the adjudicated construct\.

## 2Setup: Hierarchical Label Noise and Target Estimand

### Observed data\.

We observe an i\.i\.d\. sample of sizeNN, where each unitiicontributes

Wi=\(𝑿i,Qi,Ri,Gi,1,Gi,2,Vi,Ai\),W\_\{i\}\\;=\\;\\bigl\(\\bm\{X\}\_\{i\},\\,Q\_\{i\},\\,R\_\{i\},\\,G\_\{i,1\},\\,G\_\{i,2\},\\,V\_\{i\},\\,A\_\{i\}\\bigr\),\(1\)generated jointly with a latent binary truthYi⋆∈\{0,1\}Y^\{\\star\}\_\{i\}\\in\\\{0,1\\\}that the analyst does*not*observe\. Here𝑿i∈ℝp\+1\\bm\{X\}\_\{i\}\\in\\mathbb\{R\}^\{p\+1\}is a covariate vector with leading intercept;Qi∈\[0,1\]Q\_\{i\}\\in\[0,1\]is a real\-valued surrogate score;Ri∈\{0,1\}R\_\{i\}\\in\\\{0,1\\\}is the audit indicator with known sampling probabilityπi∈\(0,1\)\\pi\_\{i\}\\in\(0,1\);Gi,1,Gi,2∈\{0,1\}G\_\{i,1\},G\_\{i,2\}\\in\\\{0,1\\\}are two human coder labels observed iffRi=1R\_\{i\}=1;Vi∈\{0,1\}V\_\{i\}\\in\\\{0,1\\\}is the adjudication indicator with known probabilityρi≡ℙ\(Vi=1∣Ri=1,Zi\)∈\(0,1\)\\rho\_\{i\}\\equiv\\mathbb\{P\}\(V\_\{i\}=1\\mid R\_\{i\}=1,Z\_\{i\}\)\\in\(0,1\)a known function of the inner features defined just below; andAi∈\{0,1\}A\_\{i\}\\in\\\{0,1\\\}is the adjudicated label, observed iffRi​Vi=1R\_\{i\}V\_\{i\}=1\. The scoreQiQ\_\{i\}need not be calibrated; it is treated as an observed surrogate feature whose systematic errors are corrected by the audit design\. We writeBi=\(Qi,𝑿i⊤\)⊤B\_\{i\}=\(Q\_\{i\},\\bm\{X\}\_\{i\}^\{\\top\}\)^\{\\top\}for the*outer*feature vector andZi=\(Qi,𝑿i⊤,Gi,1,Gi,2\)⊤Z\_\{i\}=\(Q\_\{i\},\\bm\{X\}\_\{i\}^\{\\top\},G\_\{i,1\},G\_\{i,2\}\)^\{\\top\}for the*inner*feature vector\. We use two coder labels as the running case because this is a common annotation workflow and makes disagreement\-based adjudication easy to describe\. However, the construction does not require exactly two labels: more generally,ZiZ\_\{i\}may contain any audit\-tier information observed whenRi=1R\_\{i\}=1, includingKiK\_\{i\}coder labels, coder identities, agreement indicators, vote shares, uncertainty scores, item metadata, or other features used in the adjudication design\. The probabilitiesπi\\pi\_\{i\}andρi\\rho\_\{i\}are design parameters chosen by the researcher, so they are*known*, not estimated\.

[Figure˜1](https://arxiv.org/html/2607.15455#S2.F1)summarizes the nested three\-tier sampling design\. Throughout, the tiers refer to nested measurement availability, not mutually exclusive samples\. Tier 1 contains allNNunits and includes\(Xi,Qi\)\(X\_\{i\},Q\_\{i\}\)\. Tier 2 is the audit subset\{i:Ri=1\}\\\{i:R\_\{i\}=1\\\}, which also includes noisy coder labels\(Gi​1,Gi​2\)\(G\_\{i1\},G\_\{i2\}\)\. Tier 3 is the adjudicated subset\{i:Ri​Vi=1\}\\\{i:R\_\{i\}V\_\{i\}=1\\\}, which also includes the adjudicated labelAiA\_\{i\}\. Hence tier 3⊂\\subsettier 2⊂\\subsettier 1\.

Step 1Three\-tier SampleQQXXG1G\_\{1\}G2G\_\{2\}AAStep 2Inner Pass \(Eq\. 3\)QQXXG1G\_\{1\}G2G\_\{2\}AAM^\\widehat\{M\}Step 3Outer Pass: DSL onM^\\widehat\{M\}\(Eq\. 4\)QQXXM^\\widehat\{M\}Y~\\widetilde\{Y\}β^\\widehat\{\\beta\}Figure 1:PA\-DSL pipeline\. Step 1 shows the three nested measurement tiers\. Every unit has covariates and an automated score\(X,Q\)\(X,Q\)\(Tier 1, gray\); a randomized audit sample with known probabilityπi\\pi\_\{i\}adds two noisy human labels\(G1,G2\)\(G\_\{1\},G\_\{2\}\)\(Tier 2, blue\); and a randomized subset of audited units with known probabilityρi\\rho\_\{i\}receives an adjudicated labelAA\(Tier 3, amber\)\. Bar heights encode measurement tier, and dashed outlines mark labels that are unobserved\. Step 2 is the new inner correction, which uses the adjudicated cases to turn the noisy audit information into an audit\-tier pseudo\-labelM^\\widehat\{M\}\(purple\)\. Step 3 applies standard DSL toM^\\widehat\{M\}, producing the final pseudo\-outcomeY~\\widetilde\{Y\}\(coral\) used to estimateβ^\\widehat\{\\beta\}\.
### Target estimand\.

In the applications motivating this paper, the scientific target is often not whether each individual text or record is classified correctly, but how a target construct varies across groups, treatments, or covariates\. For subjective annotation tasks, we interpretYi⋆Y^\{\\star\}\_\{i\}as the label that would be assigned to unitiiunder the target adjudication protocol, rather than as an externally verified objective truth\. We therefore target a population\-level regression summary for this adjudication\-defined construct: the logistic projection ofYi⋆Y^\{\\star\}\_\{i\}onto the covariates\. Formally, we focus on the parameterβ∈ℝp\+1\\beta\\in\\mathbb\{R\}^\{p\+1\}defined as the unique solution of the population score equation

𝔼​\[𝑿i​\(Yi⋆−expit⁡\(𝑿i⊤​β\)\)\]=0,\\mathbb\{E\}\\bigl\[\\bm\{X\}\_\{i\}\\,\\bigl\(Y^\{\\star\}\_\{i\}\-\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\beta\)\\bigr\)\\bigr\]\\;=\\;\\mathbf\{0\},\(2\)reporting the slopeβ1\\beta\_\{1\}on the first non\-intercept covariate\. This is the logistic\-regression M\-estimand forY⋆Y^\{\\star\}given𝑿\\bm\{X\}\. Validity for an objective construct beyond the protocol\-defined target requires the additional measurement\-validity assumption that the adjudication protocol itself captures that construct\.

The working model serves only as a specification choice for the score equation; it has a literal conditional\-probability interpretation if and only if it holds exactly, which we do not assume\.[Equation˜2](https://arxiv.org/html/2607.15455#S2.E2)is*outcome\-affine*: the score is linear inYi⋆Y^\{\\star\}\_\{i\}for any fixedβ\\beta\. The construction in Section[3](https://arxiv.org/html/2607.15455#S3)produces an unbiased pseudo\-outcomeY~i\\widetilde\{Y\}\_\{i\}forYi⋆Y^\{\\star\}\_\{i\}, so any outcome\-affine estimating equation \(the population mean, generalized linear regression, inverse\-propensity weighted treatment effects, etc\.\) can be solved by substitutingY~i\\widetilde\{Y\}\_\{i\}forYi⋆Y^\{\\star\}\_\{i\}\. We treat the population prevalence𝔼​\[Y⋆\]\\mathbb\{E\}\[Y^\{\\star\}\]as the special case in which the score isYi⋆−ψY^\{\\star\}\_\{i\}\-\\psi; see[Appendix˜A](https://arxiv.org/html/2607.15455#A1)for that variant and its empirical results\.

Importantly, we do not assume thatQiQ\_\{i\}is calibrated, unbiased, or a consistent estimator ofYi⋆Y\_\{i\}^\{\\star\}\. The surrogate score is treated as an observed feature available on the full frame\. Its systematic errors are corrected by the audit/adjudication design, as in standard DSL\. Identification comes from the known audit and adjudication probabilities together with the ignorability/correctness assumptions below, not from any correctness property ofQiQ\_\{i\}\. The quality ofQiQ\_\{i\}affects efficiency through the nuisance functions, but not design validity\.

### Identifying assumptions\.

The audit/adjudication design supplies four assumptions; together they identify the population means we will need\.

###### Assumption 1\(Audit ignorability\)\.

Ri⟂⟂Yi⋆∣\(𝑿i,Qi\)R\_\{i\}\\perp\\\!\\\!\\\!\\perp Y^\{\\star\}\_\{i\}\\mid\(\\bm\{X\}\_\{i\},Q\_\{i\}\), withπi\\pi\_\{i\}bounded away from0and11\. Holds by construction whenever audit assignment is a randomized function of observed design variables\.

###### Assumption 2\(Adjudication ignorability\)\.

Vi⟂⟂Yi⋆∣\(Ri=1,Zi\)V\_\{i\}\\perp\\\!\\\!\\\!\\perp Y^\{\\star\}\_\{i\}\\mid\(R\_\{i\}=1,Z\_\{i\}\), withρi\\rho\_\{i\}bounded away from0and11\. Holds whenever adjudication is randomized with known probability as a known function ofZiZ\_\{i\}— for example, high\-probability adjudication on coder disagreement and lower but nonzero adjudication on agreement cases\. Strictly deterministic prioritization rules require an additional positive sampling floor in every design stratum used for inference \([Appendix˜H](https://arxiv.org/html/2607.15455#A8)sweeps this floor\)\.

###### Assumption 3\(Protocol\-level Adjudication correctness\)\.

On adjudicated units,Ai=Yi⋆A\_\{i\}=Y\_\{i\}^\{\\star\}, whereYi⋆Y\_\{i\}^\{\\star\}denotes the label that would be assigned under the target adjudication protocol\.

###### Assumption 4\(Coder labels are observed noisy features\)\.

The coder labels\(Gi,1,Gi,2\)\(G\_\{i,1\},G\_\{i,2\}\)and any other audit\-tier metadata included inZiZ\_\{i\}are observed on the audit set and may be arbitrary noisy functions of the latent truth, item difficulty, coder identity, task context, or other factors\. We do not require coder independence, unbiasedness, a fixed number of coders, or a parametric coder\-error model\.

Assumption 4 is a measurement condition rather than an identifying model: All audit\-tier information, including\(Gi,1,Gi,2\)\(G\_\{i,1\},G\_\{i,2\}\), enter only as observed features inZiZ\_\{i\}\. Their bias, dependence, and heterogeneity are absorbed intoμ0​\(Zi\)\\mu\_\{0\}\(Z\_\{i\}\); design validity comes from the known adjudication probabilityρi\\rho\_\{i\}and adjudication correctness\.

## 3Estimator: Partially Adjudicated DSL

Standard DSL\(Egamiet al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib14)\)replaces the unobservedYi⋆Y^\{\\star\}\_\{i\}with a pseudo\-outcome of the formg^​\(Bi\)\+\(Ri/πi\)​\(Yigold−g^​\(Bi\)\)\\hat\{g\}\(B\_\{i\}\)\+\(R\_\{i\}/\\pi\_\{i\}\)\\bigl\(Y\_\{i\}^\{\\text\{gold\}\}\-\\hat\{g\}\(B\_\{i\}\)\\bigr\), whereg^\\hat\{g\}is a cross\-fit nuisance regression of the gold label onBiB\_\{i\}and the Horvitz–Thompson correction debiases that regression at the known audit weight\. The construction assumesYigold=Yi⋆Y\_\{i\}^\{\\text\{gold\}\}=Y^\{\\star\}\_\{i\}on the audit set; when the audit labels are themselves noisy, substituting any one ofGi,1G\_\{i,1\},Gi,2G\_\{i,2\}, or their majority vote inherits the human bias\. We address this by nesting a second design\-based correction inside, exploiting the fact that the adjudicated labelAiA\_\{i\}is unbiased forYi⋆Y^\{\\star\}\_\{i\}on\{Ri​Vi=1\}\\\{R\_\{i\}V\_\{i\}=1\\\}with known sampling weightρi\\rho\_\{i\}\.

### Step 1 — Inner correction\.

Given any cross\-fit estimatorμ^​\(⋅\)\\hat\{\\mu\}\(\\cdot\)of the inner nuisanceμ0​\(z\)=𝔼​\[Ai∣Zi=z,Ri=1,Vi=1\]\\mu\_\{0\}\(z\)=\\mathbb\{E\}\[A\_\{i\}\\mid Z\_\{i\}=z,R\_\{i\}=1,V\_\{i\}=1\]trained on the adjudicated subset\{Ri​Vi=1\}\\\{R\_\{i\}V\_\{i\}=1\\\}, define the audit\-level pseudo\-label

M^i=μ^​\(Zi\)\+Viρi​\(Ai−μ^​\(Zi\)\),i∈\{Ri=1\}\.\\widehat\{M\}\_\{i\}\\;=\\;\\hat\{\\mu\}\(Z\_\{i\}\)\\;\+\\;\\frac\{V\_\{i\}\}\{\\rho\_\{i\}\}\\,\\bigl\(A\_\{i\}\-\\hat\{\\mu\}\(Z\_\{i\}\)\\bigr\),\\qquad i\\in\\\{R\_\{i\}=1\\\}\.\(3\)Hereμ^​\(Zi\)\\hat\{\\mu\}\(Z\_\{i\}\)uses the surrogate, the covariates, and the two coder labels to predict adjudicated truth, and theVi/ρiV\_\{i\}/\\rho\_\{i\}term debiases that prediction on the adjudicated subset\. This definition is unchanged ifZiZ\_\{i\}contains more than two coder labels or additional audit\-tier metadata; the inner correction only requires that the adjudication probabilityρi=P\(Vi=1∣Ri=1,Zi\)\\rho\_\{i\}=P\(V\_\{i\}=1\\mid R\_\{i\}=1,Z\_\{i\}\)be known by design and bounded away from zero\.

### Step 2 — Outer correction\.

Given a cross\-fit estimatorg^​\(⋅\)\\hat\{g\}\(\\cdot\)of the outer nuisanceg0​\(b\)=𝔼​\{μ0​\(Zi\)∣Bi=b,Ri=1\}g\_\{0\}\(b\)=\\mathbb\{E\}\\\{\\mu\_\{0\}\(Z\_\{i\}\)\\mid B\_\{i\}=b,R\_\{i\}=1\\\}trained on the audit set\{Ri=1\}\\\{R\_\{i\}=1\\\}usingM^i\\widehat\{M\}\_\{i\}as response, define the final pseudo\-outcome

Y~i=g^​\(Bi\)\+Riπi​\(M^i−g^​\(Bi\)\),i=1,…,N\.\\widetilde\{Y\}\_\{i\}\\;=\\;\\hat\{g\}\(B\_\{i\}\)\\;\+\\;\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\,\\bigl\(\\widehat\{M\}\_\{i\}\-\\hat\{g\}\(B\_\{i\}\)\\bigr\),\\qquad i=1,\\dots,N\.\(4\)Under Assumptions 1–3, this target satisfiesg0​\(Bi\)=𝔼​\(Yi⋆∣Bi\)g\_\{0\}\(B\_\{i\}\)=\\mathbb\{E\}\(Y\_\{i\}^\{\\star\}\\mid B\_\{i\}\); the realized pseudo\-labelM^i\\widehat\{M\}\_\{i\}is used only to estimate this population nuisance\.

Substituting[Eq\.˜3](https://arxiv.org/html/2607.15455#S3.E3)into[Eq\.˜4](https://arxiv.org/html/2607.15455#S3.E4)gives the closed form below, which makes the three roles transparent

Y~i=g^​\(Bi\)⏟full\-frame regression\+Riπi​\[μ^​\(Zi\)−g^​\(Bi\)\]⏟audit correction\+Ri​Viπi​ρi​\[Ai−μ^​\(Zi\)\]⏟adjudication correction\\boxed\{\\;\\widetilde\{Y\}\_\{i\}\\;=\\;\\underbrace\{\\hat\{g\}\(B\_\{i\}\)\}\_\{\\text\{full\-frame regression\}\}\\;\+\\;\\underbrace\{\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\,\\bigl\[\\hat\{\\mu\}\(Z\_\{i\}\)\-\\hat\{g\}\(B\_\{i\}\)\\bigr\]\}\_\{\\text\{audit correction\}\}\\;\+\\;\\underbrace\{\\frac\{R\_\{i\}V\_\{i\}\}\{\\pi\_\{i\}\\rho\_\{i\}\}\\,\\bigl\[A\_\{i\}\-\\hat\{\\mu\}\(Z\_\{i\}\)\\bigr\]\}\_\{\\text\{adjudication correction\}\}\\;\}\(5\)
Thus, PA\-DSL uses adjudication only to correct the part of the audit information that cannot be trusted, while still allowing the unadjudicated human labels to improve efficiency throughμ^​\(Zi\)\\hat\{\\mu\}\(Z\_\{i\}\)\.

### Step 3 — Downstream estimating equation\.

SubstituteY~i\\widetilde\{Y\}\_\{i\}forYi⋆Y^\{\\star\}\_\{i\}in the population score[Eq\.˜2](https://arxiv.org/html/2607.15455#S2.E2)and solve the empirical analog:

1N​∑i=1N𝑿i​\(Y~i−expit⁡\(𝑿i⊤​β^\)\)=0,\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\bm\{X\}\_\{i\}\\,\\bigl\(\\widetilde\{Y\}\_\{i\}\-\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\hat\{\\beta\}\)\\bigr\)\\;=\\;\\mathbf\{0\},\(6\)returning the cross\-fit estimatorβ^\\hat\{\\beta\}\. Standard errors come from the sandwich formula evaluated at\(𝑿i,Y~i\)\(\\bm\{X\}\_\{i\},\\widetilde\{Y\}\_\{i\}\)\.

### WhyY~i\\widetilde\{Y\}\_\{i\}is design\-valid\.

Both corrections in[Eqs\.˜3](https://arxiv.org/html/2607.15455#S3.E3)and[4](https://arxiv.org/html/2607.15455#S3.E4)are AIPW\-style augmentations\(Robinset al\.,[1994](https://arxiv.org/html/2607.15455#bib.bib5)\)with known design weights\. Under[Assumptions1](https://arxiv.org/html/2607.15455#Thmassumption1)to[3](https://arxiv.org/html/2607.15455#Thmassumption3), iterated expectations give𝔼​\[M^i∣Ri=1,Zi\]=𝔼​\[Yi⋆∣Ri=1,Zi\]\\mathbb\{E\}\[\\widehat\{M\}\_\{i\}\\mid R\_\{i\}=1,Z\_\{i\}\]=\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid R\_\{i\}=1,Z\_\{i\}\]and𝔼​\[Y~i∣Bi\]=𝔼​\[Yi⋆∣Bi\]\\mathbb\{E\}\[\\widetilde\{Y\}\_\{i\}\\mid B\_\{i\}\]=\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid B\_\{i\}\], regardless of nuisance specification\. Thus the nuisances affect efficiency, not design validity; full algebra and influence functions are in[Appendix˜B](https://arxiv.org/html/2607.15455#A2)\.

Algorithm 1PA\-DSLcross\-fit pseudo\-outcome\.Input:data\{\(𝑿i,Qi,Ri,Gi,1,Gi,2,Vi,Ai\)\}i=1N\\\{\(\\bm\{X\}\_\{i\},Q\_\{i\},R\_\{i\},G\_\{i,1\},G\_\{i,2\},V\_\{i\},A\_\{i\}\)\\\}\_\{i=1\}^\{N\}; design weights\{πi,ρi\}i=1N\\\{\\pi\_\{i\},\\rho\_\{i\}\\\}\_\{i=1\}^\{N\}; foldsKK; nuisance learners forμ^,g^\\hat\{\\mu\},\\hat\{g\}\. Output:pseudo\-outcome\{Y~i\}i=1N\\\{\\widetilde\{Y\}\_\{i\}\\\}\_\{i=1\}^\{N\}\.

1. 1\.Partition\{1,…,N\}\\\{1,\\dots,N\\\}into foldsℱ1,…,ℱK\\mathcal\{F\}\_\{1\},\\dots,\\mathcal\{F\}\_\{K\}\. Letfold​\(i\)\\mathrm\{fold\}\(i\)denote the fold containing unitii\.
2. 2\.Inner pass\.Fork=1,…,Kk=1,\\dots,K: 1. \(a\)Fitμ^\(−k\)\\hat\{\\mu\}^\{\(\-k\)\}on\{\(Zi,Ai\):Ri​Vi=1,i∉ℱk\}\\\{\(Z\_\{i\},A\_\{i\}\):R\_\{i\}V\_\{i\}=1,\\;i\\notin\\mathcal\{F\}\_\{k\}\\\}\. 2. \(b\)For audited unitsi∈ℱki\\in\\mathcal\{F\}\_\{k\}\(Ri=1R\_\{i\}=1\), setM^i=μ^\(−k\)​\(Zi\)\+\(Vi/ρi\)​\(Ai−μ^\(−k\)​\(Zi\)\)\\widehat\{M\}\_\{i\}=\\hat\{\\mu\}^\{\(\-k\)\}\(Z\_\{i\}\)\+\(V\_\{i\}/\\rho\_\{i\}\)\\bigl\(A\_\{i\}\-\\hat\{\\mu\}^\{\(\-k\)\}\(Z\_\{i\}\)\\bigr\)\.
3. 3\.Outer pass\.Fork=1,…,Kk=1,\\dots,K: 1. \(a\)Fitg^\(−k\)\\hat\{g\}^\{\(\-k\)\}on\{\(Bi,M^i\):Ri=1,i∉ℱk\}\\\{\(B\_\{i\},\\widehat\{M\}\_\{i\}\):R\_\{i\}=1,\\;i\\notin\\mathcal\{F\}\_\{k\}\\\}, using the out\-of\-foldM^i\\widehat\{M\}\_\{i\}from step 2\(b\)\. 2. \(b\)For*all*unitsi∈ℱki\\in\\mathcal\{F\}\_\{k\}, setY~i=g^\(−k\)​\(Bi\)\+\(Ri/πi\)​\(M^i−g^\(−k\)​\(Bi\)\)\\widetilde\{Y\}\_\{i\}=\\hat\{g\}^\{\(\-k\)\}\(B\_\{i\}\)\+\(R\_\{i\}/\\pi\_\{i\}\)\\bigl\(\\widehat\{M\}\_\{i\}\-\\hat\{g\}^\{\(\-k\)\}\(B\_\{i\}\)\\bigr\)\.
4. 4\.Return\{Y~i\}i=1N\\\{\\widetilde\{Y\}\_\{i\}\\\}\_\{i=1\}^\{N\}\.

### Cross\-fitting\.

Algorithm[1](https://arxiv.org/html/2607.15455#algorithm1)uses a shared fold partition for the inner and outer passes, which is the implementation used in the experiments\. For the cleanest theoretical argument, Appendix[B](https://arxiv.org/html/2607.15455#A2)first analyzes a leakage\-free nested variant in which the outer nuisance for a held\-out fold is trained using audit pseudo\-labels constructed without any information from that fold\. The shared\-partition version differs only through an indirect dependence of the outer nuisance on the held\-out fold through the inner pseudo\-labels used in outer training\. Appendix[B](https://arxiv.org/html/2607.15455#A2)isolates this difference as a stability remainder and states the high\-level condition under which the shared\-partition implementation has the same first\-order expansion\.

Writem​\(Wi;β,η\)=𝑿i​\(Y~i​\(η\)−expit⁡\(𝑿i⊤​β\)\)m\(W\_\{i\};\\beta,\\eta\)=\\bm\{X\}\_\{i\}\\,\(\\widetilde\{Y\}\_\{i\}\(\\eta\)\-\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\beta\)\)for the estimating function, withη=\(μ,g\)\\eta=\(\\mu,g\)now an explicit argument, and letβ0\\beta\_\{0\}denote the population value ofβ\\betadefined by[Eq\.˜2](https://arxiv.org/html/2607.15455#S2.E2)\.

###### Proposition 1\(Design validity and asymptotic normality\)\.

Suppose[Assumptions1](https://arxiv.org/html/2607.15455#Thmassumption1)to[4](https://arxiv.org/html/2607.15455#Thmassumption4)hold, the design weightsπi,ρi\\pi\_\{i\},\\rho\_\{i\}are known and bounded away from zero and one,𝔼​‖𝐗i‖2\+δ<∞\\mathbb\{E\}\\\|\\bm\{X\}\_\{i\}\\\|^\{2\+\\delta\}<\\inftyfor someδ\>0\\delta\>0, and

𝐇=𝔼​\[𝑿i​𝑿i⊤​expit′⁡\(𝑿i⊤​β0\)\]\\mathbf\{H\}=\\mathbb\{E\}\\\!\\left\[\\bm\{X\}\_\{i\}\\bm\{X\}\_\{i\}^\{\\top\}\\operatorname\{expit\}^\{\\prime\}\\\!\\left\(\\bm\{X\}\_\{i\}^\{\\top\}\\beta\_\{0\}\\right\)\\right\]is nonsingular\. For the leakage\-free nested cross\-fit estimator, under cross\-fitL2L^\{2\}\-consistent nuisance estimation,β^\\hat\{\\beta\}defined by[Eq\.˜6](https://arxiv.org/html/2607.15455#S3.E6)satisfies

N​\(β^−β0\)=𝐇−1​1N​∑i=1Nm​\(Wi;β0,η0\)\+op​\(1\),\\sqrt\{N\}\(\\hat\{\\beta\}\-\\beta\_\{0\}\)=\\mathbf\{H\}^\{\-1\}\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{i=1\}^\{N\}m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\+o\_\{p\}\(1\),and hence

N​\(β^−β0\)↝𝒩​\(𝟎,𝐇−1​Ω​𝐇−1⊤\),\\sqrt\{N\}\(\\hat\{\\beta\}\-\\beta\_\{0\}\)\\rightsquigarrow\\mathcal\{N\}\\\!\\left\(\\mathbf\{0\},\\,\\mathbf\{H\}^\{\-1\}\\Omega\\mathbf\{H\}^\{\-1\\top\}\\right\),where

Ω=𝔼​\[m​\(Wi;β0,η0\)​m​\(Wi;β0,η0\)⊤\]=Var⁡\{m​\(Wi;β0,η0\)\},η0=\(μ0,g0\)\.\\Omega=\\mathbb\{E\}\\\!\\left\[m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)^\{\\top\}\\right\]=\\operatorname\{Var\}\\\{m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\\},\\qquad\\eta\_\{0\}=\(\\mu\_\{0\},g\_\{0\}\)\.The equality to the variance holds because𝔼​\{m​\(Wi;β0,η0\)\}=𝟎\\mathbb\{E\}\\\{m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\\}=\\mathbf\{0\}\. The covariance matrix𝐇−1​Ω​𝐇−1⊤\\mathbf\{H\}^\{\-1\}\\Omega\\mathbf\{H\}^\{\-1\\top\}is consistently estimated by the empirical cross\-fit sandwich estimator\. Moreover, for any fixed nuisance pairη=\(μ,g\)\\eta=\(\\mu,g\),

𝔼​\[m​\(Wi;β0,η\)\]=𝟎,\\mathbb\{E\}\\\!\\left\[m\(W\_\{i\};\\beta\_\{0\},\\eta\)\\right\]=\\mathbf\{0\},so the score is design\-unbiased atβ0\\beta\_\{0\}regardless of nuisance specification\. The same expansion also holds for the shared\-partition implementation in[Algorithm˜1](https://arxiv.org/html/2607.15455#algorithm1)under the high\-level stability condition stated in[Appendix˜B](https://arxiv.org/html/2607.15455#A2)\. Full assumptions and the influence\-function derivation for the leakage\-free nested estimator are given in[Appendix˜B](https://arxiv.org/html/2607.15455#A2)\.

Unlike doubly robust settings with estimated sampling weights, no product\-rate condition is needed because the known design weights make the linear nuisance remainder conditionally mean\-zero; Appendix[B](https://arxiv.org/html/2607.15455#A2)gives the full argument and the shared\-partition stability condition\.

We instantiate[Algorithm˜1](https://arxiv.org/html/2607.15455#algorithm1)withK=5K=5cross\-fitting folds and useL2L^\{2\}\-regularized logistic regression for bothμ^\\hat\{\\mu\}andg^\\hat\{g\}\. In the experiments, we balance folds on the twentyQQ\-quintile×\\timesBdiffB\_\{\\mathrm\{diff\}\}\-quartile cells as a finite\-sample stabilization heuristic\. This fold stratification is not part of the identifying design and is not required for design validity; ordinary random folds or other balanced fold constructions could also be used\. Prediction clipping, the full estimator catalogue, and an ablation over the inner feature set appear in[Appendices˜D](https://arxiv.org/html/2607.15455#A4)and[F](https://arxiv.org/html/2607.15455#A6)\.

## 4Experimental Design

BecauseY⋆Y^\{\\star\}is latent in real annotation problems, empirical benchmarks must either assume a proxy truth or move to simulation\. We therefore combine a synthetic Monte Carlo study, whereY⋆Y^\{\\star\}is known by construction and the measurement design can be varied, with a semi\-synthetic study on the Wikipedia Detox corpus\(Wulczynet al\.,[2017](https://arxiv.org/html/2607.15455#bib.bib4)\), where real text and crowdworker labels replace the parametric annotation model but the target is necessarily a reference\-panel proxy\. The two studies respectively test design validity against a known target and robustness under real annotator disagreement\.

### Synthetic data\-generating process\.

Each replication generatesN=50,000N=50\{,\}000units\. The design is intentionally richer than independent label\-flip noise\. Motivated by work on human label variation and learning from disagreement, we allow some items to be intrinsically ambiguous, some coders to be systematically more or less reliable, and machines and humans to have partially distinct error modes\(Plank,[2022](https://arxiv.org/html/2607.15455#bib.bib19); Umaet al\.,[2021](https://arxiv.org/html/2607.15455#bib.bib20); Jiang and de Marneffe,[2022](https://arxiv.org/html/2607.15455#bib.bib21)\)\. Latent labelsYi⋆Y\_\{i\}^\{\\star\}are drawn from a logistic model inp=5p=5correlated covariates\. Three latent difficulty channels then modulate human and surrogate accuracy:DiD\_\{i\}, a shared item\-difficulty channel affecting both humans and machines;HiH\_\{i\}, a human\-only nuisance channel; andMiM\_\{i\}, a machine\-only nuisance channel\. The surrogateQiQ\_\{i\}depends onYi⋆Y\_\{i\}^\{\\star\}, covariates,DiD\_\{i\}, andMiM\_\{i\}; the coder labelsGi​1,Gi​2G\_\{i1\},G\_\{i2\}depend onYi⋆Y\_\{i\}^\{\\star\}, coder\-specific sensitivity/specificity,DiD\_\{i\}, andHiH\_\{i\}\. Thus coder errors are heterogeneous and correlated through hard items, and surrogate errors are allowed to differ from human errors\. This structure mirrors empirical annotation settings in which disagreement reflects a mixture of ambiguity, annotator effects, and task artifacts, while downstream regression remains vulnerable to response misclassification\(Neuhaus,[1999](https://arxiv.org/html/2607.15455#bib.bib18); Wanget al\.,[2020](https://arxiv.org/html/2607.15455#bib.bib13)\)\. The audit indicator is uniform Bernoulli\(π=0\.10\\pi=0\.10\), reflecting a modest manual\-review budget\. Adjudication is either uniform at mean rateρ¯\\bar\{\\rho\}or disagreement\-weighted to meanρ¯\\bar\{\\rho\}\. The latter targets cases where the two coders disagree, motivated by evidence that disagreement often reflects item ambiguity, task subjectivity, or guideline instability rather than only idiosyncratic annotator noise\(Plank,[2022](https://arxiv.org/html/2607.15455#bib.bib19); Umaet al\.,[2021](https://arxiv.org/html/2607.15455#bib.bib20); Jiang and de Marneffe,[2022](https://arxiv.org/html/2607.15455#bib.bib21)\)\. We summarize the simulation design with three primary scenarios:*benign*\(high\-quality surrogate, low coder noise, uniform adjudication,ρ¯=0\.25\\bar\{\\rho\}=0\.25\);*realistic*\(medium\-quality surrogate, moderate coder noise, disagreement\-driven adjudication,ρ¯=0\.25\\bar\{\\rho\}=0\.25\); and*hard*\(weak surrogate, high coder heterogeneity, sparse adjudication,ρ¯=0\.10\\bar\{\\rho\}=0\.10\)\. Full equations, default parameter values, and the scenario parameter table appear in[Appendix˜E](https://arxiv.org/html/2607.15455#A5)\.

### Semi\-synthetic study \(Wikipedia Detox\)\.

The Wikipedia Detox corpus contains talk\-page comments labeled by crowdworkers for whether the comment contains a personal attack \(i\.e\., hostile or insulting language directed at another contributor\)\. In our notation,Yi⋆Y^\{\\star\}\_\{i\}denotes whether commentiiwould be judged to contain a personal attack under the target labeling standard\. Because this status is unobserved in the Wikipedia data, the study is not a field validation against external ground truth\. We instead evaluate against a researcher\-constructed proxy truth estimated from a held\-out reference panel of crowdworkers\. Thus, the Detox study evaluates recovery of a reference\-panel operationalization of the construct under real annotator disagreement, not validity against an external oracle for personal attack\. In each replication, we re\-draw the audit and adjudication samples under known design probabilities\. The corpus comprisesN≈1\.16×105N\\approx 1\.16\\times 10^\{5\}talk\-page comments with multiple crowdworker attack ratings per comment\. Real\-world covariates retained for the analysis are the editor’s logged\-in status, namespace indicator, comment year \(standardized\), and standardized log comment length; the surrogateQiQ\_\{i\}is a TF–IDF logistic classifier fit out\-of\-fold on the comment text\. We partition the crowdworkers into a reference panel \(used for the proxy truth\) and two audit panels \(used asGi,1,Gi,2G\_\{i,1\},G\_\{i,2\}\)\. The main\-text proxy truth is a binary Dawid–Skene estimate from a held\-out reference panel\(Dawid and Skene,[1979](https://arxiv.org/html/2607.15455#bib.bib3)\); results under hard\-majority and soft\-mean\-share proxies appear in[Appendix˜G](https://arxiv.org/html/2607.15455#A7)\. Audit and adjudication indicators are re\-drawn each replication fromRi∼Bernoulli​\(0\.10\)R\_\{i\}\\sim\\mathrm\{Bernoulli\}\(0\.10\)and disagreement\-weightedρi\\rho\_\{i\}withρ¯=0\.25\\bar\{\\rho\}=0\.25; we useR=200R=200replications\.

### Estimators\.

[Table˜1](https://arxiv.org/html/2607.15455#S4.T1)summarizes the main estimators\.PA\-DSLuses[Eq\.˜5](https://arxiv.org/html/2607.15455#S3.E5)with inner featuresZi=\(Qi,𝑿i,Gi,1,Gi,2\)Z\_\{i\}=\(Q\_\{i\},\\bm\{X\}\_\{i\},G\_\{i,1\},G\_\{i,2\}\)and L2\-regularized logistic nuisances\. Appendix[Appendix˜F](https://arxiv.org/html/2607.15455#A6)reports a variant that augments the inner feature set with coder\-pair indicators\.

Table 1:Estimators compared in the main experiments\.

## 5Results

We organize the empirical results around four claims, focusing on the logistic\-regression slopeβ1\\beta\_\{1\}throughout\. The synthetic simulation analysis usesR=1,000R=1\{,\}000replications per scenario with trueβ1=1\\beta\_\{1\}=1; the Detox study usesR=200R=200replications with the Dawid–Skene proxy truth as the target\. Prevalence and the inner\-stage ablation are tabulated in[Appendices˜A](https://arxiv.org/html/2607.15455#A1)and[F](https://arxiv.org/html/2607.15455#A6)\.

### Claim 1: noisy plug\-ins fail\.

[Figure˜2](https://arxiv.org/html/2607.15455#S5.F2)shows thatSurrogate\-only,Human\-naive, andDSL\-MajVoteare all substantially biased, with bias ranging from−0\.41\-0\.41in the benign scenario to−0\.85\-0\.85in the hard scenario and empirical95%95\\%coverage equal to0\.0000\.000in every cell\. Standard DSL does not repair this failure when its “gold” audit label is itself noisy: the correction removes surrogate\-side bias relative to the audit label, and therefore inherits bias in that label\. Majority vote also fails because coder errors are correlated through shared item difficulty, so the bias common to both coders survives aggregation\.

![Refer to caption](https://arxiv.org/html/2607.15455v1/figures/bias_coverage_beta1.png)Figure 2:Synthetic study,β1\\beta\_\{1\}across three scenarios \(R=1,000R=1\{,\}000Monte Carlo replications, trueβ1=1\\beta\_\{1\}=1\)\.*Left:*bias\.*Right:*empirical95%95\\%coverage\. The four plug\-in baselines on the left of each panel are biased−0\.41\-0\.41to−0\.85\-0\.85with zero coverage; the three audit\-corrected estimators on the right \(DSL\-AdjOnly,PA\-DSL,DSL\-oracle\) recover near\-zero bias and nominal coverage in every scenario\.
### Claim 2: audit\-corrected estimators recover validity\.

[Figure˜2](https://arxiv.org/html/2607.15455#S5.F2)also shows thatDSL\-AdjOnly,PA\-DSL, and the infeasibleDSL\-oracleall sit within Monte Carlo error of zero bias and report empirical coverage in\[0\.94,0\.96\]\[0\.94,0\.96\]across all three scenarios\. Adjudication\-only DSL is design\-valid because it uses only the adjudicated subset\{Ri​Vi=1\}\\\{R\_\{i\}V\_\{i\}=1\\\}with the joint sampling weight1/\(π​ρi\)\.1/\(\\pi\\rho\_\{i\}\)\.The price it pays is that the unadjudicated coder labelsGi,1,Gi,2G\_\{i,1\},G\_\{i,2\}on the rest of the audit set are simply discarded\.

### Claim 3:PA\-DSLimproves efficiency where the inner layer has signal\.

[Figure˜3](https://arxiv.org/html/2607.15455#S5.F3)reports the efficiency ratios ofPA\-DSLversusDSL\-AdjOnly\(the strongest feasible baseline\) andDSL\-oracle\(the infeasible benchmark\), with paired\-bootstrap95%95\\%CIs over the1,0001\{,\}000Monte Carlo replications\. Relative toDSL\-AdjOnly, the RMSE ratio is1\.211\.21\(95%95\\%CI\[1\.17,1\.25\]\[1\.17,1\.25\]\) in benign,1\.111\.11\(\[1\.08,1\.14\]\[1\.08,1\.14\]\) in realistic, and1\.011\.01\(\[1\.00,1\.02\]\[1\.00,1\.02\]\) in hard, with corresponding variance ratios1\.461\.46\(\[1\.36,1\.57\]\[1\.36,1\.57\]\),1\.211\.21\(\[1\.16,1\.27\]\[1\.16,1\.27\]\), and1\.021\.02\(\[1\.00,1\.04\]\[1\.00,1\.04\]\)\. The benign and realistic gains are clear; the hard\-regime ratio’s lower bound sits within5×10−45\\times 10^\{\-4\}of11, soPA\-DSL*reverts to the adjudication\-only benchmark*when the inner layer has no recoverable signal\. In this scenario, the surrogate is weak, the coders are noisy, and the adjudication budget is sparse, so predictingAiA\_\{i\}from\(Qi,Xi,Gi,1,Gi,2\)\(Q\_\{i\},X\_\{i\},G\_\{i,1\},G\_\{i,2\}\)buys little leverage beyond theπi​ρi\\pi\_\{i\}\\rho\_\{i\}\-weighted adjudicated subset\. The reversion is the right behavior:PA\-DSLdoes not manufacture signal absent from the inner features\. Classification\-oriented diagnostics in Appendix[L](https://arxiv.org/html/2607.15455#A12)show the same pattern at the prediction level:μ^​\(Zi\)\\widehat\{\\mu\}\(Z\_\{i\}\)improves ong^​\(Bi\)\\widehat\{g\}\(B\_\{i\}\)in the benign and realistic settings, but adds little signal in the hard setting\.

![Refer to caption](https://arxiv.org/html/2607.15455v1/figures/efficiency_beta1.png)Figure 3:Relative efficiency ofPA\-DSLonβ1\\beta\_\{1\}\. Bars above one favorPA\-DSL; whiskers are paired\-bootstrap95%95\\%CIs over the1,0001\{,\}000Monte Carlo replications\. AgainstDSL\-AdjOnly\(blue\),PA\-DSLachieves materially smaller variance and CI length in benign and realistic; in hard the inner stage extracts essentially no additional signal and the lower CI bound sits within5×10−45\\times 10^\{\-4\}of11\. AgainstDSL\-oracle\(orange\) the gap to the infeasible benchmark is roughly1\.6×1\.6\\timesin benign and widens to6\.3×6\.3\\timesin hard, the price paid for not observingY⋆Y^\{\\star\}on the audit set\.
### Claim 4: Detox replicates the qualitative pattern\.

[Table˜2](https://arxiv.org/html/2607.15455#S5.T2)shows the same comparison on the Wikipedia Detox corpus with the Dawid–Skene proxy truth \(β1=−1\.42\\beta\_\{1\}=\-1\.42onlogged\_in\)\. The plug\-in baselines and the standard\-DSL variants fed with noisy gold are biased by0\.330\.33–0\.390\.39with zero empirical coverage;DSL\-AdjOnlyandPA\-DSLboth achieve near\-zero bias and coverage at or slightly above95%95\\%\.PA\-DSL’s RMSE ratio againstDSL\-AdjOnlyis1\.181\.18\(95%95\\%paired\-bootstrap CI\[1\.10,1\.27\]\[1\.10,1\.27\]\) and its CI\-length ratio is1\.131\.13\(\[1\.12,1\.13\]\[1\.12,1\.13\]\); both are clearly above11, so the efficiency gain is statistically distinguishable from the no\-gain point\. The same qualitative ranking is reproduced under the hard\-majority and soft\-mean proxies in[Appendix˜G](https://arxiv.org/html/2607.15455#A7)\. As above, this is evidence for the sampling\-and\-adjudication machinery under real text and real crowdworker disagreement, with respect to the constructed proxy truth rather than an observed latent construct\.

Table 2:Detox semi\-synthetic study,β1\\beta\_\{1\}onlogged\_inunder the Dawid–Skene proxy truth \(R=200R=200replications, targetβ1=−1\.42\\beta\_\{1\}=\-1\.42\)\.Boldmarks the column\-best for MC SD, RMSE, and CI length among design\-valid estimators \(DSL\-AdjOnly,PA\-DSL\); coverage is a calibration diagnostic and bias is reported but not bolded\. The light rule separates the infeasibleDSL\-oraclebenchmark\.

## 6Discussion

PA\-DSLextends design\-based supervised learning to hierarchical annotation pipelines in which adjudication is treated as the gold\-standard label but is available only for a subset of audited cases\. The results suggest several practical lessons for designing and analyzing such pipelines\.

### When the proposal helps and when it does not\.

PA\-DSLdelivers efficiency gains over adjudication\-only DSL when the audit\-tier featuresZi=\(Qi,Xi,Gi,1,Gi,2\)Z\_\{i\}=\(Q\_\{i\},X\_\{i\},G\_\{i,1\},G\_\{i,2\}\)predict the adjudicated labelAiA\_\{i\}beyond the outer featuresBi=\(Qi,Xi\)B\_\{i\}=\(Q\_\{i\},X\_\{i\}\)\. When this inner layer contains little additional signal,PA\-DSLbehaves like the adjudication\-only benchmark, as in the hard synthetic regime\. This pattern is formalized by the variance comparison in[Appendix˜B](https://arxiv.org/html/2607.15455#A2.SS0.SSS0.Px10), where the gain over adjudication\-only DSL is proportional to theZZ\-explainable componentμ0​\(Zi\)−g0​\(Bi\)\\mu\_\{0\}\(Z\_\{i\}\)\-g\_\{0\}\(B\_\{i\}\)\. The inner ablations in[Appendix˜I](https://arxiv.org/html/2607.15455#A9)also show that the augmented\-IPW form is not merely cosmetic: regression\-only correction produced severe undercoverage in the realistic and hard regimes, underscoring the need to combine the outcome model with the known adjudication design\.

### Design implications\.

The design sensitivity analysis in[Appendix˜H](https://arxiv.org/html/2607.15455#A8)suggests a practical operating range in the settings we studied: audit rates of roughlyπ∈\[0\.10,0\.20\]\\pi\\in\[0\.10,0\.20\]combined with disagreement\-driven adjudication atρ¯∈\[0\.10,0\.25\]\\bar\{\\rho\}\\in\[0\.10,0\.25\]\. Disagreement\-driven adjudication should include a positive adjudication floor even among agreement cases; otherwise, the design cannot identify error patterns in apparently easy items\. More generally, the two\-coder setup used in our experiments is not required by the framework:ZiZ\_\{i\}may include any audit\-tier information observed whenRi=1R\_\{i\}=1, includingKiK\_\{i\}coder labels, coder identities, agreement indicators, vote shares, uncertainty scores, or other metadata\. The same inner correction applies as long as the adjudication probabilityρi=P\(Vi=1∣Ri=1,Zi\)\\rho\_\{i\}=P\(V\_\{i\}=1\\mid R\_\{i\}=1,Z\_\{i\}\)is known and bounded away from zero\. The realized audit and adjudication probabilities should also be recorded with the labels, sincePA\-DSLrelies on knownπi\\pi\_\{i\}andρi\\rho\_\{i\}rather than estimating them after the fact\. A pilot audit can help assess whether coder labels contain enough signal about the adjudicated construct to justify collecting additional human labels before increasing the adjudication budget\.

### Scope and limitations\.

Although LLM\-based annotation is one motivating use case, PA\-DSL is a classifier\-agnostic design\-based correction: the surrogateQiQ\_\{i\}may come from an LLM, a supervised classifier, a rules\-based score, or any other automated labeling system observed on the full frame\. However,PA\-DSLis intended for prospective, known\-probability audit/adjudication designs, or for settings in which the audit and adjudication probabilities can be reconstructed from the design record\. It is not a generic post hoc correction for arbitrary deterministic review pipelines: if audited or adjudicated cases were selected using undocumented rules, or if some design strata have zero adjudication probability, the required positivity and ignorability conditions may fail\.

Validity also relies on[Assumption˜3](https://arxiv.org/html/2607.15455#Thmassumption3)\. We interpret this assumption as requiring adjudication to recover the target label under the adjudication protocol\.[Appendix˜K](https://arxiv.org/html/2607.15455#A11)shows thatPA\-DSLinherits adjudicator bias relative toYi⋆Y^\{\\star\}\_\{i\}when[Assumption˜3](https://arxiv.org/html/2607.15455#Thmassumption3)fails\. Thus,PA\-DSLcorrects the sampling and measurement structure only relative to the adjudicated construct; it cannot by itself guarantee that the adjudication protocol captures the substantive construct researchers intended\. Estimators that remain valid under imperfect adjudication, treatingAiA\_\{i\}as a noisy proxy with a known design weight, are a natural next step\.

The Detox study should be interpreted in this same spirit\. It is a semi\-synthetic evaluation using real text and real crowdworker disagreement, but the audit/adjudication indicators are re\-drawn with known probabilities and the target is a held\-out reference\-panel operationalization rather than an external field\-validated oracle\. Non\-binary outcomes, scores nonlinear inYi⋆Y^\{\\star\}\_\{i\}, and dependent\-class missingness lie outside the current scope\.

## References

- A\. N\. Angelopoulos, S\. Bates, C\. Fannjiang, M\. I\. Jordan, and T\. Zrnic \(2023\)Prediction\-powered inference\.Science382\(6671\),pp\. 669–674\.Cited by:[Appendix C](https://arxiv.org/html/2607.15455#A3.p1.1),[§1](https://arxiv.org/html/2607.15455#S1.p2.1)\.
- L\. Aroyo and C\. Welty \(2015\)Truth is a lie: crowd truth and the seven myths of human annotation\.AI magazine36\(1\),pp\. 15–24\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- H\. Chau, S\. Balaneshin, K\. Liu, and O\. Linda \(2020\)Understanding the tradeoff between cost and quality of expert annotations for keyphrase extraction\.InProceedings of the 14th Linguistic Annotation Workshop,pp\. 74–86\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- R\. Chew, J\. Bollenbacher, M\. Wenger, J\. Speer, and A\. Kim \(2023\)LLM\-assisted content analysis: using large language models to support deductive coding\.arXiv preprint arXiv:2306\.14924\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1)\.
- R\. Chew, S\. Eckman, C\. Kern, and F\. Kreuter \(2026\)From ground truth to measurement: a statistical framework for human labeling\.arXiv preprint arXiv:2604\.07591\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- A\. P\. Dawid and A\. M\. Skene \(1979\)Maximum likelihood estimation of observer error\-rates using the EM algorithm\.Journal of the Royal Statistical Society\. Series C \(Applied Statistics\)28\(1\),pp\. 20–28\.Cited by:[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px2.p1.9)\.
- N\. A\. de Pieuchon, A\. Daoud, C\. T\. Jerzak, M\. Johansson, and R\. Johansson \(2025\)Benchmarking debiasing methods for llm\-based parameter estimates\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 19768–19783\.Cited by:[Appendix C](https://arxiv.org/html/2607.15455#A3.p1.1)\.
- N\. Egami, M\. Hinck, B\. Stewart, and H\. Wei \(2023\)Using imperfect surrogates for downstream inference: design\-based supervised learning for social science applications of large language models\.Advances in Neural Information Processing Systems36,pp\. 68589–68601\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p2.1),[§3](https://arxiv.org/html/2607.15455#S3.p1.11)\.
- M\. A\. Finlayson and T\. Erjavec \(2017\)Overview of annotation creation: processes and tools\.InHandbook of linguistic annotation,pp\. 167–191\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- C\. Fong and M\. Tyler \(2021\)Machine learning predictions as regression covariates\.Political Analysis29\(4\),pp\. 467–484\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p2.1)\.
- F\. Gilardi, M\. Alizadeh, and M\. Kubli \(2023\)ChatGPT outperforms crowd workers for text\-annotation tasks\.Proceedings of the National Academy of Sciences120\(30\),pp\. e2305016120\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1)\.
- J\. Grimmer and B\. M\. Stewart \(2013\)Text as data: the promise and pitfalls of automatic content analysis methods for political texts\.Political analysis21\(3\),pp\. 267–297\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1)\.
- C\. Held \(2019\)When do we need clinical endpoint adjudication in clinical trials?\.Upsala Journal of Medical Sciences124\(1\),pp\. 42–45\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- D\. J\. Hopkins and G\. King \(2010\)A method of automated nonparametric content analysis for social science\.American Journal of Political Science54\(1\),pp\. 229–247\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p2.1)\.
- N\. Jiang and M\. de Marneffe \(2022\)Investigating reasons for disagreement in natural language inference\.Transactions of the Association for Computational Linguistics10,pp\. 1357–1374\.Cited by:[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px1.p1.20)\.
- K\. Krippendorff \(2018\)Content analysis: an introduction to its methodology\.Sage publications\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1)\.
- M\. Lombard, J\. Snyder\-Duch, and C\. C\. Bracken \(2002\)Content analysis in mass communication: assessment and reporting of intercoder reliability\.Human communication research28\(4\),pp\. 587–604\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- J\. M\. Neuhaus \(1999\)Bias and efficiency loss due to misclassified responses in binary regression\.Biometrika86\(4\),pp\. 843–855\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1),[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px1.p1.20)\.
- M\. J\. Page, D\. Moher, P\. M\. Bossuyt, I\. Boutron, T\. C\. Hoffmann, C\. D\. Mulrow, L\. Shamseer, J\. M\. Tetzlaff, E\. A\. Akl, S\. E\. Brennan,et al\.\(2021\)PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews\.bmj372\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- B\. Plank \(2022\)The “problem” of human label variation: on ground truth in data, modeling and evaluation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px1.p1.20)\.
- J\. M\. Robins, A\. Rotnitzky, and L\. P\. Zhao \(1994\)Estimation of regression coefficients when some regressors are not always observed\.Journal of the American Statistical Association89\(427\),pp\. 846–866\.Cited by:[§3](https://arxiv.org/html/2607.15455#S3.SS0.SSS0.Px4.p1.2)\.
- R\. Snow, B\. O’connor, D\. Jurafsky, and A\. Y\. Ng \(2008\)Cheap and fast–but is it good? evaluating non\-expert annotations for natural language tasks\.InProceedings of the 2008 conference on empirical methods in natural language processing,pp\. 254–263\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1)\.
- P\. Törnberg \(2025\)Large language models outperform expert coders and supervised classifiers at annotating political social media messages\.Social Science Computer Review43\(6\),pp\. 1181–1195\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1)\.
- A\. N\. Uma, T\. Fornaciari, D\. Hovy, S\. Paun, B\. Plank, and M\. Poesio \(2021\)Learning from disagreement: a survey\.Journal of Artificial Intelligence Research72,pp\. 1385–1470\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p3.1),[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px1.p1.20)\.
- S\. Wang, T\. H\. McCormick, and J\. T\. Leek \(2020\)Methods for correcting inference based on outcomes predicted by machine learning\.Proceedings of the National Academy of Sciences117\(48\),pp\. 30266–30275\.Cited by:[§1](https://arxiv.org/html/2607.15455#S1.p1.1),[§1](https://arxiv.org/html/2607.15455#S1.p2.1),[§4](https://arxiv.org/html/2607.15455#S4.SS0.SSS0.Px1.p1.20)\.
- E\. Wulczyn, N\. Thain, and L\. Dixon \(2017\)Ex machina: personal attacks seen at scale\.InProceedings of the 26th International Conference on World Wide Web,pp\. 1391–1399\.Cited by:[§4](https://arxiv.org/html/2607.15455#S4.p1.2)\.

## Appendix APrevalence as a Special Case

The population prevalenceψ=𝔼​\[Y⋆\]\\psi=\\mathbb\{E\}\[Y^\{\\star\}\]is the special case of the outcome\-affine framework in which the score isYi⋆−ψY^\{\\star\}\_\{i\}\-\\psi\. ThePA\-DSLestimator instantiates asψ^=N−1​∑iY~i\\hat\{\\psi\}=N^\{\-1\}\\sum\_\{i\}\\widetilde\{Y\}\_\{i\}withY~i\\widetilde\{Y\}\_\{i\}from[Eq\.˜5](https://arxiv.org/html/2607.15455#S3.E5), and the empirical\-variance standard errorSE^​\(ψ^\)=\(N​\(N−1\)\)−1/2​\(∑i\(Y~i−ψ^\)2\)1/2\\widehat\{\\mathrm\{SE\}\}\(\\hat\{\\psi\}\)=\(N\(N\-1\)\)^\{\-1/2\}\\bigl\(\\sum\_\{i\}\(\\widetilde\{Y\}\_\{i\}\-\\hat\{\\psi\}\)^\{2\}\\bigr\)^\{1/2\}is consistent forN−1/2N^\{\-1/2\}times the standard deviation of the influence function in[Eq\.˜7](https://arxiv.org/html/2607.15455#A2.E7)\.[Table˜3](https://arxiv.org/html/2607.15455#A1.T3)reports the synthetic results for this target across the three scenarios; the qualitative ranking matches theβ1\\beta\_\{1\}results in the main text\.

Table 3:Synthetic study, prevalenceψ\\psi\(R=1,000R=1\{,\}000replications, realized truthY¯⋆≈0\.426\\bar\{Y\}^\{\\star\}\\approx 0\.426\)\.Boldmarks the column\-best for MC SD, RMSE, and CI length per scenario among design\-valid estimators \(DSL\-AdjOnly,PA\-DSL\); coverage is a calibration diagnostic and bias is reported but not bolded\. The light rule separates the infeasibleDSL\-oraclebenchmark\.
## Appendix BIdentification, Influence Function, and Variance

### Notation and population nuisances\.

For notational simplicity, this appendix writes the audit and adjudication probabilities asπ​\(Bi\)\\pi\(B\_\{i\}\)andρ​\(Zi\)\\rho\(Z\_\{i\}\)\. If the actual audit probability depends on additional observed design variables, those variables are either included inBiB\_\{i\}or added to the conditioning set below\. Likewise, any observed variables used to set adjudication probabilities are included inZiZ\_\{i\}\. The arguments require only that the relevant conditional inclusion probabilities are known by design and bounded away from zero\.

We write the population nuisance functions as

μ0​\(z\)=𝔼​\(Ai∣Zi=z,Ri=1,Vi=1\),g0​\(b\)=𝔼​\{μ0​\(Zi\)∣Bi=b,Ri=1\}\.\\mu\_\{0\}\(z\)=\\mathbb\{E\}\(A\_\{i\}\\mid Z\_\{i\}=z,R\_\{i\}=1,V\_\{i\}=1\),\\qquad g\_\{0\}\(b\)=\\mathbb\{E\}\\\{\\mu\_\{0\}\(Z\_\{i\}\)\\mid B\_\{i\}=b,R\_\{i\}=1\\\}\.Under Assumptions 1–3,μ0​\(Zi\)=𝔼​\(Yi⋆∣Zi,Ri=1\)\\mu\_\{0\}\(Z\_\{i\}\)=\\mathbb\{E\}\(Y\_\{i\}^\{\\star\}\\mid Z\_\{i\},R\_\{i\}=1\)andg0​\(Bi\)=𝔼​\(Yi⋆∣Bi\)g\_\{0\}\(B\_\{i\}\)=\\mathbb\{E\}\(Y\_\{i\}^\{\\star\}\\mid B\_\{i\}\)\. Thusg0g\_\{0\}is a population target; it does not depend on the realized nuisance estimate or on a particular cross\-fit pseudo\-label\.

### Pointwise identification\.

Under[Assumption˜1](https://arxiv.org/html/2607.15455#Thmassumption1),𝔼​\[Yi⋆∣𝑿i,Qi\]=𝔼​\[Yi⋆∣𝑿i,Qi,Ri=1\]\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid\\bm\{X\}\_\{i\},Q\_\{i\}\]=\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid\\bm\{X\}\_\{i\},Q\_\{i\},R\_\{i\}=1\]\. Under[Assumptions˜2](https://arxiv.org/html/2607.15455#Thmassumption2)and[3](https://arxiv.org/html/2607.15455#Thmassumption3),𝔼​\[Yi⋆∣Zi,Ri=1\]=𝔼​\[Ai∣Zi,Ri​Vi=1\]=μ0​\(Zi\)\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid Z\_\{i\},R\_\{i\}=1\]=\\mathbb\{E\}\[A\_\{i\}\\mid Z\_\{i\},R\_\{i\}V\_\{i\}=1\]=\\mu\_\{0\}\(Z\_\{i\}\)\. Composing,𝔼​\[μ0​\(Zi\)∣Bi,Ri=1\]=𝔼​\[Yi⋆∣Bi,Ri=1\]=𝔼​\[Yi⋆∣Bi\]=g0​\(Bi\)\\mathbb\{E\}\[\\mu\_\{0\}\(Z\_\{i\}\)\\mid B\_\{i\},R\_\{i\}=1\]=\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid B\_\{i\},R\_\{i\}=1\]=\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid B\_\{i\}\]=g\_\{0\}\(B\_\{i\}\)\.

### Design\-validity ofY~i\\tilde\{Y\}\_\{i\}\.

Fix arbitrary nuisance functionsη=\(μ,g\)\\eta=\(\\mu,g\)\. For audited units define

Mi​\(μ\)=μ​\(Zi\)\+Viρ​\(Zi\)​\{Ai−μ​\(Zi\)\}\.M\_\{i\}\(\\mu\)=\\mu\(Z\_\{i\}\)\+\\frac\{V\_\{i\}\}\{\\rho\(Z\_\{i\}\)\}\\\{A\_\{i\}\-\\mu\(Z\_\{i\}\)\\\}\.When the nuisances are estimated by cross\-fitting, the following argument is understood conditionally on the training folds, so thatμ\\muandggare fixed functions for the evaluation fold\.

First consider the inner correction\. Conditional onRi=1R\_\{i\}=1andZiZ\_\{i\},

E​\{Mi​\(μ\)∣Ri=1,Zi\}\\displaystyle E\\\{M\_\{i\}\(\\mu\)\\mid R\_\{i\}=1,Z\_\{i\}\\\}=μ​\(Zi\)\+E​\[Viρ​\(Zi\)​\{Ai−μ​\(Zi\)\}∣Ri=1,Zi\]\\displaystyle=\\mu\(Z\_\{i\}\)\+E\\left\[\\frac\{V\_\{i\}\}\{\\rho\(Z\_\{i\}\)\}\\\{A\_\{i\}\-\\mu\(Z\_\{i\}\)\\\}\\mid R\_\{i\}=1,Z\_\{i\}\\right\]=μ​\(Zi\)\+E​\{Ai−μ​\(Zi\)∣Ri=1,Zi,Vi=1\}\\displaystyle=\\mu\(Z\_\{i\}\)\+E\\\{A\_\{i\}\-\\mu\(Z\_\{i\}\)\\mid R\_\{i\}=1,Z\_\{i\},V\_\{i\}=1\\\}=E​\(Ai∣Ri=1,Zi,Vi=1\)\.\\displaystyle=E\(A\_\{i\}\\mid R\_\{i\}=1,Z\_\{i\},V\_\{i\}=1\)\.By adjudication correctness,Ai=Yi⋆A\_\{i\}=Y\_\{i\}^\{\\star\}on\{Ri​Vi=1\}\\\{R\_\{i\}V\_\{i\}=1\\\}, and by adjudication ignorability,

E​\(Ai∣Ri=1,Zi,Vi=1\)=E​\(Yi⋆∣Ri=1,Zi\)\.E\(A\_\{i\}\\mid R\_\{i\}=1,Z\_\{i\},V\_\{i\}=1\)=E\(Y\_\{i\}^\{\\star\}\\mid R\_\{i\}=1,Z\_\{i\}\)\.Hence

E​\{Mi​\(μ\)∣Ri=1,Zi\}=E​\(Yi⋆∣Ri=1,Zi\)\.E\\\{M\_\{i\}\(\\mu\)\\mid R\_\{i\}=1,Z\_\{i\}\\\}=E\(Y\_\{i\}^\{\\star\}\\mid R\_\{i\}=1,Z\_\{i\}\)\.
No restriction on the joint distribution of\(Gi,1,Gi,2\)\(G\_\{i,1\},G\_\{i,2\}\)is used here: once we condition onZiZ\_\{i\}, the coder labels are simply observed covariates, and all coder dependence or bias is part of the conditional meanμ0​\(Zi\)\\mu\_\{0\}\(Z\_\{i\}\)\.

Now consider the outer correction\. Conditional onBiB\_\{i\},

E​\{Y~i​\(η\)∣Bi\}\\displaystyle E\\\{\\tilde\{Y\}\_\{i\}\(\\eta\)\\mid B\_\{i\}\\\}=g​\(Bi\)\+E​\[Riπ​\(Bi\)​\{Mi​\(μ\)−g​\(Bi\)\}∣Bi\]\\displaystyle=g\(B\_\{i\}\)\+E\\left\[\\frac\{R\_\{i\}\}\{\\pi\(B\_\{i\}\)\}\\\{M\_\{i\}\(\\mu\)\-g\(B\_\{i\}\)\\\}\\mid B\_\{i\}\\right\]=g​\(Bi\)\+E​\{Mi​\(μ\)−g​\(Bi\)∣Ri=1,Bi\}\\displaystyle=g\(B\_\{i\}\)\+E\\\{M\_\{i\}\(\\mu\)\-g\(B\_\{i\}\)\\mid R\_\{i\}=1,B\_\{i\}\\\}=E​\{Mi​\(μ\)∣Ri=1,Bi\}\.\\displaystyle=E\\\{M\_\{i\}\(\\mu\)\\mid R\_\{i\}=1,B\_\{i\}\\\}\.Using iterated expectations and the inner result,

E​\{Mi​\(μ\)∣Ri=1,Bi\}\\displaystyle E\\\{M\_\{i\}\(\\mu\)\\mid R\_\{i\}=1,B\_\{i\}\\\}=E​\[E​\{Mi​\(μ\)∣Ri=1,Zi\}∣Ri=1,Bi\]\\displaystyle=E\\left\[E\\\{M\_\{i\}\(\\mu\)\\mid R\_\{i\}=1,Z\_\{i\}\\\}\\mid R\_\{i\}=1,B\_\{i\}\\right\]=E​\[E​\(Yi⋆∣Ri=1,Zi\)∣Ri=1,Bi\]\\displaystyle=E\\left\[E\(Y\_\{i\}^\{\\star\}\\mid R\_\{i\}=1,Z\_\{i\}\)\\mid R\_\{i\}=1,B\_\{i\}\\right\]=E​\(Yi⋆∣Ri=1,Bi\)\.\\displaystyle=E\(Y\_\{i\}^\{\\star\}\\mid R\_\{i\}=1,B\_\{i\}\)\.Finally, by audit ignorability,

E​\(Yi⋆∣Ri=1,Bi\)=E​\(Yi⋆∣Bi\)\.E\(Y\_\{i\}^\{\\star\}\\mid R\_\{i\}=1,B\_\{i\}\)=E\(Y\_\{i\}^\{\\star\}\\mid B\_\{i\}\)\.Therefore

E​\{Y~i​\(η\)∣Bi\}=E​\(Yi⋆∣Bi\)\.E\\\{\\tilde\{Y\}\_\{i\}\(\\eta\)\\mid B\_\{i\}\\\}=E\(Y\_\{i\}^\{\\star\}\\mid B\_\{i\}\)\.SinceXiX\_\{i\}is measurable with respect toBiB\_\{i\},

E​\[Xi​\{Y~i​\(η\)−expit⁡\(Xi⊤​β\)\}\]=E​\[Xi​\{Yi⋆−expit⁡\(Xi⊤​β\)\}\]\.E\\left\[X\_\{i\}\\\{\\tilde\{Y\}\_\{i\}\(\\eta\)\-\\operatorname\{expit\}\(X\_\{i\}^\{\\top\}\\beta\)\\\}\\right\]=E\\left\[X\_\{i\}\\\{Y\_\{i\}^\{\\star\}\-\\operatorname\{expit\}\(X\_\{i\}^\{\\top\}\\beta\)\\\}\\right\]\.Atβ=β0\\beta=\\beta\_\{0\}, the right\-hand side is zero by definition ofβ0\\beta\_\{0\}\. ThusE​\{m​\(Wi;β0,η\)\}=0E\\\{m\(W\_\{i\};\\beta\_\{0\},\\eta\)\\\}=0for every fixed nuisance pairη\\eta\. This fixed\-nuisance statement establishes design\-unbiasedness of the score, but it does not imply that arbitrary inconsistent nuisance estimators have the same first\-order expansion\. The influence\-function representation below is obtained by expanding aroundη0=\(μ0,g0\)\\eta\_\{0\}=\(\\mu\_\{0\},g\_\{0\}\)and therefore requires the stated cross\-fit nuisance convergence conditions\.

### Influence function\.

The cross\-fit estimatorψ^\\hat\{\\psi\}admits the asymptotic linear expansionN​\(ψ^−ψ\)=N−1/2​∑iφ​\(Wi;μ0,g0\)\+op​\(1\)\\sqrt\{N\}\(\\hat\{\\psi\}\-\\psi\)=N^\{\-1/2\}\\sum\_\{i\}\\varphi\(W\_\{i\};\\mu\_\{0\},g\_\{0\}\)\+o\_\{p\}\(1\)with influence function

φ​\(Wi;μ0,g0\)=g0​\(Bi\)−ψ\+Riπi​\[μ0​\(Zi\)−g0​\(Bi\)\]\+Ri​Viπi​ρi​\[Ai−μ0​\(Zi\)\]\.\\varphi\(W\_\{i\};\\mu\_\{0\},g\_\{0\}\)=g\_\{0\}\(B\_\{i\}\)\-\\psi\+\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\bigl\[\\mu\_\{0\}\(Z\_\{i\}\)\-g\_\{0\}\(B\_\{i\}\)\\bigr\]\+\\frac\{R\_\{i\}V\_\{i\}\}\{\\pi\_\{i\}\\rho\_\{i\}\}\\bigl\[A\_\{i\}\-\\mu\_\{0\}\(Z\_\{i\}\)\\bigr\]\.\(7\)The estimatorβ^\\hat\{\\beta\}from[Eq\.˜6](https://arxiv.org/html/2607.15455#S3.E6)has influence function

𝐇−1​𝑿i​\(Y~i​\(η0\)−expit⁡\(𝑿i⊤​β0\)\),𝐇=𝔼​\[𝑿i​𝑿i⊤​expit′⁡\(𝑿i⊤​β0\)\],\\mathbf\{H\}^\{\-1\}\\,\\bm\{X\}\_\{i\}\\,\\bigl\(\\widetilde\{Y\}\_\{i\}\(\\eta\_\{0\}\)\-\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\beta\_\{0\}\)\\bigr\),\\qquad\\mathbf\{H\}=\\mathbb\{E\}\[\\bm\{X\}\_\{i\}\\bm\{X\}\_\{i\}^\{\\top\}\\,\\operatorname\{expit\}^\{\\prime\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\beta\_\{0\}\)\],\(8\)obtained directly from the M\-estimator scorem​\(Wi;β0,η0\)m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\); the prevalence\-targetφ\\varphiis not the right object forβ^\\hat\{\\beta\}because the logistic score is not the prevalence influence function\.

### Why no product\-rate condition is required\.

Letδg​\(B\)=g^​\(B\)−g0​\(B\)\\delta\_\{g\}\(B\)=\\hat\{g\}\(B\)\-g\_\{0\}\(B\)andδμ​\(Z\)=μ^​\(Z\)−μ0​\(Z\)\\delta\_\{\\mu\}\(Z\)=\\hat\{\\mu\}\(Z\)\-\\mu\_\{0\}\(Z\)\. Direct expansion ofY~i\\widetilde\{Y\}\_\{i\}gives

Y~i​\(η^\)−Y~i​\(η0\)=\(1−Riπi\)​δg​\(Bi\)\+Riπi​\(1−Viρi\)​δμ​\(Zi\),\\widetilde\{Y\}\_\{i\}\(\\hat\{\\eta\}\)\-\\widetilde\{Y\}\_\{i\}\(\\eta\_\{0\}\)=\\Bigl\(1\-\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\Bigr\)\\delta\_\{g\}\(B\_\{i\}\)\+\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\Bigl\(1\-\\frac\{V\_\{i\}\}\{\\rho\_\{i\}\}\\Bigr\)\\delta\_\{\\mu\}\(Z\_\{i\}\),\(9\)so thatm​\(Wi;β0,η^\)−m​\(Wi;β0,η0\)=𝑿i⋅\[r\.h\.s\. of[Eq\.˜9](https://arxiv.org/html/2607.15455#A2.E9)\]m\(W\_\{i\};\\beta\_\{0\},\\hat\{\\eta\}\)\-m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)=\\bm\{X\}\_\{i\}\\cdot\[\\text\{r\.h\.s\.\\ of \\lx@cref\{creftype~refnum\}\{eq:nuis\-perturb\}\}\]\. The two nuisance errors enter linearly, but each is multiplied by a known\-design, conditionally mean\-zero sampling residual\. This is why the argument requires onlyL2L^\{2\}consistency of the nuisance estimators, rather than a product\-rate condition\.

### Leakage\-free nested cross\-fitting\.

We first analyze a leakage\-free nested version of the estimator\. For a held\-out evaluation foldℱk\\mathcal\{F\}\_\{k\}, the inner nuisance used to construct audit pseudo\-labels for the outer training sample is fit without using observations fromℱk\\mathcal\{F\}\_\{k\}\. The outer nuisance used to evaluateℱk\\mathcal\{F\}\_\{k\}is therefore trained on pseudo\-labels that contain no information from the held\-out fold\. Conditional on the training data for that fold, the nuisance functions are fixed with respect to the evaluation observations\.

For such an evaluation observation, the conditional means of both bracketed factors in[Eq\.˜9](https://arxiv.org/html/2607.15455#A2.E9)are zero:

𝔼\[1−Riπi\|Bi\]=0\\mathbb\{E\}\\\!\\left\[1\-\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\,\\middle\|\\,B\_\{i\}\\right\]=0by[Assumption˜1](https://arxiv.org/html/2607.15455#Thmassumption1), and

𝔼\[Riπi\(1−Viρi\)\|Zi,Bi\]=0\\mathbb\{E\}\\\!\\left\[\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\Bigl\(1\-\\frac\{V\_\{i\}\}\{\\rho\_\{i\}\}\\Bigr\)\\,\\middle\|\\,Z\_\{i\},B\_\{i\}\\right\]=0by[Assumption˜2](https://arxiv.org/html/2607.15455#Thmassumption2); the inner factor has conditional mean zero givenRi=1R\_\{i\}=1andZiZ\_\{i\}\. Hence

𝔼​\[m​\(Wi;β0,η^\)−m​\(Wi;β0,η0\)\|training data\]=𝟎\.\\mathbb\{E\}\\\!\\Bigl\[m\(W\_\{i\};\\beta\_\{0\},\\hat\{\\eta\}\)\-m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\,\\Big\|\\,\\text\{training data\}\\Bigr\]=\\mathbf\{0\}\.\(10\)Thus the nuisance error contributes only a centered empirical\-process remainder\.

Becauseπi\\pi\_\{i\}andρi\\rho\_\{i\}are bounded away from zero, and𝔼​‖𝑿i‖2\+δ<∞\\mathbb\{E\}\\\|\\bm\{X\}\_\{i\}\\\|^\{2\+\\delta\}<\\infty, there is a finite constantCCsuch that

𝔼​\[‖m​\(Wi;β0,η^\)−m​\(Wi;β0,η0\)‖2\|training data\]≤C​\(‖g^−g0‖2,X,B2\+‖μ^−μ0‖2,X,Z2\),\\mathbb\{E\}\\\!\\Bigl\[\\bigl\\\|m\(W\_\{i\};\\beta\_\{0\},\\hat\{\\eta\}\)\-m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\bigr\\\|^\{2\}\\,\\Big\|\\,\\text\{training data\}\\Bigr\]\\leq C\\left\(\\\|\\hat\{g\}\-g\_\{0\}\\\|\_\{2,X,B\}^\{2\}\+\\\|\\hat\{\\mu\}\-\\mu\_\{0\}\\\|\_\{2,X,Z\}^\{2\}\\right\),\(11\)where

‖h‖2,X,B2=𝔼​\[‖𝑿i‖2​h​\(Bi\)2\],‖r‖2,X,Z2=𝔼​\[‖𝑿i‖2​r​\(Zi\)2\]\.\\\|h\\\|\_\{2,X,B\}^\{2\}=\\mathbb\{E\}\\\!\\left\[\\\|\\bm\{X\}\_\{i\}\\\|^\{2\}h\(B\_\{i\}\)^\{2\}\\right\],\\qquad\\\|r\\\|\_\{2,X,Z\}^\{2\}=\\mathbb\{E\}\\\!\\left\[\\\|\\bm\{X\}\_\{i\}\\\|^\{2\}r\(Z\_\{i\}\)^\{2\}\\right\]\.Therefore,

N​\(PN−P\)​\{m​\(W;β0,η^\)−m​\(W;β0,η0\)\}=Op​\(‖g^−g0‖2,X,B\+‖μ^−μ0‖2,X,Z\)=op​\(1\),\\sqrt\{N\}\(P\_\{N\}\-P\)\\\{m\(W;\\beta\_\{0\},\\hat\{\\eta\}\)\-m\(W;\\beta\_\{0\},\\eta\_\{0\}\)\\\}=O\_\{p\}\\\!\\left\(\\\|\\hat\{g\}\-g\_\{0\}\\\|\_\{2,X,B\}\+\\\|\\hat\{\\mu\}\-\\mu\_\{0\}\\\|\_\{2,X,Z\}\\right\)=o\_\{p\}\(1\),under cross\-fitXX\-weightedL2L^\{2\}\-consistent nuisance estimation\.

Combining this centered empirical\-process bound with the fixed\-nuisance design\-unbiasedness result above and a standard mean\-value expansion inβ\\betaaroundβ0\\beta\_\{0\}gives the asymptotic linear expansion in[Proposition˜1](https://arxiv.org/html/2607.15455#Thmproposition1)for the leakage\-free nested estimator\. No product\-rate condition appears because the known design weights make the linear nuisance remainder conditionally mean\-zero\.

### Shared\-partition cross\-fitting\.

Algorithm[1](https://arxiv.org/html/2607.15455#algorithm1)uses a computationally simpler shared\-partition implementation, in which the same folds are used in the inner and outer passes\. For foldℱk\\mathcal\{F\}\_\{k\}, writeg^sp\(−k\)\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}for the outer nuisance actually used by the algorithm\. Letg^lf\(−k\)\\hat\{g\}\_\{\\mathrm\{lf\}\}^\{\(\-k\)\}denote the leakage\-free counterpart: the outer nuisance that would be obtained by constructing each training pseudo\-labelM^j\\widehat\{M\}\_\{j\},j∉ℱkj\\notin\\mathcal\{F\}\_\{k\}, using an inner fit that excludes bothℱk\\mathcal\{F\}\_\{k\}andℱfold​\(j\)\\mathcal\{F\}\_\{\\mathrm\{fold\}\(j\)\}\.

The shared\-partition estimator has the same first\-order expansion as the leakage\-free estimator provided the leakage remainder satisfies

1N​∑k=1K∑i∈ℱkXi​\(1−Riπi\)​\{g^sp\(−k\)​\(Bi\)−g^lf\(−k\)​\(Bi\)\}=op​\(1\)\.\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{F\}\_\{k\}\}X\_\{i\}\\Bigl\(1\-\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\Bigr\)\\bigl\\\{\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}\(B\_\{i\}\)\-\\hat\{g\}\_\{\\mathrm\{lf\}\}^\{\(\-k\)\}\(B\_\{i\}\)\\bigr\\\}=o\_\{p\}\(1\)\.\(SP\)To see this, add and subtract the leakage\-free score:

1N​∑k=1K∑i∈ℱk\[m​\(Wi;β0,μ^\(−k\),g^sp\(−k\)\)−m​\(Wi;β0,η0\)\]\\displaystyle\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{F\}\_\{k\}\}\\left\[m\\bigl\(W\_\{i\};\\beta\_\{0\},\\hat\{\\mu\}^\{\(\-k\)\},\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}\\bigr\)\-m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\right\]=1N​∑k=1K∑i∈ℱk\[m​\(Wi;β0,μ^\(−k\),g^lf\(−k\)\)−m​\(Wi;β0,η0\)\]\\displaystyle\\quad=\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{F\}\_\{k\}\}\\left\[m\\bigl\(W\_\{i\};\\beta\_\{0\},\\hat\{\\mu\}^\{\(\-k\)\},\\hat\{g\}\_\{\\mathrm\{lf\}\}^\{\(\-k\)\}\\bigr\)\-m\(W\_\{i\};\\beta\_\{0\},\\eta\_\{0\}\)\\right\]\+1N​∑k=1K∑i∈ℱkXi​\(1−Riπi\)​\{g^sp\(−k\)​\(Bi\)−g^lf\(−k\)​\(Bi\)\}\.\\displaystyle\\qquad\+\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{F\}\_\{k\}\}X\_\{i\}\\Bigl\(1\-\\frac\{R\_\{i\}\}\{\\pi\_\{i\}\}\\Bigr\)\\bigl\\\{\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}\(B\_\{i\}\)\-\\hat\{g\}\_\{\\mathrm\{lf\}\}^\{\(\-k\)\}\(B\_\{i\}\)\\bigr\\\}\.\(12\)The first term isop​\(1\)o\_\{p\}\(1\)by the leakage\-free argument\. The second term isop​\(1\)o\_\{p\}\(1\)by condition \([SP](https://arxiv.org/html/2607.15455#A2.Ex20)\)\. Hence the shared\-partition estimator has the same asymptotic linear representation as the leakage\-free estimator whenever condition \([SP](https://arxiv.org/html/2607.15455#A2.Ex20)\) holds\.

### Discussion of the shared\-partition stability condition\.

Condition \([SP](https://arxiv.org/html/2607.15455#A2.Ex20)\) is a stability condition on the indirect leakage created by reusing the same fold partition in the inner and outer stages\. It is expected to hold when the inner learner is stable under adding or removing one fold, the outer learner is Lipschitz\-stable in its training responses, and the resulting difference

‖g^sp\(−k\)−g^lf\(−k\)‖2,X=op​\(1\)\\\|\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}\-\\hat\{g\}\_\{\\mathrm\{lf\}\}^\{\(\-k\)\}\\\|\_\{2,X\}=o\_\{p\}\(1\)is sufficiently small that the weighted empirical average in \([SP](https://arxiv.org/html/2607.15455#A2.Ex20)\) isop​\(1\)o\_\{p\}\(1\)\. This is plausible for stable nuisance learners in fixed\-dimensional smooth classes, such as the fixed\-hyperparameterL2L^\{2\}\-regularized logistic regressions used in the experiments, under standard nonsingularity and bounded\-moment conditions\. A formal verification requires learner\-specific deletion stability controlling the effect of excludingℱk\\mathcal\{F\}\_\{k\}from the inner fits on the resulting outer nuisance\. The condition may fail for highly adaptive nuisance procedures, including tuning, stacking, or model selection performed inside the cross\-fitting loop, because such procedures can transmit evaluation\-fold information intog^sp\(−k\)\\hat\{g\}\_\{\\mathrm\{sp\}\}^\{\(\-k\)\}\.

### Variance decomposition\.

Let

H​\(Z,B\)=μ0​\(Z\)−g0​\(B\),ε=A−μ0​\(Z\),H\(Z,B\)\\;=\\;\\mu\_\{0\}\(Z\)\-g\_\{0\}\(B\),\\qquad\\varepsilon\\;=\\;A\-\\mu\_\{0\}\(Z\),\(13\)so thatφ​\(W;μ0,g0\)=\(g0​\(B\)−ψ\)\+Rπ​\(B\)​H​\(Z,B\)\+R​Vπ​\(B\)​ρ​\(Z\)​ε\\varphi\(W;\\mu\_\{0\},g\_\{0\}\)=\\bigl\(g\_\{0\}\(B\)\-\\psi\\bigr\)\+\\tfrac\{R\}\{\\pi\(B\)\}H\(Z,B\)\+\\tfrac\{RV\}\{\\pi\(B\)\\,\\rho\(Z\)\}\\varepsilon\. Because𝔼​\[H​\(Z,B\)∣B,R=1\]=0\\mathbb\{E\}\[H\(Z,B\)\\mid B,R=1\]=0\(by definition ofg0g\_\{0\}\) and𝔼​\[ε∣Z,R=1,V=1\]=0\\mathbb\{E\}\[\\varepsilon\\mid Z,R=1,V=1\]=0\(by definition ofμ0\\mu\_\{0\}\), the three summands are mutually orthogonal martingale increments\. Iterated expectations give𝔼​\[\(R/π​\(B\)\)2​H​\(Z,B\)2\]=𝔼​\[H​\(Z,B\)2/π​\(B\)\]\\mathbb\{E\}\[\(R/\\pi\(B\)\)^\{2\}H\(Z,B\)^\{2\}\]=\\mathbb\{E\}\[H\(Z,B\)^\{2\}/\\pi\(B\)\]and𝔼​\[\(R​V/\(π​\(B\)​ρ​\(Z\)\)\)2​ε2\]=𝔼​\[σA2​\(Z\)/\(π​\(B\)​ρ​\(Z\)\)\]\\mathbb\{E\}\[\(RV/\(\\pi\(B\)\\rho\(Z\)\)\)^\{2\}\\varepsilon^\{2\}\]=\\mathbb\{E\}\[\\sigma\_\{A\}^\{2\}\(Z\)/\(\\pi\(B\)\\,\\rho\(Z\)\)\]whereσA2​\(Z\)=Var​\(A∣Z,R=1,V=1\)\\sigma\_\{A\}^\{2\}\(Z\)=\\mathrm\{Var\}\(A\\mid Z,R=1,V=1\), so

Var​\[φ\]=Var​\[g0​\(B\)\]\+𝔼​\[H​\(Z,B\)2π​\(B\)\]\+𝔼​\[σA2​\(Z\)π​\(B\)​ρ​\(Z\)\]\.\\mathrm\{Var\}\[\\varphi\]\\;=\\;\\mathrm\{Var\}\[g\_\{0\}\(B\)\]\\;\+\\;\\mathbb\{E\}\\\!\\left\[\\frac\{H\(Z,B\)^\{2\}\}\{\\pi\(B\)\}\\right\]\\;\+\\;\\mathbb\{E\}\\\!\\left\[\\frac\{\\sigma\_\{A\}^\{2\}\(Z\)\}\{\\pi\(B\)\\,\\rho\(Z\)\}\\right\]\.\(14\)For constantπ​\(B\)≡π\\pi\(B\)\\equiv\\piandρ​\(Z\)≡ρ\\rho\(Z\)\\equiv\\rho,Var​\[φ\]=Var​\[g0​\(B\)\]\+π−1​𝔼​\[\(μ0​\(Z\)−g0​\(B\)\)2\]\+\(π​ρ\)−1​𝔼​\[σA2​\(Z\)\]\\mathrm\{Var\}\[\\varphi\]=\\mathrm\{Var\}\[g\_\{0\}\(B\)\]\+\\pi^\{\-1\}\\mathbb\{E\}\[\(\\mu\_\{0\}\(Z\)\-g\_\{0\}\(B\)\)^\{2\}\]\+\(\\pi\\rho\)^\{\-1\}\\mathbb\{E\}\[\\sigma\_\{A\}^\{2\}\(Z\)\]\. Atπ=ρ=1\\pi=\\rho=1this collapses toVar​\(A\)\\mathrm\{Var\}\(A\)via the law of total variance, as it must\.

### Efficiency gain overDSL\-AdjOnly\.

The adjudication\-only DSL pseudo\-outcome treatsRi​ViR\_\{i\}V\_\{i\}as the sampling indicator and uses the joint design weight1/\(π​\(Bi\)​ρ​\(Zi\)\)1/\(\\pi\(B\_\{i\}\)\\,\\rho\(Z\_\{i\}\)\)to debias the outer regressiong^\\hat\{g\}, giving influence functionφadj=\(g0​\(B\)−ψ\)\+\(R​V/\(π​\(B\)​ρ​\(Z\)\)\)​\(A−g0​\(B\)\)\\varphi\_\{\\mathrm\{adj\}\}=\(g\_\{0\}\(B\)\-\\psi\)\+\(RV/\(\\pi\(B\)\\,\\rho\(Z\)\)\)\(A\-g\_\{0\}\(B\)\)— structurally a single\-stage AIPW with the joint sampling indicator\. DecomposingA−g0​\(B\)=H​\(Z,B\)\+εA\-g\_\{0\}\(B\)=H\(Z,B\)\+\\varepsilonand applying the same orthogonality argument,

Var​\[φadj\]=Var​\[g0​\(B\)\]\+𝔼​\[H​\(Z,B\)2π​\(B\)​ρ​\(Z\)\]\+𝔼​\[σA2​\(Z\)π​\(B\)​ρ​\(Z\)\]\.\\mathrm\{Var\}\[\\varphi\_\{\\mathrm\{adj\}\}\]\\;=\\;\\mathrm\{Var\}\[g\_\{0\}\(B\)\]\\;\+\\;\\mathbb\{E\}\\\!\\left\[\\frac\{H\(Z,B\)^\{2\}\}\{\\pi\(B\)\\,\\rho\(Z\)\}\\right\]\\;\+\\;\\mathbb\{E\}\\\!\\left\[\\frac\{\\sigma\_\{A\}^\{2\}\(Z\)\}\{\\pi\(B\)\\,\\rho\(Z\)\}\\right\]\.\(15\)Subtracting[Eq\.˜14](https://arxiv.org/html/2607.15455#A2.E14)from[Eq\.˜15](https://arxiv.org/html/2607.15455#A2.E15)gives

Var​\[φadj\]−Var​\[φ\]=𝔼​\[1−ρ​\(Z\)π​\(B\)​ρ​\(Z\)​\(μ0​\(Z\)−g0​\(B\)\)2\]≥0,\\mathrm\{Var\}\[\\varphi\_\{\\mathrm\{adj\}\}\]\-\\mathrm\{Var\}\[\\varphi\]\\;=\\;\\mathbb\{E\}\\\!\\left\[\\frac\{1\-\\rho\(Z\)\}\{\\pi\(B\)\\,\\rho\(Z\)\}\\,\\bigl\(\\mu\_\{0\}\(Z\)\-g\_\{0\}\(B\)\\bigr\)^\{2\}\\right\]\\;\\geq\\;0,\(16\)which simplifies to\(\(1−ρ\)/\(π​ρ\)\)​𝔼​\[\(μ0​\(Z\)−g0​\(B\)\)2\]\(\(1\-\\rho\)/\(\\pi\\rho\)\)\\,\\mathbb\{E\}\[\(\\mu\_\{0\}\(Z\)\-g\_\{0\}\(B\)\)^\{2\}\]for constantπ,ρ\\pi,\\rho\.

For the logistic\-regression target in the main text, the same comparison applies to the covariance of the score influence functionXi​\{Y~i​\(η0\)−expit⁡\(Xi⊤​β0\)\}X\_\{i\}\\\{\\widetilde\{Y\}\_\{i\}\(\\eta\_\{0\}\)\-\\operatorname\{expit\}\(X\_\{i\}^\{\\top\}\\beta\_\{0\}\)\\\}\. In particular, the corresponding difference in score covariance matrices is

Ωadj−Ω=𝔼​\[Xi​Xi⊤​1−ρ​\(Zi\)π​\(Bi\)​ρ​\(Zi\)​\(μ0​\(Zi\)−g0​\(Bi\)\)2\],\\Omega\_\{\\mathrm\{adj\}\}\-\\Omega\\;=\\;\\mathbb\{E\}\\\!\\left\[X\_\{i\}X\_\{i\}^\{\\top\}\\frac\{1\-\\rho\(Z\_\{i\}\)\}\{\\pi\(B\_\{i\}\)\\,\\rho\(Z\_\{i\}\)\}\\,\\bigl\(\\mu\_\{0\}\(Z\_\{i\}\)\-g\_\{0\}\(B\_\{i\}\)\\bigr\)^\{2\}\\right\],\(17\)which is positive semidefinite\. Applying the usual M\-estimation transformation, the corresponding asymptotic covariance difference forβ^\\hat\{\\beta\}is

H−1​\(Ωadj−Ω\)​H−1⊤,H^\{\-1\}\(\\Omega\_\{\\mathrm\{adj\}\}\-\\Omega\)H^\{\-1\\top\},also positive semidefinite\. Thus the scalar comparison in[Eq\.˜16](https://arxiv.org/html/2607.15455#A2.E16)has the same interpretation for the main logistic\-regression estimand:PA\-DSLremoves the extraρ​\(Zi\)−1\\rho\(Z\_\{i\}\)^\{\-1\}inflation from the component of adjudicated truth explained by audit\-tier features beyondBiB\_\{i\}\.

ThePA\-DSLestimator does*not*reduce the residual adjudication noise termσA2​\(Z\)\\sigma\_\{A\}^\{2\}\(Z\), which can only be learned from adjudicated units and remains weighted by1/\(π​\(B\)​ρ​\(Z\)\)1/\(\\pi\(B\)\\rho\(Z\)\)in both estimators\. WhatPA\-DSLdoes is move theZZ\-explainable componentμ0​\(Z\)−g0​\(B\)\\mu\_\{0\}\(Z\)\-g\_\{0\}\(B\)from the sparse adjudication weight1/\(π​\(B\)​ρ​\(Z\)\)1/\(\\pi\(B\)\\rho\(Z\)\)to the larger audit weight1/π​\(B\)1/\\pi\(B\)\. The gain in[Eq\.˜16](https://arxiv.org/html/2607.15455#A2.E16)is therefore large precisely when \(i\) adjudication is sparse, soρ​\(Z\)\\rho\(Z\)is small, and \(ii\) the audit\-level featuresZZexplain substantial variation in the adjudicated label, and collapses to zero whenZZhas no incremental predictive power overBB\.

## Appendix CWhy build on DSL rather than PPI?

Prediction Powered Inference \(PPI\) is a closely related prediction\-assisted inference framework\[Angelopouloset al\.,[2023](https://arxiv.org/html/2607.15455#bib.bib15)\]\. We build on DSL because our measurement process is explicitly design\-based: the audit and adjudication samples are drawn with known probabilities chosen by the analyst\. The DSL pseudo\-outcome is therefore the natural object to extend, and partial adjudication is handled by nesting a second known\-weight correction inside the usual audit correction\. This is not a claim that PPI is inappropriate; rather, deriving a partially adjudicated PPI analogue is a separate extension\. Recent benchmarking finds that DSL can be competitive with, and often more efficient than, PPI for LLM\-based parameter estimation, though performance varies by setting\[de Pieuchonet al\.,[2025](https://arxiv.org/html/2607.15455#bib.bib30)\]\.

## Appendix DImplementation Details

All nuisances are L2\-regularized logistic regression unless otherwise noted \(C=1C=1, max\-iter20002000\); predictions are clipped to\[10−3,1−10−3\]\[10^\{\-3\},1\-10^\{\-3\}\]to bound the importance weights in[Eq\.˜5](https://arxiv.org/html/2607.15455#S3.E5)\. Cross\-fitting usesK=5K=5folds, stratified on twentyQQ\-quintile×\\timesBdiffB\_\{\\mathrm\{diff\}\}\-quartile strata whereBdiff=1−2​\|p⋆−0\.5\|B\_\{\\mathrm\{diff\}\}=1\-2\|p^\{\\star\}\-0\.5\|is the boundary statistic\.β\\betais fit by damped iteratively reweighted least squares with a10−610^\{\-6\}ridge for numerical stability; standard errors use the empirical sandwich on the cross\-fit pseudo\-outcome\.

The estimators evaluated in the main text and appendices are:Surrogate\-only,Human\-naive,DSL\-MajVote\(two coders, ties randomly broken\),DSL\-AdjOnly,DSL\-oracle,PA\-DSL\(logistic, baseline inner features\),PA\-DSL\-coder\(logistic, coder\-pair features\),PA\-DSL\-IPW\-only\(innerμ^≡0\\hat\{\\mu\}\\equiv 0; pure Horvitz–Thompson at the inner stage\), andPA\-DSL\-Reg\-only\(noV/ρV/\\rhodebias at the inner stage; pure regression\)\.

## Appendix ESynthetic Data\-Generating Process

Each replication generatesN=50,000N=50\{,\}000units\. Covariates𝑿i\\bm\{X\}\_\{i\}comprise an intercept andp=5p=5continuous features drawn from a multivariate normal with unit variances and equicorrelationρcov=0\.2\\rho\_\{\\mathrm\{cov\}\}=0\.2\. Latent labels areYi⋆∣𝑿i∼Bernoulli​\(expit⁡\(𝑿i⊤​𝜷\)\)Y^\{\\star\}\_\{i\}\\mid\\bm\{X\}\_\{i\}\\sim\\text\{Bernoulli\}\(\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\bm\{\\beta\}\)\), with𝜷=\(−0\.4,1\.0,−0\.8,0\.6,0\.0,0\.0\)⊤\\bm\{\\beta\}=\(\-0\.4,\\,1\.0,\\,\-0\.8,\\,0\.6,\\,0\.0,\\,0\.0\)^\{\\top\}\. The boundary statistic isBi,diff=1−2​\|expit⁡\(𝑿i⊤​𝜷\)−0\.5\|B\_\{i,\\mathrm\{diff\}\}=1\-2\|\\operatorname\{expit\}\(\\bm\{X\}\_\{i\}^\{\\top\}\\bm\{\\beta\}\)\-0\.5\|\.

Three latent difficulty channels are

Di\\displaystyle D\_\{i\}=δ0\+δ1​Bi,diff\+δ2​\|Xi,1\|\+δ3​\|Xi,2\|\+Ui,Ui∼N​\(0,σD2\),\\displaystyle=\\delta\_\{0\}\+\\delta\_\{1\}B\_\{i,\\mathrm\{diff\}\}\+\\delta\_\{2\}\|X\_\{i,1\}\|\+\\delta\_\{3\}\|X\_\{i,2\}\|\+U\_\{i\},\\quad U\_\{i\}\\sim N\(0,\\sigma\_\{D\}^\{2\}\),Hi\\displaystyle H\_\{i\}=η1​Xi,3\+η2​Xi,4\+ξi,ξi∼N​\(0,σH2\),\\displaystyle=\\eta\_\{1\}X\_\{i,3\}\+\\eta\_\{2\}X\_\{i,4\}\+\\xi\_\{i\},\\quad\\xi\_\{i\}\\sim N\(0,\\sigma\_\{H\}^\{2\}\),Mi\\displaystyle M\_\{i\}=ζ1​Xi,4\+ζ2​Xi,5\+νi,νi∼N​\(0,σM2\)\.\\displaystyle=\\zeta\_\{1\}X\_\{i,4\}\+\\zeta\_\{2\}X\_\{i,5\}\+\\nu\_\{i\},\\quad\\nu\_\{i\}\\sim N\(0,\\sigma\_\{M\}^\{2\}\)\.The surrogate isQi=expit⁡\(τ0\+τ1​Yi⋆\+τ2​Xi,1\+τ3​Xi,2−λD​Di−λM​Mi\+εi\)Q\_\{i\}=\\operatorname\{expit\}\(\\tau\_\{0\}\+\\tau\_\{1\}Y^\{\\star\}\_\{i\}\+\\tau\_\{2\}X\_\{i,1\}\+\\tau\_\{3\}X\_\{i,2\}\-\\lambda\_\{D\}D\_\{i\}\-\\lambda\_\{M\}M\_\{i\}\+\\varepsilon\_\{i\}\)withεi∼N​\(0,σQ2\)\\varepsilon\_\{i\}\\sim N\(0,\\sigma\_\{Q\}^\{2\}\)\. Coderjjhas sensitivity interceptαj\(1\)∼N​\(asens,σcoder2\)\\alpha^\{\(1\)\}\_\{j\}\\sim N\(a\_\{\\text\{sens\}\},\\sigma\_\{\\text\{coder\}\}^\{2\}\)and specificity interceptαj\(0\)∼N​\(aspec,σcoder2\)\\alpha^\{\(0\)\}\_\{j\}\\sim N\(a\_\{\\text\{spec\}\},\\sigma\_\{\\text\{coder\}\}^\{2\}\); conditional onYi⋆Y^\{\\star\}\_\{i\}and the difficulty channels,ℙ​\(Gi​j=1∣Yi⋆=1\)=expit⁡\(αj\(1\)−γD​Di−γH​Hi\)\\mathbb\{P\}\(G\_\{ij\}=1\\mid Y^\{\\star\}\_\{i\}=1\)=\\operatorname\{expit\}\(\\alpha^\{\(1\)\}\_\{j\}\-\\gamma\_\{D\}D\_\{i\}\-\\gamma\_\{H\}H\_\{i\}\)and the symmetric expression onYi⋆=0Y^\{\\star\}\_\{i\}=0\. The audit indicator is uniform Bernoulli\(π\\pi\); audited units draw two distinct coders without replacement\. Under uniform adjudicationρi≡ρ¯\\rho\_\{i\}\\equiv\\bar\{\\rho\}; under disagreement\-based adjudicationρi∝d0\+d1​𝟏​\{Gi,1≠Gi,2\}\\rho\_\{i\}\\propto d\_\{0\}\+d\_\{1\}\\mathbf\{1\}\\\{G\_\{i,1\}\\neq G\_\{i,2\}\\\}rescaled to meanρ¯\\bar\{\\rho\}\.

Table 4:Scenario parameters varied across the synthetic sweep\. Default values not listed:δ0=0\\delta\_\{0\}=0,δ1=1\.5\\delta\_\{1\}=1\.5,δ2=δ3=0\.4\\delta\_\{2\}=\\delta\_\{3\}=0\.4,σD=0\.75\\sigma\_\{D\}=0\.75,η1=0\.8\\eta\_\{1\}=0\.8,η2=−0\.5\\eta\_\{2\}=\-0\.5,σH=0\.5\\sigma\_\{H\}=0\.5,ζ1=0\.6\\zeta\_\{1\}=0\.6,ζ2=0\.5\\zeta\_\{2\}=0\.5,σM=0\.5\\sigma\_\{M\}=0\.5,τ0=0\\tau\_\{0\}=0,τ2=0\.3\\tau\_\{2\}=0\.3,τ3=−0\.2\\tau\_\{3\}=\-0\.2,asens=2\.4a\_\{\\text\{sens\}\}=2\.4,aspec=2\.6a\_\{\\text\{spec\}\}=2\.6,J=8J=8,d0=0\.1d\_\{0\}=0\.1,d1=1\.0d\_\{1\}=1\.0\.
## Appendix FInner\-Stage Feature Ablation

[Table˜5](https://arxiv.org/html/2607.15455#A6.T5)ablates the inner nuisance feature set \(R=200R=200replications\)\. Coder\-pair features add no detectable improvement over the baseline feature set in any scenario\. We adopt the baseline\-feature logistic variant as the defaultPA\-DSLthroughout the main text\.

Table 5:Inner\-stage feature ablation,β1\\beta\_\{1\}\(R=200R=200replications, trueβ1=1\\beta\_\{1\}=1\)\.Boldmarks the column\-best for MC SD, RMSE, and CI length among the design\-valid estimators \(DSL\-AdjOnly,PA\-DSLvariants\); coverage is a calibration diagnostic and bias is reported but not bolded\.
## Appendix GDetox: Additional Proxy Truths

[Table˜6](https://arxiv.org/html/2607.15455#A7.T6)reports the Detoxβ1\\beta\_\{1\}comparison under the two proxy\-truth constructions not shown in the main text: hard majority vote on the reference panel \(R=50R=50replications, targetβ1=−1\.50\\beta\_\{1\}=\-1\.50\) and the soft mean attack share \(R=50R=50, targetβ1=−1\.10\\beta\_\{1\}=\-1\.10\)\. The qualitative ranking is unchanged: the plug\-in baselines andDSL\-MajVotefed with noisy gold miss the proxy with zero coverage on the hard\-majority proxy;DSL\-AdjOnlyandPA\-DSLboth achieve nominal coverage;PA\-DSLreduces RMSE relative toDSL\-AdjOnlyby1313–16%16\\%\. BecauseY⋆Y^\{\\star\}is unobservable in the field, the displayed*truth*is the proxy refit on the full reference panel;DSL\-oracleis therefore an upper bound on achievable accuracy under each proxy, not an oracle for the latent attack construct\.

Table 6:Detox semi\-synthetic study,β1\\beta\_\{1\}onlogged\_inunder the two proxy\-truth constructions deferred from the main text\.Boldmarks the column\-best for MC SD, RMSE, and CI length per proxy among design\-valid estimators \(DSL\-AdjOnly,PA\-DSL\); coverage is a calibration diagnostic and bias is reported but not bolded\. The light rule separates the infeasibleDSL\-oraclebenchmark\.
## Appendix HDesign Curves: Audit and Adjudication Sweep

The three benign / realistic / hard scenarios in the main text are static design points\. To answer the practical question “*how much adjudication should an analyst buy, and how should adjudicated items be selected?*", we sweep the audit rateπ∈\{0\.05,0\.10,0\.20\}\\pi\\in\\\{0\.05,0\.10,0\.20\\\}, the mean adjudication rateρ¯∈\{0\.10,0\.25,0\.50\}\\bar\{\\rho\}\\in\\\{0\.10,0\.25,0\.50\\\}, the adjudication policy∈\{\\in\\\{uniform, disagreement\}\\\}, and an optional positivity floorρmin,agree∈\{0,0\.05\}\\rho\_\{\\min,\\text\{agree\}\}\\in\\\{0,0\.05\\\}on agreement cases \(under disagreement\-driven adjudication, units withGi,1=Gi,2G\_\{i,1\}=G\_\{i,2\}would otherwise receiveρ≈d0/\(d0\+d1\)​ρ¯\\rho\\approx d\_\{0\}/\(d\_\{0\}\+d\_\{1\}\)\\,\\bar\{\\rho\}, which can be very small\)\. Other parameters are fixed at the realistic\-scenario values \([Table˜4](https://arxiv.org/html/2607.15455#A5.T4)\); each cell usesR=200R=200replications\.

[Figure˜4](https://arxiv.org/html/2607.15455#A8.F4)reports theβ1\\beta\_\{1\}RMSE ratioRMSE​\(DSL\-AdjOnly\)/RMSE​\(PA\-DSL\)\\mathrm\{RMSE\}\(\\textsc\{DSL\-AdjOnly\}\)/\\mathrm\{RMSE\}\(\\textsc\{PA\-DSL\}\{\}\)across the sweep, with paired\-bootstrap95%95\\%CIs\. Three patterns emerge\.*First, disagreement\-driven adjudication is consistently higher in point estimate than uniform adjudication*across the grid: the disagreement panels \(bottom row\) sit roughly55–1010percentage points above the uniform panels at matched mean budget, with overlapping but offset CIs atR=200R=200replications per cell\. The mechanism is that adjudication concentrates on the cases where coder labels disagree — exactly where the innerμ^\\hat\{\\mu\}has the largest residual to debias and the most leverage to add overDSL\-AdjOnly\.*Second, the gain is largest at the moderate audit rateπ=0\.10\\pi=0\.10under disagreement*\(orange line, peak ratio1\.151\.15atρ¯=0\.10\\bar\{\\rho\}=0\.10\);π=0\.05\\pi=0\.05leaves too little inner training data, whileπ=0\.20\\pi=0\.20already buys enough adjudicated truth that the inner correction has less to add and the ratio drops toward1\.061\.06atρ¯=0\.50\\bar\{\\rho\}=0\.50\.*Third, the positivity floor on agreement cases has only small, mostly within\-CI effects*on the realistic\-scenario sweep\. Theρmin,agree=0\.05\\rho\_\{\\min,\\text\{agree\}\}=0\.05panels are nearly indistinguishable from the no\-floor panels\. The floor would matter more at scenarios with very smalld0/\(d0\+d1\)d\_\{0\}/\(d\_\{0\}\+d\_\{1\}\)ratios \(where the unfloored agreement\-caseρi\\rho\_\{i\}is close to zero\); we leave that exploration to future work\. For practical pipelines, the sweep suggests targetingπ∈\[0\.10,0\.20\]\\pi\\in\[0\.10,0\.20\]with disagreement\-driven adjudication atρ¯∈\[0\.10,0\.25\]\\bar\{\\rho\}\\in\[0\.10,0\.25\]as a sweet spot of audit cost vs\. inferential efficiency\.

![Refer to caption](https://arxiv.org/html/2607.15455v1/figures/design_curves.png)Figure 4:Design\-curve sweep on the realistic scenario\. RMSE ratioRMSE​\(DSL\-AdjOnly\)/RMSE​\(PA\-DSL\)\\mathrm\{RMSE\}\(\\textsc\{DSL\-AdjOnly\}\)/\\mathrm\{RMSE\}\(\\textsc\{PA\-DSL\}\{\}\)forβ1\\beta\_\{1\}, with95%95\\%paired\-bootstrap CIs \(R=200R=200reps per cell,10001000bootstrap draws\)\. Higher values favorPA\-DSL\. The dashed line at1\.01\.0marks the no\-gain point\.
## Appendix IInner\-Stage AIPW Ablation

PA\-DSL’s inner stage combines an outcome modelμ^​\(Z\)\\hat\{\\mu\}\(Z\)with a Horvitz–Thompson correctionV/ρV/\\rhoaround it\. We isolate the contribution of each piece by re\-running the synthetic analysis with inner\-stage variants:

- •IPW\-only\.Setμ^≡0\\hat\{\\mu\}\\equiv 0, soM^i=\(Vi/ρi\)​Ai\\widehat\{M\}\_\{i\}=\(V\_\{i\}/\\rho\_\{i\}\)\\,A\_\{i\}\. Pure Horvitz–Thompson at the inner stage\. Design\-valid \(unbiased for any nuisance choice\) but does not exploit the surrogate or coder labels at the inner stage, and is sensitive to smallρi\\rho\_\{i\}values\.
- •IPW\-only \(trunc\)\.As IPW\-only but with the truncated weightρi↦max⁡\(ρi,0\.05\)\\rho\_\{i\}\\mapsto\\max\(\\rho\_\{i\},0\.05\), capping1/ρi≤201/\\rho\_\{i\}\\leq 20\. This is a numerical sensitivity check that disentangles weight\-stability failures from fundamental design issues; truncation breaks design\-validity under the original sampling weights\.
- •Reg\-only\.Skip theV/ρV/\\rhodebias; setM^i=μ^​\(Zi\)\\widehat\{M\}\_\{i\}=\\hat\{\\mu\}\(Z\_\{i\}\)\. Pure regression\. Lower variance than IPW\-only whenμ^\\hat\{\\mu\}fits well, but the sandwich SE no longer accounts for the nuisance estimation noise, so confidence intervals can severely undercover\.

[Table˜7](https://arxiv.org/html/2607.15455#A9.T7)reports the comparison on the three synthetic scenarios atR=200R=200\.

The IPW\-only variant pays a steep variance price for not exploiting the inner features\. In benign and realistic, RMSE roughly triples; in hard the estimator becomes*numerically unstable*\. The disagreement\-weighted adjudication policy atρ¯=0\.10\\bar\{\\rho\}=0\.10produces audited units withρi\\rho\_\{i\}as low as∼0\.02\\sim\\\!0\.02\(1/ρi≈451/\\rho\_\{i\}\\approx 45\), and theY~i\\widetilde\{Y\}\_\{i\}pseudo\-outcomes inherit those weights\. Two ofR=200R=200hard\-scenario replications produce pseudo\-outcome distributions extreme enough to push the logistic IRLS solver outside its convergence regime, yielding\|β^1\|\|\\hat\{\\beta\}\_\{1\}\|on the order of10510^\{5\}–10610^\{6\}and dominating the column means\.111The truncated variantPA\-DSL\-IPW\-only\-trunccaps1/ρi≤201/\\rho\_\{i\}\\leq 20and restores numerical stability on the hard scenario at the cost of a small bias relative to the original design\. We report it as a sensitivity check that disentangles a small\-ρ\\rhostability artefact from a fundamental design defect; under the original \(un\-truncated\) sampling weights, IPW\-only is the design\-valid pure\-Horvitz–Thompson baseline\.

The Reg\-only variant achieves lower MC variance than fullPA\-DSLin benign and realistic, and looks nearly unbiased, but the empirical coverage drops to0\.6450\.645in benign,0\.2600\.260in realistic, and0\.0700\.070in hard\. The sandwich SE on\(𝑿i,Y~i\)\(\\bm\{X\}\_\{i\},\\widetilde\{Y\}\_\{i\}\)is computed as ifM^i=μ^​\(Zi\)\\widehat\{M\}\_\{i\}=\\hat\{\\mu\}\(Z\_\{i\}\)were a fixed quantity, butμ^\\hat\{\\mu\}is itself a noisy estimator; without theV/ρV/\\rhocorrection term to absorb that noise into the estimating equation, the SE underestimates the true variability and CIs miss the truth far more often than nominal\.

The fullPA\-DSLestimator avoids both failure modes: the regression componentμ^​\(Z\)\\hat\{\\mu\}\(Z\)stabilizes the inner pseudo\-label relative to IPW\-only \(cutting variance by an order of magnitude\), while theV/ρV/\\rhocorrection restores design validity and makes the sandwich variance reflect the adjudication\-stage uncertainty\.

Table 7:Inner\-stage AIPW ablation,β1\\beta\_\{1\}across the three synthetic scenarios \(R=200R=200replications, trueβ1=1\\beta\_\{1\}=1\)\.*IPW\-only:*innerμ^≡0\\hat\{\\mu\}\\equiv 0\.*IPW\-only \(trunc\):*IPW\-only with truncated weights1/ρ≤201/\\rho\\leq 20— a numerical sensitivity check that breaks design\-validity under the original sampling weights, included only to disentangle weight\-stability failures from fundamental design issues\.†The truncation engages only when someρi<0\.05\\rho\_\{i\}<0\.05, which happens only on the hard scenario; on benign and realistic the IPW\-only\-trunc row reproduces IPW\-only verbatim\.*Reg\-only:*noV/ρV/\\rhodebias at the inner stage\.Boldmarks the column\-best for MC SD, RMSE, and CI length per scenario among design\-valid estimators \(excluding Reg\-only and IPW\-only\-trunc, which are design\-invalid by construction\); coverage is a calibration diagnostic and bias is reported but not bolded\. The light rule separates the infeasibleDSL\-oraclebenchmark\.
## Appendix JShared\-Partition vs\.K2K^\{2\}\-fold Cross\-fit

[Proposition˜1](https://arxiv.org/html/2607.15455#Thmproposition1)\(b\) covers two implementations: leakage\-free nested cross\-fitting and the shared\-partition[Algorithm˜1](https://arxiv.org/html/2607.15455#algorithm1)under stability condition[Eq\.˜SP](https://arxiv.org/html/2607.15455#A2.Ex20)\.[Table˜8](https://arxiv.org/html/2607.15455#A10.T8)reports a paired sanity check on the realistic scenario \(N=50,000N=50\{,\}000,ρ¯=0\.25\\bar\{\\rho\}=0\.25, disagreement\-weighted adjudication,R=200R=200replications, paired per\-rep seeds\): the two implementations give the same point estimate to within Monte Carlo error, consistent with[Eq\.˜SP](https://arxiv.org/html/2607.15455#A2.Ex20)holding for the fixed\-hyperparameter L2\-regularized logistic nuisances used here\.

Table 8:Shared\-partition vs\.K2K^\{2\}\-fold double cross\-fit,β1\\beta\_\{1\}on the realistic scenario \(R=200R=200paired replications\)\. The “mean abs\. diff\.” column is the per\-replication\|β^1sp−β^1K2\|\|\\hat\{\\beta\}\_\{1\}^\{\\mathrm\{sp\}\}\-\\hat\{\\beta\}\_\{1\}^\{\\mathrm\{K^\{2\}\}\}\|averaged across replications\.The mean absolute paired difference0\.00090\.0009is two orders of magnitude smaller than either implementation’s MC SD \(≈0\.116\\approx 0\.116\), and the sample correlation between the two paired estimates is0\.999950\.99995\. The shared\-partition shortcut reproduces the leakage\-free estimator essentially exactly at the cost ofK−1K\-1fewer inner fits per outer fold\.

## Appendix KImperfect\-Adjudication Sensitivity

[Assumption˜3](https://arxiv.org/html/2607.15455#Thmassumption3)\(Ai=Yi⋆A\_\{i\}=Y^\{\\star\}\_\{i\}on adjudicated units\) is load\-bearing for design\-validity with respect toYi⋆Y^\{\\star\}\_\{i\}\. To illustrate what happens when adjudication is systematically wrong,[Table˜9](https://arxiv.org/html/2607.15455#A11.T9)compares the realistic scenario \(N=50,000N=50\{,\}000,R=500R=500replications\) under correct adjudication against a biased\-adjudication mechanism that flipsAiA\_\{i\}with probability0\.0750\.075whenXi,1\>0X\_\{i,1\}\>0and0\.0250\.025otherwise — a 5% average error correlated with the regression’s primary covariate\.

Under correct adjudication, bothDSL\-AdjOnlyandPA\-DSLare unbiased and have nominal coverage\. Under biased adjudication, both inherit the adjudicator bias relative toYi⋆Y^\{\\star\}\_\{i\}and miss the truth at higher rates\.PA\-DSLretains its design\-validity for the*adjudicated construct*\(the function𝔼​\[Ai∣𝑿i\]\\mathbb\{E\}\[A\_\{i\}\\mid\\bm\{X\}\_\{i\}\]\), but that construct is no longer𝔼​\[Yi⋆∣𝑿i\]\\mathbb\{E\}\[Y^\{\\star\}\_\{i\}\\mid\\bm\{X\}\_\{i\}\]when adjudicators systematically disagree with the latent label\. The audit/adjudication design probabilities cannot recover information adjudicators do not provide; designs that further relax[Assumption˜3](https://arxiv.org/html/2607.15455#Thmassumption3)require an additional measurement tier \(e\.g\. treatingAiA\_\{i\}as a noisy proxy ofYi⋆Y^\{\\star\}\_\{i\}with a known design weight\)\.

Table 9:Imperfect\-adjudication sensitivity,β1\\beta\_\{1\}on the realistic scenario \(R=500R=500replications, targetβ1=1\\beta\_\{1\}=1\)\.*Correct A=Y\**:[Assumption˜3](https://arxiv.org/html/2607.15455#Thmassumption3)holds\.*Biased A*:AiA\_\{i\}flipped with probability0\.0750\.075whenXi,1\>0X\_\{i,1\}\>0else0\.0250\.025\.Boldmarks the column\-best for MC SD, RMSE, and CI length within each adjudication\-target block among design\-valid estimators \(DSL\-AdjOnly,PA\-DSL\)\. The light rule separates the infeasibleDSL\-oraclebenchmark\.
## Appendix LClassification\-oriented predictor diagnostics\.

Although PA\-DSL targets downstream estimating equations rather than individual classification, we report supplemental prediction diagnostics to help interpret the efficiency mechanism\. Table[10](https://arxiv.org/html/2607.15455#A12.T10)compares the full\-frame surrogateQiQ\_\{i\}, the outer nuisance predictiong^​\(Bi\)\\widehat\{g\}\(B\_\{i\}\), and the inner nuisance predictionμ^​\(Zi\)\\widehat\{\\mu\}\(Z\_\{i\}\), evaluated against the adjudicated/proxy target\. The final AIPW pseudo\-outcomes are not included as classifiers because they are estimating\-equation devices and need not be calibrated probabilities or lie in\[0,1\]\[0,1\]\.

The diagnostics support the mechanism described in Appendix[B](https://arxiv.org/html/2607.15455#A2)\. In the benign and realistic synthetic settings,μ^​\(Zi\)\\widehat\{\\mu\}\(Z\_\{i\}\)improves prediction relative tog^​\(Bi\)\\widehat\{g\}\(B\_\{i\}\), indicating that the audit\-tier features add adjudication\-relevant information beyondBi=\(Qi,Xi\)B\_\{i\}=\(Q\_\{i\},X\_\{i\}\)\. In the hard setting, this incremental signal is small and the inner prediction does not improve ong^​\(Bi\)\\widehat\{g\}\(B\_\{i\}\), matching the near\-zero efficiency gain reported in the main results\. The Detox study shows a modest but consistent improvement fromμ^​\(Zi\)\\widehat\{\\mu\}\(Z\_\{i\}\), consistent with the observed PA\-DSL efficiency gain\.

Table 10:Classification\-oriented diagnostics for predictive components\. AUC is higher\-is\-better; Brier score is lower\-is\-better\. The RMS difference is\{n−1​∑i\(μ^​\(Zi\)−g^​\(Bi\)\)2\}1/2\\\{n^\{\-1\}\\sum\_\{i\}\(\\widehat\{\\mu\}\(Z\_\{i\}\)\-\\widehat\{g\}\(B\_\{i\}\)\)^\{2\}\\\}^\{1/2\}, a descriptive measure of incremental audit\-tier signal\. Diagnostics are evaluated against the adjudicated/proxy target and are not used as inferential validity criteria\.