BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges

# BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges
Source: [https://arxiv.org/html/2607.16239](https://arxiv.org/html/2607.16239)
Anlan ZhangAdobe ResearchRita LyuUniversity of California, BerkeleyZhengmian HuAdobe ResearchTong YuAdobe ResearchDavid ArbourAdobe Research Avi FellerUniversity of California, BerkeleySaayan MitraAdobe ResearchRitwik SinhaAdobe Research

###### Abstract

AI judges are increasingly used to reduce the cost of human evaluation\. While they provide a scalable and inexpensive alternative, their outputs can be biased relative to human preferences and highly item\-dependent, with substantial variation across judges, tasks, and domains\. When practitioners rely on uncalibrated AI evaluations for model ranking, item scoring, or population\-level quality reporting, systematic bias can propagate directly into downstream decisions and make the conclusions unreliable\. We propose BACON, a four\-stage pipeline that combines budgeted human calibration with multiple AI\-judge outputs to obtain more accurate annotations\. BACON first constructs full\-coverage auxiliary features for every item in the evaluation pool, including multi\-judge scores, token\-level uncertainty statistics, and contextual embeddings\. It then obtains human labels for a small sampled subset and trains a cross\-fitted outcome model to produce calibrated item\-level surrogate predictions\. These predictions support two downstream modes: the first is population\-level estimation of summary metrics, such as means, quantiles, or other estimands, using an augmented estimating\-equation estimator with valid confidence intervals; and the second is individual\-level surrogate scoring for item scoring and ranking\. Throughout, BACON treats AI judges as auxiliary measurements rather than ground truth: human labels provide the calibration anchor, while AI\-derived signals serve as surrogate predictions and improve efficiency\. We validate BACON on a variety of tasks and domains\. Across settings and labeling budgets, the cross\-fitted outcome model improves predictive accuracy and ranking consistency, while the estimating\-equation estimator reduces bias and variance relative to raw AI outputs and purely human\-label\-based methods\. These results suggest that BACON provides a practical framework for scalable, statistically grounded evaluation with limited human annotation\.

## 1Introduction

### 1\.1Background: Bias in AI\-assisted evaluations

Many important decisions rely on large\-scale human evaluations: grading student writing, scoring persuasive essays, and rating the aesthetics, trustworthiness, and usability of websites, among others\. Human labels are expensive, especially when the goal is not only to score individual items but also to estimate population\-level summary statistics, such as the mean quality of a corpus, under a fixed budget\. Using generative AI as scalable judges has become a popular practice, providing low\-cost scores for large item pools\. However, AI\-judge outputs can be systematically biased relative to human preferences, and their errors are often item\-dependent and model\-specific\. Below, we give three examples showing that this mismatch appears across tasks and modalities\.

Case Study 1: Essay scoring with PERSUADE Dataset\. The PERSUADE dataset\(Crossleyet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib18)\)contains over 25,000 argumentative essays from U\.S\. students in grades 6–12, each assigned a holistic human score from 1 to 6\. We prompt several LLMs to score the same essays and compare their outputs to the human labels \(prompt in Appendix[B](https://arxiv.org/html/2607.16239#A2)\)\. Figure[1\(a\)](https://arxiv.org/html/2607.16239#S1.F1.sf1)compares human and LLM score distributions on PERSUADE\. Human scores are more heavy\-tailed than the AI scores, and some AI models \(e\.g\. llama\-3\.1\-8b\) concentrate around an entirely different peak\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/persuade_grade10_human_vs_llm_score_distribution.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/mqm_human_model_alignment_histograms.png)\(b\)

Figure 1:Score distribution comparisons between human raters and AI judges for PERSUADE and MQM\. Panel \(a\): Grade\-10 PERSUADE score proportions for human labels and all 8 LLM judges \(scores 1–6\)\. Human shown as solid line; LLMs as dashed lines\. Panel \(b\): MQM segment\-level score distributions for human raters and six LLM judges on WMT 2020 en–de\. Both datasets show systematic disagreement between human labels and AI judges\.Case Study 2: Machine translation evaluation\. We also study the WMT 2020 English–German MQM dataset\(Freitaget al\.,[2021](https://arxiv.org/html/2607.16239#bib.bib19)\), where expert raters annotate translation errors by category and severity and each segment receives a non\-negative MQM score \(lower is better\)\. We use six LLM judges \(listed in Figure[1\(b\)](https://arxiv.org/html/2607.16239#S1.F1.sf2)\) to produce MQM\-style annotations across ten MT systems, including three human translations\. Figure[1\(b\)](https://arxiv.org/html/2607.16239#S1.F1.sf2)shows systematic miscalibration: some LLM judges largely underestimate or overestimate error rates \(Gemini\-2\.5\-Flash and Llama\-3\.1\-8B\), while others demonstrate a two peak distribution that is not observed in human labels\.

Case Study 3: Web\-design scoring\. Finally, we consider visual evaluation of website screenshots, where models must judge perceptual qualities such as aesthetics, trustworthiness, typicality, and usability\(Miniukovich and Figl,[2023](https://arxiv.org/html/2607.16239#bib.bib7)\)\. Figure[2](https://arxiv.org/html/2607.16239#S1.F2)shows the score distributions for the Universities subset of the WebDesign dataset and reveals substantial disagreement between human labels and VLM judges\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/unis_human_vs_vlm_score_distribution.png)Figure 2:WebDesign Universities score distributions for six perceptual outcomes \(scores−3\-3to33\)\. Human scores are shown as KDE curve; VLM judges as proportion lines\.Together, these examples show a common problem: AI scores usually don’t target the same measurement as the human labels\. If practitioners simply average AI scores or use them to rank items or systems, they are making an implicit and usually unvalidated measurement assumption\. We propose BACON to address this problem\.

### 1\.2BACON framework overview and our contributions

BACON is an evaluation protocol for making human\-aligned claims under limited labels\. For a target item population, the evaluator specifies the human outcome and the evaluation report of interest, such as a population mean, system ranking, or item\-level score\. BACON collects full\-coverage auxiliary measurements from multiple AI judges, including scores, uncertainty statistics, and item embeddings\. A small portion of samples is then labeled by humans and used to fit a cross\-fitted calibration model from AI measurements to the target human outcome\. BACON reports calibrated item\-level surrogate scores and aggregate human\-aligned estimates with uncertainty intervals\. Throughout the protocol, AI judges are treated as auxiliary measurements rather than replacements for human labels: human calibration anchors the validity of the final claim, while AI measurements improve efficiency when they are predictive\.

To summarize, we make the following contribution:

\(1\)Human\-calibrated AI\-judge evaluation protocol\.We formulate AI\-assisted evaluation as a budgeted human\-calibration problem in which multiple AI judges are treated as auxiliary measurements, not ground truth\. AI judge scores provide full\-coverage outcome predictions, and a small uniform human sample supplies a bias correction, decoupling estimation accuracy from the quality of the outcome model\.

\(2\)Multi\-judge calibration recipes\.BACON distinguishes two evaluation tasks: empirically calibrated item\-level surrogate scoring and statistically valid summary metric estimation\. For item\-level evaluation, we propose building outcome models from multiple judge scores, uncertainty summaries, and context embeddings, and discuss practical model choices for different type of outcomes such as ordinal, zero\-inflated, continuous, and categorical evaluation outcomes\. For summary statistics report, BACON follows an estimating equation\-based statistical framework to deliver estimators that are consistent for human assessment and provide valid uncertainty quantification through confidence intervals\.

\(3\)Cross\-domain empirical study of AI\-judge calibration\.We validate BACON across three real\-world datasets spanning multiple tasks and modalities, including PERSUADE essay scoring, MQM machine translation evaluation, and WebDesign web\-design scoring\. The results demonstrate substantial improvements in judge\-model predictive accuracy for individual item quality and the estimation efficiency for summary metrics over several baselines\. The code for replication is hosted in the following GitHub repo \(anonymized for review\):[https://github\.com/diaryofnewton/bacon\-calibration](https://github.com/diaryofnewton/bacon-calibration)\.

### 1\.3Related Works

LLM\-as\-a\-judge\.Zhenget al\.\([2023](https://arxiv.org/html/2607.16239#bib.bib30)\)establish the canonical LLM\-as\-judge paradigm via MT\-Bench and Chatbot Arena;Liuet al\.\([2023](https://arxiv.org/html/2607.16239#bib.bib31)\)extend it with chain\-of\-thought prompting for text\-generation evaluation; andDuboiset al\.\([2024](https://arxiv.org/html/2607.16239#bib.bib35)\)develop AlpacaEval for instruction\-following comparison\. Despite their utility, LLM judges exhibit systematic biases in terms of position, verbosity, and self\-enhancement\(Wanget al\.,[2023a](https://arxiv.org/html/2607.16239#bib.bib32)\), that distort rankings; surveys\(Liet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib33); Guet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib34)\)catalog these failure modes\. BACON highlights such biases as a starting point and provides a principled calibration framework to correct them using a small sample of human labeled data\.

Multi\-judge aggregation\.Self\-consistency\(Wanget al\.,[2023b](https://arxiv.org/html/2607.16239#bib.bib39)\)and multi\-agent debate\(Duet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib36); Lianget al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib37)\)use ensembles of reasoning paths or models to improve reliability\. In the evaluation setting, ChatEval\(Chanet al\.,[2023](https://arxiv.org/html/2607.16239#bib.bib38)\), jury\-style panels\(Vergaet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib40)\), and structured multi\-agent collaboration\(Qianet al\.,[2025](https://arxiv.org/html/2607.16239#bib.bib41)\)show that aggregating diverse judges reduces individual biases\. BACON differs: rather than debate or ad\-hoc ensembling, we treat judge scores as features in a supervised outcome model fit against human labels, yielding statistically grounded estimates\.

Prediction\-powered inference \(PPI\) and semi\-supervised estimation\.Angelopouloset al\.\([2023a](https://arxiv.org/html/2607.16239#bib.bib23)\)introduce PPI, using model predictions plus a small labeled set to form valid confidence intervals;Angelopouloset al\.\([2023b](https://arxiv.org/html/2607.16239#bib.bib24)\)extend this to PPI\+\+ with adaptive weighting;Zrnic \([2024](https://arxiv.org/html/2607.16239#bib.bib25)\)remove the held\-out requirement via cross\-fitting\. These ideas root in classical survey regression estimators\(Särndalet al\.,[1992](https://arxiv.org/html/2607.16239#bib.bib26); Casselet al\.,[1976](https://arxiv.org/html/2607.16239#bib.bib27)\)and semi\-supervised inference\(Zhanget al\.,[2019](https://arxiv.org/html/2607.16239#bib.bib28); Cai and Guo,[2020](https://arxiv.org/html/2607.16239#bib.bib29)\)\. In the application of summary metrics estimation, BACON follows directly this line of work\. For the special case of estimating a population mean, our estimator is mathematically equivalent to a PPI / regression\-adjusted estimator with a learned prediction function\. The contribution of BACON is therefore not a new mean estimator\. Instead, BACON addresses a problem left open by generic PPI formulations: how to construct reliable auxiliary predictions in AI\-assisted evaluation when the available predictors are multiple biased AI judges, uncertainty statistics, as well as item embeddings, and how to report both aggregate inferential quantities and item\-level surrogate scores without treating AI judgments as ground truth, which serves as the protocol layer between LLM\-as\-judge and PPI\.

Training and fine\-tuning LLM judges\.JudgeLM\(Zhuet al\.,[2025](https://arxiv.org/html/2607.16239#bib.bib42)\), Prometheus\(Kimet al\.,[2024a](https://arxiv.org/html/2607.16239#bib.bib43),[b](https://arxiv.org/html/2607.16239#bib.bib44)\), and PandaLM\(Wanget al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib45)\)fine\-tune open\-source models as scalable evaluators, whileLiet al\.\([2023](https://arxiv.org/html/2607.16239#bib.bib46)\)produce evaluative rationales rather than scalar scores\. However,Huanget al\.\([2025](https://arxiv.org/html/2607.16239#bib.bib47)\)find that fine\-tuned judges often underperform GPT\-4 out\-of\-domain\. BACON is training\-free: off\-the\-shelf judge outputs feed a lightweight calibration layer, requiring no judge\-specific training data and supporting closed\-source models\.

## 2BACON Framework

### 2\.1Overview and notation setup

We present a training\-free pipeline for AI\-assisted evaluation\. BACON integrates three components to mitigate bias in AI judge outputs: \(1\) aggregating predictions from multiple judges to capture cross\-judge variability; \(2\) leveraging contextual features to debias judge predictions; and \(3\) calibrating AI evaluations using a small sample of human\-labeled data\.

We study AI\-assisted modeling and evaluation under limited human labeling budgets\. Suppose we have a pool of items, each associated with a scalar or vector\-valued human outcome\. Our goal is to estimate population\-level performance and improve item\-level prediction by combining AI judge outputs in a principled way, with a small amount of human labels for calibration\.

Formally, for itemi∈\{1,…,N\}i\\in\\\{1,\\dots,N\\\}, letyiy\_\{i\}denote the human score \(or one outcome component\), and letℓi∈ℝM\\mathbf\{\\ell\}\_\{i\}\\in\\mathbb\{R\}^\{M\}collectMMmodel\-judge scores\. We also construct contextual embedding featureseie\_\{i\}\(e\.g\., text embeddings for essays and image embeddings for screenshots\)\. We then fit predictive models foryiy\_\{i\}using judge scores, uncertainty features, and embedding features to capture their relationship with human evaluations\.

### 2\.2Budgeted Evaluation Framework

The proposed evaluation framework is as follows; Figure[3](https://arxiv.org/html/2607.16239#S2.F3)gives an overview\.

Stage 1Model InputPreparationStage 2Human LabelSamplingStage 3Outcome ModelFittingStage 4aSummaryMetricsStage 4bItem Scoresand Rankingsembeddingseie\_\{i\}scoresℓi\\ell\_\{i\}, uncertaintyuiu\_\{i\}uniform oradaptivecross\-fittingf^​\(ei,ℓi,ui\)\\widehat\{f\}\(e\_\{i\},\\ell\_\{i\},u\_\{i\}\)μ^EE\\widehat\{\\mu\}^\{\\mathrm\{EE\}\}, CIsy^i\\widehat\{y\}\_\{i\}, rankFigure 3:BACON evaluation pipeline\. Stage 1 prepares inputs \(context features and AI jury scores\) for all items; Stage 2 collects a small human label sample; Stage 3 fits the outcome model with cross\-fitting; Stage 4 routes to summary\-metric estimation or individual scoring and ranking\.\(1\) Stage 1: Model input preparation\. \(i\) Context features extraction\. For each item in the evaluation pool, we extract some context features\. This includes the content embeddings, summary statistics, emotion and psychological attributes, among others\(Mozer and Miratrix,[2025](https://arxiv.org/html/2607.16239#bib.bib21)\)\. \(ii\) AI Jury Scoring\. We use multiple AI judges to score all the items in the pool\. We can record auxiliary information such as the token entropy and perplexity\. We can also run the AI judges for multiple rounds\.

\(2\) Stage 2: Human Label Sampling\. We sample a small number of items to get human labels\. The basic approach is to sample items uniformly at random, given a fixed budget\. Also, we can apply adaptive sampling strategies to improve the sampling efficiency\. We leave an detailed discussion of adaptive sampling strategies to Appendix[D](https://arxiv.org/html/2607.16239#A4)\.

\(3\) Stage 3: Outcome Model Fitting\. We fit an outcome model to the human labels and the AI scores\. For this part, we use cross\-fitting to obtain out\-of\-fold predictions to avoid overfitting\.

\(4\) Stage 4: Route the model for downstream tasks\. Here we list two representative tasks to use the scores: \(i\) Estimate summary metrics\. We build regression\-adjusted estimators to estimate aggregated metrics such as the population average of human scores, as well as providing uncertainty quantifications through confidence intervals\. \(ii\) Get individual item predictions and ranks\. We can use the model to predict surrogate human scores and rank the items\. The surrogate scores can be used to flag low quality items\.

### 2\.3Details of the BACON Framework

Joint modeling of context features and multi\-judge evaluations\.Overall, we try to fit a predictive model foryiy\_\{i\}from the context featureseie\_\{i\}and the judge evaluationsℓi\\ell\_\{i\}\. Moreover, we can incorporate additional featuresuiu\_\{i\}which encodes the uncertainty of the judge evaluations\. We propose to fit a joint model foryiy\_\{i\}from the context featureseie\_\{i\}, the judge evaluationsℓi\\ell\_\{i\}with uncertainty featuresuiu\_\{i\}:

y^i=f​\(ei,ℓi,ui\),\\displaystyle\\widehat\{y\}\_\{i\}=f\(e\_\{i\},\\ell\_\{i\},u\_\{i\}\),whereffis a function that we need to learn from data\. We will discuss the choice offflater\. Intuitively, we hope the functionffcan serve as a meta\-judge that aggregates the single\-judge evaluations and steer the prediction towards the human preferences\.

Embedding featureseie\_\{i\}capture item\-level effects, such as topic, style, visual composition, that are not explained by judge scores alone, acting as fixed\-effect controls\. Multiple judge scoresℓi\\ell\_\{i\}are combined via model fitting rather than ad\-hoc majority voting or averaging, allowing the outcome model to down\-weight biased or unreliable judges\. Uncertainty featuresuiu\_\{i\}encode judge confidence: token\-level entropy \(available when log\-probabilities are accessible\) or sample\-level variance from multiple judge runs \(applicable to any model\)\.

Example models\.The choice offfdepends on the score pattern of the outcome\. Table[1](https://arxiv.org/html/2607.16239#S2.T1)summarizes several instantiations; all share the same feature inputs\(ei,ℓi,ui\)\(e\_\{i\},\\ell\_\{i\},u\_\{i\}\)\.

Table 1:Example outcome models in BACON\. All models use embeddingseie\_\{i\}, judge scoresℓi\\ell\_\{i\}, and uncertainty featuresuiu\_\{i\}as inputs\.Score patternExample modelKey ideaReal\-valuedRidge regressionLinear predictory^i=α0\+αe⊤​ei\+αℓ⊤​ℓi\+αu⊤​ui\\widehat\{y\}\_\{i\}=\\alpha\_\{0\}\+\\alpha\_\{e\}^\{\\top\}e\_\{i\}\+\\alpha\_\{\\ell\}^\{\\top\}\\ell\_\{i\}\+\\alpha\_\{u\}^\{\\top\}u\_\{i\};ℓ2\\ell\_\{2\}penalty controls judge collinearity\.Non\-negative, zero\-inflatedHurdle model \(logistic gate\+\+conditional ridge\)GatePr⁡\(zi=1\)\\Pr\(z\_\{i\}\{=\}1\)predicts whetheryi\>0y\_\{i\}\{\>\}0; conditional ridge predictsy^i\+\\widehat\{y\}\_\{i\}^\{\+\}on positive examples; final predictiony^i=Pr⁡\(zi=1\)⋅y^i\+\\widehat\{y\}\_\{i\}=\\Pr\(z\_\{i\}\{=\}1\)\\cdot\\widehat\{y\}\_\{i\}^\{\+\}\.OrdinalProportional\-odds \(ordered logit\)Linear predictorηi=α0\+αe⊤​ei\+αℓ⊤​ℓi\+αu⊤​ui\\eta\_\{i\}=\\alpha\_\{0\}\+\\alpha\_\{e\}^\{\\top\}e\_\{i\}\+\\alpha\_\{\\ell\}^\{\\top\}\\ell\_\{i\}\+\\alpha\_\{u\}^\{\\top\}u\_\{i\}withK−1K\{\-\}1ordered cutpointsθ1<⋯<θK−1\\theta\_\{1\}\{<\}\\cdots\{<\}\\theta\_\{K\-1\};Pr⁡\(Yi≤k\)=σ​\(θk−ηi\)\\Pr\(Y\_\{i\}\{\\leq\}k\)=\\sigma\(\\theta\_\{k\}\-\\eta\_\{i\}\)\.CategoricalMultinomial regressionPr⁡\(Yi=k\)=exp⁡\(ηi\(k\)\)/∑j=1Kexp⁡\(ηi\(j\)\)\\Pr\(Y\_\{i\}\{=\}k\)=\\exp\(\\eta\_\{i\}^\{\(k\)\}\)/\\sum\_\{j=1\}^\{K\}\\exp\(\\eta\_\{i\}^\{\(j\)\}\), whereηi\(k\)=α0\(k\)\+αe\(k\)⊤​ei\+αℓ\(k\)⊤​ℓi\+αu\(k\)⊤​ui\\eta\_\{i\}^\{\(k\)\}=\\alpha\_\{0\}^\{\(k\)\}\+\\alpha\_\{e\}^\{\(k\)\\top\}e\_\{i\}\+\\alpha\_\{\\ell\}^\{\(k\)\\top\}\\ell\_\{i\}\+\\alpha\_\{u\}^\{\(k\)\\top\}u\_\{i\}\.Cross\-fitting for outcome model fitting\.A key requirement is that the outcome modelf^\\widehat\{f\}must not be fitted on the same sampled observations whose residuals are used for bias correction\. If the same data is used for both fitting and prediction, the in\-sample fit can be overly optimistic—a model that memorises its training labels will produce near\-zero residuals, inflating bias for complex, low\-regularization models\. Cross\-fitting\(Chernozhukovet al\.,[2018](https://arxiv.org/html/2607.16239#bib.bib22)\)resolves this by partitioning the sampled set𝒮=\{i:Si=1\}\\mathcal\{S\}=\\\{i:S\_\{i\}=1\\\}intoKKfolds𝒮1,…,𝒮K\\mathcal\{S\}\_\{1\},\\ldots,\\mathcal\{S\}\_\{K\}\. For each foldkk, the outcome modelf^\(−k\)\\widehat\{f\}^\{\(\-k\)\}is trained on all sampled observations*except*foldkk, and its predictions on foldkkare used as the out\-of\-fold \(OOF\) predictions\. Formally, fori∈𝒮ki\\in\\mathcal\{S\}\_\{k\}:

y^i=f^\(−k\)​\(ei,ℓi,ui\),f^\(−k\)=fit⁡\(\{\(ej,ℓj,uj,Yj\):j∈𝒮∖𝒮k\}\)\.\\widehat\{y\}\_\{i\}=\\widehat\{f\}^\{\(\-k\)\}\(e\_\{i\},\\ell\_\{i\},u\_\{i\}\),\\qquad\\widehat\{f\}^\{\(\-k\)\}=\\operatorname\{fit\}\\\!\\left\(\\\{\(e\_\{j\},\\ell\_\{j\},u\_\{j\},Y\_\{j\}\):j\\in\\mathcal\{S\}\\setminus\\mathcal\{S\}\_\{k\}\\\}\\right\)\.Then we fit the an aggregated modelf^\\widehat\{f\}on the full sampleSSand use it to predict the labels for items that are not inSS\. Through this formulation, the predictionsy^i\\widehat\{y\}\_\{i\}’s cover allNNitems\. Becausef^\(−k\)\\widehat\{f\}^\{\(\-k\)\}is fitted on a held\-out split, the residualsYi−y^iY\_\{i\}\-\\widehat\{y\}\_\{i\}are honest as they reflect genuine generalisation error rather than in\-sample fit\. This prevents overfit models from artificially suppressing the bias\-correction variance, and makes the variance estimate more stable when the outcome model is complex\. In practice, we useK=5K\{=\}5folds, each trained on 80% of the sampled data, which balances the bias–variance trade\-off for the OOF predictions\.

Summary metrics estimation with uncertainty quantification\.Summary metrics include population mean, median/quantiles, variance, etc\. We can unify the estimation of common summary metrics using an estimating equation framework\. LetYYbe the observed human scores \(withNAif no human label is collected\),SSbe the indicator variable indicating whether a human label is collected,XXbe some measurements for the context, such as the embedding features, the judge scores and uncertainty summary metrics, etc\. Below we provide an estimating equation framework\(Hardin and Hilbe,[2002](https://arxiv.org/html/2607.16239#bib.bib14)\)\.

Suppose we are interested in a target parameterθ⋆=θ​\(ℙY,S,X\)\\theta^\{\\star\}=\\theta\(\\mathbb\{P\}\_\{Y,S,X\}\), which is the solution to the following estimating equation:𝔼​\{m​\(Y,S,X;θ⋆,γ⋆\)\}=0,\{\\mathbb\{E\}\\left\\\{m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\\right\\\}\}=0,wherem​\(Y,S,X;θ,γ\)m\(Y,S,X;\\theta,\\gamma\)is the estimating equation, andγ⋆\\gamma^\{\\star\}is a nuisance parameter\. For example, for the mean andτ\\tau\-th quantile, we have

mmean​\(Y,S,X;θ,γ\)=S​Yπ​\(X\)−θ,andmquantile​\(Y,S,X;θ,γ\)=S​𝟏​\{Y≤θ\}π​\(X\)−τ,m\_\{\\text\{mean\}\}\(Y,S,X;\\theta,\\gamma\)=\\frac\{SY\}\{\\pi\(X\)\}\-\\theta,\\quad\\text\{and\}\\quad m\_\{\\text\{quantile\}\}\(Y,S,X;\\theta,\\gamma\)=\\frac\{S\\bm\{1\}\\left\\\{Y\\leq\\theta\\right\\\}\}\{\\pi\(X\)\}\-\\tau,respectively\. Hereπ​\(X\)\\pi\(X\)is the labeling propensity and serves as a given nuisance parameterγ⋆\\gamma^\{\\star\}\. In practice, we can apply many classical sampling schemes to collect the human labels, which corresponds to different choices ofπ​\(X\)\\pi\(X\)\. For example, we can draw items uniformly at random, or perform stratified sampling based on some context featuresXX\. Alternatively, we may learn the labeling propensity from some pilot data using a propensity model\. See Appendix[D](https://arxiv.org/html/2607.16239#A4)for more details\.

It is usually desirable to incorporate a regression adjustment term to the estimating equation to improve the efficiency of the estimator\. For the above examples, we can augment the estimating equation as:

mmean​\(Y,S,X;θ,γ\)=S​\(Y−μ​\(X\)\)π​\(X\)\+μ​\(X\)−θ,\\displaystyle m\_\{\\text\{mean\}\}\(Y,S,X;\\theta,\\gamma\)=\\frac\{S\(Y\-\\mu\(X\)\)\}\{\\pi\(X\)\}\+\\mu\(X\)\-\\theta,mquantile​\(Y,S,X;θ,γ\)=S​\(𝟏​\{Y≤θ\}−ϕ​\(X;θ\)\)π​\(X\)\+ϕ​\(X;θ\)−τ,\\displaystyle m\_\{\\text\{quantile\}\}\(Y,S,X;\\theta,\\gamma\)=\\frac\{S\(\\bm\{1\}\\left\\\{Y\\leq\\theta\\right\\\}\-\\phi\(X;\\theta\)\)\}\{\\pi\(X\)\}\+\\phi\(X;\\theta\)\-\\tau,respectively\. Hereμ​\(X\)\\mu\(X\)is a regression adjustment for the human scores andϕ​\(X;θ\)\\phi\(X;\\theta\)is a regression adjustment for the quantile\. We can then estimate the target parameterθ⋆\\theta^\{\\star\}by solving the following estimating equation:𝔼N​\(m​\(Y,S,X;θ^,γ^\)\)=N−1​∑i=1Nm​\(Yi,Si,Xi;θ^,γ^\)=0\\mathbb\{E\}\_\{N\}\(m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)\)=N^\{\-1\}\\sum\_\{i=1\}^\{N\}m\(Y\_\{i\},S\_\{i\},X\_\{i\};\\widehat\{\\theta\},\\widehat\{\\gamma\}\)=0, which we call as the augmented estimating\-equation \(AEE\) estimator\. Under some regularity conditions \(Appendix[C\.1](https://arxiv.org/html/2607.16239#A3.SS1)\), the estimatorθ^\\widehat\{\\theta\}is consistent and asymptotically normal:N​\(θ^−θ⋆\)↝𝒩​\(0,Σθ\)\.\\sqrt\{N\}\(\\widehat\{\\theta\}\-\\theta^\{\\star\}\)\\rightsquigarrow\\mathcal\{N\}\(0,\\Sigma\_\{\\theta\}\)\.The asymptotic covariance takes the sandwich formΣθ=A−1​B​A−T\\Sigma\_\{\\theta\}=A^\{\-1\}BA^\{\-T\}, whereAAandBBare given by

A=𝔼​\{∇θm​\(Y,S,X;θ⋆,γ⋆\)\},andB=𝔼​\{m​\(Y,S,X;θ⋆,γ⋆\)​m​\(Y,S,X;θ⋆,γ⋆\)⊤\}\.A=\{\\mathbb\{E\}\\left\\\{\\nabla\_\{\\theta\}m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\\right\\\}\},\\quad\\text\{and\}\\quad B=\{\\mathbb\{E\}\\left\\\{m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\\,m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)^\{\\top\}\\right\\\}\}\.Accordingly, a plug\-in sandwich estimator ofVar​\{θ^\}\\textup\{Var\}\\\{\\widehat\{\\theta\}\\\}isv^=1n​A^−1​B^​A^−T\\widehat\{v\}=\\frac\{1\}\{n\}\\widehat\{A\}^\{\-1\}\\widehat\{B\}\\widehat\{A\}^\{\-T\}, withA^\\widehat\{A\}andB^\\widehat\{B\}given by

A^=𝔼N​∇θm​\(Y,S,X;θ^,γ^\),andB^=𝔼N​m​\(Y,S,X;θ^,γ^\)​m​\(Y,S,X;θ^,γ^\)⊤\.\\widehat\{A\}=\\mathbb\{E\}\_\{N\}\{\\nabla\_\{\\theta\}m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)\},\\quad\\text\{and\}\\quad\\widehat\{B\}=\\mathbb\{E\}\_\{N\}\{m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)\\,m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)^\{\\top\}\}\.Then we can construct a confidence interval forθ⋆\\theta^\{\\star\}asθ^±zα/2​v^\\widehat\{\\theta\}\\pm z\_\{\\alpha/2\}\\sqrt\{\\widehat\{v\}\}with confidence level1−α1\-\\alpha\. When the target is the population mean, the AEE estimator is equivalent to the inverse\-propensity weighted estimator upon regression adjustment in survey sampling\(Särndalet al\.,[1992](https://arxiv.org/html/2607.16239#bib.bib26); Casselet al\.,[1976](https://arxiv.org/html/2607.16239#bib.bib27)\)or predictive powered inference\(Angelopouloset al\.,[2023a](https://arxiv.org/html/2607.16239#bib.bib23)\)\. For a rigorous technical presentation, please refer to Appendix[C\.1](https://arxiv.org/html/2607.16239#A3.SS1)\.

## 3Real\-World Results

### 3\.1Setup

Datasets and Outcomes\.\(i\)PERSUADE\.We use the PERSUADE training corpus with essay\-level holistic labels\(Crossleyet al\.,[2024](https://arxiv.org/html/2607.16239#bib.bib18)\)\. In particular, main experiments target grade\-10 essays\. \(ii\)Machine translation\.We use the WMT 2020 English–German MQM \(Multidimensional Quality Metrics\) dataset\(Freitaget al\.,[2021](https://arxiv.org/html/2607.16239#bib.bib19)\), where expert human raters annotate translation errors by category and severity\. Each segment receives a non\-negative MQM score \(lower is better; zero means no errors detected\)\. \(iii\)WebDesign\.We use screenshot datasets from multiple universities\(Miniukovich and Figl,[2023](https://arxiv.org/html/2607.16239#bib.bib7)\)\. For each screenshot, we model six human\-rated outcomes: Aesthetics, Trustworthiness, Typicality, Exemplar Goodness, Family Resemblance, and Usability\. Due to space limit, we defer the WebDesign results to Appendix[A\.2](https://arxiv.org/html/2607.16239#A1.SS2)\.

Embedding Features\.\(i\)Text embeddings\.Essays/Machine translation segments are embedded usingtext\-embedding\-3\-large111[https://developers\.openai\.com/api/docs/models/text\-embedding\-3\-large](https://developers.openai.com/api/docs/models/text-embedding-3-large)from OpenAI\. \(ii\)Image embeddings\.Screenshots are embedded with SigLIP\(Zhaiet al\.,[2023](https://arxiv.org/html/2607.16239#bib.bib20)\)\. To stabilize downstream linear models, we apply PCA to high\-dimensional embeddings, retaining a tunable number of principal components\.

Baselines and general metrics\.We compare BACON against both uncalibrated AI\-judge baselines and calibrated outputs\. The uncalibrated baselines use raw judge scores directly, including single\-judge scores and averages across judges\. For calibrated models, to isolate the value of different signals, we evaluate calibrated models using only judge scores and uncertainty features, only embedding features, or the full joint feature set\. For item\-level prediction, we reportR2R^\{2\}, root mean squared error \(RMSE\), and Spearman correlation\. For summary\-metric estimation, we report bias, standard deviation, and RMSE across repeated sampling trials\. Monte Carlo experiments are run on a server with an Intel Xeon Platinum 8275CL CPU \(96 cores, 3\.00 GHz, 1\.1 TB RAM\)\.

### 3\.2PERSUADE

Oracle performance\. Table[2](https://arxiv.org/html/2607.16239#S3.T2)reports out\-of\-fold judge performance when fitting the outcome model on grade\-10 essays under six configurations with all human labels revealed, in order to evaluate the oracle performance of the model\. We use an ordinal regression model\. The full 8\-judge model combined with embeddings achieves the best performance\. Single\-judge results reveal a slight performance drop with embeddings, and a large performance drop without embeddings, indicating that embedding features carry substantial predictive signal\. As uncalibrated baselines, simply averaging all judges’ raw scores \(no outcome model\) yieldsR2=0\.361R^\{2\}\{=\}0\.361, and averaging per\-judge raw metrics yieldsR2=0\.150R^\{2\}\{=\}0\.150—substantially below all calibrated configurations\.

Table 2:PERSUADE judge\-model performance \(5\-fold OOF CV, ordinal regression\)\. Lower RMSE is better; higherR2R^\{2\}and Spearmanρ\\rhoare better\.ConfigurationTypeR2R^\{2\}RMSESpearmanρ\\rhoEight judges \+ embeddingCalibrated0\.6990\.6020\.831Eight judges \(no embedding\)Calibrated0\.3990\.8510\.619Single judge \+ embeddingCalibrated0\.6830\.6180\.822Single judge avg\. \(no embedding\)Calibrated0\.2690\.9380\.460Embedding\-only \(ordinal\)Calibrated0\.6740\.6270\.816Intercept\-only constantCalibrated−\-0\.0001\.098−\-0\.020Raw avg\. of all judgesUncalibrated0\.3610\.8770\.601Raw single\-judge avg\.Uncalibrated0\.1501\.0070\.502Simulation on summary metrics\.We evaluate how well several configurations support the bias and efficiency claim of the AEE estimator under a uniform sampling budgetη∈\{0\.05,…,0\.30\}\\eta\\in\\\{0\.05,\\ldots,0\.30\\\}\. In each Monte Carlo trial, a fractionη\\etaof grade\-10 essays is sampled uniformly at random\. We run 200 independent trials per budget and report estimator bias, standard deviation \(SD\), and RMSE across trials\.Figure[4\(a\)](https://arxiv.org/html/2607.16239#S3.F4.sf1)shows the AEE estimator quality as a function ofη\\eta\. All calibrated methods are approximately unbiased when the sampling budget is appropriate, while the uncalibrated baselines are biased\. The full hybrid model mostly achieves the lowest SD and RMSE across budgets, demonstrating that a better outcome model directly translates to a more efficient estimator\. Across all budgets, the reduction in estimator variance from the full model over the intercept\-only baseline reflects the predictiveR2R^\{2\}gains seen in Table[2](https://arxiv.org/html/2607.16239#S3.T2)\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/persuade_uniform_dr_bias_sd_rmse.png)\(a\)Estimating\-equation estimator quality
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/persuade_uniform_dr_outcome_r2_rmse_spearman.png)\(b\)Outcome model quality vs\. budgetη\\eta\.

Figure 4:PERSUADE experimental results\. Top: bias, standard deviation, and RMSE as a function ofη\\eta\(200 MC trials\) for mean estimation\. Bottom: Outcome model quality vs\.η\\eta: out\-of\-foldR2R^\{2\}, RMSE, and Spearmanρ\\rhofitted on the sampled essays\.Outcome model quality vs\. budget\.A key question is how well the outcome model itself can be estimated from the sampled essays\. Figure[4\(b\)](https://arxiv.org/html/2607.16239#S3.F4.sf2)traces out\-of\-foldR2R^\{2\}, RMSE, and Spearmanρ\\rhoof the judge model asη\\etaincreases, using only the sampled subset for fitting\. At the smallest budget \(η=0\.05\\eta\{=\}0\.05, roughly 230 essays\), the full model already achieves meaningful predictive performance; quality improves steadily as more essays are observed\. This confirms that the joint model provides sufficient signal for reliable outcome modelling even at modest sampling rates, supporting the practical viability of BACON in annotation\-scarce settings\.

### 3\.3Machine translation

We apply the BACON framework to machine translation quality evaluation using the WMT 2020 English–German MQM dataset\.

Simulation on summary metrics\.We evaluate each outcome model as a plug\-in for the AEE estimator under uniform sampling budgetsη∈\{0\.05,…,0\.30\}\\eta\\in\\\{0\.05,\\ldots,0\.30\\\}across all ten MT systems\. In each of 200 Monte Carlo trials, a fractionη\\etaof segments per system is sampled uniformly; 5\-fold cross\-fitting with winsorised LLM score features is used in the evaluation\. Figure[5](https://arxiv.org/html/2607.16239#S3.F5)reports estimator bias, SD, and RMSE across trials\.

All methods remain approximately unbiased throughout \(max\|bias\|<0\.02\|\\text\{bias\}\|<0\.02\), confirming that the AEE estimator corrects for outcome\-model misspecification\. The hurdle models \(both full and no\-emb variants\) reduce estimator SD relative to the intercept\-only baseline at every budget\. The embedding\-only hurdle is slightly less efficient than the full model at larger budgets, suggesting that LLM judge scores contribute residual predictive signal beyond text embeddings alone\. Besides, all calibrated methods maintain Bonferroni\-corrected CI coverage at or above the nominal 95% level throughout, which suggests the validity of the uncertainty quantification\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/mqm_uniform_dr_ranking_bias_sd_rmse_crossfit.png)Figure 5:MQM AEE estimator quality vs\. sampling budgetη\\etain terms of bias, SD, and RMSE and Bonferroni\-corrected 95% CI coverage\.Translation system comparison under budgeted sampling\.One key practical goal is correctly ranking the MT systems by quality given limited human labels\. Figure[6](https://arxiv.org/html/2607.16239#S3.F6)reports Spearmanρ\\rho, Kendallτ\\tau, and mean absolute rank error \(MAE\) of rankings derived from AEE estimates vs\. the true system ranking under budgets\. All methods produce increasingly accurate rankings asη\\etagrows\. For this goal, hurdle models with both full and multiple\-judge\-only configurations achieve strong performance at only small budgets\. This suggests that system ranking can be achieved with limited human label and proper calibration\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/mqm_uniform_dr_ranking_quality_crossfit.png)Figure 6:MQM system ranking quality vs\. sampling budgetη\\eta\(200 MC trials, 5\-fold cross\-fitting\)\. Spearmanρ\\rho, Kendallτ\\tau, and mean absolute rank error \(MAE\) comparing rankings estimated from AEE to the true system ranking\. Hurdle models improve over the intercept baseline at all budgets\.Additional evaluation results\.Section[A\.1](https://arxiv.org/html/2607.16239#A1.SS1)provides more details on the MQM evaluation, including the oracle performance of the outcome model and the individual item score quality\.

## 4Discussion

What BACON adds\.BACON is a pipeline that aims to provide better guidance for the use of AI judges in evaluation\. The item\-wise labeling part is closely related to LLM\-as\-a\-judge approaches and LLM collaboration literature, while aggregated estimators are closely related to regression\-adjusted survey estimators and PPI\-style estimators\. The contribution is to bridge these practices in an unified framework and make them operational\. Our experiments suggest that this calibration layer matters: raw judge averages can be misaligned with human labels, while calibrated outcome models mitigate the bias by combining the auxiliary signals from multiple sources\.

Limitations\.BACON’s two output modes have different guarantees\. Aggregate reports, such as corpus means or system\-level averages, are anchored by human labels and inherit uncertainty quantification under the sampling assumptions\. Item\-level BACON scores are only calibrated surrogate predictions: they are useful for triage, ranking, and auditing, but should not be interpreted as unbiased measurements for each item\. Moreover, calibration may degrade under distribution shift in items, prompts, model versions, or rating rubrics; practical deployment requires periodic recalibration, version logging, and residual monitoring on fresh human labels\.

Future work\.Natural extensions include video evaluation, mobile UI assessment, and agentic task evaluation\. Future work should also develop stronger item\-level uncertainty tools, such as conformal prediction sets, and adaptive sampling strategies that improve efficiency without creating extreme propensities or hiding subgroup failures\.

## References

- A\. N\. Angelopoulos, S\. Bates, C\. Fannjiang, M\. I\. Jordan, and T\. Zrnic \(2023a\)Prediction\-powered inference\.Science382\(6671\),pp\. 669–674\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1),[§2\.3](https://arxiv.org/html/2607.16239#S2.SS3.p7.16)\.
- A\. N\. Angelopoulos, J\. C\. Duchi, and T\. Zrnic \(2023b\)PPI\+\+: efficient prediction\-powered inference\.arXiv preprint arXiv:2311\.01453\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1)\.
- T\. T\. Cai and Z\. Guo \(2020\)Semi\-supervised inference for explained variance in high\-dimensional linear regression and its applications\.Journal of the Royal Statistical Society: Series B82\(2\),pp\. 391–419\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1)\.
- C\. M\. Cassel, C\. Särndal, and J\. H\. Wretman \(1976\)Some results on generalized difference estimation and generalized regression estimation for finite populations\.Biometrika63\(3\),pp\. 615–620\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1),[§2\.3](https://arxiv.org/html/2607.16239#S2.SS3.p7.16)\.
- C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu \(2023\)ChatEval: towards better LLM\-based evaluators through multi\-agent debate\.arXiv preprint arXiv:2308\.07201\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- V\. Chernozhukov, D\. Chetverikov, M\. Demirer, E\. Duflo, C\. Hansen, W\. Newey, and J\. Robins \(2018\)Double/debiased machine learning for treatment and structural parameters\.The Econometrics Journal21\(1\),pp\. C1–C68\.Cited by:[§2\.3](https://arxiv.org/html/2607.16239#S2.SS3.p4.9)\.
- S\. A\. Crossley, Y\. Tian, P\. Baffour, A\. Franklin, M\. Benner, and U\. Boser \(2024\)A large\-scale corpus for assessing written argumentation: persuade 2\.0\.Assessing Writing61,pp\. 100865\.Cited by:[§1\.1](https://arxiv.org/html/2607.16239#S1.SS1.p2.1),[§3\.1](https://arxiv.org/html/2607.16239#S3.SS1.p1.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2024\)Improving factuality and reasoning in language models through multiagent debate\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- Y\. Dubois, X\. Li, R\. Taori, T\. Zhang, I\. Gulrajani, J\. Ba, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2024\)AlpacaEval: an automatic evaluator of instruction\-following models\.Note:[https://github\.com/tatsu\-lab/alpaca\_eval](https://github.com/tatsu-lab/alpaca_eval)Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- M\. Freitag, G\. Foster, D\. Grangier, V\. Ratnakar, Q\. Tan, and W\. Macherey \(2021\)Experts, errors, and context: a large\-scale study of human evaluation for machine translation\.Transactions of the Association for Computational Linguistics9,pp\. 1460–1474\.Cited by:[§1\.1](https://arxiv.org/html/2607.16239#S1.SS1.p3.1),[§3\.1](https://arxiv.org/html/2607.16239#S3.SS1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, L\. Shen, S\. Zhu, F\. Cheng, and J\. Ma \(2024\)A survey on LLM\-as\-a\-judge\.arXiv preprint arXiv:2411\.15594\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- M\. Hamilton, J\. Lai, W\. Zhao, S\. Maji, and D\. Sheldon \(2025\)Active measurement: efficient estimation at scale\.arXiv preprint arXiv:2507\.01372\.Cited by:[§D\.1](https://arxiv.org/html/2607.16239#A4.SS1.p3.1)\.
- J\. W\. Hardin and J\. M\. Hilbe \(2002\)Generalized estimating equations\.chapman and hall/CRC\.Cited by:[§2\.3](https://arxiv.org/html/2607.16239#S2.SS3.p5.3)\.
- H\. Huang, X\. Bu, H\. Zhou, K\. Li, X\. Fan, Y\. Guo, H\. Fei, Q\. He, J\. Li, and F\. Mi \(2025\)An empirical study of LLM\-as\-a\-judge for LLM evaluation: fine\-tuned judge model is not a general substitute for GPT\-4\.InFindings of the Association for Computational Linguistics: ACL 2025,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- S\. Kim, J\. Shin, Y\. Cho, J\. Han, S\. Cha, M\. Lee, and M\. Seo \(2024a\)Prometheus: inducing fine\-grained evaluation capability in language models\.InInternational Conference on Learning Representations,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024b\)Prometheus 2: an open source language model specialized in evaluating other language models\.arXiv preprint arXiv:2405\.01535\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- H\. Li, Q\. Dong, J\. Chen, H\. Su, Y\. Hui, Y\. Shi, S\. Fang, X\. Zhu, Q\. Liu, and Y\. Liu \(2024\)LLMs\-as\-judges: a comprehensive survey on LLM\-based evaluation of NLP\.arXiv preprint arXiv:2402\.08862\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. Liu \(2023\)Generative judge for evaluating alignment\.arXiv preprint arXiv:2310\.05470\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- P\. Li, T\. Zrnic, and E\. Candès \(2025\)Robust sampling for active statistical inference\.arXiv preprint arXiv:2511\.08991\.Cited by:[§D\.1](https://arxiv.org/html/2607.16239#A4.SS1.p3.1)\.
- T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. Tu \(2024\)Encouraging divergent thinking in large language models through multi\-agent debate\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- A\. Miniukovich and K\. Figl \(2023\)The effect of prototypicality on webpage aesthetics, usability, and trustworthiness\.International Journal of Human\-Computer Studies179,pp\. 103103\.Cited by:[§1\.1](https://arxiv.org/html/2607.16239#S1.SS1.p4.1),[§3\.1](https://arxiv.org/html/2607.16239#S3.SS1.p1.1)\.
- R\. Mozer and L\. Miratrix \(2025\)More power to you: using machine learning to augment human coding for more efficient inference in text\-based randomized trials\.The Annals of Applied Statistics19\(1\),pp\. 440–464\.Cited by:[§2\.2](https://arxiv.org/html/2607.16239#S2.SS2.p2.1)\.
- Y\. Qian, S\. Zhang, Y\. Zhou, A\. Balakrishnan, J\. Li, S\. K\. Jauhar, A\. Kannan, and R\. Tian \(2025\)Enhancing LLM\-as\-a\-judge via multi\-agent collaboration\.arXiv preprint arXiv:2501\.05366\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- C\. Särndal, B\. Swensson, and J\. Wretman \(1992\)Model assisted survey sampling\.Springer,New York\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1),[§2\.3](https://arxiv.org/html/2607.16239#S2.SS3.p7.16)\.
- P\. Verga, S\. Hofstätter, S\. Althammer, Y\. Su, A\. Piktus, N\. Arkil, P\. Ferão, M\. Z\. Razavi, S\. Saeidi, S\. Ruder, and P\. Roth \(2024\)Replacing judges with juries: evaluating LLM generations with a panel of diverse models\.arXiv preprint arXiv:2404\.18796\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- P\. Wang, L\. Li, L\. Chen, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023a\)Large language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023b\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p2.1)\.
- Y\. Wang, Z\. Yu, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. Zhang \(2024\)PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.InInternational Conference on Learning Representations,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§3\.1](https://arxiv.org/html/2607.16239#S3.SS1.p2.1)\.
- A\. Zhang, L\. D\. Brown, and T\. T\. Cai \(2019\)Semi\-supervised inference: general theory and estimation of means\.The Annals of Statistics47\(5\),pp\. 2538–2566\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p1.1)\.
- L\. Zhu, X\. Wang, and X\. Wang \(2025\)JudgeLM: fine\-tuned large language models are scalable judges\.InInternational Conference on Learning Representations,Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p4.1)\.
- T\. Zrnic \(2024\)Cross\-prediction\-powered inference\.Proceedings of the National Academy of Sciences121\(26\),pp\. e2322083121\.Cited by:[§1\.3](https://arxiv.org/html/2607.16239#S1.SS3.p3.1)\.

## Appendix AAdditional Evaluation Results

### A\.1More on MQM evaluation

##### Importance of fitting the appropriate outcome model\.

Table[3](https://arxiv.org/html/2607.16239#A1.T3)reports pooled 5\-fold OOF metrics for predicting human MQM scores\. Features include per\-LLM mean scores and variance\-based uncertainty, with embeddings from a large text embedding model\. Because MQM scores are non\-negative with a substantial zero mass \(∼\\sim13% pooled, up to 29% for human translations\), we compare the standard Ridge model against a*Hurdle model*\(logistic gate predictingP​\(MQM\>0\)P\(\\text\{MQM\}\>0\)combined with Ridge on the positive subset\)\. We also compare two variance encodings: the originallog⁡\(Var\+ϵ\)\\log\(\\mathrm\{Var\}\+\\epsilon\)transform versusVar\\sqrt\{\\mathrm\{Var\}\}with a binary has\-variance indicator, since 57–67% of items have zero LLM variance \(all raters agree\), creating an artificial bimodal cliff atlog⁡\(10−8\)≈−18\.4\\log\(10^\{\-8\}\)\\approx\-18\.4in the log encoding\.

Table 3:MQM pooled judge\-model ablation \(5\-fold OOF, all 10 systems,n=14,180n\{=\}14\{,\}180\)\. The Hurdle model withVar\\sqrt\{\\mathrm\{Var\}\}encoding achieves the bestR2R^\{2\}while eliminating negative predictions\.ConfigurationR2R^\{2\}SpearmanFull \(sqrt, hurdle\)0\.3690\.657Full \(sqrt, ridge\)0\.3550\.646Full \(log, hurdle\)0\.3650\.646Full \(log, ridge\)0\.3510\.634LLM scores only \(sqrt, ridge\)0\.2870\.557Embedding\-only ridge0\.1670\.497Intercept\-only baseline0\.0000\.000TheVar\\sqrt\{\\mathrm\{Var\}\}encoding resolves pathological failures on high\-quality systems: forHuman\-B\.0\(29% zeros\), Ridge with log\-variance yieldsR2=−0\.14R^\{2\}=\-0\.14due to extreme predictions driven by the−18\.4\-18\.4floor, whereasVar\\sqrt\{\\mathrm\{Var\}\}with Hurdle achievesR2=0\.24R^\{2\}=0\.24\. For MT systems with fewer zeros, both encodings perform similarly\.

##### Per\-system ablation\.

Table[4](https://arxiv.org/html/2607.16239#A1.T4)reports the Hurdle model withVar\\sqrt\{\\mathrm\{Var\}\}encoding across all ten translation systems and the pooled corpus\. The full model \(LLM \+ uncertainty \+ embeddings\) consistently dominates the LLM\-only and embedding\-only ablations\. LLM scores alone explain much of the variance for MT systems \(R2R^\{2\}0\.36–0\.44\) but substantially less for human translations \(R2R^\{2\}0\.15–0\.21\), where inter\-rater noise is higher\. Embeddings alone provide a complementary signal \(R2R^\{2\}0\.10–0\.20\) that is especially valuable for the human\-translation systems where LLM scores are weakest\. Combining both sources yields consistent gains across all systems\.

Table 4:Per\-system MQM judge\-model performance \(5\-fold OOF, Hurdle model,Var\\sqrt\{\\mathrm\{Var\}\}encoding\)\.R2R^\{2\}and Spearmanρ\\rhoare reported for each feature ablation\.FullLLM onlyEmbedding onlyIntercept onlySystemR2R^\{2\}ρ\\rhoR2R^\{2\}ρ\\rhoR2R^\{2\}ρ\\rhoR2R^\{2\}ρ\\rhoHuman\-A\.00\.2730\.5540\.1910\.4270\.1500\.4480\.0000\.000Human\-B\.00\.2360\.5160\.1500\.3590\.1390\.4520\.0000\.000Human\-P\.00\.3280\.5810\.2070\.4780\.1970\.4910\.0000\.000Huoshan\_Translate0\.3850\.6600\.3840\.6440\.1050\.3980\.0000\.000OPPO0\.4250\.6910\.3880\.6490\.1600\.4700\.0000\.000Online\-A0\.4410\.7030\.3910\.6910\.1120\.3750\.0000\.000Online\-B0\.3690\.6460\.3990\.6410\.1160\.3650\.0000\.000Tencent\_Translation0\.4350\.6900\.3980\.6420\.1600\.4610\.0000\.000Tohoku\-AIP\-NTT0\.3770\.6560\.3600\.6360\.1300\.4050\.0000\.000eTranslation0\.4490\.7070\.4360\.6730\.1300\.4410\.0000\.000Pooled0\.3690\.6570\.2870\.5570\.1670\.4970\.0000\.000

### A\.2WebDesign evaluation

For the Universities subset, we ran the same judge ablation protocol across six outcomes \(Aesthetics, Trustworthiness, Typicality, Exemplar Goodness, Family Resemblance, Usability\), including a constant intercept\-only baseline\. Table[5](https://arxiv.org/html/2607.16239#A1.T5)reports average 5\-fold OOF metrics across outcomes\. The full hybrid model \(VLM scores \+ uncertainty \+ SigLIP intercept\) achieves the best overall performance, improving over both the VLM\-only and SigLIP\-only ablations\. As uncalibrated baselines, using raw VLM scores directly yields highly negativeR2R^\{2\}\(due to a scale mismatch: human ratings are z\-scored while VLM scores are on their original integer scale\), though Spearman correlation remains moderate \(0\.56\), confirming that outcome\-model calibration is essential in this domain\.

Table 5:WebDesign \(Universities\) judge\-model ablation results\. Values are averaged across six outcomes\. Lower RMSE is better; higherR2R^\{2\}and Spearman are better\. Uncalibrated rows use raw VLM scores directly \(no outcome\-model fitting\); note that human ratings are z\-scored while VLM scores are on their original integer scale, so scale mismatch explains the negativeR2R^\{2\}\.ConfigurationTypeAvg\.R2R^\{2\}Avg\. RMSEAvg\. SpearmanVLM scores \+ uncertainty \+ SigLIPCalibrated0\.5550\.4620\.727VLM scores \+ uncertainty \(no SigLIP\)Calibrated0\.3900\.5440\.591SigLIP\-onlyCalibrated0\.4970\.4920\.690Intercept\-only constantCalibrated−\-0\.0010\.699−\-0\.047Raw avg\. of all judgesUncalibrated−\-9\.7192\.2580\.561Raw single\-judge avg\.Uncalibrated−\-10\.1512\.2910\.340##### Simulation on summary metrics\.

We evaluate each configuration as an AEE outcome model under uniform sampling budgetsη∈\{0\.05,…,0\.30\}\\eta\\in\\\{0\.05,\\ldots,0\.30\\\}across all six perceptual outcomes\. In each trial a fractionη\\etaof websites is sampled uniformly; the AEE estimator combines population\-level predictions from the judge model with an in\-sample bias correction\. Figures[7](https://arxiv.org/html/2607.16239#A1.F7)and[8](https://arxiv.org/html/2607.16239#A1.F8)report estimator and outcome\-model quality per outcome and on average\.

##### Estimator quality \(Figure[7](https://arxiv.org/html/2607.16239#A1.F7)\)\.

All models are approximately unbiased across budgets\. The full hybrid model \(VLM scores \+ uncertainty \+ SigLIP intercept\) achieves the lowest standard deviation and RMSE for most outcomes, confirming that the variance\-reduction gains from the judge model carry through to the AEE estimator\. Perceptual outcomes vary considerably in difficulty: Aesthetics and Usability are better predicted than Exemplar Goodness and Typicality, and this ordering is reflected in estimator efficiency\. The AVG panel shows that the full model consistently outperforms both the VLM\-only and SigLIP\-only ablations when averaged across outcomes\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_AE.png)\(a\)Aesthetics
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_TRU.png)\(b\)Trustworthiness
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_TYP.png)\(c\)Typicality
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_EXMPL.png)\(d\)Exemplar Goodness
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_US.png)\(e\)Usability
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_bias_sd_rmse_AVG.png)\(f\)Average \(across all outcomes\)

Figure 7:WebDesign \(Universities\) AEE estimator quality: bias, SD, and RMSE vs\. sampling budgetη\\eta, per perceptual outcome and averaged\. Lower is better\. The full hybrid model achieves the lowest estimator variance for most outcomes\.
##### Outcome model quality \(Figure[8](https://arxiv.org/html/2607.16239#A1.F8)\)\.

Figure[8](https://arxiv.org/html/2607.16239#A1.F8)traces the out\-of\-fold predictive performance of each outcome model as a function ofη\\eta\. Even at the smallest budget \(η=0\.05\\eta\{=\}0\.05, roughly 30–40 websites\), the full model achieves nontrivialR2R^\{2\}and Spearman correlation for several outcomes, demonstrating that VLM judge scores and SigLIP embeddings together provide a strong prior that makes efficient use of limited human labels\. Outcome\-model quality improves consistently with budget, and the full hybrid model leads onR2R^\{2\}and Spearman for most outcomes, matching the pattern observed in the static ablation \(Table[5](https://arxiv.org/html/2607.16239#A1.T5)\)\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_AE.png)\(a\)Aesthetics
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_TRU.png)\(b\)Trustworthiness
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_TYP.png)\(c\)Typicality
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_EXMPL.png)\(d\)Exemplar Goodness
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_US.png)\(e\)Usability
![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/webdesign_uniform_dr_outcome_r2_rmse_spearman_AVG.png)\(f\)Average \(across all outcomes\)

Figure 8:WebDesign \(Universities\) outcome model quality vs\. sampling budgetη\\eta: out\-of\-foldR2R^\{2\}, RMSE, and Spearmanρ\\rhoper perceptual outcome and averaged\. HigherR2R^\{2\}/Spearman and lower RMSE are better\. The full model leads on most outcomes\.

## Appendix BModel\-Judge Scoring Protocol

For each item, we query a panel of language or vision\-language judges through a unified API endpoint\. Prompts enforce strict scalar outputs to reduce parse ambiguity and improve comparability across models\. We use fixed templates:

Essay scoring \(LLMs\)\.

> You are an essay grader\. You will be given a student essay and its writing prompt\. Rate the essay’s holistic quality on a scale of 1 to 6: 1 = Very poor, 2 = Poor, 3 = Below average, 4 = Above average, 5 = Good, 6 = Excellent\. You MUST respond with ONLY a single digit \(1, 2, 3, 4, 5, or 6\)\. No explanation, no reasoning, no other text \- just the number\.

> Essay prompt: \{prompt\_name\} Essay: \{full\_text\}

Each judge receives this system and user text; we take the first substring matching a digit in\{1,…,6\}\\\{1,\\dots,6\\\}as the score \(same rule as in the scoring script\)\.

Web\-design scoring \(VLMs\)\.

> You are a strict visual evaluator\. Return only the requested six lines\.

> Rate this screenshot from a \{domain\_label\} on six dimensions\. Return ONLY six integer scores in \[\-3, \-2, \-1, 0, 1, 2, 3\], one per line, exactly: AE: <score\> TRU: <score\> TYP: <score\> EXMPL: <score\> AVG: <score\> US: <score\> Definitions: \- AE \(Aesthetics\): visually appealing overall design\. \- TRU \(Trustworthiness\): appears credible and trustworthy\. \- TYP \(Typicality\): \{prompt\_typ\} \- EXMPL \(Exemplar Goodness\): \{prompt\_exmpl\} \- AVG \(Family Resemblance\): \{prompt\_avg\} \- US \(Usability\): easy to navigate and understand\.

The placeholders \{domain\_label\}, \{prompt\_typ\}, \{prompt\_exmpl\}, and \{prompt\_avg\} are filled from a domain table \(university/college, commercial\-bank, online fashion\-shopping, online homeware\-shopping\) matching our dataset splits\. Each judge receives this user text together with one screenshot \(system role as above\); responses are parsed as six lines of the formAE:…\\ldots,TRU:…\\ldots, etc\.

Machine translation scoring \(LLMs\)\.

> You are a professional MQM annotator following the official WMT MQM human evaluation guidelines at the segment level\. Identify up to five target\-side errors, each with a category \(Accuracy, Fluency, Terminology, Style, Locale conventions, or Other, including subcategories such as Mistranslation or Grammar; also Source error or Non\-translation when applicable\), a severity \(Major, Minor, or Neutral\), and the exact span from the source or hypothesis\. If the translation is too garbled or unrelated to the source, return exactly one error: Non\-translation with severity Major\. Source errors are listed separately and do not count toward the five target\-error limit\. Do not double\-count the same underlying issue; when multiple issues overlap one span, keep only the most severe; when severities tie, prefer Accuracy\>\>Fluency\>\>Terminology\>\>Style\>\>Locale conventions\>\>Other\. Be conservative and return JSON only \(no prose outside the object\)\.

> \[Source\] \{source\} \[Hypothesis\] \{hypothesis\} Return JSON only: a single object with an errors array; each entry has string fields category, severity, and span\.

Each judge receives the source segment and the system hypothesis \(machine translation\) and returns structured segment\-level MQM annotations in that JSON format\.

For every API response, we log token\-level statistics \(including per\-token log probabilities and top alternatives when available\) and derive uncertainty summaries such as mean token entropy and perplexity\. Failed requests are retried with bounded concurrency and backoff; unresolvable failures are markedNAand excluded from successful\-completion counts during resume runs\.

## Appendix CAdditional results

### C\.1Properties of the estimating equation estimator

In this section, we will present the theoretical properties of the estimating equation estimator\.

###### Theorem 1\(Consistency and asymptotic normality of the AEE estimator\)\.

Letθ⋆∈Θ⊂ℝp\\theta^\{\\star\}\\in\\Theta\\subset\{\\mathbb\{R\}\}^\{p\}be the unique solution to

Ψ​\(θ,γ⋆\):=𝔼​\{m​\(Y,S,X;θ,γ⋆\)\}=0\.\\Psi\(\\theta,\\gamma^\{\\star\}\):=\{\\mathbb\{E\}\\left\\\{m\(Y,S,X;\\theta,\\gamma^\{\\star\}\)\\right\\\}\}=0\.Assume:

1. 1\.Identification\.θ⋆\\theta^\{\\star\}is the unique zero ofΨ​\(θ,γ⋆\)\\Psi\(\\theta,\\gamma^\{\\star\}\)in a neighborhood of the truth\.
2. 2\.Positivity and MAR\.There existsϵ\>0\\epsilon\>0such thatπ⋆​\(X\)≥ϵ\\pi^\{\\star\}\(X\)\\geq\\epsilonalmost surely, andS⟂Y∣XS\\perp Y\\mid X\.
3. 3\.Smoothness\.The mapθ↦m​\(Y,S,X;θ,γ\)\\theta\\mapsto m\(Y,S,X;\\theta,\\gamma\)is continuously differentiable nearθ⋆\\theta^\{\\star\}, and A:=𝔼​\{∇θm​\(Y,S,X;θ⋆,γ⋆\)\}A:=\{\\mathbb\{E\}\\left\\\{\\nabla\_\{\\theta\}m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\\right\\\}\}exists and is nonsingular\.
4. 4\.Nuisance estimation\.‖γ^−γ⋆‖=op​\(1\)\\\|\\widehat\{\\gamma\}\-\\gamma^\{\\star\}\\\|=o\_\{p\}\(1\)and 𝔼N​\[m​\(Y,S,X;θ⋆,γ^\)\]−𝔼N​\[m​\(Y,S,X;θ⋆,γ⋆\)\]=op​\(N−1/2\)\.\\mathbb\{E\}\_\{N\}\\\!\\left\[m\(Y,S,X;\\theta^\{\\star\},\\widehat\{\\gamma\}\)\\right\]\-\\mathbb\{E\}\_\{N\}\\\!\\left\[m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\\right\]=o\_\{p\}\(N^\{\-1/2\}\)\.This holds, for example, under Neyman orthogonality together with standard consistency rates for the nuisance estimators\.
5. 5\.Empirical process control\.A uniform law of large numbers and a central limit theorem hold for the relevant class of score functions in a neighborhood of\(θ⋆,γ⋆\)\(\\theta^\{\\star\},\\gamma^\{\\star\}\)\.

Ifθ^\\widehat\{\\theta\}satisfies

𝔼N​\[m​\(Y,S,X;θ^,γ^\)\]=op​\(N−1/2\),\\mathbb\{E\}\_\{N\}\\\!\\left\[m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)\\right\]=o\_\{p\}\(N^\{\-1/2\}\),thenθ^​→𝑝​θ⋆\\widehat\{\\theta\}\\overset\{p\}\{\\to\}\\theta^\{\\star\}\. Moreover,

N​\(θ^−θ⋆\)=−A−1​1N​∑i=1Nm​\(Yi,Si,Xi;θ⋆,γ⋆\)\+op​\(1\),\\sqrt\{N\}\(\\widehat\{\\theta\}\-\\theta^\{\\star\}\)=\-A^\{\-1\}\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{i=1\}^\{N\}m\(Y\_\{i\},S\_\{i\},X\_\{i\};\\theta^\{\\star\},\\gamma^\{\\star\}\)\+o\_\{p\}\(1\),and hence

N​\(θ^−θ⋆\)↝𝒩​\(0,Σθ\),Σθ=A−1​B​A−T,\\sqrt\{N\}\(\\widehat\{\\theta\}\-\\theta^\{\\star\}\)\\rightsquigarrow\\mathcal\{N\}\(0,\\Sigma\_\{\\theta\}\),\\qquad\\Sigma\_\{\\theta\}=A^\{\-1\}BA^\{\-T\},where

B=𝔼​\{m​\(Y,S,X;θ⋆,γ⋆\)​m​\(Y,S,X;θ⋆,γ⋆\)⊤\}\.B=\{\\mathbb\{E\}\\left\\\{m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)^\{\\top\}\\right\\\}\}\.

###### Corollary 1\(Consistency of the sandwich variance estimator\)\.

If, in addition,A^​→𝑝​A\\widehat\{A\}\\overset\{p\}\{\\to\}AandB^​→𝑝​B\\widehat\{B\}\\overset\{p\}\{\\to\}B, then

Var⁡\{θ^\}^=1n​A^−1​B^​A^−T\\widehat\{\{\\operatorname\{Var\}\\left\\\{\\widehat\{\\theta\}\\right\\\}\}\}=\\frac\{1\}\{n\}\\widehat\{A\}^\{\-1\}\\widehat\{B\}\\widehat\{A\}^\{\-T\}is a consistent estimator of the asymptotic covariance ofθ^\\widehat\{\\theta\}\.

### C\.2Proof of Theorem[1](https://arxiv.org/html/2607.16239#Thmtheorem1)

###### Proof\.

Consistency follows from standardZZ\-estimation arguments\. By the uniform law of large numbers and nuisance consistency,

supθ∈Θ‖𝔼N​\[m​\(Y,S,X;θ,γ^\)\]−Ψ​\(θ,γ⋆\)‖=op​\(1\)\.\\sup\_\{\\theta\\in\\Theta\}\\left\\\|\\mathbb\{E\}\_\{N\}\[m\(Y,S,X;\\theta,\\widehat\{\\gamma\}\)\]\-\\Psi\(\\theta,\\gamma^\{\\star\}\)\\right\\\|=o\_\{p\}\(1\)\.SinceΨ​\(θ,γ⋆\)\\Psi\(\\theta,\\gamma^\{\\star\}\)has a unique zero atθ⋆\\theta^\{\\star\}, any approximate empirical root must converge in probability toθ⋆\\theta^\{\\star\}\.

For asymptotic normality, expand the empirical moment around\(θ⋆,γ⋆\)\(\\theta^\{\\star\},\\gamma^\{\\star\}\):

0=𝔼N​\[m​\(Y,S,X;θ^,γ^\)\]=𝔼N​\[m​\(Y,S,X;θ⋆,γ⋆\)\]\+A​\(θ^−θ⋆\)\+rN,0=\\mathbb\{E\}\_\{N\}\[m\(Y,S,X;\\widehat\{\\theta\},\\widehat\{\\gamma\}\)\]=\\mathbb\{E\}\_\{N\}\[m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\]\+A\(\\widehat\{\\theta\}\-\\theta^\{\\star\}\)\+r\_\{N\},wherern=op​\(n−1/2\)r\_\{n\}=o\_\{p\}\(n^\{\-1/2\}\)by differentiability, stochastic equicontinuity, and the assumed negligible first\-order effect of nuisance estimation\. Rearranging gives

N​\(θ^−θ⋆\)=−A−1​N​𝔼N​\[m​\(Y,S,X;θ⋆,γ⋆\)\]\+op​\(1\)\.\\sqrt\{N\}\(\\widehat\{\\theta\}\-\\theta^\{\\star\}\)=\-A^\{\-1\}\\sqrt\{N\}\\,\\mathbb\{E\}\_\{N\}\[m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\]\+o\_\{p\}\(1\)\.The multivariate central limit theorem yields

N​𝔼N​\[m​\(Y,S,X;θ⋆,γ⋆\)\]↝𝒩​\(0,B\),\\sqrt\{N\}\\,\\mathbb\{E\}\_\{N\}\[m\(Y,S,X;\\theta^\{\\star\},\\gamma^\{\\star\}\)\]\\rightsquigarrow\\mathcal\{N\}\(0,B\),and Slutsky’s theorem implies the stated Gaussian limit with covarianceA−1​B​A−TA^\{\-1\}BA^\{\-T\}\. ∎

## Appendix DExtension: constrained Weighting Optimization for Adaptive Sampling

### D\.1Framework

Suppose we can learn a good outcome model from the one round of labeling, we can then solve a constrained weighting objective to optimize future sampling schema under a fixed budget when the goal is to estimate the population summary metrics\. We use population means as an example\. Formally, we want to solve the following problem:

minσ​∑iri2e​\(xi\)subject to1N​∑ie​\(xi\)≤η,\\min\_\{\\sigma\}\\sum\_\{i\}\\frac\{r\_\{i\}^\{2\}\}\{e\(x\_\{i\}\)\}\\quad\\text\{subject to\}\\quad\\frac\{1\}\{N\}\\sum\_\{i\}e\(x\_\{i\}\)\\leq\\eta,wheree​\(xi\)=π\+\(1−π\)​σ​\(b\+xi⊤​β\)e\(x\_\{i\}\)=\\pi\+\(1\-\\pi\)\\sigma\(b\+x\_\{i\}^\{\\top\}\\beta\)is the inclusion probability for itemii, andbbandβ\\betaare the parameters of the outcome model\. We use an augmented Lagrangian approach with a hinge\-squared penalty for constraint violations\. This formulation encourages larger weights on high\-residual items while enforcing a global sampling\-rate cap\.

We propose the following finite\-population labeling workflow:

1. 1\.Draw an initial random sample \(stage 1\), collect human labels, and fit an outcome modelff\.
2. 2\.Fit a propensity modelσ\\sigmafrom features \(primarily embedding PCs\) to approximate hardness\-informed inclusion scores\.
3. 3\.Sample additional items adaptively \(stage 2\) using probabilities derived fromσ\\sigmaunder an overall budgetη\\eta\.
4. 4\.Estimate the population mean using the AEE estimator that combines inverse\-propensity correction with outcome\-model augmentation\.

This objective prioritizes high\-residual items while enforcing a global sampling\-rate cap\. This is relevant to active statistical inference literature:\(Liet al\.,[2025](https://arxiv.org/html/2607.16239#bib.bib16); Hamiltonet al\.,[2025](https://arxiv.org/html/2607.16239#bib.bib15)\)\.

### D\.2Variance characterization of the AEE estimator

First, let’s think about the variance of the AEE estimator\. For simplicity, let’s assume that the outcomes are i\.i\.d\. draws from a super population\(y,x\)\(y,x\)from some distributionPP\. Then, we can write the AEE estimator as

μ^=1N​∑iSi​\(yi−f​\(xi\)\)e​\(xi\)\+f​\(xi\)\.\\displaystyle\\widehat\{\\mu\}=\\frac\{1\}\{N\}\\sum\_\{i\}\\frac\{S\_\{i\}\(y\_\{i\}\-f\(x\_\{i\}\)\)\}\{e\(x\_\{i\}\)\}\+f\(x\_\{i\}\)\.Due to independence, the variance ofμ^\\widehat\{\\mu\}is given by

1N​Var⁡\{S​\(y−f​\(x\)\)e​\(x\)\+f​\(x\)\}\.\\displaystyle\\frac\{1\}\{N\}\{\\operatorname\{Var\}\\left\\\{\\frac\{S\(y\-f\(x\)\)\}\{e\(x\)\}\+f\(x\)\\right\\\}\}\.Now we compute the variance term\. First, notice the following decomposition:

S​\(y−f​\(x\)\)e​\(x\)\+f​\(x\)\\displaystyle\\frac\{S\(y\-f\(x\)\)\}\{e\(x\)\}\+f\(x\)=\\displaystyle=S​\(y−f⋆​\(x\)\)e​\(x\)⏟Term I\+f⋆​\(x\)⏟Term II\+\(S−e​\(x\)\)​\(f⋆​\(x\)−f​\(x\)\)e​\(x\)⏟Term III,\\displaystyle\\underbrace\{\\frac\{S\(y\-f^\{\\star\}\(x\)\)\}\{e\(x\)\}\}\_\{\\text\{Term I\}\}\+\\underbrace\{f^\{\\star\}\(x\)\}\_\{\\text\{Term II\}\}\+\\underbrace\{\\frac\{\(S\-e\(x\)\)\(f^\{\\star\}\(x\)\-f\(x\)\)\}\{e\(x\)\}\}\_\{\\text\{Term III\}\},Note that Term I, II and III are mutually orthogonal, i\.e\.

Cov⁡\{Term I,Term II\}=\\displaystyle\{\\operatorname\{Cov\}\\left\\\{\\text\{Term I\},\\text\{Term II\}\\right\\\}\}=0\\displaystyle 0Cov⁡\{Term I,Term III\}=\\displaystyle\{\\operatorname\{Cov\}\\left\\\{\\text\{Term I\},\\text\{Term III\}\\right\\\}\}=0\\displaystyle 0Cov⁡\{Term II,Term III\}=\\displaystyle\{\\operatorname\{Cov\}\\left\\\{\\text\{Term II\},\\text\{Term III\}\\right\\\}\}=0\.\\displaystyle 0\.
Therefore, the variance ofμ^\\widehat\{\\mu\}is given by

1N​Var⁡\{Term I\}\+1N​Var⁡\{Term II\}\+1N​Var⁡\{Term III\}\.\\displaystyle\\frac\{1\}\{N\}\{\\operatorname\{Var\}\\left\\\{\\text\{Term I\}\\right\\\}\}\+\\frac\{1\}\{N\}\{\\operatorname\{Var\}\\left\\\{\\text\{Term II\}\\right\\\}\}\+\\frac\{1\}\{N\}\{\\operatorname\{Var\}\\left\\\{\\text\{Term III\}\\right\\\}\}\.Now we compute the variance of each term\.

Var⁡\{Term I\}=𝔼​\{σ∗2​\(x\)e​\(x\)\}\.\\displaystyle\{\\operatorname\{Var\}\\left\\\{\\text\{Term I\}\\right\\\}\}=\{\\mathbb\{E\}\\left\\\{\\frac\{\\sigma^\{\*2\}\(x\)\}\{e\(x\)\}\\right\\\}\}\.For Term II, we have

Var⁡\{Term II\}=𝔼​\{\(f⋆​\(x\)−μ\)2\}\.\\displaystyle\{\\operatorname\{Var\}\\left\\\{\\text\{Term II\}\\right\\\}\}=\{\\mathbb\{E\}\\left\\\{\(f^\{\\star\}\(x\)\-\\mu\)^\{2\}\\right\\\}\}\.For Term III, we have

Var⁡\{Term III\}=𝔼​\{\(1−e​\(x\)\)​\(f⋆​\(x\)−f​\(x\)\)2e​\(x\)\}\.\\displaystyle\{\\operatorname\{Var\}\\left\\\{\\text\{Term III\}\\right\\\}\}=\{\\mathbb\{E\}\\left\\\{\\frac\{\(1\-e\(x\)\)\(f^\{\\star\}\(x\)\-f\(x\)\)^\{2\}\}\{e\(x\)\}\\right\\\}\}\.
Therefore we summarize two forms of the variance of the AEE estimator:

- •Form 1: Var⁡\{μ^\}=1N​𝔼​\{σ∗2​\(x\)e​\(x\)\+\(f⋆​\(x\)−μ\)2\+\(1−e​\(x\)\)​\(f⋆​\(x\)−f​\(x\)\)2e​\(x\)\}\.\\displaystyle\{\\operatorname\{Var\}\\left\\\{\\widehat\{\\mu\}\\right\\\}\}=\\frac\{1\}\{N\}\{\\mathbb\{E\}\\left\\\{\\frac\{\\sigma^\{\*2\}\(x\)\}\{e\(x\)\}\+\(f^\{\\star\}\(x\)\-\\mu\)^\{2\}\+\\frac\{\(1\-e\(x\)\)\(f^\{\\star\}\(x\)\-f\(x\)\)^\{2\}\}\{e\(x\)\}\\right\\\}\}\.
- •Form 2: Var⁡\{μ^\}=1N​𝔼​\{\(y−f​\(x\)\)2e​\(x\)\+\(f⋆​\(x\)−μ\)2−\(f⋆​\(x\)−f​\(x\)\)2\}\.\\displaystyle\{\\operatorname\{Var\}\\left\\\{\\widehat\{\\mu\}\\right\\\}\}=\\frac\{1\}\{N\}\{\\mathbb\{E\}\\left\\\{\\frac\{\(y\-f\(x\)\)^\{2\}\}\{e\(x\)\}\+\(f^\{\\star\}\(x\)\-\\mu\)^\{2\}\-\(f^\{\\star\}\(x\)\-f\(x\)\)^\{2\}\\right\\\}\}\.

##### What are we actually optimizing when we optimize the propensity score?

From Form 1, we can see that given a fixed propensity model, we can benefit from fitting a better outcome model to reduce the variance of the AEE estimator\.

From Form 2, we can see that given a fixed outcome model, we can benefit from fitting a better propensity model to reduce the variance of the AEE estimator\. More importantly, the best choice of propensity model should be proportional to the square root of the conditional quadratic error of the outcome model\. To show this, let’s try to optimize the propensity model based on Form 2\. Define the conditional squared error as:

σf2​\(x\)=𝔼​\{\(y−f​\(x\)\)2∣x\}\.\\displaystyle\\sigma^\{2\}\_\{f\}\(x\)=\{\\mathbb\{E\}\\left\\\{\(y\-f\(x\)\)^\{2\}\\mid x\\right\\\}\}\.Whenf=f⋆f=f^\{\\star\}, we haveσf2​\(x\)=σ∗2​\(x\)\\sigma^\{2\}\_\{f\}\(x\)=\\sigma^\{\*2\}\(x\)\. By Cauchy\-Schwarz inequality, we have

𝔼​\{σf​\(x\)\}2≤𝔼​\{σf2​\(x\)e​\(x\)\}⋅𝔼​\{e​\(x\)\}≤η​𝔼​\{σf2​\(x\)e​\(x\)\}\.\\displaystyle\{\\mathbb\{E\}\\left\\\{\\sigma\_\{f\}\(x\)\\right\\\}\}^\{2\}\\leq\{\\mathbb\{E\}\\left\\\{\\frac\{\\sigma^\{2\}\_\{f\}\(x\)\}\{e\(x\)\}\\right\\\}\}\\cdot\{\\mathbb\{E\}\\left\\\{e\(x\)\\right\\\}\}\\leq\\eta\{\\mathbb\{E\}\\left\\\{\\frac\{\\sigma^\{2\}\_\{f\}\(x\)\}\{e\(x\)\}\\right\\\}\}\.Therefore, we have the following lower bound:

𝔼​\{σf2​\(x\)e​\(x\)\}≥𝔼​\{σf​\(x\)\}2η\.\\displaystyle\{\\mathbb\{E\}\\left\\\{\\frac\{\\sigma^\{2\}\_\{f\}\(x\)\}\{e\(x\)\}\\right\\\}\}\\geq\\frac\{\{\\mathbb\{E\}\\left\\\{\\sigma\_\{f\}\(x\)\\right\\\}\}^\{2\}\}\{\\eta\}\.The equality holds when

e​\(x\)∝σf​\(x\)\.\\displaystyle e\(x\)\\propto\\sigma\_\{f\}\(x\)\.That is, the best choice of propensity model should be proportional to the square root of the conditional quadratic error of the outcome model\. This is consistent with the intuition that we should sample more items that are harder to predict\.

### D\.3Synthetic experiments on summary metrics

To isolate the benefit of adaptive sampling under controlled heteroskedasticity, we construct a synthetic labeling environment that inherits the covariate structure of the real PERSUADE corpus but replaces human scores with outcomes drawn from a known data\-generating process \(DGP\)\.

#### D\.3\.1Data\-Generating Process

Letei∈ℝde\_\{i\}\\in\\mathbb\{R\}^\{d\}denote the OpenAI text embedding for essayii, and letℓ¯i\\bar\{\\ell\}\_\{i\}be the average LLM\-judge score across the model panel\. We compute principal components from the standardised embedding matrix and define the conditional mean

μi=∑j=15PCi​j\+ℓ¯i,\\mu\_\{i\}\\;=\\;\\sum\_\{j=1\}^\{5\}\\mathrm\{PC\}\_\{ij\}\\;\+\\;\\bar\{\\ell\}\_\{i\},so that the outcome depends on both intrinsic essay features \(through the first five PCs\) and the aggregate model\-judge signal\. Conditional variance is heteroskedastic in the first principal component:

σi2=0\.1​exp⁡\(λ​PCi​1\),\\sigma\_\{i\}^\{2\}\\;=\\;0\.1\\,\\exp\\bigl\(\\lambda\\,\\mathrm\{PC\}\_\{i1\}\\bigr\),withλ\>0\\lambda\>0governing the severity of heteroskedasticity\. For each item we draw

Yi∼𝒩​\(μi,σi2\)\.Y\_\{i\}\\;\\sim\\;\\mathcal\{N\}\\\!\\bigl\(\\mu\_\{i\},\\;\\sigma\_\{i\}^\{2\}\\bigr\)\.This DGP makes high\-PC1 items intrinsically noisier while low\-PC1 items remain relatively clean, creating a directional heteroskedasticity that the propensity model can exploit by selectively oversampling the noisy tail\.

#### D\.3\.2Simulation Protocol

We sweepλ∈\{0\.5,1\.0,1\.5,2\.0,2\.5,3\.0\}\\lambda\\in\\\{0\.5,1\.0,1\.5,2\.0,2\.5,3\.0\\\}to trace the effect of increasing heteroskedasticity\. For each value ofλ\\lambda, the synthetic outcomes are generated once and held fixed across Monte Carlo trials\. Each trial then executes one of three estimation strategies under a common total labeling budgetη\\eta:

1. 1\.Baseline 1 \(Direct Sample Mean\)\.Draw⌊η​N⌋\\lfloor\\eta N\\rflooritems uniformly at random, collect their labels, and report the sample mean\.
2. 2\.Baseline 2 \(One\-Stage estimation\)\.Draw⌊η​N⌋\\lfloor\\eta N\\rflooritems uniformly, fit an outcome modelf^\\widehat\{f\}by ridge regression on embedding PCs, and form an AEE estimator with constant propensitye​\(xi\)=ηe\(x\_\{i\}\)=\\eta\.
3. 3\.Two\-Stage Adaptive estimation\.Draw a stage\-1 pilot sample at rateπ<η\\pi<\\eta\. Fit an outcome modelf^\\widehat\{f\}and compute squared residualsri2=\(Yi−f^​\(xi\)\)2r\_\{i\}^\{2\}=\(Y\_\{i\}\-\\widehat\{f\}\(x\_\{i\}\)\)^\{2\}on the pilot\. Fit a propensity modelσ​\(xi\)\\sigma\(x\_\{i\}\)on embedding PCs to predict hardness, then draw stage\-2 items with inclusion probabilitye​\(xi\)=π\+\(1−π\)​σ​\(xi\)e\(x\_\{i\}\)=\\pi\+\(1\-\\pi\)\\sigma\(x\_\{i\}\)subject to a mean\-budget constraint1N​∑ie​\(xi\)≤η\\frac\{1\}\{N\}\\sum\_\{i\}e\(x\_\{i\}\)\\leq\\eta\. Combine both stages via an AEE estimator with item\-specific propensitiese​\(xi\)e\(x\_\{i\}\)\.

The propensity model solves the constrained weighting problem described above using an augmented Lagrangian optimizer, wheree​\(xi\)=π\+\(1−π\)​σ​\(b\+xi⊤​β\)e\(x\_\{i\}\)=\\pi\+\(1\-\\pi\)\\,\\sigma\(b\+x\_\{i\}^\{\\top\}\\beta\)and\(b,β\)\(b,\\beta\)are chosen to minimize∑iri2/e​\(xi\)\\sum\_\{i\}r\_\{i\}^\{2\}/e\(x\_\{i\}\)subject to1N​∑ie​\(xi\)≤η\\frac\{1\}\{N\}\\sum\_\{i\}e\(x\_\{i\}\)\\leq\\eta\. We allow separate PCA dimensionalities for the outcome model and the propensity model\.

Performance is assessed by the root mean squared error \(RMSE\) of each estimator relative to the true population meanμ=N−1​∑iμi\\mu=N^\{\-1\}\\sum\_\{i\}\\mu\_\{i\}, computed over repeated Monte Carlo trials\. We additionally report bias and standard deviation to diagnose whether improvements stem from variance reduction or bias correction\.

#### D\.3\.3Synthetic Simulation Results

Figure[9](https://arxiv.org/html/2607.16239#A4.F9)reports the bias, variance, and MSE of all three estimators asλ\\lambdaincreases\. The direct sample mean \(Baseline 1\) maintains near\-zero bias but its variance stays flat and high across allλ\\lambdavalues, as uniform sampling cannot adapt to the heteroskedastic noise structure\. The one\-stage AEE estimator \(Baseline 2\) substantially reduces variance relative to the direct mean at lowλ\\lambdaby leveraging the outcome model, but its variance grows steadily withλ\\lambdaas residual heteroskedasticity overwhelms the constant\-propensity correction\. The two\-stage adaptive AEE estimator achieves the lowest MSE across allλ\\lambdavalues\. Its advantage widens as heteroskedasticity increases: atλ=3\.0\\lambda=3\.0the two\-stage MSE is roughly half that of the one\-stage AEE estimator and over an order of magnitude below the direct mean\. All three estimators remain approximately unbiased throughout the sweep, confirming that the MSE gains are driven entirely by variance reduction through adaptive allocation of the labeling budget toward the noisy tail of the PC1 distribution\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/synthetic_bias_variance_mse_vs_lambda.png)Figure 9:Synthetic simulation: bias and MSE of three population\-mean estimators as a function of the heteroskedasticity parameterλ\\lambda\. Budgetη=0\.40\\eta=0\.40, pilot rateπ=0\.15\\pi=0\.15, 400 Monte Carlo trials perλ\\lambdavalue\.
#### D\.3\.4Sensitivity to Pilot Rate and Budget

We next fixλ=1\.5\\lambda=1\.5and study how estimation quality depends on the pilot rateπ\\piand the total budgetη\\eta\.

##### Varyingπ\\pi\(fixedη=0\.40\\eta=0\.40\)\.

Figure[10](https://arxiv.org/html/2607.16239#A4.F10)sweepsπ∈\{0\.05,0\.10,…,0\.30\}\\pi\\in\\\{0\.05,0\.10,\\ldots,0\.30\\\}\. Because the baselines use the full budgetη\\etain a single uniform draw, their MSE is invariant toπ\\pi\. The two\-stage AEE estimator is relatively stable across this range, with a slight optimum nearπ=0\.15\\pi=0\.15; very small pilots \(π=0\.05\\pi=0\.05\) leave the outcome model under\-trained, while large pilots \(π≥0\.25\\pi\\geq 0\.25\) consume too much budget before the adaptive stage can act\. All estimators remain approximately unbiased\.

##### Varyingη\\eta\(fixedπ=0\.15\\pi=0\.15\)\.

Figure[11](https://arxiv.org/html/2607.16239#A4.F11)sweepsη∈\{0\.20,0\.25,…,0\.45\}\\eta\\in\\\{0\.20,0\.25,\\ldots,0\.45\\\}\. As expected, MSE decreases monotonically with budget for all three methods\. The two\-stage AEE estimator maintains a consistent advantage over both baselines at every budget level: even at the tightest budget \(η=0\.20\\eta=0\.20\) it achieves lower MSE than the direct mean atη=0\.45\\eta=0\.45\. The relative improvement of two\-stage AEE estimator over one\-stage AEE estimator is largest at smallη\\eta, where the adaptive allocation of the scarce remaining budget after the pilot is most valuable\.

![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/synthetic_sweep_pi_bias_mse.png)Figure 10:Sensitivity to pilot rateπ\\piwith fixed budgetη=0\.40\\eta=0\.40andλ=1\.5\\lambda=1\.5\. Left: bias; right: MSE \(log scale\)\. 400 trials per setting\.![Refer to caption](https://arxiv.org/html/2607.16239v1/figures/synthetic_sweep_eta_bias_mse.png)Figure 11:Sensitivity to total budgetη\\etawith fixed pilot rateπ=0\.15\\pi=0\.15andλ=1\.5\\lambda=1\.5\. Left: bias; right: MSE \(log scale\)\. 400 trials per setting\.