Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

# the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Source: [https://arxiv.org/html/2606.15474](https://arxiv.org/html/2606.15474)
###### Abstract

Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down\. But the judge is itself a model behind an API, and a silent version bump or scoring\-prompt update changes how it scores — so every drift alarm is ambiguous between a worse product and a changed judge\. We resolve the ambiguity with a fixed, human\-labeled*anchor set*that the*current*judge re\-scores at a steady interleave, a second betting e\-process on the judge\-versus\-human gap, and a guard\-window rule returning a verdict in\{none,system,judge\}\\\{\\texttt\{none\},\\texttt\{system\},\\texttt\{judge\}\\\}\. We prove anytime\-validity, one\-way identification \(only the judge can move the anchors\), an attribution race whose design law is that the anchors must out\-run the main process they guard, and process orthogonality\. On two*real*judge changes, a silent version bump is detected as judge drift in60/6060/60runs with zero judge\-to\-system misattribution, and a contaminating strict\-prompt change is correctly attributed on110110of120120runs at guard width300300— while the industry\-default rollingzz\-test false\-alarms on75%75\\%of drift\-free streams\. Every experiment replicates on a second domain \(TL;DR summarization\) with nothing re\-tuned, and where the domains differ the differences are the ones the race predicts: the strict\-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect \(240/240240/240\)\. The monitor runs at≈0\.64\\approx 0\.64of the cost of strong\-judging every item, or0\.210\.21in a cheaper\-but\-deafer regime\.

## 1Introduction

Large language model products are now evaluated continuously by other large language models\. An LLM\-as\-judge\(Zheng et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib27)\)scores each interaction, a dashboard tracks the rolling mean, and an on\-call team is paged when the score drifts down\. The implicit contract is that the judge is a fixed yardstick, so a falling score means a falling product\. But the strong judge is itself a model behind an API: a version bump or a silent prompt\-and\-policy update can change how it scores, with no announcement and no change to the product at all\. When the monitor fires, the team faces a question the dashboard cannot answer —*which thing drifted, the system or the judge?*Acting on the wrong answer is expensive in both directions: rolling back a healthy product, or shipping a real regression because the alarm was dismissed as “just the judge\.”

This is not a hypothetical edge case; it is a property of the tools already in use\. Consider the de\-facto industry practice for watching a judge: re\-test the judge\-versus\-human gap on a rolling window with azz\-test atα=0\.05\\alpha=0\.05, every time a new observation arrives\. Run on the same drift\-free stream our anchor process consumes, that procedure false\-alarms on75%75\\%of streams where nothing ever changes, and1616of6060of its “detections” fire*before the change point even exists*\([Section˜5](https://arxiv.org/html/2606.15474#S5)\)\. Its apparent vigilance is the absence of error control, not sensitivity: a fixed\-α\\alphatest re\-run at every step has no validity guarantee at a data\-dependent stopping time\. Continuous monitoring demands*anytime\-valid*tooling — e\-values and test supermartingales that may be inspected at every step without inflating the false\-alarm rate\(Ville,[1939](https://arxiv.org/html/2606.15474#bib.bib22); Waudby\-Smith and Ramdas,[2022](https://arxiv.org/html/2606.15474#bib.bib24); Howard et al\.,[2022](https://arxiv.org/html/2606.15474#bib.bib12)\)\. Once those are in place, the detector reliably tells you*that*something drifted\.*Attribution*— whether it was the system or the judge — is the remaining hole, and it is the one we fill\.

#### The construction\.

We start from the prediction\-powered monitor a cost\-conscious team would deploy anyway: a cheap judgeμ\\muscored on*every*item, the expensive strong judgeyysampled under a budget, and a per\-stratum prediction\-powered e\-process\(Csillag et al\.,[2025b](https://arxiv.org/html/2606.15474#bib.bib8); Angelopoulos et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib1)\)that accrues anytime\-valid evidence of system drift \([Section˜3](https://arxiv.org/html/2606.15474#S3)\)\. To this we add one component: a fixed, human\-labeled*anchor set*𝒜\\mathcal\{A\}, held out before monitoring begins and re\-scored by the*current*judge at a steady interleave, driving a second e\-process that watches the judge’s gap to its frozen human labels \([Section˜4](https://arxiv.org/html/2606.15474#S4)\)\. A guard\-window rule combines the two stopping times into a verdict in\{none,system,judge\}\\\{\\texttt\{none\},\\texttt\{system\},\\texttt\{judge\}\\\}\. The construction turns on a deliberate asymmetry\. The anchors are frozen, so the product’s drift can never reach them — improve the system or wreck it, the gap to human labels on those fixed items does not move\. Only a change in the*judge*can move it\. The anchor process is therefore a one\-way mirror: it sees judge drift, and nothing else\.

#### Contributions\.

1. 1\.The anchor construction and guard\-window attribution rule\([Section˜4](https://arxiv.org/html/2606.15474#S4)\) — a human\-anchored, anytime\-valid e\-process that disentangles judge drift from system drift, emitting an explicit\{none,system,judge\}\\\{\\texttt\{none\},\\texttt\{system\},\\texttt\{judge\}\\\}verdict rather than an undifferentiated alarm\.
2. 2\.Four propositions\([Section˜4](https://arxiv.org/html/2606.15474#S4)\): anytime\-validity of the anchor family, one\-way identification \(only the judge can move the anchors\), the attribution race, and process orthogonality\. The race proposition exposes a design law — the anchor process must be provisioned to*out\-run*the main process it guards, since a contaminated reference can otherwise fire a spurious system alarm before the anchors catch up\.
3. 3\.Two*real*judge changes, not only synthetic drift \([Section˜5](https://arxiv.org/html/2606.15474#S5)\): a silent version bump \(gemini\-3\.1\-pro→\\to3\.5\-flash\) detected as judge drift in60/6060/60runs with0judge\-to\-system misattribution at the featured configuration, and a strict\-prompt policy change that genuinely contaminates the main monitor yet is correctly attributed on110110of120120contaminated runs at guard widthW=300W=300— both replicated exhibit\-for\-exhibit on a second domain \(TL;DR summarization\) with nothing re\-tuned\. The replication is a test of the theory, not only of robustness: the invariances transfer \(the same version bump leaves the same\+0\.03\+0\.03to\+0\.07\+0\.07signature in both domains\), and where the domains differ, the differences are the ones the propositions predict — the strict\-prompt shift is larger on summaries, so the anchors fire faster, attribution becomes perfect \(240/240240/240\), and the co\-provisioning band moves toward slower anchor rates, exactly the attribution race\.
4. 4\.Operational evidence that classical alternatives fail\([Section˜5](https://arxiv.org/html/2606.15474#S5)\): on the identical anchor stream, naive repeatedzz\-testing false\-alarms on75%75\\%of drift\-free streams, and a carefully calibrated Page–Hinkley detector holds its false\-alarm budget but detects the real lenient bump only8%8\\%of the time and requires held\-out calibration streams the e\-process does not\.
5. 5\.The cost\-aware foundation\([Section˜3](https://arxiv.org/html/2606.15474#S3)\): a stratified prediction\-powered monitor that detects localized blind\-spot regressions at roughly one third the cost of strong\-evaluating every item, with an e\-wealth escalation trigger that is far cheaper than covariate\-triggered prior rules \(which oversample to near full\-evaluation cost or read inert under a biased cheap judge\), while merely tying a matched fixed budget on the faintest drift\.

[Section˜2](https://arxiv.org/html/2606.15474#S2)situates these contributions;[Section˜3](https://arxiv.org/html/2606.15474#S3)builds the budgeted main monitor;[Section˜4](https://arxiv.org/html/2606.15474#S4)adds the anchor construction and its guarantees; and[Section˜5](https://arxiv.org/html/2606.15474#S5)validates all of them on two datasets and two real judge changes before[Section˜6](https://arxiv.org/html/2606.15474#S6)discusses limitations\.

## 2Related Work

#### Prediction\-powered inference and anytime\-valid testing\.

Prediction\-powered inference\(Angelopoulos et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib1),[2024](https://arxiv.org/html/2606.15474#bib.bib2)\)uses a cheap predictor to debias a small set of expensive labels, and has been extended in many directions: cross\-fitting\(Zrnic and Candès,[2024](https://arxiv.org/html/2606.15474#bib.bib28)\), frequentist–assisted–Bayes constructions\(Kilian et al\.,[2025](https://arxiv.org/html/2606.15474#bib.bib14)\), federated\-and\-bagged variants\(Cortinovis and Caron,[2025](https://arxiv.org/html/2606.15474#bib.bib6)\), conformal procedures\(Csillag et al\.,[2025a](https://arxiv.org/html/2606.15474#bib.bib7)\), and power analysis\(Chen et al\.,[2026](https://arxiv.org/html/2606.15474#bib.bib4)\)\. Our main monitor builds on the*prediction\-powered e\-value*ofCsillag et al\. \([2025b](https://arxiv.org/html/2606.15474#bib.bib8)\), whose Thm\. 2\.1 our per\-stratum process inherits, and sits in the broader anytime\-valid tradition: test \(super\)martingales and Ville’s inequality\(Ville,[1939](https://arxiv.org/html/2606.15474#bib.bib22)\), time\-uniform concentration\(Howard et al\.,[2025](https://arxiv.org/html/2606.15474#bib.bib13),[2022](https://arxiv.org/html/2606.15474#bib.bib12)\), betting confidence sequences\(Waudby\-Smith and Ramdas,[2022](https://arxiv.org/html/2606.15474#bib.bib24)\), e\-detectors for sequential change\(Shin et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib19)\), sequential comparison of forecasters\(Choe and Ramdas,[2023](https://arxiv.org/html/2606.15474#bib.bib5)\), and prediction\-powered*risk*monitoring of deployed models\(Zhang et al\.,[2026](https://arxiv.org/html/2606.15474#bib.bib26)\)\. We claim no new validity theory here: the combination of a cheap\-proxy PPI estimator, an anytime\-valid e\-process, and drift detection already exists in this cluster\. Our[Section˜3](https://arxiv.org/html/2606.15474#S3)contribution is the specific assembly — a per\-\(rubric×\\timesstratum\) Bonferroni e\-process for*localized blind\-spot*drift, plus an e\-wealth escalation trigger — benchmarked against this cluster on a single cost\-versus\-latency frontier\. The genuinely new material is the anchor construction and attribution analysis of[Section˜4](https://arxiv.org/html/2606.15474#S4)\.

#### Active and adaptive acquisition\.

Several lines couple a sampling rule to a downstream inference or detection target: active inference that labels where uncertainty is highest\(Zrnic and Candès,[2026](https://arxiv.org/html/2606.15474#bib.bib29)\), the active\-acquisition rule in Csillag et al\.’s Appendix B\.2 \(which we include as a baseline\), round\-robin active change detection\(Chaudhuri et al\.,[2024](https://arxiv.org/html/2606.15474#bib.bib3)\), adaptive\-sampling change detection\(Yi and Yang,[2025](https://arxiv.org/html/2606.15474#bib.bib25)\), and drift\-to\-action controllers that pair a detector with an acquisition policy\(Lamaakal et al\.,[2026](https://arxiv.org/html/2606.15474#bib.bib15)\)\. A cautionary result frames our escalation finding:Sfyraki and Wang \([2026](https://arxiv.org/html/2606.15474#bib.bib18)\)prove that for prediction\-powered mean estimation, uncertainty\-driven adaptive sampling is asymptotically no better than a fixed budget\. Our result is*consistent*with theirs — our e\-wealth trigger merely ties a matched fixed budget on the faintest drift — and our claim is narrowly about cost\-efficiency relative to other adaptive*triggers*\(covariate\-driven rules waste budget or read inert under a biased cheap judge\), not asymptotic dominance over fixed\-τ\\tau\.

#### LLM\-as\-judge reliability\.

A growing literature studies the reliability of LLM judges\(Zheng et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib27)\): correctly reporting results obtained with an imperfect judge\(Lee et al\.,[2026](https://arxiv.org/html/2606.15474#bib.bib16)\), valid downstream inference under noisy judge labels\(Feng et al\.,[2026](https://arxiv.org/html/2606.15474#bib.bib9)\), and quantifying judge bias and uncertainty\(Fiedler,[2026](https://arxiv.org/html/2606.15474#bib.bib10)\)\. That work corrects or qualifies a*static*judge — it treats the judge’s errors as a fixed, if unknown, distortion to be debiased\. We address a different failure mode: a judge whose behavior*changes over time*, and the problem of separating those changes from changes in the system under evaluation\. To our knowledge, no prior work runs a human\-anchored e\-process specifically to detect and attribute drift in the judge’s identity\.

## 3Cost\-Aware Monitoring with a Budgeted Strong Judge

### 3\.1Stream model

A deployment emits itemsi=1,2,…i=1,2,\\dotsin a stream\. Each item is scored onRRrubrics, with every score normalized to\[0,1\]\[0,1\], and carries an observable stratum labelt∈\{1,…,K\}t\\in\\\{1,\\dots,K\\\}\(here a topic\)\. Two judges are available\. A*cheap*judgeμ\\muis run on*every*item and returns a vectorμi∈\[0,1\]R\\mu\_\{i\}\\in\[0,1\]^\{R\}; a*strong*judgeyyreturnsyi∈\[0,1\]Ry\_\{i\}\\in\[0,1\]^\{R\}but is expensive and is therefore queried only on a sampled subset of items, under a budget\. In this sectionyyis the trusted reference: a drop inyyis, by assumption, a real drop in quality\.

### 3\.2A per\-stratum prediction\-powered e\-process

The monitor maintains one e\-process per \(rubric×\\timesstratum\) cell, testing the one\-sided null that the cell mean has not fallen below a healthy bar,

H0sys:𝔼\[yi\[r\]∣i∈cell\(r,t\)\]≥q0\[r,t\]−0\.05,H\_\{0\}^\{\\mathrm\{sys\}\}:\\quad\\mathbb\{E\}\[y\_\{i\}\[r\]\\mid i\\in\\text\{cell \}\(r,t\)\]\\;\\geq\\;q\_\{0\}\[r,t\]\-0\.05,\(1\)where the per\-cell barq0​\[r,t\]q\_\{0\}\[r,t\]is the mean strong\-judge score over in\-control items of stratumtt, estimated once from held\-out calibration data, and the0\.050\.05slack absorbs calibration noise\. Each cell accumulates wealth through prediction\-powered e\-values\(Csillag et al\.,[2025b](https://arxiv.org/html/2606.15474#bib.bib8)\): the cheap scoreμi​\[r\]\\mu\_\{i\}\[r\]supplies a debiasing prediction, and on sampled items the strong label corrects it, so a small number of strong queries yields a valid e\-value with the cheap judge as free side information\(Angelopoulos et al\.,[2023](https://arxiv.org/html/2606.15474#bib.bib1)\)\. The bets are*predictable*— chosen from the past — and capped at an effectiveλmax​\(πi\)\\lambda\_\{\\max\}\(\\pi\_\{i\}\)that depends only on the \(predictable\) sampling probabilityπi\\pi\_\{i\}, which keeps each cell’s wealth a nonnegative supermartingale under[Equation˜1](https://arxiv.org/html/2606.15474#S3.E1)\. An alarm fires when any cell’s wealth crosses the Bonferroni thresholdR​K/αsysRK/\\alpha\_\{\\mathrm\{sys\}\}\. Validity is then inherited fromCsillag et al\. \([2025b](https://arxiv.org/html/2606.15474#bib.bib8), Thm\. 2\.1\)together with the predictability ofπ\\piand Ville’s inequality\(Ville,[1939](https://arxiv.org/html/2606.15474#bib.bib22)\): the family’s probability of ever falsely alarming, at any data\-dependent stopping time, is at mostαsys\\alpha\_\{\\mathrm\{sys\}\}\.

The per\-stratum design is not a refinement but a necessity\. A localized blind\-spot drift — a quality regression confined to one stratum, on a rubric where the cheap judge is blind — is diluted to invisibility when pooled\. Empirically, under the mild drift a single*global*per\-rubric PPI e\-process never detects at any budget \(censoring rate1\.001\.00at strong\-sampling rates of0\.10\.1,0\.20\.2, and0\.30\.3\), whereas the stratified monitor detects at the same budgets; the localization is what makes the signal visible\.

### 3\.3Acquisition modes

The budget can be spent two ways\. Under a*fixed budget*the strong judge is queried with probabilityπi=τ\\pi\_\{i\}=\\tau\(allocated uniformly, by stratum, or by a bandit\)\. Under*evidence\-triggered escalation*the probability rises from a floorπ0\\pi\_\{0\}toward11as a stratum’s accumulated e\-wealth approaches its alarm threshold,

πi=π0\+\(1−π0\)⋅clip⁡\(𝒲i−1threshold,0,1\)\.\\pi\_\{i\}\\;=\\;\\pi\_\{0\}\+\(1\-\\pi\_\{0\}\)\\cdot\\operatorname\{clip\}\\\!\\Big\(\\frac\{\\mathcal\{W\}\_\{i\-1\}\}\{\\text\{threshold\}\},\\,0,\\,1\\Big\)\.\(2\)Crucially,πi\\pi\_\{i\}is a function of*past*wealth only, so it is predictable and anytime\-validity is preserved by construction\. The trigger keys on accumulated e\-evidence rather than on the cheap covariate, because the cheap judge is blind to exactly the drift we are trying to catch\.

### 3\.4Cost model

Each judge call costs\(input tokens\)⋅\(input price\)\+\(output tokens\)⋅\(output price\)\\text\{\(input tokens\)\}\\cdot\\text\{\(input price\)\}\+\\text\{\(output tokens\)\}\\cdot\\text\{\(output price\)\}, with representative counts of≈600\\approx 600input and≈30\\approx 30output tokens\. At the measured prices this is$​0\.000195\\mathdollar 0\.000195per cheap call and$​0\.00156\\mathdollar 0\.00156per strong call \(the cheap judge runs on allLLprocessed items; the strong judge onSSsampled items\)\. Relative to strong\-evaluating every item, the cost fraction is

cost\-fraction=L​cμ\+S​cyL​cy=0\.125\+SL,\\text\{cost\-fraction\}\\;=\\;\\frac\{L\\,c\_\{\\mu\}\+S\\,c\_\{y\}\}\{L\\,c\_\{y\}\}\\;=\\;0\.125\+\\frac\{S\}\{L\},\(3\)where full evaluation \(π≡1\\pi\\equiv 1\) is1\.01\.0by definition\. The0\.1250\.125is the irreducible cheap floor andS/LS/Lis the realized strong\-sampling rate\.

### 3\.5What the foundation buys

On a semi\-synthetic localized drift, every variant of the detector catches the regression at cost\-fraction0\.210\.21–0\.430\.43— about one third of full evaluation — with measured false\-alarm0\.0000\.000across all1616policy points\. The two modes trace a cost–reliability frontier: escalation occupies the cheap corner \(0\.210\.21–0\.280\.28, sitting near the0\.125\+π00\.125\+\\pi\_\{0\}floor when nothing is wrong\), fixed budgets buy reliability \(fixed@0\.30\.3reaches censoring0\.280\.28at cost0\.430\.43\), and full evaluation sets the latency floor \(median327327steps to detection\) at full price\. On the cost–detection plane \([Figure˜1](https://arxiv.org/html/2606.15474#S3.F1)\) this ordering is visible on both datasets: evidence\-triggered escalation lies up\-and\-left of the fixed\-budget family — more detection per strong\-judge dollar onceπ0\\pi\_\{0\}clears the hard\-drift floor — while paying in detection*latency*, its weak axis, which the plot omits and[Table˜6](https://arxiv.org/html/2606.15474#A2.T6)records\. We defer the frontier table and the full ten\-method comparison to[Appendix˜B](https://arxiv.org/html/2606.15474#A2)\. One contrast from that comparison matters here: keying acquisition on the cheap covariate is wasteful or inert — a boundary\-proximity trigger detects only by oversampling to cost≈0\.87\\approx 0\.87–0\.890\.89\. The published predicted\-e\-growth rule, meanwhile, reads identically zero under an overrating cheap judge, whereas e\-wealth escalation detects at roughly one third of that cost\. On faint drift, escalation merely ties a matched fixed budget, consistent with the result ofSfyraki and Wang \([2026](https://arxiv.org/html/2606.15474#bib.bib18)\)that uncertainty\-driven sampling is asymptotically no better than fixed\-τ\\taufor prediction\-powered mean estimation\.

![Refer to caption](https://arxiv.org/html/2606.15474v1/x1.png)Figure 1:The C1 cost–detection frontier on HelpSteer2 \(left\) and TL;DR \(right\): detection power \(1−censor1\-\\text\{censor\}\) against cost\-fraction \(log axis; full strong\-evaluation=1\.0=1\.0\)\. Evidence\-triggered escalation \(ours, blue\) lies up\-and\-left of the fixed\-budget family — whose three allocation policies \(uniform, stratified, bandit\) coincide, so they are drawn as one centroid line with a hairline min–max band — detecting more per strong\-judge dollar across theπ0\\pi\_\{0\}ladder; full evaluation \(⋆\\star\) buys the final increment of power at full price\. Detection latency, escalation’s weak axis, is omitted here and tabulated per policy in[Table˜6](https://arxiv.org/html/2606.15474#A2.T6)\. The TL;DR panel is the nothing\-re\-tuned replication detailed in[Section˜5](https://arxiv.org/html/2606.15474#S5)\.Everything above rests on trustingyy\. The next section drops that assumption\.

## 4Disentangling Judge Drift: the Anchor Construction

### 4\.1The problem with a drifting reference

The monitor of[Section˜3](https://arxiv.org/html/2606.15474#S3)treats the strong judgeyyas ground truth\. Butyyis itself an LLM service: a version bump or a silent prompt/policy update can change its scoring without notice\. When that happens, a main\-process alarm is ambiguous — did the system get worse, or did the judge change? — and the failure is compounding: the contaminatedyyfeeds the very process meant to certify system drift, so the detector can confidently report a system regression that is really an artifact of its own reference moving\. We disentangle the two with a second, independent e\-process anchored to fixed human labels\.

### 4\.2The anchor set

###### Definition 1\.

An*anchor set*𝒜=\{a1,…,ak\}\\mathcal\{A\}=\\\{a\_\{1\},\\dots,a\_\{k\}\\\}is a fixed, held\-out collection of items with frozen human labelsh​\(a\)∈\[0,1\]Rh\(a\)\\in\[0,1\]^\{R\}, drawn before monitoring starts and excluded from all main\-process calibration\. At every positioni≡0\(modρ\)i\\equiv 0\\pmod\{\\rho\}\(an*interleave rate*1/ρ1/\\rho\), one anchoraais drawn uniformly from𝒜\\mathcal\{A\}and re\-scored by the*current*strong judge, yieldingyi​\(a\)y\_\{i\}\(a\)\.

Because the anchor items and their human labels never change, the only thing that can moveyi​\(a\)y\_\{i\}\(a\)is the judge\. We summarize each re\-judgement by a rescaled gap statistic

g~i=clip⁡\(yi​\(a\)−h​\(a\)\+12,0,1\)∈\[0,1\]R,\\tilde\{g\}\_\{i\}\\;=\\;\\operatorname\{clip\}\\\!\\Big\(\\tfrac\{y\_\{i\}\(a\)\-h\(a\)\+1\}\{2\},\\,0,\\,1\\Big\)\\;\\in\\;\[0,1\]^\{R\},\(4\)with baselineq~0∈\(0,1\)R\\tilde\{q\}\_\{0\}\\in\(0,1\)^\{R\}\(a per\-rubric anchor baseline, not to be confused with the per\-cell quality barsq0​\[r,t\]q\_\{0\}\[r,t\]of[Section˜3](https://arxiv.org/html/2606.15474#S3)\), the mean rescaled gap of the*time\-0*judge over the full anchor set\. The baseline is computed once before monitoring; because the time\-0 judge scores are cached,q~0\\tilde\{q\}\_\{0\}is the exact mean over𝒜\\mathcal\{A\}and carries no finite\-sample error\.*This is the asymmetry the construction exploits: system drift changes the incoming items but leaves the fixed anchors untouched, whereas a judge change moves both the main scores and the anchors — the anchors see judge drift and only judge drift\.*

### 4\.3The anchor e\-process

For each rubricrrand directiond∈\{below,above\}d\\in\\\{\\text\{below\},\\text\{above\}\\\}we run a betting e\-process\(Waudby\-Smith and Ramdas,[2022](https://arxiv.org/html/2606.15474#bib.bib24); Csillag et al\.,[2025b](https://arxiv.org/html/2606.15474#bib.bib8)\)with e\-values

ei=1\+λi​sd​\(q~0​\[r\]−g~i​\[r\]\),sbelow=\+1,sabove=−1,e\_\{i\}\\;=\\;1\+\\lambda\_\{i\}\\,s\_\{d\}\\,\\big\(\\tilde\{q\}\_\{0\}\[r\]\-\\tilde\{g\}\_\{i\}\[r\]\\big\),\\qquad s\_\{\\text\{below\}\}=\+1,\\;s\_\{\\text\{above\}\}=\-1,\(5\)where the betλi∈\[0,λmax​\(q~0​\[r\],d\)\]\\lambda\_\{i\}\\in\[0,\\lambda\_\{\\max\}\(\\tilde\{q\}\_\{0\}\[r\],d\)\]is*predictable*\(a function ofg~1,…,g~i−1\\tilde\{g\}\_\{1\},\\dots,\\tilde\{g\}\_\{i\-1\}only; in code the plug\-inλ≈mean⁡\(d\)/mean⁡\(d2\)\\lambda\\approx\\operatorname\{mean\}\(d\)/\\operatorname\{mean\}\(d^\{2\}\)\)\. The wealth is𝒲n\(r,d\)=∏i≤nei\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}=\\prod\_\{i\\leq n\}e\_\{i\}, and the family alarms whenmaxr,d⁡𝒲n\(r,d\)≥2​R/αjudge\\max\_\{r,d\}\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\\geq 2R/\\alpha\_\{\\mathrm\{judge\}\}\(Bonferroni over the2​R2Rrubric–direction pairs\)\. The two directions test, respectively, that the judge has become more lenient or more harsh on the anchors relative to the human labels\.

### 4\.4The attribution rule

Letτsys\\tau\_\{\\mathrm\{sys\}\}andτanc\\tau\_\{\\mathrm\{anc\}\}be the first alarm positions of the main and anchor families \(∞\\inftyif neither ever fires\), and fix a guard windowW≥0W\\geq 0\. The verdict is

verdict=\{noneif neither family fires,judgeif​τanc<∞​and​\(τsys=∞​or​τanc≤τsys\+W\),systemif​τsys<∞​and​\(τanc=∞​or​τanc\>τsys\+W\)\.\\text\{verdict\}=\\begin\{cases\}\\texttt\{none\}&\\text\{if neither family fires,\}\\\\\[2\.0pt\] \\texttt\{judge\}&\\text\{if \}\\tau\_\{\\mathrm\{anc\}\}<\\infty\\text\{ and \}\(\\tau\_\{\\mathrm\{sys\}\}=\\infty\\text\{ or \}\\tau\_\{\\mathrm\{anc\}\}\\leq\\tau\_\{\\mathrm\{sys\}\}\+W\),\\\\\[2\.0pt\] \\texttt\{system\}&\\text\{if \}\\tau\_\{\\mathrm\{sys\}\}<\\infty\\text\{ and \}\(\\tau\_\{\\mathrm\{anc\}\}=\\infty\\text\{ or \}\\tau\_\{\\mathrm\{anc\}\}\>\\tau\_\{\\mathrm\{sys\}\}\+W\)\.\\end\{cases\}\(6\)The rule tests two nulls\. The anchor nullH0ancH\_\{0\}^\{\\mathrm\{anc\}\}\(no judge change\) states that, conditional on the past, each observed rescaled gap has meanq~0​\[r\]\\tilde\{q\}\_\{0\}\[r\]for every rubric — the current judge’s gap\-to\-human distribution on the fixed anchor set matches the time\-0 baseline\. The system nullH0sysH\_\{0\}^\{\\mathrm\{sys\}\}is the per\-stratum main\-process null of[Equation˜1](https://arxiv.org/html/2606.15474#S3.E1)\. When both families fire within the guard window the verdict isjudge*by design*: a contaminatedyyinvalidates the main alarm, so we do not attempt simultaneous attribution\.

### 4\.5Guarantees

The validity claims below are inherited from standard test\-supermartingale results; the contribution is the construction these results apply to, not new probability theory\. We state each proposition and a one\-line proof idea, deferring full proofs to[Appendix˜A](https://arxiv.org/html/2606.15474#A1)\.

###### Proposition 1\(Anytime\-validity of the anchor process\)\.

UnderH0ancH\_\{0\}^\{\\mathrm\{anc\}\}, for every rubric–direction pair\(𝒲n\(r,d\)\)n\(\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\)\_\{n\}is a nonnegative supermartingale with𝒲0=1\\mathcal\{W\}\_\{0\}=1, and consequently

ℙ\(∃n:maxr,d𝒲n\(r,d\)≥2R/αjudge\)≤αjudge,\\mathbb\{P\}\\Big\(\\exists\\,n:\\ \\max\_\{r,d\}\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\\geq 2R/\\alpha\_\{\\mathrm\{judge\}\}\\Big\)\\;\\leq\\;\\alpha\_\{\\mathrm\{judge\}\},uniformly over all stopping times: at any data\-dependent monitoring horizon, the probability that the anchor family*ever*falsely fires is at mostαjudge\\alpha\_\{\\mathrm\{judge\}\}\.

###### Proof idea\.

Boundedness ofg~i\\tilde\{g\}\_\{i\}and the directional bound onλi\\lambda\_\{i\}make eachei≥0e\_\{i\}\\geq 0; predictability ofλi\\lambda\_\{i\}together with𝔼​\[g~i​\[r\]∣past\]=q~0​\[r\]\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}\[r\]\\mid\\text\{past\}\]=\\tilde\{q\}\_\{0\}\[r\]underH0ancH\_\{0\}^\{\\mathrm\{anc\}\}gives𝔼​\[ei∣past\]=1\\mathbb\{E\}\[e\_\{i\}\\mid\\text\{past\}\]=1, so each wealth is a nonnegative supermartingale\. Ville’s inequality\(Ville,[1939](https://arxiv.org/html/2606.15474#bib.bib22)\)bounds each pair’s crossing probability byαjudge/2​R\\alpha\_\{\\mathrm\{judge\}\}/2R, and a union bound over the2​R2Rpairs finishes \(full proof in[Appendix˜A](https://arxiv.org/html/2606.15474#A1)\)\. ∎

###### Proposition 2\(Identification\)\.

The anchor observations depend only on the fixed items𝒜\\mathcal\{A\}, their frozen labelshh, and the current judge\. System drift — any change in the distribution of new items or their true quality — leaves the anchor gap distribution unchanged, soH0ancH\_\{0\}^\{\\mathrm\{anc\}\}can be violated only by a change in the judge\. Consequently:\(a\)under pure system drift,ℙ​\(verdict=judge\)≤αjudge\\mathbb\{P\}\(\\text\{verdict\}=\\texttt\{judge\}\)\\leq\\alpha\_\{\\mathrm\{judge\}\}, uniformly inWW; and\(b\)under no drift at all,ℙ​\(verdict≠none\)≤αsys\+αjudge\\mathbb\{P\}\(\\text\{verdict\}\\neq\\texttt\{none\}\)\\leq\\alpha\_\{\\mathrm\{sys\}\}\+\\alpha\_\{\\mathrm\{judge\}\}\.

###### Proof idea\.

A falsejudgeverdict requires the anchor family to fire, whose probability[Proposition˜1](https://arxiv.org/html/2606.15474#Thmproposition1)bounds byαjudge\\alpha\_\{\\mathrm\{judge\}\}irrespective ofWW; \(b\) is a union bound over the two families’ separate false\-fire events\. See[Appendix˜A](https://arxiv.org/html/2606.15474#A1)\. ∎

###### Proposition 3\(The attribution race\)\.

Under judge drift \(with or without simultaneous system drift\) the misattribution event\{verdict=system\}\\\{\\text\{verdict\}=\\texttt\{system\}\\\}requires the contaminated main process to fire more thanWWpositions before the anchors:

ℙ​\(verdict=system∣judge drift\)=ℙ​\(τsys\+W<τanc\),\\mathbb\{P\}\(\\text\{verdict\}=\\texttt\{system\}\\mid\\text\{judge drift\}\)\\;=\\;\\mathbb\{P\}\(\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\),which is monotonically non\-increasing in the guard widthWW, in the anchor interleave rate1/ρ1/\\rho, and in the anchor set sizekk\(more anchor evidence per position yields stochastically smallerτanc\\tau\_\{\\mathrm\{anc\}\}\), and monotonically non\-decreasing in the main process’s sampling power \(more strong calls on a contaminatedyyyield stochastically smallerτsys\\tau\_\{\\mathrm\{sys\}\}\)\.

###### Proof idea\.

The event\{verdict=system\}\\\{\\text\{verdict\}=\\texttt\{system\}\\\}under judge drift is exactly\{τsys\+W<τanc\}\\\{\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\\\}by[Equation˜6](https://arxiv.org/html/2606.15474#S4.E6); the monotonicities follow from stochastic ordering of the stopping times in each knob\. See[Appendix˜A](https://arxiv.org/html/2606.15474#A1)\. ∎

###### Proposition 4\(Orthogonality\)\.

τanc\\tau\_\{\\mathrm\{anc\}\}is independent of the main\-process configuration \(policy, budget,π\\pi\), and the main process’s distribution is independent of the anchor interleave: anchor re\-judgements are out\-of\-band calls on held\-out items that never enter the main e\-processes or their calibration\. The design therefore splits cleanly — the anchor budget\(k,ρ\)\(k,\\rho\)and guardWWgovern judge\-drift latency and the race, the main configuration governs system power, andαjudge,αsys\\alpha\_\{\\mathrm\{judge\}\},\\alpha\_\{\\mathrm\{sys\}\}are separate Bonferroni budgets\.

###### Proof idea\.

The anchor observations are a deterministic function of\(𝒜,h,current judge\)\(\\mathcal\{A\},h,\\text\{current judge\}\)and the interleave schedule, none of which depend on the main acquisition rule; symmetrically, held\-out anchor calls never enter the main statistics\. See[Appendix˜A](https://arxiv.org/html/2606.15474#A1)\. ∎

We verify these guarantees empirically in[Section˜5](https://arxiv.org/html/2606.15474#S5):[Proposition˜1](https://arxiv.org/html/2606.15474#Thmproposition1)and[Proposition˜2](https://arxiv.org/html/2606.15474#Thmproposition2)as false\-judgerates at or belowαjudge\\alpha\_\{\\mathrm\{judge\}\}under pure system drift and no drift;[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)through the guard\-window sweep and the cost frontier; and[Proposition˜4](https://arxiv.org/html/2606.15474#Thmproposition4)through the constant judge\-latency rows and constant system\-power columns of that frontier\. Two boundaries are worth stating plainly\. The construction claims no optimality — the plug\-inλ\\lambdaapproximates the log\-optimal bet, and the attribution rule is not claimed to minimize misattribution at a fixed budget — and it assumes the human labelshhremain a valid reference: if the anchor items’ true quality standard itself drifts conceptually, the anchor process correctly reports judge\-vs\-anchor disagreement, but its reading as judge drift weakens until a periodic anchor refresh re\-establishes the baseline\.

## 5Experiments

We evaluate four claims empirically: the attribution rule disentangles judge drift from system drift \([Proposition˜2](https://arxiv.org/html/2606.15474#Thmproposition2)\); the misattribution rate is a tunable race in the guard width, anchor rate, and anchor size \([Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)\); the two processes are orthogonal and must be co\-provisioned \([Proposition˜4](https://arxiv.org/html/2606.15474#Thmproposition4)\); and the anytime\-valid anchor process dominates the classical alternatives operationally\. Every experiment is run on two datasets — HelpSteer2 assistant\-response evals and TL;DR summarization evals — with*nothing*re\-tuned between them: the same judge roster, featured configuration, thresholds, andα\\alphabudgets\. The second dataset thereby functions as an out\-of\-sample test of the propositions, not only a robustness check: invariances must transfer, and cross\-domain differences must land where the theory says \([Section˜5\.6](https://arxiv.org/html/2606.15474#S5.SS6)\)\. We close by showing every property also generalizes across all topic–rubric cells of both datasets\.

### 5\.1Setup

We use two datasets\. The primary is HelpSteer2\(Wang et al\.,[2024](https://arxiv.org/html/2606.15474#bib.bib23)\):14981498items, each scored onR=5R=5rubrics \(helpfulness, correctness, coherence, complexity, verbosity\) and carrying a topic label overK=6K=6strata\. The second is the TL;DR summarization axis evals\(Stiennon et al\.,[2022](https://arxiv.org/html/2606.15474#bib.bib21)\):14981498summarization items scored onR=4R=4axes \(overall, accuracy, coverage, coherence\) with subreddit strata \(K=5K=5\)\. On both, the cheap judgeμ\\muisgemini\-3\.1\-flash\-liteand the strong judgeyyisgemini\-3\.1\-pro\-preview\(Google DeepMind,[2025](https://arxiv.org/html/2606.15474#bib.bib11)\); the frozen anchor labelshhare each dataset’s human ratings \(HelpSteer2 per\-attribute0–44, TL;DR77\-point Likert, both rescaled to\[0,1\]\[0,1\]— the anchor baseline gap absorbs the constant scale offset\)\. We monitor streams of lengthN=1200N=1200with the change point at position300300, averaging over6060seeds, and setαsys=αjudge=0\.1\\alpha\_\{\\mathrm\{sys\}\}=\\alpha\_\{\\mathrm\{judge\}\}=0\.1\.

We inject three judge drifts spanning the realism axis\.*Real\-lenient*: every item is re\-judged bygemini\-3\.5\-flash, an actual silent version bump, which runs uniformly more lenient than the baseline judge \(per\-rubric mean gap shift\+0\.022\+0\.022to\+0\.072\+0\.072on HelpSteer2,\+0\.031\+0\.031to\+0\.071\+0\.071on TL;DR — the same bump, the same signature, in both domains\)\.*Real\-harsh*: the*same*model is re\-prompted with a stricter scoring prompt \(“v2\-strict”\), a real judge\-behavior change that shifts scores down by−0\.043\-0\.043to−0\.152\-0\.152on HelpSteer2 \(verbosity the lone exception at\+0\.025\+0\.025\) and by−0\.097\-0\.097to−0\.230\-0\.230on TL;DR \(summaries offer more flaws to penalize\)\.*Synthetic*: a tunable shift−0\.25\-0\.25plus noise\. System drift is injected by drawing the target cell \(creative×\\timeshelpfulness on HelpSteer2, relationships×\\timesoverall on TL;DR\) post\-change from a regressed\-blind pool \(yylow,μ\\muhigh\), the exact blind spot that makes the cheap judge alone insufficient\. Crossing the two drifts gives four ground\-truth conditions\{none,system,judge,both\}\\\{\\texttt\{none\},\\texttt\{system\},\\texttt\{judge\},\\texttt\{both\}\\\}\.

Unless a knob is swept, we report a single*featured configuration*: a fixed\-budget main process at sampling rate0\.30\.3\(the high\-power point on the cost frontier of[Section˜3](https://arxiv.org/html/2606.15474#S3)\) withk=200k=200anchors interleaved at rate1/51/5and guardW=300W=300\(justified in[Section˜5\.4](https://arxiv.org/html/2606.15474#S5.SS4)\)\. This configuration runs at cost\-fraction≈0\.64\\approx 0\.64\(HelpSteer2\) /0\.630\.63–0\.650\.65\(TL;DR\) of strong\-evaluating every item \(the0\.1250\.125cheap floor,0\.300\.30main sampling, and1/51/5anchor share\), versus1\.01\.0for full evaluation; a cheap\-but\-deaf alternative \(escalationπ0=0\.02\\pi\_\{0\}=0\.02with rate\-2020anchors\) costs0\.210\.21but misses most system regressions\.

### 5\.2Disentanglement and detection

[Table˜1](https://arxiv.org/html/2606.15474#S5.T1)reports the verdict confusion matrices under the synthetic harsh shift on both datasets\. With matched anchors, system detection is41/6041/60\(HelpSteer2\) and56/6056/60\(TL;DR\) — exactly each fixed@0\.30\.3main’s power from[Section˜3](https://arxiv.org/html/2606.15474#S3)— while judge and both are detected60/6060/60and5555–59/6059/60\. On HelpSteer2 the purejudgeandbothrows*never*land insystem\(judge→\\tosystem0/1200/120\); on TL;DR a small spill remains at the default guard \(5\+1/1205\{\+\}1/120\) — the guard race of[Section˜5\.3](https://arxiv.org/html/2606.15474#S5.SS3), where wideningWWcloses it\. Falsejudgeverdicts stay at or belowαjudge\\alpha\_\{\\mathrm\{judge\}\}on both datasets:2/602/60onnoneand2/602/60onsystem\(rate≈0\.03<0\.1\\approx 0\.03<0\.1, as expected — the rate\-55interleave gives the anchor family four times the observations, so a fewα\\alpha\-budget false fires appear by design\)\.

The right half of[Table˜1](https://arxiv.org/html/2606.15474#S5.T1)is the co\-provisioning lesson\. With under\-provisioned anchors \(k=50k=50, rate1/201/20\), the high\-power main reads the contaminatedyyso fast that it fires roughly300300items before the anchors catch up:4848–50/6050/60\(HelpSteer2\) and5454–58/6058/60\(TL;DR\) of the genuine judge\-drift runs land insystemand the disentanglement collapses\. The anchor process must be provisioned to*out\-run the main process it guards*— anchor detection latency \(120120/140140items for the matched anchors, versus420420/440440for the slow ones\) has to come in under the main’s contaminated\-fire latency plusWW\.

Table 1:Verdict confusion \(60 seeds, synthetic judge shift−0\.25\-0\.25, fixed@0\.3 main, guardW=50W\{=\}50\), both datasets\. Left: matched anchors \(k=200k\{=\}200, rate1/51/5\)\. Right: under\-provisioned anchors \(k=50k\{=\}50, rate1/201/20\) — the contaminated main out\-runs the anchors and the disentanglement collapses on both datasets\.[Table˜2](https://arxiv.org/html/2606.15474#S5.T2)turns to a*real*silent version bump and the anchor\-budget frontier\. The lenientgemini\-3\.5\-flashbump is moderate \(mean shift≤\+0\.072\\leq\+0\.072, magnitude comparable to the synthetic−0\.05\-0\.05/−0\.10\-0\.10sweep points\), so detection depends on anchor budget — on both datasets:7%7\\%atk=50k=50/rate\-2020on each, rising to100%100\\%\(60/6060/60\) on HelpSteer2 and83%83\\%\(50/6050/60\) on TL;DR at the featuredk=200k=200/rate\-55\(the bump is subtler per\-axis on TL;DR’s featured cell, so the same budget buys less of the frontier; misses fail safe tonone\)\. The pure\-judgerow never lands insystemat any budget on either dataset, and false\-judge rates stay underαjudge\\alpha\_\{\\mathrm\{judge\}\}\. The featured fixed@0\.30\.3replications hold: judge detection60/6060/60at latency498498\(HelpSteer2\) and50/6050/60at latency555555\(TL;DR\), judge→\\tosystem0\.

Table 2:A real version bump \(gemini\-3\.1\-pro→\\to3\.5\-flash\) vs anchor budget, both datasets\. 60 seeds; escalation main \(featured fixed@0\.3 replications: 60/60 at latency 498 on HelpSteer2, 50/60 at latency 555 on TL;DR, judge→\\tosystem 0 on both\)\.anchorsratedetection \(judge / both\)judge→\\tosysnone→\\tojudgelatency*HelpSteer2*\(mean shift≤\+0\.072\\leq\{\+\}0\.072\)k=50k\{=\}501/201/207% \(4/60\) / 7%00/60660k=150k\{=\}1501/81/862% \(37/60\) / 60%01/60652k=200k\{=\}2001/51/5100% \(60/60\)/ 98%02/60498*TL;DR*\(mean shift≤\+0\.071\\leq\{\+\}0\.071\)k=50k\{=\}501/201/207% \(4/60\) / 7%00/60810k=150k\{=\}1501/81/838% \(23/60\) / 38%02/60516k=200k\{=\}2001/51/583% \(50/60\)/ 83%02/60555#### The one\-sided caveat\.

The lenient bump scoresy′≥yy^\{\\prime\}\\geq yon every rubric, and the main monitor is one\-sided “below” — it fires only on quality*drops*\. A lenient judge therefore cannot push the main toward asystemalarm at all, so the judge→\\tosystem=0=0of[Table˜2](https://arxiv.org/html/2606.15474#S5.T2)is*partly structural*for this drift direction, not solely the anchors’ doing\.[Table˜2](https://arxiv.org/html/2606.15474#S5.T2)thus demonstrates*detection*of a subtle real bump; the genuine test of disentanglement\-under\-contamination is a*harsher*judge change that does push the main downward, which is the subject of[Section˜5\.3](https://arxiv.org/html/2606.15474#S5.SS3)\. Both demonstrations are needed; neither alone supports the full claim\.

### 5\.3A real harsh judge change: contamination and the guard race

The v2\-strict re\-judge is a real LLM judge\-behavior change: the same model is instructed to “reserve44for flawless work … when torn, choose the lower\.”[Table˜3](https://arxiv.org/html/2606.15474#S5.T3)characterizes both real drifts per rubric on both datasets — the lenient bump is a uniform leniency shift \(≤\+0\.072\\leq\+0\.072\) with the*same*\+0\.03\+0\.03to\+0\.07\+0\.07signature in both domains \(a property of the version bump, not the dataset\), while the harsh re\-judge shifts strictly downward: into the−0\.10\-0\.10to−0\.15\-0\.15band on HelpSteer2 \(verbosity the lone, near\-noise exception at\+0\.025\+0\.025\) and deeper,−0\.10\-0\.10to−0\.23\-0\.23, on TL;DR\. This is the synthetic sweep’s hardest regime: slow anchors against a fast contaminated main\. Note that item\-level agreement degrades far more than the means suggest \(exact agreement2626–84%84\\%\), and the rubrics where the two strong judges agree least \(verbosity, complexity, corr0\.460\.46–0\.600\.60\) mirror the cheap judge’s own blind spots — the anchor process fires on the systematic component, not the per\-item noise, which is why it needsk≈200k\\approx 200anchors for the subtle bump\.

Table 3:Strong\-judge gapy′−yy^\{\\prime\}\-yper rubric, both real drifts, both datasets \(≈1496\\approx 1496–14981498items judged by both judges\)\.*shift*is the mean gap shift the anchor process monitors;*corr*and*agree%*are item\-level\. The lenient bump moves scores up uniformly with the same signature in both domains; the harsh re\-judge moves them down — harder on TL;DR, where summaries offer more flaws to penalize\.lenient \(3\.5\-flash\)harsh \(v2\-strict\)rubricshiftcorragree%shiftcorragree%*HelpSteer2*helpfulness\+0\.064\+0\.0640\.8863\.6%−0\.152\-0\.1520\.9045\.0%correctness−0\.008\-0\.0080\.8670\.0%−0\.120\-0\.1200\.8855\.1%coherence\+0\.022\+0\.0220\.7682\.5%−0\.092\-0\.0920\.7962\.9%complexity\+0\.050\+0\.0500\.5972\.5%−0\.043\-0\.0430\.5777\.5%verbosity\+0\.072\+0\.0720\.4662\.6%\+0\.025\+0\.0250\.6071\.0%*TL;DR*overall\+0\.048\+0\.0480\.8969\.7%−0\.230\-0\.2300\.8426\.2%accuracy\+0\.031\+0\.0310\.9184\.4%−0\.097\-0\.0970\.8965\.7%coverage\+0\.071\+0\.0710\.8765\.6%−0\.193\-0\.1930\.8331\.9%coherence\+0\.037\+0\.0370\.8178\.6%−0\.134\-0\.1340\.7851\.0%[Table˜4](https://arxiv.org/html/2606.15474#S5.T4)reports the featured config under this real contamination, sweeping the guard windowWWon both datasets\. The harsh shift genuinely contaminates the one\-sided main, but the anchors flip the large majority of contaminated fires to the correctjudgeverdict, and the residual spill is a tunable race, not a defect: judge→\\tosystem shrinks monotonically inWW—29→18→1029\\to 18\\to 10of120120on HelpSteer2 and5→1→05\\to 1\\to 0on TL;DR — exactly the monotonicity of[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3), because spills happen only when the contaminated main fires more thanWWitems before the anchors\. TL;DR reaches a*perfect*240/240240/240over its two drift rows atW=300W=300: its harsh shift is larger \(overall−0\.23\-0\.23\), so the anchors are faster \(median latency205205versus370370\) and win the race outright — a bigger judge change is*easier*to attribute\. Critically, thenoneandsystemrows are identical at everyWWon both datasets \(58/0/258/0/2and17/41/217/41/2;58/0/258/0/2and2/56/22/56/2\): widening the guard costs nothing in system detection or false\-judge rate — only attribution latency and a sliver of anchor cost\. The guard window is the third knob, after anchor size and interleave rate, and the cheapest: it buys correctness with latency rather than dollars\.

Table 4:Real harsh judge change \(v2\-strict re\-judge\), featured config \(fixed@0\.3 main,k=200k\{=\}200/rate\-55anchors\), 60 seeds, sweeping guardWW, both datasets\. Spill = judge→\\tosystem total over the two drift rows \(of 120\)\. Thenoneandsystemrows are identical at everyWWon both datasets: guard width costs only attribution latency\. TL;DR’s larger shift makes the anchors faster, closing the spill entirely atW=300W\{=\}300\.guardWWjudge row \(none/sys/judge\)both row \(none/sys/judge\)spill /120latencycost*HelpSteer2*\(shifts−0\.04\-0\.04to−0\.15\-0\.15\)500 / 10 / 500 / 19 / 41293700\.641500 / 8 / 520 / 10 / 50183700\.66–0\.673000 /3/570 /7/53103700\.68–0\.71*TL;DR*\(shifts−0\.10\-0\.10to−0\.23\-0\.23\)500 / 3 / 570 / 2 / 5852050\.63–0\.651500 / 1 / 590 / 0 / 6012050\.63–0\.683000 /0/600 /0/6002050\.63–0\.73[Figure˜2](https://arxiv.org/html/2606.15474#S5.F2)shows one seed of the featured config under this real harsh drift on each dataset\. The change point is at300300\. On HelpSteer2 the anchor process crosses its threshold \(≈4\.61\\approx 4\.61\) at position735735, well before the contaminated main crosses its threshold \(≈5\.70\\approx 5\.70\) at10981098; on TL;DR the larger shift accelerates*both*processes — anchors at370370, contaminated main at567567— but the anchors keep winning\. In both panels the verdict isjudge: the contaminated main fire lands inside the guard window \(shaded\) opened by the anchor alarm\.

![Refer to caption](https://arxiv.org/html/2606.15474v1/x2.png)Figure 2:The attribution race under real harsh judge drift \(one seed, featured config\), on HelpSteer2 \(left\) and TL;DR \(right\)\. Both e\-process wealth trajectories are shown against their Bonferroni alarm thresholds; the change point is at300300\. The anchor process alarms first on both datasets \(735735vs10981098on HelpSteer2;370370vs567567on TL;DR\), so the main fire falls inside the guard window \(shaded\) and the verdict is correctlyjudge\.
### 5\.4Co\-provisioning: main power×\\timesanchor rate

[Figure˜3](https://arxiv.org/html/2606.15474#S5.F3)sweeps five main configurations against three anchor interleave rates \(real harsh drift,k=200k=200,W=50W=50,3030seeds\) on both datasets, establishing four structural facts\. First, the two processes are*orthogonal*as designed \([Proposition˜4](https://arxiv.org/html/2606.15474#Thmproposition4)\): judge latency depends only on the anchor rate \(830/695/430830/695/430items at rate20/10/520/10/5on HelpSteer2;640/330/202640/330/202on TL;DR — across every main config\), system power depends only on the main config \(constant down each main’s rows\), and the false\-judge rate is flat at≤0\.07≤αjudge\\leq 0\.07\\leq\\alpha\_\{\\mathrm\{judge\}\}\. Second, at under\-anchored points the sum of judge\-detection and spill is≈1\\approx 1: those points do not*miss*the drift, they*mislabel*it — detection and attribution are separate\. Third, the spill grows monotonically with main power at fixed rate — on HelpSteer2’s rate\-55column,0\.03→0\.07→0\.18→0\.330\.03\\to 0\.07\\to 0\.18\\to 0\.33\(escalationπ0=0\.02\\pi\_\{0\}=0\.02, fixed@0\.10\.1, fixed@0\.20\.2, fixed@0\.30\.3\) — so the feasible region is a diagonal band and the anchor budget must be co\-provisioned with main power, not bolted on\. The TL;DR panel shows the same geometry entered from the fast\-anchor side: its larger harsh shift lets rate\-55anchors out\-run even the fixed@0\.30\.3main \(spill≤0\.05\\leq 0\.05\), and the band sits at rates1010–2020instead \(0\.420\.42at fixed@0\.30\.3/rate\-1010,0\.900\.90at rate\-2020\) — the race of[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3), with the drift magnitude setting where the band falls\. Fourth, the guardWWis the orthogonal escape hatch: the sweep of[Section˜5\.3](https://arxiv.org/html/2606.15474#S5.SS3)pulled the HelpSteer2 corner’s spill from2929to1010of120120atW=300W=300\(and TL;DR’s from55to0\), so points outside the band are recoverable with attribution latency instead of anchor dollars\. Together these justify the featured configuration: a high\-power main matched with rate\-55anchors and a wide guard\.

![Refer to caption](https://arxiv.org/html/2606.15474v1/x3.png)Figure 3:Co\-provisioning frontier \(real harsh drift,k=200k\{=\}200,W=50W\{=\}50, 30 seeds\), on HelpSteer2 \(left\) and TL;DR \(right\)\. Judge latency depends only on the anchor rate and system power only on the main config \([Proposition˜4](https://arxiv.org/html/2606.15474#Thmproposition4)\); the misattribution spill grows along the main\-power axis at fixed anchor rate, so the feasible region is a diagonal band — main power and anchor rate must be raised together\. TL;DR’s larger harsh shift shifts the band toward slower anchor rates: the same geometry, set by the drift magnitude\.
### 5\.5Baselines: why anytime\-validity matters operationally

We compare the anchor e\-process against two classical change detectors consuming the*identical*anchor observation stream \(k=200k=200, rate1/51/5,6060seeds, both datasets\);[Table˜5](https://arxiv.org/html/2606.15474#S5.T5)reports false\-alarm rate, detection rate, and latency on each drift\. The default practice fails outright\.*naive\-z*— a rolling\-windowzz\-test against baseline atα=0\.05\\alpha=0\.05, re\-tested every observation with no sequential correction, the industry standard — false\-alarms on75%75\\%\(HelpSteer2\) and67%67\\%\(TL;DR\) of no\-drift streams, and16/6016/60and10/6010/60of its “detections” fire*before the change even happens*: its apparent speed is the absence of error control, not aggression\.*ph\-calib*is Page–Hinkley\(Page,[1954](https://arxiv.org/html/2606.15474#bib.bib17)\)per rubric×\\timesdirection, with the family threshold calibrated on6060held\-outH0H\_\{0\}anchor streams to FWER0\.10\.1— the strongest classical recipe\. It holds its false\-alarm budget \(0\.080\.08/0\.030\.03\) but is blind to the subtle real lenient bump \(8%8\\%on HelpSteer2,40%40\\%on TL;DR\), and where it does detect it is slower than the e\-process \(295295versus120120on the synthetic shift;248248versus205205and165165versus140140on TL;DR\)\. The anchor e\-process dominates on both datasets: false\-alarm0\.03≤αjudge0\.03\\leq\\alpha\_\{\\mathrm\{judge\}\}with no calibration data and a horizon\-free guarantee,97%97\\%detection on all three HelpSteer2 drifts and8383–100%100\\%on TL;DR \(the83%83\\%on the lenient bump*is*the anchor\-budget frontier of[Table˜2](https://arxiv.org/html/2606.15474#S5.T2), not a method limitation\), and the best latency at every drift among false\-alarm\-controlled methods\. Anytime\-validity is not a theoretical nicety: the default practice is unusable for continuous monitoring, and the properly\-calibrated classical alternative pays with both blindness to subtle real drift and a calibration\-data tax\.

![Refer to caption](https://arxiv.org/html/2606.15474v1/x4.png)Figure 4:The anchor e\-process versus classical change detectors on the identical anchor stream, on HelpSteer2 \(left\) and TL;DR \(right\)\. naive\-z \(rollingzz\-test, the industry default\) has no sequential error control and false\-alarms on75%75\\%/67%67\\%of no\-drift streams; calibrated Page–Hinkley holdsα\\alphabut misses the subtle real lenient bump; the anytime\-valid e\-process controls false alarms atαjudge\\alpha\_\{\\mathrm\{judge\}\}with no calibration data and detects all drifts up to its anchor budget\.Table 5:Anchor\-process baselines on the identical anchor stream \(60 seeds\), both datasets\. FA = false\-alarm rate on no\-drift; detection rate / median latency on each of the three judge drifts\. naive\-z additionally fires16/6016/60\(HelpSteer2\) and10/6010/60\(TL;DR\) alarms*before*the change point\.detection ratemedian latencymethodFAlenientharshsynthlenientharshsynth*HelpSteer2*eprocess0\.030\.970\.970\.97498370120naive\-z0\.750\.730\.730\.7313013840ph\-calib0\.080\.080\.651\.00555530295*TL;DR*eprocess0\.030\.831\.001\.00555205140naive\-z0\.670\.830\.830\.831687250ph\-calib0\.030\.401\.001\.00550248165
### 5\.6Generalization: every cell of both datasets

#### Every topic–rubric cell\.

Running the featured config across all2828runnable HelpSteer2 topic×\\timesrubric cells \(two near\-ceiling coherence cells are skipped — their regressed\-blind pools are empty\), judge detection is1\.001\.00in27/2827/28cells \(0\.970\.97in factual\_qa:coherence\) at latency9898–128128items, judge→\\tosystem is0\.000\.00in27/2827/28\(0\.030\.03in that same cell\), and the false\-judge rate is at most0\.100\.10\. On TL;DR’s19/2019/20runnable subreddit×\\timesaxis cells the picture is the same: judge detection0\.850\.85–0\.970\.97at latency118118–145145in every cell, judge→\\tosystem≤0\.15\\leq 0\.15atW=50W=50, and false\-judge≤0\.10=αjudge\\leq 0\.10=\\alpha\_\{\\mathrm\{judge\}\}\. The C2 properties are cell\-independent by construction — the anchor process never looks at the main\-process cell\. Only C1 system power varies \(range0\.000\.00–1\.001\.00across HelpSteer2 cells,0\.400\.40–1\.001\.00on TL;DR, weakest whereμ\\muis most generous; missed system regressions fail safe tonone, never a falsejudge\)\. The full per\-cell table is in[Appendix˜C](https://arxiv.org/html/2606.15474#A3)\.

#### Cross\-dataset reads\.

The exhibit\-by\-exhibit replication tests the theory in two distinct ways: the*invariances*must transfer, and the*differences*must be the ones the propositions predict\. Both happen\. On the invariance side, the same real version bump shows the same\+0\.03\+0\.03to\+0\.07\+0\.07per\-rubric signature in both domains \([Table˜3](https://arxiv.org/html/2606.15474#S5.T3)\) — a property of the bump, not the dataset — turning the lenient case from a case study into a property of the version bump itself; and each dataset has one low\-resolution rubric \(verbosity on HelpSteer2, coherence on TL;DR\) with identical phenomenology — weakμ\\mucorrelation, near\-ceiling humans, weakest judge–human agreement — a property of rubric design, not of the monitor\. On the prediction side, the domains differ in exactly one relevant input — the strict\-prompt shift is larger on summaries — and[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)says a larger gap shift makes the anchors stochastically faster, so attribution should improve everywhere that race appears\. It does, three times over, with nothing re\-tuned: anchor latency drops \(205205versus370370\), the guard sweep closes to zero spill atW=300W=300\([Table˜4](https://arxiv.org/html/2606.15474#S5.T4)\), and the co\-provisioning band moves toward slower anchor rates \([Figure˜3](https://arxiv.org/html/2606.15474#S5.F3)\)\. These are out\-of\-sample structural predictions coming true, not robustness checks\. The one quantity that is*not*predicted to transfer, C1 system power, indeed varies for its own reason: it is*higher*on TL;DR \(56/6056/60versus41/6041/60,[Table˜1](https://arxiv.org/html/2606.15474#S5.T1)\) because all four axes are informative forμ\\mu, unlike HelpSteer2’s complexity and verbosity\.

## 6Discussion and Limitations

The construction buys a sharp guarantee — judge drift is never silently charged to the system — at the price of several scoped assumptions, which we state plainly\.

#### Anchor staleness\.

Identification \([Proposition˜2](https://arxiv.org/html/2606.15474#Thmproposition2)\) assumes the frozen human labelshhremain a valid reference\. If the anchor items’*true*quality standard drifts conceptually — the meaning of a rubric shifts, say — the anchor process still correctly reports judge\-versus\-anchor disagreement, but its interpretation as*judge*drift weakens, since the disagreement could now be the world moving under a fixed yardstick\. A periodic anchor refresh, re\-labeling against the current standard, re\-establishes the baseline\.

#### Pooled anchor family\.

The anchor e\-process aggregates over rubric×\\timesdirection only; it carries no topic dimension\. This is a deliberate power/budget choice: judge drift is plausibly global, and a per\-topic anchor family would split thekkanchors across cells and multiply the Bonferroni factor byKK\. The cost is reduced power against*topic\-localized*judge drift — a judge that changed only on, say, code\-heavy items\. Topic stratification lives entirely on the system\-process side\.

#### bothis reported asjudge, by design\.

When both families fire within the guard window the verdict isjudge\([Equation˜6](https://arxiv.org/html/2606.15474#S4.E6)\): a contaminatedyyinvalidates the main alarm, so we do not attempt simultaneous attribution of a co\-occurring system regression\. The honest reading is that a confirmed judge change suspends trust in the main verdict until the judge is re\-anchored\.

#### No optimality\.

We claim validity and identification, not efficiency\. The plug\-in betλ\\lambdaapproximates the log\-optimal bet rather than realizing it, and the attribution rule is not claimed to minimize misattribution at a fixed anchor budget\. The race monotonicities of[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)say which direction each knob moves the spill, not that the knobs are set optimally\.

#### System power is cell\-dependent, and the horizon is single\.

The disentanglement properties are cell\-independent by construction, but the main monitor’s*system*power is not: it is near zero on the cheap\-judge\-blind rubrics \(complexity, verbosity\) and on the math cells, where the regressed\-blind pool is empty or the cheap judge is uninformative\. A missed system regression fails safe tonone— never to a falsejudge\. The experiments use a single monitoring horizon \(N=1200N=1200, change point at300300\) and a representative token\-cost model \(input/output token counts at measured prices\), which ignores topic\-assignment overhead; absolute cost fractions should be read as representative, not universal\.

#### The honest cost menu\.

There is no single price\. The system traces a menu against strong\-evaluating every item \(1\.01\.0\): a cheap\-but\-deaf configuration \(escalationπ0\\pi\_\{0\}with sparse rate\-2020anchors\) monitors at≈0\.21\\approx 0\.21but misses most system regressions at the hard cell; the featured powered\-and\-guarded configuration \(fixed@0\.30\.3main with rate\-55anchors\) costs≈0\.64\\approx 0\.64and catches system drift with zero misattribution\. Notably, the anchor budget is the*smaller*part of the increment from cheap to powered: of the0\.430\.43cost\-fraction gap, the anchor share is only0\.200\.20— most of the added cost buys main\-process power, not anchor coverage\.

#### A practical recipe\.

These limitations compose into a deployment order, which is the payoff for a practitioner\. First, pick the main configuration on the cost/power frontier of[Section˜3](https://arxiv.org/html/2606.15474#S3)— this fixes the system power you can afford and, with it, the contaminated\-fire latency the anchors must beat\. Second, size the anchor budget\(k\(k, interleave rate\)\)so that anchor detection latency, at the drift magnitudes you actually care about, comes in*under*the main’s contaminated\-fire latency — the co\-provisioning band of[Section˜5\.4](https://arxiv.org/html/2606.15474#S5.SS4), not a bolted\-on afterthought\. Third, widen the guardWWto absorb the residual spill: it is the cheapest knob, buying attribution correctness with latency rather than dollars, and \([Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)\) it can never manufacture a falsejudgeverdict\.

## 7Conclusion

Every team that monitors an LLM product with an LLM judge eventually hits the same question: when the dashboard turns red, is it the product or the judge? We have shown that a small, fixed set of human\-labeled anchors, re\-scored by the current judge and watched by one extra anytime\-valid e\-process, answers it — separating system drift from judge drift into an explicit three\-state verdict with three transparent knobs \(anchor sizekk, interleave rate1/ρ1/\\rho, and guard widthWW\) and a guarantee that judge drift is never charged to the system beyondαjudge\\alpha\_\{\\mathrm\{judge\}\}\. The construction is validated not only on synthetic drift but on two*real*judge changes — a silent version bump and a strict\-prompt policy update — across two domains, with nothing re\-tuned between them; where the domains differ, the differences are themselves predicted by the attribution race \(a larger judge shift yields faster anchors, perfect attribution, and a co\-provisioning band shifted toward slower anchor rates\)\. Three directions remain open: principled anchor\-refresh policies that keephha valid reference as standards evolve; per\-stratum anchor families that recover power against topic\-localized judge drift; and optimal betting and attribution rules that minimize misattribution at a fixed anchor budget\.

## References

- Angelopoulos et al\. \[2023\]Anastasios N\. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I\. Jordan, and Tijana Zrnic\.Prediction\-powered inference, 2023\.URLhttps://arxiv\.org/abs/2301\.09633\.
- Angelopoulos et al\. \[2024\]Anastasios N\. Angelopoulos, John C\. Duchi, and Tijana Zrnic\.PPI\+\+: Efficient prediction\-powered inference, 2024\.URLhttps://arxiv\.org/abs/2311\.01453\.
- Chaudhuri et al\. \[2024\]Anamitra Chaudhuri, Georgios Fellouris, and Ali Tajer\.Round robin active sequential change detection for dependent multi\-channel data, 2024\.URLhttps://arxiv\.org/abs/2403\.16297\.
- Chen et al\. \[2026\]Yiqun T\. Chen, Moran Guo, and Shengy Li\.Power analysis for prediction\-powered inference, 2026\.URLhttps://arxiv\.org/abs/2603\.16041\.
- Choe and Ramdas \[2023\]Yo Joong Choe and Aaditya Ramdas\.Comparing sequential forecasters, 2023\.URLhttps://arxiv\.org/abs/2110\.00115\.
- Cortinovis and Caron \[2025\]Stefano Cortinovis and François Caron\.FAB\-PPI: Frequentist, assisted by Bayes, prediction\-powered inference, 2025\.URLhttps://arxiv\.org/abs/2502\.02363\.
- Csillag et al\. \[2025a\]Daniel Csillag, Pedro Dall’Antonia, Claudio José Struchiner, and Guilherme Tegoni Goedert\.Extending prediction\-powered inference through conformal prediction, 2025a\.URLhttps://arxiv\.org/abs/2510\.16166\.
- Csillag et al\. \[2025b\]Daniel Csillag, Claudio José Struchiner, and Guilherme Tegoni Goedert\.Prediction\-powered E\-Values, 2025b\.URLhttps://arxiv\.org/abs/2502\.04294\.
- Feng et al\. \[2026\]Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner\-Beuerle, and Miguel R\. D\. Rodrigues\.Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges, 2026\.URLhttps://arxiv\.org/abs/2601\.20913\.
- Fiedler \[2026\]James Fiedler\.Bias and uncertainty in LLM\-as\-a\-Judge estimation, 2026\.URLhttps://arxiv\.org/abs/2605\.06939\.
- Google DeepMind \[2025\]Google DeepMind\.Gemini model family\.https://deepmind\.google/technologies/gemini/, 2025\.
- Howard et al\. \[2022\]Steven R\. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon\.Time\-uniform, nonparametric, nonasymptotic confidence sequences, 2022\.URLhttps://arxiv\.org/abs/1810\.08240\.
- Howard et al\. \[2025\]Steven R\. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon\.Time\-uniform Chernoff bounds via nonnegative supermartingales, 2025\.URLhttps://arxiv\.org/abs/1808\.03204\.
- Kilian et al\. \[2025\]Valentin Kilian, Stefano Cortinovis, and François Caron\.Anytime\-valid, Bayes\-assisted, prediction\-powered inference, 2025\.URLhttps://arxiv\.org/abs/2505\.18000\.
- Lamaakal et al\. \[2026\]Ismail Lamaakal, Chaymae Yahyati, Khalid El Makkaoui, Ibrahim Ouahbi, and Yassine Maleh\.Drift\-to\-Action controllers: Budgeted interventions with online risk certificates, 2026\.URLhttps://arxiv\.org/abs/2603\.08578\.
- Lee et al\. \[2026\]Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy\-yong Sohn, and Kangwook Lee\.How to correctly report LLM\-as\-a\-Judge evaluations, 2026\.URLhttps://arxiv\.org/abs/2511\.21140\.
- Page \[1954\]E\. S\. Page\.Continuous inspection schemes\.*Biometrika*, 41\(1/2\):100–115, 1954\.
- Sfyraki and Wang \[2026\]Maria\-Eleni Sfyraki and Jun\-Kun Wang\.Revisiting active sequential prediction\-powered mean estimation, 2026\.URLhttps://arxiv\.org/abs/2604\.18569\.
- Shin et al\. \[2023\]Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo\.E\-detectors: a nonparametric framework for sequential change detection, 2023\.URLhttps://arxiv\.org/abs/2203\.03532\.
- Shiryaev \[1963\]Albert N\. Shiryaev\.On optimum methods in quickest detection problems\.*Theory of Probability & Its Applications*, 8\(1\):22–46, 1963\.
- Stiennon et al\. \[2022\]Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M\. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano\.Learning to summarize from human feedback, 2022\.URLhttps://arxiv\.org/abs/2009\.01325\.
- Ville \[1939\]Jean Ville\.Étude critique de la notion de collectif\.*Gauthier\-Villars, Paris*, 1939\.
- Wang et al\. \[2024\]Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J\. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev\.HelpSteer2: Open\-source dataset for training top\-performing reward models, 2024\.URLhttps://arxiv\.org/abs/2406\.08673\.
- Waudby\-Smith and Ramdas \[2022\]Ian Waudby\-Smith and Aaditya Ramdas\.Estimating means of bounded random variables by betting, 2022\.URLhttps://arxiv\.org/abs/2010\.09686\.
- Yi and Yang \[2025\]Yanqing Yi and Su\-Fen Yang\.Change detection with adaptive sampling for binary responses, 2025\.URLhttps://arxiv\.org/abs/2512\.15507\.
- Zhang et al\. \[2026\]Guangyi Zhang, Yunlong Cai, Guanding Yu, and Osvaldo Simeone\.Prediction\-powered risk monitoring of deployed models for detecting harmful distribution shifts, 2026\.URLhttps://arxiv\.org/abs/2602\.02229\.
- Zheng et al\. \[2023\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena, 2023\.URLhttps://arxiv\.org/abs/2306\.05685\.
- Zrnic and Candès \[2024\]Tijana Zrnic and Emmanuel J\. Candès\.Cross\-prediction\-powered inference, 2024\.URLhttps://arxiv\.org/abs/2309\.16598\.
- Zrnic and Candès \[2026\]Tijana Zrnic and Emmanuel J\. Candès\.Active statistical inference, 2026\.URLhttps://arxiv\.org/abs/2403\.03208\.

## Appendix AProofs

We prove Propositions[1](https://arxiv.org/html/2606.15474#Thmproposition1)–[4](https://arxiv.org/html/2606.15474#Thmproposition4)as stated in[Section˜4](https://arxiv.org/html/2606.15474#S4), in the same notation\. Throughout, “past” at positioniidenotes theσ\\sigma\-algebraℱi−1\\mathcal\{F\}\_\{i\-1\}generated by the anchor observationsg~1,…,g~i−1\\tilde\{g\}\_\{1\},\\dots,\\tilde\{g\}\_\{i\-1\}\(and, for the main process, the stream and acquisition history up toi−1i\-1\)\. All validity claims are inherited from standard test\-\(super\)martingale results; the contribution is the construction they apply to, not new probability theory\.

###### Proof of[Proposition˜1](https://arxiv.org/html/2606.15474#Thmproposition1)\(anytime\-validity\)\.

Fix a rubric–direction pair\(r,d\)\(r,d\)and writeei=1\+λi​sd​\(q~0​\[r\]−g~i​\[r\]\)e\_\{i\}=1\+\\lambda\_\{i\}\\,s\_\{d\}\\,\(\\tilde\{q\}\_\{0\}\[r\]\-\\tilde\{g\}\_\{i\}\[r\]\)withsbelow=\+1s\_\{\\text\{below\}\}=\+1,sabove=−1s\_\{\\text\{above\}\}=\-1\([Equation˜5](https://arxiv.org/html/2606.15474#S4.E5)\)\.

*Nonnegativity\.*By[Equation˜4](https://arxiv.org/html/2606.15474#S4.E4),g~i​\[r\]∈\[0,1\]\\tilde\{g\}\_\{i\}\[r\]\\in\[0,1\]andq~0​\[r\]∈\(0,1\)\\tilde\{q\}\_\{0\}\[r\]\\in\(0,1\), so the signed gapsd​\(q~0​\[r\]−g~i​\[r\]\)s\_\{d\}\\,\(\\tilde\{q\}\_\{0\}\[r\]\-\\tilde\{g\}\_\{i\}\[r\]\)lies in\[−1,1\]\[\-1,1\]\. The bet is constrained toλi∈\[0,λmax​\(q~0​\[r\],d\)\]\\lambda\_\{i\}\\in\[0,\\lambda\_\{\\max\}\(\\tilde\{q\}\_\{0\}\[r\],d\)\], where the directional capλmax​\(q~0​\[r\],d\)\\lambda\_\{\\max\}\(\\tilde\{q\}\_\{0\}\[r\],d\)is exactly the largestλ≥0\\lambda\\geq 0for which1\+λ⋅sd​\(q~0​\[r\]−g\)1\+\\lambda\\cdot s\_\{d\}\\,\(\\tilde\{q\}\_\{0\}\[r\]\-g\)stays nonnegative for every attainableg∈\[0,1\]g\\in\[0,1\]\(ford=belowd=\\text\{below\}the binding case isg=1g=1, givingλmax=1/\(1−q~0​\[r\]\)\\lambda\_\{\\max\}=1/\(1\-\\tilde\{q\}\_\{0\}\[r\]\); ford=aboved=\\text\{above\},g=0g=0, givingλmax=1/q~0​\[r\]\\lambda\_\{\\max\}=1/\\tilde\{q\}\_\{0\}\[r\]\)\. Henceei≥0e\_\{i\}\\geq 0for allii\.

*Conditional mean\.*The betλi\\lambda\_\{i\}is predictable — a function ofg~1,…,g~i−1\\tilde\{g\}\_\{1\},\\dots,\\tilde\{g\}\_\{i\-1\}only \(in code the plug\-inλ≈mean⁡\(d\)/mean⁡\(d2\)\\lambda\\approx\\operatorname\{mean\}\(d\)/\\operatorname\{mean\}\(d^\{2\}\)computed from past gaps\)\. UnderH0ancH\_\{0\}^\{\\mathrm\{anc\}\}we have𝔼​\[g~i​\[r\]∣ℱi−1\]=q~0​\[r\]\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}\[r\]\\mid\\mathcal\{F\}\_\{i\-1\}\]=\\tilde\{q\}\_\{0\}\[r\], so

𝔼​\[ei∣ℱi−1\]=1\+λi​sd​\(q~0​\[r\]−𝔼​\[g~i​\[r\]∣ℱi−1\]\)=1\+λi​sd⋅0=1,\\mathbb\{E\}\[e\_\{i\}\\mid\\mathcal\{F\}\_\{i\-1\}\]=1\+\\lambda\_\{i\}\\,s\_\{d\}\\,\\big\(\\tilde\{q\}\_\{0\}\[r\]\-\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}\[r\]\\mid\\mathcal\{F\}\_\{i\-1\}\]\\big\)=1\+\\lambda\_\{i\}\\,s\_\{d\}\\cdot 0=1,using predictability ofλi\\lambda\_\{i\}to pull it outside the conditional expectation\.

*Supermartingale\.*Therefore𝔼​\[𝒲n\(r,d\)∣ℱn−1\]=𝒲n−1\(r,d\)⋅𝔼​\[en∣ℱn−1\]=𝒲n−1\(r,d\)\\mathbb\{E\}\[\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\\mid\\mathcal\{F\}\_\{n\-1\}\]=\\mathcal\{W\}\_\{n\-1\}^\{\(r,d\)\}\\cdot\\mathbb\{E\}\[e\_\{n\}\\mid\\mathcal\{F\}\_\{n\-1\}\]=\\mathcal\{W\}\_\{n\-1\}^\{\(r,d\)\}, with𝒲0\(r,d\)=1\\mathcal\{W\}\_\{0\}^\{\(r,d\)\}=1\(empty product\)\. Each\(𝒲n\(r,d\)\)n\(\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\)\_\{n\}is a nonnegative martingale, hence a nonnegative supermartingale\.

*Crossing bound and union\.*Ville’s inequality\[Ville,[1939](https://arxiv.org/html/2606.15474#bib.bib22)\]for a nonnegative supermartingale with unit initial value gives, for any levelβ\>0\\beta\>0,ℙ\(∃n:𝒲n\(r,d\)≥β\)≤1/β\\mathbb\{P\}\(\\exists n:\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\\geq\\beta\)\\leq 1/\\beta\. Takingβ=2​R/αjudge\\beta=2R/\\alpha\_\{\\mathrm\{judge\}\}bounds each pair’s ever\-crossing probability byαjudge/2​R\\alpha\_\{\\mathrm\{judge\}\}/2R\. A union bound over the2​R2Rrubric–direction pairs gives

ℙ\(∃n:maxr,d𝒲n\(r,d\)≥2R/αjudge\)≤2R⋅αjudge2​R=αjudge\.\\mathbb\{P\}\\Big\(\\exists n:\\max\_\{r,d\}\\mathcal\{W\}\_\{n\}^\{\(r,d\)\}\\geq 2R/\\alpha\_\{\\mathrm\{judge\}\}\\Big\)\\;\\leq\\;2R\\cdot\\frac\{\\alpha\_\{\\mathrm\{judge\}\}\}\{2R\}=\\alpha\_\{\\mathrm\{judge\}\}\.Ville’s inequality is time\-uniform, so the bound holds at any data\-dependent stopping time\. ∎

###### Proof of[Proposition˜2](https://arxiv.org/html/2606.15474#Thmproposition2)\(identification\)\.

Each anchor observation at any position is a deterministic function of \(a\) the fixed items𝒜\\mathcal\{A\}, \(b\) their frozen labelshh, and \(c\) the current judge, through[Equation˜4](https://arxiv.org/html/2606.15474#S4.E4)\. System drift — any change in the distribution of*new*stream items or in their true quality — alters none of \(a\)–\(c\), so it leaves the anchor gap distribution unchanged\. HenceH0ancH\_\{0\}^\{\\mathrm\{anc\}\}can be violated only by a change in the judge\.

*\(a\)*Under pure system drift the judge is unchanged, soH0ancH\_\{0\}^\{\\mathrm\{anc\}\}holds\. Ajudgeverdict \([Equation˜6](https://arxiv.org/html/2606.15474#S4.E6)\) requiresτanc<∞\\tau\_\{\\mathrm\{anc\}\}<\\infty, i\.e\. the anchor family fires;[Proposition˜1](https://arxiv.org/html/2606.15474#Thmproposition1)bounds the probability of that event byαjudge\\alpha\_\{\\mathrm\{judge\}\}at any horizon, and this bound does not involveWW\. Thereforeℙ​\(verdict=judge∣pure system drift\)≤αjudge\\mathbb\{P\}\(\\text\{verdict\}=\\texttt\{judge\}\\mid\\text\{pure system drift\}\)\\leq\\alpha\_\{\\mathrm\{judge\}\}, uniformly inWW\.

*\(b\)*Under no drift at all, both nulls hold\. The event\{verdict≠none\}\\\{\\text\{verdict\}\\neq\\texttt\{none\}\\\}requires at least one family to fire, i\.e\.\{τsys<∞\}∪\{τanc<∞\}\\\{\\tau\_\{\\mathrm\{sys\}\}<\\infty\\\}\\cup\\\{\\tau\_\{\\mathrm\{anc\}\}<\\infty\\\}\. By the main\-process guarantee \([Section˜3](https://arxiv.org/html/2606.15474#S3), inherited fromCsillag et al\. \[[2025b](https://arxiv.org/html/2606.15474#bib.bib8), Thm\. 2\.1\]with predictableπ\\piand Ville’s inequality\)ℙ​\(τsys<∞\)≤αsys\\mathbb\{P\}\(\\tau\_\{\\mathrm\{sys\}\}<\\infty\)\\leq\\alpha\_\{\\mathrm\{sys\}\}, and by[Proposition˜1](https://arxiv.org/html/2606.15474#Thmproposition1)ℙ​\(τanc<∞\)≤αjudge\\mathbb\{P\}\(\\tau\_\{\\mathrm\{anc\}\}<\\infty\)\\leq\\alpha\_\{\\mathrm\{judge\}\}\. A union bound givesℙ​\(verdict≠none\)≤αsys\+αjudge\\mathbb\{P\}\(\\text\{verdict\}\\neq\\texttt\{none\}\)\\leq\\alpha\_\{\\mathrm\{sys\}\}\+\\alpha\_\{\\mathrm\{judge\}\}; the two budgets are separate by design\. ∎

###### Proof of[Proposition˜3](https://arxiv.org/html/2606.15474#Thmproposition3)\(the attribution race\)\.

Suppose judge drift has occurred \(with or without simultaneous system drift\)\. By the verdict rule[Equation˜6](https://arxiv.org/html/2606.15474#S4.E6),verdict=system\\text\{verdict\}=\\texttt\{system\}holds exactly whenτsys<∞\\tau\_\{\\mathrm\{sys\}\}<\\inftyand \(τanc=∞\\tau\_\{\\mathrm\{anc\}\}=\\inftyorτanc\>τsys\+W\\tau\_\{\\mathrm\{anc\}\}\>\\tau\_\{\\mathrm\{sys\}\}\+W\), which is precisely the event\{τsys\+W<τanc\}\\\{\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\\\}\(adopting the conventionτsys\+W<∞=τanc\\tau\_\{\\mathrm\{sys\}\}\+W<\\infty=\\tau\_\{\\mathrm\{anc\}\}when the anchors never fire\)\. Hence

ℙ​\(verdict=system∣judge drift\)=ℙ​\(τsys\+W<τanc\)\.\\mathbb\{P\}\(\\text\{verdict\}=\\texttt\{system\}\\mid\\text\{judge drift\}\)=\\mathbb\{P\}\(\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\)\.
*Monotonicity inWW\.*For fixed sample paths of the two stopping times, the indicator𝟏​\{τsys\+W<τanc\}\\mathbf\{1\}\\\{\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\\\}is non\-increasing inWWpointwise; taking expectations preserves the order, so the misattribution probability is monotonically non\-increasing inWW\.

*Monotonicity in the anchor budget\.*Increasing the interleave rate1/ρ1/\\rhoor the anchor set sizekksupplies more anchor evidence per position, which stochastically decreasesτanc\\tau\_\{\\mathrm\{anc\}\}\(first\-order stochastic dominance: more accumulated wealth per step reaches the threshold2​R/αjudge2R/\\alpha\_\{\\mathrm\{judge\}\}no later\)\. A stochastically smallerτanc\\tau\_\{\\mathrm\{anc\}\}makes the event\{τsys\+W<τanc\}\\\{\\tau\_\{\\mathrm\{sys\}\}\+W<\\tau\_\{\\mathrm\{anc\}\}\\\}less likely, so the misattribution probability is non\-increasing in1/ρ1/\\rhoand inkk\.

*Monotonicity in main power\.*Increasing the main process’s sampling power \(more strong calls on the contaminatedyy\) stochastically decreasesτsys\\tau\_\{\\mathrm\{sys\}\}, by the same first\-order argument applied to the main e\-process: more strong labels per step grow the contaminated wealth no slower toward its threshold\. A stochastically smallerτsys\\tau\_\{\\mathrm\{sys\}\}makes the misattribution event more likely, so the probability is non\-decreasing in main power\.

These are stochastic\-dominance arguments at the level of rigor of the formal\-statements source; we do not claim a sharper coupling\. ∎

###### Proof of[Proposition˜4](https://arxiv.org/html/2606.15474#Thmproposition4)\(orthogonality\)\.

The anchor observationsg~i\\tilde\{g\}\_\{i\}are a deterministic function of\(𝒜,h,current judge\)\(\\mathcal\{A\},h,\\text\{current judge\}\)and the fixed interleave schedule \([Equation˜4](https://arxiv.org/html/2606.15474#S4.E4)\), none of which depend on the main acquisition rule \(policy, budget, orπ\\pi\)\. Therefore the law of the anchor process — and in particularτanc\\tau\_\{\\mathrm\{anc\}\}— does not depend on the main\-process configuration\. Symmetrically, the anchor re\-judgements are out\-of\-band calls on held\-out items that, by the anchor\-set definition \([Section˜4](https://arxiv.org/html/2606.15474#S4)\), are excluded from all main\-process calibration and never enter the main e\-values; the anchor process never reads stream items\. Hence the main process’s distribution does not depend on the anchor interleave\. Given the judge state, the two stopping times are functions of disjoint randomness, and the design splits cleanly: the anchor budget\(k,ρ\)\(k,\\rho\)and guardWWgovern judge\-drift latency and the race; the main configuration governs system power; andαjudge,αsys\\alpha\_\{\\mathrm\{judge\}\},\\alpha\_\{\\mathrm\{sys\}\}are separate Bonferroni budgets\. ∎

#### What is not claimed\.

Two boundaries, transcribed from the formal\-statements source\.*No optimality*: the plug\-inλ\\lambdaapproximates the log\-optimal bet, and the attribution rule is not claimed to minimize misattribution at a fixed budget\.*Anchor staleness*:[Proposition˜2](https://arxiv.org/html/2606.15474#Thmproposition2)assumes the human labelshhremain a valid reference; if the anchor items’*true*quality standard drifts conceptually \(e\.g\. the rubric meaning changes\), the anchor process correctly reports judge\-vs\-anchor disagreement, but its reading as judge drift weakens until a periodic anchor refresh re\-establishes the baseline\. Finally, the anchor family is pooled across strata \(rubric×\\timesdirection only\), so a judge change confined to one stratum is detected at reduced power; topic stratification lives entirely on the system\-process side\.

## Appendix BThe Cost\-Aware Foundation in Detail

This appendix gives the full cost\-frontier table, the ten\-method prior\-work comparison, and the trigger ablations summarized in[Section˜3](https://arxiv.org/html/2606.15474#S3)\. All numbers are for the semi\-synthetic localized blind\-spot drift in the helpfulness×\\timescreative cell of HelpSteer2 \(streamN=1200N=1200, change at300300\), averaged over6060Monte\-Carlo seeds, atαsys=0\.1\\alpha\_\{\\mathrm\{sys\}\}=0\.1\.

#### Cost model recap\.

Each judge call costs\(input tokens\)⋅\(input price\)\+\(output tokens\)⋅\(output price\)\\text\{\(input tokens\)\}\\cdot\\text\{\(input price\)\}\+\\text\{\(output tokens\)\}\\cdot\\text\{\(output price\)\}at representative counts of≈600\\approx 600input and≈30\\approx 30output tokens, giving$​0\.000195\\mathdollar 0\.000195per cheap call and$​0\.00156\\mathdollar 0\.00156per strong call\. The cheap judge runs on allLLprocessed items and the strong judge onSSsampled items, so relative to strong\-evaluating every item thecost\-fraction=\(L​cμ\+S​cy\)/\(L​cy\)=0\.125\+S/L\\text\{cost\-fraction\}=\(Lc\_\{\\mu\}\+Sc\_\{y\}\)/\(Lc\_\{y\}\)=0\.125\+S/L, with full evaluation \(π≡1\\pi\\equiv 1\) equal to1\.01\.0by definition\. The0\.1250\.125is the irreducible cheap floor andS/LS/Lis the realized strong\-sampling rate\.

### B\.1The cost–reliability frontier

[Table˜6](https://arxiv.org/html/2606.15474#A2.T6)reports the per\-stratum monitor across both acquisition modes\. Latency is the median steps from the change point over uncensored reps; censor is the fraction of reps that never alarmed within the horizon\.

Table 6:Cost–reliability frontier for the per\-stratum monitor \(helpfulness×\\timescreative,ylow=0\.25y\_\{\\mathrm\{low\}\}=0\.25, 60 seeds\)\. Latency = median steps to detection over uncensored reps; censor = fraction never alarming within the horizon \(N=1200N=1200, change at300300\)\.Every detector catches the drift at cost\-fraction0\.210\.21–0\.430\.43, about one third of full strong evaluation, with false\-alarm0\.0000\.000across all1616policy points \(empirically anytime\-valid\)\. The two modes trace a frontier: escalation is cheapest \(0\.210\.21–0\.280\.28\), sitting at the0\.125\+π00\.125\+\\pi\_\{0\}floor in\-control — near\-zero strong calls when nothing is wrong; fixed budgets buy reliability \(@0\.2→\\tocensor0\.500\.50, @0\.3→\\tocensor0\.280\.28\); full evaluation is fastest \(latency327327\) at full price\. Escalationπ0=0\.1\\pi\_\{0\}=0\.1\(cost0\.280\.28, censor0\.530\.53\) is about as reliable as fixed @0\.2 \(cost0\.330\.33, censor0\.500\.50\) but cheaper, missing the strict censor\-≤0\.5\\leq 0\.5cut by one rep\. This drift is intrinsically hard \(even full\-eval censors2%2\\%\), which compresses escalation’s margin; a stronger or cleaner drift, or a higherπ0\\pi\_\{0\}, lets escalation dominate the frontier outright\.

### B\.2The ten\-method prior\-work comparison

We place the four prior\-work families in one harness on one frontier, calibrated to the same false\-alarm level\.[Table˜7](https://arxiv.org/html/2606.15474#A2.T7)lists what each method sees and acquires; “seesμ\\mu” means it runs the cheap judge on every item, and the cost\-fraction charges each method only for what it calls\.

Table 7:The ten\-method roster\. “isolates” names the design axis each method isolates\.The prior\-work mapping: A2/A3 are the concept\-drift family \(Page–Hinkley\[Page,[1954](https://arxiv.org/html/2606.15474#bib.bib17)\]/Shiryaev–Roberts\[Shiryaev,[1963](https://arxiv.org/html/2606.15474#bib.bib20)\]\); B1 is label\-only SAVI; C1 is the global \(un\-stratified\) PPI e\-process ofCsillag et al\. \[[2025b](https://arxiv.org/html/2606.15474#bib.bib8)\]; D2 is Zrnic–Candès uncertainty\-triggered active inference\[Zrnic and Candès,[2026](https://arxiv.org/html/2606.15474#bib.bib29)\]; D3 is the Csillag App B\.2 active\-PPI rule; D4 is ours; E1 is the naive ground\-truth\-everything monitor\. Every e\-process method is false\-alarm\-valid by construction \(Bonferroni\), and all four per\-stratum PPI rules \(D1–D4\) share the*same*per\-item bet capλmax​\(πi\)\\lambda\_\{\\max\}\(\\pi\_\{i\}\)on their realized \(predictable\) acquisition probability, so no method gets a betting head\-start\.

#### The acquisition\-rule headline\.

[Table˜8](https://arxiv.org/html/2606.15474#A2.T8)ranks the four per\-stratum PPI rules, which differ*only*in the acquisition rule, against drift strength\. “Detects” means censor≤0\.5\\leq 0\.5and false\-alarm≤α\\leq\\alpha\. Because the strong labelyylives on the HelpSteer2 5\-level grid\{0,0\.25,0\.5,0\.75,1\.0\}\\\{0,0\.25,0\.5,0\.75,1\.0\\\}, the four\-point drift sweep \(ylow∈\{0\.15,0\.25,0\.35,0\.45\}y\_\{\\mathrm\{low\}\}\\in\\\{0\.15,0\.25,0\.35,0\.45\\\}\) collapses to two effective magnitudes in this cell:ylow=0\.15y\_\{\\mathrm\{low\}\}=0\.15draws a pool of one extreme item \(severe\), whileylow∈\{0\.25,0\.35,0\.45\}y\_\{\\mathrm\{low\}\}\\in\\\{0\.25,0\.35,0\.45\\\}draw the*identical*21\-item pool \(mild, byte\-identical in the raw JSON\)\. We report the comparison as severe vs mild\.

Table 8:Acquisition\-rule ranking vs drift strength \(per\-stratum PPI; D1–D4 differ only in the trigger, sharing the per\-item bet cap\)\. Severe =ylow=0\.15y\_\{\\mathrm\{low\}\}=0\.15; mild =ylow≥0\.25y\_\{\\mathrm\{low\}\}\\geq 0\.25\.The covariate\-triggered prior\-art rules do not fail to*detect*— they fail to detect*efficiently*, each for a distinct reason\. D2 \(boundary\) detects on both drifts but only by oversampling to cost≈0\.87\\approx 0\.87–0\.890\.89\(near full\-eval\): its “label near the decision boundary” heuristic fires near*every*rubric’s bar across the3030cells, soπ\\piis pushed high regardless of where the drift is\. D3 \(the published Csillag App B\.2 rule\) is inert: its predicted\-ee\-growth trigger reads identically zero because the cheap judge overrates*every*healthy bar \(q0−μ<0q\_\{0\}\-\\mu<0on all five rubrics: helpfulness−0\.18\-0\.18, complexity−0\.23\-0\.23, verbosity−0\.27\-0\.27\), so it degenerates to fixed\-budget at rateπ0\\pi\_\{0\}and fails on the mild drift\. D4 \(ours\), keying on accumulatedee\-evidence rather than the blind cheap covariate, detects at cost0\.280\.28— roughly one third of D2’s — by spending only once evidence accrues; on the faint mild drift it merely*ties*a matched fixed budget \(D4@0\.1 censor 0\.53 cost 0\.28≈\\approxD1@0\.2 censor 0\.50 cost 0\.33\), exactly the regime ofSfyraki and Wang \[[2026](https://arxiv.org/html/2606.15474#bib.bib18)\]where an adaptive trigger buys essentially nothing over fixed\-τ\\taufor prediction\-powered mean estimation\. We report the tie as\-is\. The central result, restated: keying acquisition on the blind cheap covariate is either wasteful \(D2 oversamples to≈0\.88\\approx 0\.88\) or inert \(D3 reads zero under the overrating bias\); keying on accumulatedee\-evidence \(D4\) detects at one third of that cost\.

#### Two further contrasts\.

*Stratification vs global*: per\-\(rubric×\\timestopic\) stratification is decisive\. At every matched budget, global PPI \(C1\) censors strictly worse than per\-stratum fixed\-budget \(D1\) — severe @0\.2 global0\.680\.68vs stratified0\.120\.12, @0\.30\.580\.58vs0\.050\.05; on the mild drift global PPI censors1\.001\.00at all three budgets while stratified detects at @0\.2 \(0\.500\.50\) and @0\.3 \(0\.270\.27\)\. A localized blind\-spot drift is diluted to invisibility when pooled\.*Classic detectors do not transfer*: the two concept\-drift detectors are threshold\-calibrated to family\-wise false\-alarm≈α\\approx\\alphaon a held\-out, disjointH0H\_\{0\}seed block, yet off that sample held\-out Page–Hinkley records false\-alarm0\.1170\.117\(just overα\\alpha\) and held\-out Shiryaev–Roberts0\.2670\.267\(well overα\\alpha\) — the realized maximum false\-alarm across all methods and settings,0\.2670\.267, is driven entirely by the classic SR detector, while the e\-process spine holds at false\-alarm≤0\.017\\leq 0\.017\(the lone non\-zero e\-process value is covariate\-boundaryπ0=0\.1\\pi\_\{0\}=0\.1at0\.0170\.017, oneH0H\_\{0\}alarm in6060, withinα\\alpha\)\. The cheap proxy alone never sees this drift: all three cheap\-only methods censor near1\.01\.0everywhere \(A1 ablation1\.001\.00, A3 SR1\.001\.00, A2 Page–Hinkley0\.980\.98severe /0\.920\.92mild\), by construction of the blind\-spot drift\.

## Appendix CAll\-Cells Results

[Table˜9](https://arxiv.org/html/2606.15474#A3.T9)gives the full per\-\(topic×\\timesrubric\) cell table summarized in[Section˜5\.6](https://arxiv.org/html/2606.15474#S5.SS6), for the featured configuration \(fixed@0\.3 main,k=200k=200anchors at rate1/51/5, guardW=50W=50, synthetic shift−0\.25\-0\.25,2020seeds\) on HelpSteer2\. The two near\-ceiling coherence cells coding:coherence and math:coherence are skipped: their regressed\-blind pool \(y≤0\.5∧μ≥0\.6y\\leq 0\.5\\wedge\\mu\\geq 0\.6\) is empty because coherence sits at∼0\.92\{\\sim\}0\.92\. Columns: judge detection rate, median judge\-drift latency \(items\), judge→\\tosystem spill rate, false\-judge rate, and system detection rate\.

The C2 properties are cell\-independent by construction — the anchor process never looks at the main\-process cell\. Judge detection is1\.001\.00in27/2827/28cells \(and0\.970\.97in factual\_qa:coherence\), at latency9898–128128items everywhere; judge→\\tosystem is0\.000\.00in27/2827/28\(0\.030\.03in that same cell\); and the false\-judge rate is at most0\.100\.10, underαjudge\\alpha\_\{\\mathrm\{judge\}\}\. Only the system detection rate varies, a pure C1 sensitivity property: correctness and coherence cells detect at0\.800\.80–1\.001\.00, helpfulness in the middle \(0\.450\.45–0\.950\.95\), and complexity/verbosity≈0\\approx 0everywhere \(the rubrics whereμ\\muis blind and the regressed pools are thin\), with all math cells≈0\\approx 0\. Every missed system regression fails safe tonone, never a falsejudge\.

Table 9:Every topic×\\timesrubric cell at the featured config \(HelpSteer2, fixed@0\.3 main,k=200k\{=\}200/rate\-55anchors,W=50W\{=\}50, synthetic−0\.25\-0\.25, 20 seeds\)\. 28 runnable cells in two column blocks; coding:coherence and math:coherence are skipped \(empty regressed\-blind pool\)\. jdet = judge detection, jlat = median judge latency \(items\), j→\\tos = judge→\\tosystem spill, fj = false\-judge, sdet = system detection\.The TL;DR replication of this sweep \([Section˜5\.6](https://arxiv.org/html/2606.15474#S5.SS6)\) runs19/2019/20cells \(AskReddit:coherence skipped, near\-ceiling\): judge detection0\.850\.85–0\.970\.97at latency118118–145145in every cell, judge→\\tosystem≤0\.15\\leq 0\.15atW=50W=50, false\-judge≤0\.10=αjudge\\leq 0\.10=\\alpha\_\{\\mathrm\{judge\}\}, and system detection0\.400\.40–1\.001\.00\(weakest on coverage cells, the rubric whereμ\\muis most generous\)\.

## Appendix DCheap\-Judge Calibration

The cheap judgeμ\\mu\(gemini\-3\.1\-flash\-lite\) is run on every item as free side information for the prediction\-powered main process\.[Table˜10](https://arxiv.org/html/2606.15474#A4.T10)reports how much signal it carries per rubric, from the fitted\-calibration study: a per\-rubricμ→y\\mu\\to ycorrection \(shift / affine / isotonic\) is fit on the first600600HelpSteer2 items and measured on the held\-out next600600\.

Table 10:Cheap\-judge calibration against the strong judge, held\-out \(n=600n=600test\)\. Bias=μ−y=\\mu\-y; corr is Pearson; affine slopeaais the fittedμ→y\\mu\\to ygain; RMSE↓\\downarrowis the held\-out RMSE reduction of the affine correction vs rawμ\\mu\.The corrections transfer cleanly: held\-out residual bias≈0\\approx 0on every rubric, and calibration RMSE≈\\approxtest RMSE \(no overfit\)\. The affine fit≈\\approxisotonic fit \(sinceμ\\mulives on a 5\-level grid\), and a 1\-parameter*shift*already removes most of the error; the blind\-spot structure is stable across the split \(\|Δ​bias\|≤0\.01\|\\Delta\\text\{bias\}\|\\leq 0\.01,\|Δ​blind%\|≤5\|\\Delta\\text\{blind\\%\}\|\\leq 5points\)\. The takeaway: calibration removes the cheap judge’s systematic generosity but cannot create resolution it never had\. The affine slopeaameasures how muchμ\\muis worth:a≈1a\\approx 1\(helpfulness\) means keep its signal and just shift it;a≈0a\\approx 0\(verbosity\) meansμ\\muis essentially the base rate, so the strong judge is required there\. Helpfulness and complexity become good calibrated proxies; verbosity does not — which is exactly the rubric where the localized blind\-spot drift of[Appendix˜B](https://arxiv.org/html/2606.15474#A2)hides, and where the large judge\-version gap of[Table˜3](https://arxiv.org/html/2606.15474#S5.T3)appears\.

## Appendix EJudge Prompts

The strong and cheap judges share a scoring prompt; only the model snapshot differs\. Drift is injected on the strong judge in two ways: a real version bump \(re\-judging with a smaller model\) needs no prompt change, while the real*harsh*change re\-judges with the same model under a deliberately stricter scoring prompt\. We quote both prompt versions verbatim from the registry \(driftjudge/data/judges\.py\); the rubric list and prompt/response are interpolated at the marked fields\.

#### Baseline scoring prompt \(v1\)\.

> You are a strict evaluator\. Score the assistant RESPONSE to the PROMPT on each rubric using an integer 0\-4 \(HelpSteer2 scale: 0 worst, 4 best\)\. Reply with ONLY a JSON object mapping each rubric name to its integer score\. Rubrics: \{rubrics\} PROMPT: \{prompt\} RESPONSE: \{response\}

#### Harsh scoring prompt \(v2\-strict\)\.

A deliberately harsher scoring policy on the same scale and model, emulating a real silent scoring\-policy update \(C2’s contamination direction\)\. The money clause is the strict tie\-break:

> You are an exacting, skeptical evaluator\. Score the assistant RESPONSE to the PROMPT on each rubric using an integer 0\-4 \(HelpSteer2 scale: 0 worst, 4 best\)\. Be strict: reserve 4 for flawless work, penalize every error, omission, or unsupported claim you notice, and when torn between two adjacent scores always choose the lower\. Reply with ONLY a JSON object mapping each rubric name to its integer score\. Rubrics: \{rubrics\} PROMPT: \{prompt\} RESPONSE: \{response\}

The TL;DR summarization domain uses parallel variantsv1\-summandv2\-strict\-summ\(same instructions, with POST/SUMMARY framing and the four summarization axes substituted for the rubric meanings\)\. An unknownprompt\_versionraises\. The harsh re\-judge has33HelpSteer2 items \(and11TL;DR item\) that the strong endpoint cannot score \(content filter\); these fall back to the undriftedyy, so the contaminated stream is1495/14981495/1498\(HelpSteer2\) and1497/14981497/1498\(TL;DR\)\.

## Appendix FReproducibility

The pipeline has two phases\. The*data layer*is a one\-time paid Gemini cache \(the only network/API phase\): every item is scored once by the cheap judge, once by the strong judge, and once per drift slice, and the human labels are joined in\. All experiments afterward are*compute\-only*: they read the cached scores fromartifacts/and re\-run the monitor in simulation, so reproducing every number below requires no API access once the cache exists\. The cached artifacts are*not committed to the source tree*\(artifacts/is gitignored\); re\-collecting scores from a later judge version would yield different numbers — which is precisely the paper’s point\. For exact public reproduction, the intended release vehicle is a separate versioned artifact archive, not committed source files\. Figures regenerate from the checked\-in computed outputs viamake figures\(scripts/paper\_figures\.py\)\.

#### Datasets and licensing\.

HelpSteer2\[Wang et al\.,[2024](https://arxiv.org/html/2606.15474#bib.bib23)\]is released under CC\-BY\-4\.0 \(© NVIDIA\); the OpenAI summarize\-from\-feedback feedback data is released under OpenAI’s modified MIT license; and the underlying Webis TLDR corpus is listed under CC\-BY\-4\.0\. The reproducibility artifact redistributes only derived judge scores and human labels/metadata, with attribution, not raw source text\. Gemini scores are Gemini\-API outputs\.

#### Data layer \(one\-time, paid\)\.

```
HF_HUB_OFFLINE=1 uv run --extra pipeline python scripts/add_human_labels.py
GEMINI_API_KEY=... uv run --extra pipeline python scripts/judge_drift_data.py \
    --drift-snapshot gemini-3.5-flash --limit 1498
GEMINI_API_KEY=... HF_HUB_OFFLINE=1 uv run --extra pipeline python \
    scripts/judge_drift_data.py \
    --drift-snapshot gemini-3.1-pro-preview --prompt-version v2-strict --limit 1498
```

#### HelpSteer2 experiments \(compute\-only\)\.

```
# Result 1b -- high-power main (fixed @0.3), matched vs old anchor budget:
uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \
    --main-mode fixed --main-budget 0.3
uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 50 --anchor-rate 20 \
    --main-mode fixed --main-budget 0.3
# Result 3 -- real lenient bump, featured [email protected] main:
uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \
    --main-mode fixed --main-budget 0.3 --real-drift-snapshot gemini-3.5-flash
# Result 4 -- real harsh drift, guard sweep (data study first):
uv run python scripts/study_judge_version_gap.py \
    --snapshot gemini-3.1-pro-preview --prompt-version v2-strict
for W in 50 150 300; do \
  uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \
      --guard $W --main-mode fixed --main-budget 0.3 \
      --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict; done
# Result 5 -- co-provisioning frontier (real harsh drift):
uv run python scripts/c2_anchor_experiment.py --seeds 30 --k 200 --guard 50 --frontier \
    --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict
# Result 6 -- every cell at the featured config -> artifacts/c2_cells.json:
uv run python scripts/c2_anchor_experiment.py --seeds 20 --k 200 --anchor-rate 5 \
    --main-mode fixed --main-budget 0.3 --all-cells
# Result 7 -- baselines on the identical anchor stream:
uv run python scripts/c2_baseline_comparison.py --seeds 60
```

#### TL;DR second dataset\.

The TL;DR data layer is a separate one\-time paid run \(raw files fetched perdriftjudge/data/tldr\.py\); every experiment reuses the same drivers with\-\-dataset tldr\(results land in\*\_tldr\.jsonartifacts, so the HelpSteer2 result JSONs are untouched\)\. The TL;DR halves of every table and figure mirror the HelpSteer2 commands above:

```
# data layer (paid, one-time, ~6k calls):
GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256
GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256 \
    --drift-snapshot gemini-3.5-flash --prompt-version v1-summ
GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256 \
    --drift-snapshot gemini-3.1-pro-preview --prompt-version v2-strict-summ
# experiments (compute-only):
E="uv run python scripts/c2_anchor_experiment.py --dataset tldr"
# confusion (matched + under-provisioned):
$E --seeds 60 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3
$E --seeds 60 --k 50 --anchor-rate 20 --main-mode fixed --main-budget 0.3
# lenient bump: anchor-budget sweep (escalation main) + featured [email protected]:
$E --seeds 60 --k 50 --anchor-rate 20 --real-drift-snapshot gemini-3.5-flash
$E --seeds 60 --k 150 --anchor-rate 8 --real-drift-snapshot gemini-3.5-flash
$E --seeds 60 --k 200 --anchor-rate 5 --real-drift-snapshot gemini-3.5-flash
$E --seeds 60 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3 \
    --real-drift-snapshot gemini-3.5-flash
# harsh drift: per-axis study + guard sweep:
uv run python scripts/study_judge_version_gap.py --dataset tldr
uv run python scripts/study_judge_version_gap.py --dataset tldr \
    --snapshot gemini-3.1-pro-preview --prompt-version v2-strict
for W in 50 150 300; do \
  $E --seeds 60 --k 200 --anchor-rate 5 --guard $W --main-mode fixed \
      --main-budget 0.3 --real-drift-snapshot gemini-3.1-pro-preview \
      --drift-prompt-version v2-strict; done
# co-provisioning frontier, every cell, and baselines:
$E --seeds 30 --k 200 --guard 50 --frontier \
    --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict
$E --seeds 20 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3 --all-cells
uv run python scripts/c2_baseline_comparison.py --dataset tldr --seeds 60
```

All experiment outputs land in gitignored JSON underartifacts/\(e\.g\.c2\_confusion\.json,c2\_frontier\.json,c2\_cells\.json,c2\_baselines\.json\); the C1 cost frontier is reproduced byuv run python scripts/c1\_stratified\_experiment\.py \-\-seeds 60and the ten\-method comparison byuv run python scripts/c1\_method\_comparison\.py \-\-seeds 60\.