Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

# LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
Source: [https://arxiv.org/html/2606.10315](https://arxiv.org/html/2606.10315)
Sawyer ZhangAlexander WangSophie Lei Lumivate \(Lumi\)

\(June 8, 2026\)

###### Abstract

LLM\-as\-judge has become the default instrument for evaluating conversational agents, yet its reliability is almost always reported as*agreement with human ratings*, not as*recall of real defects*\. We study a deployed, multi\-turn food\-and\-beverage ordering agent and measure how many genuine quality problems its built\-in LLM judge actually catches, using exhaustive human transcript review as ground truth\. Across three review batches the judge surfaces well under a quarter of human\-confirmed systematic problems—2 of 9 patterns \(22%22\\%\) in one batch, and its operational gate flagged*zero*of 100 rounds in a batch where human review confirmed 23 distinct defects and 7 new cross\-cutting patterns\. We contribute a*blind\-spot taxonomy*showing the failure is structured rather than random: the judge reliably catches*turn\-local*issues \(a fabricated statistic, a wrong language\) but systematically misses*cross\-turn state*issues \(confirm\-gate lockout, cart hallucination, escalation lockout, stale referents\)\. We trace a concrete mechanism: the judge’s scoring rubric exposes only three coarse axes \(intent, brand\-voice, personalization\) and has no category for the behavioural dimensions—state\-tracking, guardrails, recovery—where most confirmed defects cluster\. The failure is one of routing, not perception: across the corpus, 113 of 114 rounds whose raw judge note describes a confirm\-gate or cart\-state defect are scored “brand voice”, and none reach an operational failure—the shipping gate is wired to hangs and hard assertions, not to the rubric—so the0%0\\%is a routing\-and\-wiring failure, not blindness\. The consequence for prevalence estimation is sharp: when the gate’s apparent defect rate is zero the Rogan–Gladen correction degenerates—no apparent\-prevalence signal can recover the true rate at all, a stronger statement than any fixed multiplier—while where the gate reports a nonzero rate the same estimator implies a33–6×6\\times\(order\-of\-magnitude\) undercount under our measured sensitivity\. We argue that for production multi\-turn agents automated judging is a regression floor, not a substitute for human review\.

## 1Introduction

LLM\-as\-judge\(Zheng et al\.,[2023](https://arxiv.org/html/2606.10315#bib.bib17); Liu et al\.,[2023](https://arxiv.org/html/2606.10315#bib.bib9)\)is now standard for evaluating open\-ended and agentic systems: it scales, it is cheap, and on single\-turn preference tasks it correlates well with human raters\. A large literature studies its*biases*—position, verbosity, self\-preference\(Ye et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib16); Shi et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib13); Wu et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib14)\)—and its*agreement*with humans\(Gu et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib3); Han et al\.,[2025](https://arxiv.org/html/2606.10315#bib.bib5)\)\. Far less is known about a property that matters more for shipping software: when an LLM judge runs as the automated quality gate on a production agent’s live traffic,*what fraction of the real problems does it actually catch?*

We answer this for one concrete, deployed system: a per\-tenant, multi\-turn ordering agent for South\-East\-Asian food\-and\-beverage brands, evaluated on a randomized multi\-turn corpus with exhaustive human transcript review as ground truth\. Our findings are uncomfortable and, we argue, generalizable in shape if not in exact value:

- •Low recall, not just low agreement\.The built\-in judge catches well under a quarter of human\-confirmed systematic problems\. In one batch it auto\-captured 2 of 9 human\-identified patterns \(22%22\\%\); in a 100\-round batch it flagged*zero*turns as failures while human review confirmed 23 distinct ledger defects and promoted 7 new patterns\.
- •The blind spot is structured\.Recall degrades monotonically with how “cross\-turn” a problem is\. Turn\-local, surface\-checkable issues \(fabricated statistics, wrong\-language replies\) are caught; issues that require reasoning over the conversation arc \(confirm\-gate state, cart contents, escalation flags, last\-proposal referents\) are missed\.
- •A mechanism, not a mystery\.The judge’s own rubric has three axes \(intent / brand\-voice / personalization\)\. It has*no*bucket for state\-tracking, guardrails, or recovery—the dimensions that hold most confirmed defects—so confirm\-gate violations get filed as “brand voice” or slip through\.

We position this against, and distinguish it from, the two closest lines of work—single\-turn error\-detection studies that already report low judge recall\(Kamoi et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib7)\), and the one benchmark that judges the*judges of agents*\(Lù et al\.,[2025](https://arxiv.org/html/2606.10315#bib.bib10)\)—in §[2](https://arxiv.org/html/2606.10315#S2)\.

## 2Related Work

#### LLM\-as\-judge: bias and agreement\.

The canonical formulation\(Zheng et al\.,[2023](https://arxiv.org/html/2606.10315#bib.bib17)\)and CoT variants\(Liu et al\.,[2023](https://arxiv.org/html/2606.10315#bib.bib9)\)report reliability as correlation/agreement with human preference\. A rich follow\-up quantifies systematic*biases*\(Ye et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib16); Shi et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib13); Wu et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib14)\)and revisits agreement with stricter statistics\(Han et al\.,[2025](https://arxiv.org/html/2606.10315#bib.bib5); Gu et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib3)\)\. This line answers “does the judge rank like a human?”—not “what does the judge miss?”\.

#### Judge recall / false negatives\.

Closest in spirit,Kamoi et al\. \([2024](https://arxiv.org/html/2606.10315#bib.bib7)\)show GPT\-4/Claude detect real errors in LLM responses at*very low recall*, far below humans, and that self\-consistency does not help;Jain et al\. \([2025](https://arxiv.org/html/2606.10315#bib.bib6)\)report true\-negative rates below25%25\\%and note class imbalance inflates accuracy\. Both are*single\-turn, objective\-error*settings\. Our contribution is the*multi\-turn, production transaction\-agent*instantiation and, crucially, a taxonomy explaining*which*defect classes are missed\.

#### Judging agent trajectories\.

Lù et al\. \([2025](https://arxiv.org/html/2606.10315#bib.bib10)\)is the genuine near\-twin: the first benchmark assessing LLM judges of agent trajectories \(expert\-reviewed web\-agent runs\), finding no judge excels everywhere\. It covers single\-trajectory web agents and does not report a production false\-negative\-rate\-versus\-human\-review headline, nor a turn\-local\-versus\-cross\-turn\-state taxonomy—the two gaps we fill\.

#### Agent evaluation by outcome\.

Mainstream agent benchmarks\(Yao et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib15); Zhou et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib18); Liu et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib8); Ma et al\.,[2024](https://arxiv.org/html/2606.10315#bib.bib11); Guan et al\.,[2025](https://arxiv.org/html/2606.10315#bib.bib4)\)score end\-to\-end task success\. NotablyYao et al\. \([2024](https://arxiv.org/html/2606.10315#bib.bib15)\), the closest to our transaction setting, uses database\-state goal\-matching rather than an LLM judge—an outcome oracle that, like our findings, sidesteps judge recall but cannot diagnose*where*behaviour broke\.

#### Correcting noisy judges\.

Using a calibrated noisy classifier to recover true prevalence is the Rogan–Gladen estimator\(Rogan and Gladen,[1978](https://arxiv.org/html/2606.10315#bib.bib12)\); its application to LLM judges is very recent\(Chen et al\.,[2026](https://arxiv.org/html/2606.10315#bib.bib1); Feng et al\.,[2026](https://arxiv.org/html/2606.10315#bib.bib2)\)\. We adopt this framing \(§[5\.6](https://arxiv.org/html/2606.10315#S5.SS6)\) to argue that a measured judge false\-negative rate should*correct*, not merely accompany, reported quality\.

## 3Setting: A Production Multi\-Turn Ordering Agent

The system under study is a per\-tenant brand agent that takes food\-and\-drink orders in natural language over multi\-turn conversations: it searches a menu, builds and mutates a cart, answers product questions, applies dietary constraints, and places orders behind an explicit confirmation gate\. The evaluation tenant is a Singapore coffee brand with a bilingual \(English / Mandarin, with code\-switching\) customer base\.

#### Corpus\.

We use a randomized multi\-turn corpus generated by a loop driver that samples a scenario*family*\(e\.g\.the\_usual,deep\_dietary,large\_group,cancel,slot\_conflict\), a locale \(en\-SG/zh\-SG\), an identity mode \(anonymous / verified\-with\-favourites\), and a depth of 5–9 user turns\. The batches analysed below span 34 scenario families, balanced 50/50 across locales, with conversation depths concentrated at T5–T9 \(median latency7373s, p95192192s per round\)\.

#### The built\-in judge\.

During each loop round an LLM judge scores the transcript against a rubric that emits three*capability*axes—intent,brand\_voice,personalization—each a pass/fail bit with a free\-text reason, plus an optional free\-textsuspected\_bugnote\. These axis scores feed an auto\-export*regression ledger*: a case is recorded whenever an axis is scored0and, once it recurs, auto\-promoted to a fixture\. Separately, the loop emits the single*operational gate*verdict,live\_passed—the output that actually ships and blocks\. The two are not merely reported separately; they are*architecturally disconnected*\. We verified in the loop driver thatlive\_passedis*not*a function of the three axes: it flips to fail only on an operational failure \(the agent raises an exception or returns no response\) or a hard per\-turn assertion—never on a low quality score, however severe\. So the quality rubric and the gate that ships have no wiring between them, and we measure them separately because they diverge sharply \(§[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)\)\. “Judge recall” in our headline refers to the operational gate—the output that ships—unless stated otherwise\.

#### Ground truth\.

Our ground truth is exhaustive human transcript review: an analyst \(with, in the largest batch, an adversarial multi\-reviewer workflow—one reviewer per family followed by a skeptic verifier prompted to*refute*each finding before synthesis\) reads full turns plus tool calls and records a*pattern*only when a behaviour recurs across multiple rounds, families, or locales\. A pattern is the symptom of a*layer*, not a one\-off\. This yields a conservative denominator: single occurrences are excluded\.

## 4Method

We compare, per batch, the set of human\-confirmed problems against the set the judge flagged\. We report \(i\)*pattern recall*: of the systematic patterns human review confirmed, how many the judge’s pipeline captured; \(ii\) the*distribution of confirmed problems over behavioural dimensions*versus the judge’s available rubric axes; and \(iii\) a*blind\-spot taxonomy*keyed on whether a defect is decidable from a single turn or requires the conversation arc\. All numbers below are extracted from the project’s pattern log and auto\-captured regression ledger; the exact batch identifiers are retained for reproducibility\.

## 5Results

### 5\.1Recall is well under a quarter

#### Batch B1 \(R166–R196, deep\-flow\)\.

Exhaustive human review of the R166–R196 deep\-flow batch surfaced 9 systematic patterns\. The operational gate auto\-captured 2 of them—a22%22\\%pattern recall\(Wilson95%95\\%CI\[6\.3%,54\.7%\]\[6\.3\\%,54\.7\\%\]; the interval is wide atn=9n\{=\}9\)\. The point of B1 is not a precise rate but that the upper end of even a generous interval sits near half\. The≈18%\\approx\\\!18\\%figure we quote elsewhere as a*running*estimate is the program\-level pooled ratio—auto\-captured patterns as a fraction of human\-confirmed patterns, pooled across loop batches rather than computed on B1 alone—and it sits just below the B1 point because later batches confirmed more patterns than the gate captured; we use it only as an order\-of\-magnitude anchor, never as a precise rate\.

#### Batch B2 \(R166–R265, 100 deep rounds\)\.

On the closed 100\-round batch the operational gate’s failure rate \(live\_passed\) was0%0\\%: it surfaced no round as failing\. Human review of the same rounds confirmed23 distinct defectsin the ledger and promoted7 new patternsbeyond B1\. With zero of 23 confirmed defects surfaced, the gate’s recall on this batch is0%0\\%with a rule\-of\-three95%95\\%upper bound of only13%13\\%\(Wilson\[0%,14\.3%\]\[0\\%,14\.3\\%\]\); at the round level,0/1000/100rounds flagged gives an upper bound of3\.7%3\.7\\%\. On this batch the operational gate’s recall for the multi\-turn defect classes involved is statistically indistinguishable from zero\. Crucially, this0%0\\%is*not*the raw judge falling silent—§[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)shows the raw judge frequently*notices*these defects in free text but the rubric vocabulary and a gate disconnected from it prevent the notice from becoming a surfaced failure\.

#### Batch B3 \(R1–R160, post\-fix, fresh pool\)\.

An adversarial human\-in\-the\-loop review \(one reviewer per family→\\rightarrowrefute\-first skeptic verifier→\\rightarrowsynthesis\) raised147 suspected issues, of which85 were adversarially confirmed\(22 explicitly refuted as reviewer/judge artefacts\), skewed to high severity—money, state, and guardrail P0/P1 defects dominate the confirmed set\. This batch established 14 further patterns \(P17–P30\), several of which were*regressions*of B1/B2 fixes—defects the automated gate again did not surface as failures\.

Across batches \(Table[1](https://arxiv.org/html/2606.10315#S5.T1)\), the consistent picture is an operational gate that surfaces a small minority of the systematic, multi\-turn problems a human finds\. We deliberately do*not*collapse this to a single clean number: the three batches enumerate different units \(systematic patterns, ledger\-confirmed defects, rounds\), so22%22\\%\(B1\),≈0%\\approx 0\\%\(B2\), and the≈18%\\approx 18\\%pooled running estimate are not the same quantity—they bound the same*conclusion*, with every interval’s upper edge well under half\. The one unit common to a batch is the round \(B2:0/1000/100\); a fully turn\-level recall comparable across batches needs per\-turn human labels and is part of the calibration pass of §[5\.6](https://arxiv.org/html/2606.10315#S5.SS6)\.

Table 1:Built\-in judge recall against exhaustive human review, per batch \(Wilson intervals, in %\)\. Denominators differ by what each batch’s human review enumerated \(systematic patterns / ledger\-confirmed defects / rounds\); all bound the same conclusion—recall well under a quarter, upper edge under half even atn=9n\{=\}9\.

### 5\.2The defect distribution the judge cannot see

Table[2](https://arxiv.org/html/2606.10315#S5.T2)and Figure[1](https://arxiv.org/html/2606.10315#S5.F1)map the 30 confirmed patterns \(P1–P30\) onto five behavioural dimensions and contrast them with the three axes the judge’s rubric can emit\. The mismatch is the mechanism:15 of 30 confirmed patterns \(50%50\\%\)—every state\-tracking, guardrail, recovery, and safety pattern \(8/3/2/2\)—fall into dimensions the judge’s rubric has no category for, a clean50/5050/50split against the brand\-voice and knowledge patterns it can at least partially name\. Inspection of the auto\-captured ledger confirms the leakage concretely: confirm\-flow and order\-completion failures \(guardrail/state defects—the agent mishandling an explicit “confirm”\) appear in the ledger taggedbrand\_voice\(e\.g\. thestacked\_mutation\_confirmandorderfamilies\)\.

Table 2:Confirmed systematic patterns by behavioural dimension versus the judge’s available rubric axes\. The judge can emit onlyintent/brand\_voice/personalization; it has no native category for state\-tracking, guardrails, recovery, or safety—which together hold50%50\\%of confirmed patterns\. Patterns carrying more than one dimension tag are counted under their primary dimension\.![Refer to caption](https://arxiv.org/html/2606.10315v1/x1.png)Figure 1:The coverage gap, visualized\. Each bar is a behavioural dimension’s share of the 30 confirmed patterns, colored by whether the judge’s three\-axis rubric can name it: green = a dedicated axis \(brand\_voice\); amber = partial \(folded intointent\); red =*no*axis\. Every red dimension—state\-tracking, guardrails, recovery, safety,15/3015/30patterns—is structurally invisible to the judge’s scoring vocabulary\.The judge’s own captured ledger \(74 family×\\timescapability entries\) is distributedbrand\_voice43,intent28,personalization3—i\.e\. the judge funnels the multi\-turn failures it*does*notice into “brand voice”, the one rubric axis loose enough to absorb them, which both mislabels the defect and prevents per\-dimension triage\.

### 5\.3What the judge notices but cannot surface

The0%0\\%operational recall \(§[5\.1](https://arxiv.org/html/2606.10315#S5.SS1)\) could mean two very different things: the raw judge is*blind*\(never registers the defect\), or the raw judge*registers but cannot route*it—it notices the problem, has no rubric axis to file it under, and the shipping gate never consumes the notice at all\. These have opposite design implications, so we separate them directly using the judge’s raw per\-axis scores and free\-textsuspected\_bugnotes, logged alongside the operational verdict\. Over the current 220\-round corpus the split is unambiguous:

- •Theoperational gate\(live\_passed\) reports0/2200/220failures—consistent with the per\-batch recall above\.
- •Theraw judge is far from silent: it emits a free\-textsuspected\_bugon125/220125/220rounds \(57%57\\%\)\.
- •Of those notes,114114describe a confirm\-gate or cart\-state defect—precisely the cross\-turn classes—and113113of114114\(99%99\\%\) are scored underbrand\_voice, the only loose axis available\.
- •None of the125125notices became an operational failure\(0/2200/220live\_passedfails\)—and not because a disposition step weighed and dropped them, but becauselive\_passedis not wired to the quality axes at all \(it fails only on a hang or a hard assertion, §[3](https://arxiv.org/html/2606.10315#S3)\)\.
- •This is structural, not a one\-run artifact\.On an independent earlier corpus \(381381rounds\) the gate*did*fail—121121times—but every one was an operational hang \(the agent returned no response\), not a quality catch; and of the144144rounds there where the agent*responded*and the judge*emitted*a quality note, all144144\(100%100\\%\) still passed the gate\. Pooling both corpora,269269responded\-with\-a\-quality\-note rounds producedzerogate failures\. The gate’s blindness to quality is a property of the wiring, not of any particular run\.

Table[3](https://arxiv.org/html/2606.10315#S5.T3)breaks the125125notices down by what the note*describes*\(a transparent keyword heuristic over the judge’s own free text\) against what the rubric*scored*and whether the gate escalated\. Two rows are the mechanism\. Notes that genuinely describe a tone/voice problem*should*scorebrand\_voice, and do—that is correct routing\. But the confirm\-gate/guardrail and cart/state notes, which describe behavioural defects with no matching axis, also collapse ontobrand\_voice\(113/114113/114\); and the final column is uniformly zero—not one of the125125notices, of any theme, survives to become a gate failure \(the gate never reads them\)\. The misrouting and the dissolution are visible in the same table\.

Table 3:Raw\-judgesuspected\_bugnotes \(125/220125/220rounds\), classified by theme, against the rubric axis scored and the operational outcome\. Themes are assigned by a keyword heuristic over the free\-text note \(the script and the per\-round audit list are released\); the load\-bearing rows are robust to classifier slop because the behavioural\-vs\-brand\_voiceshare is near\-saturated \(113/114113/114\) and the final column is identically zero regardless of theme\. Reproduced on the current 220\-round corpus byeval/experiments/p1\_judge\_routing\.py\.So the blind spot is not \(only\) cognitive\. The raw judge routinely*detects*the confirm\-gate and state defects; what fails is*recording and acting*: the rubric has no state/guardrail axis, so a detected guardrail violation is written as “brand voice” \(113/114113/114\), and the shipping gate—which never reads the axes—cannot act on it regardless\. This is the mechanism of §[5\.1](https://arxiv.org/html/2606.10315#S5.SS1)pinned to data—a rubric\-routing failure compounded by a gate disconnected from the quality signal, not merely a perception failure—and it sharpens the fix \(§[7](https://arxiv.org/html/2606.10315#S7)\): giving the rubric the missing axes is not cosmetic relabeling but the prerequisite for the notice the judge*already produces*to reach the gate at all\.

### 5\.4Blind\-spot taxonomy: turn\-local vs\. cross\-turn

Table[4](https://arxiv.org/html/2606.10315#S5.T4)arranges representative confirmed patterns by how much conversational context is needed to decide the defect\. Judge recall falls as the required context widens from a single turn to the full session arc\. Turn\-local, surface\-checkable defects \(a tilde\-prefixed fabricated statistic; a Vietnamese reply in an English session\) are caught or are at least matchable by a regex; cross\-turn\-state defects \(confirm\-gate lockout, cart hallucination, escalation lockout, stale last\-proposal\) require holding and comparing state across turns—exactly what a per\-turn\-scored judge does not do\.

Table 4:Blind\-spot taxonomy\. Recall degrades as the context required to decide the defect widens\.†Price defects are nominally single\-turn but require reconciling the quoted number against tool\-returned cart state, which the judge does not do\.
### 5\.5Three case studies the judge passed

The taxonomy is concrete in the data\. Below are three real scenarios \(verbatim user turns from the corpus; the agent’s defect is the documented pattern; the judge’s verdict is from the live run\) where the judge emitted no failure—and, where it tagged anything at all, taggedbrand\_voice, a dimension that does not describe the defect\.

Case 1 — confirm\-gate, taggedbrand\_voice\(P1/P17\)\.User:“what’s my cart and total?∣\\midanything change? if it’s all good, confirm the order\.”The defect class is a guardrail/state contract: either an order is placed without an unambiguous confirm, or \(post\-fix\) an explicit confirm fails to place it\. Deciding it requires comparing the confirm\-utterance against the prior proposal and the resulting tool calls—two turns of state\. The judge logged this family underbrand\_voice\.

Case 2 — mid\-session allergen \+ confirm, taggedbrand\_voice\(P15/safety\)\.User:“what’s in my cart?∣\\midactually I just realised I’m allergic to dairy — go ahead and confirm anyway\.”The safe behaviour requires holding the newly\-declared allergen against the dairy items already in cart before honouring the confirm—a cross\-turn safety\+guardrail decision\. A per\-turn judge scoring the final reply for tone has no allergen or guardrail axis; this landed underbrand\_voice\.

Case 3 — stale last\-proposal across an interruption \(P5/P25\)\.User:“what’s the total for my flat white?∣\\midhow much caffeine is in a flat white?∣\\midand is it stronger than a latte?∣\\midok cool, just confirm my order\.”“Confirm my order” refers to a proposal three turns back, across two knowledge questions\. Resolving the referent—and noticing when the agent silently re\-prices or loses it—needs the whole session arc\. The judge, scoring the last turn, cannot see the broken referent\.

In all three the evidence of the defect is a*relation between turns*, not a property of the final response; and in all three the judge’s only available non\-null label isbrand\_voice\. This is the mechanism of Section[5\.1](https://arxiv.org/html/2606.10315#S5.SS1)made concrete: wrong unit of judgement, wrong vocabulary\.

### 5\.6Prevalence correction, and where it breaks

A gate with a false\-negative rate this high does not just under\-count; it*biases*any quality number computed from it\. The standard tool for de\-biasing a noisy binary classifier is the Rogan–Gladen estimator\(Rogan and Gladen,[1978](https://arxiv.org/html/2606.10315#bib.bib12)\): given sensitivityses\_\{e\}\(true\-positive rate\) and specificitysps\_\{p\}\(true\-negative rate\), it recovers true prevalenceπ\\pifrom the apparent \(gate\-reported\) prevalencepappp\_\{\\text\{app\}\}:

π^=papp\+sp−1se\+sp−1\.\\hat\{\\pi\}\\;=\\;\\frac\{p\_\{\\text\{app\}\}\+s\_\{p\}\-1\}\{s\_\{e\}\+s\_\{p\}\-1\}\.
#### Our headline result is the estimator’s degenerate point—and that is the stronger statement\.

On B2 the apparent prevalence is*itself*zero \(papp=0p\_\{\\text\{app\}\}=0\)\. Withsp≤1s\_\{p\}\\\!\\leq\\\!1the numerator is≤0\\leq 0, soπ^≤0\\hat\{\\pi\}\\leq 0: the correction recovers no defects even though human review confirmed 23\. This is not a flaw in our analysis but a structural fact about apparent\-prevalence correction—when the gate emits no positive signal,*no*reweighting of that signal can recover the truth\.The right conclusion is therefore sharper than “the headline number is some multiple too low”: for a silent gate the multiple is*undefined*, because there is no apparent signal to scale\. A consumer handed a0%0\\%judge rate cannot bound the true rate*at all*without independent \(human\) data—a stronger and cleaner claim than any finite33–6×6\\times\.

#### Where the gate is non\-silent, the correction is large but only order\-of\-magnitude\.

When a deployment’s gate*does*report a nonzeropappp\_\{\\text\{app\}\}, the same estimator quantifies the undercount\. Table[5](https://arxiv.org/html/2606.10315#S5.T5)sweeps it as an*illustrative*sensitivity analysis: at the optimistic limitsp=1s\_\{p\}\\\!=\\\!1,π^=papp/se\\hat\{\\pi\}=p\_\{\\text\{app\}\}/s\_\{e\}, so a gate reporting10%10\\%withse≈0\.2s\_\{e\}\\\!\\approx\\\!0\.2implies≈50%\\approx\\\!50\\%true prevalence—a33–6×6\\timesundercount across the assumedsps\_\{p\}range\. Two honesty limits keep this*directional, not calibrated*\. \(i\)sps\_\{p\}is unmeasured; a calibrated point estimate needs a dedicated human\-labeling pass \(N≈200N\\\!\\approx\\\!200turn\-level triples; tooling in\(Chen et al\.,[2026](https://arxiv.org/html/2606.10315#bib.bib1); Feng et al\.,[2026](https://arxiv.org/html/2606.10315#bib.bib2)\)\)\. \(ii\)Granularity mismatch: these≈0\.2s\_\{e\}\\\!\\approx\\\!0\.2we substitute is a*pattern\-level*recall \(§[5\.1](https://arxiv.org/html/2606.10315#S5.SS1)\), whereas Rogan–Gladen requires a*turn\-level*sensitivity at the same grain aspappp\_\{\\text\{app\}\}; mixing grains is why33–6×6\\timesis an order\-of\-magnitude illustration, not a measured coefficient\. Both limits are exactly what the calibration pass resolves\. The robust takeaway needs neithersps\_\{p\}nor a turn\-levelses\_\{e\}:*a near\-zero gate rate is uninformative about true prevalence, and any nonzero rate read off a low\-sensitivity gate is an undercount of unknown but large size\.*

Table 5:*Illustrative*Rogan–Gladen correction for the*non\-silent*regime \(papp\>0p\_\{\\text\{app\}\}\>0\)\. The headline B2 batch haspapp=0p\_\{\\text\{app\}\}=0, where the estimator degenerates and no correction is possible \(§[5\.6](https://arxiv.org/html/2606.10315#S5.SS6)\)\. True prevalenceπ^\\hat\{\\pi\}implied by an apparent ratepapp=10%p\_\{\\text\{app\}\}=10\\%, under apattern\-levelrecall substituted asse≈0\.18s\_\{e\}\\approx 0\.18–0\.220\.22\(§[5\.1](https://arxiv.org/html/2606.10315#S5.SS1)\) and a range ofassumedspecificitiessps\_\{p\}\. A gate reporting one defect in ten implies2929–56%56\\%true prevalence \(33–6×6\\times\)\. Two caveats make this directional, not calibrated:sps\_\{p\}is unmeasured, andses\_\{e\}is pattern\- not turn\-level, so the coefficient is order\-of\-magnitude\. See §[5\.6](https://arxiv.org/html/2606.10315#S5.SS6)\.

## 6Discussion

Why does a capable LLM gate miss so much? Three compounding reasons—and only the first is about the model’s perception\.\(1\) The unit of judgement is wrong\.A judge scored on the final response of a turn cannot see a defect whose evidence is the*difference*between turnttand turnt−3t\{\-\}3\(a phantom cart add, a referent that broke two turns ago, an escalation flag that never reset\)\.\(2\) The rubric cannot name the failure\.A rubric with axes intent / brand\-voice / personalization has no place to record a guardrail or state\-tracking violation, so a failure the judge*does*notice is mislabelled into “brand voice” \(113/114113/114confirm\-gate/state notices in our corpus; §[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)\) rather than triaged\.\(3\) The shipping gate ignores the quality axes\.The operational verdict that ships is wired only to hangs and hard assertions, not to the rubric, so a scored\-low axis or an explicitsuspected\_bugnote cannot flip it—the notice the judge produces never reaches the gate \(0/2200/220rounds failed despite125125raw notes; across two corpora,269269responded\-with\-a\-note rounds yielded zero gate fails\)\. The salient point is that \(2\)–\(3\), not \(1\), drive most of the miss in our data: the model often*sees*the defect; the scoring vocabulary and the gate throw the observation away\. These are not prompt\-tuning problems; they are structural limits of per\-turn, coarse\-rubric, pass\-biased judging\.

The practical consequence: for production multi\-turn agents, automated judging should be treated as a*regression floor*—cheap, always\-on, good at catching turn\-local recurrences of already\-known defects—not as the primary signal for whether the agent is good\. The primary signal remains exhaustive human transcript review, which is what surfaced every cross\-turn P0 in our data\. A corollary for anyone reporting an agent’s quality from an LLM judge: report the judge’s measured recall \(or a Rogan–Gladen\-corrected prevalence\), or the headline number is an unknown multiple too low\.

## 7Toward State\-Aware Judging

Our diagnosis points to two concrete, testable fixes; we state them as hypotheses for the community rather than results\.\(1\) Change the unit of judgement from turn to arc\.A judge given the*state diff*across turns—cart before/after, the open proposal, the escalation flag, the declared\-allergen set—can in principle decide the cross\-turn defects a final\-response judge cannot\. The cheap version is a tool\-augmented judge that calls the samereview\_cart/state accessors the agent does and asserts consistency; the structured version scores a trajectory, not a message\.\(2\) Give the rubric the missing axes—and let them escalate\.Add explicit*state\-tracking*,*guardrail*,*recovery*, and*safety*categories so a noticed failure has somewhere to go other than “brand\_voice”, and wire those axes to flip the operational verdict when one of them \(or asuspected\_bugnote\) fires—the connection that does not exist today\. Our data \(§[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)\) makes this the cheapest high\-yield fix: the judge already emits the right observation on a majority of defective rounds; today it is discarded by a vocabulary with no slot and a gate that never reads it\. This is a prerequisite for, not a substitute for, \(1\): a category with no state to reason over still cannot be scored\. The strongest experiment—left to future work because it needs a labeled slice—is an*ablation*: extend the rubric with the four missing axes, re\-judge a human\-labeled batch, and measure how much of the∼80%\\sim\\\!80\\%gap each fix recovers\. Our prediction is that rubric axes alone recover the*labeling*of guardrail failures but little of the*recall*of cross\-turn ones, which need the arc\-level unit; the experiment would falsify or confirm it\.

## 8Threats to Validity

We group threats along the four standard empirical classes\.Construct validity\.Is “pattern recall against human review” a fair measure of judge quality? Human review is itself imperfect, so it is an upper\-bound proxy, not an oracle; but the asymmetry we report \(judge≪\\llhuman\) is robust to that—a noisier human ground truth would only*narrow*the measured gap, and the gap is large\. The per\-dimension mechanism \(Table[2](https://arxiv.org/html/2606.10315#S5.T2)\) is a structural fact about the rubric, independent of labeling noise\.Internal validity\.Could the judge be “catching” defects we credit to humans? The ledger is the judge’s*own*auto\-captured output, so its catches are counted; B2’s0/230/23is measured on the judge’s actual flags\. Ground\-truth subjectivity is mitigated by a multi\-occurrence promotion rule and, in B3, an adversarial refute\-first verifier, which also yields a measured reliability control: the independent verifier confirmed85/14785/147\(58%58\\%\) of proposer\-flagged candidates and explicitly refuted2222, bounding the proposer’s false\-discovery contribution\. This is a two\-stage*adversarial*agreement; we additionally ran the parallel*blind*double\-annotation it lacked\. Two annotators independently labeled an8080\-round sample of the loop corpus from the transcripts alone—judge scores,suspected\_bug, and the operational verdict all stripped—against a shared codebook\. Inter\-annotator agreement isCohen’sκ=0\.61\\kappa=0\.61on defect presence \(“substantial”; raw agreement90%90\\%,72/8072/80;n=80n\{=\}80; bootstrap95%95\\%CI\[0\.33,0\.83\]\[0\.33,0\.83\]\) andκ=1\.0\\kappa=1\.0on the turn\-local\-vs\-cross\-turn class for the88co\-identified defects \(perfect, butn=8n\{=\}8with77cross\-turn, so it bounds the taxonomy’s boundary\-crispness only where both annotators saw a defect\)\. We report the*pre*\-reconciliationκ\\kappa\. Defects are rare in the sample, so chance agreement is high \(pe=0\.74p\_\{e\}\{=\}0\.74\) andκ\\kappais correspondingly conservative relative to the90%90\\%raw agreement—the standard high\-agreement, skewed\-prevalence regime, which we flag rather than paper over\. Tellingly, the two annotators flagged overlapping but*non\-identical*defect sets \(1515vs\.99\);77of the88disagreements were a genuine defect one annotator caught and the other missed\. Two careful humans each under\-detect—so the single\-analyst ground truth is a conservative*floor*on true defect prevalence, not a ceiling, and labeling noise can only*widen*the judge–human gap we report, never manufacture it\. B3 already carries an adversarial verifier; the marquee B2 \(0/230/23\) rests on a single analyst plus the multi\-occurrence promotion rule, so a re\-annotation of the exact B2 rounds with full disagreement reconciliation is the residual camera\-ready step \(the kit, codebook, andκ\\kappacalculator are released\)\. The same B2 batch anchors a companion methodology paper, so this hardening propagates to two results, not one\. The raw\-vs\-gate split \(§[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)\) is independent of this concern: it is computed mechanically from logged judge fields, with no human labeling in the loop\.External validity\.All data come from one F&B ordering agent on one bilingual tenant, and the corpus is adversarially loop\-sampled rather than organic traffic; exact recall values will differ elsewhere, and post\-pilot replay on live traffic is future work\. The sampling bias has a specific, admitted direction: the loop over\-samples cross\-turn\-heavy families \(deep\_dietary,slot\_conflict,cancel\), i\.e\. it enriches exactly the defect classes the gate is worst at, so on organic traffic the turn\-local share would be higher, the gate would catch relatively more, and the prevalence undercount of §[5\.6](https://arxiv.org/html/2606.10315#S5.SS6)would*shrink*\. We therefore treat the33–6×6\\timesas directional, not a population estimate; the*degenerate*\-case conclusion \(a silent gate is uninformative\) is immune to this bias because it does not depend on the sampled defect mix at all\. We claim the*shape*\(cross\-turn≫\\ggturn\-local blind spot; rubric\-axis mismatch; a shipping gate disconnected from the rubric\) generalizes, not the precise percentages\.Conclusion validity\.Per\-batchnnis small \(B1n=9n\{=\}9\); we therefore report Wilson intervals \(Table[1](https://arxiv.org/html/2606.10315#S5.T1)\) rather than bare points and draw the conclusion only from the bound their upper edges share\. Results are also tied to the specific judge model and rubric in production over this period \(reported as such\); stronger judges—juries, debate, tool\-augmented state\-diffing judges \(§[7](https://arxiv.org/html/2606.10315#S7)\)—may narrow the gap and are the natural next experiment\.

## 9Conclusion

On a deployed multi\-turn transaction agent, the built\-in LLM judge’s operational gate surfaces well under a quarter of the systematic quality problems exhaustive human review finds, and the miss is structured: it loses precisely the cross\-turn state, guardrail, and recovery defects that matter most for a transaction agent\. The cause is mostly not blindness—the raw judge notices these defects on a majority of rounds—but routing: its rubric has no category in which to record them \(113/114113/114filed as “brand voice”\) and its shipping gate is not wired to the rubric, so it never surfaces the ones it notices \(0/2200/220, and zero across two corpora\)\. We offer a blind\-spot taxonomy, a data\-grounded mechanism, and a prevalence\-correction contract whose*degenerate*case—a silent gate licenses no estimate of true prevalence—is the sharpest practical warning\. Until judges reason over conversation state, rubrics can name state/guardrail/recovery failures, and gates escalate them, LLM\-as\-judge belongs under human review, not in place of it\.

## Reproducibility

Pattern logs, the auto\-captured regression ledger, the per\-dimension counts of Table[2](https://arxiv.org/html/2606.10315#S5.T2), the raw per\-axis judge scores andsuspected\_bugnotes underlying the raw\-vs\-gate split of §[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)and Table[3](https://arxiv.org/html/2606.10315#S5.T3)\(125/220125/220,113/114113/114,0/2200/220\), and the judge calibration script are retained in the project repository\. The full raw\-vs\-gate analysis—theme classifier, cross\-tab, and the per\-round audit of every note not scoredbrand\_voice—is a single script,eval/experiments/p1\_judge\_routing\.py, run against one corpus \(eval/loop/runs/loop\_runs\.jsonl\), so every number in §[5\.3](https://arxiv.org/html/2606.10315#S5.SS3)is reproducible from one source without re\-deriving the older per\-batch recall\. Table[5](https://arxiv.org/html/2606.10315#S5.T5)is an*illustrative*sensitivity analysis; turning it into a calibrated point estimate \(a measuredsps\_\{p\}and a turn\-levelses\_\{e\}\) is the documented human\-labeling pass left to camera\-ready\.

## References

- Chen et al\. \(2026\)Yiqun T\. Chen, Sizhu Lu, Sijia Li, Moran Guo, and Shengyi Li\.Efficient inference for noisy LLM\-as\-a\-judge evaluation\.*arXiv preprint arXiv:2601\.05420*, 2026\.
- Feng et al\. \(2026\)Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner\-Beuerle, and Miguel R\. D\. Rodrigues\.Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges\.*arXiv preprint arXiv:2601\.20913*, 2026\.
- Gu et al\. \(2024\)Jiawei Gu et al\.A survey on LLM\-as\-a\-judge\.*arXiv preprint arXiv:2411\.15594*, 2024\.
- Guan et al\. \(2025\)Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian\-guang Lou, and Haoyi Xiong\.Evaluating LLM\-based agents for multi\-turn conversations: A survey\.*arXiv preprint arXiv:2503\.22458*, 2025\.
- Han et al\. \(2025\)Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou\.Judge’s verdict: A comprehensive analysis of LLM judge capability through human agreement\.*arXiv preprint arXiv:2510\.09738*, 2025\.
- Jain et al\. \(2025\)Suryaansh Jain, Umair Z\. Ahmed, Shubham Sahai, and Ben Leong\.Beyond consensus: Mitigating the agreeableness bias in LLM judge evaluations\.*arXiv preprint arXiv:2510\.11822*, 2025\.
- Kamoi et al\. \(2024\)Ryo Kamoi et al\.Evaluating LLMs at detecting errors in LLM responses\.In*Conference on Language Modeling \(COLM\)*, 2024\.arXiv:2404\.03602\.
- Liu et al\. \(2024\)Xiao Liu et al\.AgentBench: Evaluating LLMs as agents\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2308\.03688\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.In*Proceedings of EMNLP*, 2023\.arXiv:2303\.16634\.
- Lù et al\. \(2025\)Xing Han Lù, Amirhossein Kazemnejad, et al\.AgentRewardBench: Evaluating automatic evaluations of web agent trajectories\.*arXiv preprint arXiv:2504\.08942*, 2025\.
- Ma et al\. \(2024\)Chang Ma et al\.AgentBoard: An analytical evaluation board of multi\-turn LLM agents\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2024\.arXiv:2401\.13178\.
- Rogan and Gladen \(1978\)Walter J\. Rogan and Beth Gladen\.Estimating prevalence from the results of a screening test\.*American Journal of Epidemiology*, 107\(1\):71–76, 1978\.
- Shi et al\. \(2024\)Lin Shi et al\.Judging the judges: A systematic study of position bias in LLM\-as\-a\-judge\.*arXiv preprint arXiv:2406\.07791*, 2024\.
- Wu et al\. \(2024\)Koki Wu et al\.Self\-preference bias in LLM\-as\-a\-judge\.*arXiv preprint arXiv:2410\.21819*, 2024\.
- Yao et al\. \(2024\)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.*arXiv preprint arXiv:2406\.12045*, 2024\.
- Ye et al\. \(2024\)Jiayi Ye et al\.Justice or prejudice? quantifying biases in LLM\-as\-a\-judge\.*arXiv preprint arXiv:2410\.02736*, 2024\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2023\.arXiv:2306\.05685\.
- Zhou et al\. \(2024\)Shuyan Zhou et al\.WebArena: A realistic web environment for building autonomous agents\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2307\.13854\.