Stopping and Routing LLM Judge Panels

# Stopping and Routing LLM Judge Panels
Source: [https://arxiv.org/html/2608.19802](https://arxiv.org/html/2608.19802)
Yi XieYanghui RaoThanks:Corresponding author\.Affiliation:School of Computer Science and EngineeringAffiliation:Sun Yat\-sen University, Guangzhou, ChinaAffiliation:zhub35@mail2\.sysu\.edu\.cn xiey299@mail2\.sysu\.edu\.cnEmail:[raoyangh@mail\.sysu\.edu\.cn](mailto:)

###### Abstract

LLM evaluation pipelines often have many candidate judges: general LLM\-as\-a\-judge prompts, reward models, safety classifiers, confidence variants, and task\-specific verifiers\. The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop\. We formulate judge\-panel design as a role\-conditioned allocation problem\. From a small labeled audit set, declared slices, and judge costs, the method estimates target\-relative roles:*copies*add no conditional information,*complements*improve the global panel, and*specialists*help only on slices\. These roles induce a policy: drop copies, add complements globally, route specialists conditionally, and stop when validation gain falls below a threshold\. Across reasoning, code, safety, preference, reward\-model, summarization, and math audits, the method is compared with single judges, flat panels, matched diversity heuristics, full\-call stacking, reliability juries, and frugal cascades\. The result is a regime map for judge calls: route specialists on deployable slices, stop in saturated verifier regimes, keep broad ensembles when their risk benefit is worth the cost, and ignore conditional copies\. The output is a reusable, auditable call plan for the next evaluation batch\.

## 1Introduction

LLM\-as\-a\-judge systems are now common in model evaluation\([45](https://arxiv.org/html/2608.19802#bib.bib20);[30](https://arxiv.org/html/2608.19802#bib.bib13);[46](https://arxiv.org/html/2608.19802#bib.bib21);[22](https://arxiv.org/html/2608.19802#bib.bib11);[40](https://arxiv.org/html/2608.19802#bib.bib17);[23](https://arxiv.org/html/2608.19802#bib.bib29);[15](https://arxiv.org/html/2608.19802#bib.bib24);[6](https://arxiv.org/html/2608.19802#bib.bib23)\)\. A realistic evaluation pipeline may include a general judge, a rubric prompt, a reward model, a safety classifier, a confidence variant, and a deterministic verifier\. For every new evaluation batch, the researcher must make a concrete operating decision: call the whole panel, call a cheap verifier and stop, route a safety judge only to risky cases, or drop a redundant prompt entirely\. A static judge ranking does not answer that question\. The value of a judge is conditional on the current panel, the target distribution, and the slice of examples where it will be used\.

This creates a useful opportunity\. A small labeled audit set can turn judge diversity from a descriptive property into a calling policy\. A safety judge can become a specialist on jailbreak failures; a verifier can make several LLM judges redundant; and a full ensemble can still be the right endpoint on broad math or reward\-model regimes\([43](https://arxiv.org/html/2608.19802#bib.bib18);[5](https://arxiv.org/html/2608.19802#bib.bib22)\)\. The goal is to identify these cases before paying for judge calls on the next batch\.

We turn the taxonomy of*copy*,*complement*, and*specialist*into an allocation method\. The output is a calling policyπ⁡\(x\)⊆𝒥\\pi\(x\)\\subseteq\\mathcal\{J\}with a validation\-based stopping record\. The method asks whether each candidate reduces held\-out calibration risk after conditioning on the current panel, and whether that gain is global or slice\-specific\. Copies are dropped, broad complements are added to the global panel, specialists are routed to their slices, and construction stops when no remaining candidate clears a declared gain threshold\.

#### Contributions\.

We make three claims\. First, judge diversity should be target\-relative and conditional, not nominal\. Second, copy/complement/specialist roles can be converted into a practical policy with costs, slices, and stopping conditions\. Third, the empirical value of the method is a regime map for deployment: it identifies when to route specialists, when to keep a cheap stopped panel, when to stop after a verifier, and when to pay for the full panel\.

Although this paper and the companion*A Finite\-Calibration Regime Map for LLM Judge Panels*share part of the benchmark judge\-output matrices and judge pool, they address distinct deployment decisions: this paper selects conditional calls and stopping, whereas the companion selects a panel prefix and aggregation family after candidate outputs are available\.

## 2Role\-Conditioned Allocation

LetXXbe an evaluated item andY∈\[0,1\]Y\\in\[0,1\]the audit label\. A finite candidate pool𝒥\\mathcal\{J\}contains judge signalsZjZ\_\{j\}such ascorrect/incorrect,safe/unsafe, orA/B\. For a panelS⊆𝒥S\\subseteq\\mathcal\{J\}, letZSZ\_\{S\}be the joint output pattern\. The researcher declares slicesℱ\\mathcal\{F\}that matter for the target distribution, such as LLMBar subsets\([44](https://arxiv.org/html/2608.19802#bib.bib19)\), safety failure modes\([7](https://arxiv.org/html/2608.19802#bib.bib3)\), generator type, or difficulty level\([19](https://arxiv.org/html/2608.19802#bib.bib10)\)\. The goal is a policyπ⁡\(x,𝒥,ℱ\)⊆𝒥\\pi\(x;\\mathcal\{J\},\\mathcal\{F\}\)\\subseteq\\mathcal\{J\}that decides which judges to call onxx\.

Fix a target distributionPP\. For a panelSS, define the oracle predictor

ηP,S​\(z\)=𝔼P​\[Y∣ZS=z\]\\eta\_\{P,S\}\(z\)=\\mathbb\{E\}\_\{P\}\[Y\\mid Z\_\{S\}=z\]and its squared\-loss oracle risk

ℛP,S⋆=𝔼P​\[\(Y−ηP,S​\(ZS\)\)2\]\.\\mathcal\{R\}^\{\\star\}\_\{P,S\}=\\mathbb\{E\}\_\{P\}\[\(Y\-\\eta\_\{P,S\}\(Z\_\{S\}\)\)^\{2\}\]\.The conditional value of adding judgej∉Sj\\notin Sis

gP​\(j∣S\)=ℛP,S⋆−ℛP,S∪\{j\}⋆\.g\_\{P\}\(j\\mid S\)=\\mathcal\{R\}^\{\\star\}\_\{P,S\}\-\\mathcal\{R\}^\{\\star\}\_\{P,S\\cup\\\{j\\\}\}\.This is the target information injjthat is not already present inSS\.

###### Lemma 1\(Projection gain identity\)\.

For any finite panelSSand candidate judgej∉Sj\\notin S,

gP​\(j∣S\)=𝔼P​\[\(ηP,S∪\{j\}​\(ZS∪\{j\}\)−ηP,S​\(ZS\)\)2\]≥0\.g\_\{P\}\(j\\mid S\)=\\mathbb\{E\}\_\{P\}\\left\[\\left\(\\eta\_\{P,S\\cup\\\{j\\\}\}\(Z\_\{S\\cup\\\{j\\\}\}\)\-\\eta\_\{P,S\}\(Z\_\{S\}\)\\right\)^\{2\}\\right\]\\geq 0\.The identity requires no independence assumption among judges\.

For slicesf∈ℱf\\in\\mathcal\{F\}, define broad gainCP​\(j∣S\)=gP​\(j∣S\)C\_\{P\}\(j\\mid S\)=g\_\{P\}\(j\\mid S\)and slice gainAf​\(j∣S\)=gPf​\(j∣S\)A\_\{f\}\(j\\mid S\)=g\_\{P\_\{f\}\}\(j\\mid S\)\. The role profileprofileP,ℱ,S⁡\(j\)=\(CP,\{Af\}f∈ℱ\)\\operatorname\{profile\}\_\{P,\\mathcal\{F\},S\}\(j\)=\(C\_\{P\},\\\{A\_\{f\}\\\}\_\{f\\in\\mathcal\{F\}\}\)is multi\-label: a judge may be both a broad complement and a slice specialist, and its role can change after another judge enters the panel\.

The profile is deliberately an action interface rather than a naming scheme\. For example, a reward model that is weak as a standalone preference judge can still be a complement after a rubric prompt enters the panel if it separates cases the prompt collapses\. Conversely, a second prompt from the same model family can become a copy if its conditional gain vanishes after the first prompt\. Slice roles are evaluated in the same target\-relative way\. We also track a diagnostic specialization ratio

ρf​\(j∣S\)=gPf​\(j∣S\)gP​\(j∣S\)\+ϵ0,\\rho\_\{f\}\(j\\mid S\)=\\frac\{g\_\{P\_\{f\}\}\(j\\mid S\)\}\{g\_\{P\}\(j\\mid S\)\+\\epsilon\_\{0\}\},withϵ0\>0\\epsilon\_\{0\}\>0only to avoid division by zero\. The ratio does not make roles mutually exclusive; it flags concentration of value\. A judge may be a broad complement and still be especially worth inspecting on one declared slice\.

![Refer to caption](https://arxiv.org/html/2608.19802v1/copy_complement_specialist_v15.png)Figure 1:Action\-oriented role taxonomy\. A copy is redundant after conditioning on the current panel, a complement adds broad residual information, and a specialist adds value mainly on a deployable slice\.Table 1:Role taxonomy as a policy interface\. Roles are target\-relative, conditional on the current panel, and may overlap\.#### Construction rule\.

We split each audit set into construction\-fit, construction\-validation, and final\-test parts\. Pattern calibrators estimateηP,S\\eta\_\{P,S\}by cell means on canonicalized joint judge\-output patterns on the fit split; unseen validation or test patterns fall back to the fit\-split label mean\. Selection uses validation gain only; reported results use the final\-test split only\. Given current global panelSS, costscjc\_\{j\}, and thresholdτP\\tau\_\{P\}, add a global judge only if

maxj∈J∖S⁡\[g^Pval​\(j∣S\)−λ​cj\]\>τP\.\\max\_\{j\\in J\\setminus S\}\\left\[\\widehat\{g\}^\{\\mathrm\{val\}\}\_\{P\}\(j\\mid S\)\-\\lambda c\_\{j\}\\right\]\>\\tau\_\{P\}\.For each slice, add a routed specialist only if

maxj∈J∖\(S∪Sf\)⁡\[g^Pfval​\(j∣S∪Sf\)−λf​cj\]\>τf\.\\max\_\{j\\in J\\setminus\(S\\cup S\_\{f\}\)\}\\left\[\\widehat\{g\}^\{\\mathrm\{val\}\}\_\{P\_\{f\}\}\(j\\mid S\\cup S\_\{f\}\)\-\\lambda\_\{f\}c\_\{j\}\\right\]\>\\tau\_\{f\}\.The deployed policy invokesπ⁡\(x\)=S∪Sf⁡\(x\)\\pi\(x\)=S\\cup S\_\{f\(x\)\}for examples in slicef⁡\(x\)f\(x\), andSSotherwise\. The slice function used in deployment must be computable before the routed judge call\. Ground\-truth labels may define audit strata for analysis, but they are not valid inputs toπ⁡\(x\)\\pi\(x\)on a new example; deployable routes must use metadata, verifier outputs, classifier outputs, or already\-observed judge disagreement\. If no remaining candidate clears threshold, the policy stops and records a validation\-based stopping report that every unused broad or slice gain is below the declared threshold\. This report is operational rather than asymptotic: it says that, under the audit split and cost model, the panel is usable without further judge calls\.

Algorithmically, global construction is a greedy validation procedure\. Starting from an empty or user\-seeded panel, we fit the current pattern calibrator, score each unused candidate by cost\-adjusted validation gain, add the best candidate only if it clearsτP\\tau\_\{P\}, and repeat\. After the global panel stops, each slice runs the same greedy search with the selected global panel fixed\. The final policy is then refit on the full construction split and evaluated once on held\-out final\-test examples\. The stopping report is the collection of failed inequalities for unused broad and routed candidates\. It records the decision actually made by the deployment policy: under the finite audit split, declared slices, thresholds, and costs, no remaining single judge call is worth adding to the current plan\. If a deployment owner wants to search for pairwise or higher\-order complementarity, the same validation\-gain objective can be run with beam or subset proposals; the stopping report then documents that expanded search space\.

This finite\-split design makes the policy usable as a deployment audit\. Judge selection happens on construction data, final\-test examples are held out for reporting, and the comparisons include both cheap baselines and full\-call aggregation endpoints\. The intended use is simple: before paying for future judge calls, use a labeled audit set to decide whether a candidate adds information conditional on the panel that will actually be invoked\.

## 3Experimental Protocol

We evaluate non\-saturated settings where a single judge is not already perfect: hard GSM8K rationale audits\([12](https://arxiv.org/html/2608.19802#bib.bib7)\), MBPP public\-test overfit audits\([2](https://arxiv.org/html/2608.19802#bib.bib1)\), JailbreakBench safety\([7](https://arxiv.org/html/2608.19802#bib.bib3)\), LLMBar preference under DeepSeek, Qwen3, and JudgeLM anchors\([44](https://arxiv.org/html/2608.19802#bib.bib19);[14](https://arxiv.org/html/2608.19802#bib.bib25);[37](https://arxiv.org/html/2608.19802#bib.bib33);[46](https://arxiv.org/html/2608.19802#bib.bib21)\), RewardBench\([27](https://arxiv.org/html/2608.19802#bib.bib12)\), Arena100K\([11](https://arxiv.org/html/2608.19802#bib.bib6)\), SummEval\([16](https://arxiv.org/html/2608.19802#bib.bib9)\), and MATH\-500\([19](https://arxiv.org/html/2608.19802#bib.bib10);[29](https://arxiv.org/html/2608.19802#bib.bib35)\)\. HumanEval and ordinary GSM8K are used only as saturated stopping checks\([9](https://arxiv.org/html/2608.19802#bib.bib4);[12](https://arxiv.org/html/2608.19802#bib.bib7)\)\. The concrete pool uses Qwen2\.5 Instruct 7B, Llama 3\.1 Instruct 8B, Mistral v0\.3 7B, Prometheus 2 v2\.0 7B, Gemma 3 IT 12B, Atla Selene Mini \(Llama 3\.1, 8B\), and the DeepSeek V4 Flash API model \(284B total parameters, 13B active parameters\), with task\-specific subsets where noted\. LLM judge calls have normalized cost1\.01\.0, and deterministic verifiers have cost0\.10\.1\. Route keys are treated as pre\-available metadata, verifier outputs, classifier outputs, or already\-observed judge signals; an additional model call needed to obtain a route key must be added to the cost model\.

Baselines cover the options a practitioner would plausibly deploy: single best validation judge; flat all\-judge panels; matched\-KKtop\-kk, correlation diversity, and quality\-diversity panels; full\-call ridge/logistic stacking\([43](https://arxiv.org/html/2608.19802#bib.bib18);[5](https://arxiv.org/html/2608.19802#bib.bib22)\); Dawid–Skene\-style reliability juries\([13](https://arxiv.org/html/2608.19802#bib.bib8);[3](https://arxiv.org/html/2608.19802#bib.bib2);[38](https://arxiv.org/html/2608.19802#bib.bib32);[42](https://arxiv.org/html/2608.19802#bib.bib40)\); and FrugalGPT/RouteLLM\-style confidence cascades\([8](https://arxiv.org/html/2608.19802#bib.bib5);[34](https://arxiv.org/html/2608.19802#bib.bib16)\)\. All results are averaged over 10 random splits; Appendix[E](https://arxiv.org/html/2608.19802#A5)reports split\-level standard deviations and 95% confidence intervals for the main risk comparisons\. Unless noted,τP=τf=0\.005\\tau\_\{P\}=\\tau\_\{f\}=0\.005\. Appendix[C](https://arxiv.org/html/2608.19802#A3)lists the judge pools, route keys, and cheap verifiers used in each setting\.

Table 2:Experimental matrix\. Each setting is included because it exercises a different deployment decision: add complements, route specialists, stop early, drop copies, or accept a full\-call boundary\. Human labels may define audit slices for analysis, but only metadata, verifier outputs, classifier outputs, or judge\-disagreement proxies are deployable route signals\.The evaluation metric is held\-out squared calibration risk for all tasks and accuracy where labels are binary\. Risk is the primary metric because the method selects judges through calibrated conditional gain; accuracy is reported to make the results legible for standard correctness, safety, and preference audits\([18](https://arxiv.org/html/2608.19802#bib.bib28)\)\. Average cost and average number of judge calls are reported because the paper’s object is a deployment policy rather than an unconstrained aggregator\.

The datasets are chosen to prevent a single story from explaining every result\. Hard GSM8K rationale and MBPP public\-overfit test whether a cheap verifier and a few LLM judges can be combined without defaulting to all calls\. LLMBar tests deployable conditional routing on natural and adversarial subsets, while JBB tests whether safety value is concentrated on deployable classifier or disagreement proxies and on human\-label audit strata\. Human labels are used only for audit evaluation, not for deployment\-time routing\. Arena100K and SummEval test stopping in non\-saturated settings where extra judges can worsen calibration\. RewardBench and MATH\-500 are boundary cases where broad aggregation can remain attractive\. HumanEval and ordinary GSM8K are saturated sanity checks: after an objective verifier solves the audit target, the correct policy action is to stop\.

The baselines are similarly separated by deployment question\. The flat panel and full\-call stacking baselines answer “what if we call every judge?” and therefore form strong risk endpoints at high cost\. Matched\-size non\-role panels answer whether ordinary quality or correlation diversity can match the same call budget without role conditioning\. Reliability jury answers whether global judge trustworthiness is enough\. Frugal cascade answers whether a single quality order with an uncertainty trigger is enough\. Role allocation should win only when the missing ingredient is conditional value relative to the current panel or slice\.

## 4Results

The evidence chain follows the deployment actions induced by the role profile\. Table[3](https://arxiv.org/html/2608.19802#S4.T3)asks which call plan each setting supports\. Tables[4](https://arxiv.org/html/2608.19802#S4.T4)and[6](https://arxiv.org/html/2608.19802#S4.T6)then separate the regimes: where conditional specialists should be routed, where a cheap stopped panel is enough, where copied signals should be dropped, and where the right endpoint is still a broad full\-call ensemble\.

Table 3:Main held\-out policy comparison across hard reasoning audits, code overfit audits, safety, pairwise preference, reward modeling, and scalar summarization\. Role policies expose few\-judge complement panels, one\-step stopping, specialist routing, and broad\-ensemble endpoints\.Table[3](https://arxiv.org/html/2608.19802#S4.T3)translates held\-out metrics into deployment decisions\. On hard GSM8K rationales, MBPP overfit, safety audit/proxy slices, and LLMBar, role policies recover useful accuracy with fewer than a flat panel’s calls\. Arena100K and SummEval produce a different action: keep the strong single judge because expansion adds little value\. RewardBench and MATH\-500 expose the full\-panel endpoint, where extra broad signals can be worth their cost when the researcher wants the lowest risk\.

The complement regimes show why conditioning matters\. In hard GSM8K rationale audits, ordinary answer checking is not the target: the policy must decide whether the reasoning is valid\. The stopped role policy improves accuracy over both the single\-best and flat\-all panels while invoking about three judges\. In MBPP public\-overfit, the hidden\-test verifier is cheap and strong, but it does not eliminate all residual audit uncertainty\. The role policy reaches the best reported accuracy at a cost close to one and a half calls, illustrating the intended combination of verifier\-first stopping with selective LLM additions\.

Table 4:Strong baseline comparison\. Full\-call aggregation can be the best risk endpoint, but it invokes every judge\. Role policies solve the deployment problem of deciding when to buy a small stopped panel, when to route specialists, and when to keep the full\-call endpoint\.Full\-call stacking is a strong endpoint because it sees all judge outputs before predicting\. Role allocation answers the preceding operational question: which outputs should be purchased in the first place? Table[4](https://arxiv.org/html/2608.19802#S4.T4)shows that role policies add the most value when information is conditional on the current panel or route signal, as in safety and LLMBar\. On hard GSM8K, RewardBench, and MATH\-500, matched panels or full\-call endpoints can be just as competitive\. That is the intended regime\-map reading: the policy tells the researcher whether to buy conditional specialists, stop early, or pay for broad aggregation\. The JBB proxy\-routing audit in Table[10](https://arxiv.org/html/2608.19802#A2.T10)isolates the deployable safety case: routing ongpt4\_cf, not human labels, reaches 0\.1094 risk at 2\.29 calls versus the 7\-call stack at 0\.1069\.

The matched\-KKcomparison is the key judge\-count fairness check\. A top\-kkpanel can reuse the same number of judges, but its realized call cost can differ; it selects judges by standalone validation quality rather than conditional value\. Correlation\-diverse and quality\-diverse panels also spend a similar budget, but their notion of diversity is nominal or pairwise rather than target\-conditional\. The gains on MBPP, the safety audit/proxy setting, and the three LLMBar anchors show what the role profile adds: it spends the same budget on judges whose residual information is useful for the current panel and target slice\([26](https://arxiv.org/html/2608.19802#bib.bib31)\)\.

Table 5:SOTA\-style allocation baselines\. Reliability jury is full\-call multi\-annotator aggregation; frugal cascade is confidence\-triggered budgeted routing\. Role policies are most informative when useful judges are slice\-conditional, as in deployable LLMBar slices and safety proxy/audit slices\.Reliability jury estimates which judges are globally trustworthy, but does not decide that a judge should be called only on a slice\. Frugal cascade decides when to call another globally ordered judge based on uncertainty, but it does not model specialist roles\. This explains Table[5](https://arxiv.org/html/2608.19802#S4.T5): the baselines are strong in broad\-complement regimes, while role routing is the natural deployment action when failure modes are conditional\.

These comparisons reveal two useful operating modes\. In broad\-complement settings such as hard GSM8K and MBPP, full\-call reliability juries can be the lowest\-risk endpoints because every signal contributes to the aggregate\. In slice\-conditional settings such as safety proxy/audit slices and LLMBar, useful information concentrates on failure modes\. There the role policy is both cheaper than the full\-call jury and lower risk than the cascade\.

Figure 2:Risk\-cost frontier across representative settings\. Each point is a held\-out policy evaluation averaged over 10 splits\. Full\-call stacking and full\-call jury can be low\-risk endpoints in broad\-ensemble regimes, but require invoking every judge\. Role policies occupy useful frontier regions when specialists or cheap verifiers matter\.Table 6:Mechanism evidence for the four actions induced by the taxonomy: route, stop, drop, and select broad\-ensemble endpoints\.#### Routed specialists\.

Table[7](https://arxiv.org/html/2608.19802#S4.T7)checks that routing is not merely an expensive global panel in disguise\. On LLMBar, the policy repeatedly sends different judges to declared adversarial and natural subsets across three anchors\. This is the mechanism missing from global reliability juries and confidence cascades: a judge can fail to clear the broad threshold while still clearing a slice threshold\.

Table 7:Representative routed specialists on LLMBar\. Frequencies count how often a judge\-slice route appears across 10 random splits\.
#### Stopping thresholds\.

Table[8](https://arxiv.org/html/2608.19802#S4.T8)varies the validation threshold\. The pattern is not that one threshold is universally best; it is thatτ\\taugives the researcher a transparent risk\-cost dial\. Higher thresholds save calls on hard GSM8K, MBPP, JBB, and MATH\-500\. LLMBar is a useful exception: a more conservative threshold also improves risk by avoiding sparse or redundant expansions\.

Table 8:Threshold sensitivity for role\-routed stopping\. Conservative thresholds reduce calls and provide an explicit stopping condition: add no remaining judge whose validation gain is belowτ\\tau\.
#### Copy stress test\.

The copy role should change deployment, not merely interpretation\. We therefore add four exact copies of an existing DeepSeek judge to LLMBar and JBB\. The role policy is unchanged because the copies have zero conditional validation gain after the original signal is present\. Full\-call baselines still pay for the copies, and reliability jury becomes worse because duplicated votes are overweighted\.

Table 9:Redundant\-copy stress test\. Four exact copies of an existing judge increase the apparent pool size, but role\-conditioned stopping ignores them after their conditional gain vanishes\.Taken together, the results support a policy interpretation of judge diversity\. When a new judge supplies broad conditional information, it should enter the global panel\. When its value is concentrated on a declared slice, it should be routed rather than called everywhere\. When its gain vanishes after conditioning on the current panel, it should be dropped even if it increases nominal model diversity\. When full\-call aggregation remains lower risk and the cost is acceptable, the regime map marks the full panel as the deployment endpoint\.

## 5Related Work

#### LLM\-as\-a\-judge evaluation\.

LLM judges are widely used for open\-ended generation, instruction following, translation quality, preference comparison, and rubric scoring\([45](https://arxiv.org/html/2608.19802#bib.bib20);[30](https://arxiv.org/html/2608.19802#bib.bib13);[23](https://arxiv.org/html/2608.19802#bib.bib29);[46](https://arxiv.org/html/2608.19802#bib.bib21);[22](https://arxiv.org/html/2608.19802#bib.bib11);[15](https://arxiv.org/html/2608.19802#bib.bib24)\)\. This line of work establishes that LLMs can be useful evaluators, but also documents evaluator\-specific biases such as position and length effects\([41](https://arxiv.org/html/2608.19802#bib.bib38);[15](https://arxiv.org/html/2608.19802#bib.bib24)\)\. Benchmarks such as LLMBar, RewardBench, Arena100K, SummEval, MATH/MATH\-500, GSM8K, MBPP, and HumanEval define the targets and stress cases used in this paper\([44](https://arxiv.org/html/2608.19802#bib.bib19);[27](https://arxiv.org/html/2608.19802#bib.bib12);[11](https://arxiv.org/html/2608.19802#bib.bib6);[16](https://arxiv.org/html/2608.19802#bib.bib9);[19](https://arxiv.org/html/2608.19802#bib.bib10);[29](https://arxiv.org/html/2608.19802#bib.bib35);[12](https://arxiv.org/html/2608.19802#bib.bib7);[2](https://arxiv.org/html/2608.19802#bib.bib1);[9](https://arxiv.org/html/2608.19802#bib.bib4)\)\. Our contribution is not a new judge benchmark\. It is a policy for deciding which available judge signals to call under a finite audit budget\.

The same literature also motivates the need for conditional allocation rather than one global judge ranking\. LLMBar constructs adversarial preference cases that expose evaluator failures\([44](https://arxiv.org/html/2608.19802#bib.bib19)\); AlpacaEval and FairEval document length and position biases in automatic judges\([15](https://arxiv.org/html/2608.19802#bib.bib24);[41](https://arxiv.org/html/2608.19802#bib.bib38)\); and JailbreakBench separates safety evaluation from ordinary helpfulness or preference evaluation\([7](https://arxiv.org/html/2608.19802#bib.bib3)\)\. These findings imply that a judge’s usefulness can depend sharply on the failure mode\. Role\-conditioned routing treats that dependence as a deployment object: a judge may be worth calling on adversarial, safety\-proxy, or disagreement regions without being worth calling everywhere\.

#### Judge panels and multi\-agent evaluators\.

Several works study using more than one LLM evaluator, either as panels or multi\-agent discussions\([40](https://arxiv.org/html/2608.19802#bib.bib17);[6](https://arxiv.org/html/2608.19802#bib.bib23)\)\. These methods motivate judge diversity, but they do not by themselves determine whether an additional judge should be called globally, routed to a slice, or dropped as a copy\. Role\-conditioned allocation treats panel construction as a conditional value problem: the value of a candidate depends on the current panel, the target distribution, the slice, and the cost\. Correlated\-error audits show that nominal panel size can substantially overstate effective information\([25](https://arxiv.org/html/2608.19802#bib.bib45)\), while calibrated full\-panel results show that weak but nonredundant judges can remain useful when their signals are learnable\([28](https://arxiv.org/html/2608.19802#bib.bib46)\)\. Our policy reconciles these observations at deployment time by retaining a signal only when its finite\-sample conditional gain justifies its call cost\.

#### Annotator aggregation and preference models\.

Reliability\-based aggregation has a long history in multi\-annotator learning\([13](https://arxiv.org/html/2608.19802#bib.bib8);[38](https://arxiv.org/html/2608.19802#bib.bib32);[42](https://arxiv.org/html/2608.19802#bib.bib40)\), and pairwise comparison models such as Bradley–Terry remain standard tools for preference aggregation\([3](https://arxiv.org/html/2608.19802#bib.bib2)\)\. These methods estimate global or item\-conditioned annotator reliability from observed labels\. Our setting is different because a deployment system must decide which judge outputs to observe in the first place\. A judge can be low\-reliability globally but useful on one slice, or high\-reliability but redundant after a verifier enters the panel\.

#### Ensembles, deferral, cascades, and calibration\.

Stacked generalization and ensemble selection show how to combine many model outputs once they are observed\([43](https://arxiv.org/html/2608.19802#bib.bib18);[5](https://arxiv.org/html/2608.19802#bib.bib22)\)\. Classifier\-ensemble work also studies diversity measures and their limits\([26](https://arxiv.org/html/2608.19802#bib.bib31)\)\. Learning\-to\-defer methods train systems to route examples to a human or expert when delegation improves task performance\([32](https://arxiv.org/html/2608.19802#bib.bib14);[33](https://arxiv.org/html/2608.19802#bib.bib15)\)\. Model cascades and routers reduce inference cost by calling stronger models only when needed\([8](https://arxiv.org/html/2608.19802#bib.bib5);[34](https://arxiv.org/html/2608.19802#bib.bib16)\)\. More generally, wrapper selection, conditional\-redundancy criteria, and budgeted classifier cascades select predictive signals under validation or acquisition costs\([24](https://arxiv.org/html/2608.19802#bib.bib41);[4](https://arxiv.org/html/2608.19802#bib.bib42);[10](https://arxiv.org/html/2608.19802#bib.bib43)\); active feature acquisition makes the corresponding per\-example decision about which costly features to observe\([39](https://arxiv.org/html/2608.19802#bib.bib39)\)\. Cascaded Selective Evaluation applies confidence\-based escalation to LLM judges with a target human\-agreement guarantee\([21](https://arxiv.org/html/2608.19802#bib.bib44)\)\. In contrast, our setting selects a set\-valued panel policy from finite audit data, allowing global complements, slice\-routed specialists, and unused conditional copies\. Role\-conditioned allocation applies the same conditional\-computation question to evaluation itself\. Its units are auditable judge calls: use calibrated validation gain\([18](https://arxiv.org/html/2608.19802#bib.bib28)\)to decide whether an output should be obtained globally, routed to a declared slice, or left uncalled\. The LLM\-judge setting adds three constraints that ordinary routers do not address together: route signals must be deployable before the routed judge call, human\-label strata can be audit diagnostics without being route inputs, and nominally different prompts or models may be conditional copies after the current panel has already been observed\.

## 6Discussion

Judge diversity becomes useful when it changes a deployment action\. A model family, prompt template, or reward head can look diverse on paper and still be a copy after conditioning on the current panel; a weak standalone judge can be exactly the call worth making on one failure slice\. The practical workflow is a calibration loop: collect a labeled audit set, run the candidate judge pool once, declare decision\-relevant slices, and fit the stopped role policy\. The output is a call plan with global judges, routed specialists, and unused judges whose validation gain did not justify their cost\.

That plan has a direct operational reading\. If a deterministic verifier dominates, keep it and stop\. If broad gains plateau but slice gains remain, route specialists\. If copied signals appear, drop them without changing the rest of the policy\. If the full panel is still the lowest\-risk endpoint and the budget allows it, pay for that endpoint\. Across repeated audit splits, risk\-cost stability and exact plan identity should be read separately: stable risk with variable routes calls for frequency checking, keeping recurring calls and collecting more audit labels before relying on low\-frequency specialists\. This frequency audit is a deployment diagnostic, not a statistical guarantee\.

The threshold is the risk\-cost dial\. Lower thresholds keep marginal calls when evaluation errors are expensive; higher thresholds produce leaner policies when latency or budget dominates\. If small threshold changes alter the selected panel, the audit set is signaling instability; if the same copy, route, or stop decisions persist, the call plan is more credible for the next evaluation batch\.

## 7Deployment Extensions

The same policy interface scales along three axes\. Larger judge pools can keep the role profile while replacing finite\-cell means with smoothed, cross\-fitted, or parametric calibrators when joint output cells become sparse\. Declared slices can be extended by automatic slice discovery: discover candidate failure regions, validate whether any judge has conditional value there, and route only specialists that clear threshold\. Normalized call costs can also be replaced by actual API prices, latency, safety\-review budget, or carbon budget; the output remains a validation\-backed plan of global calls, routed specialists, and stopped candidates\.

## 8Conclusion

We proposed role\-conditioned panel policies for LLM judge allocation\. The shift is from describing judge diversity to deciding judge calls\. Target\-relative profiles identify copies, complements, and specialists; validation\-stopped construction turns them into global and routed policies; held\-out evaluation reports the resulting risk\-cost tradeoff\. The resulting regime map tells LLM researchers when to drop, add, route, stop, or pay for a full panel\.

The broader point is that a larger panel is not automatically more reliable, and a smaller panel is not automatically more efficient if it drops conditional information\. Future judge\-panel studies should therefore report not only which evaluator scored best, but also which additional judges were worth calling, where they were worth routing, and why panel construction stopped\.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China \(62372483\)\.

## References

- Alexandruet al\.\(2025\)A\. Alexandru, A\. Calvi, H\. Broomfield, J\. Golden, K\. Dai, M\. Leys, M\. Burger, M\. Bartolo, R\. Engeler, S\. Pisupati, T\. Drane, and Y\. S\. ParkAtla Selene Mini: a general purpose evaluation model\.arXiv preprint arXiv:2501\.17195\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px3.p1.1)\.
- Brownet al\.\(2012\)G\. Brown, A\. Pocock, M\. Zhao, and M\. LujánConditional likelihood maximisation: a unifying framework for information theoretic feature selection\.Journal of Machine Learning Research13,pp\. 27–66\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Caruanaet al\.\(2004\)R\. Caruana, A\. Niculescu\-Mizil, G\. Crew, and A\. KsikesEnsemble selection from libraries of models\.InProceedings of the 21st International Conference on Machine Learning,External Links:[Document](https://dx.doi.org/10.1145/1015330.1015432)Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p2.1),[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Chanet al\.\(2024\)C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. LiuChatEval: towards better LLM\-based evaluators through multi\-agent debate\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px2.p1.1)\.
- Chaoet al\.\(2024\)P\. Chao, E\. Debenedetti, A\. Robey, M\. Andriushchenko, F\. Croitoru, V\. Sehwag, E\. Dobriban, N\. Flammarion, G\. J\. Pappas, F\. Tramer, H\. Hassani, and E\. WongJailbreakBench: an open robustness benchmark for jailbreaking large language models\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[Table 10](https://arxiv.org/html/2608.19802#A2.T10),[§2](https://arxiv.org/html/2608.19802#S2.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p2.1)\.
- Chenet al\.\(2024\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2012\)M\. Chen, Z\. Xu, K\. Q\. Weinberger, O\. Chapelle, and D\. KedemClassifier cascade for minimizing feature evaluation cost\.InProceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics,Vol\.22,pp\. 218–226\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Chianget al\.\(2024\)W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez, and I\. StoicaChatbot Arena: an open platform for evaluating LLMs by human preference\.arXiv preprint arXiv:2403\.04132\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Dawid and Skene \(1979\)A\. P\. Dawid and A\. M\. SkeneMaximum likelihood estimation of observer error\-rates using the EM algorithm\.Journal of the Royal Statistical Society\. Series C \(Applied Statistics\)28\(1\),pp\. 20–28\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px3.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.InConference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p2.1)\.
- Fabbriet al\.\(2021\)A\. R\. Fabbri, W\. Kryscinski, B\. McCann, C\. Xiong, R\. Socher, and D\. RadevSummEval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics9,pp\. 391–409\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Gemma Team \(2025\)Gemma TeamGemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 1321–1330\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p3.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InNeurIPS Datasets and Benchmarks,Cited by:[§2](https://arxiv.org/html/2608.19802#S2.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. Le Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El SayedMistral 7B\.arXiv preprint arXiv:2310\.06825\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1)\.
- Junget al\.\(2025\)J\. Jung, F\. Brahman, and Y\. ChoiTrust or escalate: LLM judges with provable guarantees for human agreement\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Kimet al\.\(2024\)S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. SeoPrometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1),[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Kocmi and Federmann \(2023\)T\. Kocmi and C\. FedermannLarge language models are state\-of\-the\-art evaluators of translation quality\.InProceedings of the 24th Annual Conference of the European Association for Machine Translation,pp\. 193–203\.Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Kohavi and John \(1997\)R\. Kohavi and G\. H\. JohnWrappers for feature subset selection\.Artificial Intelligence97\(1–2\),pp\. 273–324\.External Links:[Document](https://dx.doi.org/10.1016/S0004-3702%2897%2900043-X)Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Kohli \(2026\)G\. KohliNine judges, two effective votes: correlated errors undermine LLM evaluation panels\.arXiv preprint arXiv:2605\.29800\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px2.p1.1)\.
- Kuncheva and Whitaker \(2003\)L\. I\. Kuncheva and C\. J\. WhitakerMeasures of diversity in classifier ensembles and their relationship with the ensemble accuracy\.Machine Learning51,pp\. 181–207\.Cited by:[§4](https://arxiv.org/html/2608.19802#S4.p5.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Lambertet al\.\(2024\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Li \(2026\)Y\. LiCalibrate, don’t curate: label\-efficient estimation from noisy LLM judges\.arXiv preprint arXiv:2605\.09702\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px2.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Llama Team \(2024\)Llama TeamThe Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1)\.
- Madraset al\.\(2018\)D\. Madras, T\. Pitassi, and R\. ZemelPredict responsibly: improving fairness and accuracy by learning to defer\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Mozannar and Sontag \(2020\)H\. Mozannar and D\. SontagConsistent estimators for learning to defer to an expert\.InProceedings of the 37th International Conference on Machine Learning,pp\. 7076–7087\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Onget al\.\(2025\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to route LLMs from preference data\.InInternational Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- OpenAI \(2023\)OpenAIGPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[Table 10](https://arxiv.org/html/2608.19802#A2.T10)\.
- Qwen Team \(2024\)Qwen TeamQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1)\.
- Raykaret al\.\(2010\)V\. C\. Raykar, S\. Yu, L\. H\. Zhao, G\. H\. Valadez, C\. Florin, L\. Bogoni, and L\. MoyLearning from crowds\.Journal of Machine Learning Research11,pp\. 1297–1322\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px3.p1.1)\.
- Shimet al\.\(2018\)H\. Shim, S\. J\. Hwang, and E\. YangJoint active feature acquisition and classification with variable\-size set encoding\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Vergaet al\.\(2024\)P\. Verga, S\. Hofstatter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. LewisReplacing judges with juries: evaluating LLM generations with a panel of diverse models\.arXiv preprint arXiv:2404\.18796\.Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 9440–9450\.Cited by:[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p2.1)\.
- Whitehillet al\.\(2009\)J\. Whitehill, T\. Wu, J\. Bergsma, J\. R\. Movellan, and P\. L\. RuvoloWhose vote should count more: optimal integration of labels from labelers of unknown expertise\.InAdvances in Neural Information Processing Systems,Vol\.22\.Cited by:[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px3.p1.1)\.
- Wolpert \(1992\)D\. H\. WolpertStacked generalization\.Neural Networks5\(2\),pp\. 241–259\.Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p2.1),[§3](https://arxiv.org/html/2608.19802#S3.p2.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px4.p1.1)\.
- Zenget al\.\(2024\)Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. ChenEvaluating large language models at evaluating instruction following\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.19802#S2.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.arXiv preprint arXiv:2306\.05685\.Cited by:[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2023\)L\. Zhu, X\. Wang, and X\. WangJudgeLM: fine\-tuned large language models are scalable judges\.arXiv preprint arXiv:2310\.17631\.Cited by:[Appendix C](https://arxiv.org/html/2608.19802#A3.p1.1),[§1](https://arxiv.org/html/2608.19802#S1.p1.1),[§3](https://arxiv.org/html/2608.19802#S3.p1.1),[§5](https://arxiv.org/html/2608.19802#S5.SS0.SSS0.Px1.p1.1)\.

## Appendix AFull Role Profile and Construction Details

### A\.1Role Profiles With Specialization Ratios

The main text uses the compact role profileprofileP,ℱ,S⁡\(j\)=\(CP,\{Af\}f∈ℱ\)\\operatorname\{profile\}\_\{P,\\mathcal\{F\},S\}\(j\)=\(C\_\{P\},\\\{A\_\{f\}\\\}\_\{f\\in\\mathcal\{F\}\}\)\. In the experiments we also track a specialization ratio

ρf​\(j∣S\)=gPf​\(j∣S\)gP​\(j∣S\)\+ϵ0,\\rho\_\{f\}\(j\\mid S\)=\\frac\{g\_\{P\_\{f\}\}\(j\\mid S\)\}\{g\_\{P\}\(j\\mid S\)\+\\epsilon\_\{0\}\},whereϵ0\>0\\epsilon\_\{0\}\>0prevents division by zero\. The ratio is not used to make complement and specialist mutually exclusive\. Instead, it is a diagnostic for concentration: a judge may have positive broad gain and still be unusually valuable on one declared slice\. This is why the role table in the main text allows a complement\-plus\-specialist role\.

The role interpretation used throughout the experiments is:

- •Copy:broad gain and all slice gains fall below the declared threshold after conditioning on the current panel\.
- •Broad complement:cost\-adjusted broad validation gain exceedsτP\\tau\_\{P\}, so the judge is added to the global panel\.
- •Slice specialist:cost\-adjusted slice validation gain exceedsτf\\tau\_\{f\}, so the judge is invoked only on examples routed to that slice\.
- •Complement plus specialist:broad gain is positive and at least one slice gain is concentrated; the judge can enter globally and can also be prioritized for interpretation on that slice\.

### A\.2Construction Algorithm

The complete construction procedure is:

1. 1\.Collect a labeled audit set and run all candidate judges on it\.
2. 2\.Declare slicesℱ\\mathcal\{F\}that are meaningful for the target deployment distribution\.
3. 3\.Split the audit set into construction\-fit, construction\-validation, and final\-test portions\.
4. 4\.Start from an empty global panel or a user\-specified seed panel\.
5. 5\.Fit the pattern calibrator for the current global panel on the fit split\.
6. 6\.Estimate each remaining candidate’s broad validation gain on the validation split\.
7. 7\.Add the candidate with the largest positive cost\-adjusted broad gain if that gain exceedsτP\\tau\_\{P\}; otherwise stop global construction\.
8. 8\.For each slice, repeat the same greedy search after conditioning on the selected global panel and any specialists already assigned to that slice\.
9. 9\.Refit calibrators for the selected global and routed paths on the full construction split\.
10. 10\.Evaluate the resulting calling policy on final\-test examples only\.

Judge outputs are canonicalized before pattern construction: binary and preference labels are mapped to normalized symbols, confidence suffixes are stripped for the pattern table, and numeric rubric scores such as SummEval’s 1–5 judgments remain ordinal cell labels for the pattern calibrator\. Numeric scores are normalized to\[0,1\]\[0,1\]only for scalar stacking and cascade baselines\. Experiments use complete\-case rows for the declared judge pool; rows with missing or unparseable selected judge outputs are excluded before splitting\. When two candidates have the same cost\-adjusted gain, ties are resolved by raw gain and then by the judge identifier, making the construction deterministic for a fixed split\.

This procedure produces both a policy and a stopping report\. The stopping report is the set of failed inequalities: after stopping, every unused broad candidate is belowτP\\tau\_\{P\}, and every unused routed candidate is below the corresponding slice threshold\. The stopping report is tied to the search space used by the audit\. With the default greedy search, it records that no single additional call is justified under the finite audit set, threshold, and cost model\. With beam, pair\-addition, or subset proposals, the same reporting format certifies the expanded candidate moves\.

## Appendix BDataset and Slice Details

#### Hard GSM8K rationale audits\.

Ordinary GSM8K answer checking is too easy for the main claim because an answer verifier can saturate the target\. The hard rationale setting instead asks whether a candidate solution rationale is valid\. This creates a complement regime: the verifier is cheap and useful, but LLM judges can still provide conditional information about reasoning validity\.

#### MBPP public\-test overfit audits\.

The code setting asks whether a candidate program has overfit public tests or generalizes to hidden tests\. The hidden\-test verifier is cheap and often dominant, but LLM judges may still help on residual code\-audit cases\. This setting tests whether the policy can combine a deterministic verifier with a small number of complementary LLM calls instead of always invoking the full panel\.

#### JailbreakBench safety\.

The safety setting includes human\-labeled safe and unsafe responses\. Slices include human\-label strata for audit analysis and deployable proxy regions defined by safety\-classifier output or judge disagreement\. Human labels are not available at deployment time, so they serve as audit strata rather than route inputs\. The practical deployment question is whether a safety judge should be invoked globally or only on proxy regions where the current panel is unreliable; label\-conditioned JBB tables report audit\-slice evidence, while classifier and disagreement proxies define deployable routes\.

Table 10:JailbreakBench safety proxy\-routing audit\. The route signal is the dataset’s GPT\-4 classifier fieldgpt4\_cf\([7](https://arxiv.org/html/2608.19802#bib.bib3);[35](https://arxiv.org/html/2608.19802#bib.bib36)\), which is stored separately from human\-majority labels and is used here as a deployable proxy slice signal\. Across 10 splits, the routed policy selected extra specialists on the classifier\-safe proxy slice in 4 splits and on the classifier\-unsafe proxy slice in 2 splits\.This audit is the safety version of the paper’s central deployment question\. The full\-call stack remains the best risk endpoint because it sees all seven judge outputs, but role routing nearly reaches that endpoint while buying about one third of the calls\. The selected specialists are not fixed globally:mistral\_7b\_safety,deepseek\_v4\_flash\_safety,prometheus\_7b\_safety, andselene\_8b\_safetyappear on differentgpt4\_cfproxy slices across splits\. This is the desired behavior for a live safety audit: use a cheap classifier proxy to decide where the panel needs extra scrutiny, and keep the stopped global panel elsewhere\.

#### LLMBar preference\.

LLMBar is the main specialist\-routing benchmark\. Its natural and adversarial subsets induce different failure modes\. The repeated routes in the main text show that the policy selects different judges for adversarial instruction, adversarial output, adversarial neighbor, and natural subsets, rather than expanding the full panel uniformly\.

#### RewardBench and Arena100K preference\.

These preference settings test broad\-ensemble behavior\. Role policies expose cheap frontier points, and full\-call aggregation can remain the lowest\-risk endpoint\. The regime map identifies when to pay for a broad ensemble and when a stopped policy is already sufficient\.

#### SummEval scalar judging\.

SummEval evaluates scalar summary quality\. It is useful because additional judges can worsen or barely improve the risk\-cost tradeoff\. In this regime, a one\-step stopped policy is a meaningful outcome rather than a failed panel\. The pattern policy treats each 1–5 rubric output as a discrete cell symbol, while scalar baselines use the normalized score value\.

#### MATH\-500 correctness\.

MATH\-500 tests whether hard math correctness benefits from broad ensembles\. The stopped role policy gives a cheaper point, while full\-call stacking can be lower risk\. This supports the regime\-map framing\.

#### HumanEval and ordinary GSM8K\.

These are saturated stopping checks\. If a unit\-test verifier or answer verifier already solves the audit target, the correct allocation decision is to stop rather than to claim artificial panel gains\.

## Appendix CJudge Pool Disclosure

Table[11](https://arxiv.org/html/2608.19802#A3.T11)lists the candidate signals used by the main experiments\. Names with suffixes\_v,\_s,\_safety, or task\-specific correctness suffixes are direct\-schema judge outputs\. Verifiers have normalized cost0\.10\.1; all LLM judge calls have normalized cost1\.01\.0\. Route keys are treated as pre\-available metadata, verifier outputs, classifier outputs, or already\-observed proxy signals\. If a deployment must call an additional model to compute a route key, that call should be added to the cost model before refitting the policy\. The candidate model families are DeepSeek, Qwen2\.5/Qwen3, Gemma, Llama, Mistral, Prometheus, JudgeLM, and Selene\([14](https://arxiv.org/html/2608.19802#bib.bib25);[36](https://arxiv.org/html/2608.19802#bib.bib34);[37](https://arxiv.org/html/2608.19802#bib.bib33);[17](https://arxiv.org/html/2608.19802#bib.bib26);[31](https://arxiv.org/html/2608.19802#bib.bib27);[20](https://arxiv.org/html/2608.19802#bib.bib30);[22](https://arxiv.org/html/2608.19802#bib.bib11);[46](https://arxiv.org/html/2608.19802#bib.bib21);[1](https://arxiv.org/html/2608.19802#bib.bib37)\)\.

Table 11:Judge\-pool disclosure for the main experiments\. The table reports the signals available to the allocation policy before it selects global calls, routed specialists, or stopped candidates\.Table[12](https://arxiv.org/html/2608.19802#A3.T12)reports the complete\-case filtering used before random splitting\. The main seven\-judge panels drop at most two rows\. The near\-duplicate prompt\-variant audit has a higher drop rate because one letter\-prompt judge has many unparseable outputs; that audit is therefore read as a complete\-case prompt\-variant stress test rather than as a claim about parse robustness\.

Table 12:Complete\-case filtering before policy construction\. Rows with missing or unparseable outputs for the declared judge pool are excluded before the construction/final\-test split\.
## Appendix DBaseline Implementation Details

All baseline choices use the same construction\-validation split as the role policy, and all reported numbers are computed on the final\-test split only\. This matters because full\-call aggregation and cascades have enough flexibility to overfit a small audit if their regularization, order, or thresholds are chosen after looking at final\-test outcomes\. We therefore treat baseline selection as part of the deployment procedure rather than as an oracle leaderboard\.

#### Single best and flat all\.

The single\-best baseline selects the judge with lowest validation risk and invokes only that judge on final\-test examples\. The flat\-all baseline invokes every candidate judge for every example and calibrates on the joint output pattern\.

#### Matched\-size non\-role panels\.

Matched\-size panels use the same average number of calls as the stopped role policy but select judges without role conditioning\. The top\-kkversion uses standalone validation quality\. The correlation\-diverse version discourages highly correlated judge outputs\. The quality\-diverse version balances standalone quality with nominal diversity\. These baselines ask whether copy/complement/specialist roles add value beyond ordinary diversity heuristics\. The matched budget is fixed from construction\-validation behavior, then evaluated once on final\-test examples\.

#### Full\-call stacking\.

Full\-call ridge and logistic stacking are supervised aggregation endpoints: they observe every judge output before predicting\. Pairwise variants include features derived from pairwise judge\-output interactions\. These methods can be excellent low\-risk endpoints, but they answer a different question from the allocation policy because their call cost is fixed at the full panel\. The regularized linear or logistic variant used for a setting is selected on construction\-validation risk, not on final\-test risk\.

#### Reliability jury\.

The reliability jury treats judges as noisy annotators and estimates label\-conditional error behavior from construction data\. It is a strong multi\-annotator aggregation baseline, especially when judge reliability is mostly global\. It does not decide that a judge should be called only on one slice\.

#### Frugal cascade\.

The cascade orders judges by validation quality and invokes additional judges when the current calibrated prediction is uncertain\. It is a strong cost\-aware baseline when one global order is adequate\. It differs from role routing because it does not identify slice specialists that should be called only on declared failure modes\. Its uncertainty threshold is chosen on the construction\-validation split under the same normalized cost model as the role policy\.

## Appendix ESplit\-Level Variation

Table[13](https://arxiv.org/html/2608.19802#A5.T13)reports the split\-level uncertainty behind the main risk table\. Each entry is the mean held\-out risk over 10 random splits with a 95% confidence interval computed asr¯±t0\.975,9​s/10\\bar\{r\}\\pm t\_\{0\.975,9\}\\,s/\\sqrt\{10\}, wheressis the sample standard deviation across splits\. These intervals measure random split variation, not uncertainty over future datasets or changing judge models\.

Table 13:Split\-level 95% confidence intervals for the main held\-out risk comparisons\. The role column uses the routed policy when routing is selected and the global stopped policy in one\-step stopping regimes\.
## Appendix FAdditional Interpretation of Main Results

#### Few\-judge complement regimes\.

Hard GSM8K rationale checking is neither saturated answer verification nor pure slice routing\. The single\-best judge reaches 0\.6253 accuracy, the flat panel reaches 0\.6670, and the stopped role policy reaches 0\.6843 with about 2\.9 calls\. The best full\-call and reliability\-jury endpoints can be lower risk, but the role policy recovers much of the benefit without paying for every judge\.

MBPP public\-overfit is easier but still non\-saturated\. The role policy reaches 0\.9900 accuracy at cost 1\.52, exceeding flat\-all accuracy while using far fewer calls\. The result illustrates a practical pattern: a cheap verifier can be a dominant signal while a small number of LLM judges remain useful\.

#### Specialist\-routing regimes\.

LLMBar is the clearest case where the taxonomy becomes a calling policy\. Routing slice specialists improves accuracy from 0\.6692 for flat all and 0\.6822 for single best to 0\.7334 at 3\.46 calls\. The same qualitative pattern appears under Qwen3 and JudgeLM anchors\. JBB shows a related safety pattern: the role policy is near the best full\-call risk endpoint while using roughly a third of the full\-panel cost on proxy/audit slices\. Only proxy slices based on classifier outputs or judge disagreement are valid deployment\-time route signals; human\-label strata are audit diagnostics\.

#### One\-step stopping regimes\.

Arena100K and SummEval demonstrate one\-step stopping in non\-saturated settings\. Expanding the panel worsens or barely improves the risk\-cost tradeoff, so the policy keeps a strong single judge\. HumanEval and ordinary GSM8K are stronger sanity checks: once a verifier solves the target, all remaining LLM judges have zero useful validation gain\.

#### Broad\-ensemble boundary regimes\.

RewardBench and MATH\-500 are boundary cases\. The stopped role policies are cheaper, but full\-call stacking can remain lower risk\. This is the intended regime diagnosis: if the target still benefits from broad ensemble information and cost is acceptable, the policy tells the researcher to keep the full panel\.

## Appendix GDeployment Robustness Audits

After the initial risk\-cost frontier is known, the deployment owner should run three compact checks: whether selected pattern tables are sparse, whether the call plan is stable with fewer audit labels, and whether the expected slice mix matches the construction audit set\. The owner then applies the relevant cost model\. Because our deployable calibrator is a finite pattern table, we first audit how often the selected call plan sees a joint judge pattern absent from its fitting split and therefore falls back to the split mean\. Table[14](https://arxiv.org/html/2608.19802#A7.T14)shows that the sparse\-pattern pressure is concentrated in routed LLMBar/JBB specialists; MBPP, MATH\-500, and SummEval stop at short paths and have essentially no fallback\. The diagnostic is operational: if fallback is high on a target slice, collect more slice labels or cap route depth before deploying that route\.

Table 14:Pattern\-table sparsity for the selected role policy\. Each row averages 10 random construction/test splits\. Cells are occupied response\-pattern cells in the refit construction calibrator; route cells report the largest selected slice\-specific table in the split\. Fallback is the fraction of validation or final\-test examples whose invoked pattern was unseen in the corresponding fit table\.Table[15](https://arxiv.org/html/2608.19802#A7.T15)fixes the held\-out test split and varies the fraction of the construction audit set used for fitting and validation\. The half\-audit policies already recover the main LLMBar and JBB proxy risk\-cost behavior, while exact call\-plan identity is less stable\. This is a useful diagnostic: the policy can be deployed when risk and cost are stable, and more audit labels should be collected when the exact route set matters\.

Table 15:Audit\-size stability\. The held\-out test split is fixed for each seed; the construction audit set is reduced before fitting the role policy\. Plan Jaccard compares the global and routed call set with the full\-audit plan for the same split\. Entries are means over 10 random splits\.Table[16](https://arxiv.org/html/2608.19802#A7.T16)changes the next\-batch slice mix while fitting the call plan on a different construction mix\. This is the deployment reading: route policies remain useful when the construction audit contains enough examples from the deployment slice, and the table tells the owner when to buy more slice labels before trusting low\-frequency routes\. On LLMBar, an adversarial\-heavy construction set transfers cleanly to a natural\-heavy batch, while natural\-heavy construction is not the right audit for an adversarial\-heavy next batch\. On JBB, proxy\-unsafe construction transfers to a proxy\-safe batch and selects a useful routed plan\.

Table 16:Slice\-mix shift audit\. Construction and deployment splits are disjoint and intentionally use different route\-signal proportions\. Entries are mean held\-out risk/accuracy over 10 random shifted splits\. The table is read as a deployment check on whether the audit labels match the next evaluation batch\.Table[17](https://arxiv.org/html/2608.19802#A7.T17)turns the same audit into a deployment diagnostic\. Rather than deploying every split\-specific route, the owner can keep calls that appear in at least half of the calibration splits\. LLMBar yields a stable global pair and a small number of recurring slice specialists\. JBB has a stable global safety judge but no routed proxy specialist above the same frequency threshold, so the conservative action is to deploy the global call and collect more proxy\-slice audit labels before adding routes\.

Table 17:Frequency audit for conservative deployment\. Calls are listed when they appear in at least 5/10 random splits\. Route entries useslice:judge; unqualified entries are global calls\. Low\-frequency routes are not deployment failures; they are prompts to collect more audit labels before relying on that specialist\. This table is not a statistical guarantee\.Table[18](https://arxiv.org/html/2608.19802#A7.T18)replaces uniform call costs with a production\-style cost model and sweepsλ\\lambda\. For LLMBar and JBB, the DeepSeek API anchor has cost 5 and local judges have cost 1\. For MBPP, local LLM judges have cost 3, the DeepSeek API anchor has cost 5, and the hidden\-unit verifier has cost 0\.05\. Increasingλ\\lambdaasks for a leaner policy under the same validation\-gain objective\. The resulting plans keep the same operational form–global calls, routed specialists, and stopped candidates–but move along the deployment owner’s risk\-cost frontier\.

Table 18:Production\-cost sensitivity\. Costs are normalized but non\-uniform: API anchors are expensive, local judges are cheaper, and deterministic verifiers are cheapest\. Entries are role\-routed policy means over 10 random splits\.
## Appendix HNear\-Duplicate and Search\-Space Audits

Exact copies are the easiest redundancy case\. A more useful deployment question is whether prompt variants of the same judge family should be treated as extra judges\. Table[19](https://arxiv.org/html/2608.19802#A8.T19)forms a ten\-call LLMBar pool by pairing JSON\-schema and letter\-prompt outputs from five local judge families\. The audit uses the 680 complete\-case rows reported in Table[12](https://arxiv.org/html/2608.19802#A3.T12)\. The result separates two effects\. Calling every variant is not diversity: flat aggregation over all variants raises risk to0\.26330\.2633and costs ten calls\. Yet a prompt variant can still become a slice specialist\. The role\-routed policy reaches0\.71430\.7143accuracy at3\.643\.64calls, exceeding the full\-call logistic stack accuracy of0\.68620\.6862while using fewer calls\. The operational rule is therefore not to delete all near\-duplicates, nor to buy all variants; it is to route variants only where their conditional validation gain clears the deployment threshold\.

Table 19:Near\-duplicate prompt\-variant audit on LLMBar\. The pool contains JSON\-schema and letter\-prompt outputs from five judge families\. Role routing keeps prompt variants as conditional specialists rather than treating them as automatic diversity\.The stopping report can also be audited for pairwise complementarity\. After the greedy path stops, Table[20](https://arxiv.org/html/2608.19802#A8.T20)checks whether any remaining pair clears the same validation\-gain threshold when no remaining single judge does\. LLMBar and MBPP have no pair\-only misses across ten splits\. JBB and the prompt\-variant audit expose a small number of pair\-only moves, so a deployment owner who wants to search beyond single additions can widen the stopping report to beam or subset moves on those regimes\.

Table 20:Pair\-addition audit after greedy stopping\. A pair\-only move is a pair whose validation gain exceedsτ=0\.005\\tau=0\.005after every remaining single addition falls below threshold\.
## Appendix IStopping Reports

The method produces auditable stopping decisions\. On HumanEval, the first global step adds the unit\-test verifier, reducing construction\-validation risk to zero; all remaining LLM judges then have zero validation gain and the panel stops\. This report is deployment\-relevant: if the verifier is available and matches the target, the default deployment call is the verifier alone\.

On LLMBar, the report has a different shape\. The global panel adds a small set of broad complements, then stops globally when the next broad gain falls below threshold\. Slice construction then routes different specialists to adversarial and natural subsets\. This demonstrates that a judge can fail to be worth invoking globally while still being worth invoking conditionally\.

On the redundant\-copy stress test, the report states that the injected copies have no conditional validation gain after the original judge is present\. This is why role risk and role cost are unchanged when the candidate pool grows from seven to eleven judges\.