Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions

# Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator–Agent Conditions
Source: [https://arxiv.org/html/2607.00304](https://arxiv.org/html/2607.00304)
###### Abstract

The bias\-reliability tradeoff conjectures that LLM evaluation systems are constrained in\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)space, where evaluator coupling \(γ\\gamma\), strategy diversity \(HH\), and small\-sample measurement reliability \(CV\(N\)\(N\)\) cannot be simultaneously optimized at fixed sample sizeNN\. Prior evidence rests onn=5n\{=\}5conditions with complete metrics from a single study\. We expand the empirical base to 11 conditions, measuringγ\\gammaandHHfor all 11 \(nine with valid weight vectors\) and CV\(N=5\)\(N\{=\}5\)for seven with sufficient seeds \(N≥5N\\geq 5\)\. Five conditions provide the complete\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)triple\. The data confirm the trade\-off: conditions with low evaluator coupling \(γ<0\.2\\gamma<0\.2\) exhibit high measurement noise \(CV\(N=5\)\>1\.0\(N\{=\}5\)\>1\.0\), while conditions with strong coupling \(γ\>0\.9\\gamma\>0\.9\) achieve low noise \(CV\(N=5\)<0\.16\(N\{=\}5\)<0\.16\)\. The correlationr​\(H,γ\)=−0\.989r\(H,\\gamma\)=\-0\.989\(n=5n\{=\}5, excluding GPT\-4o conditions discussed below\) confirms that evaluator coupling suppresses strategy diversity\. Four GPT\-4o conditions showγ=0\.000\\gamma\{=\}0\.000andH=1\.000H\{=\}1\.000across all seeds—a pattern we attribute to insufficient evaluator signal in the June 2026 GPT\-4o API version, consistent with previously documented version drift\. No condition occupies the region\{γ<0\.2,CV​\(N=5\)<0\.3\}\\\{\\gamma<0\.2,\\text\{CV\}\(N\{=\}5\)<0\.3\\\}\. We release all per\-condition metrics as a standardized benchmark dataset for evaluator comparison\.

## 1Introduction

LLM evaluation faces a structural challenge: the properties that make an evaluator desirable—unbiasedness, reliability at small sample sizes, and encouragement of diverse agent strategies—trade off against each other\.Anonymous \([2026a](https://arxiv.org/html/2607.00304#bib.bib1)\)formalized this as a constrained triangle in\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)space, where:

- •γ≥0\\gamma\\geq 0is the evaluator coupling coefficient—the normalizedL2L^\{2\}distance between evaluator\-influenced strategy weights and baseline \(task\-only\) weights\.γ=0\\gamma=0indicates zero evaluator influence;γ\>1\\gamma\>1indicates the evaluator’s effect exceeds the baseline strategy norm\.
- •H∈\[0,1\]H\\in\[0,1\]is the normalized Shannon entropy of the strategy weight distribution\.H=1H=1corresponds to a uniform distribution \(all strategies equally viable\);H=0H=0corresponds to strategy collapse\.
- •CV​\(N\)=std​\(γ^N\)/𝔼​\[γ^N\]\\text\{CV\}\(N\)=\\text\{std\}\(\\hat\{\\gamma\}\_\{N\}\)/\\mathbb\{E\}\[\\hat\{\\gamma\}\_\{N\}\]is the coefficient of variation of coupling estimates at sample sizeNN, measuring small\-sample reliability\.CV​\(N\)≪1\\text\{CV\}\(N\)\\ll 1indicates stable estimates;CV​\(N\)≫1\\text\{CV\}\(N\)\\gg 1indicates noise\-dominated estimates\.

The trade\-off mechanism is evaluator\-induced strategy concentration: stronger evaluator preferences \(γ↑\\gamma\\uparrow\) suppress strategy diversity \(H↓H\\downarrow\), which in turn reduces across\-seed variance and improves measurement reliability \(CV↓\\text\{CV\}\\downarrow\)\. The cost of unbiased evaluation \(γ≈0\\gamma\\approx 0\) is high strategy diversity \(H≈1H\\approx 1\) and consequently high measurement noise\.

The original evidence for this trade\-off came fromn=5n\{=\}5conditions with complete\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)metricsAnonymous \([2026a](https://arxiv.org/html/2607.00304#bib.bib1)\)\. While the correlations were strong \(r​\(H,γ\)=−0\.987r\(H,\\gamma\)=\-0\.987\), five conditions are insufficient to characterize the shape of the empirical frontier or assess generality across evaluator models and protocols\.

This paper extends the empirical base\. We survey all 11 evaluator–agent conditions from the multi\-experiment dataset ofAnonymous \([2026b](https://arxiv.org/html/2607.00304#bib.bib2)\), spanning four evaluator models \(GPT\-4o, DeepSeek\-V3, Qwen\-3\.7, Claude\-3\.5\), three executor models, and two experimental protocols\. We compute standardized\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)metrics for each condition, identify the empirical Pareto frontier, and characterize three distinct regimes in the trade\-off space\. We release all per\-condition data as a benchmark for evaluator comparison\.

## 2Methods

### 2\.1Data Source and Metric Computation

We draw on the full dataset ofAnonymous \([2026b](https://arxiv.org/html/2607.00304#bib.bib2)\), which contains per\-seed strategy weight vectors and coupling coefficients for 11 evaluator–agent conditions, withN=5N=5–30 seeds per condition\. Each seed executed 30 rounds of Test\-Time Reinforcement Learning \(TTRL\) across 16 tasks \(8 text, 8 visual\) usingn=11n=11candidate strategies\.

For each condition, we compute:

- •γ\\gamma: mean of per\-seed coupling coefficients \(γTV\\gamma\_\{\\text\{TV\}\}orgTVg\_\{\\text\{TV\}\}depending on data format\)\.
- •HH: mean normalized Shannon entropy of per\-seed baseline \(task\-only\) strategy weight vectors\. Conditions lacking weight vectors are marked as missing\.
- •CV​\(N=5\)\\text\{CV\}\(N\{=\}5\): bootstrap coefficient of variation \(5,000 resamples\) ofγ\\gammaestimates at sample size 5\. Conditions withN<5N<5seeds are marked as missing\.

The full analysis pipeline is provided in the supplementary material \(triangle\_verification\.pyfromAnonymous \([2026a](https://arxiv.org/html/2607.00304#bib.bib1)\)\)\.

### 2\.2Caveat: GPT\-4o Conditions

Four conditions using GPT\-4o as evaluator \(June 2026 API version\) produceγ=0\.000\\gamma=0\.000andH=1\.000H=1\.000for all seeds—a pattern consistent with the version drift documented inAnonymous \([2026b](https://arxiv.org/html/2607.00304#bib.bib2)\), where GPT\-4o’s evaluator behavior changed substantially between May and June 2026\. The uniform weights \(H=1\.0H=1\.0with zero variance\) suggest that the current GPT\-4o API exerts negligible evaluator influence—its judgments are either absent or orthogonal to the agent’s strategy distribution\. We exclude these four conditions from the primaryHH–γ\\gammacorrelation analysis \(where they would artifactually inflate the correlation by clustering at the origin\) but retain them in the full condition table for completeness\.

## 3Results

### 3\.1Condition Survey

Table[1](https://arxiv.org/html/2607.00304#S3.T1)presents all 11 conditions\. Five provide the complete\(γ,H,CV\)\(\\gamma,H,\\text\{CV\}\)triple; two additional conditions provideγ\\gammaand CV \(but lack weight vectors forHH\); four GPT\-4o conditions provideγ\\gammaandHH\(but theHHvalues are artifactual\)\.

Table 1:Complete condition survey\.†\\daggerGPT\-4o conditions excluded from primary analysis \(see §[2\.2](https://arxiv.org/html/2607.00304#S2.SS2)\)\.‡\\ddaggerWeight vectors not available for entropy computation\.
### 3\.2The Empirical Frontier

Figure[1](https://arxiv.org/html/2607.00304#S3.F1)maps the five conditions with complete\(γ,CV\)\(\\gamma,\\text\{CV\}\)metrics\. Despite the limited sample, a clear structure emerges:

![Refer to caption](https://arxiv.org/html/2607.00304v1/x1.png)Figure 1:The empirical evaluation frontier\. Points show five conditions with complete\(γ,CV\)\(\\gamma,\\text\{CV\}\)metrics\. Color indicates strategy entropyHH\. The red\-shaded region \(lowγ\\gamma, low CV\) is empirically empty\.Low\-coupling regime\(γ<0\.2\\gamma<0\.2\)\. DS self\-eval \(γ=0\.033\\gamma=0\.033, CV = 2\.42\) occupies the “unbiased, unreliable” corner\. With near\-zero evaluator coupling, measurement noise is extreme: the standard deviation ofγ\\gammaestimates atN=5N\{=\}5is more than twice the mean\.

High\-coupling regime\(γ\>0\.9\\gamma\>0\.9\)\. DS self\-eval r30, Ablation max, and Qwen 3\.7 cluster at highγ\\gamma\(0\.94–1\.06\) and low CV \(0\.08–0\.16\)\. These conditions produce stable rankings—CV\(N=5\)<0\.16\(N\{=\}5\)<0\.16in all cases—but the rankings primarily reflect evaluator preferences\.

Intermediate regime\.Only DS×\\timesQwen \(γ=0\.187\\gamma=0\.187, CV = 1\.025\) occupies the transition zone between the two clusters\. This regime is severely undersampled\.

Empty region\.The region\{γ<0\.2,CV​\(N=5\)<0\.3\}\\\{\\gamma<0\.2,\\text\{CV\}\(N\{=\}5\)<0\.3\\\}is empty\. No evaluator–agent pair in our sample achieves both low bias and high reliability atN=5N\{=\}5\.

### 3\.3Strategy Entropy Gradient

Among the five conditions with validHHmeasurements \(excluding GPT\-4o artifacts\), entropy decreases with coupling:r​\(H,γ\)=−0\.989r\(H,\\gamma\)=\-0\.989\(p=0\.001p=0\.001,n=5n=5\)\. The DS self\-eval condition exhibits near\-maximal entropy \(H=0\.992H=0\.992\) under minimal coupling, while Ablation max shows substantially reduced entropy \(H=0\.753H=0\.753\) under strong coupling \(γ=1\.038\\gamma=1\.038\)\. The Ablation no\-S0 condition \(γ=0\.979\\gamma=0\.979,H=0\.788H=0\.788\), with onlyN=5N\{=\}5seeds, provides an additional data point consistent with the trend\.

## 4Discussion

The missing middle\.The empirical frontier is bimodal: conditions cluster at either very low or very highγ\\gamma, with a sparsely sampled intermediate regime\. This reflects current experimental practice—self\-evaluation \(γ≈0\\gamma\\approx 0\) and strong external evaluation \(γ\>0\.9\\gamma\>0\.9\) are the dominant paradigms\. Deliberately designing evaluators with intermediate coupling \(e\.g\., weak evaluators, ensemble evaluators with partial bias cancellation\) would populate this regime and enable more precise characterization of the trade\-off curve\.

GPT\-4o version drift\.The four GPT\-4o conditions all exhibitγ=0\.000\\gamma=0\.000andH=1\.000H=1\.000—the evaluator exerts zero measurable influence on the agent’s strategy distribution\. This is consistent with the version drift documented inAnonymous \([2026b](https://arxiv.org/html/2607.00304#bib.bib2)\): GPT\-4o’s May 2026 version showed strong coupling \(γ≈1\.176\\gamma\\approx 1\.176\), while the June 2026 version shows none\. From the perspective of the trade\-off, this positions GPT\-4o as simultaneously the most “unbiased” and the least “reliable” evaluator—its rankings are uncorrelated with agent strategy, providing no signal for evaluation\.

Limitations\.Our survey has three principal limitations\. First, all conditions come from a single research group’s experiments, limiting generality\. Independent replication with different models, tasks, and protocols is needed\. Second, the sample size of five conditions with complete metrics is insufficient for reliable estimation of the trade\-off curve’s functional form\. Third, the GPT\-4o conditions produce degenerate metrics \(γ=0\\gamma=0,H=1H=1\) that may reflect API version artifacts rather than genuine evaluator behavior; these conditions should be re\-measured with a stable API version or alternative evaluator models\.

Benchmark release\.We release all per\-condition metrics as a standardized JSON dataset \(p16\_data\.jsonin supplementary material\)\. Each entry contains the condition name,γ\\gammamean and standard deviation,HHmean, standard deviation, and range, CV\(N=5\)\(N\{=\}5\), and number of seeds\. We encourage the community to contribute additional evaluator–agent conditions to this benchmark using the standardized pipeline, following the model of multi\-metric LLM evaluation established byLiang et al\. \([2023](https://arxiv.org/html/2607.00304#bib.bib4)\)\.

## 5Conclusion

An 11\-condition empirical survey of the bias\-reliability tradeoff confirms that evaluator coupling \(γ\\gamma\) and measurement reliability \(CV\) are inversely related across diverse evaluator–agent pairs, withr​\(H,γ\)=−0\.989r\(H,\\gamma\)=\-0\.989\(n=5n=5complete conditions\)\. The data reveal a bimodal empirical frontier—self\-evaluation at the low\-γ\\gamma, high\-CV extreme, strong external evaluation at the high\-γ\\gamma, low\-CV extreme—with a sparsely sampled intermediate regime\. GPT\-4o’s June 2026 version exhibits zero measurable evaluator coupling, consistent with documented version drift\. All data are released as a public benchmark\.

## Broader Impact Statement

This paper characterizes evaluator behavior using quantitative metrics\. The framework could be misused to justify biased evaluation \(“highγ\\gammais acceptable because it improves reliability”\), which we explicitly caution against: the trade\-off should motivate larger sample sizes for unbiased evaluators, not acceptance of bias\. The benchmark dataset may be used to compare evaluator models; such comparisons should account for API version effects \(as demonstrated by the GPT\-4o drift\) and not be treated as stable over time\.

## Reproducibility Statement

All data are drawn from the publicly available dataset ofAnonymous \([2026b](https://arxiv.org/html/2607.00304#bib.bib2)\)\. The analysis pipeline \(triangle\_verification\.py\) fromAnonymous \([2026a](https://arxiv.org/html/2607.00304#bib.bib1)\)is included in the supplementary material\. The per\-condition benchmark dataset \(p16\_data\.json\) is provided in machine\-readable JSON format\.

## References

- Anonymous \(2026a\)Anonymous\.The Bias\-Reliability Tradeoff in LLM Evaluation: A Conjectured Impossibility Triangle\.TMLR submission, 2026\.
- Anonymous \(2026b\)Anonymous\.A Diagnostic Framework and Multi\-Evaluator Audit of Evaluator\-Driven Preference Dynamics\.TMLR submission, 2026\.
- Anonymous \(2026c\)Anonymous\.N\-Sensitivity: Small\-Sample Measurement Instability as a General Property of Complex Evaluation Systems\.TMLR submission, 2026\.
- Liang et al\. \(2023\)P\. Liang, R\. Bommasani, T\. Lee, et al\.Holistic Evaluation of Language Models\.TMLR, 2023\.
- Zheng et al\. \(2023\)L\. Zheng, W\.\-L\. Chiang, Y\. Sheng, et al\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.NeurIPS, 2023\.