The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

arXiv cs.CL Papers
llm-as-judge human-alignment inter-llm-consensus geometric-analysis indic-languages calibration evaluation-methods

Summary

This paper geometrically analyzes why LLMs acting as judges agree strongly with each other but weakly with humans, finding that inter-LLM consensus reflects a collapsed subspace rather than true human alignment on subjective rubrics. Post-hoc calibration on human data improves alignment, but even calibrated LLMs fall short of human reliability.

arXiv:2606.03043v1 Announce Type: new Abstract: LMs-as-judges are now standard, yet judges agree strongly with one another while agreeing only weakly with humans. We test whether this reflects shared signal or shared bias by measuring four geometric quantities on the standard LLM-as-judge stack across four community-built Indic datasets, eight Indic languages, and 41 LLM judges: score spread, effective rank, principal angle to the human subspace, and stacked correlations among judges and humans, all with bootstrap confidence intervals. On subjective rubrics, judges use less than half the human score range ($\sigma_J / \sigma_H \approx 0.3$--$0.5$). Their evaluation axis is nearly orthogonal to the human one and noticeably further from humans than humans are from each other ($87^\circ$--$89^\circ$ versus $78^\circ$--$81^\circ$). Inter-LLM agreement exceeds LLM--human agreement ($r_{LL} \approx 0.35$ versus $r_{LH} \approx 0.27$--$0.32$). On a rubric with a verifiable factual answer, the same diagnostics fall back into the human range (axis $58.5^\circ$; $r_{LH} = 0.519$). Fine-tuning and preference optimization recover spread ($0.32 \rightarrow 1.08$) but barely move the axis (still $87^\circ$--$88^\circ$). Only post-hoc calibration on a small human-anchored set improves all four community-health rubrics together, placing a calibrated 24B Indic judge ($r = 0.184$) ahead of GPT-5.5 ($r = 0.123$), yet still short of human reliability (human-human $r = 0.474$ on the verifiable rubric). We argue that inter-LLM agreement should be considered evidence of human alignment only when a direct geometric check on the judge's score subspace passes; otherwise, the consensus reflects agreement within a collapsed subspace.
Original Article
View Cached Full Text

Cached at: 06/03/26, 09:36 AM 

# The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
Source: [https://arxiv.org/abs/2606.03043](https://arxiv.org/abs/2606.03043)
[View PDF](https://arxiv.org/pdf/2606.03043)

> Abstract:LMs\-as\-judges are now standard, yet judges agree strongly with one another while agreeing only weakly with humans\. We test whether this reflects shared signal or shared bias by measuring four geometric quantities on the standard LLM\-as\-judge stack across four community\-built Indic datasets, eight Indic languages, and 41 LLM judges: score spread, effective rank, principal angle to the human subspace, and stacked correlations among judges and humans, all with bootstrap confidence intervals\. On subjective rubrics, judges use less than half the human score range \($\\sigma\_J / \\sigma\_H \\approx 0\.3$\-\-$0\.5$\)\. Their evaluation axis is nearly orthogonal to the human one and noticeably further from humans than humans are from each other \($87^\\circ$\-\-$89^\\circ$ versus $78^\\circ$\-\-$81^\\circ$\)\. Inter\-LLM agreement exceeds LLM\-\-human agreement \($r\_\{LL\} \\approx 0\.35$ versus $r\_\{LH\} \\approx 0\.27$\-\-$0\.32$\)\. On a rubric with a verifiable factual answer, the same diagnostics fall back into the human range \(axis $58\.5^\\circ$; $r\_\{LH\} = 0\.519$\)\. Fine\-tuning and preference optimization recover spread \($0\.32 \\rightarrow 1\.08$\) but barely move the axis \(still $87^\\circ$\-\-$88^\\circ$\)\. Only post\-hoc calibration on a small human\-anchored set improves all four community\-health rubrics together, placing a calibrated 24B Indic judge \($r = 0\.184$\) ahead of GPT\-5\.5 \($r = 0\.123$\), yet still short of human reliability \(human\-human $r = 0\.474$ on the verifiable rubric\)\. We argue that inter\-LLM agreement should be considered evidence of human alignment only when a direct geometric check on the judge's score subspace passes; otherwise, the consensus reflects agreement within a collapsed subspace\.

## Submission history

From: Sourabrata Mukherjee \[[view email](https://arxiv.org/show-email/5aa1a3b3/2606.03043)\] **\[v1\]**Tue, 2 Jun 2026 02:26:18 UTC \(1,384 KB\)

Similar Articles

Review Arcade: On the Human Alignment and Gameability of LLM Reviews

Hugging Face Daily Papers

This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.

Re-Centering Humans in LLM Personalization

Hugging Face Daily Papers

This paper investigates the effectiveness of LLM personalization by putting real humans back into the evaluation loop, revealing systematic gaps between human judgments and LLM outputs at every stage of the personalization pipeline, and highlighting the limitations of synthetic data and LLM judges.

LLM Judges Can Be Too Generous When There Is No Reference Answer

arXiv cs.CL

This paper shows that LLM judges tend to over-credit incorrect answers when no reference answer is provided, and adding a reference can flip verdicts by up to 85%, aligning more with human judgments. The authors propose calibration steps for using LLM judges in reference-free settings.

Review Arcade: On the Human Alignment and Gameability of LLM Reviews

arXiv cs.AI

This paper empirically evaluates the alignment between LLM-generated and human reviews for scientific papers, finding limited and variable alignment. It also shows that authors can 'game' LLM reviews by iteratively revising papers to improve scores, with up to 35% of papers seeing statistically significant score increases.