An LLM judge consistently returned a fixed confidence score of 0.72 in evaluations, but switching to categorical labels improved score distribution, showing that models are better at classification than numerical estimation for assessments.
Been running an eval setup where two LLM outputs get compared by a third model, and hit a failure mode I haven't seen written up much. Sharing the numbers because the fix was counterintuitive. Setup: two agents answer the same prompt, a judge model sees both answers anonymized in randomized order, returns a winner + a confidence score 0-1. Standard pairwise LLM-as-judge stuff. The problem: first 16 evals came back with 10 of them at exactly 0.72. Not clustered around it. The identical number, whether one answer was clearly stronger or they were near-indistinguishable. The score was carrying zero information. What didn't work: I gave it explicit anchors in the prompt 0.50-0.65 near-identical quality 0.66-0.80 a real but modest edge 0.81-0.92 clearly better on the criteria 0.93-1.00 one answer failed the task Use the full range. Still 0.72. Told it directly that returning the same number made the score meaningless. Still 0.72. What worked: stopped asking for a number. Asked for a label instead coin_flip | slight | clear | decisive — and mapped label→score in code. Naming a category is classification. Estimating a probability is not, and models are noticeably worse at the second. Over the next 44 evals the distribution actually spread across the range instead of piling on one value. Second thing, same lesson. I told the judge to return coin_flip whenever both answers reached the same conclusion (both solved the puzzle, both picked the same number). It kept returning slight while its own written summary said "both reach the correct solution, but B presents it more clearly." It recognizes the convergence and rewards presentation anyway. Couldn't prompt my way out of that one either. Ended up asking for a boolean (same_conclusion: true/false) and doing the downgrade in code. Same principle: ask the model to classify, decide in code. Third thing I'm less sure about, posting in case someone has data. Broke confidence down by task type across 60 evals: creativity 0.80 ← highest persuasion 0.76 logic 0.71 prediction 0.69 strategy 0.68 negotiation 0.67 The judge is most decisive on the most subjective category. My read is that it's rewarding concrete, quantified language over evocative language, and creative prompts produce the widest spread between those two styles so the gap looks bigger to it. Anecdote that made me suspect this: prompt was "describe the sound of a place you've never been, so precisely that it becomes real." One answer did prose about a souk at dawn, pigeons, silk, the muezzin call. The other wrote "cicadas at 85-90 decibels, layered; a lion's rumble travels through ground vibration before reaching ears." Judge picked the decibels, reasoning that specific acoustic detail beat evocative language given the prompt said precisely. Defensible! But it's one data point and n=13 on creativity is nothing. If anyone's measured judge confidence by task type I'd like to compare. TL;DR — if your LLM judge returns suspiciously stable scores, check the actual distribution before trusting it. Numeric self-assessment is where I'd look first, and swapping it for a categorical label plus code-side mapping is a cheap fix.
This paper introduces a margin-based confidence ranking method for LLM-as-a-judge systems, learning a dedicated estimator to ensure monotonicity between confidence and human-disagreement risk, with generalization guarantees and improved ranking accuracy across datasets.
This article benchmarks various blackbox and whitebox methods for LLM confidence estimation, including verbalized confidence, linguistic uncertainty, reasoning-length, P(Answer), P(True), and self-consistency, comparing their effectiveness for tasks like active learning and safety classification.
This paper audits the reliability of LLM-as-judge evaluation by showing that changing the evaluator model can shift scores even when candidate responses are fixed, and it examines scaling and upgrade paths for Qwen3 and MiniMax models, concluding that judge upgrades are not interchangeable and proposing best practices for reporting.
This paper proposes a risk-controlled framework for using LLMs as judges in factual evaluation, calibrating uncertainty thresholds to maintain a user-specified error rate and routing to retrieval-augmented mode when needed, achieving higher coverage with provable reliability guarantees.
This paper investigates the run-to-run reliability of LLM-as-a-Judge evaluations, finding that pairwise preferences flip 13.6% of the time on average, with significant first-position bias in GPT-4o-mini, and recommends multi-trial aggregation and position randomization.