@HamelHusain: Yes! binary judges are far more practical for most people, because likert scales (or scores) have too many footguns All…
Summary
Hamel Husain shares flashcards and insights from an AI evaluation course, advocating for binary judges over Likert scales for practical LLM evaluation.
View Cached Full Text
Cached at: 06/29/26, 04:23 AM
Yes! binary judges are far more practical for most people, because likert scales (or scores) have too many footguns
All the flashcards are here (inspired by @chrisalbon ‘s flashcards) https://t.co/qfB4WJgX5n https://t.co/OvSdVi5rbB
AI Eval Flashcards | Hamel Husain & Shreya Shankar on Maven
Source: https://maven.com/parlance-labs/o/540bd8 Digital asset
Hamel Husain
ML Engineer with 20 years of experience
Shreya Shankar
ML Systems & Applied AI Evals Researcher
See all products fromHamel Husain & Shreya Shankar
.png&w=1536&q=75)
12 pages with visual bite-sized takeaways of the most important learnings from our course.
elvis (@omarsar0): If you use LLM-as-judge, this one is worth reading.
(bookmark it)
It’s actually one of the most effective ways to use LLM-as-a-Judge for evals.
Holistic judge scores hide both their reasoning and their ceiling effects.
BINEVAL decomposes each evaluation criterion into atomic
Similar Articles
@omarsar0: If you use LLM-as-judge, this one is worth reading. (bookmark it) It's actually one of the most effective ways to use L…
BinEval is a new framework that decomposes LLM evaluation criteria into atomic binary questions, improving interpretability and enabling targeted prompt optimization, achieving strong results on factual consistency benchmarks.
@omarsar0: LLM review weirdness indeed. Avoid using scores with LLM judges, or be extremely careful if you do. Use binary labels w…
A tweet discussing a discovered quirk where renaming a paper PDF to a longer, positive title improves LLM judge scores, advising caution with score-based LLM evaluation and recommending binary labels instead.
LLM-as-judge anchored on one confidence value in 10 of 16 evals. Asking for a label fixed it.
An LLM judge consistently returned a fixed confidence score of 0.72 in evaluations, but switching to categorical labels improved score distribution, showing that models are better at classification than numerical estimation for assessments.
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
This paper investigates the run-to-run reliability of LLM-as-a-Judge evaluations, finding that pairwise preferences flip 13.6% of the time on average, with significant first-position bias in GPT-4o-mini, and recommends multi-trial aggregation and position randomization.
Binarization Flattens the Score Space
This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.