llm-judge

This position paper argues that language model evaluation scores should be treated as perishable knowledge claims, not ground truth, and proposes explicit metadata such as formality tier, scope declaration, and expiration date to counter 'trust inflation' caused by averaging weak and strong signals.

This article explains how Similarweb evaluates long-form agent research reports using LangSmith, combining deterministic checks for tool calls and LLM-as-judge scoring for quality, with a focus on making regressions inspectable and enabling A/B comparisons.

Joseph Decker evaluates 16 AI models on truthfulness for his product Condensr, discovers that the leaderboard winner fabricated content five times in an audit, and instead ships the second-place model which had zero fabrications. The post details the evaluation process, a bug in the LLM judge that penalized accurate summaries due to truncated transcripts, and the importance of custom evals over generic benchmarks.

SIFT introduces a self-improving document classifier that uses a cheap SPLADE-LightGBM pipeline and an LLM judge to continuously teach itself, while a frozen-gate safety mechanism prevents silent regression, enabling autonomous retraining without human labeling overhead.

This paper extends missing-information stress-testing to open-ended medical conversation, finding that LLM judge choice materially changes apparent safety and that LLM judges are more permissive than clinicians.

EduPanel is a rubric-grounded, learner-conditioned LLM judge that uses three specialized agents to evaluate teaching video quality, achieving reliability comparable to human experts and improving scoring accuracy while maintaining detectability of unreliable outputs.

This paper introduces Kahneman4Review, a benchmark of 3,563 peer reviews rated along nine theoretically motivated textual dimensions, eight bias diagnostics, and a continuous reasoning-quality score, to evaluate whether LLM judges can distinguish analytical form from genuine epistemic quality in peer review.

CAFE is an open-source platform that applies design-of-experiments principles to evaluate compound AI systems, attributing answer quality variance to components and their interactions using factorial designs and mixed-effects models.

This paper benchmarks 8 LLM judges for citation quality in deep-research systems, finding that cheaper models remain competitive with frontier models on source relevance and factual support, but differ in directional bias which matters for RL training.

This paper studies which signals best predict correctness in text-to-SQL for selective prediction. It finds that verification-based signals from LLM judges outperform black-box statistical signals like self-consistency, and that a two-provider ensemble achieves 0.82 AUROC with well-calibrated probabilities.

This paper introduces TheraJudge and TheraAgent, a framework that uses multi-dimensional human-aligned evaluation to improve therapeutic response generation in LLMs, showing significant gains in quality and safety.

Details an approach to train a small LLM judge for evaluating agent outputs, replacing costly frontier models, with a Claude Code plugin for deployment.

This paper introduces approach-level diversity for LLM math reasoning, showing that surface-level diversity metrics are unreliable proxies and that directly optimizing for approach diversity remains an open problem.

Hamel Husain shares flashcards and insights from an AI evaluation course, advocating for binary judges over Likert scales for practical LLM evaluation.

This paper introduces 'second-order bias', the bias LLMs exhibit when judging biased content, and proposes a reasoning task grounded in epistemic entitlement to evaluate it. Experiments show that the task evades safety guardrails and reveals systematic demographic biases in LLM judges.

Personal lessons on evaluating AI agents in production, including mapping symptoms to layers, using trajectory evaluation, calibrating LLM judges, converting failures to test cases, and performing adversarial testing.

This paper introduces a psychometric datasheet protocol for evaluating LLM judges as measurement instruments, measuring dark current, positional false preference, stable cross-sensitivity, and target sensitivity. A case study on three open-weight models reveals significant differences in judge quality and behavior.

RealMath-Eval is a benchmark of 224 real-world high school math exam responses that reveals a significant 'Evaluation Gap': state-of-the-art LLM judges perform poorly on authentic human reasoning (MSE ~2.96) compared to synthetic LLM-generated solutions (MSE ~1.17), due to higher diversity and surprisal in human error patterns.

Aigon is an open-source tool that runs multiple AI coding agents in parallel on the same feature specified in a markdown spec and uses an LLM judge to select the best implementation, with a visual dashboard and optional scheduling.

A detailed evaluation of a RAG customer support chatbot reveals that retrieval issues often masquerade as LLM problems, heuristic evaluators are misleading, deduplication improves quality, stricter grounding trades helpfulness for accuracy, and model sweeping can dramatically reduce cost while improving performance.