llm-judges

A practical guide on writing effective evaluations for AI agents, focusing on starting from observed failures and using a mix of deterministic checks and LLM judges.

This paper introduces a benchmark for evaluating LLM judges' ability to detect omissions in AI-generated clinical notes, finding that standard judges struggle with omissions but restructuring the task into per-fact verification improves detection.

SAGE is a novel evaluation framework for task-oriented dialogue agents that grounds assessments in dialogue state changes and uses abstention to provide cost-effective, accurate turn-level judgments without expensive LLM calls.

This paper examines cultural alignment in AI-generated multimodal stories across five African communities through community-grounded evaluation, developing a taxonomy of misalignment and assessing the reliability of automated judges.

JuryProbe introduces an empirical diagnostic for assessing consensus risk in reference-free LLM judge panels for factuality checking, and uses a calibration-based routing policy to ground high-risk decisions with trusted references, reducing false accepts.

This talk by Will Brown of Primordial AI discusses techniques for scaling Reinforcement Learning to complex, real-world tasks where rewards are not verifiable, using methods like anchoring, LLM judges, and simulation.

Alexander Yue introduces a new browser-use benchmark where Opus 5 and GPT-5.6 Sol show similar performance, emphasizing the benchmark's robust design with verified rubrics for LLM judges.

The paper presents EvalCEGAR, a method for automatically evolving evaluation metrics using a pool of Python operators that flag specific defects in AI outputs, improving accuracy over hand-written operators and LLM judges.

This paper investigates bidirectional bias in LLM judges induced by self- and other-labels, showing that labels alone can shift evaluation scores regardless of actual source, with contributions to understanding authorship attribution and controlled evaluation tasks.

This paper addresses rubric interference in LLM judges when evaluating multiple rubrics simultaneously and proposes Self-Anchored Rubric Alignment (SARA) to improve consistency through on-policy self-distillation.

Introduces the Wiggle Framework, a unified stress test for the epistemic stability of LLM judges, measuring mechanical consistency, single-turn conviction, and multi-turn persistence across 14 tasks. Finds that all 9 studied frontier models flip verdicts substantially under pressure, and that successful pressure is usually net-corrupting relative to ground truth.

The paper introduces a behavioral alignment framework for personalized LLM judges in recommendation evaluation, addressing bidirectional rationalization where off-the-shelf LLMs argue both for and against user engagement on the same item. Their fine-tuned and preference-optimized approach achieves a 32.19% Macro-F1 lift over zero-shot and matches production feature-engineered baselines.

This paper introduces MobileJudgeBench, a benchmark with 931 human-annotated trajectories for systematically evaluating LLM-based judges on mobile agent tasks. It finds that simple baseline judges with sampled screenshots rival purpose-built methods, with the LLM backbone being the primary driver of quality.

A tweet discussing a discovered quirk where renaming a paper PDF to a longer, positive title improves LLM judge scores, advising caution with score-based LLM evaluation and recommending binary labels instead.

This paper from Stony Brook University identifies 'Averaging Bias' in human faithfulness annotations for text summarization: global human labels correlate better with the average of per-sentence LLM judgments than with a strict conjunctive rule, meaning humans often label summaries as faithful even when they contain local factual errors.

A research paper introducing Chain-of-Models (CoM), an automated pipeline where a second LLM audits a first model's reasoning trace to correct cognitive biases. It finds that auditor effectiveness depends on model family and bias type, and proposes a bias-specific auditor selection rule that improves judgment accuracy.

A six-step guide to building evaluation gates that let AI agents merge changes autonomously, covering judge bias, runtime evals, trajectory grading, and more.

This paper shows that LLM judges tend to over-credit incorrect answers when no reference answer is provided, and adding a reference can flip verdicts by up to 85%, aligning more with human judgments. The authors propose calibration steps for using LLM judges in reference-free settings.

A blog post from Parlance Labs tests automated AI evaluation tools (Braintrust Loop, Arize Alyx, LangSmith Engine) on real production data, finding they catch 87% of issues humans flag but miss domain-specific failures and add noise, recommending iterative human-in-the-loop use.

Prime Intellect engineers demonstrated a method to train reasoning models in 30 minutes using distributed RL over the open internet, utilizing Prime-RL, LLM judges, and multi-cloud GPUs, enabling open models to compete with closed labs without owning data centers.