llm-as-judge

Introduces PAJAMA, a hybrid evaluation system that improves upon the LLM-as-a-judge approach by extracting rubrics and executing them programmatically, pushing the Pareto frontier of speed, cost, and transparency.

This paper introduces MamaBench and MamaRetrieval, two benchmarks for evaluating medical retrieval-augmented generation in maternal, neonatal, and reproductive health, addressing gaps in existing QA and retrieval datasets.

This paper introduces a signal-coverage matrix that decomposes type-correctness gains in autoformalization into four strata, revealing the mechanisms behind LLM refinements and showing that headline metrics can obscure which errors are actually resolved.

This paper investigates the assumption that setting LLM judge temperature to 0 ensures deterministic safety evaluations. It finds that in practice, many harnesses do not set temperature or seed, leading to high variance, and even with temperature=0, non-determinism persists due to provider-level randomness and API changes.

This paper evaluates the reliability of automated judges used to measure attack success rates (ASR) in LLM jailbreak research, finding that both safety classifiers and LLM-as-judges have significant calibration and adversarial robustness issues that undermine reported ASR numbers.

AdversaBench introduces an automated LLM red-teaming pipeline that uses five mutation operators and a three-judge panel with a meta-judge tiebreaker to confirm failures, revealing that attack difficulty varies by category and that adversarial prompts transfer from smaller to larger models.

Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.

This paper introduces HistoRAG, a framework that translates historiographical principles (decoupled retrieval/generation, temporal windowing, LLM-as-judge evaluation) into architectural interventions for standard RAG, applied to a corpus of 102,189 Der Spiegel articles to address interpretive rather than factual question-answering needs.

This paper presents a curriculum-grounded LLM-as-Judge pipeline for automated question-level marking in high-stakes exam preparation, using syllabus artefacts and marking guidelines to improve consistency and transparency, with preliminary evaluation showing outcomes comparable to human tutors.

A discussion on the challenges of testing AI agent harnesses with non-deterministic components, exploring approaches like golden output diffing and using an LLM as a judge, while questioning the validity of such methods.

Jugmax is a tool that evaluates AI agents by analyzing their full execution trajectory rather than just final outputs, identifying inefficiencies, errors, and wasted tokens. The founders offer free evaluation for two production agents to validate their product.

Proposes an anytime-valid attribution method that uses a human-labeled anchor set and a betting e-process to distinguish whether score drift in LLM evaluation pipelines comes from the system or the judge, resolving the ambiguity caused by silent judge changes.

The post outlines a future agent recipe for building scalable intelligence by fine-tuning efficient, specialized open models to surpass frontier performance on LLM-as-a-judge tasks, and applying this to extract signals from trace data for continual learning. LangChain Labs and FireworksAI release new work demonstrating this approach.

This paper proposes Judge-LS, a protocol to evaluate whether LLM-as-a-judge models are invariant to language switching between English and Chinese. It finds that switching languages causes 10.7-14.4% preference flips and that judges achieve their highest accuracy in English.

This paper studies the relationship between token-level log-probability distributions, LLM-as-judge rubric scores, and final task accuracy in multi-agent debate systems. It finds a consistent four-phase confidence trajectory and role asymmetry between Constructor and Auditor agents.

POLARIS is a training recipe using GRPO with LLM-as-judge rewards and human-reference injection to improve long-form story generation in small models. Applied to Qwen3.5-9B, the resulting POLARIS-9B model matches Qwen3.5-27B performance on creative writing benchmarks while better adhering to length instructions.

AICompanionBench introduces the first publicly available benchmark dataset of 2,123 real-world AI companion conversations annotated across nine safety risk categories, used to evaluate 20 LLMs as safety judges. Results show strong models handle explicit harmful content well but struggle with nuanced risks like manipulation and false positives on benign conversations.

This paper geometrically analyzes why LLMs acting as judges agree strongly with each other but weakly with humans, finding that inter-LLM consensus reflects a collapsed subspace rather than true human alignment on subjective rubrics. Post-hoc calibration on human data improves alignment, but even calibrated LLMs fall short of human reliability.

This paper presents an empirical study adapting the small language model Phi Silica for short-form text rewriting through dataset curation, prompt distillation, and parameter-efficient fine-tuning, showing that targeted adaptation significantly improves semantic fidelity and reduces hallucinations.

GLIDE is an open-source Python library that unifies state-of-the-art Prediction-Powered Inference methods for debiased evaluation of generative AI and agentic systems, enabling annotation savings with valid uncertainty estimates.