Tag
This paper introduces TheraJudge and TheraAgent, a framework that uses multi-dimensional human-aligned evaluation to improve therapeutic response generation in LLMs, showing significant gains in quality and safety.
Details an approach to train a small LLM judge for evaluating agent outputs, replacing costly frontier models, with a Claude Code plugin for deployment.
This paper introduces approach-level diversity for LLM math reasoning, showing that surface-level diversity metrics are unreliable proxies and that directly optimizing for approach diversity remains an open problem.
Hamel Husain shares flashcards and insights from an AI evaluation course, advocating for binary judges over Likert scales for practical LLM evaluation.
This paper introduces 'second-order bias', the bias LLMs exhibit when judging biased content, and proposes a reasoning task grounded in epistemic entitlement to evaluate it. Experiments show that the task evades safety guardrails and reveals systematic demographic biases in LLM judges.
Personal lessons on evaluating AI agents in production, including mapping symptoms to layers, using trajectory evaluation, calibrating LLM judges, converting failures to test cases, and performing adversarial testing.
This paper introduces a psychometric datasheet protocol for evaluating LLM judges as measurement instruments, measuring dark current, positional false preference, stable cross-sensitivity, and target sensitivity. A case study on three open-weight models reveals significant differences in judge quality and behavior.
RealMath-Eval is a benchmark of 224 real-world high school math exam responses that reveals a significant 'Evaluation Gap': state-of-the-art LLM judges perform poorly on authentic human reasoning (MSE ~2.96) compared to synthetic LLM-generated solutions (MSE ~1.17), due to higher diversity and surprisal in human error patterns.
Aigon is an open-source tool that runs multiple AI coding agents in parallel on the same feature specified in a markdown spec and uses an LLM judge to select the best implementation, with a visual dashboard and optional scheduling.
A detailed evaluation of a RAG customer support chatbot reveals that retrieval issues often masquerade as LLM problems, heuristic evaluators are misleading, deduplication improves quality, stricter grounding trades helpfulness for accuracy, and model sweeping can dramatically reduce cost while improving performance.
Brex open-sources CrabTrap, an LLM-as-a-judge HTTP proxy that filters and secures AI agent traffic before it reaches production services.
A curated list of 11 links shared daily to help people learn AI evaluation techniques, covering evals, observability, LLM-as-judge, and agent evaluation.