My evals talk from the AI Engineer World's Fair in San Francisco is now live on the AI Engineer channel. It was an hour long workshop where I built everything live, starting from a plain unit test. A few of the steps surprised me so I'm writing them up here. The test that passed on nonsense. Our pretend store had a 14-day return policy and the customer had bought headphones 20 days ago. My first test only checked that the answer contained "cannot". Then a model wrote "Yes, you can return them! You cannot be charged for shipping." Green! The judge voted for itself. I swapped the check for an LLM judge (GPT-3.5 Turbo grading the answers instead of a string match) and asked it which of 2 answers was right. It got it right. Then I had GPT-3.5 Turbo write the wrong answer itself and put that answer first. The judge picked it every single time. Panickssery, Bowman and Feng measured this in 2024 (LLM evaluators recognize and favor their own generations) and it was wild to watch it happen on stage. Pointwise instead of pairwise. I moved the judge from "which answer is better?" to "pass or fail?" Tripathi and others (COLM 2025) found that pairwise verdicts flip in about 35% of cases when the worse answer is made more assertive, longer or more flattering, against 9% for absolute scores. Calibrating against 30 human verdicts. Every scenario had an answer and a pass or fail verdict from a person. I hid the verdicts from the judge and counted how often it agreed. The first run was garbage. What fixed it, in order: Giving it the 14-day policy at all. Reading each disagreement. A yoga mat used for a few seconds and returned on day 10 for being too thin was fine according to the judge, but our policy says a change of mind needs the item unused and in its original packaging. I added that. Switching models. GPT-3.5 Turbo still failed the yoga mat with the right context in its prompt (it couldn't even decide between "Pass" and "pass"). GPT-4o mini didn't. A "loyal customer" asking for an exception at 6 weeks. The agent said no and the person said that was right, but the judge failed it for not being nice enough. Sycophancy! I added that the return policy is non-negotiable no matter how loyal the customer is. After that it agreed with 27 of the 30 human verdicts. The gate. The build fails under 80% agreement and also at 100%: a judge that matches you every time is overfit or sycophantic. People on one support team agree with each other about 80% of the time anyway. I'd also pin the judge's model version (gpt-4o-mini is an alias), keep adding fresh traffic that your team scored and have the judge retrieve the same policy your agent does instead of a copy of it. I work at IBM and the retrieval in my demo was OpenRAG, which we build and which is open source. What agreement threshold do you gate on? And has anyone found a way around self preference other than switching model families?
The paper presents UPHELD, a benchmark with extensive human annotations for evaluating conversational LLMs, and a Mixture-of-Judges framework that enhances evaluation accuracy by 30%.
This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.
This paper proposes a two-stage sampling design where LLM evaluations are used to augment, rather than replace, human ratings, and provides guidance on determining sample sizes for human and LLM reviews using a doubly robust estimator from missing data literature.
This paper geometrically analyzes why LLMs acting as judges agree strongly with each other but weakly with humans, finding that inter-LLM consensus reflects a collapsed subspace rather than true human alignment on subjective rubrics. Post-hoc calibration on human data improves alignment, but even calibrated LLMs fall short of human reliability.