Tag
作者在 AI Engineer World's Fair 上的 evals 演讲总结，记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致：包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。
This paper investigates bidirectional bias in LLM judges induced by self- and other-labels, showing that labels alone can shift evaluation scores regardless of actual source, with contributions to understanding authorship attribution and controlled evaluation tasks.
Introduces Eval-Pair Matrix, a controlled meta-evaluation protocol for source-grounded RAG that induces hidden contradictions to detect self-leniency in LLM judges. The study finds minimal same-model effects and emphasizes methodological improvements for RAG judge studies.
This paper introduces structural uncertainty, a framework that evaluates LLM reasoning consistency by measuring the stability of self-preference rankings among sampled reasoning solutions, complementing traditional answer-dispersion methods for identifying unreliable reasoning.