Tag
A company building memory infrastructure introduces 'glasshouse', a draft long-term memory benchmark with 2,847 questions over a 1.97M token conversation in 10 languages, reporting 17 separate axes (including fact-update consistency and conflict resolution) without a single headline score, and asks the agent-memory community for feedback.
LangChain 发布关于 Agent 评估（Agent Evals）的实用指南，建议从真实 trace 中选取 10–20 个失败案例构建小型评估集，结合代码检查与 LLM Judge 逐步完善评估流程。
A practitioner asks the community how they test tool-calling AI agents before and after deployment, comparing scenario tests, LLM-as-judge, tool-call trajectory checks, and regression suites while raising doubts about coverage, judge reliability, and nondeterminism.
论文提出 semCER，一种利用 LLM 评审按严重程度对字符编辑加权的 OCR 评估指标，相比标准 CER 更能贴合人类评分，并在英、越、中等 16 种语言上验证有效，同时发布了 6100 万条已评分编辑数据集与蒸馏打分模型。
MA-Bench introduces the first benchmark for evaluating how well language models perform mission analysis (military operational planning) by turning incomplete, contradictory document packets into structured plans graded against 1,257 expert-written requirements.
该论文提出一种无标签的评分标准（rubric）修复流程：通过跨评审模型的分歧自动定位有争议的评分条款并由编辑模型重写。实验表明该方法提升了评审间一致性（Krippendorff's α 提升约 0.07），但并未提升与人类专家评分的一致性，甚至点估计略低于随机条款编辑对照组。
这篇文章科普了微软研究院的论文《G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment》，介绍其通过链式思维生成评分清单并结合各分数的概率加权平均，解决 LLM 评分不稳定的问题。
作者在 AI Engineer World's Fair 上的 evals 演讲总结，记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致：包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。
文章主张评测（evals）是企业成为 AI-native 的最大障碍，并给出了 eval-driven development 的指南：定义好的行为、对 agent 进行测试、将失败案例转化为回归测试，并介绍了 grader、case、run、trace 四个核心概念以及确定性检查、参考答案比对等四种打分方式。
A discussion on how unreliable LLM memory benchmarks can be, noting that scores swing wildly depending on the evaluator model (e.g., 51.4 under token-overlap F1 vs. 75.8 under an LLM judge), the reader model, and run counts — while many results are vendor-reported with unclear noise levels.
介绍 Jev 如何作为 RAG 管道中的“裁判”，在检索与生成之间评估候选段落的相关性、可回答性和潜在提示注入，并附上基于 Comet Opik 的开源 Jev-as-a-Judge 项目。
SCOUT introduces a two-stage agentic safety verifier that combines reasoning-intensive rubric generation with tool-intensive evidence gathering to detect unsafe behaviors in computer-use agents, improving unsafe detection on AutoElicit-Bench and reducing unsafe execution rates from 30.2% to 17.2% via test-time reflection.
研究人员提出一个用于评测车载对话助手（ICA）多轮对话能力的自动化黑盒测试框架，采用策略引导的用户模拟器、对抗性策略管理器和两级 LLM 评审器，在工业级 ICA 上验证该方法与人类标注高度一致，并发现 2.96 倍的独特失败类型。
The author benchmarked the Jev judge on a dataset and reduced its calibration error by 68.1% through learning from human-labelled examples, improving confidence alignment for production use without significantly changing classification accuracy.
OSWorld-Pro introduces a process-based benchmark with over 300 tasks and 2,800 subgoals, grounded in 67,000+ human annotations, to evaluate computer-use agents step-by-step rather than only by end-state outcomes. It reveals failure modes even in state-of-the-art models, with top performers like Claude Opus 5 scoring 75.7% versus 83.4% on OSWorld.
The paper introduces Self-Improvement via Fast Tree-search (SIFT), a framework that uses an LLM-as-a-judge to efficiently evaluate self-modifications in coding agents, achieving better benchmark performance with significantly reduced CPU hours and API costs.
This paper measures family-conditioned preference in LLM-as-Judge panels, finding that judges from the same model family as the candidate show a significant positive bias, and introduces a corrected estimator to quantify this effect across four model families.
DuplexDrama is the first synthesized spoken dialogue dataset that covers personas, scenarios, full-duplex behaviors, expressive speech, and sound events, designed to advance full-duplex dialogue model research with over 2,000 hours of audio data.
This study investigates capability-dependent biases in LLM judges and introduces a calibrated weighted majority voting ensemble method to enhance automated evaluation reliability without requiring labeled data.
GAUGE is a reusable offline protocol that measures the validity of using LLM-as-a-Judge in evaluating task-oriented agents, revealing a satisfaction–success gap and proposing a calibration method to improve accuracy.