evaluation

A Meta 与密歇根州立大学的新论文发现，LLM 构建的推理图质量差异很大，其错误会传导到基于这些图计算出的答案和置信度中；研究测试了 9 个模型，指出图的质量更多取决于模型家族与图结构而非规模。

This paper shows that asking vision-language models which blocks a director can see before acting significantly changes their choices in the Director Task, raising hidden-block selections by 12-88 percentage points across GPT models and 17 points on Qwen3.8-27B. The authors argue perspective-taking benchmarks need paired baselines that differ only by the report question to avoid misreading failures of perspective use.

The paper introduces MOSAIC, an omni-modal benchmark of 923 questions over persona-grounded multi-session histories spanning text, image, video, and audio, designed to diagnose where conversational agents lose evidence across write, retrieve, deliver, and read stages rather than just producing a single leaderboard score.

该论文提出 DiagSafe 诊断式评估框架，用提示级评分标准（prompt-level rubrics）替代二元安全标签，构建了中英双语、基于 84 类安全分类体系的基准，使 LLM-as-a-judge 与人工标注的一致性达到 95.34%（Cohen's κ=0.864），并揭示了主流模型在中文与多轮对话场景下的独特漏洞。

该论文提出 RE-ZS-CoT 框架，通过整合领域专家角色来增强大语言模型的零样本链式推理，并在 20 个 LLM 和 7 个基准上评估，最高提升达 44 个百分点。

这篇论文介绍了一种"typed decision"System 1 接口：通过一次前向传播直接从模型 logits 或小分类头输出固定选项的概率，用于替代昂贵的 frontier LLM 判官。作者部署了一个自托管的 rank-16 LoRA 判官来分流 14.8 万条 agent 编码会话记录，成本仅为前沿模型判官的约 1%；并在 JevBench 上用 35B 开源模型达到 79.3%，与商业参考系统（73.0%）无显著差异。

SANE introduces an agent-based framework that automatically derives scorer-specific perturbation recipes and expected score relations for testing LLM judges, producing 3.7x more validated test families than the Judge Reliability Harness and achieving strong human alignment in judge ranking.

Walmart Global Tech and university researchers propose an explicit feedback contract for iterative ML-agent harnesses that only releases validation scores above a noise threshold computed from paired standard error, reducing adaptive overfitting to the validation split and preventing agents from chasing noise across repeated submissions.

该论文重新评估了基于LLM的文本属性图节点分类方法，发现其相对简单基线（如Correct & Smooth、MLP）的报告优势往往很小，并记录了测试集调参、标签泄漏等评估风险。

CellTrack-Bench introduces a microscopic cell tracking benchmark evaluating 22 vision-language models against specialized trackers across 15 datasets and 4 imaging modalities, revealing significant reasoning gaps in frontier models.

The author built a test store with 7 common dark patterns and ran 4 AI shopping agents (ChatGPT, Claude Opus/Sonnet/Haiku 5.5 via Buddy) through it; the top models scored A+ by avoiding deceptive pricing, pre-ticked add-ons, and urgency tricks, with a harder round 2 planned.

本文评估语言模型智能体的上下文压缩方法在'后续问题'上的表现，发现针对当前问题选择的句子在后续问题上并不优于随机删除，而 LLMingua-2 等词级压缩方法在多个基准上表现更佳。

RealControlBench is a new benchmark for evaluating interpretability-informed fine-grained steering of LLMs and agents across five control dimensions, finding that existing steering methods are unreliable in realistic settings and often degrade model capabilities.

该论文提出两阶段诊断评估方法，检验以图结构表示的LLM推理链的结构可靠性，发现图的可靠性随拓扑、提示设计和模型族而变化，且图结构对不确定性估计的校准和节点级诊断有帮助，但作为提示上下文时收益有限。

The paper introduces LoWAgent, a low-resource multilingual benchmark evaluating web agents across English, Hindi, Bengali, Urdu, and Amharic on six localized websites with 3,000 tasks, finding notable performance drops for low-resource languages—especially open-source models—and showing that post-training on benchmark traces improves agentic performance.

A comparative benchmark study evaluating ten LLMs on 70 academic prompts across four complexity levels, measuring hallucination and response deviation. Findings highlight a trade-off between response completeness and factual reliability, with Perplexity and DeepSeek V3 scoring highest on accuracy.

TRACE is a training-aware benchmark framework that reorganizes 168 text-vision embedding tasks into General, Adaptability, and Challenge categories, revealing that data adaptation improves adaptability but fails to solve challenging multilingual and visual-document tasks.

This paper proposes a state-bound runtime assurance layer for AI agents that revalidates prior approvals at dispatch time, closing the gap between when an approval is evaluated and when it is executed. In a 42-case replay, it identifies 15 later-invalidated approvals and blocks 14 affected actions before dispatch.

The paper presents ComplianceAgent, the legal compliance sub-agent of the COMPASS multi-agent framework, which uses an LLM to score texts against legal corpora with corrective rewrite loops and human escalation, evaluated on a Canadian and Quebec legal corpus with a hand-annotated gold standard.

Salesforce AI Research 釋出 EGI-Bench，一套整合八個企業級 agent 基準測試的統一基準套件，並提出六維度的 Enterprise General Intelligence (EGI) 分數，評測包括 GPT、Claude、Gemini 在內的十四個前沿與高性價比模型。