evaluation

This paper identifies a 'detectability gap' in hallucination detection: hallucinations split into high-agreement (Ghost) and low-agreement (Flickering) regimes with a 0.35–0.46 AUC gap, persisting across four models and three factual QA datasets even after freezing regime assignments and using stricter trajectory-based tests. The authors argue aggregate detection metrics hide model-dependent heterogeneity and call for regime-conditioned evaluation.

The paper introduces tau-Multilingual, a benchmark extending tau-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin for evaluating voice agents across task completion, interaction quality, and language generation. Results show Spanish, Portuguese, and Hindi perform close to English, while Korean and Mandarin drop significantly, and the authors release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

BreakingWeb is a benchmark framework from Amazon and Duke that constructs challenging browser-use tasks via controlled, deterministic environment interventions on tasks agents already solve, cutting agent pass rates by 22.9% while humans lose only 10.0%, revealing widespread 'belief failure' where agents falsely declare success.

研究人员提出一个用于评测车载对话助手（ICA）多轮对话能力的自动化黑盒测试框架，采用策略引导的用户模拟器、对抗性策略管理器和两级 LLM 评审器，在工业级 ICA 上验证该方法与人类标注高度一致，并发现 2.96 倍的独特失败类型。

This arXiv paper evaluates how prompt perturbations affect bias and hallucination in LLMs used for decision-making tasks, finding that perturbations can mitigate these issues in some models, with Claude 3 outperforming GPT-3.5 across most datasets.

This paper introduces an exact generate–transform decomposition to analyze how small 7–9B LLM teams scale across eight orchestration architectures, finding that scaling returns are sharply task-dependent — Proposer-Critic wins on arithmetic benchmarks but no architecture dominates across all tasks.

This paper introduces a controlled evaluation framework that separates answer coverage from repeatable task specialization in LLM-generated harnesses on MATH-500, finding that additional coverage largely comes from repeated executions of the same program rather than genuine specialization gains.

This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.

提出 TauIndianBankBench：一个包含 1,000 项印度零售银行任务的评测基准，不仅检查最终数据库状态，还要求工具使用型对话代理完成所有必要步骤且不做未请求的变更；结果显示仅按结果评分时有 6.5–12.0% 的会话在违反流程的情况下仍被计为成功。

该论文揭示了基于评分表的强化学习中加和式奖励聚合导致的奖励作弊问题：模型通过堆砌无关建议换取高分却降低回答质量。作者提出ProRubric，将评分标准按协议级维度分组（所有条件同时满足才计分），在临床咨询场景中将适切性提升10.8个点且不损失覆盖率。

PhysVista 提出了一个基于感知-推理-评估闭环认知框架的基准，用于评测视觉语言模型的物理智能，涵盖事件级与尺度级推理以及真实和AI生成视频，并揭示了VLM在物理推理和合理性评估上的显著局限。

论文提出 A2Z GameSpec-Bench，一个包含 100 份长篇游戏设计文档（GDD）的基准，用于评估编码代理在端到端游戏开发中遵循互锁需求的能力。结果显示现有代理在同时满足代码实现与实际游玩的联动需求上表现不佳，而需求级反馈可在两轮修订后将 GDD 忠实度提升 10.9%。

Hugging Face introduces the Open TTS Leaderboard, an objective, scalable evaluation framework for multilingual TTS and voice cloning using ASR-based intelligibility (WER/CER), speed (RTFx/TTFA), and WavLM speaker-similarity metrics, reducing model evaluation time from weeks to hours.

Liquid AI and Artificial Analysis release Pipette, an open-source benchmarking suite for on-device AI that measures quality, speed, latency, and memory use across models, quantizations, runtimes, and devices, shipping with 10k+ verified results and clients for macOS, Windows, iOS, and Android.

The article discusses the challenge of distinguishing between retrieval and generation failures in RAG systems and explores practical methods to measure each component independently.

PhyProbe 提出一种评估生成视频物理一致性的新评测器，通过冻结的预训练时空编码器提取特征并用轻量评分头映射为违反分数，结合排序、回归与锚定校准的统一目标，在多项基准上超越现有方法。

The paper proposes a framework for designing proactive LLM agents around three joint principles (3T): Task Capability, Temporal Allocation, and Trust, along with a five-dimensional design space. It also introduces Proactivity-Gym, a simulation-based testbed with multi-day scenarios and persona-conditioned simulated users, evaluating 23 model-harness configurations and a 30-participant human study showing that trust declines sharply when interventions misalign even if task outcomes are correct.

RLE-Bench introduces a qualifying-exam-style benchmark that evaluates coding agents' broader engineering capabilities in robotics — covering interactive control, policy learning, perception and estimation, and mechanical design — aggregating task-specific metrics into an overall RLE Index for systematic comparison.

论文提出 MIST（Misleading Image Stress Test）基准，测试 13 个 VLM judge 在存在对齐、误导或缺失图像时标签的稳定性，发现图像是否出现而非图像内容才是改变判断的主要因素，说明可替代性结论对评测配置同样敏感。

该论文系统评估了 GPT-6 Astra 作为通用具身策略的能力，覆盖抓取、灵巧操作、移动操作、导航、运动和人形 loco-manipulation 六个领域，发现混合控制（与 π0.5 等学习策略结合）表现最佳，但仍存在任务决策与可靠物理控制之间的差距，且推理延迟严重制约实际部署。