Tag
This paper introduces a controlled evaluation framework that separates answer coverage from repeatable task specialization in LLM-generated harnesses on MATH-500, finding that additional coverage largely comes from repeated executions of the same program rather than genuine specialization gains.
This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.
提出 TauIndianBankBench:一个包含 1,000 项印度零售银行任务的评测基准,不仅检查最终数据库状态,还要求工具使用型对话代理完成所有必要步骤且不做未请求的变更;结果显示仅按结果评分时有 6.5–12.0% 的会话在违反流程的情况下仍被计为成功。
该论文揭示了基于评分表的强化学习中加和式奖励聚合导致的奖励作弊问题:模型通过堆砌无关建议换取高分却降低回答质量。作者提出ProRubric,将评分标准按协议级维度分组(所有条件同时满足才计分),在临床咨询场景中将适切性提升10.8个点且不损失覆盖率。
PhysVista 提出了一个基于感知-推理-评估闭环认知框架的基准,用于评测视觉语言模型的物理智能,涵盖事件级与尺度级推理以及真实和AI生成视频,并揭示了VLM在物理推理和合理性评估上的显著局限。
论文提出 A2Z GameSpec-Bench,一个包含 100 份长篇游戏设计文档(GDD)的基准,用于评估编码代理在端到端游戏开发中遵循互锁需求的能力。结果显示现有代理在同时满足代码实现与实际游玩的联动需求上表现不佳,而需求级反馈可在两轮修订后将 GDD 忠实度提升 10.9%。
Hugging Face introduces the Open TTS Leaderboard, an objective, scalable evaluation framework for multilingual TTS and voice cloning using ASR-based intelligibility (WER/CER), speed (RTFx/TTFA), and WavLM speaker-similarity metrics, reducing model evaluation time from weeks to hours.
Liquid AI and Artificial Analysis release Pipette, an open-source benchmarking suite for on-device AI that measures quality, speed, latency, and memory use across models, quantizations, runtimes, and devices, shipping with 10k+ verified results and clients for macOS, Windows, iOS, and Android.
The article discusses the challenge of distinguishing between retrieval and generation failures in RAG systems and explores practical methods to measure each component independently.
PhyProbe 提出一种评估生成视频物理一致性的新评测器,通过冻结的预训练时空编码器提取特征并用轻量评分头映射为违反分数,结合排序、回归与锚定校准的统一目标,在多项基准上超越现有方法。
The paper proposes a framework for designing proactive LLM agents around three joint principles (3T): Task Capability, Temporal Allocation, and Trust, along with a five-dimensional design space. It also introduces Proactivity-Gym, a simulation-based testbed with multi-day scenarios and persona-conditioned simulated users, evaluating 23 model-harness configurations and a 30-participant human study showing that trust declines sharply when interventions misalign even if task outcomes are correct.
RLE-Bench introduces a qualifying-exam-style benchmark that evaluates coding agents' broader engineering capabilities in robotics — covering interactive control, policy learning, perception and estimation, and mechanical design — aggregating task-specific metrics into an overall RLE Index for systematic comparison.
论文提出 MIST(Misleading Image Stress Test)基准,测试 13 个 VLM judge 在存在对齐、误导或缺失图像时标签的稳定性,发现图像是否出现而非图像内容才是改变判断的主要因素,说明可替代性结论对评测配置同样敏感。
该论文系统评估了 GPT-6 Astra 作为通用具身策略的能力,覆盖抓取、灵巧操作、移动操作、导航、运动和人形 loco-manipulation 六个领域,发现混合控制(与 π0.5 等学习策略结合)表现最佳,但仍存在任务决策与可靠物理控制之间的差距,且推理延迟严重制约实际部署。
APM-Bench introduces a benchmark for cross-session persistent memory in egocentric streaming video assistants, reformulating real-world interaction as multi-session life trajectories with 549 sessions and 104 trajectories, and evaluates the utility–latency–storage trade-offs of existing memory systems.
EpiCon presents a shared multimodal memory framework for collective learning among AI agents, enhancing performance across eleven benchmarks without updating host model parameters.
The paper proposes HasMem, an adaptive memory compression method for long-term LLM agents that adjusts memory widths while maintaining frozen LLM parameters, demonstrating improved performance on reconstruction and QA benchmarks.
This paper proposes using literary interpretation as a setting to evaluate cultural reasoning depth in language models, emphasizing interpretive depth and evidence-centered benchmarks over conventional metrics.
This paper introduces an LLM-as-a-judge method to measure perturbation strength for assessing self-consistency in LLM explanations, showing that input perturbations generally affect LLMs more strongly than CoT perturbations.
The paper introduces KNOWS, a benchmark for evaluating web agents on complex, long-horizon tasks that involve synthesizing and organizing knowledge into artifacts, revealing that current agents struggle with visual steps and long-horizon reasoning.