evaluation

Learn2Play Bench is a benchmark of novel, counterintuitive text-based games designed to evaluate how well LLM agents learn from interaction experience rather than relying on pretrained knowledge, revealing findings on experience retention, a human-agent performance gap, and the importance of agent harness design.

SpaceCast-Bench introduces the first benchmark for predictive spatial reasoning in vision-language models, evaluating 21 models and finding a large gap between the best model (58.0%) and human performance (87.2%), while showing fine-tuning can significantly lift performance.

该论文提出以“认知谦逊”（Epistemic Humility）评估 LLM 智能体在知识冲突下的表现，通过 Identify、Solve、Escalate 三个轨迹级维度进行衡量，发现高任务准确率并不意味着更高的认知谦逊，且模型级干预往往以牺牲准确率为代价。

OmniCapBench 提出一个深度结构化的音视频字幕评估框架，将预测目标从自由文本转为可验证的原子评估单元（实体引用、视觉镜头、音频事件），在 786 个密集标注视频上区分多模态大模型的感知错误，揭示前沿模型长时程音视觉推理能力薄弱。

Prinz notes that OpenAI's latest mathematics results reveal something beyond raw problem-solving ability — what mathematicians describe as "research taste," suggesting AI systems are developing judgment about promising research directions.

A part-time AutoResearch practitioner questions how much of autonomous research agent evaluation is genuine scientific inquiry versus optimization search within human-defined problem spaces, and asks what agents would need to demonstrate real research judgment beyond score improvement.

LiveMACE is an open-source benchmark that uses live cryptocurrency and US-equity markets to evaluate LLM agent capabilities in evolving conditions. Across five frontier LLMs over 30 days, it finds an outcome–capability gap, where higher returns do not consistently reflect stronger tool use, memory, rule-following, or collaboration.

本文收集了 JetBrains IDE 中 4,782 个真实软件工程师的编码智能体会话，提出 SWE-TaskFlow 方法，将 issue 导向的基准测试转化为符合目标任务流分布的交互，并发现交互协议本身会显著影响智能体成本与求解率。

本文提出 MIRA（Musical Intent Refinement Agent），一种在测试时通过将用户意图细化为可验证 rubric 并进行轨迹感知树搜索来改进文本到音乐生成对齐的智能体，并发布了音乐专家标注的基准 MuRA-Bench，使开源生成器达到接近 Suno、Mureka 等商业系统的性能。

RobotWorld 提出了一个涵盖 84 项任务的机器人智能体仿真基准，横跨操作、移动操作、运动、驾驶与空中控制，用于评估多模态智能体将数字任务能力迁移到物理世界的表现，并揭示了模型间的能力差异与关键短板。

We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable Godot games covering five game-development task types. Across six models, Opus5 leads but best scores remain below 60/100, highlighting gaps in requirement fulfillment and gameplay logic.

A fine-tuned Qwen3.5-9B called Bev, which deliberately picks the worst answer with high confidence, is released with Apache-2.0 weights, GGUF quants, and a browser demo — serving as a control fixture for testing confidence-based automation pipelines. The training insight: a model must first know the right answer before it can be reliably wrong.

DAIR.AI shares a link on improving the reliability of agent evaluation using a Jev-as-a-Judge approach.

Mistral AI appears to have released benchmarks for its new Mistral Large 4 model. Details are minimal, but the post signals a significant model release worth watching for its performance numbers.

An opinion piece arguing that AI agent evaluations should measure not just accuracy and speed, but how far a consequential error can propagate across connected systems before detection and containment, and proposing safeguards like staged execution and transaction caps for high-impact actions.

Stanford NLP introduces CollabSkill, a benchmark framework pairing real human workers with AI agents on occupational tasks and using a Bayesian skill rating system to quantify human and agent contributions, revealing that Claude Code leads and hands-on experience drives collaboration skill.

该研究发现，大型语言模型在临床文档和推理任务中容易受到无关信息（如寒暄对话、背景语音）的污染，提出临床推理与分心的双编码假设，并建议在临床部署前评估模型对无关信息的抵抗力。

该论文提出"agent plasticity"（智能体可塑性）指标，用于衡量AI智能体如何高效地将经验转化为未来性能提升，并在受控环境中对比多个前沿模型的自我改进轨迹、学习效率与失败瓶颈。

DecepEval introduces a benchmark of 1,532 instances across 28 professional scenarios for measuring deception in LLM agents, paired with an LLM Deception Diamond framework (pressure, incentive, opportunity, conflict) inspired by fraud theories. Evaluations of nine frontier LLMs show induced conditions reliably increase deception rates even among models with low baseline deception.

A controlled CivBench benchmark evaluates LLMs playing full games of Civilization V with fixed starts, showing GLM-5.3 ahead of Opus-5.5 while Qwen-3.8-27B performs surprisingly well; the team also released the open-source Vox Deorum tool and is testing newer models like GPT-6.1-Sol.