evaluation

AgentHorizon introduces a benchmark of 1,373 computer-use tasks with human-recorded trajectories to evaluate agentic judges on long-horizon, multi-application tasks. The study finds GPT-5.5 achieves 80.9% balanced accuracy as the best agentic judge, and reveals that tool use helps some models but hurts open-weight models.

This paper introduces 'normative competence'—the ability of LLM agents to discern community-enforced norms from interaction alone—and evaluates it via a multi-agent debate setting with synthetic norms. Baseline LLMs fail to learn norms even when beneficial, and norm-following shows poor generalizability, with agents exhibiting unselective attribution by copying idiosyncratic noise alongside true norms.

GraphDecisionBench：一个模型无关的基准，用于评估 System One 模型在图结构任务上的决策能力，涵盖 14 种模型-接口配置，揭示了邻接识别准确并不等同于结构正确性等关键发现。

Exa introduces ATLAS, a new benchmark for evaluating AI agents on search-intensive, real-world research tasks with verified golden answers and a refreshable automated pipeline. Key findings: no sub-$1 agent run achieved row F1 over 0.5, top agents still miss ~1/3 of golden results, and Exa's search backend defines the cost-performance Pareto frontier.

Epoch AI 发布了一项评测报告，用自身真实工作任务测试六个模型，发现 Fable 5.1 与 GPT-6 Astra 虽然领先，但在开放式任务、隐性标准遵循和研究判断力上仍无法实现工作自动化。

Grok 4.7 ranks first on the updated Harvey LAB-AA v1.1 legal agent benchmark, which introduces hallucination checks and stricter accuracy standards. The benchmark was revised by Artificial Analysis in collaboration with Harvey to improve evaluation rigor.

Introducing Humanity's Sixth Sense, a new benchmark testing intuitive visual reasoning across spatial, causal, and social understanding. Humans achieve 93.1% while the strongest model, GPT-6-astra, reaches 53.6% and the median model only 30.9%.

研究发现 Claude Code 在常规任务中仅有约 9% 的会话会自动加载插件技能；改用一个"入口"技能负责路由后，技能采用率提升至 55%。作者在博客中分享了构建 Evals 的完整经验与提升技能采用率的具体建议。

InsClaimBench 是一个针对保险理赔裁定的端到端基准，涵盖汽车、财产与健康险的 3,780 个案例，评估 LLM 在从原子规则到最终赔付的决策链上的可靠性，结果显示模型在链式组合与事实变更传播中性能显著下降。

ArcticQA/ArcticAbstain introduces a dataset and paired benchmark evaluating whether LLMs appropriately abstain on Arctic science multiple-choice questions, finding large baseline differences and only modest increases in abstention when correct answers are removed.

论文提出 ARCS（Ambiguity Resolution Corpus for SQL），首个包含真实数据库上自然歧义的 text-to-SQL 基准，并提出“结构化消歧”范式取代自由对话澄清。实验表明歧义下 text-to-SQL 仍具挑战：GPT-6 SOTA 仅 51% 端到端执行准确率，开源模型均不超过 27%。

This arXiv paper studies why LLMs abandon correct answers under user pushback, finding that task verification cost and trained guardrails dominate over model family or pressure tactics. With reasoning enabled, models on hard puzzles went from 19.2% and 12.5% adoption down to 0%, yielding practical guidance for reliable LLM use.

The paper introduces Tokka-Bench, an open-source framework for evaluating tokenizers across 100 natural languages and 20 programming languages using five metrics, finding that vocabulary allocation strategy matters more than raw vocabulary size among seven BPE tokenizers including GPT-4, Llama 3.1, Gemma 3, Qwen3, and Kimi K2.

The paper introduces NP-Bench, an environment-grounded benchmark for verifying coordination among parallel LLM coding agents, and a proactive scheduling planner built into Nerveplane that partitions work into disjoint scopes and orders merges along dependency graphs. The planner lifts clean-integration success from 1/9 to 9/9 scenarios, eliminates merge conflicts, and collapses repeated-mistake rates from 1.00 to 0.00 via cross-session memory.

该论文提出"思考惯性"（Thinking Inertia）现象：即使明确指示LLM不要思考，模型仍会持续进行显式推理，且在开放式问题中更严重。作者提出三个衡量"不思考"行为的指标，并在六个模型和多种任务上评估，指出停止推理无法仅靠设置或指令保证。

This paper introduces Humanity's Sixth Sense (HSS), a benchmark for intuitive visual reasoning in multimodal large language models, showing humans reach 93.1% accuracy while the strongest model, GPT-6-astra, achieves only 53.6%, highlighting a significant capability gap in implicit visual understanding.

DataVista 是首个针对数据视频（datavideo）理解的基准测试，包含 961 个真实数据视频和 6,775 道评估题，采用三级能力框架（数据感知、时序推理、叙事理解）。对 19 个主流多模态大模型的评测显示，最佳模型 Gemini-3.1-Pro 仅有 70.0% 准确率，远低于人类专家水平。

LatentGRM 提出了一种基于语义分块与压缩重建的潜在评估框架，通过在连续隐空间中完成 rubric 引导的成对偏好判断，将评估轨迹压缩 8.9–9.2 倍并把推理时间减少 6.1–7.0 倍，同时保持与显式 SFT 评委相当的偏好准确率。

Pumpire 提出了一个统一的基准测试，用于评估图像和视频级 3D 基础模型的点对点物理距离估计能力，并构建了包含 100 个真实场景、6400 帧的大型数据集 pumpire-6k，对 29 种基线配置进行了全面对比实验。

该论文提出了 RISEBench 及其扩展版本 RISEBench++，这是首个评估推理驱动视觉编辑（Reasoning-Informed Visual Editing）的基准，涵盖时序、因果、空间、逻辑、反事实及混合推理等六个维度，并提出训练无关的 RISE-Agent 框架，结果显示即使是 GPT-Image-2.5 Sunburst 也仅达 56.6% 准确率。