evaluation

Muse Wonder (high) achieves 93.4 on the Extended NYT Connections Benchmark, surpassing Muse Spark 1.3 (high) at 85.1, though it ranks among the most expensive models tested.

研究者提出名为 Into the Unknown 的基准，让 AI 代理针对未发表的科学方向撰写后续论文，结果发现 9 个前沿代理的 135 份投稿均未达到专家标准，且多数存在虚构主张或与证据矛盾的数据。

MA-Bench introduces the first benchmark for evaluating how well language models perform mission analysis (military operational planning) by turning incomplete, contradictory document packets into structured plans graded against 1,257 expert-written requirements.

论文提出「推导鸿沟」（derivation gap）现象：LLM 工具调用评审器会因输入中缺少可参考的原文，而拒绝接受正确但需推导或为默认值的参数。作者提出 DeRef 方法，通过独立模型生成参考答案来显著提升评审的 AUROC 表现。

该论文研究在智能体循环中，System One 模型选择错误动作的原因，发现大多数错误动作是在正确读取状态事实后仍行动错误，而非误解输入，并揭示模型无法组合已知事实来确定下一步动作。

LifeSearch introduces a benchmark of 203 Chinese and English tasks for evaluating deep research agents in real-world life and productivity settings, with over 4,000 expert-written rubrics and experiments on 12 advanced agents, finding that even the best frontier agent (GPT-5.6 Sol) scores only 63.4%.

Dan Shipper notes a disappointing 52% showing for Muse Wonder on the Senior Engineer Benchmark (which tests rewriting vibe-coded repos from scratch), far behind GPT-Sol 6.1 at 71%.

Scale AI introduces DistressBench, a clinician-authored benchmark testing whether AI models provide adequate help during suicide or self-harm crises, finding that 25 frontier models often recognize danger yet still fail to deliver needed resources and response steps.

Dan Shipper says he's deleting his tweet about Muse's performance on Every's senior engineer benchmark after finding an issue in the run and planning to restart it.

该论文提出一种无标签的评分标准（rubric）修复流程：通过跨评审模型的分歧自动定位有争议的评分条款并由编辑模型重写。实验表明该方法提升了评审间一致性（Krippendorff's α 提升约 0.07），但并未提升与人类专家评分的一致性，甚至点估计略低于随机条款编辑对照组。

The paper introduces CoBe, a benchmark of 2000 questions evaluating models' zero-shot ability to perform conversational counterfactual text editing, finding that frontier models (averaging 58.05%) often default to associational generation rather than preserving causal structure, with performance not scaling with model size or token length.

该论文提出将LLM创造力形式化为参考输入-输出分布的长尾性能，通过15个基准和26个开源模型验证，并引入低成本创造力诊断指标 q@Tail(α)。

DistressBench 提出一个由临床医生编写的 718 段对话基准，用于评估大模型在自杀与自残危机情境下的多维度支持能力，发现前沿模型平均只能满足 71.4% 的必要照护标准，且常在识别危机后未能引导用户寻求人工帮助。

GeoFitBench-LLM 是一个评估大语言模型在地理分析任务中推断地理空间数据需求能力的新基准，围绕时空关系与语义落地两个推理维度，实验显示 GPT-5.4 Thinking 在资源识别上表现最佳（66.67%），但属性级适用性判定仍具挑战。

Researchers introduce GeopoliticalQA, a benchmark of 5,525 QA pairs over 415 geopolitical and strategic intelligence reports, evaluating 15 retrieval systems and six open-weight LLMs. They find mid-size encoder bge-m3 matches or beats much larger and commercial embedding models at chunk-level retrieval, while answer generation remains hard even with gold evidence for inferential and risk-oriented questions.

SWE-Duel 提出一种自我扩展的编码智能体竞技场，两个智能体通过镜像红蓝回合互相评估：红色智能体开发新功能并植入隐藏缺陷，蓝色智能体在不知情的情况下审查并修复。在 15 个仓库、11 个模型-工具组合上进行了 1,500 回合的循环赛，显示任务难度可随作者能力扩展。

We present SmartSalesBench, the first multi-turn conversational benchmark designed to evaluate LLM sales agents against psychologically grounded, resistant buyer personas, with nine evaluation metrics and experiments across four models totaling 13,680 sessions.

MathVis introduces a 512-task benchmark evaluating LLMs' ability to generate mathematical diagrams and perform diagram-assisted reasoning, with GPT-6-astra, Claude Fable 5, and Gemini 3.1 Pro achieving only 31.25%, 26.56%, and 22.85% Pass@1, revealing that models still rely primarily on symbolic reasoning rather than visual diagrams.

SpireBench introduces a long-horizon benchmark in which agentic LLMs play the full commercial deck-builder Slay the Spire. Frontier models beat heuristic baselines and match average human players but fall far short of experts, struggling with per-fight play, risk assessment, and cross-run memory.

论文指出 LLM 反编译器以可重编译性和重执行性作为主要指标时会产生误导：代码虽然能通过所有自带测试，却可能在其他合法输入上行为发散甚至使已披露漏洞悄然消失。作者提出 Decompile-Diverge 行为比较预言机，通过合成驱动程序和模糊测试语料检测行为差异，发现部分系统发散率高达 13%。