evaluation

Surge AI 发布 DAYJOB: Tax 基准，使用 50 项专业税务任务评估 AI 智能体，Claude Sonnet 5.5 以 10.8% 通过率领先。

A practical write-up on CoworkEnv, discussing how to design agent RL environments by backtracking from "what capability do we want to train." It distills 11 lessons covering environment consistency, scorer flaws, trajectory persistence, and training interruptions. An 8B LoRA SFT model improved Claw-Eval Overall from 46.6 to 64.1, but still trails Claude Opus 4.6's 80.4.

该论文提出了 EPSBench，一个用于评估 LLM 智能体季度盈利预测能力的基准，涵盖 86 家美国公司的 149 个公司季度数据，结果表明各模型的预测区间覆盖率显著低于标称的 80%。

作者质疑当前用任务完成率来衡量AI agent表现的做法，指出即使agent成功关闭了大量工单，也可能因未真正解决问题而导致客户体验下降。文章探讨应如何转向衡量下游业务结果、修正率和客户满意度等更实际的指标。

该论文构建了涵盖五种幻觉类型的二元基准（来自六个数据集的9,852个样本），比较了五种决策模型与生成式裁判gpt-6-sol在幻觉检测上的表现，发现决策模型在部分类型上以1-3%的成本匹敌或超越生成式裁判，但在逻辑与指令不一致类型上仍有差距且概率存在过度自信问题。

DocXEval introduces the first comprehensive explainability framework for multimodal document understanding, evaluating 13 XAI methods across 3 tasks, 8 datasets, 4 models, and 17 metrics, finding Occlusion as the most faithful and interpretable method overall.

该论文提出 Recursive Knowledge Improvement (RKI)，将知识改进建模为序列化、状态感知的干预过程，通过区分知识状态别名问题来指导干预选择，在 HotpotEval 和 SafeInternalEval 上平均提升 18.41 分。

This paper systematically studies LLM-based verification of Olympiad-level mathematical proofs, introducing a human-curated USAMO 2025 dataset (Trojan-Bench), an adversarial proof generation pipeline (TROJAN), and MABGrader, a Multi-Armed Bandit grading framework that outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa across 20 evaluated models.

论文提出 InfraDev 基准，包含 36 个源自 Megatron-LM、vLLM 和 verl 真实 PR 的任务，用于评估编码代理开发生产级 AI 基础设施的能力。目前最佳前沿模型仅解决 38.9% 的任务，揭示了代理在分布式系统开发中的显著短板。

ETC-Bench is the first comprehensive multilingual benchmark for evaluation-side transcript canonicalization, covering 181 phenomenon types across 12 languages, along with a new SENA metric and a taxonomy-guided agent method (TACS) that outperforms baseline systems.

This arXiv paper argues that classification accuracy is insufficient for evaluating AI agent failure/restart monitors, introducing a decision-value framework based on Blackwell's comparison of experiments and a model-free decision audit, applied to SWE-bench Verified traces showing that apparent restart gains often lack statistical support.

论文提出 Handoff State Continuity Benchmark (HSC-Bench)，包含 1,039 个任务，用于评估 AI 代理在交接时的任务接管能力，涵盖仓库证据恢复与从既有对话、工具和文件状态继续工作的两个方面，并对比了六个接收模型的表现。

提出一个可验证框架，将LLM在滑块拼图上的原生规划能力分解为成功加权规划效率（SPE=R×Q×E），通过轨迹回放与认证最优解量化执行可靠性、规划协调性与动作经济性，揭示长视野规划的局限。

该论文精确计算了语言模型在被要求随机选数时对每个整数的完整概率分布，发现指令调谐使分布更不均匀：安装了避免轮数的启发式并强化了避免端点值的倾向，但未产生跨模型共享的偏好整数。

MARQ is a modular, LLM-agnostic prototype that infuses expert-derived systems engineering knowledge into LLM-based requirement quality assessment, showing significant F1 improvements over off-the-shelf baselines on an expert-labeled NASA requirement set across Claude Opus 5, GPT-5.6 Sol, and Gemma 3 12B, though gains are model-dependent.

This arXiv paper examines what it truly means for AI assistance to save time, arguing for a 'ledger view' that weighs time saved against errors that slip through rather than a simple stopwatch metric. It replicates METR's developer study showing estimand-dependent slowdowns, analyzes GDPval's review-time constraints on GPT-5 speedups, and runs a pre-specified experiment showing that over-trusting an AI drafter leads reviewers to let more errors through while appearing faster.

A research post builds CAT-4B, a Qwen3.5-4B model fine-tuned with a LoRA to obsess over cats, as a control showing that common AI welfare evidence criteria (probe AUC, steering, ablation, self-other dissociation, relief buttons) cannot distinguish an installed disposition from a genuine target state like pain or frustration.

Arize blog post showing how to detect indirect prompt injection in AI agents using Phoenix traces and LLM-as-a-judge evaluations, block cross-account data leaks with scope guardrails, and monitor injection rates with Arize AX online evaluations and alerts.

IntReal introduces IntReal-100K, a real-world interleaved multimodal dataset, and a seven-dimension evaluation suite for diagnosing unified multimodal models, showing that modality-specific training can improve interleaved text-image generation.

Cognition 发布 FrontierCode 2.0 编码基准，采用更广泛的仓库覆盖、更严格的可合并性标准和更大的任务规模，结果显示顶级模型 Claude Opus 5.5 也仅得 45.5%，表明智能体软件工程远未被解决。