evaluation

An open-source document-parsing benchmark called OmniParse Bench was released, running 16,288 tests across 2,937 pages in 92 languages to compare OCR solutions like DeepSeek-OCR, PaddleOCR-VL, and Surya, showing no single parser wins on every document type.

A developer describes their nightly job that mines Claude Code and Codex sessions for corrections and turns them into approved lessons, including a bug that silently skipped verification for a week and the open question of how to measure whether lessons actually change agent behavior.

该论文批判了传统机器翻译评估方法（MQM、ESA）的局限，提出了RATE（Refined Assessment for Translation Evaluation）协议和由专业语言学家标注的高质量英俄翻译评估数据集，发现SOTA机器翻译系统在准确度上可能已超越人类翻译，但在流畅度上仍存在差距。

The paper introduces a controlled benchmark of French surface variants (apostrophe omission, diacritic removal, keyboard typos) across seven tasks with 12,600 inputs, showing that accuracy drops and Brier score/NLL rise with perturbation intensity, while calibration combined with input restoration improves reliability.

This paper introduces BoardSafety, a benchmark built from vendor datasheets to evaluate whether LLMs can judge if an operation is safe for a specific development board. Results show current models, even with access to board-specific rules, cannot reliably make safe determinations.

该论文提出一个统一评估框架，使用 Jev 1.13.0 作为案例研究 Structured System 1 模型在时间序列分类、事件类型预测、油温方向及股价走势预测等任务上的表现，结果显示其在显式时间模式和事件预测上较强，但在许多数值序列分类和股票走势预测上接近随机水平。

该论文审计了 1999 至 2026 年间的 113 项研究，发现只有 9 项主要评测真正测量了真实人类对系统所选行动的响应，其中仅 2 项还将显式的响应预测与实测响应进行比较；在 25 项语言模型相关工作的全文检查中没有发现符合这些条件的实验。作者提出了一套直接预测评测标准，并设计了一个用于谈判助手的单步实验来区分任务成功、当前状态推断与真实响应预测。

NewsBench 是一个由专家主导的基准，围绕新闻业、政策和情报分析的编辑标准，从来源质量、事实性和中立性三个维度评估前沿 AI 系统的新闻报道认知质量，并揭示了四个前沿模型各不相同的失败模式。

该论文提出一种将专家判断蒸馏为可规模化自动评估系统的方法，构建专家网络并将其推理过程编码为结构化编辑标准，在AI新闻报道上验证了专家校准的自动评审显著优于未经校准的前沿模型。

该论文提出实体级分析作为补充视角来评估历史文档OCR质量，使用Berrutti档案馆175份文档及5,209条人工标注实体，比较了序列识别OCR系统、OCR专用视觉-语言模型和通用视觉-语言模型，发现不同评价标准下OCR排名会变化。

AOS-Bench, from Tsinghua University and Columbia researchers, evaluates LLM coding agents as workload managers handling streams of tasks under shared CPU and memory limits, finding that isolated task success doesn't carry over to streaming scenarios and that simple fixed-concurrency policies often match elaborate agent-managed ones.

A controlled evaluation of five public EEG foundation models plus an in-house JEPA-style model, showing that random-initialisation, band-power baselines, and validation-selected probe layers substantially alter reported pretraining gains on sleep, emotion, and motor imagery benchmarks.

This NeurReps workshop extended abstract evaluates whether pretrained LeWorldModel preserves local action geometry, measuring action sensitivity via Gram matrices across perturbation scales on Reacher and OGBench Cube, finding significant directional discrepancies at small scales and recommending scale-specific evaluation.

VibeVideoBench introduces a benchmark of 239 tasks evaluating multimodal agents that generate videos by writing rendering code, finding that GPT-6 Astra and Claude Opus 5.5 lead but that time-management failures cause most lost videos.

Simon Willison publishes a research report evaluating webframe, Jeff Lindsay's lightweight Go tool for running web pages like programs in a headless system web view, finding it fast and comparable to shot-scraper but with rough edges like CSP failures and missing HTTP metadata.

This paper proposes a new method for measuring whether LLM agents have internalized skills from public skill hubs, arguing that gain-based retirement rules fail to distinguish internalized skills from uncovered ones. It introduces a probe-based unaided scoring approach with cross-model validation and releases 1,251 skills with task data across four models.

DMAIC-Bench 提出「自我克制」（self-restraint）能力评测，包含 15 个工业案例共 97 个任务及配对反事实变体，发现 13 个模型在 NO-GO 决策上的执行路由准确率平均仅 19%，远低于 GO 决策的 92%。

SWE-Shift 是一个评估编码智能体在执行过程中遭遇环境、依赖与需求变化时表现的新基准，包含120个场景和12类变化。在评测的17个编码智能体系统中，Shifted 条件下的成功率均低于 Clean 条件，GPT-6 Astra 从 67.35% 骤降至 24.79%。

PersonaScienceBench introduces a benchmark of 200 tasks across four scientific domains that evaluates how AI agents collaborate with simulated researchers holding diverse personas, measuring interaction quality alongside task completion.

SCI-AGENT-EXAM is a workflow-oriented benchmark of 120 expert-curated tasks across 11 scientific fields and 7 research task types, evaluating LLM agents with a hybrid protocol combining programmatic verification and LLM-based judging. Experiments on 10 frontier agents show GPT-6 Astra leads with 80.9 overall but only 26% of runs exceed the 0.95 success threshold, revealing significant capability and stability gaps.