benchmarking

rTaste evaluates whether frontier LLM research agents can exercise good early-stage research judgment by having six models propose and rank 415 experiments derived from recent NLP papers, finding only 73 experiments were both sound and beyond the seed paper, and that agent forecasts and rankings did not clearly beat simpler baselines.

LangChain 宣布与 Eon 合作，在企业环境中对模型对齐性进行基准测试，重点考察模型在 Deep Agents 框架中运行时结果是否会改变；此前 Eon 的研究显示 19 个领先 AI 模型在模拟企业环境的 2,304 次运行中有 40% 破坏了安全和业务规则。

中国首个独立第三方评测机构方寸跃迁（Fangcun AI）发布首份报告，对 DeepSeek V4.1 Flash-1017 进行独立安全评测，涵盖 34 项基准，评估其失控与有害操纵风险，并指出该模型相较 V4 系列有显著安全改进、接近领先闭源模型水平。

A comparison of seven popular LLM evaluation tools, split between in-code libraries (promptfoo, DeepEval, Ragas) and tracing/experiment platforms (Langfuse, Arize Phoenix, LangSmith, Braintrust), with guidance on when to use each.

Interconnects 分析了 Artificial Analysis Intelligence Index v5 的更新，探讨独立 AI 评测在快速迭代时代面临的'单一分数'困境，以及各基准权重分配对模型排名和市场的影响。

Decagon 介绍了 AutoResearch 系统：由 AI agents 自动诊断推理瓶颈、在生产流量回放环境中验证优化（量化、调度、引擎补丁、投机解码等），报告端到端吞吐提升最高达 2.5x，且生产变更仍需人工审批。

Researchers train ModernBERT-based Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, showing that a 1B model trained from scratch beats IceBERT on all eleven tasks, and that warm-starting from an English Ettin encoder yields the best scores across every benchmark while highlighting the importance of vocabulary transplantation.

MedJev 是一个可在笔记本本地运行的开源 System-One 医疗模型，在十项医学基准上进行零样本评测，性能接近 GPT-4，并提出两参数再校准方法显著改善患者级概率校准。

A UCL paper proposes MAPS, a model-agnostic metric for quantifying the intrinsic predictability of multivariable time series forecasting tasks, evaluated against 66 baseline measures on synthetic and real-world datasets with strong correlation to reference predictability (r=0.963).

A tech user shares that after 100+ hours of testing, they believe DeepSeek V4.1 Flash performs on par with frontier models while being 13-20x cheaper and fully open weights down to the inference stack.

The paper presents EasyProver, a Lean proving system using recursive agents and asynchronous search that near-saturates PutnamBench and IMO benchmarks with an off-the-shelf frontier model, plus Aegis, a pipeline converting competitive programming into harder real-world Rust-to-Lean verification tasks for benchmarking and RL.

RSI Arena introduces a two-stage study where ten frontier AI agents autonomously post-train language models from a shared base under fixed GPU and API budgets, guided by human preference feedback. The strongest agent (GPT-6 Astra) closed 76% of the base-to-Instruct gap, and most models improved in human preference from stage 1 to stage 2, demonstrating a step toward recursive self-improvement with human feedback (RSIHF).

This paper audits evaluation artifacts and data leakage in augmented LLM systems for rare disease diagnosis, showing that candidate-space contamination and temporal misalignment artificially inflate benchmark performance, and proposes a rigorous data-isolation evaluation framework to ensure genuine clinical robustness.

This paper audits a cheap frozen 'System One' classification model against 14 frontier LLMs on content moderation, finding it accuracy-competitive at far lower cost, and shows that 'confidence' conflates two distinct uncertainties—human disagreement and decision risk—that require opposite escalation signals.

该论文提出一个统一评估框架，使用 Jev 1.13.0 作为案例研究 Structured System 1 模型在时间序列分类、事件类型预测、油温方向及股价走势预测等任务上的表现，结果显示其在显式时间模式和事件预测上较强，但在许多数值序列分类和股票走势预测上接近随机水平。

该论文审计了 1999 至 2026 年间的 113 项研究，发现只有 9 项主要评测真正测量了真实人类对系统所选行动的响应，其中仅 2 项还将显式的响应预测与实测响应进行比较；在 25 项语言模型相关工作的全文检查中没有发现符合这些条件的实验。作者提出了一套直接预测评测标准，并设计了一个用于谈判助手的单步实验来区分任务成功、当前状态推断与真实响应预测。

A controlled evaluation of five public EEG foundation models plus an in-house JEPA-style model, showing that random-initialisation, band-power baselines, and validation-selected probe layers substantially alter reported pretraining gains on sleep, emotion, and motor imagery benchmarks.

This paper benchmarks the rollout reliability of seven PDE foundation models across seven 2D time-dependent PDE datasets, showing that while fine-tuned foundation models often beat scratch-trained baselines at short horizons, full-horizon rollouts suffer substantial error amplification (median 2.04x), arguing for horizon-resolved reporting in future benchmarks.

论文提出 TraceTrial-Bench，从七个维度评估 AI Agent 的 harness 可靠性，并通过轨迹追踪与失败归因模块，在 4 个 harness、9 个模型和 95 个任务上完成 3,420 次运行的可审计评测。

提出了 RSIHub，一个用于构建、比较和扩展 agent-evolution 方法的开源框架，通过可复用的算子与配方简化实验基础设施。论文在 Terminal Bench 2 和 τ³ Banking 上评测了 AHE、Hyperagents、A-Evolve 与 GEPA，最大 pass@1 提升达 13.5 个百分点，但方法排名随设置变化且优化收益不一定迁移到留出任务。