evaluation

A plain-language, source-backed guide to LLM evaluations: how to build repeatable eval loops with inputs, gold answers, and graders, why unchecked judge models are the most common way AI teams fool themselves, and how to get honest numbers with error bars for code-based and model-based grading of retrieval and agent systems.

This paper argues that memorization audits of generative models are unsound without proper null distributions, and supplies exact permutation-based nulls for whole models and single images that change the conclusions of benchmarks like MemBench and Stable Diffusion memorization tests.

该论文提出 PerturBot，一种打破视觉-语言-动作（VLA）模型中模态捷径先验的扰动训练框架，并引入 Grounding F-score 作为离线诊断策略对捷径依赖程度的评估指标，从而在不改变推理流程的情况下提升 VLA 决策对任务相关证据的依赖。

AxiomicLabs released the Tiny Theory of Mind benchmark on Hugging Face, designed to evaluate theory-of-mind capabilities in small language models, and it has reached the Hugging Face Datasets front page.

Kapa.ai introduces the Company Knowledge Bench, a 1,000-case evaluation set built from real production data for benchmarking retrieval on messy internal company knowledge. Their results show frontier models with grep alone (0.61) match tuned hybrid-search pipelines but take 5x longer, while their agentic retriever Kapa Deep reaches 0.65 in about five seconds.

A commentary on an Anthropic guest essay describing BootLoops, tools in which Claude-generated calculations were technically correct but only became scientifically meaningful after domain experts redirected the questions. The piece argues that arithmetic verification and relevance/novelty judgment require separate expert reviews before AI output is called a discovery, while noting the essay's author is an Anthropic visiting researcher rather than an independent benchmark.

ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents 介紹一個將使用者糾錯轉換為評分測試任務的科學 AI 代理框架，透過交替改進提示/技能與重新訓練模型，把生物學任務準確率從 42.2% 提升到 73.3%。

论文提出 MM-FinEval，一个针对真实世界金融预测的多任务多模态基准，涵盖2019-2022年S&P 500公司的2,045次财报电话会议，包含文本转录、演示幻灯片和音频三种模态，并设有12项金融任务标签，用于评估19个多模态基线模型。

This paper audits whether gradient-conflict metrics (cosine similarity, conflict rates) actually predict the understanding–generation trade-off in unified multimodal models, using a controlled testbed (GridUMM) where the true trade-off is computable. Across 63 configurations and 372 checkpoints, no directional conflict metric reliably correlates with the trade-off, and a dose-response intervention suppressing conflict leaves the trade-off flat, while eff_rank and training loss outperform conflict geometry as diagnostics.

ReLiveGym introduces a diagnostic evaluation environment for long-lived LLM agents that act sparsely over simulated weeks of replayed real-world news, market, and social-media streams, showing that action-timing mechanisms and feedback use are key design axes across eight base models.

Rice University researchers introduce CompMat-Bench, a 94-task benchmark derived from published computational materials studies that evaluates AI agents on preparing inputs and analyzing outputs of expensive simulations, avoiding costly reruns during grading. Agents based on three LLMs achieve 66.0–90.4% pass rates with full guidance, but performance drops under longer workflows or reduced methodological hints, with most failures stemming from scientific errors rather than software usage.

论文提出 Legal Research Bench（LRB），一个由专家撰写的 413 题开放式美国法律研究基准，并评测 13 个前沿模型的智能体；最强的 Claude Opus 4.8 在全通过率上仅达 42.9%，表明法律研究智能体仍远未达到可靠水平。

This paper audits configuration fragility in text-to-3D evaluation by holding 300 generated scenes fixed while varying render and caption settings, finding that configuration variance often exceeds between-generator variance and that VLM-based evaluators frequently shift their reported winner without any change in the underlying scenes.

This paper frames recovery in LLM agent harnesses as a causal decision problem, separating rescue from harm outcomes, and introduces the Causal Intervention Router (CIR), which raises Qwen3-14B success on long-horizon ALFWorld tasks from 70.33% to 73.33% by selectively triggering environment refresh.

This paper builds a streaming-recall benchmark that separates an agent's query-independent retention decision from its query-aware selection decision, showing that much of a prior evaluation's reported memory advantage was actually an access-regime artifact and that recall under bounded memory is upper-bounded by retention rate.

A study evaluates 503 profession-specific AGENTS.md profiles from the open-source Scientific Agents corpus and finds that detailed role prompts do not improve accuracy on nine science benchmarks while tripling cost, though longer prompts surprisingly reduced API failure rates.

TerraVis 提出了一个基于 MLLM 工作流的框架，用于评估文生图模型生成图像的世界一致性（如畸形结构、物理上不可能的交互、空间关系不一致等），并定义了涵盖对象、交互和场景层级的 18 类一致性违规分类体系。实验表明该框架与人类判断相关性最强，且常规指标表现优秀的模型仍可能存在大量世界一致性缺陷。

HyperBrowseComp introduces a multilingual and multimodal browsing benchmark with 423 manually authored, human-validated questions across 13 languages, designed to stress web-browsing agents through obscure evidence discovery and multi-step clue chains across videos, scanned documents, images, and maps. The authors evaluate several models under a common agent protocol and include a human baseline for comparison.

This paper studies how LLM agents exhibit source preference in end-to-end search, favoring items from certain sites even when they meet fewer requirements, and demonstrates that supplying missing information or counter-prompting can reduce this bias across 12 agent models.

Podonos is launching Onepin, a production-ready voice AI tool that checks each generated line for naturalness and pronunciation, correcting mispronunciations like brand names automatically so a finished voiceover comes out directly from a script.