evaluation

提出 RIVER 框架，将 LLM 多智能体交互分解为原子主张，评估收敛、探索与保留三个信息动态维度，并在 40 个新闻生成案例上比较五种交互策略的信息流动机制。

论文提出 SAGE——一个统一的10候选图像选择基准，将 VWSD、VerSe 和 WikiDiverse 重构为共享的词项中心协议，并提出训练无关的 Gloss-Augmented Reasoning (GAR) 框架，使用 Gemma3-12B 将 Hits@1 从 65.90 提升至 71.22。

This arXiv paper studies how to distributionally align LLM output distributions with human reference distributions and evaluates how well this alignment generalizes to unseen questions, finding that same-domain generalization works but cross-topic generalization is unstable and can degrade model capabilities.

This paper introduces the Schema Coherence Inventory (SCI), a 96-item forced-choice psychometric tool measuring whether beliefs assigned to persona-conditioned language models actually organize their cognition versus merely inducing a mood, validated across 107 conditions and five model backbones.

SOMA is a new benchmark for fine-grained egocentric body-motion understanding in multimodal models, with 1,632 human-verified questions evaluating 37 MLLMs. Humans reach 68.5% accuracy while the best model, Qwen3.6-27B, achieves only 50.0% zero-shot, revealing significant gaps in body-motion reasoning.

Co-Design Bench is a benchmark of 273 real-user multimodal coding tasks with 2,331 expert acceptance criteria, evaluating agents via a judge-then-refute protocol across 10 model endpoints, where the best endpoint (gpt-6-astra) passes 69.9% of criteria but fully satisfies only 26.4% of tasks.

论文提出IEC框架，研究语言模型在受控指令负载（如无意义的单词替换规则）下的行为与性能退化，揭示了规则数量、替换密度、规则位置等因素遵循的数学规律，并在GPT-4o mini上验证，为模型建立可复现的"能力护照"。

Independent safety evaluation of DeepSeek V4.1 Flash-1017 across 34 benchmarks finds substantial safety improvements over prior versions and near-parity with leading closed-source models, recommending process-level observability, incremental safety testing, and private benchmarks to address loss-of-control and harmful manipulation risks.

该论文分析了结果奖励的强化学习（Outcome RL）对大模型数学与物理推理能力的影响，发现 pass@1 的提升主要来自更快完成回答（finishing）和更高可靠性（reliability），而极少带来真正解决新问题的能力（reach）。

Omni-InteractScope introduces a benchmark of 1,479 human-annotated interactions across eight task groups to evaluate how reliably omnimodal LLMs deliver timely tool use, query-free reminders, and sustained guidance in continuous audiovisual streams, finding current models fall far short.

一篇关于大模型时代可信递归自我改进（RSI）的综述，从改进有效性、约束保持与有效监督三个视角组织了自我改进中的风险、保障措施与评估框架，并讨论了开放研究问题。

iPrefBench is a user-centric benchmark evaluating whether personal mobile-use agents can infer missing preferences from activity history or ask for clarification in an executable iOS environment with a simulated user. Evaluation of 7 frontier agents shows a large gap: high action completion (up to 91.6%) but low full task success (up to 47.7%), with agents especially struggling to request necessary ask-only information.

该论文提出了一个统一的“工作智能体”（work agents）框架，将基础模型与运行时框架结合以可靠执行复杂数字任务，引入持久性与开放性两个维度，并系统分析能力族、模型与框架优化及评估方法。

该论文将生成模型的推理 effort 视为'信任品'，通过反事实实验量化不同 effort 级别对成本与准确率的影响，发现从低 effort 起步、失败后再升级的策略在多数模型和基准上以更低成本达到或超过固定最高 effort 的效果。

The paper introduces a framework of 'memory benefit' and 'memory tax' to measure the net value of dedicated agent memory systems over naive RAG, finding that many memory systems underperform direct retrieval across benchmarks. It analyzes how memory processes can remove or obscure evidence and calls for evaluating the net value of memory rather than standalone performance.

AgentProofBench introduces a benchmark for reusable verification in web agents, linking frozen acceptance rules to execution traces across 97 tasks and 79 families on four web services, establishing baselines for program-gate correctness and reuse using DeepSeek Flash/Pro.

AdaPRA introduces a persistent coding agent harness with adaptive rubric-guided process supervision that accumulates knowledge across attempts, boosting an 80B Qwen model's leaderboard percentile by 24.5 points on DSPredict-Easy and 14.8 on DSPredict-Hard while reducing process errors in data science agents.

RSIHub 是一个开源框架，为智能体进化（agent-evolution）研究提供可组合的共享基础设施，标准化方法的构建、比较与扩展，并在 Terminal Bench 2 和 τ3 Banking 上评测了 AHE、Hyperagents、A-Evolve 和 GEPA 等方法。

该论文提出 CRM-Agent-Bench，一个基于执行的基准，包含 555 个有状态多轮任务，用于通过严格的契约检查评估工具使用型语言代理在九个企业业务领域中的结果、轨迹、可运行性与合规性；在十六个系统上评测中严格成功率介于 64.32% 到 83.60% 之间。

An open benchmark placed 13 AI models in a simulated GP consultation (195 runs across 5 hand-written cases), showing models nearly always named the right diagnosis but varied widely (27–85%) on history-taking, safety and appropriate testing — with notable failures like prescribing penicillin to an allergic patient. Results, transcripts and code are public, with a paper planned for arXiv.