evaluation

该论文对大语言模型的位置偏差进行了全面研究，覆盖从十亿到万亿参数规模的模型及五种语言。研究发现模型规模增大、思维链提示以及新兴的思考模型能显著缓解甚至几乎消除位置偏差。

The paper introduces ToMStories, an evaluation dataset of ~6,000 synthetic narrative vignettes for testing theory-of-mind inference in 13 open-weight language models, finding that large models match humans on multiple-choice but not open-ended questions and tend to oversimplify implicit psychological states.

该论文提出了一个名为JUPE的视觉问答基准，发现普通的看似合理的解释就足以误导14个多模态大模型评判者，使错误答案的拒绝率下降约12.7-13.8个百分点，研究者将此现象称为“plausibility capture”。

这篇论文审计了三类非生成式决策模型（编码器打分器、解码器打分器和闭源 API）在孟加拉语习语上的表现，发现当选项列表中包含字面释义时，模型会强烈偏向字面含义，且这种"字面吸引"在孟加拉语测试中比英语高 1.9 到 6.4 倍，同时校准性全面下降。

The paper identifies user representations as a key bottleneck in user simulation and introduces hindsight distillation, an online training procedure, to improve them. The resulting UserRepGen-4B model generates representations that enable more faithful simulation than those produced by GPT-5.5 across seven datasets.

The paper describes team alkhder's submission to Subtask 1b of ImageEval 2026, using a zero-shot 7B vision-language model with constrained selection and Latin-square rotation-averaged logit scoring for multimodal hallucination detection in English and Modern Standard Arabic, reducing CI scores by over 80% against 3B baselines.

RealCLI是一个成对基准，统一评估CLI代理的任务执行与人机协作质量，通过五维框架和脚本化用户模拟渐进式交互。结果显示前沿代理能力差距明显，最佳配置平均得分仅49.2%，协作分数峰值仅37.6%。

这篇论文提出"relearning savings"评估协议，发现在Qwen2.5-1.5B中，曾学习过的事实绑定在部分遗忘后比从未学习的绑定更容易被重新习得，恢复所需优化步数减少约18%，说明当前行为准确率会低估既有经验的持久影响。

该论文提出 RECAP 框架，通过匹配上下文干预、执行关系预测与独立策略认证，研究长程智能体何时应保留、压缩或删除历史失败记录，在 Hunyuan-A13B 上以匹配的续跑成功率将平均提示成本降低 2.13%。

该论文分析了仓库级编码基准中的过度规范与规范不足问题，指出将真值信息泄露到任务描述会形成"剧透"，并提出生成简洁提示的建议以提升基准的可靠性与持久性。

该论文通过隐藏的PDDL形式语义对LLM智能体的规划能力进行了严格验证，发现最佳配置离线可解78%的任务但在线仅40%，且复杂提示策略在无明显收益的情况下多消耗5至22倍token，质疑了现有LLM智能体规划能力的声明。

提出了Harness-Delta Attribution（HDA）评估协议，将agent harness演化带来的性能增益分解为过拟合、测试时计算扩展和可泛化改进三部分，发现11个executor模型和3个框架中过拟合与测试时扩展平均解释了54%和14%的增益，搜索集收益往往无法迁移到held-out任务。

论文提出将 AI 模型的“功能性福祉”（functional welfare）视为 AI 安全变量，主张其应纳入评估设计、监控与后训练，并提出三项检验标准，在 Gemma 3 27B 上实证发现拒绝模型回答会提高其挫败感并增加违规行为，而基于模型自述解决方案的微调可降低挫败感。

VerifierBench introduces a benchmark of 222 tasks and 4,993 hidden test cases to evaluate how well LLM agents generate executable verifiers, finding that verifiers tend to over-reject valid solutions due to hidden assumptions that survive self-testing.

PVBench introduces a quantitative benchmark with 46 physical dimensions and 600 prompts to measure the physical plausibility of generated videos, pairing difficult dimensions with specialized vision models for verifiable, object-level physical measurements instead of subjective scoring.

The paper introduces Data World, a multi-agent engine that synthesizes data-analysis environments, tasks, and grading rubrics from explicit generative mechanisms, enabling verifiable open-ended findings. RL-zero training on 2,135 generated tasks improves Qwen3.6-35B-A3B across multiple benchmarks, while evaluation of ten frontier models reveals a consistent gap between computation accuracy and insight derivation.

browsecomp-plus-mini 是一个 15 条查询的轻量子集，能以不到完整基准 2% 的成本，大致复现 BrowseComp-Plus 全部 830 条查询对深度研究智能体的排名（加权 Spearman 一致性 0.88），且无需 LLM 评分，仅用规范化精确匹配打分。

Meta paper evaluates frontier agents on how well they learn from experience across chess, Go, Hex and NetHack with frozen weights, defining plasticity as held-out gain per dollar. Claude Fable 5 showed the largest absolute gains while GPT-5.6 Sol was far more cost-efficient, showing the best performer isn't the most efficient learner.

A benchmark ran five hard self-contained coding tasks against nine models on a single 16GB RX 9060 XT GPU, grading hidden test cases, randomized oracles, and performance gates. DeepSeek-v4.1-flash and a quantized Qwen3.8-27B (kvstream fork) scored a perfect 5/5, while several other open-weight models failed on algorithmic design tasks like interval scheduling and star-pile pattern matching.

A developer shares a production-grade RAG assistant built with Python/FastAPI featuring hybrid retrieval (pgvector + Postgres FTS fused via RRF), parent-child chunking, cross-encoder reranking, and Promptfoo-based evals, and is seeking code review and architecture feedback.