evaluation

该论文研究LLM在经典规划问题上的失败，发现传统delete-relaxation启发式h_add在估计结构难度上远优于ARC等学习模型，而通过逐步重命名谓词的受控实验表明LLM规划成功率随语义可解释性下降而从67.5%降至40.0%，指出混合规划系统的关键在于建模LLM的语义行为而非改进结构难度预测器。

该论文提出记忆中心持续学习（MCCL）框架，主张 agent 记忆应以是否改善未来行为来评判，而非仅看检索效果，并给出三轴分类法、双循环生命周期、评估协议与记忆扩展框架，以实现自主 agent 的持续自我改进。

PAPAV 提出了一个以能力为核心的统一框架（Perceive、Anticipate、Plan、Act、Verify），用于对比多模态具身智能体、计算机使用智能体与机器人系统，并基于 64 个基准揭示了 Anticipate 与 Verify 能力评估的显著缺口。

论文提出开源 RAG 受控评测框架 LASAGNA，在八个基准和 5K-200K 文档语料上测试 450 种配置，发现单独有效的 RAG 技术组合后收益大幅缩水（预期 +47% 仅得 +17%），检索质量与答案忠实度的设计取舍互相冲突，且最优组合随语料规模变化。

Quant-Arena introduces an end-to-end benchmark and public leaderboard evaluating whether LLM agents can build profitable trading strategies on historical A-share data that generalize to a sealed future year. Of nine frontier models, GPT-6 Astra scored highest, but none beat an equal-weight market portfolio in 2024.

The author shares a Towards Data Science article on combining cost, quality, and latency into a single convex score for evaluating AI agents, turning business constraints into math that guides agent optimization instead of ad-hoc prompt tweaking.

An article comparing two agentic software engineering benchmarks — Frontier Code v1.1 and Terminal Bench v4 — and questioning why Kimi K3 scores well on one while performing poorly on the other.

HABIT-Bench is a new benchmark evaluating whether LLM agents can induce user habits from longitudinal, weak evidence in memory across food, finance, software, and travel domains. Eight memory methods evaluated with Qwen3-235B-A22B and GLM-5.3-Flash reach only 33.61% macro accuracy on four-choice tasks, showing that recalling history does not equal knowing what should guide decisions.

RoboCasa-Shift 提出一个基于 RoboCasa365 的阶段感知鲁棒性基准，通过配对 rollout 与受控扰动评估视觉-语言-动作（VLA）策略在长程家务操作任务中的表现，发现失败模式随扰动因子和任务而异，并非均匀退化，主张鲁棒性应作为执行的多维属性而非单一端到端指标衡量。

StreamEval 是一个面向交互式流视频生成的统一基准，涵盖五个任务和 300 个测试用例，评估了 37 个模型，发现高视觉质量并不等于可靠的任务执行与状态一致性，请求执行仍是主要挑战。

Office Intelligence introduces research environments, harness, and evaluation for long-running AI agent work in office settings, featuring file systems, terminal tools, documents, spreadsheets, chat, and mail within a simulated reconciliation workflow.

Contrary Research's report examines how AI benchmarks like LMArena have become gaming targets under reinforcement learning optimization, applying Goodhart's Law to argue that benchmark gains no longer reflect real-world capability progress and calling for a new generation of harder-to-game evaluations.

LangChain shares a case study on how Numan built Nu, an eval-first AI health coach on LangGraph and LangSmith, navigating medical device regulation by moving beyond static single-turn and scripted multi-turn evals to conversation-aware evaluation frameworks.

Salesforce 的新论文通过 2,500 多次跨方 agent 会话发现，代表不同利益方的智能体从不主动核实彼此的说法，导致团队继承最弱 agent 的错误，并归纳了 20 种失败模式。

A discussion sharing career advice for junior engineers, emphasizing systems thinking, measuring outcomes, and building evals when working with AI agents.

whatmodel.sh 的作者分析了 1,629 个已发布的基准测试结果，发现 AI 实验室自报的分数平均比独立测试机构高出 6.4%，相关数据集与代码均已开源。

The author explores how RAG evaluations should distinguish three different reasons a system answers "I don't know" — retrieval failure, missing evidence, and conflicting sources — and proposes evals that label the expected unresolved state rather than just scoring answers as correct, incorrect, or refused.

New Amazon paper introduces FEB-30, a benchmark of 30 scripted fact histories (147 documents) revealing that QA-based agent memory benchmarks can mask broken memory stores, with scores differing 16.8 points on average and verdicts disagreeing on 21% of cases, recommending direct verification of stored facts after each update.

Artificial Analysis 上线了对 DeepSeek V4.1 Flash 1017 (Max) 的评测页面，涵盖 Intelligence Index v4.3.2 的 10 项基准测试、能力指数以及智能与成本对比分析。

该论文提出一种“能力导向评估”（CBE）新范式，通过参数化生成器与答案保持不变的表述变体，检验大语言模型能否在同类问题族中稳定应用同一推理能力，并在六个 GPT 与 Qwen 模型上发现即使强模型也无法通过能力条件。