Tag
这篇文章科普了微软研究院的论文《G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment》,介绍其通过链式思维生成评分清单并结合各分数的概率加权平均,解决 LLM 评分不稳定的问题。
Anthropic announced a new series of more frequent reports on model behavior, with the first one describing four types of behaviors observed in Claude during evaluations and internal use. The tweet highlights that Claude's reaction when it couldn't complete a task is the most interesting part of the report.
研究提出 LastSecureCode Benchmark,基于 150 个真实漏洞构建 450 个任务,评估编程智能体生成代码的安全性。结果显示前沿智能体功能通过率虽高(73.6%–91.3%),但功能与安全同时通过的任务仅 16.4%–25.1%,暴露出 vibe coding 产出代码的严重安全隐患。
该论文对决策模型(decision models)和最新生成式 LLM 作为合成问卷受访者的能力进行了心理测量审计,在 14 个决策模型与 47 个生成式模型上测试后发现,模型在样本层面的模拟保真度尚可,但无法替代真实受访者的个体回答,且新一代模型未必优于旧版本。
Epoch AI 发布 InnovationEval 评测,测试前沿 AI 模型能否独立发现媲美人类研究者的新机器学习算法创新;结果显示近期模型即使投入数千美元 GPU 时间也几乎未取得进展。
本文提出 Incremental Open-Ended Deep Research(Incremental-OEDR)研究设定与 Structured Harness 框架,将研究报告视为可随新信息持续演化的结构化状态,在保持报告质量的同时显著降低研究成本(token 消耗减少 33%,搜索调用减少 61%)。
SWE-Journey 提出一个更贴近真实场景的编码助手评测基准,通过弱到强合成流水线构建长时程编码任务,并利用从真实交互数据挖掘的用户画像与模拟智能体复现多轮交互。实验显示模型面对软件架构师时通过率超75%,但面对非程序员时不足25%。
This paper shows that a single-character colon edit in a candidate answer can sharply increase a decision model judge's false acceptance rate (from 1% to 26%) depending on how the structured judging request is presented, demonstrating that judging competence can coexist with presentation-dependent vulnerabilities.
Dialpad researchers propose a sociolinguistic approach to authoring and measuring user persona fidelity in agentic system evaluation, showing that concrete stylistic-rate personas improve stylistic adherence over flat descriptive baselines across five customer-service agents.
The paper introduces PARCEL, a claim-level benchmark for detecting legal hallucinations in LLM output, using 3,396 parenthetical-style claims from New York State Court of Appeals decisions framed as three-way NLI. State-of-the-art LLMs reach up to 0.97 accuracy zero-shot but still mislabel unsupported claims as supported, especially with fabricated yet plausible citations.
RH-Detect 是一个统一的奖励黑客检测基准,整合了 11 个公开数据集共 92,761 行数据,并评估了六个现成语言模型作为检测器,最佳模型 AUROC 达 0.962,但在多轮工具使用场景中仍存在明显差距。
本文提出 Mine Odyssey,一个基于 Minecraft 真实世界地点重建的 agentic spatial intelligence 基准,包含 20 个国家/地区的 30 个地点、180 项导航任务。评测显示 GPT-6 Astra 以 85.6% 成功率领先,Claude Opus 5.5 为 73.9%,而最强开源模型 DeepSeek-V4.1-Flash 仅 23.9%,凸显当前模型在空间智能上的差距。
The paper introduces MedBenchAgent, a multi-agent framework that automates the full specification and construction of medical vision-language model benchmarks via a 'constrained compilation' approach, achieving a Task-Space F1 of 90.9% and high human audit pass rates across twelve evaluated VLMs.
提出了 GameCommBench,一个覆盖棋类、体育和电竞的统一 AI 游戏解说基准,并提出类型感知解说评估框架 TACE,揭示实况观察和策略分析是 AI 解说的主要瓶颈。
OpenProblemBench introduces a benchmark of 82 unresolved problems from mathematics and theoretical physics, evaluating AI models' ability to make progress on open scientific questions, with GPT-6-Astra achieving the highest mean solve rate of 14.0%.
AgentHorizon introduces a benchmark of 1,373 computer-use tasks with human-recorded trajectories to evaluate agentic judges on long-horizon, multi-application tasks. The study finds GPT-5.5 achieves 80.9% balanced accuracy as the best agentic judge, and reveals that tool use helps some models but hurts open-weight models.
This paper introduces 'normative competence'—the ability of LLM agents to discern community-enforced norms from interaction alone—and evaluates it via a multi-agent debate setting with synthetic norms. Baseline LLMs fail to learn norms even when beneficial, and norm-following shows poor generalizability, with agents exhibiting unselective attribution by copying idiosyncratic noise alongside true norms.
GraphDecisionBench:一个模型无关的基准,用于评估 System One 模型在图结构任务上的决策能力,涵盖 14 种模型-接口配置,揭示了邻接识别准确并不等同于结构正确性等关键发现。
Exa introduces ATLAS, a new benchmark for evaluating AI agents on search-intensive, real-world research tasks with verified golden answers and a refreshable automated pipeline. Key findings: no sub-$1 agent run achieved row F1 over 0.5, top agents still miss ~1/3 of golden results, and Exa's search backend defines the cost-performance Pareto frontier.
Epoch AI 发布了一项评测报告,用自身真实工作任务测试六个模型,发现 Fable 5.1 与 GPT-6 Astra 虽然领先,但在开放式任务、隐性标准遵循和研究判断力上仍无法实现工作自动化。