Tag
System1-Arena 是一个包含 92k 问题、105 项任务与 138 个闭环环境的基准，用于比较 System-1 概率模型与 System-2 推理 LLM 的离散决策能力。结果显示最大的 System-1 模型在非智能体任务上以四分之一延迟匹配快速闭源 LLM 且校准更好，而推理 LLM 精度最高但成本约为 70 倍；在闭环环境中大型开源 LLM 成功率最高（达 32.9%）。
The paper introduces ASAP, a lightweight framework for evaluating agentic behavior in pretrained base models through interactive end-to-end tasks without converting them into chat agents, demonstrating discriminative performance across software-engineering and deep-research benchmarks.
AgentHorizon introduces a benchmark of 1,373 computer-use tasks with human-recorded trajectories to evaluate agentic judges on long-horizon, multi-application tasks. The study finds GPT-5.5 achieves 80.9% balanced accuracy as the best agentic judge, and reveals that tool use helps some models but hurts open-weight models.
Video-Arena introduces a scalable benchmark that turns competitive games into verifiable tests of spatiotemporal world-state understanding and tracking in videos, evaluating models under native and tool-assisted agentic protocols and finding that both still struggle with state updates over time.
LangChain spotlights Corridor, which uses LangSmith to build agentic security evaluations, trace inference pipelines, and manage memory graphs while securing AI coding agents as they write code. Trusted by security teams at companies like Etsy, ElevenLabs, Lovable, and Mercor.
This paper introduces a ground-truth-as-code framework for evaluating data-science agents on continuously updated data, achieving improved agreement with human evaluators and reduced token consumption.
PACE introduces a proxy method to estimate LLM agentic capabilities using cheaper single-turn tasks, reducing the cost and time of full agentic evaluations.
Microsoft Research's latest newsletter highlights AgentPex, an open-source system for automated evaluation of agentic behaviors; new theoretical work on variance reduction for ranking systems; a call to shift from documents to repositories for human-agent collaboration; and a global challenge on AI value alignment.
GLIDE is an open-source Python library that unifies state-of-the-art Prediction-Powered Inference methods for debiased evaluation of generative AI and agentic systems, enabling annotation savings with valid uncertainty estimates.