Tag
Kiln的开发者发布了开源框架Seahaven，用于构建强化学习和评测用的AI agent环境，提供隔离实例、冻结fixture、状态diff和可复现运行，并附带一个可兼容官方Stripe SDK的模拟Stripe世界。
Centific Research introduces a Trust Layer for Agent Evaluation, a post-hoc framework that verifies benchmark scores via four deterministic checks—grading support, traceable computation, honest completion claims, and run-to-run stability. Applied to five agent configurations on 108 Agents' Last Exam tasks, only 22.6% of recorded passes clear all checks, revealing widespread fabricated computation, false success claims, and score instability.
提出 Agentic BBO-Bench 跨领域基准，在统一预算协议下评估 LLM 智能体做黑盒优化的能力；实验显示 agentic BBO 在五个领域均优于直接 LM 方法，并在四个领域超越最佳数值优化器，GPT-6Astra 与 DeepSeek-V4.1-Flash 位于性能-成本 Pareto 前沿。
Omar Santos presents a guide on using Jev, TypeSafe AI's decision model, as an LLM judge for agent evaluations, showing how reading full trajectories (tool calls, results, errors) rather than just final replies increases reliability and consistency for production-grade evaluation and monitoring systems.
SWE-Race is a new benchmark of 188 real concurrency bugs (races, deadlocks, cancellation issues) from merged PRs in ~100 Python projects, graded by project tests in network-free containers with git history stripped. Results show GLM-5.3 Flash (85% single attempt, 82% with retries) roughly matching GPT-5.6 Luna (81%), with most model differences concentrated in the hard half of tasks.
Era by Eon is a free tool that generates a complete simulated enterprise environment across Salesforce, Slack, Jira, Zendesk, Gong and Deel, enabling safe, repeatable testing of AI agents without real company data.
Eon's Era generates a complete simulated enterprise with exact ground-truth answers, writing records into Salesforce, Zendesk, Jira, Slack and Gong via MCP and REST interfaces, enabling deterministic evaluation of enterprise agents without human or LLM judges.
VisAudit is a new benchmark for evaluating multimodal agents on autonomously diagnosing, repairing, and verifying flawed data visualizations, spanning 1,900 flawed charts across 21 chart types and 10 flaw categories. Experiments show the strongest multimodal model fully recovers only 47.4% of flawed charts in the autonomous-repair setting, revealing a substantial gap toward human-like visualization review.
A practitioner outlines 15 must-build projects for AI Evals Engineers, covering trajectory grading, shadow routing, calibrated LLM-as-a-Judge, CI/CD regression gates, adversarial RAG testing, DPO fine-tuning flywheels, statistical significance, agent red-teaming, drift monitoring, cost-quality dashboards, counterfactual replay debugging, synthetic edge-case generation, context-window eviction tests, dataset contamination checks, and public eval methodology teardowns.
Stanford researchers develop a Bayesian variance-decomposition framework using Generalizability theory to diagnose reliability in agent benchmarks, showing that adding more tasks cannot fix uncertainty caused by scaffold coverage and that pooling diverse benchmarks is a cheaper path to reliable rankings.
Adobe paper proposes evaluating agents against code that recomputes the correct answer from live data, with an AI grader matching human experts 29% better and using 16% fewer tokens than written descriptions on 53 test cases.
A workshop paper presenting a single-instance case study in which an AI agent is asked to rediscover a hidden invariant of degree-four Blaschke curves from numerical data, yielding a homogeneous cubic that predicts unseen configurations with machine-precision residuals, while honestly noting limitations such as a deterministic polynomial-fitting baseline matching the agent's result.
This paper studies how to specialize large language models for generating executable algorithmic trading code, showing that framework-oriented continued pretraining plus supervised fine-tuning on validated request-to-code pairs greatly improves success on the QuantCode-Bench benchmark and agentic strategy generation, while also revealing a capability-retention failure where domain specialization degrades structured tool calling.
OSWorld-Science introduces a benchmark and evaluation environment for computer-using VLM agents on scientific software, covering 146 expert-designed tasks across domains like molecular drawing, pathology imaging, statistics, and physical simulation with artifact-based evaluators and a special agent harness.
The paper introduces LibraryDesignBench, a two-phase benchmark that has one AI agent write software libraries from underspecified requirements, then scores each library by how effectively other agent implementers use it to solve programming problems in minimal code. The evaluation metric combines squared pass rate with solution simplicity relative to a production-library reference.
The article highlights that model benchmarks fail to evaluate production agents accurately due to system complexities, and introduces TRACES by Apodex for a more comprehensive evaluation, referencing a research paper.
ControlScope is a method for LLM agents to revise workflows, comparing repair strategies across filesystem tasks, ALFWorld, and AppWorld to enhance reliability.
The author compared two 9B models for local agent seats, finding one model silently fails by writing tool calls as prose and making other errors that bypass detection, emphasizing the need to test for distinct failure modes.
This paper derives sharp theoretical limits for honest uncertainty in repeated evaluation under a hard budget, with applications to language model and agent benchmarking, demonstrating practical improvements in interval width and MSE.
This paper measures how AI agents contaminate stores they write to, identifying a threshold for error propagation and validating findings with synthetic and real-world Wikidata data.