Tag
CLAP proposes a closed-loop method for domain agent post-training that converts noisy business data into structured SFT and preference samples, integrates reward/KL diagnosis, offline gates, and application-chain replay to decide adapter release. Experiments on five manufacturing batches show modest average gains and highlight that regression and high KL risks require an integrated data-training-evaluation-release loop rather than relying on a single score.
MedEvoEval is a longitudinal evaluation framework for doctor agents that simulates outpatient episodes, assessing how agents acquire evidence, use resources, and evolve across episodes through memory and reflection mechanisms.
Introduces SEATauBench, the first agent-focused evaluation framework for Southeast Asian languages, adapting τ²-Bench to Mandarin, Vietnamese, Thai, Indonesian, and Filipino, and reveals a significant capability gap when moving from English to localized settings.
The article discusses the challenge of evaluating LLM-based agents that perform multi-step reasoning, noting that scoring only the final output is insufficient because agents may take wrong paths and recover by accident, and raises questions about how to evaluate the trajectory without manual review.
This paper argues that aggregate-score leaderboards for LLM agent benchmarks fail to capture deployment-relevant dimensions and show rank instability. It proposes ranking configurations by predictive validity—the correlation between in-sample and out-of-sample rank—and introduces a twelve-tier measurement apparatus along with falsifiable out-of-distribution criteria.
hermesbench v0.1 is a new benchmark for evaluating local models on Hermes Agent tool-calling, featuring real agent harness, deterministic verifiers, and hardware telemetry. It includes 43 tasks across 11 categories and is designed to measure how well models use the Hermes Agent.
Explains the concept of online evaluations for AI agents, which measure agent performance on live traffic over time, as opposed to offline evaluations that use fixed datasets.
Jugmax is a tool that evaluates AI agents by analyzing their full execution trajectory rather than just final outputs, identifying inefficiencies, errors, and wasted tokens. The founders offer free evaluation for two production agents to validate their product.
Introduces XBCP (Cross-lingual BrowseComp-Plus), a benchmark for evaluating deep research agents and retrievers in cross-lingual and multilingual settings. Results show significant performance degradation when evidence is in a different language from the query, highlighting both retrieval failures and agent-side difficulty in integrating language-mismatched evidence.
The author shares an approach to vetting library health using a deep research agent, discovering that the most valuable signal is when the agent flags disagreements among its sources rather than producing polished, false-confidence summaries. Apodex notably surfaced contradictions clearly, making it easier to adjudicate trust.
MTG Bench evaluates how well LLMs can play Magic: The Gathering using an MCP server for library operations, showing both successes and failures in complex game actions.
This paper introduces layer-isolated evaluation for LLM agents, decomposing a production agent into architectural layers each tested with a deterministic, no-LLM harness. It demonstrates that per-slice baseline testing localizes regressions that aggregate metrics mask, validated by controlled regression injections across multiple tenants.
This paper introduces OfficeEval, a benchmark based on China's National Computer Rank Examination (NCRE) to evaluate LLM agents on complex Office automation tasks. Frontier models achieve at best 36.6% in single-turn and 68.8% with agentic systems, far below human-level performance.
This paper introduces STAGE-Claw, an automated framework for building and evaluating realistic personal-agent scenarios in state-based computing environments, enabling scalable, state-based evaluation of LLM-powered agents.
Agent Arena is a new leaderboard that evaluates AI models on real-world agentic tasks such as coding, research, and file analysis, using signals like task success, steerability, and recovery, with GPT-5.5 High leading.
A new benchmark, FML-Bench, reveals that recent improvements in MLE-Bench scores are largely due to better base models and increased search budget rather than algorithmic advances.
This paper introduces the Causal Sensitivity Score (CSS), an interventional metric that evaluates whether clinical LLMs and agents appropriately update their recommendations when patient inputs change along clinically meaningful dimensions. It reveals hidden capability profiles not captured by standard coverage-based metrics, exposing safety blind spots and structural responsiveness deficits.
ToolFailBench, a diagnostic benchmark for tool-using agents, has been accepted at two ICML 2026 workshops, FAGEN and AIWILD.
MCP-Persona is a benchmark evaluating LLM agents on personalized tools interacting with individual accounts and local databases. Experiments reveal significant challenges for state-of-the-art agents in personalized tool use.
Agent Judge is an agentic evaluation harness that overcomes the limitations of simple LLM judges for long-horizon agents by handling long trajectories, verifying stateful actions against source-of-truth systems, and adapting to changing behavior.