ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts
Summary
ClawProBench introduces a trace-aware benchmark for evaluating AI agents, emphasizing runtime execution and safety to reveal limitations in final-answer rankings.
View Cached Full Text
Cached at: 08/25/26, 08:38 PM
Paper page - ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts
Source: https://huggingface.co/papers/2608.22510
Abstract
ClawProBench evaluates agent configurations via execution traces across live and frozen tracks, revealing that final-answer rankings obscure native-runtime failures and process-quality differences.
Agent benchmarksoften evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated onOpenClaw, a live agent runtime withworkspace toolsandnative surfacesfor browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored fromexecution tracesvia a safety-gated formula combining correctness,process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout,pass@k-anyoutperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.
View arXiv pageView PDFProject pageGitHub822Add to collection
Get this paper in your agent:
hf papers read 2608\.22510
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.22510 in a model README.md to link it from this page.
Datasets citing this paper1
#### xyh110sym/clawprobench Updatedabout 11 hours ago • 2
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.22510 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks
UniClawBench introduces a capability-driven benchmark for evaluating proactive agents in dynamic, real-world environments using live Docker containers and a closed-loop evaluation strategy with multiple agent roles.
OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories
This paper introduces OpenClawBench, a large-scale dataset for benchmarking process-side anomalies in real-world AI agent execution trajectories. It reveals that task success can hide process failures, with 9.33% of oracle-passing executions containing anomalies, and provides structured supervision via a novel taxonomy.
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
WildClawBench evaluates language and vision-language models on realistic long-horizon tasks using actual CLI environments with real tools. The benchmark reveals that even the best model achieves only 62.2% accuracy, indicating long-horizon agent evaluation remains challenging.
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
ClawForge is a generator-backed benchmark framework for executable command-line workflows under state conflict, evaluating LLM agents on tasks with pre-existing partial, stale, or conflicting artifacts across 17 scenarios.
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
EnterpriseClawBench presents a benchmark for enterprise agents based on real-world workplace sessions, offering 852 reproducible tasks and comprehensive evaluation metrics beyond single performance scores.