AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
Summary
AgentLens is a new open-source benchmark for evaluating coding agents that assesses the full trajectory of interactions, including instruction following, tool use, error recovery, and more, using formal verification and LLM-written reviews.
View Cached Full Text
Cached at: 07/09/26, 07:54 AM
# AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation Source: [https://arxiv.org/abs/2607.06624](https://arxiv.org/abs/2607.06624) [View PDF](https://arxiv.org/pdf/2607.06624) > Abstract:We present AgentLens, a production\-assessed benchmark for interactive code agents\. Most code\-agent benchmarks reduce a run to a single bit \-\- did the task pass? \-\- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way\. AgentLens evaluates that whole trajectory\. It pairs formal verification, where an objective check exists, with LLM\-written trajectory reviews and side\-by\-side comparisons, so that each run yields a readable explanation of why the score is what it is\. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline\. We release the benchmark as open source at[this https URL](https://github.com/agent-lens/agent-lens-bench)\. ## Submission history From: Vadim Lomshakov \[[view email](https://arxiv.org/show-email/71ed48c2/2607.06624)\] **\[v1\]**Tue, 7 Jul 2026 11:27:43 UTC \(263 KB\)
Similar Articles
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
AgentLens is a framework for process-level assessment of software engineering agent trajectories, revealing that over 10% of passing trajectories exhibit a 'Lucky Pass' behavior. It introduces AgentLens-Bench, a dataset annotated with quality scores, and shows that ranking by quality score can shift model rankings significantly.
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
AgentCompass is an open-source, lightweight, and extensible evaluation infrastructure for LLM-based agents, decoupling benchmarks, harness, and environment for flexible configurations. It supports over 20 benchmarks across five capability dimensions and provides fault-tolerant runtime and trajectory analysis tools.
REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage [R]
REAP is an automated pipeline that curates production-derived benchmarks for coding agents from real developer-agent sessions, using LLM-based classification and stability checks to ensure reliable evaluation without manual labeling.
Agent Evaluation: A Detailed Guide (53 minute read)
A comprehensive guide on evaluating LLM-based agent systems, covering fundamental concepts, evaluation frameworks, and case studies from recent benchmarks.
Agent Judge: Solving Long-Context Evals for Production Agents (10 minute read)
Agent Judge is an agentic evaluation harness that overcomes the limitations of simple LLM judges for long-horizon agents by handling long trajectories, verifying stateful actions against source-of-truth systems, and adapting to changing behavior.