AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
Summary
AgentLens is a new open-source benchmark for evaluating coding agents that assesses the full trajectory of interactions, including instruction following, tool use, error recovery, and more, using formal verification and LLM-written reviews.
View Cached Full Text
Cached at: 07/09/26, 07:54 AM
# AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation Source: [https://arxiv.org/abs/2607.06624](https://arxiv.org/abs/2607.06624) [View PDF](https://arxiv.org/pdf/2607.06624) > Abstract:We present AgentLens, a production\-assessed benchmark for interactive code agents\. Most code\-agent benchmarks reduce a run to a single bit \-\- did the task pass? \-\- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way\. AgentLens evaluates that whole trajectory\. It pairs formal verification, where an objective check exists, with LLM\-written trajectory reviews and side\-by\-side comparisons, so that each run yields a readable explanation of why the score is what it is\. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline\. We release the benchmark as open source at[this https URL](https://github.com/agent-lens/agent-lens-bench)\. ## Submission history From: Vadim Lomshakov \[[view email](https://arxiv.org/show-email/71ed48c2/2607.06624)\] **\[v1\]**Tue, 7 Jul 2026 11:27:43 UTC \(263 KB\)
Similar Articles
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
AgentLens is a framework for process-level assessment of software engineering agent trajectories, revealing that over 10% of passing trajectories exhibit a 'Lucky Pass' behavior. It introduces AgentLens-Bench, a dataset annotated with quality scores, and shows that ranking by quality score can shift model rankings significantly.
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
AgentCompass is an open-source, lightweight, and extensible evaluation infrastructure for LLM-based agents, decoupling benchmarks, harness, and environment for flexible configurations. It supports over 20 benchmarks across five capability dimensions and provides fault-tolerant runtime and trajectory analysis tools.
ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science
ClinLens is a new benchmark of 200 executable clinical data-science tasks over five linked MIMIC resources, evaluating long-horizon coding agents on longitudinal multimodal data. Results show strong code execution but poor clinical analysis correctness, highlighting a gap between runnable submissions and valid analyses.
REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage [R]
REAP is an automated pipeline that curates production-derived benchmarks for coding agents from real developer-agent sessions, using LLM-based classification and stability checks to ensure reliable evaluation without manual labeling.
Benchmarking LLM Judges for Mobile Agent Evaluation
This paper introduces MobileJudgeBench, a benchmark with 931 human-annotated trajectories for systematically evaluating LLM-based judges on mobile agent tasks. It finds that simple baseline judges with sampled screenshots rival purpose-built methods, with the LLM backbone being the primary driver of quality.