Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Hugging Face Daily Papers Papers
coding-agents retrieval-benchmark repository-context file-level-evaluation code2test trace2code selective-retrieval

Summary

Introduces Agent Retrieval Bench, a file-level benchmark evaluating how well coding agents retrieve relevant repository files during the context-acquisition stage. The benchmark includes 427 samples across 25 repositories and evaluates various retrieval methods, finding no single family dominates.

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.
Original Article
View Cached Full Text

Cached at: 07/29/26, 07:54 PM

Paper page - Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Source: https://huggingface.co/papers/2607.24882

Abstract

Moderncodingagentsareusuallyevaluatedbywhethertheyeventuallyproduceacorrectpatch,butpatchgenerationdependsonanearliercontext-acquisitionstage:findingtherepositoryfilesneededforthetask.WeintroduceAgentRetrievalBench,afile-levelbenchmarkforthisupstreamretrievalproblem.Samplesarebuiltfromrealcoding-workflowsignalsandevaluatedagainstfrozenbase-commitrepositories,withrelevancedefinedbywhatanagentneedsnextratherthandirectquery-filesemanticsimilarity.Thebenchmarkcoversfourpositive-retrievaltasks:code2test,comment2context,trace2code,andedit2ripple;afifthsubsetevaluatesselectiveretrievalusingnaturalevidence-backedno-goldcasesandcounterfactualwrong-repositorycontrols.AgentRetrievalBenchcontains427samplesacross25repositories:345positiveexamples,50naturalno-goldexamples,and32counterfactualcontrols.Thecorpusincludes308base-commitsnapshots,392,000files,and7.9millionchunks.Weevaluatelexicalretrieval,RepoMap,open-sourceembeddings,selectiveabstention,andloggedagentcontextselection.Nosingleretrievalfamilydominates:Qwen3-Embedding-4Bhasthebestsample-weightedMRRonpositivesamples,Qwen3-Embedding-8BthebestRecall@20,andRepoMapthebestbudgetedcontextyieldat8Ktokens,withtask-levelwinnersdifferingsubstantially.Selectivethresholdscalibratedwithcounterfactualcontrolsdonotimproveselectivesuccessonnaturalno-goldcases,revealingacalibrationgap.Loggedtrajectoriesalsomisseverygoldfileon27-35percentofsamples.Acontrolledseed-interventionpilotfindsthatretrieval-derivedinitialcontextyieldshigherfileF1withlesspost-seedexplorationthanrandomnon-goldcontext,whileoraclegoldcontextshowssubstantialremainingheadroom.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.24882

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.24882 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.24882 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.24882 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Hugging Face Daily Papers

SWE-Explore introduces a benchmark for evaluating coding agents' repository exploration capabilities, requiring ranked lists of relevant code regions within line budgets. Experiments show agentic exploration outperforms traditional retrieval, and line-level coverage remains a key differentiator.