ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Summary
ExecRetrieval introduces a benchmark to measure the functional-correctness gap in code-embedding retrieval, showing that top retrievers frequently rank buggy near-clone implementations above correct ones.
View Cached Full Text
Cached at: 09/03/26, 03:51 AM
Paper page - ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Source: https://huggingface.co/papers/2609.01865
Abstract
ExecRetrieval evaluates whether code embeddings can distinguish correct implementations from near-identical buggy variants, revealing that leading retrievers frequently rank incorrect near-clones above canonical solutions.
Embedding-based code retrievalis a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query’s canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals --execution-verified buggy variantsnear-identical to each canonical -- so that a retriever’s rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23dense embedding configurationsplusBM25under provider-native invocation with pairedMcNemar testsand query-levelbootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reachesexec@10= 1.00 but onlyexec@1= 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2609\.01865
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.01865 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.01865 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.01865 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Beyond Retrieval: A Multitask Benchmark and Model for Code Search
This paper introduces CoREB, a contamination-limited multitask benchmark for code search that evaluates text-to-code, code-to-text, and code-to-code retrieval with fine-tuned reranking capabilities.
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
Introduces Agent Retrieval Bench, a file-level benchmark evaluating how well coding agents retrieve relevant repository files during the context-acquisition stage. The benchmark includes 427 samples across 25 repositories and evaluates various retrieval methods, finding no single family dominates.
Hybrid retrieval + dependency-graph expansion beats embeddings-only for code RAG — measured, CI-gated
Archex is a new open-source code RAG tool that improves retrieval by combining hybrid search (BM25F + dense embeddings), cross-encoder reranking, and dependency-graph expansion, achieving much higher recall and token efficiency than pure embeddings-based approaches.
Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval
This paper benchmarks 17 deep learning models for first-stage recall in large-scale code-to-code retrieval, evaluating their precision, efficiency, and scalability across multiple programming languages and datasets. It introduces LLM-based code normalization and query rewriting schemes that improve precision for lower-performing models.
Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers
Researchers introduce CSR-L and CS-MTEB benchmarks showing that code-switching queries degrade IR system performance by up to 27%, revealing embedding-space divergence that current multilingual techniques cannot fix.