REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark

Papers with Code Trending Papers
code-generation benchmark repository-level code-llms executable-code fine-tuning

Summary

RepoExec is a new benchmark for evaluating repository-level code generation focused on executability, functional correctness via automated test generation, and cross-file dependency integration. Experiments reveal that pretrained LLMs beat instruction-tuned ones on correctness, while fine-tuned models better leverage provided code dependencies.

The ability of CodeLLMs to generate executable and functionally correct code at the repository-level scale remains largely unexplored. We introduce RepoExec, a novel benchmark for evaluating code generation at the repository-level scale. RepoExec focuses on three main aspects: executability, functional correctness through automated test case generation with high coverage rate, and carefully crafted cross-file contexts to accurately generate code. Our work explores a controlled scenario where developers specify necessary code dependencies, challenging the model to integrate these accurately. Experiments show that while pretrained LLMs outperform instruction-tuned models in correctness, the latter excel in utilizing provided dependencies and demonstrating debugging capabilities. We also introduce a new instruction-tuned dataset that focuses on code dependencies and demonstrate that CodeLLMs fine-tuned on our dataset have a better capability to leverage these dependencies effectively. RepoExec aims to provide a comprehensive evaluation of code functionality and alignment with developer intent, paving the way for more reliable and applicable CodeLLMs in real-world scenarios. The dataset and source code can be found at~https://github.com/FSoft-AI4Code/RepoExec.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:08 PM

Paper page - REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark

Source: https://huggingface.co/papers/2406.11927 Published on Jun 17, 2024

Abstract

RepoExec is a benchmark for evaluating repository-level code generation focusing on executability, functional correctness, and dependency integration.

The ability ofCodeLLMsto generateexecutableand functionally correct code at the repository-level scale remains largely unexplored. We introduceRepoExec, a novel benchmark for evaluating code generation at the repository-level scale.RepoExecfocuses on three main aspects: executability,functional correctnessthroughautomated test case generationwith high coverage rate, and carefully craftedcross-file contextsto accurately generate code. Our work explores a controlled scenario where developers specify necessarycode dependencies, challenging the model to integrate these accurately. Experiments show that while pretrainedLLMsoutperform instruction-tuned models in correctness, the latter excel in utilizing provided dependencies and demonstratingdebugging capabilities. We also introduce a new instruction-tuned dataset that focuses oncode dependenciesand demonstrate thatCodeLLMsfine-tuned on our dataset have a better capability to leverage these dependencies effectively.RepoExecaims to provide a comprehensive evaluation of code functionality and alignment with developer intent, paving the way for more reliable and applicableCodeLLMsin real-world scenarios. The dataset and source code can be found at~https://github.com/FSoft-AI4Code/RepoExec.

View arXiv pageView PDFGitHub164autoAdd to collection

Get this paper in your agent:

hf papers read 2406\.11927

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2406.11927 in a model README.md to link it from this page.

Datasets citing this paper2

#### Fsoft-AIC/RepoExec Viewer• UpdatedJun 23, 2024 • 1.07k • 654 • 6 #### Fsoft-AIC/RepoExec-Instruct Viewer• UpdatedJun 26, 2024 • 156k • 89 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2406.11927 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

@adithya_s_k: https://x.com/adithya_s_k/status/2067628584680710292

X AI KOLs Timeline

This article discusses how coding agents can cheat evaluations by copying known patches, and introduces Repo2RLEnv, a tool to create verifiable coding environments from real repositories to build robust benchmarks and training data for AI coding agents.

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Hugging Face Daily Papers

SWE-Explore introduces a benchmark for evaluating coding agents' repository exploration capabilities, requiring ranked lists of relevant code regions within line budgets. Experiments show agentic exploration outperforms traditional retrieval, and line-level coverage remains a key differentiator.

Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

arXiv cs.LG

This paper introduces AlgoREval, a benchmark of 599 problems covering 77 classical algorithms across 14 domains, 7 languages, and 4 graph-input representations, to measure LLMs' ability to reproduce well-known algorithms as 'parametric code retrieval' rather than true synthesis. Across 15 models (7B–34B), accuracy varies widely by language and input format, and the authors caution against deploying AI-generated algorithmic code without systematic validation.