Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

# Don’t Claim Benchmark-Oriented Optimization Improves General Coding Capability — Diverse Evaluation Is Required
Source: [https://arxiv.org/html/2608.13566](https://arxiv.org/html/2608.13566)
Vera KudrevskaiaTimur GalimzyanovMikhail EvtikhievAna TernaRastislav RabatinTimur KudashevTimofey BryksinArina PuchkovaPatrik BartakEgor BogomolovSergey Titov

###### Abstract

Post\-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks \(e\.g\., SWE\-bench and LiveCodeBench\) as evidence of broad “coding capability”, both for research artifacts and user\-facing systems\.We argue that optimization for these benchmarks leads to measuring task\-specific performance, creating a*meaning gap*between measured scores and claims of general coding ability\.We examine this gap with a Django\-based case study benchmark suite we create\.

Evaluating foundation models and checkpoints post\-trained on SWE\-bench trajectories, we find that benchmark rankings frequently fail to generalize\. Post\-trained checkpoints show little cross\-task transfer, and SWE\-bench optimization yields limited or no gains on our tasks or on LiveCodeBench\. Similarly, fine\-tuning on individual Django modalities fails to transfer\.

We conclude that a small number of benchmarks is insufficient for evaluating diverse models under benchmark optimization pressure\. We encourage the community to use differentiated evaluation—holistic assessment for frontier models, multi\-task suites for research, and human\-in\-the\-loop studies for narrow task applications\. Finally, we argue for creating a capability taxonomy and sustained benchmark maintenance, rather than one\-off benchmark releases\. Without reliable evaluation standards, engineers and researchers using LLMs and agents have to rely on insufficient evidence to make research, development, and deployment decisions\.

LLM, benchmarks, evaluation, machine learning for software engineering

## 1Introduction

The deep learning for code \(DL\-for\-code\) community has converged on a narrow evaluation paradigm\. On one end, researchers use self\-contained algorithmic tasks, such as HumanEval, that are convenient for benchmarking, yet only weakly resemble real\-world software development\. On the other end, SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib39)\)became the de facto standard for measuring “real\-world” coding ability, with leaderboard rankings frequently interpreted as proxies for general coding capability\. “General coding capability” is a latent factor explaining positive correlations in model performance across diverse programming tasks, distinct from narrow task\-specific skills\. It is similar to the definition of intelligence by Chollet\(Chollet,[2019](https://arxiv.org/html/2608.13566#bib.bib14)\)\. This convergence has driven a remarkable engineering effort: complex post\-training approaches, specialized agent architectures, and training pipelines specifically designed to maximize SWE\-bench scores\(Zenget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib55); Panet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib30)\)\.

SWE\-bench scores may indeed correlate with genuine progress in coding\. Recent generations of foundation models show clear improvements across many coding\-related behaviors, and agents combined with frontier proprietary models are now widely used in practice\(Christian Mürtz,[2025](https://arxiv.org/html/2608.13566#bib.bib7)\)\. However, the current structure of SWE\-bench and other coding benchmarks often cannot tell us*why*scores improve\. SWE\-bench\-style performance mixes multiple factors: repository understanding, patch synthesis, tool use, search and retrieval strategies, and adherence to a particular workflow and output format\. It is thus difficult to identify the true driver of improvement without deep analysis\. For the foundation models, improvements are typically reported across a broad range of benchmarks, including non\-coding ones, providing stronger evidence of genuine gains in general intelligence\. In contrast, many post\-training papers report improvements primarily on SWE\-bench \(or a small cluster of similar benchmarks\), providing a potentially limited and ambiguous signal about generalization and changes in underlying capabilities\.

This conflation matters because it shapes research priorities\. For example, if post\-training on SWE\-bench trajectories reliably improves general coding capabilities, this approach provides a path towards model improvement\. But if it just produces models skilled at SWE\-bench\-like tasks, the field risks optimizing for the benchmark itself rather than addressing the underlying capability or other coding tasks, severely limiting the impact of this post\-training approach\. Thus, researchers working on evaluation of coding agents and LLM coding capabilities may be confounded into misinterpreting their results, and engineers relying on their work may make suboptimal deployment decisions\.

Our evidence supports the latter interpretation: post\-training gains on SWE\-bench do not consistently transfer to other code tasks, even within the same repository\. To illustrate this, we build a Django benchmark covering code editing, generation, and completion, and evaluate community\-released checkpoints and our fine\-tuned models\. We observe a consistent pattern across both community\-released SWE\-bench\-oriented checkpoints and models we fine\-tune\. Models fine\-tuned on issue\-resolution trajectories do not generally improve on our Django\-related tasks or LiveCodeBench \(LCB\)\(Jainet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib64)\), while models fine\-tuned on one of the Django tasks do not improve on other Django tasks or LCB\. This suggests narrow training recipes drive task\-specific specialization, not general coding improvement\. Moreover, SWE\-bench \(and even multi\-benchmark\) rankings may not always reliably predict relative performance on these modalities\. Together, our findings suggest that SWE\-bench rankings may systematically misrepresent models’ relative strengths on individual SE tasks, particularly since real\-world usage extends beyond agentic issue resolution\.

In this position paper, we argue that this mismatch between performance on different benchmarks reflects a*construct validity*problem: when a small number of benchmarks is treated as a proxy for “general coding capability,” the resulting claims exceed what the measurements can actually support\. SWE\-bench targets a complex but specific behavior that involves navigating repositories, understanding issue descriptions, and producing targeted patches\. These measurements may mix multiple underlying capabilities with narrow task\-specific skills\. High performance may indicate strong coding ability, but may equally reflect proficiency at the particular format and workflow developed under benchmark optimization pressure\. Without evaluation across diverse task modalities, we cannot distinguish among these hypotheses, understand the strengths and weaknesses of our post\-training approaches, or even assess the foundation models with sufficient granularity\. The resulting misnterpretation can then spread to engineers and users working with LLMs and agents, as LLM\-assisted coding and coding agents become increasingly common, and, in the end, undermine their trust in these systems\.

We call for a fundamental shift in how the community evaluates code models\. Over\-reliance on a narrow set of benchmarks has led the community to equate task performance with general coding capability, undermining construct validity and making it impossible to distinguish genuine ability from benchmark\-specific optimization\. Evaluation frameworks should explicitly test for transfer across task modalities, separate general coding improvement from narrow specialization, and provide a complete picture of model capabilities and limitations\. Our benchmark suite and experimental methodology illustrate a possible path forward, but our main recommendation is methodological: the field must move beyond leaderboard rankings towards evaluation practices that more directly measure the constructs we claim to care about\. As a practical step in this direction, we propose a three\-pronged approach: holistic assessment for frontier models, diverse multi\-task suites for incremental research and smaller models, and human\-in\-the\-loop assessment for narrow applications\.

## 2Background

### 2\.1Coding benchmarks

The evaluation of code\-oriented LLMs broadly falls into two families\.

Self\-contained code taskspackage everything needed to solve a task within the prompt itself\. Many widely used suites in this family focus on short\-form code generation\. A typical example is HumanEval\(Chen,[2021](https://arxiv.org/html/2608.13566#bib.bib21)\)that contains 164 hand\-crafted Python problems for function\-level generation\. Performance is typically reported with pass@k \(often pass@1\),i\.e\., whether a sampled solution passes the provided tests\. For example of a broader benchmark suite, LiveCodeBench\(Jainet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib64)\)adds self\-repair, code execution, and test output prediction to code generation\.

Repository\-levelbenchmarks evaluate a model within an entire codebase\. Given a repository snapshot and a natural\-language task description, the model must produce a patch that resolves the task and passes the tests\. SWE\-bench Verified\(Chowdhuryet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib8)\)is thede factostandard in this family, comprising 500 issue resolution instances drawn from GitHub issues across 12 popular Python repositories\. While issue resolution is an important task, it does not represent the whole space of coding agent capabilities\. As of late, SWE\-bench Pro\(Denget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib66)\), which also targets issue resolution tasks, also started gaining traction\. Outside of issue resolution, the only relatively popular benchmark is TerminalBench\(Merrillet al\.,[2026](https://arxiv.org/html/2608.13566#bib.bib13)\)\.

The evaluation approaches of these families measure different capabilities, and it is unclear whether the benchmark scores should always be correlated\. Strong performance on self\-contained tasks may fail to predict success on repository\-level tasks that require navigating codebases, understanding dependencies, and producing targeted edits\. It is moreover unclear, whether these two families of benchmarks cover general coding capability\.

### 2\.2Foundation models

Modern foundation models are trained for broad, multi\-domain competence, and model cards and technical reports summarize coding performance with a small set of representative benchmarks\. The ability of model creators to benchmark these models for a particular domain is naturally limited by time and the necessity to produce concise, readable reports\. For example, the GPT\-5 model card includes reports on only two coding benchmarks \(SWE\-bench Verified and Aider Polyglot\), and the Qwen3 technical report\(Yanget al\.,[2025a](https://arxiv.org/html/2608.13566#bib.bib49)\)includes four benchmarks, of which three belong to the self\-contained algorithmic tasks, and the fourth is execution\-reasoning\-oriented CRUXEval\. The limited number of reported benchmarks, combined with the possibility of little correlation across benchmarking tasks, implies the risk that some coding capabilities may fall into a blind spot of benchmark suites used by the foundation model creators, resulting in a misleading evaluation\.

### 2\.3Post\-training foundation models

These limitations become even more pronounced for post\-training checkpoints\. Compared to releases of foundation models, post\-training papers face page limits, computational costs of evaluation\(Jordanet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib61)\), and the engineering overhead of adopting additional harnesses\. As a result, single\-benchmark evaluations are often the pragmatic default rather than an exception\.

This constraint produces a typical pattern: post\-training works tend to evaluate on benchmarks aligned with their intended contribution\. Works targeting repository\-level SE commonly report only SWE\-bench performance \(e\.g\., Lingma SWE\-GPT\(Maet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib50)\), R2EGym\-Agent\(Jainet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib51)\), SWE\-agent\-LM\(Yanget al\.,[2025b](https://arxiv.org/html/2608.13566#bib.bib52)\), Skywork\-SWE\(Zenget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib55)\), DeepSWE\-Preview\(Luoet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib56)\)\)\. On the other hand, studies focused on developing general post\-training techniques often evaluate on self\-contained suites like HumanEval \(e\.g\.,\(Weiet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib9)\),\(Tanget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib58)\),\(Yuet al\.,[2024b](https://arxiv.org/html/2608.13566#bib.bib59)\)\)\. While perfectly rational, this evaluation approach does not allow distinguishing whether the improvement comes from task\-specific optimization or reflects a general improvement of coding capability\. Capability generalization cannot be taken for granted: Yu et al\.\(Yuet al\.,[2024a](https://arxiv.org/html/2608.13566#bib.bib60)\)show cases where instruction tuning improves algorithmic benchmark scores without commensurate gains on the underlying behavior\. This cited pattern we find undesirable motivates our position\. To support it, we directly test cross\-task transfer by evaluating checkpoints trained on specific tasks across multiple SE modalities\.

## 3Benchmark measurements and capability claims

From the general coding capability point of view, any single benchmark can at best provide a noisy, partial measurement of this latent capability\. In particular, when two models are far apart in overall competence \(e\.g\., frontier systems versus small baselines\), a large and consistent performance gap on a benchmark is often a reasonable shorthand for a capability difference\. The difficulty arises when benchmark results are treated as sufficient evidence for broad capability claims, especially when models are optimized for that benchmark\.

Reporting model performance on a limited number of benchmarks is practical and reasonable\. However, there seems to be a systematic disconnect between what code benchmarks actually measure and how benchmark scores are commonly interpreted\. Because of this “meaning gap”, specific achievements on a narrow benchmark may inflate into broad assertions of general coding capability\.

We illustrate the meaning gap concept using examples from the Qwen3\-Coder\-480B\-A35B model descriptions\. The original blog post reports the model performance on eight coding\-related tasks, five of which correspond to issue resolution similar to SWE\-bench\(QwenTeam,[2025](https://arxiv.org/html/2608.13566#bib.bib57)\), and summarizes them as “exceptional performance in both coding and agentic tasks”\. Finally, an independent blog post comparing models makes a claim “Chinese models aren’t just competing—they’re winning\. Qwen 3 Coder leads at 67% on SWE\-bench, surpassing GPT\-4\.1’s 54\.6%”\(DigitalAppliedTeam,[2025](https://arxiv.org/html/2608.13566#bib.bib40)\)\. Each step is understandable in isolation, but together they broaden a narrow performance measurement into a claim about general superiority at coding that may influence deployment decisions\.

Conceptually, the meaning gap combines two processes: \(i\) generalization from benchmark scores to capability statements by the model and checkpoint creators, and \(ii\) further amplification of statements from scientific papers and technical reports by the general audience\. While the latter problem may have a strong impact on the DL\-for\-code community, it merits a separate sociological study beyond the scope of our paper\. Thus, in our study, we focus on the relationship between benchmark scores and the general coding capability that the community seeks to improve, and use the “meaning gap” term to refer to the mismatch between the two\.

The meaning gap is directly related to the construct validity problem\. In their review on LLM benchmarks, Bean et al\.\(Beanet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib63)\)provide recommendations for benchmark descriptions that would strengthen their construct validity\. Describing SWE\-bench Verified according to their checklist yields the following description\.

SWE\-bench is an issue resolution capability benchmark\. It is based on 12 Python repositories with a highly skewed task distribution, with 46% of 500 examples coming from the Django project\. SWE\-bench Verified does not address possible data contamination issues, and the common practice of reporting its scores does not include uncertainty estimates or analysis of common failure modes\.

These observations do not diminish the value of SWE\-bench Verified as a benchmark\. However, when a model achieves a high score on SWE\-bench Verified, there is no standard way to determine whether the improvement reflects \(a\) genuine SE capability that transfers to other coding tasks, \(b\) limited SE capability on this family of tasks, \(c\) inadvertent optimization for SWE\-bench’s specific format, tasks, workflow, or agentic scaffolding, \(d\) accidental contamination of training dataset, or \(e\) some mixture of all of the above\. This problem is especially pronounced in academic post\-training studies, as researchers often cannot afford to run many benchmarks and may mistake within\-benchmark gains for general improvements in capability\.

At this point, we only identify the meaning gap and argue that it can be practically consequential\. To test it, we create a benchmark suite that measures model performance across multiple in\-repository task modalities, explicitly assessing the cross\-task transfer capability\. If models with higher SWE\-bench Verified scores perform worse on our suite, then SWE\-bench Verified is a weak predictor for these tasks and is insufficient as a standalone proxy for broader coding capability\. In the next section, we describe our benchmark and evaluate a range of models and checkpoints on it\.

## 4Evidence from a Django case study

### 4\.1Benchmarks and data

To study the meaning gap in a controlled setting, we build a benchmark suite over a single real\-world repository, spanning three common SE modalities: \(i\) method generation, \(ii\) method completion, and \(iii\) program repair\. We use these experiments to test plausibility of our position\. We describe all benchmark details in the appendices\.

Repository snapshot\.We choose Django because it constitutes 46% of SWE\-bench Verified\. All tasks are derived from a snapshot at version 4\.0\.4\.111commit hash89807​f​b​d​e​8​b​7​b​17​d​00434​b​c​4695535855​e​96​f​e​7789807fbde8b7b17d00434bc4695535855e96fe77, date: 11th Apr 2022\.

This choice enables a simple and testable prediction: if SWE\-bench\-optimized checkpoints truly improve transferable repository\-level SE ability \(rather than specializing to SWE\-bench’s workflow and distribution\), then they should also improve on Django\-centric tasks within the same codebase\. Thus, we test cross\-task transfer for relatively close tasks that come from data distribution close to SWE\-bench’s\. We describe repository selection, data statistics, and data curation in appendix[B\.1](https://arxiv.org/html/2608.13566#A2.SS1)\.

Docstring standardization\.Django’s in\-code documentation has uneven quality, which can make “generate the method from its docstring” task instances noisy and inconsistently specified\. To reduce this variance, we synthesize standardized docstrings for all methods with DeepSeek\-R1\(Guoet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib42)\)and merge them with the original comments whenever applicable\. We provide extensive description of our documentation synthesis pipeline in Appendix[B\.3](https://arxiv.org/html/2608.13566#A2.SS3)\.

Task definitions\.Using this augmented data, we build three benchmarks:

- •Method generation:the model receives a target method signature, its docstring, and the source file with the target method body removed \(method declaration is provided\); it must generate the missing body\.
- •Method completion:the model receives the preceding file content with the first half of the target method; it must generate the remaining half \(in the chat format\)\.
- •Program repair:the model receives a source file in which one method has been corrupted, and a stack trace produced by running the Django test suite on the repo with this broken file\. The model must identify the broken method and output the correct version\.

We run our benchmarks without agentic scaffolding to check in isolation how model post\-training affects model coding behavior\. Appendix[C\.2](https://arxiv.org/html/2608.13566#A3.SS2)gives one scaffolding comparison to illustrate how scaffolding can affect performance without resolving the cross\-task transfer concern\.

Train/test split and leakage control\.Each benchmark is paired with a corresponding instruction\-tuning dataset constructed from the same task template\. After splitting, the dataset contains 3,180 train samples and 359 test samples\. To avoid cross\-contamination through shared class hierarchies, we split at the level of inheritance trees: all ancestor and descendant classes of each method’s defining class go to the same split\. We do not train on Django’s test suite, and we do not include tests in evaluation prompts\. We describe train/test split in appendix[B\.2](https://arxiv.org/html/2608.13566#A2.SS2), and provide prompts in appendix[F](https://arxiv.org/html/2608.13566#A6)\.

Evaluation\.We evaluate candidate outputs by running the official Django 4\.0\.4 test suite and reportpass@1\. For all benchmarks, we use greedy decoding atT=0T=0, so each instance has a single attempt\.

Out\-of\-suite comparison: LiveCodeBench\.In addition to our benchmarks, we evaluate all models and checkpoints on LiveCodeBench \(LCB\), using tasks added after October 1, 2024 \(341 examples\)\. We use LCB to validate basic code generation capabilities on self\-contained algorithmic problems, contrasting with our Django benchmarks that assess repository\-level understanding and task execution, and testing for the more distant cross\-task transfer\. We again evaluate with greedy decoding atT=0T=0\.

### 4\.2Models, checkpoints and fine\-tuning

We evaluate three groups: \(i\) foundation models, \(ii\) public SWE\-bench\-oriented checkpoints, and \(iii\) our task\-specific fine\-tunes\.

Foundation models\.We include Qwen3\-32B\(Huiet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib45)\), Qwen2\.5\-Coder\-7B\-Instruct, and Qwen2\.5\-Coder\-32B\-Instruct\(Yanget al\.,[2025a](https://arxiv.org/html/2608.13566#bib.bib49)\)as the representative strong open models spanning sizes and releases, and the popular base models for public checkpoints\.

Public checkpoints\. We evaluate public checkpoints obtained by fine\-tuning models on agentic issue\-resolution trajectories \(e\.g\., SWE\-bench\-style traces\): DeepSWE\-Preview\(Luoet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib56)\), R2EGym\-7B\-Agent\(Jainet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib51)\), SWE\-agent\-LM\-7B\(Yanget al\.,[2025b](https://arxiv.org/html/2608.13566#bib.bib52)\), Openhands LM 32B\(OpenHands,[2025](https://arxiv.org/html/2608.13566#bib.bib44)\), Skywork\-SWE\(Zenget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib55)\), SWE\-agent\-LM\-32B, R2EGym\-32B\-Agent\. Most of these checkpoints are based on the Qwen2\.5\-Coder\-Instruct model\. These checkpoints are designed to improve SWE\-bench performance\. By testing them on our Django benchmarks, we study whether improvements transfer to other repository\-level tasks in the same codebase—expected if coding capabilities genuinely improved\. LCB tests for transfer to more distant, self\-contained algorithmic tasks\.

Our fine\-tuning experiments\.To isolate the role of*task\-specific*supervision, we fine\-tune Qwen2\.5\-Coder\-7B\-Instruct and Qwen2\.5\-Coder\-32B\-Instruct on each of the three task\-specific datasets \(generation, completion, repair\)\. We choose these models because most of the public checkpoints we evaluate were built on top of one of them\. In every case, we use LoRA with learning rate1​e−51e\-5, and use the last epoch checkpoint for evaluation; we provide full training details in appendix[E](https://arxiv.org/html/2608.13566#A5)\. Wedo not aimto exhaustively optimize these models\. The goal is to test a practical, “reasonable” fine\-tuning recipe and measure*cross\-task transfer*rather than to maximize within\-task scores\.

### 4\.3Results

Table 1:Cross\-task transfer diagnostics for SWE\-bench\-oriented checkpointsSWE\-benchGenerationCompletionRepairLCBscoreΔ\\DeltascoreΔ\\DeltascoreΔ\\DeltascoreΔ\\DeltascoreΔ\\DeltaQwen2\.5\-Coder\-32BBase model7\.007\.00\\cellcolor\[HTML\]E6E6E6 –64\.0764\.07\\cellcolor\[HTML\]E6E6E6 –60\.1760\.17\\cellcolor\[HTML\]E6E6E6 –47\.3547\.35\\cellcolor\[HTML\]E6E6E6 –28\.228\.2\\cellcolor\[HTML\]E6E6E6 –Openhands LM 32B37\.2037\.20\\cellcolor\[HTML\]7CB17C30\.2030\.2066\.5766\.57\\cellcolor\[HTML\]AECFAE2\.512\.5148\.7548\.75\\cellcolor\[HTML\]D9A0A0−11\.42\-11\.4213\.9313\.93\\cellcolor\[HTML\]D9A0A0−33\.43\-33\.4325\.825\.8\\cellcolor\[HTML\]E5BBBB−2\.3\-2\.3Skywork\-SWE38\.0038\.00\\cellcolor\[HTML\]79AF7931\.0031\.0068\.8068\.80\\cellcolor\[HTML\]71AA714\.744\.7457\.6657\.66\\cellcolor\[HTML\]F5DFDF−2\.51\-2\.5141\.7841\.78\\cellcolor\[HTML\]F7E3E3−5\.57\-5\.5727\.327\.3\\cellcolor\[HTML\]F1D6D6−0\.9\-0\.9SWE\-agent\-LM\-32B40\.2040\.20\\cellcolor\[HTML\]71AA7133\.2033\.2066\.5766\.57\\cellcolor\[HTML\]AECFAE2\.512\.5155\.1555\.15\\cellcolor\[HTML\]EDCDCD−5\.01\-5\.0140\.1140\.11\\cellcolor\[HTML\]F5DFDF−7\.24\-7\.2425\.225\.2\\cellcolor\[HTML\]D9A0A0−2\.9\-2\.9R2EGym\-32B\-Agent34\.4034\.40\\cellcolor\[HTML\]87B78727\.4027\.4067\.4167\.41\\cellcolor\[HTML\]97C1973\.343\.3458\.2258\.22\\cellcolor\[HTML\]F6E3E3−1\.95\-1\.9540\.6740\.67\\cellcolor\[HTML\]F5E0E0−6\.69\-6\.6929\.629\.6\\cellcolor\[HTML\]71AA711\.51\.5Qwen2\.5\-Coder\-7BBase model1\.801\.80\\cellcolor\[HTML\]E6E6E6 –52\.9252\.92\\cellcolor\[HTML\]E6E6E6 –50\.9750\.97\\cellcolor\[HTML\]E6E6E6 –17\.8317\.83\\cellcolor\[HTML\]E6E6E6 –15\.215\.2\\cellcolor\[HTML\]E6E6E6 –R2EGym\-7B\-Agent19\.0019\.00\\cellcolor\[HTML\]71AA7117\.2017\.2054\.8754\.87\\cellcolor\[HTML\]71AA711\.951\.9541\.2341\.23\\cellcolor\[HTML\]F5E0E0−9\.75\-9\.7515\.0415\.04\\cellcolor\[HTML\]F6E3E3−2\.79\-2\.7917\.317\.3\\cellcolor\[HTML\]71AA712\.12\.1SWE\-agent\-LM\-7B15\.2015\.20\\cellcolor\[HTML\]8DBB8D13\.4013\.405\.855\.85\\cellcolor\[HTML\]D9A0A0−47\.08\-47\.082\.232\.23\\cellcolor\[HTML\]D9A0A0−48\.75\-48\.751\.671\.67\\cellcolor\[HTML\]D9A0A0−16\.16\-16\.166\.76\.7\\cellcolor\[HTML\]D9A0A0−8\.5\-8\.5Qwen3\-32BBase model23\.0023\.00\\cellcolor\[HTML\]E6E6E6 –67\.4167\.41\\cellcolor\[HTML\]E6E6E6 –43\.4543\.45\\cellcolor\[HTML\]E6E6E6 –46\.8046\.80\\cellcolor\[HTML\]E6E6E6 –61\.0061\.00\\cellcolor\[HTML\]E6E6E6 –DeepSWE\-Preview42\.2042\.20\\cellcolor\[HTML\]71AA7119\.2019\.2067\.6967\.69\\cellcolor\[HTML\]71AA710\.280\.2844\.2944\.29\\cellcolor\[HTML\]71AA710\.840\.8451\.2551\.25\\cellcolor\[HTML\]71AA714\.464\.4660\.160\.1\\cellcolor\[HTML\]D9A0A0−0\.9\-0\.9Table[1](https://arxiv.org/html/2608.13566#S4.T1)summarizes the performance of foundation models and public SWE\-bench\-oriented checkpoints on our Django suite and LiveCodeBench \(with deltas computed against the corresponding base model\)\. For each checkpoint, we show the absolute score and its change \(Δ\\Delta\) relative to the corresponding base model\. We take SWE\-bench Verified scores from the original papers and reports\. To assess whether fine\-tuning for SWE\-bench transfers to out\-of\-domain tasks, we examine the pattern of performance changes\. We use these results to diagnose the problem and illustrate failure of construct validity and risks of cross\-task transfer\. We do not interpret them as a new leaderboard\.

Foundation models: rankings are not always stable across modalities\.Even for foundation models, relative ordering can flip across tasks: a model stronger on SWE\-bench can underperform on our benchmark, illustrating that a single headline score may not reflect a full capability profile\. For example, while Qwen\-2\.5\-32B\-Coder\-Instruct outperforms Qwen3\-32B on our code completion benchmark, it has worse SWE\-bench scores\.

Public checkpoints: no cross\-task transfer\.SWE\-bench\-optimized checkpoints, while significantly outperforming their base models on SWE\-bench, generally do not improve on our Django tasks or LiveCodeBench\. Across the 28 out\-of\-domain checkpoint–benchmark comparisons \(7 checkpoints×\\times4 benchmarks\), we observe 18 degradations versus ten improvements\. Five of seven checkpoints degrade on the majority of benchmarks, none improve on all four, and only DeepSWE\-Preview shows more improvements than degradations overall\. Because we report single greedy evaluations on stochastic benchmarks, small deltas should not be over\-interpreted\. The overall pattern is consistent with the absence of change in cross\-task capabilities, with occasional regressions\. Importantly, these regressions do not always come from output\-format failures: most checkpoints follow task formats, with only SWE\-agent\-LM\-7B and OpenHands\-LM\-32B sometimes failing to follow instructions\. To illustrate our point on cross\-task transfer for both format and substance mistakes, we provide error\-flow analysis in appendix[C\.1](https://arxiv.org/html/2608.13566#A3.SS1)\.

Table 2:Cross\-task transfer diagnostics for task\-specific fine\-tuningGenerationCompletionRepairLCBFine\-tuningscoreΔ\\DeltascoreΔ\\DeltascoreΔ\\DeltascoreΔ\\DeltaQwen2\.5\-Coder\-32BBase Model64\.0764\.07\\cellcolor\[HTML\]E6E6E6 –60\.1760\.17\\cellcolor\[HTML\]E6E6E6 –47\.3547\.35\\cellcolor\[HTML\]E6E6E6 –28\.2028\.20\\cellcolor\[HTML\]E6E6E6 –Generation71\.0371\.03\\cellcolor\[HTML\]71AA716\.966\.9654\.6054\.60\\cellcolor\[HTML\]D9A0A0−5\.57\-5\.5742\.9042\.90\\cellcolor\[HTML\]F5E1E1−4\.45\-4\.4522\.9022\.90\\cellcolor\[HTML\]D9A0A0−5\.30\-5\.30Completion66\.0266\.02\\cellcolor\[HTML\]CEE2CE1\.951\.9574\.6574\.65\\cellcolor\[HTML\]71AA7114\.4814\.4824\.5124\.51\\cellcolor\[HTML\]D9A0A0−22\.84\-22\.8425\.2025\.20\\cellcolor\[HTML\]E9C4C4−2\.90\-2\.90Repair69\.0869\.08\\cellcolor\[HTML\]95C0955\.015\.0156\.8256\.82\\cellcolor\[HTML\]E7C0C0−3\.35\-3\.3556\.2756\.27\\cellcolor\[HTML\]71AA718\.928\.9225\.5025\.50\\cellcolor\[HTML\]EBC9C9−2\.60\-2\.60Qwen2\.5\-Coder\-7BBase Model52\.9252\.92\\cellcolor\[HTML\]E6E6E6 –50\.9750\.97\\cellcolor\[HTML\]E6E6E6 –17\.8317\.83\\cellcolor\[HTML\]E6E6E6 –15\.2015\.20\\cellcolor\[HTML\]E6E6E6 –Generation58\.2258\.22\\cellcolor\[HTML\]71AA715\.305\.3050\.1450\.14\\cellcolor\[HTML\]D9A0A0−0\.83\-0\.8315\.8815\.88\\cellcolor\[HTML\]F8E7E7−1\.95\-1\.9516\.4016\.40\\cellcolor\[HTML\]71AA711\.201\.20Completion19\.7719\.77\\cellcolor\[HTML\]D9A0A0−33\.15\-33\.1564\.0764\.07\\cellcolor\[HTML\]71AA7113\.1013\.101\.391\.39\\cellcolor\[HTML\]D9A0A0−16\.44\-16\.4415\.0015\.00\\cellcolor\[HTML\]D9A0A0−0\.3\-0\.3Repair55\.7155\.71\\cellcolor\[HTML\]AECFAE2\.792\.7951\.2551\.25\\cellcolor\[HTML\]F0F7F00\.280\.2840\.9540\.95\\cellcolor\[HTML\]71AA7123\.1223\.1215\.0015\.00\\cellcolor\[HTML\]D9A0A0−0\.3\-0\.3Task\-specific fine\-tuning: strong within\-task gains, weak transfer\.Table[2](https://arxiv.org/html/2608.13566#S4.T2)reports our fine\-tuning experiments\. In most cases, our fine\-tuning reliably improves within\-task model performance \(e\.g\., repair\-tuned models improve on repair\), but we do not observe consistent improvements on other modalities, including LiveCodeBench\. Across all six settings, fine\-tuning improves performance on the*trained*modality \(six improvements and no degradations\)\. For cross\-task transfer, we observe a pattern consistent with no improvement in capabilities, similar to public checkpoints\. Across 18 cross\-task comparisons \(each fine\-tuned checkpoint evaluated on three held\-out benchmarks\), we observe five improvements and 13 degradations\. The low scores for Qwen2\.5\-Coder\-7B\-Instruct fine\-tuned on code completion reflect overfitting to the output format\.

These results support our claim: reasonable fine\-tuning with within\-task evaluation \(practical from an effort standpoint\) may mislead practitioners into perceiving a ”capability jump” that largely reflects task\-specific specialization rather than transferable coding ability\. Notably, excluding output format failures, we observe no difference in cross\-task transfer between our benchmarks and LiveCodeBench despite their widely different task distributions\. This cross\-task regression matters for LLM and coding agent development reliability: a checkpoint optimized for one workflow may be worse on other ones\.

## 5Discussion

### 5\.1Themeaning gapthesis

Our experiments provide empirical support for the*meaning gap*: performance on a popular coding benchmark can fail to predict performance on other coding tasks, even if they are closely related\. As a result, broad capability claims based on a narrow set of leaderboards are often overstated\.

Limited evidence of cross\-benchmark rank reversals for foundation models\.If a single benchmark is a good proxy for “general coding capability,” the relative ordering of models should be broadly stable across different coding evaluations\. In our limited comparison \(Qwen2\.5 vs\. Qwen3\), we observe a rank reversal: Qwen3\-32B scores higher on SWE\-bench, yet performs substantially worse than Qwen2\.5\-Coder\-32B\-Instruct models on our code completion benchmark\. This*suggestive rather than definitive*evidence illustrates the core risk: leaderboard position on one benchmark can be an unreliable signal for another task that also corresponds to a facet of general coding capability\.

Post\-trained checkpoints show near\-universal failure of cross\-task transfer\.The meaning gap is more pronounced for post\-trained checkpoints\. Across both community\-released SWE\-bench\-optimized checkpoints and our fine\-tuned variants, improvements are largely confined to the training/evaluation distribution, with little to no consistent gains on our Django benchmark suite and LiveCodeBench\. In several cases, performance severely degrades on other tasks \(e\.g\., SWE\-agent\-LM\-7B and OpenHands\-LM\-32B underperform their base models across multiple benchmarks, failing to follow instructions\)\. Moreover, failures are not uniform: some checkpoints follow instructions and formats reasonably well, yet still do not improve out\-of\-distribution, making the lack of transfer difficult to detect via qualitative “sanity checks” alone\. As LiveCodeBench and our benchmark suite use very different data and test models on different tasks, we speculate this may mean lack of cross\-task improvement on other coding tasks as well\.

Benchmarks capture coarse capability differences, but are brittle for fine\-grained choices\.We do not claim benchmarks are useless\. Across model*tiers*\(e\.g\., small vs\. mid\-size vs\. frontier systems\), stronger models tend to outperform weaker ones on most tasks, indicating that coding capabilities*do*improve over time and scale\. The practical problem arises precisely where users most need guidance: choosing between models that are close in headline benchmark scores \(e\.g\., selecting the ”best 8B model”\) or, more importantly, checkpoints that are fine\-tuned for a specific task\. In this regime, a model that edges ahead on SWE\-bench can lag behind on completion or repair, so single\-leaderboard rankings become unreliable guides for practitioners’ and researchers’ decisions\. This is where the meaning gap causes the most harm and where multi\-task evaluation or task\-specific human\-in\-the\-loop studies are most necessary\.

### 5\.2Possible paths forward

We see several complementary approaches to addressing the meaning gap, each with distinct tradeoffs\. We broadly divide them into four groups:\(i\)single\-score benchmarks,\(ii\)diverse benchmark suites,\(iii\)human\-in\-the\-loop studies on real tasks, and\(iv\)holistic open\-ended evaluations\. Single\-score benchmarks remain useful for fast iteration \(regression tests, ablations\), but are much weaker as final assessments or evidence for broad capability claims\.

Benchmark suites \(beyond a single score\)\.The most direct response to single\-benchmark limitations is broader coverage: a suite spanning completion, editing, bug localization, question answering, and repository navigation\. MTEB\(Muennighoffet al\.,[2023](https://arxiv.org/html/2608.13566#bib.bib47)\)exemplifies this for text embeddings, where diverse tasks constrain overgeneralization\. However, this approach faces two obstacles\.

First,benchmarks are high\-effort community servicethat is weakly rewarded, especially for maintenance, whichmatters even more than creation\. Each conference introduces many new benchmarks, but very few are ever widely adopted, and fewer are still maintained as models, data, and contamination risks evolve\. High\-quality suites need sustained curation, robust evaluation harnesses, contamination monitoring, and regular refresh\(Beanet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib63)\)\.

We believe this is addressableinstitutionally: targeted grants \(including industry sponsorship\), conference tracks and workshops that reward ongoing maintenance \(e\.g\., new versions, repaired instances, and contamination audits\), not just initial releases, treating benchmarks as first\-class research artifacts\. Without recognizing benchmark maintenance as essential research infrastructure, the existence and support of benchmarks like SWE\-Bench\-Pro, SWE\-rebench\(Badertdinovet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib53)\), or LiveCodeBench depend on the chance and goodwill of a small number of individuals\.

The second obstacle is aconstruct validityproblem: a benchmark suite is a good proxy for “general coding capability” only to the extent that it is built on a clear and correct taxonomy of coding skills and behaviours it aims to measure\. In addition to the general construct validity questions raised by\(Beanet al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib63)\), a robust suite requires the community to answer \(at least\) the questions below:

- •Define the construct:what behaviours count as “coding capability” for the target use cases?
- •Choose domains:what distributions matter \(open\-source vs\. proprietary ecosystems\(Denget al\.,[2025](https://arxiv.org/html/2608.13566#bib.bib66)\)\), and how should we handle contamination risks across them?
- •Choose modalities and coverage:which task types and programming languages \(and language families\) must be represented?
- •Decide aggregation:how should we combine axes \(tasks, repos, languages\) to report an interpretable*profile*rather than an over\-compressed single score?

Developing such a taxonomy is substantial research, but other areas of science \(e\.g\., psychometrics\) suggest it is feasible: latent constructs are estimated through batteries of partially correlated tests, not single measurement\. We sketch how this taxonomy can look like in Appendix[A](https://arxiv.org/html/2608.13566#A1)\. We stress this taxonomy is not ready to use, as creating usable taxonomy requires extensive empirical and qualitative work beyond the scope of our paper\.

Human\-in\-the\-loop validation\.A complementary approach is evaluation on genuinely real\-world tasks with human judgment\. For SE tasks, this could mean sampling open GitHub issues or internal tickets and collecting structured assessments from maintainers or domain experts\. This can be uncontaminated by design \(new issues are not in training data\) and better reflect practical distributions, including cases that are underspecified, unsolvable with available context, or unnecessary to address\. Moreover, human in the loop validation can also assess the perception of a coding agent, which affects the efficiency of human\-agent collaboration\. However, human evaluation has its own failure modes: expense, limited scalability, inconsistent standards across projects, and the risk of preferring style over substance\(Wu and Aji,[2025](https://arxiv.org/html/2608.13566#bib.bib67)\)\.

Holistic open\-ended evaluation\.A more radical departure from fixed tasks is holistic evaluation in open\-ended scenarios\. E\.g\. to succeed in Anthropic’s Project Vend\(Anthropic,[2025](https://arxiv.org/html/2608.13566#bib.bib41)\)a model has to integrate many skills under events that no benchmark designer would anticipate\. For coding, analogous setups can be controlled hackathons or bug\-squashing days where teams are randomly assigned AI assistants, with models judged on outcomes*and*user experience\.

These can be run as model–vs–model competitions with standardized rules, potentially sponsored by model developers\. The results can be reported as a public leaderboard if the protocol is transparent and robust enough to avoid becoming another easily optimized proxy\. While costly, a holistic evaluation targets the meaning gap directly: it measures performance when tasks are underspecified, requirements drift, and success requires adaptive problem\-solving\.

Evaluation approaches in practice\.These approaches are complementary\. For frontier foundation models, holistic assessments \(and carefully designed human studies\) should be the most informative\. For research on incremental techniques and smaller models, benchmark*suites*should provide a practical and scalable way to detect narrow vs\. broad improvements\. Single\-score benchmarks are useful for rapid iteration, but should not be treated as decisive evidence\. For narrow practical tasks such as fine\-tuning a model on a private repository for internal usage, task\-specific human\-in\-the\-loop setups should be the most actionable: general leaderboards are unreliable guides for models close in capability\. This approach can surface narrow failure modes that public benchmarks miss\.

## 6Alternative views

In this paper, we propose a three\-pronged evaluation approach: holistic assessment of frontier models, multi\-task benchmarks for academic research, and task\-specific user studies for narrow practical tasks\. However, there are alternative views on model evaluation, which we present and discuss below\.

Jordan et al\.\(Jordanet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib61)\)argue that rigorous benchmarking can be prohibitively expensive and propose usingscientific testinginstead to understand how algorithms work\. While we agree that mechanistic understanding is valuable in academic research, we believe benchmarking model performance outside controlled experiments is equally necessary for algorithm evaluation—complementing rather than replacing scientific testing\. To paraphrase a famous statement, “all DL approaches are not general, but some are useful”, and benchmarking is how one can estimate the limits of usefulness\.

A related alternative is qualitative evaluation, exemplified by Yang et al\.\(Yanget al\.,[2025b](https://arxiv.org/html/2608.13566#bib.bib52)\), which examines how and why models fail, what distinguishes success from failure, and what patterns emerge across cases\. Such analysis reveals capability boundaries invisible in summary statistics—whether models fail on long\-range dependencies, specification understanding, or semantic correctness despite syntactic validity\. We view qualitative analysis as a valuable complement to quantitative evaluation, not a replacement, and believe qualitative findings can inform better benchmark design that addresses common failure modes\. However, the need for reliable quantitative comparison remains\.

Another possible counterargument is that the evaluation system is self\-correcting\. For example, the emergence of SWE\-bench and LiveCodeBench addresses the limitations of HumanEval: limited task scope and contamination, respectively\. We are less optimistic: benchmark gains are now read as capability gains, masking failures to transfer and creating selection pressure for Goodhart’s law\-like effects to appear\(Manheim and Garrabrant,[2018](https://arxiv.org/html/2608.13566#bib.bib29)\)\. It also takes time for the community to change the benchmark of choice: while first concerns about leaks and test quality in SWE\-bench first appeared in 2024\(Aleithanet al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib62)\), SWE\-bench is still the most popular benchmark for evaluating LLMs on SE tasks\. One of the goals of our contribution is to show the limitations of self\-correction feedback\.

## 7Conclusions: what should be done

In this position paper, we argue that there is no “one size fits all” approach to DL\-for\-code model evaluation, and that using a single or a limited number of benchmarks to assess model performance does not provide sufficient information about model capabilities\. We suggest using four complementary approaches:single\-score benchmarksfor iteration,benchmark suitesfor capability profiles,human\-in\-the\-loop studiesfor narrow deployments, andholistic open\-ended evaluationsfor frontier models\. To make benchmark suites reliable, we suggest treating them as research infrastructure: funding creation and long\-term maintenance \(e\.g\., targeted grants and industry sponsorship\) and rewarding versioned updates via dedicated conference tracks\. We also urge the community to ground benchmark suites in a shared capability taxonomy that covers tasks, domains, languages, and aggregation to ensure construct validity, and acknowledge the limitations of the benchmarking methods for the task at hand\. Finally, we also call on DL for code researchers working on training recipes and coding agents to explicitly report what capabilities transfer across tasks and to characterize measurement limitations of their evaluations\.

## References

- R\. Aleithan, H\. Xue, M\. M\. Mohajer, E\. Nnorom, G\. Uddin, and S\. Wang \(2024\)Swe\-bench\+: enhanced coding benchmark for llms\.arXiv preprint arXiv:2410\.06992\.Cited by:[§6](https://arxiv.org/html/2608.13566#S6.p4.1)\.
- Anthropic \(2025\)Project vend: can claude run a small shop? \(and why does that matter?\)\.Note:https://www\.anthropic\.com/research/project\-vend\-1Cited by:[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p7.1)\.
- I\. Badertdinov, A\. Golubev, M\. Nekrashevich, A\. Shevtsov, S\. Karasik, A\. Andriushchenko, M\. Trofimova, D\. Litvintseva, and B\. Yangel \(2025\)SWE\-rebench: an automated pipeline for task collection and decontaminated evaluation of software engineering agents\.arXiv preprint arXiv:2505\.20411\.Cited by:[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p4.1)\.
- A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner, N\. Foroutan, C\. Schmitz, K\. Korgul, H\. Batra,et al\.\(2025\)Measuring what matters: construct validity in large language model benchmarks\.arXiv preprint arXiv:2511\.04703\.Cited by:[§3](https://arxiv.org/html/2608.13566#S3.p5.1),[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p3.1),[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p5.1)\.
- M\. e\. al\. Chen \(2021\)Evaluating large language models trained on code\.arXiv:2107\.03374\.Cited by:[§2\.1](https://arxiv.org/html/2608.13566#S2.SS1.p2.1)\.
- F\. Chollet \(2019\)On the measure of intelligence\.arXiv preprint arXiv:1911\.01547\.Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p1.1)\.
- N\. Chowdhury, J\. Aung, C\. J\. Shern, O\. Jaffe, D\. Sherburn, G\. Starace, E\. Mays, R\. Dias, M\. Aljubeh, M\. Glaese, C\. E\. Jimenez, J\. Yang, L\. Ho, T\. Patwardhan, K\. Liu, and A\. Madry \(2024\)Introducing SWE\-bench verified\.External Links:[Link](https://openai.com/index/introducing-swe-bench-verified/)Cited by:[§2\.1](https://arxiv.org/html/2608.13566#S2.SS1.p3.1)\.
- M\. N\. M\. Christian Mürtz \(2025\)Agents in the wild \- dashboard\.Note:Interactive web dashboard\. Code available at[https://github\.com/logic\-star\-ai/insights](https://github.com/logic-star-ai/insights)[https://insights\.logicstar\.ai](https://insights.logicstar.ai/)External Links:[Document](https://dx.doi.org/10.5281/zenodo.15846865),[Link](https://doi.org/10.5281/zenodo.15846865)Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p2.1)\.
- X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane,et al\.\(2025\)SWE\-bench pro: can ai agents solve long\-horizon software engineering tasks?\.arXiv preprint arXiv:2509\.16941\.Cited by:[§2\.1](https://arxiv.org/html/2608.13566#S2.SS1.p3.1),[2nd item](https://arxiv.org/html/2608.13566#S5.I1.i2.p1.1)\.
- DigitalAppliedTeam \(2025\)Chinese ai models beat gpt\-4: kimi k2, qwen 3, glm 4\.5\.Note:[Digital Applied Blog](https://www.digitalapplied.com/blog/chinese-ai-models-kimi-k2-qwen-3-coder-glm-4-5)Cited by:[§3](https://arxiv.org/html/2608.13566#S3.p3.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§4\.1](https://arxiv.org/html/2608.13566#S4.SS1.p4.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p2.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)Livecodebench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.13566#S2.SS1.p2.1)\.
- N\. Jain, J\. Singh, M\. Shetty, L\. Zheng, K\. Sen, and I\. Stoica \(2025\)R2e\-gym: procedural environments and hybrid verifiers for scaling open\-weights swe agents\.arXiv preprint arXiv:2504\.07164\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1),[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p3.1)\.
- J\. H\. Jimenez, N\. Sundaresan, S\. Gu, K\. Talamadupula, B\. Tate, S\. Nair, C\. Bhagavatula, K\. Lee, M\. Heard, A\. Grewal,et al\.\(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p1.1)\.
- S\. M\. Jordan, A\. White, B\. C\. Da Silva, M\. White, and P\. S\. Thomas \(2024\)Position: benchmarking is limited in reinforcement learning research\.arXiv preprint arXiv:2406\.16241\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p1.1),[§6](https://arxiv.org/html/2608.13566#S6.p2.1)\.
- M\. Luo, N\. Jain, J\. Singh, S\. Tan, A\. Patel, Q\. Wu, A\. Ariyak, C\. Cai, T\. Venkat, S\. Zhu, B\. Athiwaratkun, M\. Roongta, C\. Zhang, L\. E\. Li, R\. A\. Popa, K\. Sen, and I\. Stoica \(2025\)DeepSWE: training a state\-of\-the\-art coding agent from scratch by scaling rl\.Note:[DeepSWE announcement](https://pretty-radio-b75.notion.site/DeepSWE-Training-a-Fully-Open-sourced-State-of-the-Art-Coding-Agent-by-Scaling-RL-22281902c1468193aabbe9a8c59bbe33)Notion BlogCited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1),[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p3.1)\.
- Y\. Ma, R\. Cao, Y\. Cao, Y\. Zhang, J\. Chen, Y\. Liu, Y\. Liu, B\. Li, F\. Huang, and Y\. Li \(2024\)Lingma swe\-gpt: an open development\-process\-centric language model for automated software improvement\.arXiv preprint arXiv:2411\.00622\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1)\.
- D\. Manheim and S\. Garrabrant \(2018\)Categorizing variants of goodhart’s law\.arXiv preprint arXiv:1803\.04585\.Cited by:[§6](https://arxiv.org/html/2608.13566#S6.p4.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[§2\.1](https://arxiv.org/html/2608.13566#S2.SS1.p3.1)\.
- N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers \(2023\)Mteb: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.Cited by:[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p2.1)\.
- OpenHands \(2025\)Introducing openhands lm 32b – a strong, open coding agent model\.Note:[OpenHands Blog](https://openhands.dev/blog/introducing-openhands-lm-32b-a-strong-open-coding-agent-model)Cited by:[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p3.1)\.
- Y\. Ouyang, J\. Yang, and L\. Zhang \(2024\)Benchmarking automated program repair: an extensive study on both real\-world and artificial bugs\.InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,pp\. 440–452\.Cited by:[§D\.3](https://arxiv.org/html/2608.13566#A4.SS3.p2.1)\.
- J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. Zhang \(2024\)Training software engineering agents and verifiers with swe\-gym\.arXiv preprint arXiv:2412\.21139\.Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p1.1)\.
- QwenTeam \(2025\)Qwen3\-coder: agentic coding in the world\.Note:[https://qwen\.ai/blog?id=qwen3\-coder](https://qwen.ai/blog?id=qwen3-coder)qwen\.ai blogCited by:[§3](https://arxiv.org/html/2608.13566#S3.p3.1)\.
- Y\. Tang, K\. Zheng, G\. Synnaeve, and R\. Munos \(2025\)Optimizing language models for inference time objectives using reinforcement learning\.arXiv preprint arXiv:2503\.19595\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1)\.
- Y\. Wei, F\. Cassano, J\. Liu, Y\. Ding, N\. Jain, Z\. Mueller, H\. de Vries, L\. Von Werra, A\. Guha, and L\. Zhang \(2024\)Selfcodealign: self\-alignment for code generation\.Advances in Neural Information Processing Systems37,pp\. 62787–62874\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1)\.
- M\. Wu and A\. F\. Aji \(2025\)Style over substance: evaluation biases for large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 297–312\.Cited by:[§5\.2](https://arxiv.org/html/2608.13566#S5.SS2.p6.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§2\.2](https://arxiv.org/html/2608.13566#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p2.1)\.
- J\. Yang, K\. Lieret, C\. E\. Jimenez, A\. Wettig, K\. Khandpur, Y\. Zhang, B\. Hui, O\. Press, L\. Schmidt, and D\. Yang \(2025b\)Swe\-smith: scaling data for software engineering agents\.arXiv preprint arXiv:2504\.21798\.Cited by:[§D\.3](https://arxiv.org/html/2608.13566#A4.SS3.p2.1),[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1),[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p3.1),[§6](https://arxiv.org/html/2608.13566#S6.p3.1)\.
- Z\. Yu, Y\. Zhao, A\. Cohan, and X\. Zhang \(2024a\)Humaneval pro and mbpp pro: evaluating large language models on self\-invoking code generation\.arXiv preprint arXiv:2412\.21199\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1)\.
- Z\. Yu, W\. Gu, Y\. Wang, X\. Jiang, Z\. Zeng, J\. Wang, W\. Ye, and S\. Zhang \(2024b\)Reasoning through execution: unifying process and outcome rewards for code generation\.arXiv preprint arXiv:2412\.15118\.Cited by:[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1)\.
- L\. Zeng, Y\. Li, Y\. Xiao, C\. Li, C\. Y\. Liu, R\. Yan, T\. Wei, J\. He, X\. Song, Y\. Liu,et al\.\(2025\)Skywork\-swe: unveiling data scaling laws for software engineering in llms\.arXiv preprint arXiv:2506\.19290\.Cited by:[§1](https://arxiv.org/html/2608.13566#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.13566#S2.SS3.p2.1),[§4\.2](https://arxiv.org/html/2608.13566#S4.SS2.p3.1)\.

## Appendix ACoding capability taxonomy illustration

To illustrate what a coding task taxonomy could look like, we include a sample taxonomy, see Tables[3](https://arxiv.org/html/2608.13566#A1.T3),[4](https://arxiv.org/html/2608.13566#A1.T4)\. This taxonomy was created in a discussion between two of the paper authors and isnotgrounded in qualitative and quantitative research\. As such, it is merely an illustration of what kind of a taxonomy would be helpful to benchmark creators so that they can ensure the construct validity of their benchmarks and assess their limitations\. We urgenotto use this taxonomy as is and call for research community to develop a proper taxonomy with research methods from other areas of science such as psychometrics and social anthropology\.

Task familyIndividual tasksHow to gatherDesign & planningArchitecture & planningAPI, schema, module design, subsystem decompositionMine ADRs \(Architecture Decision Record\), RFCs \(Request for Comments\), epic issues, and linked PR series; author tasks from real design docsRequirement analysisProblem decomposition, requirement verification \(e\.g\., checking for non\-contradiction\)Mine development roadmaps and issue trackers; mine design discussions for requirements verificationUnderstanding & communicationRepo understandingRetrieve relevant files; answer a repo questionPRs, issues, commit\-message to changed\-files pair, comments or discussions in GitHub, synthetic generationDocumentation & communicationPR summary, commit message, code explanation, README\.md, changelogs, project documentationMine accepted PR descriptions, docs diffs, maintainer answers, README updates, project documentation and changelogsData wranglingInput and output prediction for single and many\-step pipelineCreate diverse outputs via fuzzing testing on various data pipelinesCoding & feature developmentRepo\-level implementation / featuresAdd endpoint/page/workflow across multiple filesMine enhancement/feature PRs; filter out bug fixesRepo\-level synthesis & completionFill a function/class/file\-sized hole using info on the repoMask historical commits; create repo hole\-filling tasks with hidden tests\. Hide method implementations, synthetically generating docs for it\.Local synthesis & completionFill a function/class/file\-sized holeUse fresh contest problemsValidation, diagnosis & feedbackDebugging & repairReproduce bug, localize root cause, patch itMine bug reports with fix commits, CI failures with later fixes, stack traces, and failing testsTesting & verificationGenerate unit, integration, regression testsHide existing tests, mine test\-adding commits, or preserve new regression tests from bug\-fix PRsReview & critiqueReview a diff and identify concrete issues, decide whether to accept PRMine merged PRs with human review comments and suggestionsTable 3:Task families, individual software\-engineering tasks, and possible data\-gathering strategies, part 1\.Task familyIndividual tasksHow to gatherCodebase evolution & adaptationRefactoringBehavior\-preserving extract, rename, move, modularizeMine refactor\-only commits/PRs using detectors plus post\-filteringMigration & compatibilityUpgrade runtime, library, framework, API, language versionMine upgrade PRs, version bumps, deprecation\-fix commits, lockfile changesCode translationSwitch codebase to another programming languageMine code transformation PRsHardening & optimizationSecurity hardeningFix a vulnerability or produce secure repo\-level codeMine CVEs \(Common Vulnerabilities and Exposures\), advisories \(vulnerability reports\), Static/Dynamic Application Security Testing findings, and security\-fix PRsPerformance optimizationSpeed up code while preserving behavior on repo\-level tasksMine performance\-improving PRs or pair tasks with optimized baselines\. Synthetically generate the code optimization\.Local performance optimizationSpeed up code while preserving behavior on stand\-alone tasksUse fresh contest problems, mine solutions with various time/memory consumptionDelivery infrastructure & lifecycleEnvironment / CI / maintenanceSet up repo, fix failing CI, keep code green over iterationsMine setup docs/workflows, failing CI snapshots, and long commit historiesTool and skill usageCodebase navigationParsing multi\-file dependencies, finding relevant methodCreate synthetic datasets with code parsers such as tree\-sitterTool usageRunning correct commands in terminal, calling external API, using tools such as GitMine repos with API usages, mine tool and terminal usage patterns from manuals and educational tasksStyle followingFollowing code style, following documentation guidesMine documentation guides, code style guides such as PEP, diffs before / after linter usageTable 4:Task families, individual software\-engineering tasks, and possible data\-gathering strategies, part 2\.
## Appendix BDjango Benchmark Suite Details

In this section we provide a detailed description of our benchmark suite construction, expanding on the methodology outlined in Section[4](https://arxiv.org/html/2608.13566#S4)\.

### B\.1Repository Selection and Leakage Control

We base our benchmark suite on a single, frozen snapshot of Django at version 4\.0\.4, commit hash89807​f​b​d​e​8​b​7​b​17​d​00434​b​c​4695535855​e​96​f​e​7789807fbde8b7b17d00434bc4695535855e96fe77, dated 11th Apr 2022\. We select this snapshot for several reasons\. We choose Django because it constitutes 46% of SWE\-bench Verified, allowing us to test a stronger version of the meaning gap thesis: if external checkpoints perform well on SWE\-bench Verified, they should perform well on Django issue resolution by extension\. Therefore, failure to improve on our benchmark suite indicates failure of cross\-task transfer, suggesting even stronger limitations on performance in truly generalized setups with non\-SWE\-bench repositories and tasks other than issue\-resolution\. We choose Django 4 specifically because it represents the largest share of major Django releases in SWE\-bench Verified, and Django introduces breaking changes between major versions\. Finally, we select this particular commit because it corresponds to the last release before Django 4\.1 development began\. We hypothesize this version is most likely to have addressed issues from the initial Django 4\.0 release while remaining unaffected by parallel development of the next version\. As we have no prior knowledge of Django development timeline, this motivation allows us to choose snapshot that is likely stable and representative\.

### B\.2Train\-Test Split

After cloning the repository, we split the source files into train and test splits while tracking inheritance structure\. For each class, we place the class, its ancestors, and its descendants into the same split\. This prevents cross\-split contamination via inherited methods, mixins, and abstract base classes\. The code outside classes, such as standalone functions or module\-level imports, is assigned to a particular file and never duplicates across splits\. We do not use tests \(code that lies in thetestsfolder\) for training or inference purposes\. Table[5](https://arxiv.org/html/2608.13566#A2.T5)summarizes the key statistics of our training and testing datasets\.

Table 5:Dataset statistics for training and testing splits\.StatisticTrainTestNumber of methods3,180359Avg\. method length \(lines\)10\.279\.25Median method length \(lines\)44Number of unique files42558Number of inheritance chains1,585184Avg\. generated documentationlength \(chars\)243\.87236\.60Median generated documentationlength \(chars\)233\.00229\.00Methods w/o original doc\.74\.28%78\.27%The training dataset is approximately 9×\\timeslarger than the testing dataset, with similar average method lengths across both splits, indicating consistent complexity distribution\. The number of inheritance chains reflects our splitting strategy that keeps related classes together to prevent cross\-split contamination\.

### B\.3Documentation Synthesis

Django project documentation is uneven: some methods are documented thoroughly, while others have minimal or no documentation at all\. To create a uniform evaluation setup, we synthesize standardized docstrings for every method of every class using DeepSeek\-R1\. We enforce a rigid schema for all generated docstrings:

- •What it does:One or two sentences describing the method’s purpose and behavior\.
- •Main inputs:A one\-sentence description of the method’s parameters and their expected types or constraints\.
- •Outputs and side effects:A one\-sentence description of the return value, plus description of expected failure modes \(such as raising an exception in certain scenarios\)\.

We keep the original comments present in the source code and merge them with generated docstrings whenever applicable, preserving any domain\-specific information the original developers included\.

### B\.4Test Suite and Method\-Test Alignment

We build a Django test environment that runs the official 4\.0\.4 test suite\. For each method in our benchmark, we identify the tests that determine its correctness through a systematic process:

1. 1\.For each target method, we replace its body with a stub implementationreturn None\.
2. 2\.We run all Django tests and record which tests fail due to the stubbed method\.
3. 3\.We map the failed tests to the method in question\.
4. 4\.At evaluation time, we run only these mapped tests to check method validity\.

Methods for which no tests failed when stubbed withreturn Noneare excluded from the benchmark, as their correctness cannot be verified through testing\.

For a generated method to be considered correct, it must pass all mapped tests\. This approach ensures that we evaluate only the functionality directly related to each target method, rather than running the entire test suite for every evaluation\.

## Appendix CAdditional evidence for SWE\-bench limitations

### C\.1Error\-flow analysis

To support our position on possible lack of cross\-task transfer, we performed a per\-example error\-type flow analysis that traces how each test outcome changes when a base model is replaced by its fine\-tuned counterpart\.

![Refer to caption](https://arxiv.org/html/2608.13566v1/x1.png)Figure 1:Per\-example error type flow Sankey diagramAcross all seven model pairs and three benchmarks \(7 539 examples\), fine\-tuning drops the OK rate from 52\.4% to 43\.1%\. The modal dominant regression path is OK → NOT\_RUN: fine\-tuned models emit agent\-style output \(XML tags, markdown fences, preamble code\) that the harness cannot parse\. In total, 47\.2% of regressions come from format or parsing error \(NOT\_RUN \+ IndentationError\), and 52\.8% of regressions are caused by essential mistakes such as AssertionError or SyntaxError\. Thus, the impact of coding capability degradation and failure to follow the format are compatible\. Model capacity determines whether this overfitting occurs\. Comparing SWE\-agent\-LM\-7B and \-32B \(both fine\-tuned on SWE\-smith and have\>13%\>13\\%improvements on SWE\-bench\) on Method Generation:

- •7B \(52\.9% → 5\.8% OK\): the largest flow is OK → NOT\_RUN \(113 examples\)\. The model wraps output in \[object Object\]…\[object Object\] tags instead of a bare def block; 54 further examples acquire IndentationError\.
- •32B \(64\.1% → 66\.6% OK\): the dominant flow is OK → OK \(213\); 11 previously failing cases are now fixed\. The larger model compartmentalises agent formatting and retains the output format expected by the harness\. On the per\-model panels it is a long tail of rare types \(each two examples or less, e\.g\. NameError, ImportError, RecursionError\)\.

To sum up, while checkpoints fine\-tuned on agentic trajectories often fail due to format non\-adherence, failure modes are numerous and their distribution is long\-tailed\. More than half of regressions are due to conceptual errors in model generations\. These failure modes cannot be observed if a checkpoint is evaluated on a single benchmark such as SWE\-Bench Verified, supporting our position that current evaluation practices may obscure problems with cross\-task transfer\.

### C\.2Impact of agentic scaffolding

To check the impact of agentic scaffolding, we compare the SWE\-agent checkpoint to raw models in an agentic scaffolding setup on method generation task, see Table[6](https://arxiv.org/html/2608.13566#A3.T6)\. This setup is organized as follows\. SWE\-agent is placed into Django 4\.0\.4 repository\. It gets a prompt with the file path to the target file and problem statement \(description of target method to generate and system prompt to generate it\)\. The agent does not get the file contents directly\. The target file is missing the method that the agent should implement, similar to no\-scaffolding setup\. The agent gets ten steps to implement it\. The agent can only submit solution once, and the solution is either submitted by a special tool call, or submitted automatically when the step cutoff is hit\.

Model / scaffoldingScaffolding, 10 stepsNo scaffoldingQwen2\.5\-Coder\-7B\-Instruct0\.052\.92SWE\-agent\-LM\-7B24\.515\.85Qwen2\.5\-Coder\-32B\-Instruct8\.5964\.07SWE\-agent\-LM\-32B39\.6266\.57Table 6:Model performance with and without SWE\-agent scaffolding\.When both SWE\-agent checkpoints and the raw model are put into the same scaffolding, the fine\-tuned checkpoints outperform the raw model\. However, both for raw models and SWE\-agent\-LM\-32B checkpoint removal of scaffolding further improves the results in ten\-step setup\. To check further if ten steps are insufficient and whether the agent runs out of steps, we evaluate SWE\-agent\-LM\-32B checkpoint in agentic scaffolding with 75\-step limit\. We consider this particular checkpoint as it has the smallest difference with the raw model non\-agentic performance, and we do not run other checkpoints due to budget limitations\. In this setup, SWE\-agent\-LM\-32B scores63\.5163\.51, so it takes 75 steps in an agentic scaffolding for a particular SWE\-agent checkpoint to match a non\-agentic setup for either SWE\-agent checkpoint or raw model\. The difference between ten and 75\-step scaffolding performance highlights that agentic setup requires multiple tries to solve the task and suggests that in ten\-step setup running out of steps is the key failure mode\.

All in all, this indicates that regardless of the fine\-tuning approach some tasks are better solved without an agentic scaffolding\. The failure of SWE\-agent\-LM\-32B checkpoint to outperform the raw model in non\-agentic setup further supports our point on the lack of cross\-task transfer\. We stress that creators of SWE\-agent checkpoints could not have checked this without creating additional benchmarks \(a significant research and technical effort\), so this reflects a failure of the evaluation ecosystem, not of the researchers themselves\.

## Appendix DTask Descriptions

We create three distinct benchmarks from our Django repository snapshot\. Each benchmark is accompanied by a corresponding fine\-tuning dataset for instruction tuning on the same task format\. Below we describe the input\-output format and evaluation protocol for each task\.

### D\.1Method Generation

In the method generation task, the model is given a method specification and must generate the complete method body\.

##### Input format\.

The input consists of the following components in order:

1. 1\.A task description instructing the model to generate the missing method body\.
2. 2\.The target class name and the declaration \(signature\) of the method to be written\.
3. 3\.The synthesized docstring for the target method, following our standardized schema\.
4. 4\.The entire source file content, with the target method body removed\. We reorganize non\-target code blocks to place the missing method at the end of the file, but otherwise preserve the original file structure\.

##### Output format\.

The expected output is the full method code \(including its signature\) wrapped in a‘‘‘pythoncode block\. We extract the code from the last code block in the model’s response and ignore any additional text or code blocks\. Appending the correct output to the provided file context should yield a syntactically valid and functionally correct file\.

##### Evaluation\.

We evaluate with the pass@1 metric at temperatureT=0T=0\(greedy decoding\)\. For each generated method, we insert it into the source file and run only the mapped Django tests for that method\. A solution is considered correct if and only if all mapped tests pass\.

### D\.2Method Completion

The method completion task is similar to method generation, but provides partial implementation context instead of a docstring\.

##### Input format\.

The input consists of:

1. 1\.A task description instructing the model to complete the method\.
2. 2\.The source file content up to and including approximately half of the target method body\.
3. 3\.The model must generate the remaining portion of the method\.

Unlike method generation, we do not provide the synthesized docstring\. Instead, the model must infer the method’s intended behavior from the partial implementation and surrounding code context\.

##### Output format\.

The model should generate only the completion—the remaining lines of the method body that, when concatenated with the provided prefix, form a complete and correct method\. We extract the generated code and append it to the provided partial method\.

##### Evaluation\.

We use the same evaluation protocol as method generation: the completed method is inserted into the source file and validated against the mapped Django tests using pass@1\.

### D\.3Program Repair

In the program repair task, the model receives a file with a broken method and must identify and fix the bug\. To create plausible buggy methods, we use a cascading approach: we first collect failing outputs from three runs of Qwen3\-32B \(89 samples\), then from Qwen3\-8B for tasks solved by the larger model \(83 samples\), and finally from Qwen3\-0\.6B \(126 samples\)\. For the remaining 61 tasks solved by all models, we shuffle non\-control tokens within methods\.

The idea of creating synthetic program repair benchmark follows\(Ouyanget al\.,[2024](https://arxiv.org/html/2608.13566#bib.bib48)\), our approach for creating LLM\-generated corrupted methods is analogous to “LM\-rewrite” approach for SWE\-smith dataset creation\(Yanget al\.,[2025b](https://arxiv.org/html/2608.13566#bib.bib52)\), and our fallback approach of token shuffling and method candidate shuffling is inspired by the “Procedural modification” approach of SWE\-smith\.

##### Input format\.

The input consists of:

1. 1\.A task description explaining that one method in the file is broken and needs to be fixed\.
2. 2\.The complete source file containing one method with an introduced bug\.
3. 3\.The full stack trace from the Django test suite when run against the broken method, indicating which tests failed and where the error occurred\.

The model must analyze the stack trace, identify the broken method, understand the nature of the bug, and generate a corrected version\.

##### Output format\.

The model should return only the fixed method implementation\. We extract it and substitute the broken method in the source file\.

##### Evaluation\.

We run the mapped tests for the originally broken method\. A repair is considered successful if all mapped tests pass\. We evaluate with pass@1 at temperatureT=0T=0\.

### D\.4Benchmark Verification with Proprietary Models

We evaluate a diverse set of state\-of\-the\-art proprietary LLMs on our benchmark suite\. Our goal is not to rank these foundation models, but to verify our benchmarks are well\-posed and solvable by the best LLMs\.

We evaluate all the models under a setup identical to the other evaluations we did\. The only exception is the temperature for the GPT\-5 model, as we could not setT=0T=0for it\. We use evaluation protocol identical to the one we use for other model checkpoints, and observe no issues with context sizes, rate limits or any other problems that can arise when using proprietary models through API calls\.

ModelCGCCPROpenAI GPT\-4\.176\.6064\.9061\.84OpenAI GPT\-584\.6877\.9979\.67Google Gemini 2\.5 Pro83\.0151\.2567\.97Google Gemini 3 Pro86\.3578\.2761\.00Google Gemini 3 Flash85\.7973\.2662\.95Anthropic Claude Haiku 4\.574\.6558\.7761\.56Anthropic Claude Sonnet 4\.582\.7377\.4469\.36Anthropic Claude Opus 4\.588\.0276\.6078\.27Table 7:Performance of Proprietary LLMs##### Result Analysis\.

The results in Table[7](https://arxiv.org/html/2608.13566#A4.T7)provide several insights into the properties of our benchmarks\.

First, all three tasks are clearly solvable by contemporary high\-capability models, with top\-performing systems exceeding 80% pass@1 on method generation and method completion, and almost reaching this score for program repair\. This indicates that the benchmarks are well\-posed and solvable\.

Second, the results align with expected capability orderings within model families\. GPT\-5 outperforms GPT\-4\.1 across all tasks\. Claude Opus 4\.5 outperforms Sonnet 4\.5 on two tasks out of three, and Sonnet outperforms Haiku 4\.5 on all three tasks\. Gemini 3 Pro and Flash outperform Gemini 2\.5 Pro on code generation and code completion\. These orderings are consistent with the models’ relative positions on other established benchmarks\.

Third, we observe a consistent ordering of task difficulty across models\. Method generation \(CG\) is generally easier than method completion \(CC\) and program repair \(PR\), similarly to what we observe for the open\-weights models and post\-trained checkpoints\.

Fourth, the table reveals that even frontier models exhibit uneven capability profiles across tasks, with various models having different strengths and weaknesses\. For example, GPT\-5 performs significantly better than Gemini 3 Pro on code completion, but is not better on code generation and program repair\.

All these observations support our claim that the proposed benchmarks are well\-posed, pass the sanity checks, and test distinct aspects of LLM coding capabilities, which are, although connected, still distinguishable\. Thus, we can use them as a diagnostic tool to support our hypothesis on meaning gap and risk of cross\-task transfer\.

## Appendix EFine\-tuning Setup

We fine\-tune Qwen2\.5\-Coder\-7B\-Instruct and Qwen2\.5\-Coder\-32B\-Instruct on each of the three task modalities described above\. We choose these models because most of the external checkpoints we evaluate were built on top of one of these two base models, enabling direct comparison\.

We do not claim that our fine\-tuning experiments extract all possible value from the repository data\.Our goal is to demonstrate how a checkpoint created with a reasonable, standard fine\-tuning approach performs across different tasks, and not to achieve state\-of\-the\-art results through extensive hyperparameter optimization\. Our configuration represents a practical setup that practitioners might use when adapting models to new domains, revealing how task\-specific training affects cross\-task generalization\.

### E\.1LoRA Configuration

We use Low\-Rank Adaptation \(LoRA\) for parameter\-efficient fine\-tuning with the following configuration:

- •Rank:128
- •Alpha:128
- •Dropout:0\.05
- •Target modules:All attention and MLP projection layers \(q\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj,down\_proj\)

We target all linear layers in the transformer architecture to maximize the model’s capacity to adapt to the new task distribution while keeping the number of trainable parameters manageable\.

### E\.2Training Hyperparameters

- •Optimizer:AdamW with weight decay of 0\.01
- •Learning rate:1×10−51\\times 10^\{\-5\}
- •Learning rate scheduler:Cosine annealing
- •Warmup steps:30
- •Maximum gradient norm:1\.0
- •Number of epochs:3
- •Batch size: - –1 per device with gradient accumulation over 8 steps \(effective batch size of 8\) for Qwen2\.5\-Coder\-7B\-Instruct - –4 per device with gradient accumulation over 8 steps \(effective batch size of 32\) for Qwen2\.5\-Coder\-32B\-Instruct

In every case, we evaluate the last epoch checkpoint\.

These settings are*not*the result of extensive hyperparameter search\. We deliberately use standard, reasonable defaults to demonstrate that even without careful tuning, single\-task fine\-tuning produces models with improved in\-distribution performance but limited cross\-task transfer\. This supports our main thesis about the meaning gap in benchmark evaluation and risks of missing it in single\-benchmark evaluation setup\.

## Appendix FPrompt Templates

This section provides the exact prompt templates used for each task\. Each task uses a system prompt that establishes the assistant’s role and output constraints, followed by a user prompt template that provides the specific task context\. Placeholders in curly braces \(e\.g\.,\{file\_content\}\) are replaced with actual values at inference time\.

### F\.1Method Generation Prompts

##### System prompt\.

YouareahelpfulPythoncode\-generationassistant\.

\*\*Outputrules\*\*

\*Return\*\*only\*\*thetargetmethod\(signature\+body\)inside\*\*one\*\*fencedcodeblocklabeled‘python‘\.

\*Do\*\*not\*\*includethecontainingclassoranyothercode\.

\*Do\*\*not\*\*addimportsortop\-leveldefinitions\.

\*Noproseorexplanationsoutsidethecodeblock\.Commentsareallowed\*\*inside\*\*thecode\.

\*\*Signature&decoratorrules\*\*

\*Startwiththe\*\*exact\*\*methoddeclarationprovidedin‘\{method\_declaration\}‘\(name,parameters,defaults,‘async‘,returnannotation\)\-\-do\*\*not\*\*changeit\.

\*Youmayadddecorators\*\*immediatelyabove\*\*thedeclaration\*\*onlyifnecessary\*\*forcorrectness\*\*withinaclass\*\*:

\*\*Implementationrules\*\*

\*Matchtheproject’sstyle\(typing,errorhandling,logging,docstrings\)\.

\*Useonlymodulesalreadyimportedinthefile;ifatinyhelperisrequired,defineit\*\*asanestedfunctioninsidethemethod\*\*\.

\*Accessonlyattributesthatexistintheclasspertheprovidedfileunlessthedescriptionexplicitlyintroducesthem\.

\*Preferclear,test\-readylogic\.Avoidplaceholderslike‘\.\.\.‘,‘pass‘,or‘raiseNotImplementedError‘unlessexplicitlyrequired\.

\*Ensurethemethodisself\-containedandsyntacticallyvalid\.

\*\*Format\*\*

‘‘‘python

\#methoddeclaration\(withanalloweddecoratorifnecessary\)followedbythemethodbodyonly

‘‘‘

##### User prompt template\.

Generatetheimplementationofthemethod‘\{method\_name\}‘forclass‘\{class\_name\}‘accordingtothedescription\.

\*\*Returnonlythismethod\*\*\-\-startwiththe\*\*exact\*\*declarationshownbelow\(donotchangeit\)\.Ifadecoratorisnecessaryforcorrectnessinsidetheclass,placeitonthelineabovethedeclaration\.Puteverythinginsideasingle‘python‘codeblock\.

Methoddeclaration:

‘‘‘python

\{method\_declaration\}

‘‘‘

Methoddescription:

\{method\_description\}

Filecontext:

‘‘‘python

\{file\_content\}

‘‘‘

### F\.2Method Completion Prompts

##### System prompt\.

YouareahelpfulPythoncode\-completionassistant\.

\*\*Outputrules\*\*

\*Return\*\*only\*\*thenewcode\(thecontinuation\),inside\*\*one\*\*fencedcodeblocklabeled‘python‘\.

\*Do\*\*not\*\*repeatanypartoftheprovidedfile\.

\*Do\*\*not\*\*includeexplanationsortextoutsidethecodeblock\.

\*Theoutputwillbedirectlyinsertedintotheendofthefile,sothecodemustbecorrectlyformatted,andmust\*\*preserveexistingindentationlevels\*\*toavoidissues\.Thefilefromthecontext,plusthegeneratedcodeshouldcombineintoacorrectlyworkingPythonfile

\*Commentsareallowed\*\*inside\*\*thecode,sparingly\.

\*\*Completionrules\*\*

\*Continuefromtheendoftheprovidedsnippet,completingthe\*\*currentlyopenfunction/method\*\*untilthefileisvalidandthemethodisfullyimplemented\.

\*Do\*\*not\*\*generatemethoddeclarationorexistingbody,onlythecontinuation\.

\*Do\*\*not\*\*modifypriorlinesorsignatures;do\*\*not\*\*introducenewtop\-levelcode\.

\*Matchexistingstyle\(typing,naming,docstrings,errorhandling,logging\)\.

\*Useonlyalready\-importedmodulesandbuiltins\.Ifahelperisrequired,implementit\*\*inside\*\*thecompletedmethod\(local/nested\)\.

\*Ensuretheresultissyntacticallyvalid,self\-containedwithinthecompletedregion,andaimstopasstests\.

\*Avoidplaceholderslike‘\.\.\.‘,‘pass‘,‘raiseNotImplementedError‘unlesstheyaretheintendedfinalbehavior\.

\*\*Format\*\*

‘‘‘python

\#yourcontinuationonly

‘‘‘

##### User prompt template\.

YouaregivenaPythonfilewithan\*\*incompletemethod\*\*\.Complete\*\*only\*\*themissingcontinuationsothatthefilebecomesvalidandthemethodisfullyimplemented\.

\*\*Returnonlythenewcode\*\*,insideonefenced‘python‘codeblock\.\*\*Donot\*\*repeatanyexistinglinesfromthefile\.Do\*\*not\*\*includeexplanationsorexamples\.

Filecontent:

‘‘‘python

\{file\_content\}

‘‘‘

### F\.3Program Repair Prompts \(With Test Output\)

##### System prompt\.

YouareahelpfulPythonprogram\-repairassistant\.

\*\*Yourtask\*\*

\*Fromtheprovidedfileandtestoutput,identifythe\*\*single\*\*incorrectmethodandproduceafixedimplementation\.

\*\*Outputrules\*\*

\*Return\*\*only\*\*therepairedmethod\(exactsignature\+body\)inside\*\*one\*\*fencedcodeblocklabeled‘python‘\.

\*Repairexactlyonemethodintheprovidedfilethatismostlikelyincorrect\.

\*Do\*\*not\*\*includetheclasswrapper,importsoranyothercode\.

\*Do\*\*not\*\*includeexplanations,prose,orexamplesoutsidethecodeblock\.

\*Commentsareallowed\*\*inside\*\*thecodeandaftersignature\.

\*\*Repairrules\*\*

\*Keepthemethod’s\*\*signatureexactlyasinthefile\*\*\(name,parameters,defaults,‘async‘,returnannotation\)\.Do\*\*not\*\*changeit\.

\*Do\*\*not\*\*addorchangedecorators\.

\*Do\*\*not\*\*introducenewimportsortop\-leveldefinitions\.

\*Useonlysymbolsalreadyavailableinscope;ifatinyhelperisneeded,defineit\*\*asanestedfunctioninsidethemethod\*\*\.

\*Followtheproject’sstyle\(typing,errorhandling,logging\)\.

\*Usethe\*\*testoutput\*\*toguidetherepair;ensurethefixaddressesfailingcases\.

\*Avoidplaceholderslike‘\.\.\.‘,‘pass‘,or‘raiseNotImplementedError‘unlessthatistheintendedbehaviorinthetestoutputs\.

\*Ifmanymethodslooksuspicious,fixtheonethatbestexplainsthefailingtests\.

\*\*Outputformat\*\*

‘‘‘python

\#exactmethoddeclarationfromthefile,followedbytherepairedbody

‘‘‘

##### User prompt template\.

YouaregivenacompletePythonfile\.Exactly\*\*one\*\*methodisincorrect\.Youalsoreceivethefailingtestfeedback\.

Identifytheincorrectmethodandoutputits\*\*repairedimplementation\*\*\.

\*\*Returnonlythismethod\*\*,startingwiththe\*\*exactsignatureasitappearsinthefile\*\*\(noclasswrapper,nocomments\),insideasingle‘python‘codeblock\.

Testoutput\(failingtests\):

‘‘‘

\{test\_output\}

‘‘‘

Fullfilecontext:

‘‘‘python

\{file\_content\}

‘‘‘

### F\.4Examples of Synthesized Documentation

This section provides examples of documentation synthesized by DeepSeek\-R1 for Django methods\. We present both cases where no original developer documentation existed and cases where the original docstring was available and incorporated\.

#### F\.4\.1Examples Without Original Django Documentation

##### Example 1:Article\.save

Method declaration:

defsave\(self,\*args,\*\*kwargs\):

Method body:

ifnotself\.id:

self\.created=datetime\.date\.today\(\)

returnsuper\(\)\.save\(\*args,\*\*kwargs\)

Generated documentation \(DeepSeek\):

\*Whatitdoes

Setsthe’created’datetothecurrentdateiftheinstanceisnew\(whenidisnotset\)\.

\*Maininputs

\*argsand\*\*kwargspassedtoparentsavemethod\.

\*Output/sideeffects

Savesthemodelinstancetodatabaseandupdates’created’fieldfornewinstances\.

##### Example 2:CustomErrorMessage\.clean

Method body:

ifself\.name1=="FORBIDDEN\_VALUE":

raiseValidationError\(

\{"name1":\[ValidationError\("Model\.clean\(\)errormessages\."\)\]\}

\)

elifself\.name1=="FORBIDDEN\_VALUE2":

raiseValidationError\(

\{"name1":"Model\.clean\(\)errormessages\(simplersyntax\)\."\}

\)

elifself\.name1=="GLOBAL\_ERROR":

raiseValidationError\("Globalerrormessage\."\)

Generated documentation \(DeepSeek\):

\*Whatitdoes

Validatesthemodelinstancebycheckingif’name1’containsforbiddenvalues\.

Raisesfield\-specificvalidationerrorsfor"FORBIDDEN\_VALUE"\(detailederror

structure\)and"FORBIDDEN\_VALUE2"\(simplifiedsyntax\),oraglobalvalidation

errorfor"GLOBAL\_ERROR"\.

\*Maininputs

Currentvalueofthemodel’s’name1’fieldduringvalidation\.

\*Output/sideeffects

RaisesValidationErrorwithstructurederrormessagesattachedtothe’name1’

fieldorasglobalerrors,blockingmodelsaveuntilresolved\.

#### F\.4\.2Examples With Original Django Documentation

##### Example 1:StartsWithRelation\.field

Method declaration with original Django docstring:

deffield\(self\):

"""

MakesReverseManyToOneDescriptorworkinbothdirections\.

"""

Generated documentation \(DeepSeek\):

\*Whatitdoes

Providesaccesstoremote\_fieldpropertytoenablebidirectional

ReverseManyToOneDescriptor\.

\*Maininputs

None\(propertyaccessor\)\.

\*Output/sideeffects

Returnsremote\_fieldreferenceenablingdescriptorfunctionalityinboth

directions\.

##### Example 2:Article\.articles\_from\_same\_day\_2

Method declaration with original Django docstring:

defarticles\_from\_same\_day\_2\(self\):

"""

Verboseversionofget\_articles\_from\_same\_day\_1,whichdoesacustom

databasequeryforthesakeofdemonstration\.

"""

Method body:

fromdjango\.dbimportconnection

withconnection\.cursor\(\)ascursor:

cursor\.execute\(

"""

SELECTid,headline,pub\_date

FROMcustom\_methods\_article

WHEREpub\_date=%s

ANDid\!=%s""",

\[connection\.ops\.adapt\_datefield\_value\(self\.pub\_date\),self\.id\],

\)

return\[self\.\_\_class\_\_\(\*row\)forrowincursor\.fetchall\(\)\]

Generated documentation \(DeepSeek\):

\*Whatitdoes

Retrievessame\-dayarticlesusingdirectdatabasequery\.

\*Maininputs

self\(instance\)\.

\*Output/sideeffects

ReturnslistofArticleinstancescreatedfromrawqueryresults\.