Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?

# Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
Source: [https://arxiv.org/html/2609.29410](https://arxiv.org/html/2609.29410)
Alexandru Stefan StoicaTraian RebedeaAffiliation:University Politehnica of Bucharest Bucharest, Romania traian\.rebedea@upb\.roMarian Cristian MihaescuAffiliation:University of Craiova Craiova, Romania cristian\.mihaescu@edu\.ucv\.ro

###### Abstract

Recent studies have shown that Large Language Models can effectively solve problems and fix bugs in diverse programming environments, including competitive programming\. Existing approaches primarily evaluate LLM performance in problem solving or bug fixing independently, but do not explore the relationship between these two capabilities\. This work focuses on determining how much the LLM deviates from a buggy solution to fix the bug compared to a human\-written patch, and if there is a bias towards generating entirely new solutions\. We construct a dataset with all the submissions \(∼\\sim3000\) from a couple of users from Codeforces, and we match each buggy submission with its corresponding human fix\. By using the similarity between the buggy solution and the human fix as a baseline, we evaluate the quality of LLM\-generated bug fixes on 3 OpenAI GPT models \(gpt\-5\-nano, gpt\-5\-mini, gpt\-5\.1\)\. We check if the generated solutions solve the problem by using the Codeforces\-R1 dataset, an openly available dataset that has tests generated with the DeepSeek\-R1 model\. Our findings suggest that LLMs tend to modify more lines than necessary compared to human fixes and, in some cases, generate entirely new solutions\. We also observe that LLMs solve more problems correctly when allowed to generate solutions from scratch rather than patch buggy submissions, even when those submissions are close to the human patch\. This has important implications for the design of AI\-assisted programming tools, particularly in supporting user debugging processes and promoting incremental problem\-solving strategies rather than solution replacement\.

###### Index Terms:

Large Language Models, Competitive Programming, Automatic Program Repair

## IIntroduction

Competitive programming is a mind sport in which participants try to come up with solutions to given problems \(algorithmic in nature\) that must satisfy some constraints\. The submission to a given problem is evaluated through multiple test cases that capture various edge cases and inform the user if a constraint is violated\. For a given problem to be considered Accepted, one has to pass all the test cases\. Besides being a sport, it also represents a method in which a user can learn more about algorithmic design by receiving feedback on their submission by following an iterative debugging process\.

The rapid advancement of Large Language Models \(LLMs\) has driven significant interest in their application to Automatic Program Repair \(APR\)\. Prior work has demonstrated that LLMs can both generate solutions to algorithmic problems\[[1](https://arxiv.org/html/2609.29410#bib.bib15)\]and patch buggy code\[[2](https://arxiv.org/html/2609.29410#bib.bib14)\]across a variety of settings\. However, existing approaches largely evaluate these two capabilities in isolation, leaving open an important question: when an LLM is given a buggy submission and tasked with fixing it, does it behave like a human debugger — making targeted, minimal edits — or does it tend to discard the original solution and generate a new one from scratch?

This distinction matters beyond mere academic curiosity\. In practice, AI\-assisted programming tools are increasingly integrated into iterative development workflows, where preserving the user’s intent and code structure is often as important as producing a correct solution\. A tool that systematically replaces rather than repairs code undermines the debugging process and offers little support for incremental problem solving\.

In this paper, we introduce a dataset that contains all the submissions from 7 Codeforces\[[3](https://arxiv.org/html/2609.29410#bib.bib11)\]users \(∼\\sim3000 submissions\), which we use to investigate how various strong closed\-source LLMs from the OpenAI family \(gpt\-5\-nano, gpt\-5\-mini, and gpt\-5\.1\) repair their buggy submissions relative to their own fix\. In other words, we identify how well the LLM preserves the original structure of the code in the buggy submission when generating a fix, compared to a human patch\. Using 3 methods of prompting: \(1\) a simple bug fix prompt; \(2\) a prompt that just generate a solution from scratch; \(3\) an optimized prompt via Genetic\-Pareto \(GEPA\)\[[4](https://arxiv.org/html/2609.29410#bib.bib1)\], a prompt optimization technique, we compare the similarity between the buggy problem and the generated fix to the similarity between the buggy problem and the human fix\. We also use Codeforces\-R1\[[5](https://arxiv.org/html/2609.29410#bib.bib3)\]dataset to evaluate the solutions for each problem using coding tests generated by DeepSeek\-R1\[[6](https://arxiv.org/html/2609.29410#bib.bib2)\]to determine if the code is correct\. With this in mind, we investigate the following research questions:

- - –RQ1:Given a buggy submission, how well does the LLM preserve the original user’s structure when fixing the bug? - –RQ2: Are language models better at generating code that solves a problem than at fixing bugs? - –RQ3: Is there a relationship between how much the AI preserves the user’s original code structure and its ability to correctly fix the bug?

In summary, this paper makes the following contributions: \(1\) We introduce a dataset of 3,000 submissions from 7 Codeforces users, capturing complete submission histories including buggy and accepted solutions\. \(2\) We systematically evaluate three prompting strategies — bug fix, scratch generation, and GEPA\-optimized — across three OpenAI models on their ability to repair buggy code\. \(3\) We propose and apply a structural similarity analysis to measure how faithfully LLM\-generated fixes preserve the original code, comparing it against human patches\. \(4\) We provide empirical answers to the interplay between structural preservation and correctness, with implications for AI\-assisted debugging tools in educational and professional contexts\.

## IIRelated Work

### II\-AEvaluation of LLMs in Solving Competitive Programming Problems

Zheng et al\.\[[7](https://arxiv.org/html/2609.29410#bib.bib4)\]have observed that the current models excel in cases of implementation and derivation of competitive programming solutions, while they perform significantly worse on problems that require heavy observation\. At the same time, LLMs alone struggle to solve hard problems or unseen problems, but human\-LLM collaboration significantly improves task performance\[[8](https://arxiv.org/html/2609.29410#bib.bib5)\]\. The same study finds that this is also true for bug identification\. A problem identified by both papers is that data contamination is a common problem in competitive programming evaluation of LLMs\.

### II\-BEvaluation of LLMs in Fixing Bugs in Competitive Programming Problems

Some recent works explored the capabilities of language models to fix incorrect competitive programming solutions\. DebugBench\[[9](https://arxiv.org/html/2609.29410#bib.bib12)\]is a benchmark with different types of bugs that was constructed via bug implantation with various LLMs on submissions from LeetCode\. After a rigorous human evaluation, multiple LLMs are evaluated against the benchmark and the results suggest that LLMs tend to perform better on syntax and reference errors compared to logical errors\. Mavalankar et al\.\[[10](https://arxiv.org/html/2609.29410#bib.bib13)\]introduce a golden set of example buggy / fix code pairs that is then used to enhance the performance of LLMs in bug fixing\. In\[[11](https://arxiv.org/html/2609.29410#bib.bib6)\]two Language Models \(PLBart\[[12](https://arxiv.org/html/2609.29410#bib.bib7)\]and CodeT5\[[13](https://arxiv.org/html/2609.29410#bib.bib8)\]\) were evaluated in the context of program repair on n\-gram based metrics and execution based metrics based on \(bug fix, human fix\) pairs, and it was found that the n\-gram based metrics are suboptimal in comparison with the execution based metrics\. A bug fix should be as localized as possible, and it should not change the structure of the code unless the user explicitly wants this\. Most existing works focus on the capacity of LLMs to solve bugs, but not on how they are solved\. To fill this gap, our work explores the differences between the human fixes and the generated fixes as well in terms of modified lines and correctness\. To this aspect, the only similar work is from Dai et al\.\[[14](https://arxiv.org/html/2609.29410#bib.bib9)\]that has also explored fixing bugs in competitive programming with minimal modifications by defining the consistency of a fix as the ratio between preserved lines and the number of lines in the fixed code\. We further expand this metric by also penalizing the lines that are deleted from the initial code\. Even though there may be cases where these lines are not actually used anywhere, deleting them is a deviation from what the user asked\.

## IIIMethod

### III\-ADataset Creation

Detecting if a LLM is generating a solution from scratch for a buggy submission instead of actually fixing the bug requires the following: \(1\) problem specification; \(2\) a suite of tests on which a generated solution can be verified that is correct; \(3\) a buggy submission; \(4\) a human\-authored corrected submission that fixes the errors in the buggy submission\. The similarity between the buggy submission and the correct one can be used as a thresholdγ\\gammathat identifies if the generated solution deviates too much from the buggy submission or makes fewer modifications than the human to fix the bug\.

For \(1\) and \(2\), we used Codeforces\-R1, a dataset with over 10k unique problems extracted from Codeforces along with public test cases and test cases generated with Deepseek\-R1\. The team behind Codeforces\-R1 offers a dataset for submissions \(over 12M\) as well, but the distributions of verdicts is highly unbalanced, 99% of the submissions being submissions that have Accepted as a verdict\. The rest of submissions are buggy submissions that are sparse and cannot be connected with the correct Accepted submissions, so this dataset is not suitable for \(3\) and \(4\)\.

To gather data for paired buggy \(3\) and human\-authored correct \(4\) submissions, we asked various Competitive programming users to provide all the submissions that they have on Codeforces\. All the users, started competing on Codeforces when they were students at BSc level \(aged 19\) and in some cases continuing to do so at MSc level or after\. The submissions span from 2015 to 2025\. In this way, for a given problemppand a useruu, we can extract all the submissions and order them in descending order by date\. We consider a solution incorrect if it does not pass all the test cases for that specific problem, or if it does not compile\. Ordering by date/time is very important in this case, because it offers the possibility to have different snapshots of a given solution from a user, based on submissions\. Let us assume that for a given problemPP, we havennsubmissions,sis\_\{i\}, from the same useruu, where only the last submissionsns\_\{n\}is correct, and the other submissions are wrong\. Starting froms1s\_\{1\}, one needs to add or delete a couple of lines to get tos2s\_\{2\}\. The process is repeated froms2s\_\{2\}tos3s\_\{3\}and so on tillsn−1s\_\{n\-1\}andsns\_\{n\}\. In theory,sn−1s\_\{n\-1\}submission should have the minimal number of lines that need to be added or deleted to reach an Accepted solution, whiles1s\_\{1\}should have the maximal number of lines that need to be added or deleted\. Having these historical submissions, we could estimate a threshold \(γ\\gamma\) that we can use to determine if a solution is generated from scratch by the LLM or it is actually doing a set of minimal modifications to reach a correct solution – thus being a fix\. From now on, we refer tosns\_\{n\}as the anchor solution andsi,i∈1,2\.\.,n−1s\_\{i\},i\\in\{1,2\.\.,n\-1\}, as being the buggy solutions that are related tosns\_\{n\}\.

In practice, the following scenarios usually happen: \(1\) An user could have more than one Accepted solution to a given problem \(e\.g\. trying a different approach\); \(2\) The buggy solutions could be very similar to each other in relation to a given anchor \(e\.g\. there is a small error which the user is not figuring out\); \(3\) The buggy solutions could be very different from the anchor solution\. The user has multiple buggy submissions, and then it figures out that the whole approach is wrong and reimplements everything from scratch\. To solve \(1\), we order in descending order and determine the anchor in a greedy manner\. Iterating over the submissions, whenever the verdict of a given submissionsis\_\{i\}is Accepted, we create a new anchoraia\_\{i\}and we attach to it the following submissionssi\+1,si\+2​…s\_\{i\+1\},s\_\{i\+2\}\.\.\.till we encounter another Accepted solution\. Point \(2\) is not a problem but a useful property\. These submissions could be considered as small augmentations of a buggy solution that check the consistency of LLMs’ output\. For case \(3\), we choose to retain only the buggy submissions that have a similarity of at least 60 % with the correct submission\. Anything less than 60% increases the chance of having a re\-implementation, where the anchor transforms from a bug fixing solution to a new solution altogether\. Comparing the LLM’s generated solution to an anchor like this would change the focus from comparing the code in terms of bug fixing modifications to modifications that reach an alternative solution\. We consider that a threshold of 60 % is high enough to avoid cases of re\-implementations in anchors but low enough to capture cases in which a LLM genuinely comes up with a bug fixing that is closer to the original code compared to the anchor\. \(e\.g\. the user adds or removes a lot of unnecessary code to fix the bug while a LLM modifies only a couple of lines to fix the bug\)\.

Figure[1](https://arxiv.org/html/2609.29410#S3.F1)depicts the process described in this section for a given problem, where multiple anchors and buggy submissions can be seen\.

![Refer to caption](https://arxiv.org/html/2609.29410v1/figures/CalLLMsFixSubmissions_Diagram_submissions.png)Fig\. 1:For a given useruuand problempp, we order the submissions in descending order\. There are two anchors that are related to 3, respectively 1 buggy submission\.#### III\-A1Dataset

Based on the extracted submissions, we created a dataset with the following features:

- - –submissions\_id\- The id of the submission\. - –problem\_id\- The id of the problem for the given submission\. - –creation\_time\_seconds\- Timestamp when the submission was created\. - –programming\_language\- The version of the compiler used to execute the submission\. - –verdict\- The verdict of the submission on Codeforces \(Accepted, Runtime Error, Memory Limit Exceeded, Time Limit Exceeded, Wrong Answer, Compilation Error, Challenged, Skipped and Rejected\)\. - –source\_code\- The source code of the submission\. - –accepted\_anchor\- The submission id to which the given submission is connected to or \-1 \(the submission is an anchor or it is a buggy submission which has no anchor\)\. - –is\_problem\_usable\- The problem id can be found or not in Codeforces\-R1 dataset\. - –passes\_r1\_tests\- The submission passes or not all the test cases\.

We acknowledge that Codeforces problems may becontaminatedas they are usually used for training LLMs, but we are less interested in the capacity of LLMs to solve novel problems\. Our main focus is on bug fixing with minimal modifications and the relationship between bug fixing and problem solving\.

Codeforces evaluates each submission with a variety of tests that capture all the edge cases, but the test cases themselves are partially available \(they are truncated to∼\\sim400 characters\)\. For each submission, we identified if the corresponding problem exists in Codeforces R1\-Dataset and we marked if it passes all the public test cases along with the generated tests by DeepSeek\-R1\.

To preprocess the dataset, we removed all the submissions that satisfied the following conditions:

- - –All the submissions that were from problems that cannot be found in the Codeforces R1\-Dataset\. - –Buggy submissions that passed all the tests and all the anchors that failed tests\. - –Buggy submissions that had different C / C\+\+ versions, used for compiling, from the anchor\. - –Submissions that had verdicts which were different from \(Accepted, Runtime Error, Memory Limit Exceeded, Time Limit Exceeded, Wrong Answer and Compilation Error\)\. - –Buggy submissions that had a similarity lower than 60% with the anchor\. - –All the buggy submissions that did not have an anchor after or before the preprocessing\.

After preprocessing the dataset, we remain with 567 <anchor, buggy submission\> pairs\. From the∼\\sim3000 submissions,∼\\sim600 submissions didn’t have an anchor \(no Accepted submissions for that specific problem for a given user\),∼\\sim700 submissions were for problems which could not be found in Codeforces\-R1 dataset and∼\\sim900 submissions were removed due to the similarity threshold\.

### III\-BMetrics

To be able to quantify how much an LLM is deviating from a buggy submission compared to a human fix, we need to know: \(1\) the similarity between the buggy solution and the anchor;\(2\) the similarity between the buggy solution and the generated solution; \(3\) whether the generated solution passes all the tests\.

#### III\-B1Similarity

Given two source codess1∈Ss\_\{1\}\\in Sands2∈Ss\_\{2\}\\in Swithnnnumber of lines, respectivelymmnumber of lines, andSSbeing the set of all possible source codes, we are interested in a functions​i​m:S×S→\[0,1\]sim:S\\times S\\rightarrow\[0,1\]that captures the similarity between them\.

In our case, we defines​i​m​\(s1,s2\)=s1∩s2s1∪s2sim\(s\_\{1\},s\_\{2\}\)=\\frac\{s\_\{1\}\\cap s\_\{2\}\}\{s\_\{1\}\\cup s\_\{2\}\}as the Jaccard Index betweens1s\_\{1\}ands2s\_\{2\}\. To compute the Jaccard Index, we use a Diff Tool to determine the number of deleted linesddand added linesaato get froms1s\_\{1\}tos2s\_\{2\}\. We can compute the common linesccasn−dn\-d\. In this ways​i​m​\(s1,s2\)=cd\+c\+asim\(s\_\{1\},s\_\{2\}\)=\\frac\{c\}\{d\+c\+a\}\. When we compute the similarity, we strip all the lines of white spaces and tabs, remove empty new lines and comments\.

Given a buggy source codesbs\_\{b\}and an anchorsas\_\{a\}, we define the baseline similarity ass​i​m​\(sb,sa\)sim\(s\_\{b\},s\_\{a\}\)\. For a buggy source code and a generated source code, we define the generated similarity ass​i​m​\(sb,sg\)sim\(s\_\{b\},s\_\{g\}\)\.

We define the Similarity Ratio metric ass​i​mr​a​t​i​o=s​i​m​\(sb,sg\)s​i​m​\(sb,sa\)sim\_\{ratio\}=\\frac\{sim\(s\_\{b\},s\_\{g\}\)\}\{sim\(s\_\{b\},s\_\{a\}\)\}\. The metric captures two scenarios: \(1\)s​i​mr​a​t​i​o<1sim\_\{ratio\}<1: The generated code by the LLM is not as consistent as the human fix \(2\)s​i​mr​a​t​i​o\>1sim\_\{ratio\}\>1: The generated code by the LLM is more consistent than the human fix\. This can happen in situations where a human fix adds more code than necessary\. Note that thes​i​mr​a​t​i​osim\_\{ratio\}can be greater than 1 if the baseline similarity is low\. When we preprocess the dataset we choose 0\.6 as threshold to capture cases in which the human fix may be inefficient in terms on modified lines but also to remove cases when a human fix reimplements the whole solution from scratch\. Compared to\[[14](https://arxiv.org/html/2609.29410#bib.bib9)\], which defines the consistency rate asr/kr/k, whererrindicates the number of lines preserved in the fixed solution andkkthe total number of lines in the fixed solution, we use a metric based on the Jaccard Index that also penalizes the number of deleted lines from the buggy solution\.

#### III\-B2Pass

To quantify if the LLMs fixes the bug or not, we are interested to see if the generated fix passes all the tests and thus we define the Test pass for a given submissionss, andnntest cases asT​P​\(s\)=∏i=1n1tk^=tkTP\(s\)=\\prod\_\{i=1\}^\{n\}1\_\{\\hat\{t\_\{k\}\}=t\_\{k\}\}\. We choose to use an indicator function instead of a ratio of passing tests because we are interested in seeing if the LLM correctly fixes the bug and solves the problem\.

## IVExperimental Setup

Using the preprocessed dataset, we evaluate 3 strong closed\-source LLMs from the OpenAI family \(gpt\-5\-nano, gpt\-5\-mini, gpt\-5\.1\) using three prompting methods: NaiveBugFix, GenerateScratch, GEPABugFix\. For all the methods, we make use of DSPy prompt optimization framework\[[15](https://arxiv.org/html/2609.29410#bib.bib10)\]to define the prompts\. For each LLM, we use amediumreasoning effort\. In DSPy, we use a Predictor Module instead of ChainOfThought \(COT\) module, because the models already have an internal reasoning\. We limit each LLM to 32k tokens\. For gpt\-5\.1 we also investigate how the model performs with no reasoning\. We split the initial preprocessed dataset by problem into 3 folds: \(1\) train fold \- 15 % problems \(2\) validation fold \- 15 % problems \(3\) test fold \- 70 % problems\. We chose to split the preprocessed dataset by problem instead of submission to avoid data leakage\. We use the train and validation folds for GEPA prompt optimization – as GEPA is very data efficient and requires a small number of samples, and we test all the models on the test fold\.

#### IV\-1NaiveBugFix

We create a simple prompt that asks the LLM to generate a fix for a buggy submission on a given problem \(by generating the entire code\), while keeping the number of deleted or added lines as low as possible\. The prompt can be seen in Figure[2](https://arxiv.org/html/2609.29410#S4.F2)\.

NaiveBugFixPrompt
Fix the bug in the buggy code for the given competitive programming problem by adding, deleting or modifying as few lines as possible\. In other words, you must adhere to the given buggy code and change it as little as possible to make it work\. You can add, delete or modify lines of code, but you cannot rewrite the whole solution\. The more lines you change, the more points you lose\. You should try to find the bug and fix it, not to rewrite the whole solution\.

Inputs:problem\_description,input\_format,output\_format,examples,note\(optional\),programmingLanguage,submission\_verdict,buggy\_code

Output:fixed\_code— The complete fixed code encapsulated in acppblock\.Fig\. 2:TheNaiveBugFixPrompt
#### IV\-2GenerateScratch

We create a simple prompt that asks the LLM to generate a solution for a given problem\. Note that in this case, there is no bug fixing, so thes​i​mr​a​t​i​osim\_\{ratio\}will be very low\. We use this method to identify a lower bound for thes​i​mr​a​t​i​osim\_\{ratio\}and to see if there is any performance gain in the pass metric\. The prompt can be seen in Figure[3](https://arxiv.org/html/2609.29410#S4.F3)\.

GenerateScratchPrompt
Generate a solution for the given competitive programming problem\.

Inputs:problem\_description,input\_format,output\_format,examples,note\(optional\),programmingLanguage

Output:generated\_code— The generated code\. Please encapsulate it in acppblock\.Fig\. 3:TheGenerateScratchPrompt
#### IV\-3GEPABugFix

Given the NaiveBugFix as a starting point, we employ GEPA as a prompt optimization technique\. We use as a prediction model gpt\-5\-nano and as a reflection model gpt\-5\-mini, with medium reasoning, a maximum 32k tokens and 2 full evaluations over the validation dataset\. To optimize the prompt, GEPA requires a metric function that returns values between\[0,1\]\[0,1\]\. To convert ours​i​mr​a​t​i​osim\_\{ratio\}metric into a function that has values between 0 and 1, we use the following reward function:r​e​w​a​r​d=p​a​s​s∗m​i​n​\(1,s​i​mr​a​t​i​o\)reward=pass\*min\(1,sim\_\{ratio\}\)\. Note thatp​a​s​s∈\{0,1\}pass\\in\\\{0,1\\\}\. Basically, if the generated solution does not fix the bug, then we return 0, since the similarity is not relevant in this case, and thus forcing GEPA to find a prompt that prioritizes bug fixing\. If the bug is fixed, then we return thes​i​mr​a​t​i​osim\_\{ratio\}\. Note that ifs​i​mr​a​t​i​o\>1sim\_\{ratio\}\>1, we trim it to 1 since we already have a solution that is better or equal with the baseline\. GEPA allows for textual feedback in the metric function\. The entire flow can be seen in Figure[4](https://arxiv.org/html/2609.29410#S4.F4)\.

![Refer to caption](https://arxiv.org/html/2609.29410v1/figures/gepa_flow.png)Fig\. 4:GEPA Flow

## VResults and Discussion

For each method and model, we compute thes​i​mr​a​t​i​osim\_\{ratio\}of each submission\. We group the submissions based on a baseline similarity range to observe how each method / model handles various degrees of similarity\. The results can be seen in Table[I](https://arxiv.org/html/2609.29410#S5.T1)\.

TABLE I:Results for gpt\-5\-nano, gpt\-5\-mini, gpt\-5\.1\-no\-reasoning, gpt\-5\.1 for the given methods\. The Sim Range column groups the submissions based on the baseline similarity range\.s​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}mean is computed based on each submission’ss​i​mr​a​t​i​osim\_\{ratio\}in the given range\. Problems Solved shows the ratio of problems solved\. The following acronyms were used: gpt\-5\-nano \(5n\), gpt\-5\-mini\(5m\), gpt5\.1\-no\-reasoning \(5\.1nr\), gpt5\.1 \(5\.1\)\. For each Similarity Range and each column \(s​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}, Problems Solved\), the bolded text shows the highest score and the underlined text the second highest\.#### V\-1RQ1: Given a buggy submission, how well does the LLM preserve the original user’s structure when fixing the bug?

For each method and model, when the model capacity is increased \(e\.g\. gpt\-5\-nano→\\rightarrowgpt\-5\-mini\), thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}increases as well\. In submissions with a lower similarity range, GEPABugFix and NaiveBugFix methods seem to find solutions that are even better in preserving the structure than the user fix, while this improvement seem to decrease if the similarity range increases\. The high gap between GEPA \(0\.93\) and NaiveBugFix \(0\.83\) ons​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}on the \[0\.9 \- 1\] range illustrates that when a simple prompt is used and the bug is very localized, the LLMs modify other unnecessary lines\. This suggests that when asked to solve a bug, LLMs tend to alter additional code which is not part of the bug, thus deviating from the original structure\. Reasoning does not seem to affect thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}in a substantial way, with the exception of the \[0\.7\-0\.8\) range for NaiveBugFix, which suggests that the LLM is capable of following the instruction to do minimal edits\. When generating a problem from scratch, thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}is greater than 0 because there is a high probability of having some libraries or the starting point of the program in common\.

#### V\-2RQ2: Are models better at generating code that solves a problem than at fixing bugs?

In the case of gpt\-nano, there seems to be∼\\sim0\.15 drop for bug fixing compared to generating from scratch, and for gpt\-mini as well in some similarity ranges\. For the other models, this drop happens in various similarity ranges as well, with the exception of gpt 5\.1 in the \[0\.9 \- 1\] range\. This suggests that LLMs are better at generating a solution from scratch than at bug fixing\. Besides this, we can observe that increasing the model capacity increases the score on a task level but doesn’t always reduce the gap between the tasks\. This can be observed on the \[0\.6 \- 0\.7\) and \[0\.8 \- 0\.9\) ranges\. Also, reasoning seems to have a big impact on the ratio of problems solved since gpt\-5\.1\-no\-reason has low scores\. The superiority of LLMs at generating solutions from scratch over fixing bugs may create a misleading expectation in users\. Since debugging intuitively feels like a more constrained and therefore easier task than solving a problem from scratch, users may incorrectly assume that LLMs should excel at it when the evidence suggests the opposite\.

#### V\-3RQ3: Is there a relationship between how much the AI preserves the user’s original code structure and its ability to correctly fix the bug?

In the case of GEPABugFix method, even though thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}is better than in the case of NaiveBugFix method, the ratio of problems solved is lower\. This suggests that if an LLM is forced to fix a bug with minimal modifications, it tends to focus more on the structure instead of actually solving the problem correctly\. This could happen because GEPA defines many rules to minimize the number of modified lines that may obscure the task of actually fixing the bug\. This forces the user to choose between receiving solutions that have localized patches, which are more likely to be wrong or a correct solution that diverges from the original structure and has to be understood\.

#### V\-4Discussion summary

\- From the results above, it can be seen that the LLMs are better at the task of generating a solution from scratch to a given problem \(especially lower capacity models which may be overfitted on the task\) than on the task of bug fixing with minimal modifications\. Having a complex prompt that forces the model to minimize the number of modified lines when fixing a bug increases thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}but decreases the ratio of solved problems while a more simpler prompt decreases thes​i​mr​a​t​i​o¯\\overline\{sim\_\{ratio\}\}and increases the number of problems solved\. These findings point to open challenges for AI\-assisted debugging tools\. One promising direction is to adapt how strictly the model is constrained based on how large the fix needs to be — enforcing minimal edits only when the required change is small\.

## VIReproducibility

The code, GEPA model and dataset can be found on[Github](https://github.com/Arkin1/CanLLMFixSubmissions)\. All the experiments were conducted using Docker to ensure reproducibility\. Each generated solution was tested against tests by using the solution suggested on the Codeforces\-R1 Huggingface page\. For the entire study, we have consumed∼\\sim20 million tokens with a price of∼\\sim80$\.

## VIIConclusion

In this paper, we have investigated how LLMs perform when they are required to fix bugs with minimal modifications\. We have found that LLMs are better at generating solutions from scratch than at bug fixing with minimal modifications especially when the number of modified lines in the human fix is very small\. Besides this, we have also found that defining a more complex prompt to reduce the number of modifications has a negative impact on the capacity of the model to correctly solve a problem\. This suggests that LLMs may not propose bug fixes that are consistent with the initial code and perform other modifications that the user does not expect\. Since LLMs perform better at generating solutions from scratch, they may be tempted to rewrite rather than minimally fix code\. This is problematic in any domain where the original structure carries value — such as education, where it supports learning; collaborative development, where it aids code review; or legacy systems, where large rewrites introduce risk and break established conventions\.

As future work we would like to extend the study to LLMs from other vendors as well to understand if the effects replicate\. We would also like to extend the dataset to contain more users and track how the gap between scratch generation and bug fixing evolves as newer, more recent submissions are introduced\. We also want to analyze how the effects manifest based on problem difficulty and judge verdicts\. Besides this, we are also interested to evaluate the model performance on cases in which they are allowed to have more than 1 call per submission and to further explore the impact of prompt complexity on the relation between preserving the structure and fixing the buggy submission\.

## References

- \[1\]L\. et al\.\(2023\)AlphaCode 2 technical report\.Technical reportGoogle DeepMind\.External Links:[Link](https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p2.1)\.
- \[2\]C\. S\. Xia and L\. Zhang\(2024\)Automated program repair via conversation: fixing 162 out of 337 bugs for $ 0\.42 each using chatgpt\.InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,ISSTA 2024,pp\. 819–831\.External Links:[Link](https://doi.org/10.1145/3650212.3680323),[Document](https://dx.doi.org/10.1145/3650212.3680323)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p2.1)\.
- \[3\]Codeforces\(2026\)Codeforces – competitive programming platform\.Note:Accessed: 2024\-02\-14External Links:[Link](https://codeforces.com/)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p4.1)\.
- \[4\]L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab\(2025\)GEPA: reflective prompt evolution can outperform reinforcement learning\.External Links:2507\.19457,[Link](https://arxiv.org/abs/2507.19457)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p4.1)\.
- \[5\]G\. Penedo, A\. Lozhkov, H\. Kydlíček, L\. B\. Allal, E\. Beeching, A\. P\. Lajarín, Q\. Gallouédec, N\. Habib, L\. Tunstall, and L\. von Werra\(2025\)CodeForces\.Hugging Face\.Note:[https://huggingface\.co/datasets/open\-r1/codeforces](https://huggingface.co/datasets/open-r1/codeforces)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p4.1)\.
- \[6\]D\. Guo, D\. Yang, and e\. al\. Zhang\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§I](https://arxiv.org/html/2609.29410#S1.p4.1)\.
- \[7\]Z\. Zheng, Z\. Cheng, Z\. Shen, S\. Zhou, K\. Liu, H\. He, D\. Li, S\. Wei, H\. Hao, J\. Yao, P\. Sheng, Z\. Wang, W\. Chai, A\. Korolova, P\. Henderson, S\. Arora, P\. Viswanath, J\. Shang, and S\. Xie\(2025\)LiveCodeBench pro: how do olympiad medalists judge llms in competitive programming?\.External Links:2506\.11928,[Link](https://arxiv.org/abs/2506.11928)Cited by:[§II\-A](https://arxiv.org/html/2609.29410#S2.SS1.p1.1)\.
- \[8\]X\. Yang, Z\. Liu, C\. Huang, J\. Zhang, T\. Zhang, Y\. Zhang, and W\. Lei\(2025\)ELABORATION: a comprehensive benchmark on human\-LLM competitive programming\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 59–104\.External Links:[Link](https://aclanthology.org/2025.acl-long.4/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.4),ISBN 979\-8\-89176\-251\-0Cited by:[§II\-A](https://arxiv.org/html/2609.29410#S2.SS1.p1.1)\.
- \[9\]R\. Tian, Y\. Ye, Y\. Qin, X\. Cong, Y\. Lin, Y\. Pan, Y\. Wu, H\. Haotian, L\. Weichuan, Z\. Liu,et al\.\(2024\)Debugbench: evaluating debugging capability of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 4173–4198\.Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1)\.
- \[10\]A\. Mavalankar, H\. Mansoor, Z\. Marinho, M\. Samsikova, and T\. Schaul\(2025\)AuPair: golden example pairs for code repair\.arXiv preprint arXiv:2502\.18487\.Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1)\.
- \[11\]M\. M\. A\. Haque, W\. U\. Ahmad, I\. Lourentzou, and C\. Brown\(2023\)FixEval: execution\-based evaluation of program fixes for programming problems\.External Links:2206\.07796,[Link](https://arxiv.org/abs/2206.07796)Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1)\.
- \[12\]W\. Ahmad, S\. Chakraborty, B\. Ray, and K\. Chang\(2021\)Unified pre\-training for program understanding and generation\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 2655–2668\.External Links:[Link](https://www.aclweb.org/anthology/2021.naacl-main.211)Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1)\.
- \[13\]Y\. Wang, W\. Wang, S\. Joty, and S\. C\.H\. Hoi\(2021\)CodeT5: identifier\-aware unified pre\-trained encoder\-decoder models for code understanding and generation\.InEMNLP,Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1)\.
- \[14\]Z\. Dai, B\. Chen, Z\. Zhao, X\. Tang, S\. Wu, C\. Yao, Z\. Gao, and J\. Chen\(2025\)Less is more: adaptive program repair with bug localization and preference learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 128–136\.Cited by:[§II\-B](https://arxiv.org/html/2609.29410#S2.SS2.p1.1),[§III\-B1](https://arxiv.org/html/2609.29410#S3.SS2.SSS1.p4.1)\.
- \[15\]O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts\(2024\)DSPy: compiling declarative language model calls into self\-improving pipelines\.Cited by:[§IV](https://arxiv.org/html/2609.29410#S4.p1.1)\.