Function-Level Execution Feedback for Code Preference Optimization

# Function-Level Execution Feedback for Code Preference Optimization
Source: [https://arxiv.org/html/2608.23632](https://arxiv.org/html/2608.23632)
Idris NechnechSehwan KimAffiliation:Department of Electrical and Computer Engineering, Seoul National UniversityJimin SeoAffiliation:Department of Electrical and Computer Engineering, Seoul National UniversityYeongoon KimAffiliation:Department of Electrical and Computer Engineering, Seoul National UniversityMinhae OhAffiliation:Department of Electrical and Computer Engineering, Seoul National UniversitySangwoo HongAffiliation:Department of Computer Science and Engineering, Konkuk University\{inechnech,junglee\}@snu\.ac\.krJungwoo Lee

###### Abstract

Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought\. In code generation, however, process supervision remains underexplored because there is no standard notion of a step\. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize\. We proposeStep\-KTOder, a framework for code preference optimization that defines steps as module\-level functions in decomposed multi\-function programs and assigns binary correctness labels via automatically generated unit tests\. Our method provides a code\-specific instantiation of stepwise KTO, combining function\-level process supervision with outcome\-level feedback on the full program\. We evaluate on HumanEval\(\+\), MBPP\(\+\), BigCodeBench, and LiveCodeBench, showing thatStep\-KTOderimproves over outcome\-only KTO and DPO\. Further analysis shows that execution\-based labels are essential: LLM\-as\-a\-judge annotations systematically over\-predict function failures, corrupt positive step labels, and degrade downstream preference optimization\. Code is available at:[https://github\.com/inechnech/STEP\-KTODER](https://github.com/inechnech/STEP-KTODER)\.

$\\dagger$$\\dagger$footnotetext:Corresponding author\.## 1Introduction

Code generation has become a major application of Large Language Models \(LLMs\), with recent code\-specialized models achieving strong performance on a wide range of programming tasks\([8](https://arxiv.org/html/2608.23632#bib.bib9);[11](https://arxiv.org/html/2608.23632#bib.bib10)\)\. These models are typically improved through two stages: supervised fine\-tuning \(SFT\)\([30](https://arxiv.org/html/2608.23632#bib.bib31)\)on high\-quality code data, followed by an alignment stage that adapts the model toward better outputs\([24](https://arxiv.org/html/2608.23632#bib.bib30)\)\. Among alignment methods, Direct Preference Optimization \(DPO\)\([26](https://arxiv.org/html/2608.23632#bib.bib8)\)is widely used to learn from paired preference data, whereas Kahneman–Tversky Optimization \(KTO\)\([7](https://arxiv.org/html/2608.23632#bib.bib4)\)learns from binary desirability labels on individual outputs\.

However, applying these techniques to code generation is nontrivial\. These methods are typically instantiated at the level of the entire program, treating a full solution as a single unit of feedback\. This provides only a coarse training signal: a program can be mostly correct and fail only on a narrow edge case, but outcome\-level supervision still marks the entire output as undesirable\. Such supervision does not reveal which component is correct and which is responsible for the failure, motivating finer\-grained supervision for code generation\.

In mathematical reasoning,*process supervision*—providing feedback on intermediate reasoning steps—has proven highly effective\([17](https://arxiv.org/html/2608.23632#bib.bib1);[21](https://arxiv.org/html/2608.23632#bib.bib2);[29](https://arxiv.org/html/2608.23632#bib.bib11)\), with process reward models \(PRMs\) underpinning state\-of\-the\-art reasoning systems\. Building on this success,[18](https://arxiv.org/html/2608.23632#bib.bib3)combined stepwise binary feedback with KTO to jointly optimize intermediate and final\-answer quality for mathematical problem solving\. Transferring this idea to code generation, however, remains an open challenge: code does not naturally decompose into a single standardized sequence of intermediate logical steps, making the very definition of a “step” ambiguous\.

![Refer to caption](https://arxiv.org/html/2608.23632v1/figure1.png)Figure 1:Overview of theStep\-KTOderframework\. A reference solutiony⋆y^\{\\star\}is decomposed into module\-level functionss1,…,sns\_\{1\},\\ldots,s\_\{n\}, and function\-level unit tests are generated for each function\. Candidate completions are evaluated against these tests to obtain step labelsziz\_\{i\}indicating local correctness, alongside an outcome labeloofrom the dataset\-provided test suite\. The resulting labeled data is used to train the model with a joint objective that combines outcome\-level KTO \(ℒout\\mathcal\{L\}\_\{\\mathrm\{out\}\}\) and function\-level supervision \(ℒstep\\mathcal\{L\}\_\{\\mathrm\{step\}\}\), reinforcing locally correct functions while penalizing locally incorrect ones\.We address this ambiguity by defining steps as module\-level functions in decomposed programs, and proposeStep\-KTOder, a framework that brings function\-level process supervision to code generation\. Our key idea is that decomposed programs provide an executable step structure: when a solution is written as multiple functions, each can be tested independently using automatically generated unit tests, yielding binary labels of local correctness\.Step\-KTOderuses these labels to extend outcome\-level KTO with function\-level supervision, guiding training by both global program success and local correctness, as indicated in Figure[1](https://arxiv.org/html/2608.23632#S1.F1)\.

Beyond the main empirical gains, we investigate two design choices that are important toStep\-KTOder’s effectiveness\. First, programs that pass dataset\-provided tests but contain a locally incorrect function provide a form of supervision that outcome\-only training cannot express\. We find that preserving these local/global mismatches strengthens the benefit of step\-level supervision, whereas masking such mismatches substantially weakens the signal\. Second, we assess whether automatically generated unit tests are necessary for reliable step supervision by replacing execution\-based step labels with LLM\-as\-a\-judge annotations\. Although these annotations show moderate agreement with execution\-based labels, they systematically over\-predict step failures, and when used forStep\-KTOdertraining, they degrade performance\. Even for already instruction\-tuned Qwen2\.5\-Coder models,Step\-KTOderachieves substantial improvements: it improves over the base model by up to \+26\.7% on BigCodeBench Hard\([35](https://arxiv.org/html/2608.23632#bib.bib23)\)and \+27\.0% on LiveCodeBench\([12](https://arxiv.org/html/2608.23632#bib.bib24)\)\.

Our contributions are as follows:

- •We proposeStep\-KTOder, a code\-specific instantiation of stepwise KTO that combines function\-level process supervision with outcome\-level feedback\.
- •We introduce an automatic data construction pipeline that decomposes solutions into functions, generates function\-level unit tests, and derives execution\-based step labels for training\.
- •Experiments across widely used benchmarks show thatStep\-KTOderimproves post\-trained code models\. We further demonstrate that replacing execution\-based step labels with LLM\-as\-a\-judge annotations weakens the step\-level signal and degrades performance\.

## 2Related Work

### 2\.1Process Supervision for Reasoning

Process supervision has been highly effective for mathematical reasoning, where step\-level feedback improves over outcome\-only supervision\([5](https://arxiv.org/html/2608.23632#bib.bib29);[28](https://arxiv.org/html/2608.23632#bib.bib12);[17](https://arxiv.org/html/2608.23632#bib.bib1)\)\. Subsequent work has scaled this supervision:[29](https://arxiv.org/html/2608.23632#bib.bib11)introduced automatic step\-level annotation via Monte Carlo estimation, and[21](https://arxiv.org/html/2608.23632#bib.bib2)proposed a divide\-and\-conquer Monte Carlo Tree Search framework for large\-scale process supervision data\.[18](https://arxiv.org/html/2608.23632#bib.bib3)introduced Step\-KTO, which combines KTO with PRM\-labeled stepwise feedback to jointly optimize intermediate reasoning quality and final response correctness\. Our work builds on this idea but reformulates it for code: instead of using PRM\-labeled reasoning steps, we define steps as concrete, independently testable functions and supervise them directly through execution\-based unit test feedback\. This perspective is consistent with recent work on code PRMs showing that execution feedback improves the reliability of process supervision for code\([15](https://arxiv.org/html/2608.23632#bib.bib7)\)\. In contrast to PRM training, we study how execution\-based process labels can be incorporated directly into offline preference optimization\.

### 2\.2Preference Optimization for Code

Preference optimization has become a standard post\-training technique for code models\. DPO\([26](https://arxiv.org/html/2608.23632#bib.bib8)\)learns from paired preferences, while KTO\([7](https://arxiv.org/html/2608.23632#bib.bib4)\)removes the requirement for paired data using binary desirability labels grounded in prospect theory\. Recent work has explored localized and fine\-grained feedback for code:[34](https://arxiv.org/html/2608.23632#bib.bib5)concentrated the DPO loss on error\-prone token spans, while[31](https://arxiv.org/html/2608.23632#bib.bib33)introduced Target\-DPO, a focal preference alignment framework that localizes preference updates to targeted code regions\.[6](https://arxiv.org/html/2608.23632#bib.bib6)trained a PRM that provides dense line\-level feedback during code generation via reinforcement learning\.[33](https://arxiv.org/html/2608.23632#bib.bib13)introduced CodeDPO, which constructs self\-generated preference pairs to align code models for both correctness and efficiency\.

These approaches target different granularities—tokens, code blocks, lines, or whole programs—but to our knowledge, none of these approaches defines steps as semantically meaningful, independently testable functions and integrates them into a binary preference optimization objective\.

### 2\.3Solution Decomposition and Automated Testing

Fill\-in\-the\-middle objectives\([2](https://arxiv.org/html/2608.23632#bib.bib14)\)train models to complete code given surrounding context, encouraging modular structure\.[27](https://arxiv.org/html/2608.23632#bib.bib32)extend this idea to alignment, splitting code into AST\-based blocks to construct more diverse preference pairs for DPO\. Prompting\-based approaches have also emphasized modular decomposition for code generation:[25](https://arxiv.org/html/2608.23632#bib.bib27)break complex programming problems into smaller reasoning modules through hierarchical prompting\.[19](https://arxiv.org/html/2608.23632#bib.bib15)proposed generating both code and unit tests simultaneously, then using self\-validation to enhance generation quality\. Automated unit test generation has matured significantly\([3](https://arxiv.org/html/2608.23632#bib.bib17);[22](https://arxiv.org/html/2608.23632#bib.bib16)\); our pipeline leverages this capability in a function\-level setting, generating tests for individual functions and using execution outcomes as binary step labels for training\.

### 2\.4Execution\-Grounded Inference\-Time Methods

Recent execution\-grounded methods instead improve code at inference time\. S∗combines parallel sampling with iterative debugging and selects among candidates using adaptively synthesized inputs and their execution results\([14](https://arxiv.org/html/2608.23632#bib.bib34)\), while ORPS explores a tree of reasoning and code trajectories guided by execution outcomes and self\-critique\([32](https://arxiv.org/html/2608.23632#bib.bib35)\)\. Unlike these per\-query search and selection methods,Step\-KTOderuses function\-level execution during offline data construction to produce labels that update the model policy\.

## 3Method

### 3\.1Problem Formulation

Given a promptxxand a reference solutiony⋆y^\{\\star\}, we first rewritey⋆y^\{\\star\}into a decomposed multi\-function program

y~=\(s1,s2,…,sn\),\\tilde\{y\}=\(s\_\{1\},s\_\{2\},\\dots,s\_\{n\}\),using a strong code language model, where eachsis\_\{i\}denotes one function in the decomposed program andnnis the total number of such functions\. We treat each functionsis\_\{i\}as a*step*\.

Training samples are then generated from this decomposition: the target model fills the function skeleton defined byy~\\tilde\{y\}to produce a candidate solutionyy\. Each training sample receives two forms of binary supervision:

- •Anoutcome labelo∈\{0,1\}o\\in\\\{0,1\\\}, indicating whetheryypasses the dataset\-provided test suite forxx\.
- •A sequence ofstep labels𝐳=\(z1,…,zn\)\\mathbf\{z\}=\(z\_\{1\},\\dots,z\_\{n\}\), where eachzi∈\{0,1,∅\}z\_\{i\}\\in\\\{0,1,\\varnothing\\\}indicates whether the implementation ofsis\_\{i\}inyypasses its function\-level unit tests\. A label of∅\\varnothingindicates that no valid unit test is available for that step\.

This formulation distinguishes*local correctness*from*global correctness*, providing finer\-grained supervision than outcome\-only evaluation\. Rather than treating partially correct programs as uniformly desirable or undesirable, it enables the objective to reinforce correct functions while penalizing those responsible for failure\.

### 3\.2KTO Background

We build on Kahneman–Tversky Optimization \(KTO\)\([7](https://arxiv.org/html/2608.23632#bib.bib4)\), which aligns a policyπθ\\pi\_\{\\theta\}from binary feedback using a Kahneman–Tversky\-inspired value function over the log\-ratio between the policy and a frozen referenceπref\\pi\_\{\\mathrm\{ref\}\}:

rθ​\(x,y\)=log⁡πθ​\(y∣x\)πref​\(y∣x\)\.r\_\{\\theta\}\(x,y\)=\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\.
The outcome\-level reference point is

z0out=KL⁡\(πθ​\(y′∣x\)∥πref​\(y′∣x\)\),z\_\{0\}^\{\\mathrm\{out\}\}=\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\theta\}\(y^\{\\prime\}\\mid x\)\\;\\middle\\\|\\;\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\\mid x\)\\right\),wherey′y^\{\\prime\}denotes an output sequence used to estimate the divergence between the policy and reference\. Given a binary desirability labelo∈\{0,1\}o\\in\\\{0,1\\\}, the outcome\-level value function is

vout=\{λD​σ​\(βout​\(rθ−z0out\)\),o=1,λU​σ​\(βout​\(z0out−rθ\)\),o=0,v\_\{\\mathrm\{out\}\}=\\begin\{cases\}\\lambda\_\{D\}\\,\\sigma\\\!\\bigl\(\\beta\_\{\\mathrm\{out\}\}\(r\_\{\\theta\}\-z\_\{0\}^\{\\mathrm\{out\}\}\)\\bigr\),&o=1,\\\\\[2\.0pt\] \\lambda\_\{U\}\\,\\sigma\\\!\\bigl\(\\beta\_\{\\mathrm\{out\}\}\(z\_\{0\}^\{\\mathrm\{out\}\}\-r\_\{\\theta\}\)\\bigr\),&o=0,\\end\{cases\}whererθ≡rθ​\(x,y\)r\_\{\\theta\}\\equiv r\_\{\\theta\}\(x,y\)andσ⁡\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid function\.βout\>0\\beta\_\{\\mathrm\{out\}\}\>0controls the sensitivity of the outcome\-level value function, andλD\\lambda\_\{D\}andλU\\lambda\_\{U\}weight desirable and undesirable samples, respectively\. The outcome\-level KTO loss is then

ℒout​\(πθ,πref\)=𝔼\(x,y,o\)∼𝒟​\[λo−vout\],\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(\\pi\_\{\\theta\},\\pi\_\{\\mathrm\{ref\}\}\)=\\mathbb\{E\}\_\{\(x,y,o\)\\sim\\mathcal\{D\}\}\\bigl\[\\lambda\_\{o\}\-v\_\{\\mathrm\{out\}\}\\bigr\],whereλo=λD\\lambda\_\{o\}=\\lambda\_\{D\}ifo=1o=1andλo=λU\\lambda\_\{o\}=\\lambda\_\{U\}ifo=0o=0\.

### 3\.3Step\-KTO Objective

Following[18](https://arxiv.org/html/2608.23632#bib.bib3), we instantiate stepwise KTO over the function\-level steps defined in Section[3\.1](https://arxiv.org/html/2608.23632#S3.SS1)and introduce reliability masking for functions without validated local tests\. For each stepsis\_\{i\}, we compute a step\-local log\-ratio over the tokens of that function:

ri=∑t∈silog⁡πθ​\(yt∣x,y<t\)πref​\(yt∣x,y<t\)\.r\_\{i\}=\\sum\_\{t\\in s\_\{i\}\}\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\.Each step has a labelziz\_\{i\}as defined above and a maskmi∈\{0,1\}m\_\{i\}\\in\\\{0,1\\\}; only steps withmi=1m\_\{i\}=1andzi∈\{0,1\}z\_\{i\}\\in\\\{0,1\\\}contribute to the step\-level loss\.

Similarly, we define a step\-level reference point

z0step=KL⁡\(πθ​\(yi′∣x,s<i\)∥πref​\(yi′∣x,s<i\)\),z\_\{0\}^\{\\mathrm\{step\}\}=\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\theta\}\(y^\{\\prime\}\_\{i\}\\mid x,s\_\{<i\}\)\\;\\middle\\\|\\;\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\_\{i\}\\mid x,s\_\{<i\}\)\\right\),whereyi′y^\{\\prime\}\_\{i\}denotes the subsequence of generated tokens belonging to theii\-th function\. The step\-level value function is

vi=\{λD,step​σ​\(βstep​\(ri−z0step\)\),zi=1,λU,step​σ​\(βstep​\(z0step−ri\)\),zi=0\.v\_\{i\}=\\begin\{cases\}\\lambda\_\{D,\\mathrm\{step\}\}\\,\\sigma\\\!\\bigl\(\\beta\_\{\\mathrm\{step\}\}\(r\_\{i\}\-z\_\{0\}^\{\\mathrm\{step\}\}\)\\bigr\),&z\_\{i\}=1,\\\\\[2\.0pt\] \\lambda\_\{U,\\mathrm\{step\}\}\\,\\sigma\\\!\\bigl\(\\beta\_\{\\mathrm\{step\}\}\(z\_\{0\}^\{\\mathrm\{step\}\}\-r\_\{i\}\)\\bigr\),&z\_\{i\}=0\.\\end\{cases\}Here,βstep\\beta\_\{\\mathrm\{step\}\},λD,step\\lambda\_\{D,\\mathrm\{step\}\}, andλU,step\\lambda\_\{U,\\mathrm\{step\}\}are the step\-level counterparts of the outcome\-level KTO parameters\.

Letℳτ=\{i:mi=1,zi∈\{0,1\}\}\\mathcal\{M\}\_\{\\tau\}=\\\{i:m\_\{i\}=1,\\ z\_\{i\}\\in\\\{0,1\\\}\\\}denote the supervised steps for sampleτ\\tau, and letM=\|ℳτ\|M=\|\\mathcal\{M\}\_\{\\tau\}\|\. Letλzi,step\\lambda\_\{z\_\{i\},\\mathrm\{step\}\}denote the step\-level target coefficient, withλzi,step=λD,step\\lambda\_\{z\_\{i\},\\mathrm\{step\}\}=\\lambda\_\{D,\\mathrm\{step\}\}whenzi=1z\_\{i\}=1andλzi,step=λU,step\\lambda\_\{z\_\{i\},\\mathrm\{step\}\}=\\lambda\_\{U,\\mathrm\{step\}\}whenzi=0z\_\{i\}=0\. The masked step\-level loss is

ℒstep=𝔼τ∼𝒟​\[1n​∑i∈ℳτ\(λzi,step−vi\)\],\\mathcal\{L\}\_\{\\mathrm\{step\}\}=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{D\}\}\\left\[\\frac\{1\}\{n\}\\sum\_\{i\\in\\mathcal\{M\}\_\{\\tau\}\}\\left\(\\lambda\_\{z\_\{i\},\\mathrm\{step\}\}\-v\_\{i\}\\right\)\\right\],where the sum is zero whenM=0M=0, so samples without supervised steps fall back to outcome\-only KTO\.

Combining with the outcome\-level loss yields the finalStep\-KTOderobjective:

ℒStep\-KTOder=ℒout\+λstep​ℒstep\.\\mathcal\{L\}\_\{\\text\{\{Step\-KTOder\}\}\}=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\+\\lambda\_\{\\mathrm\{step\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{step\}\}\.By jointly optimizing outcome\-level and function\-level feedback,Step\-KTOderreinforces locally correct functions and penalizes locally incorrect ones, providing a more localized learning signal than outcome\-level supervision alone\. Figure[2](https://arxiv.org/html/2608.23632#S3.F2)illustrates this difference on a failing candidate: outcome\-only KTO treats the full program as undesirable, whereasStep\-KTOderassigns negative step\-level signal only to the faulty function\.

Figure 2:Training signal comparison for a failing candidate\. The wrong candidate differs from the decomposed reference solution only in the highlighted line, yielding locally correct functionsz1,z2,z3=1z\_\{1\},z\_\{2\},z\_\{3\}=1, a faulty final functionz4=0z\_\{4\}=0, and outcome labelo=0o=0\. Outcome\-only KTO penalizes all function spans, whereasStep\-KTOderpreserves the locally correct functions and penalizes only the faulty one\.
### 3\.4Data Construction Pipeline

Our pipeline consists of four stages, illustrated in Figure[1](https://arxiv.org/html/2608.23632#S1.F1)\.

#### \(1\) Reference decomposition\.

We use a strong instruction\-tuned code model to rewrite each ground\-truth solution into an interface\-preserving multi\-function program, retaining only rewrites that pass the original test suite\.

#### \(2\) Function\-level unit test generation\.

For each decomposed function, a strong code model generates several targeted unit tests for its interface and edge cases\. We retain only tests that pass on the reference implementation, invoke the target function, and contain nontrivial assertions whose pass condition depends on the function’s output\. Functions without valid tests receive a null step label \(zi=∅z\_\{i\}=\\varnothing\) and are masked from the step loss\. The prompts are provided in Appendix[O](https://arxiv.org/html/2608.23632#A15)\.

#### \(3\) On\-policy candidate generation\.

From each decomposition, we construct a*skeleton*by replacing function bodies withpasswhile preserving signatures, docstrings, imports, and module\-level context\. Then, for each skeleton we samplek=8k=8candidate completions from the model being post\-trained, using temperatureT=0\.4T=0\.4and top\-ppsampling withp=0\.95p=0\.95; this keeps the preference data close to the model’s own generation distribution\.

#### \(4\) Labeling and dataset construction\.

Each candidate is evaluated against the dataset\-provided test suite to obtain an outcome labeloo\. Step labelsziz\_\{i\}are obtained by executing each candidate function against its generated unit tests\. By default, we use*local*step labels: each function is tested after being inserted into the decomposed reference program, with all other functions kept fixed to their reference implementations\. This isolates function\-level correctness from downstream composition effects\. Across datasets, 83\.2% of testable decomposed functions retain at least one validated test\. We refer to this quantity as*validated\-test availability*; the corresponding per\-problem distribution is shown in Figure[3](https://arxiv.org/html/2608.23632#S3.F3)\. For each task, we retain up to one passing and one failing candidate, preferring candidates with richer step\-level supervision\. Rows with fewer than two supervised steps fall back to outcome\-only KTO, avoiding single\-step supervision that largely duplicates the outcome\-level signal\.

Figure 3:Distribution of per\-problem validated\-test availability\. Most problems achieve near\-full test availability\.

### 3\.5Conflict\-Preserving Step Labels

A key design question is how to handle examples where outcome\-level and step\-level supervision disagree\. Two conflict patterns arise:passing programs with negative step labels, where the full program passes dataset tests but at least one supervised function fails its function\-level tests; andfailing programs with all\-positive step labels, where the full program fails dataset tests even though all supervised functions pass their unit tests\. These examples reflect a mismatch between local and end\-to\-end correctness\. Rather than removing them, we compare masked and conflict\-preserving variants to test whether such cases provide useful training signal beyond outcome\-only supervision and ordinary partial\-correctness patterns\.

## 4Experimental Setup

### 4\.1Training Configuration

All target models are instruction\-tuned code models that have already undergone substantial post\-training\. We fine\-tune each model for 1 epoch using LoRA\([10](https://arxiv.org/html/2608.23632#bib.bib18)\)with global batch size 16\. KTO andStep\-KTOderuse a learning rate of1×10−61\\times 10^\{\-6\}, while DPO uses5×10−75\\times 10^\{\-7\}\. We setλstep=1\.0\\lambda\_\{\\mathrm\{step\}\}=1\.0by default\. Full training details are provided in Appendix[B](https://arxiv.org/html/2608.23632#A2)\.

### 4\.2Datasets

Training data is drawn from the training splits of TACO\([16](https://arxiv.org/html/2608.23632#bib.bib19)\)and APPS\([9](https://arxiv.org/html/2608.23632#bib.bib20)\)\. We construct training samples using the pipeline described in Section[3](https://arxiv.org/html/2608.23632#S3): reference solutions are decomposed with Qwen2\.5\-Coder\-32B\-Instruct\([11](https://arxiv.org/html/2608.23632#bib.bib10)\), and each target model generates on\-policy candidate completions from the resulting function skeletons\. For each problem, we retain up to one passing and one failing candidate using a ranking procedure that prioritizes positives whose supervised functions all pass their unit tests and negatives with both passing and failing function\-level labels\. Such negatives localize which components remain correct and which fail \(Appendix[A](https://arxiv.org/html/2608.23632#A1)\)\.

Table[1](https://arxiv.org/html/2608.23632#S4.T1)summarizes the resulting training sets\. For same\-family training with Qwen2\.5\-Coder\-1\.5B\-Instruct and Qwen2\.5\-Coder\-3B\-Instruct, we include decomposed reference solutions as additional positive anchors\. For cross\-family training with DeepSeek\-Coder\-6\.7B\-Instruct, we use only on\-policy generated candidates, since adding off\-policy samples degrades performance \(Appendix[M](https://arxiv.org/html/2608.23632#A13)\)\. Rows with fewer than two supervised steps fall back to outcome\-only KTO\. As a result, 82% of Qwen2\.5\-Coder\-1\.5B\-Instruct rows, 83% of Qwen2\.5\-Coder\-3B\-Instruct rows, and 78% of DeepSeek\-Coder\-6\.7B\-Instruct rows carry active step\-level supervision\. We justify this threshold in Section[6\.1](https://arxiv.org/html/2608.23632#S6.SS1)\.

### 4\.3Evaluation Benchmarks and Baselines

Table 1:Training dataset statistics\. Step\-supervised rows contain at least two supervised function\-level labels and activate the step\-level loss; fallback KTO rows use outcome\-only KTO\. Avg\. steps / active row is computed over step\-supervised rows\. AST alignment measures exact agreement with the decomposition skeleton\.We evaluate on seven code generation benchmarks: HumanEval\([4](https://arxiv.org/html/2608.23632#bib.bib21)\)and HumanEval\+\([20](https://arxiv.org/html/2608.23632#bib.bib26)\), MBPP\([1](https://arxiv.org/html/2608.23632#bib.bib22)\)and MBPP\+\([20](https://arxiv.org/html/2608.23632#bib.bib26)\), BigCodeBench Full and Hard\([35](https://arxiv.org/html/2608.23632#bib.bib23)\), and LiveCodeBench\([12](https://arxiv.org/html/2608.23632#bib.bib24)\)releasev4\_v5\. All evaluations use greedy decoding with vLLM\([13](https://arxiv.org/html/2608.23632#bib.bib25)\)\.

We compareStep\-KTOderagainst four baselines: the original instruction\-tuned model, DPO\([26](https://arxiv.org/html/2608.23632#bib.bib8)\), KTO\([7](https://arxiv.org/html/2608.23632#bib.bib4)\), and Target\-DPO\([31](https://arxiv.org/html/2608.23632#bib.bib33)\)\. Baseline construction details are provided in Appendix[D](https://arxiv.org/html/2608.23632#A4)\.

## 5Results

### 5\.1Main Results

Table 2:Pass rate across model families and scales\. Relative improvements are computed over outcome\-only KTO within each model; gains are largest on harder benchmarks\. The best results are highlighted in bold\.The experimental results are reported in Table[2](https://arxiv.org/html/2608.23632#S5.T2)\. Since all target models are already instruction\-tuned code models, easier benchmarks such as HumanEval and MBPP leave limited headroom, and the clearest gains appear on harder benchmarks\. On Qwen2\.5\-Coder\-1\.5B\-Instruct,Step\-KTOderimproves over KTO by \+11\.5% on BigCodeBench Hard and \+7\.7% on LiveCodeBench\. On Qwen2\.5\-Coder\-3B\-Instruct, the gains are similarly concentrated on the hardest benchmarks, with \+11\.3% on BigCodeBench Hard and \+9\.5% on LiveCodeBench, while matching or slightly improving KTO elsewhere\. DeepSeek\-Coder\-6\.7B\-Instruct shows the same qualitative pattern: BigCodeBench Hard improves monotonically from DPO to KTO toStep\-KTOder, withStep\-KTOderachieving a \+4\.9% gain over KTO, and LiveCodeBench improving by \+3\.2%\. On the easier benchmarks, where headroom is limited,Step\-KTOderremains competitive, obtaining the best or tied\-best result in most cases, with Target\-DPO leading slightly on HumanEval\+ for DeepSeek\-Coder\-6\.7B\-Instruct\. The gains persist across seeds:Step\-KTOderexceeds KTO on both BigCodeBench Hard and LiveCodeBench across three seeds \(Appendix[E](https://arxiv.org/html/2608.23632#A5)\)\.

Both Target\-DPO andStep\-KTOdertarget localized code failures, but they optimize different supervision signals: Target\-DPO derives preference pairs from debugging traces and localizes loss to changed token regions, whereasStep\-KTOderlabels functions directly via unit tests\.Step\-KTOderis consistently stronger on the harder benchmarks\. For Qwen2\.5\-Coder\-1\.5B\-Instruct, Target\-DPO is competitive on easier benchmarks and ties on BigCodeBench Hard, butStep\-KTOderis stronger on LiveCodeBench\. For Qwen2\.5\-Coder\-3B\-Instruct and DeepSeek\-Coder\-6\.7B\-Instruct,Step\-KTOderimproves over Target\-DPO on both BigCodeBench Hard \(\+18\.5% for both\) and LiveCodeBench \(\+13\.4% and \+17\.0%, respectively\)\. On easier benchmarks,Step\-KTOderties or improves over Target\-DPO in most cases\. Notably,Step\-KTOderuses at most 11,440 training samples, compared with Target\-DPO’s 59,000 preference pairs\.

### 5\.2Ablations

Table[3](https://arxiv.org/html/2608.23632#S6.T3)reports three core ablations on Qwen2\.5\-Coder\-3B\-Instruct\.First, conflict preservation is essential: masking conflicts reducesStep\-KTOderto KTO\-level performance on BigCodeBench Hard and decreases on LiveCodeBench as well\. This shows that apparent disagreements between outcome\-level and function\-level labels can provide useful training signal rather than noise to be removed\.Second,λstep=1\.0\\lambda\_\{\\mathrm\{step\}\}=1\.0gives the best trade\-off \(λstep=0\\lambda\_\{\\mathrm\{step\}\}=0corresponds to outcome\-only KTO\), reported above\. Lower step weights underuse the function\-level signal, while larger weights begin to regress easier benchmarks without improving the hardest ones\.Third, requiring at least two supervised steps outperforms the looser threshold of one, consistent with the redundancy of single\-step supervision discussed in Section[6\.1](https://arxiv.org/html/2608.23632#S6.SS1)\.

## 6Analysis

Table 3:Core ablations on Qwen2\.5\-Coder\-3B\-Instruct\. We vary conflict handling, the step\-loss weightλstep\\lambda\_\{\\mathrm\{step\}\}, and the minimum number of supervised steps required to activate the step loss\.### 6\.1Why Function\-Level Supervision Helps

Both DPO and KTO operate at the program level, weighting the policy–reference log\-ratio by a single binary outcome label\. They therefore cannot distinguish which parts of a generated program contributed to success or failure: when correct and incorrect functions coexist, outcome\-level methods reinforce or penalize all components together\.Step\-KTOdertargets this missing information by assigning labels to individual functions, allowing the model to reinforce locally correct functions and penalize locally incorrect ones within the same program\. This explains why conflict preservation matters\. Whenzi=oz\_\{i\}=ofor all supervised steps, the step loss is directionally aligned with the outcome loss and mainly changes where the gradient is applied\. In contrast, whenzi≠oz\_\{i\}\\neq o, the step loss provides a direction that outcome\-level supervision cannot produce, such as penalizing a specific function even when the full program passes\. Such conflicts can arise when the decomposed function interface exposes edge cases not exercised by the original test suite\. Masking these conflicts removes this directional disagreement, making the remaining step signal largely redundant with outcome\-level supervision\. The same reasoning explains the minimum\-supervised\-steps threshold\. For single\-function rows, the step span covers the full answer, sor1=routr\_\{1\}=r\_\{\\mathrm\{out\}\}\. Whenz1=oz\_\{1\}=o, the step loss adds no information beyond a rescaled outcome loss\. Requiring at least two supervised steps filters out these redundant rows\.

### 6\.2Function\-Level Labels Localize Failures

To test whether negative step labels identify functions that contribute to end\-to\-end failure, we perform a repair intervention on failing candidates whose supervised functions include both positive and negative labels\. We replace the locally negative functions with their decomposed reference implementations and evaluate the repaired candidates using the original dataset\-provided test suite\. As controls, we replace either the same number of locally positive functions or the same number of randomly selected observed functions\.

As shown in Table[4](https://arxiv.org/html/2608.23632#S6.T4), replacing locally negative functions repairs 74\.9% of failing candidates, substantially outperforming both random and locally positive replacement\. This suggests that execution\-based step labels localize repair\-relevant faults, rather than merely correlating with outcome correctness\.

Table 4:Repair intervention on failing candidates\.
### 6\.3Execution\-Based Labels Are Essential

A natural question is whether the execution\-based unit tests at the core ofStep\-KTOderare necessary, or whether an LLM\-as\-a\-judge can provide comparable step\-level supervision\. We test this by replacing execution\-based step labels with judgments from GPT\-5\.4 mini\([23](https://arxiv.org/html/2608.23632#bib.bib28)\), and we retrain Qwen2\.5\-Coder\-3B\-Instruct with identical settings\. The LLM shows moderate overall agreement with execution\-based labels \(73\.2%73\.2\\%\), but this aggregate score hides a strong asymmetry\. It agrees with execution on94\.1%94\.1\\%of failing functions, but on only52\.2%52\.2\\%of passing functions, systematically over\-predicting failure\. This bias corrupts the positive step labels needed forStep\-KTOder: training with LLM\-as\-a\-judge labels instead of execution\-based labels degrades performance, especially on BigCodeBench Hard and LiveCodeBench\. Thus, execution\-based labels are essential not only because they provide step\-level supervision: they also provide reliable step\-level supervision\. Full agreement statistics and benchmark results across all seven benchmarks are provided in Appendix[N](https://arxiv.org/html/2608.23632#A14)\.

## 7Conclusion

We proposeStep\-KTOder, a framework for code preference optimization that defines process\-supervision steps as module\-level functions in decomposed programs\. By combining outcome\-level KTO with execution\-based function labels,Step\-KTOderprovides localized feedback that reinforces correct functions while penalizing incorrect ones\. Experiments on post\-trained code models show consistent gains over outcome\-only KTO and DPO, with ablations confirming that preserving local/global label conflicts and using execution\-based labels are central to these gains\. These results suggest that function\-level execution feedback offers a practical path toward process supervision for code generation\.

## 8Limitations

#### Model and data scope\.

Step\-KTOderis most natural when solutions admit a meaningful decomposition into independently testable functions, and is most reliable when candidate solutions are generated on\-policy by the target model, which adds engineering cost for each new model\. Our experiments focus on instruction\-tuned code models that have already undergone substantial post\-training; applying function\-level supervision earlier, during SFT or on larger base\-model training runs, remains a promising direction\.

#### Difficulty distribution\.

Our filtering pipeline requires problems to have a clear function\-style structure or to admit a meaningful decomposition\. As a result, the training data is biased toward problems whose solutions can be decomposed into testable functions, while harder competition problems are less represented\. Extending the pipeline to more complex I/O formats and harder problem regimes is a natural next step\.

#### Unit\-test quality\.

Step labels depend on automatically generated unit tests, which we validate by execution against decomposed reference solutions but do not formally verify\. A mutation\-sensitivity audit \(Appendix[H](https://arxiv.org/html/2608.23632#A8)\) shows that the retained unit tests reject 92\.6% of semantically perturbed reference implementations, and our repair intervention \(Section[6\.2](https://arxiv.org/html/2608.23632#S6.SS2)\) shows that negative step labels identify functions responsible for end\-to\-end failure\. Our LLM\-as\-a\-judge comparison further shows that execution\-based labels are substantially more reliable than LLM judgments\. Still, stronger automated test\-generation or verification methods could further improve label quality\.

## Ethical Considerations

The models used in this paper, Qwen2\.5\-Coder\([11](https://arxiv.org/html/2608.23632#bib.bib10)\)and DeepSeek\-Coder\([8](https://arxiv.org/html/2608.23632#bib.bib9)\), are licensed for academic research purposes\. The training datasets, TACO\([16](https://arxiv.org/html/2608.23632#bib.bib19)\)and APPS\([9](https://arxiv.org/html/2608.23632#bib.bib20)\), and all evaluation benchmarks\([4](https://arxiv.org/html/2608.23632#bib.bib21);[20](https://arxiv.org/html/2608.23632#bib.bib26);[1](https://arxiv.org/html/2608.23632#bib.bib22);[35](https://arxiv.org/html/2608.23632#bib.bib23);[12](https://arxiv.org/html/2608.23632#bib.bib24)\)are publicly available and distributed for research use\.

## Acknowledgments

This work was supported in part by the National Research Foundation of Korea \(NRF\) grant funded by the Ministry of Science and ICT \(MSIT\) \(RS\-2024\-00451435, 20%; RS\-2024\-00413957, 20%\), the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the MSIT \(RS\-2025\-02305453, 15%; RS\-2025\-02273157, 15%; RS\-2025\-25442149, 15%; RS\-2021\-II211343, 15%\), the Institute of New Media and Communications \(INMAC\), the BK21 FOUR program funded by the Ministry of Education, the Artificial Intelligence Graduate School Program \(Seoul National University\), and the Research Program for Future ICT Pioneers at Seoul National University in 2026\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Bavarianet al\.\(2022\)M\. Bavarian, H\. Jun, N\. Tezak, J\. Schulman, C\. McLeavey, J\. Tworek, and M\. ChenEfficient training of language models to fill in the middle\.External Links:2207\.14255,[Link](https://arxiv.org/abs/2207.14255)Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- Chenet al\.\(2023\)B\. Chen, F\. Zhang, A\. Nguyen, D\. Zan, Z\. Lin, J\. Lou, and W\. ChenCodeT: code generation with generated tests\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ktrw68Cmu9c)Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Daiet al\.\(2025\)N\. Dai, Z\. Wu, R\. Zheng, Z\. Wei, W\. Shi, X\. Jin, G\. Liu, C\. Dun, L\. Huang, and L\. YanProcess supervision\-guided policy optimization for code generation\.External Links:2410\.17621,[Link](https://arxiv.org/abs/2410.17621)Cited by:[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1)\.
- Ethayarajhet al\.\(2024\)K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. KielaModel alignment as prospect theoretic optimization\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=iUwHnoENnl)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.23632#S3.SS2.p1.1),[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p2.1)\.
- Guoet al\.\(2024\)D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang, G\. Chen, X\. Bi, Y\. Wu, Y\. K\. Li, F\. Luo, Y\. Xiong, and W\. LiangDeepSeek\-coder: when the large language model meets programming – the rise of code intelligence\.External Links:2401\.14196,[Link](https://arxiv.org/abs/2401.14196)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. SteinhardtMeasuring coding challenge competence with APPS\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=sD93GOzH3i5)Cited by:[§4\.2](https://arxiv.org/html/2608.23632#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Appendix B](https://arxiv.org/html/2608.23632#A2.p1.1),[§4\.1](https://arxiv.org/html/2608.23632#S4.SS1.p1.1)\.
- Huiet al\.\(2024\)B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu, K\. Dang, Y\. Fan, Y\. Zhang, A\. Yang, R\. Men, F\. Huang, B\. Zheng, Y\. Miao, S\. Quan, Y\. Feng, X\. Ren, X\. Ren, J\. Zhou, and J\. LinQwen2\.5\-coder technical report\.External Links:2409\.12186,[Link](https://arxiv.org/abs/2409.12186)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p1.1),[§4\.2](https://arxiv.org/html/2608.23632#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=chfJJYC3iL)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p5.1),[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1)\.
- Liet al\.\(2025a\)D\. Li, S\. Cao, C\. Cao, X\. Li, S\. Tan, K\. Keutzer, J\. Xing, J\. E\. Gonzalez, and I\. StoicaS\*: test time scaling for code generation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15964–15978\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.865/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.865),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.4](https://arxiv.org/html/2608.23632#S2.SS4.p1.1)\.
- Liet al\.\(2025b\)Q\. Li, X\. Dai, X\. Li, W\. Zhang, Y\. Wang, R\. Tang, and Y\. YuCodePRM: execution feedback\-enhanced process reward model for code generation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 8169–8182\.External Links:[Link](https://aclanthology.org/2025.findings-acl.428/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.428)Cited by:[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Liet al\.\(2023\)R\. Li, J\. Fu, B\. Zhang, T\. Huang, Z\. Sun, C\. Lyu, G\. Liu, Z\. Jin, and G\. LiTACO: topics in algorithmic code generation dataset\.External Links:2312\.14852,[Link](https://arxiv.org/abs/2312.14852)Cited by:[§4\.2](https://arxiv.org/html/2608.23632#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Linet al\.\(2025a\)Y\. Lin, D\. Jin, T\. Xu, T\. Wu, S\. Sukhbaatar, C\. Zhu, Y\. He, Y\. Chen, J\. E\. Weston, Y\. Tian, A\. Rahnama, S\. Wang, H\. Ma, and H\. FangStep\-KTO: optimizing mathematical reasoning through stepwise binary feedback\.InProceedings of The 3rd Workshop on Mathematical Natural Language Processing \(MathNLP 2025\),M\. Valentino, D\. Ferreira, M\. Thayaparan, L\. Ranaldi, and A\. Freitas \(Eds\.\),Suzhou, China,pp\. 15–33\.External Links:[Link](https://aclanthology.org/2025.mathnlp-main.2/),[Document](https://dx.doi.org/10.18653/v1/2025.mathnlp-main.2)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2608.23632#S3.SS3.p1.1)\.
- Linet al\.\(2025b\)Z\. Lin, S\. Shen, J\. Shang, J\. E\. Weston, and Y\. NieLearning to solve and verify: a self\-play framework for mutually improving code and test generation\.InNeurIPS 2025 Fourth Workshop on Deep Learning for Code,External Links:[Link](https://openreview.net/forum?id=j6tMZaPWWF)Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=1qvx610Cu7)Cited by:[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.
- Luoet al\.\(2024\)L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng, J\. Sun, and A\. RastogiImprove mathematical reasoning in language models by automated process supervision\.External Links:2406\.06592,[Link](https://arxiv.org/abs/2406.06592)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Maet al\.\(2025\)Z\. Ma, X\. Zhang, J\. Zhang, J\. Yu, S\. Luo, and J\. TangDynamic scaling of unit tests for code reward modeling\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 6917–6935\.External Links:[Link](https://aclanthology.org/2025.acl-long.343/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.343),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- OpenAI \(2026\)OpenAIIntroducing GPT\-5\.4 mini and nano\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4\-mini\-and\-nano/](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)Accessed: 2026\-04\-28Cited by:[§6\.3](https://arxiv.org/html/2608.23632#S6.SS3.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Gray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=TG8KACxEON)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p1.1)\.
- Pan and Zhang \(2025\)R\. Pan and H\. ZhangModularization is better: effective code generation with modular prompting\.External Links:2503\.12483,[Link](https://arxiv.org/abs/2503.12483)Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=HPuSIXJaa9)Cited by:[Appendix D](https://arxiv.org/html/2608.23632#A4.p1.1),[§1](https://arxiv.org/html/2608.23632#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p2.1)\.
- Renet al\.\(2025\)H\. Ren, Z\. Lu, W\. Shi, H\. Hou, Y\. Yang, K\. Wang, A\. Zhou, J\. Pan, M\. Zhan, and H\. LiAlignment with fill\-in\-the\-middle for enhancing code generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8304–8320\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.419/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.419),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.3](https://arxiv.org/html/2608.23632#S2.SS3.p1.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\- and outcome\-based feedback\.External Links:2211\.14275,[Link](https://arxiv.org/abs/2211.14275)Cited by:[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. SuiMath\-shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9426–9439\.External Links:[Link](https://aclanthology.org/2024.acl-long.510/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.510)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.23632#S2.SS1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, M\. Bosma, V\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. LeFinetuned language models are zero\-shot learners\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gEZrGCozdqR)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p1.1)\.
- Wuet al\.\(2025\)J\. Wu, H\. Li, X\. Zhang, X\. Liu, Y\. Huang, J\. Luo, Y\. Zhang, Z\. Li, R\. Chu, Y\. Yang, and S\. LiTeaching your models to understand code via focal preference alignment\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 14003–14023\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.707/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.707),ISBN 979\-8\-89176\-332\-6Cited by:[Appendix D](https://arxiv.org/html/2608.23632#A4.p3.1),[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p2.1)\.
- Yuet al\.\(2025\)Z\. Yu, W\. Gu, Y\. Wang, X\. Jiang, Z\. Zeng, J\. Wang, W\. Ye, and S\. ZhangReasoning through execution: unifying process and outcome rewards for code generation\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=pLQtovjXiw)Cited by:[§2\.4](https://arxiv.org/html/2608.23632#S2.SS4.p1.1)\.
- Zhanget al\.\(2025a\)K\. Zhang, G\. Li, Y\. Dong, J\. Xu, J\. Zhang, J\. Su, Y\. Liu, and Z\. JinCodeDPO: aligning code models with self generated and verified source code\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 15854–15871\.External Links:[Link](https://aclanthology.org/2025.acl-long.771/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.771),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1)\.
- Zhanget al\.\(2025b\)K\. Zhang, G\. Li, J\. Li, Y\. Dong, J\. Li, and Z\. JinFocused\-DPO: enhancing code generation through focused preference optimization on error\-prone points\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 9578–9591\.External Links:[Link](https://aclanthology.org/2025.findings-acl.498/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.498)Cited by:[§2\.2](https://arxiv.org/html/2608.23632#S2.SS2.p1.1)\.
- Zhuoet al\.\(2025\)T\. Y\. Zhuo, V\. M\. Chien, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari, I\. N\. B\. Yusuf, H\. Zhan, J\. He, I\. Paul, S\. Brunner, C\. Gong, J\. Hoang, A\. R\. Zebaze, X\. Hong, W\. Li, J\. Kaddour, M\. Xu, Z\. Zhang, P\. Yadav, N\. Jain, A\. Gu, Z\. Cheng, J\. Liu, Q\. Liu, Z\. Wang, D\. Lo, B\. Hui, N\. Muennighoff, D\. Fried, X\. Du, H\. de Vries, and L\. V\. WerraBigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YrycTjllL0)Cited by:[§1](https://arxiv.org/html/2608.23632#S1.p5.1),[§4\.3](https://arxiv.org/html/2608.23632#S4.SS3.p1.1),[Ethical Considerations](https://arxiv.org/html/2608.23632#Sx1.p1.1)\.

## Appendix APreference Pair Selection

For each problem we generate 8 candidate completions from the target model and select up to one positive and one negative candidate for the training set, based on the outcome labelooand the step labels\.

#### Positive candidates\.

Among candidates witho=1o=1, we select the one with the cleanest step\-level signal: we prioritize candidates whose supervised functions all pass their unit tests, followed by candidates with passing labels and some unknown steps, then candidates containing both passing and failing step labels, and finally candidates without active step supervision\. Within each category, ties are broken by preferring candidates with more locally passing functions, fewer locally failing functions, valid parsing, and non\-empty outputs\.

#### Negative candidates\.

Among candidates witho=0o=0, we select the one with the richest local supervision\. We first prioritize candidates that contain both passing and failing supervised functions, since they identify which components remain correct and which are responsible for the failure\. If none are available, we fall back to candidates with at least one locally failing function\. As a secondary fallback, we use*generated\-context labels*, obtained by executing each generated function within the full generated module rather than inserting it into the decomposed reference program\. We prioritize candidates with both passing and failing generated\-context labels, then candidates with at least one generated\-context failure, and finally candidates without informative step signal\.

#### Reference\-context and generated\-context agreement\.

We compare our default local labels, computed in the reference context, with generated\-context labels on selected training rows\. They agree on89\.4%89\.4\\%of43,45843\{,\}458comparable supervised steps, consistently across models \(89\.289\.2–89\.6%89\.6\\%\), suggesting that local labels closely align with generated\-context behavior while isolating function\-level correctness\.

#### Pairing\.

Tasks with only a positive candidate contribute an unpaired positive row, and tasks with only a negative candidate contribute an unpaired negative row; both are usable by KTO but not DPO\. Tasks with both contribute one positive and one negative row\.

## Appendix BTraining Details

We fine\-tune all target models with LoRA\([10](https://arxiv.org/html/2608.23632#bib.bib18)\)using rankr=32r=32, scaling factorα=64\\alpha=64, dropout0\.050\.05, maximum sequence length 2048, and global batch size 16\. We use AdamW with\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\)and a cosine learning\-rate schedule with 50 warmup steps and minimum ratio0\.10\.1\. We use learning rate1×10−61\\times 10^\{\-6\}for KTO andStep\-KTOder, and5×10−75\\times 10^\{\-7\}for DPO\. For DPO and KTO, we setβ=0\.1\\beta=0\.1\. ForStep\-KTOder, we useβout=βstep=0\.1\\beta\_\{\\mathrm\{out\}\}=\\beta\_\{\\mathrm\{step\}\}=0\.1, setλstep=1\.0\\lambda\_\{\\mathrm\{step\}\}=1\.0, and use unit weights for all KTO value\-function coefficients:λD=λU=λD,step=λU,step=1\.0\\lambda\_\{D\}=\\lambda\_\{U\}=\\lambda\_\{D,\\mathrm\{step\}\}=\\lambda\_\{U,\\mathrm\{step\}\}=1\.0\. Sensitivity to these coefficients is analyzed in Appendix[K](https://arxiv.org/html/2608.23632#A11)\.

All experiments were run on 2 NVIDIA RTX A5000 GPUs with 24 GB of memory\.

## Appendix CData Construction Cost

For Qwen2\.5\-Coder\-3B\-Instruct, data construction required approximately 78 GPU\-hours: 24 GPU\-hours for reference decomposition, 43 for function\-level unit\-test generation, and 11 for samplingk=8k=8candidate completions\. Decompositions, skeletons, and tests are generated once and reused across target models; only candidate sampling and CPU\-based execution and labeling are model\-specific\.

Table 5:Mean pass rate±\\pmstandard deviation across three training seeds\. Bold denotes the best mean within each benchmark\.Table 6:Pass rates for the general\-purpose Qwen2\.5\-7B\-Instruct model\.
## Appendix DDPO and Baseline Construction Details

DPO\([26](https://arxiv.org/html/2608.23632#bib.bib8)\)trains from paired preferences\. Given a promptxx, a preferred solutiony\+y^\{\+\}, and a dispreferred solutiony−y^\{\-\}, its loss is

ℒDPO=−𝔼⁡\[log⁡σ⁡\(β⁡\(rθ​\(x,y\+\)−rθ​\(x,y−\)\)\)\],\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}=\-\\mathbb\{E\}\\left\[\\log\\sigma\\left\(\\beta\\left\(r\_\{\\theta\}\(x,y^\{\+\}\)\-r\_\{\\theta\}\(x,y^\{\-\}\)\\right\)\\right\)\\right\],where

rθ​\(x,y\)=log⁡πθ​\(y∣x\)πref​\(y∣x\)\.r\_\{\\theta\}\(x,y\)=\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\.Here,πθ\\pi\_\{\\theta\}is the optimized policy,πref\\pi\_\{\\mathrm\{ref\}\}is the frozen reference policy,σ⁡\(⋅\)\\sigma\(\\cdot\)is the sigmoid function, andβ\\betacontrols the KL regularization strength\.

In our experiments, DPO pairs are constructed using the same candidate pool as KTO andStep\-KTOder: a passing candidate is used asy\+y^\{\+\}and a failing candidate asy−y^\{\-\}\. Unlike DPO, KTO andStep\-KTOderoperate on individually labeled candidates, allowing them to use unpaired positives or negatives\. KTO samples are created using the same candidate rows and outcome labels asStep\-KTOder, but we discard all function\-level step labels, reducing training to outcome\-only preference optimization\.

We train Target\-DPO\([31](https://arxiv.org/html/2608.23632#bib.bib33)\)on its dataset of 59,000 preference pairs\. For comparability, we match our DPO LoRA setup:r=32r=32,α=64\\alpha=64, dropout0\.050\.05, global batch size 16, and learning rate5×10−75\\times 10^\{\-7\}\.

## Appendix EMulti\-Seed Robustness

To assess seed sensitivity, we train Qwen2\.5\-Coder\-3B\-Instruct with both KTO andStep\-KTOderacross three random seeds, holding all other settings fixed\. Table[5](https://arxiv.org/html/2608.23632#A3.T5)reports the resulting mean pass rate and standard deviation\.

Step\-KTOderexceeds seed\-matched KTO in all three runs on both BigCodeBench Hard and LiveCodeBench\. The corresponding mean gains are 0\.015 and 0\.014, respectively; on the other five benchmarks, the difference between the method means is at most 0\.002\.

## Appendix FGeneral\-Purpose Model

To test whether this pattern persists for a general\-purpose instruction\-tuned model, we additionally train Qwen2\.5\-7B\-Instruct using on\-policy candidates together with the validated ground\-truth anchors used in our Qwen experiments\. Qwen2\.5\-7B\-Instruct starts from a stronger base than our other target models on MBPP, BigCodeBench, and LiveCodeBench\.Step\-KTOderexceeds KTO on six of seven benchmarks, with the largest gains on BigCodeBench Hard and LiveCodeBench; see Table[6](https://arxiv.org/html/2608.23632#A3.T6)for full results\.

## Appendix GDataset Statistics

This section reports difficulty distributions across the three stages of our data construction pipeline—the raw TACO and APPS training splits, the problems retained after ground\-truth decomposition validation, and the final per\-source training rows used by each target model—together with quality statistics for the function\-level unit tests generated in Stage 2\.

#### Raw datasets\.

Table[7](https://arxiv.org/html/2608.23632#A7.T7)reports the difficulty composition of the raw TACO and APPS training splits as released\. Both datasets contain difficulty annotations covering a wide range from introductory exercises to competition\-level problems\. Our pipeline retains only problems with an explicit function\-style entry point—identified by a non\-emptyfn\_namefield in the dataset’s annotations—and discards problems with other interfaces, which typically contain monolithic algorithmic solutions resistant to function\-level decomposition\. As a result, higher\-difficulty buckets contribute few or no problems to our training set: in TACO, onlyeasy,medium, andmedium\_hardproblems contribute, whilehard,very\_hard, andunknown\_difficultyproblems are filtered out entirely\. In APPS, onlyintroductoryandinterviewproblems remain;competitionproblems are excluded\.

Table 7:Difficulty distribution of the raw TACO and APPS training splits before any filtering\.
#### Ground\-truth decomposition validation\.

Table[8](https://arxiv.org/html/2608.23632#A7.T8)reports per\-difficulty pass rates after Stage 1, where ground\-truth solutions are decomposed with Qwen2\.5\-Coder\-32B\-Instruct and validated by execution against the original test suite\. In the table,*Total*denotes the number of problems entering Stage 1 with a clear function\-style entry point, and*Validated*denotes the number whose decomposed solution still passes the original tests\. Only validated problems proceed to candidate generation\. Validation is much more permissive on APPS \(88\.9%88\.9\\%\) than on TACO \(43\.9%43\.9\\%\)\. The TACO drop is concentrated in the harder buckets:easyvalidates at52\.1%52\.1\\%,medium\_hardat37\.1%37\.1\\%, andmediumat only18\.8%18\.8\\%, reflecting the difficulty of producing semantically equivalent decompositions for problems with complex global state\.

SourceDifficultyTotalValidatedPass rateTACOeasy4,1122,14152\.1%medium1,20022518\.8%medium\_hard52019337\.1%Total5,8322,55943\.9%APPSintroductory2,3252,06989\.0%interview44239088\.2%Total2,7672,45988\.9%Table 8:Ground\-truth decomposition validation rates by difficulty\.Table 9:Difficulty distribution of the final training sets for each target model\. Qwen2\.5\-Coder\-1\.5B\-Instruct and Qwen2\.5\-Coder\-3B\-Instruct use the same validated decomposed reference solutions as positive anchors, but differ in their on\-policy generated candidates\. DeepSeek\-Coder\-6\.7B\-Instruct uses only on\-policy generated candidates\.
#### Final training sets\.

Table[9](https://arxiv.org/html/2608.23632#A7.T9)reports the difficulty distribution of the rows actually used to train each target model\. For Qwen2\.5\-Coder\-3B\-Instruct \(same\-family\), we retain all 5,018 validated ground\-truth decompositions as additional positive anchors and add 6,422 generated rows produced by the target model itself\. For DeepSeek\-Coder\-6\.7B\-Instruct \(cross\-family\), ground\-truth rows are discarded \(Appendix[M](https://arxiv.org/html/2608.23632#A13)\) and only on\-policy generated candidates are retained\.

The final training sets are skewed toward easier problems, reflecting both the natural distribution of decomposition\-friendly problems and the per\-difficulty validation rates from Table[8](https://arxiv.org/html/2608.23632#A7.T8)\. We discuss the implications in the*Difficulty distribution*paragraph of Section[8](https://arxiv.org/html/2608.23632#S8)\.

## Appendix HUnit Test Generation Quality

#### Generation and validation protocol\.

Function\-level unit tests are produced by Qwen2\.5\-Coder\-32B\-Instruct \(Section[3\.4](https://arxiv.org/html/2608.23632#S3.SS4), Stage 2\) and validated by execution against the decomposed reference implementation\. The generator outputs structured JSON test cases with named inputs, which are converted into aunittestclass that calls the target function on each input and compares the result to the reference output\. Tests are validated by execution: only test classes that parse, execute, and pass against the ground\-truth code are retained\. Functions for which no valid test remains receive a null step label \(zi=∅z\_\{i\}=\\varnothing\) and fall back to outcome\-only KTO supervision during training\. Table[10](https://arxiv.org/html/2608.23632#A8.T10)summarizes unit\-test generation quality, and Figure[3](https://arxiv.org/html/2608.23632#S3.F3)shows the full validated\-test\-availability distribution\.

Table 10:Function\-level unit test generation quality\. Validated\-test availability is over testable functions\.Table 11:Effect of rebalancing outcome\- and step\-levelλU\\lambda\_\{U\}on Qwen2\.5\-Coder\-3B\-Instruct\.Table 12:Step\-only objective ablation on Qwen2\.5\-Coder\-3B\-Instruct\.As shown in Table[10](https://arxiv.org/html/2608.23632#A8.T10), first\-pass parsing succeeds in over98%98\\%of calls, and covered functions have around5\.55\.5valid tests on average, providing multiple independent assertions per supervised step\. Overall,83\.2%83\.2\\%of testable decomposed functions retain at least one validated test, while mean per\-problem validated\-test availability is84\.4%84\.4\\%; the latter is visualized in Figure[3](https://arxiv.org/html/2608.23632#S3.F3)\.

#### Mutation\-sensitivity audit\.

As an additional sanity check, we evaluate whether the generated function\-level tests detect controlled perturbations rather than merely executing successfully on the reference implementation\. We audit all 11,705 functions from validated decomposed programs that retained at least one valid generated unit test across TACO and APPS, and apply syntax\-preserving mutations to the reference function, including comparison flips, Boolean\-operator flips, arithmetic\-operator changes, constant perturbations, and default\-return replacements\. Across 27,958 valid mutants, the generated tests reject 25,880 mutants, yielding a mutation kill rate of 92\.6%\. This suggests that the retained tests reliably detect local behavioral changes rather than only validating executability on the reference solution\.

## Appendix IRepresentative Function\-Level Label Cases

#### Incomplete local test\.

For a Pair of Shoes task \(TACO 11007\), the candidate checks whether the concatenated left and right\-shoe sizes are unique, rather than comparing the two size multisets\. Because the retained test forpair\_of\_shoescovers only the empty input, both local labels are positive although the full program is incorrect\. This illustrates whyStep\-KTOderretains the outcome\-level term\.

#### Useful local/global conflict\.

In a Task Scheduler problem \(APPS 171\), the candidate passes the dataset\-provided test suite, but its decomposed stepcalculate\_min\_intervalsomitslen\(tasks\)from the maximum, understating the required number of intervals on inputs where the task count exceeds the frequency\-based bound\. Its negative local label exposes a hidden defect that outcome\-only KTO cannot represent\.

## Appendix JRepair Intervention Details

Beyond the aggregate repair rates reported in Section[6\.2](https://arxiv.org/html/2608.23632#S6.SS2), we also examine how the intervention behaves across failure types\. The analysis is run on originally failing Qwen2\.5\-Coder\-3B\-Instruct selected training candidates whose supervised functions include both positive and negative step labels\. For each candidate, we replace local\-negative functions with their decomposed reference implementations and rerun the dataset\-provided tests\. As controls, we replace either the same number of local\-positive functions or the same number of randomly selected observed functions\.

The repair effect is strongest for wrong\-answer failures: replacing local\-negative functions repairs 78\.6% of such candidates on the combined APPS and TACO subset\. This suggests that local\-negative labels are especially effective at identifying semantic errors, rather than merely capturing parsing or execution artifacts\.

## Appendix KSensitivity to KTO Value\-Function Weights

The default configuration uses unit weights for all KTO value\-function coefficients \(λD=λU=λD,step=λU,step=1\.0\\lambda\_\{D\}=\\lambda\_\{U\}=\\lambda\_\{D,\\mathrm\{step\}\}=\\lambda\_\{U,\\mathrm\{step\}\}=1\.0\)\. The KTO authors recommend rebalancingλU\\lambda\_\{U\}when the positive/negative sample ratio is skewed; our Qwen2\.5\-Coder\-3B\-Instruct training set has a step\-level positive/negative ratio of approximately2\.5:12\.5\{:\}1, motivating a check on whether upweighting undesirable samples improves performance\. When one coefficient is varied, all other value\-function coefficients remain fixed at1\.01\.0\. Table[11](https://arxiv.org/html/2608.23632#A8.T11)reports two rebalancing runs against the default\.

Both deviations from uniform weights regress BigCodeBench Hard\. The outcome\-levelλU=2\.5\\lambda\_\{U\}=2\.5run additionally collapses LiveCodeBench performance, suggesting that outcome\-level rebalancing is unstable in this setting\. One possible explanation is that the observed positive/negative ratio reflects the natural pass distribution of a strong instruction\-tuned 3B model rather than simple dataset imbalance; reweighting failures too aggressively may over\-allocate gradient budget toward avoiding failures at the cost of reinforcing successful behavior, hurting generalization to harder benchmarks\.

## Appendix LAdditional Objective Ablations

Our default objective combines outcome\-level and function\-level supervision:

ℒStep\-KTOder=ℒout\+λstep​ℒstep\.\\mathcal\{L\}\_\{\\text\{\{Step\-KTOder\}\}\}=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\+\\lambda\_\{\\mathrm\{step\}\}\\mathcal\{L\}\_\{\\mathrm\{step\}\}\.To better understand the role of the outcome\-level term, we evaluate a step\-only variant on Qwen2\.5\-Coder\-3B\-Instruct\. This variant removes the outcome\-level KTO loss and optimizes only the stepwise loss, allowing us to assess how much signal is provided by local execution feedback alone\.

As shown in Table[12](https://arxiv.org/html/2608.23632#A8.T12), the step\-only variant retains useful signal from function\-level execution feedback, matching KTO on MBPP and matching the fullStep\-KTOderobjective on MBPP\+\. However, removing the outcome\-level KTO term weakens performance on HumanEval, BigCodeBench, and LiveCodeBench\. TheStep\-KTOderobjective performs best overall, with the clearest advantage on BigCodeBench Hard\. These results support our formulation ofStep\-KTOderas a joint objective: the step loss provides localized credit assignment, while the outcome\-level term preserves the end\-to\-end correctness signal needed for robust program\-level performance\.

## Appendix MOff\-Policy Training Data in Cross\-Family Training

To test whether on\-policy candidate generation is necessary, we train DeepSeek\-Coder\-6\.7B\-Instruct on candidates generated by Qwen2\.5\-Coder\-3B\-Instruct on the same set of decomposed problems used in our main DeepSeek run\. Both training sets share the same decomposed function skeletons and unit tests, generated by Qwen2\.5\-Coder\-32B\-Instruct; the only difference is the model that produced the candidate completions\. The Qwen\-generated candidates are fully off\-policy with respect to the DeepSeek reference model\.

Table 13:Effect of training with off\-policy data on DeepSeek\-Coder\-6\.7B\-Instruct\.As shown in Table[13](https://arxiv.org/html/2608.23632#A13.T13), off\-policy training degrades both methods on the hardest benchmark \(BigCodeBench Hard: KTO 0\.122 to 0\.115,Step\-KTOder0\.128 to 0\.122\)\. The effect on LiveCodeBench is asymmetric: KTO regresses substantially \(0\.127 to 0\.112\) whileStep\-KTOderregresses marginally \(0\.131 to 0\.127\), suggesting that function\-level supervision is more robust to policy mismatch than outcome\-only supervision\. Easier benchmarks \(HumanEval, HumanEval\+\) show small improvements under off\-policy training, likely reflecting the broader problem distribution covered by the Qwen model’s candidate pool\.

We attribute the BigCodeBench Hard regression to noisier log\-ratio rewards: when the candidate distribution diverges from the reference model’s, the ratiolog⁡πθ/πref\\log\\pi\_\{\\theta\}/\\pi\_\{\\text\{ref\}\}is poorly calibrated, weakening the KTO objective\. The function\-level signal partially compensates because it operates on shorter token spans, where calibration noise has less cumulative effect\. We therefore use the on\-policy configuration as our default setting\.

## Appendix NLLM\-as\-a\-Judge vs\. Execution\-Based Step Labels

To evaluate execution\-based step labeling against a common alternative, we compareStep\-KTOder’s unit\-test\-based function labels with annotations from GPT\-5\.4 mini, applied to the same candidate functions in the Qwen2\.5\-Coder\-3B\-Instruct training pipeline\. The comparison has two components: \(i\) label\-level agreement statistics on the full candidate pool, and \(ii\) a downstream training experiment in which execution\-based labels are replaced by LLM\-as\-a\-judge labels in the actualStep\-KTOdertraining set\.

#### Setup\.

For agreement analysis, we sampleN=88,469N\{=\}88\{,\}469supervised functions from the Qwen2\.5\-Coder\-3B\-Instruct candidate pool, stratified by the\(o,z\)\(o,z\)outcome–step pair, and ask GPT\-5\.4 mini to predict whether each function would pass its associated unit tests\. The judge sees the function source code and the unit tests, and returns a binary label with a one\-sentence rationale; the system prompt is reproduced in Appendix[O](https://arxiv.org/html/2608.23632#A15)\. For the downstream experiment, we apply the same procedure only to the supervised functions in the final training rows, replace execution\-based labels with LLM\-as\-a\-judge labels where available, and retrain with the same settings as the main result\.

\(a\)Agreement by\(o,z\)\(o,z\)stratum\.![Refer to caption](https://arxiv.org/html/2608.23632v1/figure5.png)\(b\)Row\-normalized confusion matrix\.

Figure 4:LLM\-as\-a\-judge agreement with execution\-based labels is high for failing functions \(z=0z\{=\}0\) but much lower for passing functions \(z=1z\{=\}1\), revealing a bias toward predicting failure\.Table 14:Effect of replacing execution\-based step labels with GPT\-5\.4 mini judgments on Qwen2\.5\-Coder\-3B\-Instruct\.
#### Agreement is asymmetric\.

Figure[4\(a\)](https://arxiv.org/html/2608.23632#A14.F4.sf1)reports agreement by\(o,z\)\(o,z\)stratum, and Figure[4\(b\)](https://arxiv.org/html/2608.23632#A14.F4.sf2)shows the row\-normalized confusion matrix\. Overall agreement is73\.2%73\.2\\%, but this aggregate score hides a strong asymmetry\. The LLM agrees with execution on94\.1%94\.1\\%of functions labeled as failing by execution \(z=0z\{=\}0\), but on only52\.2%52\.2\\%of functions labeled as passing by execution \(z=1z\{=\}1\)\. At the stratum level, agreement is high on both failing\-function strata \(o=1,z=0o\{=\}1,z\{=\}0:92\.4%92\.4\\%;o=0,z=0o\{=\}0,z\{=\}0:94\.8%94\.8\\%\), but only around52%52\\%on both passing\-function strata\.

This pattern indicates that the LLM judge has a strong false\-negative bias: it often flags true failures, but also frequently hallucinates issues in correct implementations\. This preserves many negative conflict labels while corrupting the much larger pool of clean positive step labels\.

#### Downstream impact\.

Table[14](https://arxiv.org/html/2608.23632#A14.T14)comparesStep\-KTOdertrained with execution\-based labels versus LLM\-as\-a\-judge labels on Qwen2\.5\-Coder\-3B\-Instruct\. Replacing execution labels with GPT\-5\.4 mini judgments substantially degrades the hardest benchmarks, with relative drops of25\.8%25\.8\\%on BigCodeBench Hard and11\.8%11\.8\\%on LiveCodeBench\. Notably, BigCodeBench Hard falls below the outcome\-only KTO baseline, confirming that moderate label\-level agreement can still hide systematic bias that harms downstream training\.

#### Discussion\.

Moderate aggregate agreement does not translate into useful supervision because the LLM judge’s errors are highly asymmetric: its false\-negative bias corrupts passing step labels, whichStep\-KTOderneeds to stabilize function\-level training\. Within the operating regime of our framework, execution\-based labels are not merely a convenient choice but a necessary one\.

## Appendix OPrompts

Here, we outline the key prompts used in ourStep\-KTOderdata construction pipeline\. Figure[5](https://arxiv.org/html/2608.23632#A15.F5)shows the prompts used to decompose reference solutions into multi\-function programs\. Figure[6](https://arxiv.org/html/2608.23632#A15.F6)gives the prompt used for generating function\-level unit tests, and Figure[7](https://arxiv.org/html/2608.23632#A15.F7)shows the prompt used to sample on\-policy candidate completions from function skeletons\. Finally, Figure[8](https://arxiv.org/html/2608.23632#A15.F8)gives the prompt used by the LLM\-as\-a\-judge in our labeling\-method comparison \(Appendix[N](https://arxiv.org/html/2608.23632#A14)\)\.

Prompt for Function DecompositionSYSTEM PROMPT
You are an expert Python refactoring assistant\.
Your task is to decompose a Python solution into semantically meaningful top\-level helper functions while preserving exact behavior and the exact required interface\.
When the solution contains distinct logical stages, decompose it into multiple helper functions rather than leaving everything inside one large function\.
Do not invent trivial helpers or change the algorithm unnecessarily\. Follow the requested XML schema exactly\. Return ONLY valid XML\. No explanations\.USER PROMPT
Refactor the following Python solution into structured XML with multiple <function\> blocks\.
Goal:
\- Decompose into helper functions where useful\.
\- Preserve behavior exactly\.
\- Keep the required entrypoint unchanged \(same name and signature\)\.
Required entrypoint \(must appear EXACTLY in your code\):
\{required\_entrypoint\_line\}
Rules:
1\. Do NOT introduce class Solution \(this is a plain function problem\)\.
2\. Do NOT read from stdin or print output\.
3\. Do NOT include any top\-level execution \(no main\(\), no \_\_starting\_point\(\), no if \_\_name\_\_ == "\_\_main\_\_": \.\.\.\)\.
4\. Put helper functions ABOVE the required entrypoint\.
5\. Each function definition must be in its own <function\> block\.
6\. The <code\> section should contain exactly ONE def statement \(or one class\)\.
7\. Copy all original imports at the very top of the FIRST <code\> block\.
8\. Do NOT add unit tests yet: keep <tests\> as placeholder comments\.
9\. Do NOT define nested functions inside another function or method\. Every helper must be top\-level and placed in its own <function\> block\.
10\. When all <code\> blocks are concatenated in order, the result must be a valid standalone Python solution with exactly the same behavior as the original\.
11\. When the solution contains distinct logical stages, decompose it into multiple semantically meaningful helper functions\.
12\. Avoid returning a single large function unless decomposition is genuinely unnecessary\.
13\. Do not invent trivial helpers just to increase the number of functions\.
XML format:
<function name="\.\.\."\> <docstring\>\.\.\.</docstring\> <tests\> \{tests\_placeholder\} </tests\> <code\> \.\.\. code here \.\.\. </code\> </function\>
Problem: \{problem\_description\}
Original solution: \{ground\_truth\_solution\}
Refactored XML format \(ONLY XML\):Figure 5:System and user prompts used to decompose reference solutions into behavior\-preserving multi\-function programs\.Prompt for Unit Test GenerationOutput must be a JSON object with this exact schema:
\{ "cases": \[ \{"name": "short\_name", "args": \[\.\.\.\], "kwargs": \{\.\.\.\}\}, \.\.\. \] \}
Rules:
\- args must be a JSON array; kwargs must be a JSON object \(use \{\} if none\)\.
\- Use only JSON\-serializable values: null, true/false, numbers, strings, lists, objects\.
\- Keep inputs physically SMALL \(e\.g\., arrays under 5 items, integers between \-50 and 50\) to avoid execution timeouts\.
\- The test suite MUST include at least one or two ADVERSARIAL EDGE CASES \(e\.g\., empty lists \[\], empty strings "", zero 0, or negative numbers \-1\)\.
\- Do NOT make every case an edge case\. Provide a balanced mix of typical and boundary inputs\.
\- Do NOT include expected outputs\.
\- Provide 4 to 6 diverse cases\.Figure 6:User prompt schema and constraints for generating function\-level unit tests\. The model returns only test inputs; expected outputs are derived by executing the reference implementation\.Prompt for Candidate GenerationSolve the following programming problem\.
You MUST implement the provided skeleton by replacing each pass with a correct implementation\.
Rules:
\- Keep all function/class names and signatures unchanged\.
\- Do not remove any definitions\.
\- Do not add new functions, methods, or classes\.
\- Only fill the bodies of the provided definitions\.
\- Return a single complete Python module\.
\#\#\# problem
\{question\}
\#\#\# skeleton
\{skeleton\_code\}Figure 7:User prompt for candidate generation\. The target model fills the function skeleton \(withpassas each function body\) constructed in Stage 3 of our pipeline\.Prompt for LLM\-as\-a\-JudgeYou are an expert Python programmer acting as a code reviewer\. You will be given a Python function and a set of unit tests for that function\. Your job is to decide whether the function implementation would pass all the provided unit tests when executed\.
Respond in strict JSON of the form:
\{"label": 0 or 1, "reason": "<one short sentence\>"\}
\- label = 1 means you believe the function would pass all provided tests\.
\- label = 0 means you believe at least one test would fail\.
Do not include any other text outside of the JSON object\.Figure 8:System prompt for the GPT\-5\.4 mini judge in the LLM\-as\-a\-judge experiment \(Appendix[N](https://arxiv.org/html/2608.23632#A14)\)\. The judge sees the function’s source code and its generated unit tests, and predicts whether the function passes them\.