Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms

# Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms
Source: [https://arxiv.org/html/2608.14653](https://arxiv.org/html/2608.14653)
###### Abstract

Prediction uncertainty is a widely adopted metric for quantifying model confidence, with downstream applications spanning model explanation, data selection, and prediction rollback\. Despite its demonstrated utility, the potential of uncertainty quantification to enhance code generation in large language models \(LLMs\) remains largely underexplored, raising a critical question:to what extent can uncertainty serve as an effective signal for improving LLM\-based code generation?

To answer this question, we study uncertainty\-aware rollback decoding, an inference\-time strategy that uses uncertainty signals to identify unreliable generation regions and roll back to earlier valid prefixes without retraining the model\. We evaluate this framework on seven code LLMs, five code generation benchmarks, and eight token\-level uncertainty signals under a unified decoding setup\.

Our results show that the complete rollback framework improves over equal\-budget restart across the evaluated benchmarks and model settings, with gains of up to 0\.26 in pass@1 and 0\.35 in AvgTestPassRate on functional code generation benchmarks, and an absolute improvement of up to 6\.4% in Patch\-Aligned Safe Rate on Dsec\-Python\. Among the evaluated signals, information\-theoretic measures such as token entropy and negative log\-likelihood show the most favorable overall trend, frequently achieving the best or near\-best results on standard benchmarks\. A component\-controlled ablation further shows that feedback\-guided rollback provides the main improvement, while uncertainty localization provides an additional gain when checking, budget, rollback, and branch decay are held fixed\.

## IIntroduction

Large language models \(LLMs\) have achieved strong performance on a wide range of code generation benchmarks, from function\-level synthesis tasks to more realistic programming scenarios involving execution\-based evaluation and complex instructions\[[7](https://arxiv.org/html/2608.14653#bib.bib1),[6](https://arxiv.org/html/2608.14653#bib.bib2),[24](https://arxiv.org/html/2608.14653#bib.bib3),[41](https://arxiv.org/html/2608.14653#bib.bib28)\]\. Despite this progress, code generation remains fundamentally fragile\. Unlike natural language generation, code is governed by strict syntactic, semantic, and execution constraints, so a single early mistake can invalidate an entire program\. Under standard left\-to\-right decoding, such errors are irreversibly incorporated into the generation prefix and may subsequently propagate through later statements, causing error accumulation and amplifying their downstream impact\.

Prior work improves generated code through self\-debugging, iterative refinement, execution guidance, and intermediate verification\[[8](https://arxiv.org/html/2608.14653#bib.bib4),[25](https://arxiv.org/html/2608.14653#bib.bib5),[26](https://arxiv.org/html/2608.14653#bib.bib24),[36](https://arxiv.org/html/2608.14653#bib.bib25),[20](https://arxiv.org/html/2608.14653#bib.bib30)\]\. Rollback\-based decoding further enables failures to be corrected before a complete program is produced\[[17](https://arxiv.org/html/2608.14653#bib.bib23)\], while uncertainty estimation can identify unreliable generations\[[19](https://arxiv.org/html/2608.14653#bib.bib7),[10](https://arxiv.org/html/2608.14653#bib.bib8),[27](https://arxiv.org/html/2608.14653#bib.bib34)\]\. However, post\-generation repair acts only after errors have propagated, and uncertainty has mainly been studied as a final\-answer reliability measure rather than a decoding\-time control signal\.

We therefore study whether uncertainty can guide rollback localization and regeneration\. Our unified framework incrementally generates and checks code, rolls back after detected failures, and uses uncertainty to identify unreliable regions\. We compare information\-theoretic and sampling\-based signals across multiple benchmarks and model scales under a fixed rollback mechanism\.

Our study is organized around three research questions:

- •RQ1: Does the complete rollback framework improve over equal\-budget restart across datasets and models?We answer this question by comparing the full framework with equal\-budget restart on multiple benchmarks and model families\. This comparison evaluates the complete framework rather than isolating uncertainty\.
- •RQ2: What are the individual contributions of rollback, uncertainty, and branch decay?Here, rollback denotes reverting to an earlier generation prefix after detecting failure, uncertainty denotes model\-based signals for locating unreliable generation steps, and branch decay denotes a mechanism for reducing repeated exploration of unpromising branches\. We answer this question through ablation experiments, rollback\-localization diagnostics, and same\-budget comparisons that isolate the effect of uncertainty\-guided localization from additional generation budget\.
- •RQ3: How does uncertainty behave across rollback\-triggering error types?We answer this question by analyzing the distribution of uncertainty scores across different categories of rollback\-triggering errors\.

Our results show that the complete rollback framework improves over equal\-budget restart across the evaluated settings\. The matched component ablation attributes the main improvement to feedback\-guided rollback and an additional gain to uncertainty localization\. Across the evaluated settings, information\-theoretic signals generally provide more stable benefits than sampling\-based alternatives\.

In summary, this paper makes the following contributions:

- •We formulate uncertainty\-aware rollback decoding as a decoding\-time strategy for mitigating error accumulation in LLM\-based code generation\.
- •We provide a controlled empirical comparison of information\-theoretic and sampling\-based uncertainty signals within a unified rollback framework\.
- •We present a multi\-benchmark, multi\-model evaluation of the complete rollback framework, together with a component\-controlled uncertainty ablation and diagnostics of rollback localization, token cost, and same\-budget alternatives\.

## IIProblem Statement

We formalize code generation as an autoregressive decoding process conditioned on a problem specification\. Given an input promptxx\(e\.g\., a function signature together with a natural\-language description\), a pretrained language model generates an output token sequence𝐲1:T=\(y1,…,yT\)\\mathbf\{y\}\_\{1:T\}=\(y\_\{1\},\\ldots,y\_\{T\}\)of lengthTT, whereyty\_\{t\}denotes the token at positiontt\. We use𝐲<t=𝐲1:t−1\\mathbf\{y\}\_\{<t\}=\\mathbf\{y\}\_\{1:t\-1\}to denote the generated prefix before steptt\. The conditional distribution over the full sequence is factorized as

p​\(𝐲1:T∣x\)=∏t=1Tp​\(yt∣x,𝐲<t\)\.p\(\\mathbf\{y\}\_\{1:T\}\\mid x\)=\\prod\_\{t=1\}^\{T\}p\(y\_\{t\}\\mid x,\\mathbf\{y\}\_\{<t\}\)\.
Standard left\-to\-right decoding cannot revise an erroneous prefix, so early code mistakes can propagate into later statements and execution failures\[[36](https://arxiv.org/html/2608.14653#bib.bib25),[17](https://arxiv.org/html/2608.14653#bib.bib23),[28](https://arxiv.org/html/2608.14653#bib.bib6),[7](https://arxiv.org/html/2608.14653#bib.bib1)\]\. This is especially problematic for code, where one decision can make the remaining program syntactically invalid, semantically inconsistent, or non\-executable\.

Rollback\-based decoding permits revision after a detected failure\. Execution\-based feedback has been used to assess generated code and guide inference\-time revision or selection\[[26](https://arxiv.org/html/2608.14653#bib.bib24),[36](https://arxiv.org/html/2608.14653#bib.bib25),[17](https://arxiv.org/html/2608.14653#bib.bib23),[23](https://arxiv.org/html/2608.14653#bib.bib27),[21](https://arxiv.org/html/2608.14653#bib.bib29),[39](https://arxiv.org/html/2608.14653#bib.bib32),[3](https://arxiv.org/html/2608.14653#bib.bib33)\]\. In our framework, execution feedback determines*whether*to reconsider the current prefix, while uncertainty refines*where*rollback should occur\[[19](https://arxiv.org/html/2608.14653#bib.bib7),[10](https://arxiv.org/html/2608.14653#bib.bib8)\]\. Letr<tr<tdenote a rollback position\. When the current partial program corresponding to prefix𝐲1:t\\mathbf\{y\}\_\{1:t\}fails an execution check or a test case, the generated sequence is truncated back to the earlier prefix𝐲<r=𝐲1:r−1\\mathbf\{y\}\_\{<r\}=\\mathbf\{y\}\_\{1:r\-1\}, and decoding resumes from positionrr\. This changes generation from a strictly forward process into a search with selective revision\. The key subproblem is selectingrr: without a principled criterion, rollback remains a heuristic operation\. We study uncertainty signals derived from the predictive distribution as a basis for this selection\[[19](https://arxiv.org/html/2608.14653#bib.bib7),[10](https://arxiv.org/html/2608.14653#bib.bib8),[27](https://arxiv.org/html/2608.14653#bib.bib34)\]\. During decoding, each generated tokenyty\_\{t\}is associated with an uncertainty scoreutu\_\{t\}, computed from the predictive distributionp\(⋅∣x,𝐲<t\)p\(\\cdot\\mid x,\\mathbf\{y\}\_\{<t\}\), for example using token entropy or negative log\-likelihood\. Tokens with high uncertainty indicate positions where the predictive distribution is diffuse, suggesting lower confidence and a higher likelihood of erroneous generation\.

The central question is not only whether rollback helps, but whether uncertainty can indicate where rollback should occur\. We therefore keep the rollback mechanism fixed while varying the uncertainty signal, enabling a controlled assessment against standard decoding and uncertainty\-agnostic rollback\.

## IIIUncertainty Signals

Since rollback operates online, its uncertainty estimator must be efficiently computable and localized\. We distinguish:*single\-pass token\-level signals*, which can be computed from a single forward pass at each decoding step, and*multi\-pass signals*, which require repeated forward passes to estimate predictive uncertainty more robustly\. This follows the standard separation between measures from one predictive distribution and those estimated through stochastic inference or committee\-style disagreement\[[30](https://arxiv.org/html/2608.14653#bib.bib9),[12](https://arxiv.org/html/2608.14653#bib.bib14),[31](https://arxiv.org/html/2608.14653#bib.bib13)\]\.

### III\-ASingle\-Pass Token\-Level Signals

Single\-pass token\-level signals are computed directly fromp\(⋅∣x,y<t\)p\(\\cdot\\mid x,y\_\{<t\}\)without additional stochastic sampling\. We evaluate six such signals:

- •Average Negative Log\-Likelihood \(Avg NLL\): the average surprisal of generated tokens\[[32](https://arxiv.org/html/2608.14653#bib.bib11)\]\.
- •Max Token Entropy: the maximum token\-level entropy within the current statement or span\[[32](https://arxiv.org/html/2608.14653#bib.bib11),[30](https://arxiv.org/html/2608.14653#bib.bib9)\]\.
- •Max Probability: token confidence is converted to uncertainty as1−maxv⁡p​\(v∣x,y<t\)1\-\\max\_\{v\}p\(v\\mid x,y\_\{<t\}\), and the maximum converted score within the statement is used\[[22](https://arxiv.org/html/2608.14653#bib.bib10),[30](https://arxiv.org/html/2608.14653#bib.bib9)\]\.
- •Least Confidence: the same token\-level conversion,1−maxv⁡p​\(v∣x,y<t\)1\-\\max\_\{v\}p\(v\\mid x,y\_\{<t\}\), is averaged over the statement\. Thus, the two confidence measures share a common direction but differ in statement\-level aggregation\[[22](https://arxiv.org/html/2608.14653#bib.bib10),[30](https://arxiv.org/html/2608.14653#bib.bib9)\]\.
- •Margin of Confidence:1−\(p\(1\)−p\(2\)\)1\-\(p\_\{\(1\)\}\-p\_\{\(2\)\}\), wherep\(1\)p\_\{\(1\)\}andp\(2\)p\_\{\(2\)\}are the two largest token probabilities; statement scores use the mean\[[30](https://arxiv.org/html/2608.14653#bib.bib9)\]\.
- •Gini Impurity:1−∑vp​\(v∣x,y<t\)21\-\\sum\_\{v\}p\(v\\mid x,y\_\{<t\}\)^\{2\}, capturing distributional dispersion\[[4](https://arxiv.org/html/2608.14653#bib.bib12)\]\.

Avg NLL, Least Confidence, Margin of Confidence, and Gini Impurity use statement means; Max Token Entropy and converted Max Probability use statement maxima\. Larger values consistently indicate higher uncertainty\.

### III\-BMulti\-Pass Signals

Multi\-pass signals require repeated stochastic forward passes\. We consider two such signals:

- •Predictive Entropy: the entropy of the mean predictive distribution across stochastic samples\[[12](https://arxiv.org/html/2608.14653#bib.bib14),[13](https://arxiv.org/html/2608.14653#bib.bib15)\]\.
- •Variation Ratio:1−f/N1\-f/N, whereffis the modal\-token frequency overNNsamples\[[31](https://arxiv.org/html/2608.14653#bib.bib13),[13](https://arxiv.org/html/2608.14653#bib.bib15)\]\.

These signals require repeated stochastic inference and therefore cost more than single\-pass signals\[[12](https://arxiv.org/html/2608.14653#bib.bib14)\]\.

### III\-CUsage in Rollback Decoding

Uncertainty complements rather than replaces execution feedback\. Execution checks indicate*whether*a generated prefix is problematic, while uncertainty helps locate*where*to roll back\. For single\-pass signals, token scores are aggregated over the current statement span: Avg NLL uses mean token surprisal, Max Token Entropy uses maximum token entropy, and confidence\-style signals are oriented so that larger values indicate lower confidence before statement\-level ranking\.

For multi\-pass signals, token scores are computed from multiple stochastic forward passes and averaged within each statement before rollback is triggered\[[12](https://arxiv.org/html/2608.14653#bib.bib14),[13](https://arxiv.org/html/2608.14653#bib.bib15)\]\. Table[I](https://arxiv.org/html/2608.14653#S3.T1)summarizes the uncertainty signals evaluated in this work and their grouping by computation type\.

TABLE I:Uncertainty signals considered in this work\.Computation TypeSignalsSingle\-pass token\-levelAvg NLL, Max Token Entropy,Max Probability, Least Confidence,Margin of Confidence, Gini ImpurityMulti\-pass sampling\-basedPredictive Entropy, Variation Ratio

All signals are integrated into the same rollback framework, so decoding, checking, and rollback conditions remain identical across estimators\. Our goal is not to propose a new estimator, but to compare which signals best support rollback\-based correction\. We emphasize single\-pass signals for online use, while multi\-pass estimators serve as higher\-cost references\[[12](https://arxiv.org/html/2608.14653#bib.bib14),[30](https://arxiv.org/html/2608.14653#bib.bib9)\]\.

![Refer to caption](https://arxiv.org/html/2608.14653v1/overview_v4.png)Figure 1:Overview of the study design\. The task setup combines programming prompts, visible public tests, seven code LLMs, and eight uncertainty signals\. During decoding, successful chunks extend the accepted prefix, whereas failed checks trigger feedback\- or uncertainty\-guided rollback, branch decay, prefix restoration, and regeneration\. The evaluation addresses overall effectiveness, component contributions, and uncertainty across rollback\-triggering error types\.

## IVUncertainty\-Aware Decoding Framework

Figure[1](https://arxiv.org/html/2608.14653#S3.F1)illustrates the overall workflow of our framework\.

Building on rollback\-based code decoding\[[17](https://arxiv.org/html/2608.14653#bib.bib23)\], we instantiate a unified backbone that alternates incremental generation, lightweight checking, and rollback\-based revision, while varying only the uncertainty signal\. Given a task prompt and public tests, the decoder generates local chunks rather than the whole program at once\. Execution feedback determines whether revision is needed, and uncertainty refines the rollback location\. The goal is not a new rollback mechanism, but a controlled comparison under the same generation, checking, and regeneration protocol\.

##### Incremental statement generation\.

As shown in Algorithm[1](https://arxiv.org/html/2608.14653#algorithm1),yyis the accepted prefix\. At each iteration, the model extends it with a token\-by\-token, newline\-bounded chunkss, which serves as a statement\-like unit in practice\. This enables validation after each local extension instead of waiting for the complete function\.

Algorithm[1](https://arxiv.org/html/2608.14653#algorithm1)uses the following notation\. The candidate continuation isc=y⊕sc=y\\oplus s, where⊕\\oplusdenotes concatenation\.Norm⁡\(c\)\\operatorname\{Norm\}\(c\)makes an incomplete candidate executable using the normalization rules described below\.Check\\operatorname\{Check\}returns eitherPassor a diagnosticeeafter syntax checking, public\-test execution, and repetition detection\.Loc⁡\(e\)\\operatorname\{Loc\}\(e\)maps a valid syntax offset to its reported position, a repetition error to the beginning of the repeated line, and another execution error to the beginning of its reported line; it returns⊥\\botwhen that position is invalid\. For each statementSSon the current path,U​\(S\)U\(S\)is the statement\-level score of the selected uncertainty signal, using the aggregation defined in Section III, andstart⁡\(S\)\\operatorname\{start\}\(S\)is its first token position\.Complete⁡\(c,L\)\\operatorname\{Complete\}\(c,L\)is true when generation emits the end\-of\-sequence marker or reaches the maximum lengthLL\.

The trie𝒢\\mathcal\{G\}stores generated tokens, their uncertainty values, and previously explored outgoing branches;Path⁡\(𝒢\)\\operatorname\{Path\}\(\\mathcal\{G\}\)denotes the statements on its current candidate path\. The counterbbincludes all generated tokens, including discarded continuations\. The pair\(ℓprev,k\)\(\\ell\_\{\\mathrm\{prev\}\},k\)tracks the most recent feedback location and its consecutive failure count\. Feedback localization is used for at mostK=2K=2consecutive failures at one location; otherwise, rollback selects the highest\-scoring statement on the current path\. At the restored node, branch decay multiplies the stored sampling weight of the outgoing edge that begins the failed suffix byγ=0\.9\\gamma=0\.9before regeneration\.

Input:Promptxx, modelMM, public tests𝒯\\mathcal\{T\}, maximum lengthLL, token budgetBB

Output:Completion

yy
y←∅y\\leftarrow\\emptyset,

𝒢←root\\mathcal\{G\}\\leftarrow\\operatorname\{root\},

b←0b\\leftarrow 0,

\(ℓprev,k\)←\(⊥,0\)\(\\ell\_\{\\mathrm\{prev\}\},k\)\\leftarrow\(\\bot,0\)
//Step 1: Generate and record uncertainty

while*b<Bb<Band\|y\|<L\|y\|<L*do

Generate a newline\-bounded chunk

ssfrom

M\(⋅∣x,y\)M\(\\cdot\\mid x,y\)
Record every token and its uncertainty in

𝒢\\mathcal\{G\}
b←b\+\|s\|b\\leftarrow b\+\|s\|,

c←y⊕sc\\leftarrow y\\oplus s
//Step 2: Validate the candidate

z←Norm⁡\(c\)z\\leftarrow\\operatorname\{Norm\}\(c\),

e←Check⁡\(z,𝒯\)e\\leftarrow\\operatorname\{Check\}\(z,\\mathcal\{T\}\)
if*e=Passe=\\mathrm\{Pass\}*then

y←cy\\leftarrow c,

\(ℓprev,k\)←\(⊥,0\)\(\\ell\_\{\\mathrm\{prev\}\},k\)\\leftarrow\(\\bot,0\)
if*Complete⁡\(y,L\)\\operatorname\{Complete\}\(y,L\)*then

return

yy
else

//Step 3: Select and apply rollback

ℓ←Loc⁡\(e\)\\ell\\leftarrow\\operatorname\{Loc\}\(e\)
if*ℓ=ℓprev\\ell=\\ell\_\{\\mathrm\{prev\}\}*then

k←k\+1k\\leftarrow k\+1
else

\(ℓprev,k\)←\(ℓ,1\)\(\\ell\_\{\\mathrm\{prev\}\},k\)\\leftarrow\(\\ell,1\)
if*ℓ≠⊥\\ell\\neq\\botandk≤Kk\\leq K*then

r←ℓr\\leftarrow\\ell
else

r←start⁡\(arg​maxS∈Path⁡\(𝒢\)⁡U​\(S\)\)r\\leftarrow\\operatorname\{start\}\\\!\\left\(\\operatorname\*\{arg\\,max\}\_\{S\\in\\operatorname\{Path\}\(\\mathcal\{G\}\)\}U\(S\)\\right\)
Downweight by

γ\\gammathe edge entering the failed suffix at

rr
y←y\\leftarrowthe prefix immediately preceding

rr
Restore the current node of

𝒢\\mathcal\{G\}to that prefix

return

yy

Algorithm 1Uncertainty\-Aware Rollback DecodingDuring this process, the selected signal’s token\-level values are attached to nodes in𝒢\\mathcal\{G\}and aggregated into statement scoresU​\(S\)U\(S\)\. These scores are consulted only when feedback localization is invalid or repeatedly fails\.

##### Program assembly and feedback checking\.

The candidate partial program is normalized into executable form even when the function is incomplete\. This includes completing indentation blocks and temporarily insertingbreakinside an openwhileblock to avoid non\-terminating intermediate execution\. Lightweight analysis then performs syntax checking, public\-test execution, and repetition detection\. A passing chunk is merged into the accepted prefix; otherwise, the unreliable suffix is rolled back rather than discarding the whole program\.

##### Rollback strategy\.

Unlike restart\-based regeneration, rollback preserves the verified prefix\. Reliable syntax offsets, repetition locations, and runtime error lines determine rollback directly\. However, a runtime failure can surface after its underlying mistake, and shallow rollback can repeatedly revisit the same region\. If the reported location is invalid or fails repeatedly, the decoder instead rolls back to the highest\-uncertainty statement on the current trace, allowing it to bypass a larger suspicious region\.

##### Generation trace and branch decay\.

The trie stores the accepted path together with previously explored continuations\. On rollback, the decoder restores an earlier valid node and decays the outgoing edge that begins the failed suffix\. This soft penalty reduces the probability of regenerating exactly the same continuation without permanently blocking it, preserving flexibility when the search later revisits the prefix\.

##### Termination and budget control\.

The decoding loop is bounded by a token budget and a maximum generation length\. When the model emits the end\-of\-sequence marker or reaches the maximum generation length, decoding terminates and returns the current completion\. Hidden benchmark tests are applied only afterward for final evaluation and never affect online stopping or rollback\.

Overall, the framework separates three complementary roles during decoding\. Intermediate checking determines whether the current candidate remains executable, rollback preserves previously verified prefixes instead of restarting from scratch, and uncertainty is used only when explicit feedback cannot reliably localize the underlying error\.

## VExperimental Setup

Unless otherwise stated, models are evaluated without retraining, and the rollback mechanism, checking procedure, and decoding loop remain fixed across conditions\.

### V\-ABenchmarks

We evaluate the framework on five code generation benchmarks and one security\-oriented benchmark, covering function\-level synthesis, multilingual execution, competition\-style problem solving, and secure code repair\[[40](https://arxiv.org/html/2608.14653#bib.bib41),[5](https://arxiv.org/html/2608.14653#bib.bib42)\]\.

HumanEval\[[7](https://arxiv.org/html/2608.14653#bib.bib1)\]contains 164 function\-level Python tasks with hidden unit tests\. Intermediate checking uses executable public inputs when available; final evaluation follows the benchmark\-standard pipeline\.

MBPP\[[1](https://arxiv.org/html/2608.14653#bib.bib16)\]provides shorter, structured Python synthesis tasks, complementing HumanEval with basic algorithmic and library problems\.

HumanEval\-CPP\[[38](https://arxiv.org/html/2608.14653#bib.bib17)\]extends HumanEval\-style evaluation to C\+\+ and its stricter compilation constraints, where small structural errors can immediately invalidate the program\[[5](https://arxiv.org/html/2608.14653#bib.bib42)\]\.

CodeForces2305\[[9](https://arxiv.org/html/2608.14653#bib.bib18)\]contains 90 recent competition problems emphasizing long\-range reasoning, input/output handling, and algorithm design\[[23](https://arxiv.org/html/2608.14653#bib.bib27),[41](https://arxiv.org/html/2608.14653#bib.bib28)\]\. The problems go beyond function\-level synthesis, and their release dates reduce contamination risk for many evaluated models\.

APPS \(Competition subset\)\[[16](https://arxiv.org/html/2608.14653#bib.bib19)\]provides difficult programming tasks with hidden tests and less constrained solution spaces, complementing the recent CodeForces problems with a broader standard benchmark\.

Dsec\-Python\[[15](https://arxiv.org/html/2608.14653#bib.bib22)\]evaluates whether generated security patches align with the intended secure edit region, using a patch\-level metric rather than functional pass rate\[[2](https://arxiv.org/html/2608.14653#bib.bib39),[37](https://arxiv.org/html/2608.14653#bib.bib43)\]\.

### V\-BDecoding Setup

The model generates a statement\-level continuation up to a newline or local structural boundary, then immediately checks it before generating the next statement\. This allows rollback to occur online rather than only after a full draft is produced\[[35](https://arxiv.org/html/2608.14653#bib.bib38),[18](https://arxiv.org/html/2608.14653#bib.bib44),[11](https://arxiv.org/html/2608.14653#bib.bib45)\]\.

Unless otherwise specified, decoding uses temperature=0\.0=0\.0, top\-k=50k=50, top\-p=1\.0p=1\.0, and one sample per task\. The maximum generation length is set to 768 tokens, and the total token budget is limited to twice this length\. For each task, error\-line rollback is used for at most two consecutive failures at the same location before uncertainty\-guided localization is activated\. A path\-decay factor of0\.90\.9is applied during regeneration to reduce repeated exploration of previously failed continuations\.

### V\-CIntermediate Checking and Rollback

After each statement, benchmark\-specific checks either accept the prefix or trigger rollback and regeneration from an earlier prefix\.

We maintain a strict separation among lightweight checks, public tests, and hidden evaluation tests\. Lightweight checks perform syntax validation, executable\-prefix checks, and repetition detection during generation\. Public tests are benchmark\-provided visible examples used only for online checking and rollback decisions\. Hidden tests are accessed only after generation terminates for final evaluation\. No hidden\-test result is exposed to decoding, rollback localization, branch decay, candidate selection, or early stopping\.

Rollback localization follows a hybrid strategy\. When explicit error information is available, such as syntax offsets, runtime error lines, or repetition\-triggering regions, rollback is applied near the reported failure location\. If this location is invalid or fails more than twice consecutively, the framework instead selects the statement containing the highest\-uncertainty token in the current generation trace\. This design separates two roles: execution or analysis feedback determines*whether*the current prefix is problematic, while uncertainty helps determine*where*the decoding trajectory should be revised\[[14](https://arxiv.org/html/2608.14653#bib.bib40)\]\.

Path decay downweights, rather than blocks, previously failed continuations\.

### V\-DUncertainty Signals

We evaluate the signals in Table[I](https://arxiv.org/html/2608.14653#S3.T1)\. For each token position, multi\-pass signals useN=5N=5stochastic forward passes with MC dropout and sampling temperature1\.01\.0\. During uncertainty estimation, the model is temporarily switched to training mode to activate dropout, gradients are disabled, and the fixed per\-task random seed is retained\. The model is restored to evaluation mode for main\-path generation\. Predictive Entropy uses the mean probability distribution across the five passes; Variation Ratio uses their modal\-token frequency\[[33](https://arxiv.org/html/2608.14653#bib.bib37)\]\. All uncertainty signals are used only for rollback localization; the surrounding decoding, testing, and rollback mechanism is kept unchanged across all settings\. Table[III](https://arxiv.org/html/2608.14653#S6.T3)compares predictive effectiveness under the same rollback framework, but multi\-pass signals require additional forward passes and the comparison is not compute\-normalized\.

### V\-EBaselines and Ablation Settings

Our main RQ1 baseline isequal\-budget restart\. It uses the same prompt, model, sampling configuration, and 1,536\-token total budget as the full framework\. Each attempt is limited to 768 tokens; after a failed attempt, generation restarts from the original prompt within the remaining budget, without error feedback, rollback, path decay, or uncertainty localization\.

Component ablations include rollback without uncertainty guidance and uncertainty scoring without rollback, separating the effects of rollback, localization, and branch decay\.

We also conduct a same\-budget comparison on HumanEval and CodeForces2305 to test whether the observed gains are simply caused by using more generation tokens\. All compared strategies use CodeLlama\-7B, temperature=0\.2=0\.2, and the same 1536\-token budget; for each task, they share the initial seed defined as the base seed plus the task index\. The first strategy,restart from scratch, discards a failed attempt and regenerates a complete solution from the original prompt without error feedback, path decay, or sequence blocking\. The second strategy,error\-line rollback, rolls back according to the reported error location without uncertainty\-based refinement\. Error\-line rollback and the full framework use identical checking, regeneration, and branch\-decay mechanisms; they differ only in whether uncertainty can refine the rollback location\. The third strategy is the full framework with uncertainty\-guided rollback localization\. All strategies use public tests as the stopping signal\. Under the shared token budget, restart comparison controls for additional generation, while error\-line rollback versus the full framework measures the incremental contribution of uncertainty\-guided localization\. The component ablation separately evaluates branch decay\.

For Dsec\-Python, equal\-budget restart and the full framework use the same model, sampling configuration, and total token budget\. The full framework permits at most three rollback recoveries within that budget, while restart may produce at most three candidates under the same budget\. Both methods are counted as successful if any generated candidate satisfies PASR\. PASR is computed only after generation and is not used for candidate selection, rollback, or stopping\.

### V\-FEvaluation Metrics

For HumanEval, MBPP, HumanEval\-CPP, CodeForces2305, and APPS, we reportpass@1andAvgTestPassRate\. For taskii, letTiT\_\{i\}denote its hidden tests andyiy\_\{i\}the single generated program\. Then

AvgTestPassRate=1\|D\|​∑i=1\|D\|1\|Ti\|​∑j=1\|Ti\|𝟏​\[yi⊧ti​j\]\.\\mathrm\{AvgTestPassRate\}=\\frac\{1\}\{\|D\|\}\\sum\_\{i=1\}^\{\|D\|\}\\frac\{1\}\{\|T\_\{i\}\|\}\\sum\_\{j=1\}^\{\|T\_\{i\}\|\}\\mathbf\{1\}\[y\_\{i\}\\models t\_\{ij\}\]\.pass@1records whetheryiy\_\{i\}passes all hidden tests for taskii, whereasAvgTestPassRatemacro\-averages the fraction of hidden tests passed per task\. Thus, both metrics are compatible with one generated sample per task\[[7](https://arxiv.org/html/2608.14653#bib.bib1),[1](https://arxiv.org/html/2608.14653#bib.bib16),[16](https://arxiv.org/html/2608.14653#bib.bib19)\]\.

For Dsec\-Python,Patch\-Aligned Safe Rate \(PASR\)requires line\-level IoU≥0\.5\\geq 0\.5and normalized similarity≥0\.6\\geq 0\.6in the modified region\. A task succeeds if any of at most three candidates generated within the matched total token budget meets both criteria\.

For the uncertainty\-guided rollback diagnostics, we retain only events logged asuncertainty\_fallbackoruncertainty\_refinement, excluding error\-line and other feedback\-only rollbacks\. We report the number of uncertainty\-selected events, the subset with a comparable reported error line, the within\-3\-line rate over that subset, and the fraction of uncertainty\-selected events whose next lightweight check succeeds\. The line\-distance metric remains an operational proxy rather than manually annotated bug\-localization ground truth\.

For functional\-error analysis, we useAssertionErroras a proxy for functional failures, meaning programs that execute but fail test assertions; this should not be interpreted as a manually labeled semantic\-error category\.

### V\-GImplementation Details

The framework uses PyTorch\[[29](https://arxiv.org/html/2608.14653#bib.bib20)\]and HuggingFace Transformers\[[34](https://arxiv.org/html/2608.14653#bib.bib21)\]\. The main experiments ran on four NVIDIA Tesla V100\-SXM2 GPUs \(16GB each\), 28 Intel Xeon Gold 6132 CPU cores, and 754GB RAM, using Python 3\.10, PyTorch 2\.8\.0, Transformers 4\.40\.0, and CUDA 12\.7\.

The rollback\-diagnostic and same\-budget experiments reported in Section[VI\-B](https://arxiv.org/html/2608.14653#S6.SS2)were run separately from the main multi\-model evaluation using CodeLlama\-7B and Max Token Entropy\. We therefore interpret these results as within\-environment comparisons among rollback strategies rather than as direct runtime comparisons with the main server experiments\. The diagnostic machine used an NVIDIA RTX 3090 \(24GB\), Intel Xeon Gold 6226R CPUs, 125GB RAM, and Ubuntu 20\.04\.6, with Python 3\.11\.9, PyTorch 2\.3\.0\+cu121, and Transformers 4\.40\.1\.

## VIResults

This section evaluates overall effectiveness, uncertainty signals, model\-scale effects, rollback diagnostics, component ablations, and error types\. The main results are shown in Table[II](https://arxiv.org/html/2608.14653#S6.T2), Table[III](https://arxiv.org/html/2608.14653#S6.T3), Table[IV](https://arxiv.org/html/2608.14653#S6.T4), Table[V](https://arxiv.org/html/2608.14653#S6.T5), and Table[VI](https://arxiv.org/html/2608.14653#S6.T6), with complementary analyses in Figures[2](https://arxiv.org/html/2608.14653#S6.F2)and[3](https://arxiv.org/html/2608.14653#S6.F3)\.

### VI\-ARQ1: Does the complete rollback framework improve over equal\-budget restart across datasets and models?

Table[II](https://arxiv.org/html/2608.14653#S6.T2)compares the complete rollback framework with equal\-budget restart\. Because the full framework jointly includes feedback, rollback, branch decay, and uncertainty localization, this comparison supports the complete framework rather than the isolated effect of uncertainty\. Improvements appear on HumanEval, MBPP, and HumanEval\-CPP\. For example, on HumanEval, DeepSeek\-7B improves from 0\.32/0\.49 to 0\.52/0\.69, Qwen2\.5\-7B from 0\.36/0\.56 to 0\.55/0\.77, and CodeLlama\-34B from 0\.48/0\.68 to 0\.61/0\.81\.

TABLE II:RQ1 complete\-framework results against equal\-budget restart\. The full framework uses Max Token Entropy\. Both settings use a 1,536\-token total budget and at most 768 tokens per attempt\. Generation benchmarks report pass@1 / AvgTestPassRate; Dsec\-Python reports PASR\.DatasetDSDeepSeek\-7BSCStarCoder2\-7BQQwen2\.5\-7BMMistral\-7BCLCodeLlama\-7BSSafeCoder\-7BCLCodeLlama\-34BHumanEval0\.52/0\.690\.45/0\.680\.55/0\.770\.40/0\.610\.39/0\.610\.36/0\.590\.61/0\.81MBPP0\.34/0\.550\.30/0\.500\.36/0\.580\.27/0\.460\.25/0\.430\.23/0\.410\.43/0\.66HumanEval\-CPP0\.36/0\.560\.41/0\.570\.43/0\.660\.29/0\.440\.35/0\.540\.30/0\.470\.53/0\.73CodeForces23050\.09/0\.280\.08/0\.260\.10/0\.320\.07/0\.240\.06/0\.200\.05/0\.180\.15/0\.40APPS\-Competition0\.16/0\.260\.13/0\.210\.15/0\.240\.14/0\.250\.10/0\.180\.07/0\.100\.23/0\.34Dsec\-Python0\.560\.510\.570\.480\.550\.530\.54HumanEval–Restart0\.32/0\.490\.27/0\.380\.36/0\.560\.22/0\.400\.23/0\.420\.20/0\.370\.48/0\.68MBPP–Restart0\.22/0\.350\.19/0\.320\.23/0\.370\.17/0\.290\.16/0\.270\.14/0\.260\.34/0\.54HumanEval\-CPP–Restart0\.18/0\.300\.20/0\.270\.17/0\.310\.15/0\.270\.23/0\.330\.16/0\.240\.43/0\.61CodeForces2305–Restart0\.05/0\.150\.04/0\.140\.06/0\.180\.04/0\.130\.03/0\.110\.03/0\.100\.11/0\.30APPS\-Competition–Restart0\.13/0\.240\.08/0\.170\.14/0\.190\.09/0\.150\.08/0\.180\.05/0\.160\.16/0\.34Dsec\-Python–Restart0\.500\.470\.520\.450\.490\.470\.50

The trend remains visible on harder long\-horizon benchmarks\. On CodeForces2305, StarCoder2\-7B from 0\.04/0\.14 to 0\.08/0\.26, and CodeLlama\-34B from 0\.11/0\.30 to 0\.15/0\.40\.

For Dsec\-Python, the complete framework also improves over equal\-budget restart across all evaluated models in Table[II](https://arxiv.org/html/2608.14653#S6.T2), e\.g\., from 0\.50 to 0\.56 for DeepSeek\-7B and from 0\.52 to 0\.57 for Qwen2\.5\-7B\. However, the margins are smaller than those on execution\-based benchmarks\. A likely reason is that Patch\-Aligned Safe Rate is a relatively coarse static metric, whose sensitivity to fine\-grained decoding differences is limited\.

Finding 1\.The complete rollback framework improves over equal\-budget restart across the evaluated benchmarks; this comparison does not isolate the contribution of uncertainty\.

Table[III](https://arxiv.org/html/2608.14653#S6.T3)compares the predictive effectiveness of uncertainty signals under the same rollback framework; it is not an equal\-compute comparison because multi\-pass signals require five forward passes per token position\. Overall, information\-theoretic signals, especiallyMax Token EntropyandAvg NLL, show the strongest trend on standard code\-generation benchmarks\. Max Token Entropy achieves the best or near\-best results in many settings, including HumanEval, MBPP, HumanEval\-CPP, and CodeForces2305\. Avg NLL is also consistently competitive\.

TABLE III:Predictive\-effectiveness comparison of uncertainty signals under the same rollback framework; results are not compute\-normalized\. Generation benchmarks report pass@1 / AvgTestPassRate; Dsec\-Python reports PASR\. Multi\-pass signals use five stochastic forward passes\. The best result for each dataset–model pair is shaded and boldfaced \(ties included\)\.DatasetMethodDSDeepSeek\-7BSCStarCoder2\-7BQQwen2\.5\-7BMMistral\-7BCLCodeLlama\-7BSSafeCoder\-7BCLCodeLlama\-34BHumanEvalAvg NLL0\.49/0\.670\.45/0\.620\.54/0\.720\.38/0\.560\.42/0\.590\.34/0\.550\.60/0\.77Max Token Entropy0\.52/0\.690\.45/0\.680\.55/0\.770\.40/0\.610\.39/0\.610\.36/0\.590\.61/0\.81Max Probability0\.44/0\.610\.48/0\.590\.47/0\.670\.37/0\.540\.43/0\.610\.36/0\.520\.53/0\.70Least Confidence0\.46/0\.630\.43/0\.640\.51/0\.650\.42/0\.580\.40/0\.570\.31/0\.530\.52/0\.72Margin of Confidence0\.47/0\.650\.50/0\.680\.49/0\.690\.36/0\.550\.44/0\.580\.37/0\.510\.56/0\.71Gini Impurity0\.51/0\.620\.41/0\.660\.55/0\.680\.43/0\.600\.39/0\.560\.35/0\.580\.54/0\.69Predictive Entropy0\.43/0\.660\.49/0\.610\.48/0\.730\.39/0\.610\.36/0\.530\.33/0\.590\.57/0\.78Variation Ratio0\.45/0\.600\.46/0\.670\.50/0\.640\.35/0\.570\.37/0\.540\.30/0\.500\.51/0\.74MBPPAvg NLL0\.32/0\.530\.28/0\.470\.36/0\.550\.26/0\.440\.25/0\.420\.23/0\.400\.42/0\.64Max Token Entropy0\.34/0\.550\.30/0\.500\.36/0\.580\.27/0\.460\.25/0\.430\.23/0\.410\.43/0\.66Max Probability0\.29/0\.490\.32/0\.440\.30/0\.520\.25/0\.410\.28/0\.460\.25/0\.380\.37/0\.60Least Confidence0\.31/0\.500\.27/0\.480\.33/0\.510\.29/0\.430\.23/0\.410\.21/0\.400\.36/0\.60Margin of Confidence0\.30/0\.520\.32/0\.500\.32/0\.540\.24/0\.400\.27/0\.420\.26/0\.370\.40/0\.59Gini Impurity0\.34/0\.480\.26/0\.500\.35/0\.530\.30/0\.450\.22/0\.400\.25/0\.440\.38/0\.58Predictive Entropy0\.30/0\.550\.31/0\.470\.29/0\.530\.27/0\.480\.21/0\.390\.20/0\.430\.43/0\.66Variation Ratio0\.28/0\.480\.31/0\.530\.32/0\.500\.24/0\.430\.26/0\.390\.20/0\.380\.37/0\.59HumanEval\-CPPAvg NLL0\.33/0\.570\.36/0\.510\.40/0\.680\.27/0\.390\.32/0\.560\.28/0\.430\.49/0\.74Max Token Entropy0\.36/0\.560\.41/0\.570\.43/0\.660\.29/0\.440\.35/0\.540\.30/0\.470\.53/0\.73Max Probability0\.29/0\.460\.38/0\.540\.35/0\.610\.26/0\.370\.37/0\.500\.32/0\.410\.45/0\.70Least Confidence0\.34/0\.490\.33/0\.550\.39/0\.550\.31/0\.470\.32/0\.540\.25/0\.450\.47/0\.63Margin of Confidence0\.30/0\.540\.40/0\.590\.37/0\.640\.24/0\.350\.37/0\.550\.33/0\.400\.50/0\.67Gini Impurity0\.36/0\.530\.34/0\.560\.42/0\.580\.32/0\.410\.28/0\.480\.27/0\.490\.51/0\.65Predictive Entropy0\.29/0\.530\.39/0\.470\.36/0\.630\.28/0\.460\.29/0\.440\.24/0\.470\.46/0\.72Variation Ratio0\.31/0\.430\.36/0\.580\.41/0\.530\.23/0\.460\.27/0\.410\.26/0\.380\.44/0\.71CodeForces2305Avg NLL0\.08/0\.260\.07/0\.240\.08/0\.280\.06/0\.220\.05/0\.190\.04/0\.170\.14/0\.38Max Token Entropy0\.09/0\.280\.08/0\.260\.10/0\.320\.07/0\.240\.06/0\.200\.05/0\.180\.15/0\.40Max Probability0\.07/0\.230\.08/0\.230\.07/0\.260\.06/0\.190\.06/0\.200\.05/0\.160\.12/0\.35Least Confidence0\.08/0\.240\.07/0\.250\.08/0\.250\.07/0\.200\.05/0\.180\.04/0\.170\.12/0\.36Margin of Confidence0\.07/0\.250\.09/0\.260\.07/0\.280\.05/0\.180\.07/0\.190\.06/0\.150\.13/0\.34Gini Impurity0\.09/0\.240\.06/0\.240\.08/0\.270\.08/0\.230\.04/0\.170\.05/0\.190\.11/0\.33Predictive Entropy0\.06/0\.270\.08/0\.240\.07/0\.290\.06/0\.250\.04/0\.160\.04/0\.180\.14/0\.37Variation Ratio0\.07/0\.210\.07/0\.250\.07/0\.230\.05/0\.190\.06/0\.150\.03/0\.150\.10/0\.32APPS\-CompetitionAvg NLL0\.15/0\.240\.12/0\.200\.14/0\.230\.13/0\.210\.09/0\.160\.06/0\.090\.21/0\.31Max Token Entropy0\.16/0\.260\.13/0\.210\.15/0\.240\.14/0\.250\.10/0\.180\.07/0\.100\.23/0\.34Max Probability0\.11/0\.190\.15/0\.220\.10/0\.180\.12/0\.170\.11/0\.190\.08/0\.110\.18/0\.27Least Confidence0\.13/0\.200\.10/0\.180\.13/0\.190\.15/0\.220\.08/0\.150\.05/0\.080\.17/0\.28Margin of Confidence0\.14/0\.220\.16/0\.240\.12/0\.200\.10/0\.160\.12/0\.170\.09/0\.120\.20/0\.29Gini Impurity0\.17/0\.230\.09/0\.170\.16/0\.220\.16/0\.240\.07/0\.140\.08/0\.130\.19/0\.26Predictive Entropy0\.11/0\.200\.14/0\.190\.12/0\.200\.12/0\.220\.06/0\.130\.04/0\.090\.22/0\.32Variation Ratio0\.09/0\.170\.11/0\.200\.12/0\.180\.08/0\.150\.07/0\.120\.03/0\.060\.16/0\.24Dsec\-PythonAvg NLL0\.540\.530\.560\.500\.530\.510\.55Max Token Entropy0\.560\.510\.570\.480\.550\.530\.54Max Probability0\.510\.560\.520\.510\.550\.500\.52Least Confidence0\.540\.500\.550\.520\.490\.550\.51Margin of Confidence0\.520\.570\.530\.470\.560\.510\.50Gini Impurity0\.570\.490\.550\.530\.480\.560\.53Predictive Entropy0\.530\.550\.510\.510\.520\.540\.56Variation Ratio0\.510\.540\.540\.460\.500\.490\.48

At the same time, no single signal is universally optimal\. In several cases, confidence\-based or alternative signals remain competitive\. For example, Gini Impurity performs strongly on APPS\-Competition for DeepSeek\-7B, and the best\-performing signal on Dsec\-Python varies by model\. The differences among signals also become smaller on harder settings such as CodeForces2305, APPS\-Competition, and Dsec\-Python\.

Finding 2\.In the predictive\-effectiveness comparison, Max Token Entropy and Avg NLL perform relatively better than other signals, but no signal consistently dominates; multi\-pass results are not compute\-normalized\.

The gains in Table[II](https://arxiv.org/html/2608.14653#S6.T2)are not limited to a single evaluated backbone\. Improvements appear across multiple 7B models, including DeepSeek\-7B, StarCoder2\-7B, Qwen2\.5\-7B, Mistral\-7B, CodeLlama\-7B, and SafeCoder\-7B, suggesting that the method is not tied to a specific architecture\.

The framework also remains effective for stronger models\. In Table[II](https://arxiv.org/html/2608.14653#S6.T2), CodeLlama\-34B improves from 0\.48/0\.68 to 0\.61/0\.81 on HumanEval, from 0\.34/0\.54 to 0\.43/0\.66 on MBPP, and from 0\.43/0\.61 to 0\.53/0\.73 on HumanEval\-CPP\. This suggests that rollback remains useful not only for weaker models, but also for stronger generators in the evaluated settings\.

We also fine\-tune CodeLlama\-7B on the APPS training split using 4\-bit NF4 QLoRA for two epochs \(learning rate2×10−42\\times 10^\{\-4\}, rank 16, LoRA alpha 32, seed 42\)\. Figure[2](https://arxiv.org/html/2608.14653#S6.F2)compares the merged final\-epoch model with the base model under the same rollback configuration\.

![Refer to caption](https://arxiv.org/html/2608.14653v1/ft_v3.png)Figure 2:CodeLlama\-7B and its fine\-tuned variant under the same rollback configuration\. Red labels show the maximum absolute gain in each panel\.These improvements are consistently observed across models with different parameter scales and across both functional and security\-oriented benchmarks\. Although the absolute gains vary depending on task difficulty and model capability, the overall trend suggests that rollback\-based decoding remains beneficial under diverse evaluation settings\.

Finding 3\.The proposed framework remains effective across the evaluated model scales, and the fine\-tuned variant also improves over its base model\.

### VI\-BRQ2: What are the contributions of rollback, uncertainty, and branch decay?

#### VI\-B1Rollback Diagnostics and Controlled Comparison

To examine uncertainty\-guided decisions directly, we instrumented CodeLlama\-7B with Max Token Entropy on HumanEval and on CodeForces2305 with its revised evaluation harness\. The diagnostic run uses temperature=0\.0=0\.0and seed=0=0\. It records token entropy and, when the reported line is invalid or the same location fails more than twice, rolls back to the line containing the highest\-entropy token on the current path; no fixed entropy threshold is used\. Table[IV](https://arxiv.org/html/2608.14653#S6.T4)includes onlyuncertainty\_fallbackanduncertainty\_refinementevents\. On HumanEval, 1,346 rollbacks are selected by entropy; 226 have a comparable reported error line, and 85\.84% of these selections fall within three lines\. The next lightweight check succeeds after 77\.93% of uncertainty\-selected rollbacks\. On CodeForces2305, the corresponding counts are 951 and 417, with a 72\.90% within\-3\-line rate and a 70\.14% next\-check pass rate\.

TABLE IV:Diagnostics restricted to uncertainty\-selected rollback events\. UC Ev\.: uncertainty\-selected events; Comp\.: events with a comparable reported error line;≤\\leq3: rollback within three lines\.DatasetUC Ev\.Comp\.≤\\leq3Next PassHumanEval1,34622685\.84%77\.93%CF230595141772\.90%70\.14%These measurements isolate behavior after entropy\-based localization rather than mixing it with feedback\-only rollback\. They show that uncertainty\-selected points are often close to the observed failure and frequently enable the next check to pass, although they do not by themselves constitute a counterfactual comparison against alternative localization rules\.

The runs average 11\.52 and 20\.24 rollbacks per task on HumanEval and CodeForces2305, respectively\. Uncertainty\-score computation accounts for 3\.99% and 1\.86% of runtime in these diagnostic runs; these values do not represent end\-to\-end overhead relative to standard decoding\.

We further compare the full framework against two alternatives under the same 1536\-token budget\. As shown in Table[V](https://arxiv.org/html/2608.14653#S6.T5), simply restarting from the original prompt after failure reaches 23\.17% pass rate on HumanEval, while rolling back directly to the reported error line reaches 32\.32%\. The full framework reaches 41\.46% while using fewer average generated tokens than both alternatives\. The same trend appears on CodeForces2305: restart and error\-line rollback both reach 2\.22%, while the full framework reaches 4\.44%\. Although the absolute pass rate on CodeForces2305 remains low due to task difficulty, the controlled comparison indicates that the gain is not explained merely by allowing more tokens or more regeneration attempts\.

TABLE V:Same\-budget comparison under a 1536\-token budget\. Tok\.: average generated tokens; RB: average rollback count \(not applicable to restart\)\.DatasetStrategyPassTok\.RBHumanEvalRestart from scratch23\.17%1224\.15–Error\-line rollback32\.32%935\.3472\.15Full framework41\.46%644\.2713\.40CodeForces2305Restart from scratch2\.22%1427\.47–Error\-line rollback2\.22%1399\.0096\.87Full framework4\.44%1304\.7151\.10##### Functional\-failure proxy\.

To distinguish assertion failures from parsing and runtime errors, we trackAssertionError, which indicates that a program executes but fails a test assertion\. In the temperature\-0\.00\.0diagnostic run, it accounts for 1,102/1,889 \(58\.34%\) rollback events and 69/164 \(42\.07%\) final samples on HumanEval, and 534/1,822 \(29\.31%\) rollback events and 72/90 \(80\.00%\) final samples on CodeForces2305\. Event\-level and final\-sample statistics therefore use different denominators\. In the separate temperature\-0\.20\.2same\-budget run, finalAssertionErrorproxy cases decrease from 110 to 76 on HumanEval and from 73 to 67 on CodeForces2305 when the full method replaces error\-line rollback\. This is a test\-level proxy rather than manually annotated semantic\-error ground truth\.

Finding 4\.Strictly filtered uncertainty\-guided rollbacks frequently select locations near the reported failure and are followed by high next\-check pass rates; uncertainty\-score computation occupies a small fraction of diagnostic runtime, while same\-budget comparisons show gains beyond extra generation budget and fewer finalAssertionErrorproxy cases than error\-line rollback\. An additional observation is that the framework often requires fewer generated tokens before converging to a successful solution\. Instead of repeatedly regenerating complete programs, rollback revisits only suspicious regions while preserving verified prefixes\. This behavior explains why the full framework can simultaneously achieve higher accuracy and lower average token consumption under the same overall decoding budget\. This result also suggests that rollback improves search efficiency instead of merely increasing exploration\. Because validated prefixes are retained throughout decoding, the search process can concentrate on correcting suspicious regions rather than reconstructing already verified program fragments\. Consequently, the available generation budget is allocated more effectively, allowing additional refinement attempts before the decoding budget is exhausted\.

#### VI\-B2Component Ablation

Table[VI](https://arxiv.org/html/2608.14653#S6.T6)reports the CodeLlama\-7B ablation on HumanEval\. Feedback\-Only Detection improves only slightly over Vanilla Decoding \(0\.25 vs\. 0\.23 pass@1\), while Feedback\-Guided Rollback reaches 0\.33 pass@1 and 0\.53 AvgTestPassRate\. The causal comparison for uncertainty is between Feedback\-Guided Rollback and the Full Framework: both use feedback, rollback, branch decay, the same checks, and the same budget, and differ only in uncertainty localization\. Adding uncertainty raises pass@1 from 0\.33 to 0\.39 and AvgTestPassRate from 0\.53 to 0\.61\. This matched evidence is currently limited to CodeLlama\-7B on HumanEval\.

TABLE VI:CodeLlama\-7B ablation results on HumanEval\. Feedback\-Guided Rollback and Full Framework share feedback, rollback, branch decay, checking, and budget; only the latter uses uncertainty localization\. Comp\. denotes compilation pass rate\.Variantpass@1AvgTestPassRateComp\.Vanilla Decoding0\.230\.420\.84Feedback\-Only Detection0\.250\.440\.86Feedback\-Guided Rollback0\.330\.530\.92Uncertainty\-Guided Rollback0\.270\.460\.88Full w/o Branch Decay0\.370\.590\.94Full Framework0\.390\.610\.96Finding 5\.Feedback\-guided rollback provides the main improvement, while uncertainty provides an additional gain when checking, budget, rollback, and branch decay are controlled\.

### VI\-CRQ3: How does uncertainty behave across rollback\-triggering error types?

Table[VII](https://arxiv.org/html/2608.14653#S6.T7)complements the aggregate analysis with representative corrections across rollback\-triggering error types\.

TABLE VII:Representative corrections across rollback\-triggering error types on HumanEval\. Highlighted tokens indicate the repaired fragments\.Error TypeBefore RollbackAfter RollbackNameError[⬇](data:text/plain;base64,IyBUQVNLIDQKbWVhbiA9IHN1bShudW1iZXJzKSAvIGxlbihudW1iZXJzKQpyZXR1cm4gc3VtKGFicyh4IC0gKCpAXGNoZ3ttZWFuKG51bWJlcnMpfUAqKSkgZm9yIHggaW4gbnVtYmVycyk=)mean=sum\(numbers\)/len\(numbers\)returnsum\(abs\(x\-mean\(numbers\)\)forxinnumbers\)[⬇](data:text/plain;base64,IyBUQVNLIDQKbWVhbiA9IHN1bShudW1iZXJzKSAvIGxlbihudW1iZXJzKQpyZXR1cm4gc3VtKGFicyh4IC0gKCpAXGNoZ3ttZWFufUAqKSkgZm9yIHggaW4gbnVtYmVycyk=)mean=sum\(numbers\)/len\(numbers\)returnsum\(abs\(x\-mean\)forxinnumbers\)IndexError[⬇](data:text/plain;base64,IyBUQVNLIDMzCmwuaW5zZXJ0KGksIHNvcnRlZChsW2k6aSszXSkpCmwucG9wKGkrMSkKbC5wb3AoaSsxKQooKkBcY2hne2wucG9wKGkrMSl9QCop)l\.insert\(i,sorted\(l\[i:i\+3\]\)\)l\.pop\(i\+1\)l\.pop\(i\+1\)l\.pop\(i\+1\)[⬇](data:text/plain;base64,IyBUQVNLIDMzCigqQFxjaGd7bFtpOmkrM10gPSBzb3J0ZWQobFtpOmkrM10pfUAqKQ==)l\[i:i\+3\] = sorted\(l\[i:i\+3\]\)RepeatPatternError[⬇](data:text/plain;base64,IyBUQVNLIDEKaWYgcGFyZW5fc3RyaW5nID09ICcoKSc6CiAgICByZXR1cm4gWycoKSddCmlmIHBhcmVuX3N0cmluZyA9PSAnKCkoKSc6CiAgICByZXR1cm4gWycoKScsICcoKSddCigqQFxjaGd7XHRleHR0dHtpZiBwYXJlblxfc3RyaW5nID09ICcoKSgpKCknOn19QCop)ifparen\_string==’\(\)’:return\[’\(\)’\]ifparen\_string==’\(\)\(\)’:return\[’\(\)’,’\(\)’\]if paren\_string == ’\(\)\(\)\(\)’:[⬇](data:text/plain;base64,IyBUQVNLIDEKZ3JvdXBzLCBjdXIsIGJhbCA9IFtdLCBbXSwgMApmb3IgY2ggaW4gcGFyZW5fc3RyaW5nOgogICAgYmFsICs9IDEgaWYgY2ggPT0gJygnIGVsc2UgLTEKICAgIGN1ci5hcHBlbmQoY2gpCiAgICBpZiBiYWwgPT0gMDoKICAgICAgICBncm91cHMuYXBwZW5kKCcnLmpvaW4oY3VyKSk7IGN1ciA9IFtd)groups,cur,bal=\[\],\[\],0forchinparen\_string:bal\+=1ifch==’\(’else\-1cur\.append\(ch\)ifbal==0:groups\.append\(’’\.join\(cur\)\);cur=\[\]SyntaxError[⬇](data:text/plain;base64,IyBUQVNLIDYKZm9yIGNoIGluIHBhcmVuX3N0cmluZzoKICAgIGlmIGNoID09ICcoJzoKICAgICAgICBkZXB0aCArPSAxCnJldHVybiAoKkBcY2hne1t9QCop)forchinparen\_string:ifch==’\(’:depth\+=1return\[[⬇](data:text/plain;base64,IyBUQVNLIDYKZm9yIGNoIGluIHBhcmVuX3N0cmluZzoKICAgIGlmIGNoID09ICcoJzoKICAgICAgICBkZXB0aCArPSAxCnJldHVybiAoKkBcY2hne2RlcHRofUAqKQ==)forchinparen\_string:ifch==’\(’:depth\+=1returndepthTable[VII](https://arxiv.org/html/2608.14653#S6.T7)illustrates that rollback supports both localized repairs and broader structural revisions\. NameError and SyntaxError are corrected through small fragment replacements, whereas IndexError and RepeatPatternError require rewriting a larger statement or control\-flow block, showing why rollback depth should adapt to the observed failure pattern\.

Figure[3](https://arxiv.org/html/2608.14653#S6.F3)shows the distribution of rollback\-triggering error types and their associated uncertainty values\. NameError \(n=283n=283\) and TypeError \(n=202n=202\) are among the most frequent error categories, followed by RepeatPatternError, IndexError, and ValueError\.

![Refer to caption](https://arxiv.org/html/2608.14653v1/analysis.png)Figure 3:Rollback\-triggering error types and uncertainty\. Bars show event\-level means, white circles show medians, and error bars denote one standard deviation across rollback events\. Error bars are omitted for categories withn=1n=1\.At the same time, the highest uncertainty values are associated with more structurally unstable errors\. SyntaxError has the highest mean uncertainty \(1\.85\), while IndentationError \(1\.51\) and TypeError \(1\.51\) are also clearly above the overall average\. In contrast, more common local semantic errors such as NameError \(0\.94\), IndexError \(0\.83\), and ValueError \(0\.70\) show lower uncertainty\. This suggests that uncertainty is not simply tracking how often an error occurs; instead, it better reflects how destabilizing that error is to the ongoing generation process\.

The observed uncertainty distribution also suggests that uncertainty is associated with structural instability rather than merely reflecting error frequency\. This observation is consistent with the intuition that rollback should prioritize regions where the decoding trajectory becomes unstable instead of simply revisiting the most recently generated tokens\.

This pattern supports the role of uncertainty in our framework: it is more useful as a cue for rollback localization than as a standalone predictor of final correctness\. We note that categories with fewer than five events are reported descriptively and should be interpreted cautiously; the error bars summarize dispersion and are not confidence intervals\.

Finding 6\.Rollback\-triggering uncertainty tends to be higher for structurally unstable errors than for common local semantic mistakes, supporting its use as a rollback\-localization cue\.

## VIIThreats to Validity

Statistical validity\.Due to computational cost, our experiments use a single\-run evaluation setting and do not include repeated trials, confidence intervals, or formal significance tests\. Therefore, the reported improvements should be interpreted as empirical trends on the evaluated benchmarks rather than statistically conclusive effect estimates, especially when the gaps between uncertainty signals are small\[[24](https://arxiv.org/html/2608.14653#bib.bib3),[41](https://arxiv.org/html/2608.14653#bib.bib28),[9](https://arxiv.org/html/2608.14653#bib.bib18)\]\.

Construct validity\.Our evaluation uses functional correctness metrics \(e\.g\., pass@1 and AvgTestPassRate\) for code\-generation benchmarks and Patch\-Aligned Safe Rate for Dsec\-Python\. While practical, these metrics do not fully capture all aspects of generation quality\. In particular, the Dsec\-Python metric relies on patch alignment and lightweight static validation, so its sensitivity to subtle security improvements is limited compared with end\-to\-end exploit\-level verification\. Similarly, our rollback\-line distance andAssertionErroranalyses are operational proxies rather than manually annotated ground truth for bug localization or semantic functional\-error categories\.

External validity\.Although we evaluate multiple representative open\-source code LLMs, including several 7B\-scale models and one 34B model, the study still covers only a limited portion of the model space\. In addition, most benchmarks are Python\-centered\. Therefore, the findings should be understood as evidence on the evaluated settings rather than a universal conclusion\.

Methodological validity\.Our conclusions are drawn under a fixed rollback\-based decoding framework with a specific intermediate\-checking and branch\-decay design\. Therefore, the observed effectiveness of uncertainty signals should be interpreted in this context\. Different rollback strategies, checking mechanisms, or search policies may lead to different relative behaviors of uncertainty estimators\. In addition, lightweight intermediate checking may not capture all failure modes equally well\. The diagnostic and same\-budget experiments were conducted on a separate machine from the main multi\-model evaluation, so their runtime numbers should be interpreted within that diagnostic setting rather than directly compared with the main experimental server\.

Future work may further investigate adaptive rollback policies, alternative intermediate verification strategies, and larger proprietary code models\. We also expect stronger execution feedback to further improve uncertainty\-guided localization in more realistic programming environments\.

## VIIIConclusion

This paper studied whether uncertainty signals can improve rollback\-based decoding for code generation\. The complete rollback framework improves over equal\-budget restart across the evaluated models and benchmarks, with gains of up to 0\.26 in pass@1 and 0\.35 in AvgTestPassRate on functional benchmarks and up to 6\.4% in Patch\-Aligned Safe Rate on Dsec\-Python\. These comparisons evaluate the complete framework rather than uncertainty alone\. In the matched component ablation, feedback\-guided rollback provides the main improvement, while uncertainty adds a further gain when checking, budget, rollback, and branch decay are controlled\. Diagnostics restricted to entropy\-selected events show high within\-3\-line and next\-check pass rates; uncertainty\-score computation accounts for a small fraction of diagnostic runtime but does not measure end\-to\-end overhead relative to standard decoding\. Information\-theoretic signals show the most stable predictive effectiveness, although multi\-pass comparisons are not compute\-normalized and no signal is universally best\.

## Data Availability Statement

The replication package, including the source code and evaluation scripts, is available through an anonymous repository: https://anonymous\.4open\.science/r/ICSE2027\-257C

## References

- \[1\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p3.1),[§V\-F](https://arxiv.org/html/2608.14653#S5.SS6.p1.5)\.
- \[2\]M\. H\. M\. Bhuiyan, A\. S\. Parthasarathy, N\. Vasilakis, M\. Pradel, and C\. Staicu\(2023\)SecBench\.js: an executable security benchmark suite for server\-side javascript\.InProceedings of the 45th IEEE/ACM International Conference on Software Engineering,pp\. 1059–1070\.External Links:[Document](https://dx.doi.org/10.1109/ICSE48619.2023.00096)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p7.1)\.
- \[3\]Z\. Bi, Y\. Wan, Z\. Wang, H\. Zhang, B\. Guan, F\. Lu, Z\. Zhang, Y\. Sui, H\. Jin, and X\. Shi\(2024\)Iterative refinement of project\-level code context for precise code generation with compiler feedback\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 2336–2353\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.138)Cited by:[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[4\]L\. Breiman, J\. H\. Friedman, R\. A\. Olshen, and C\. J\. Stone\(1984\)Classification and regression trees\.Wadsworth,Belmont, CA\.Cited by:[6th item](https://arxiv.org/html/2608.14653#S3.I1.i6.p1.1)\.
- \[5\]J\. Cao, Z\. Chen, J\. Wu, S\. Cheung, and C\. Xu\(2024\)JavaBench: a benchmark of object\-oriented code generation for evaluating large language models\.InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering,pp\. 870–882\.External Links:[Document](https://dx.doi.org/10.1145/3691620.3695470),[Link](https://doi.org/10.1145/3691620.3695470)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p1.1),[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p4.1)\.
- \[6\]B\. Chen, F\. Zhang, A\. Nguyen, D\. Zan, Z\. Lin, J\. Lou, and W\. Chen\(2023\)CodeT: code generation with generated tests\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ktrw68Cmu9c)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p1.1)\.
- \[7\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p1.1),[§II](https://arxiv.org/html/2608.14653#S2.p2.1),[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p2.1),[§V\-F](https://arxiv.org/html/2608.14653#S5.SS6.p1.5)\.
- \[8\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2024\)Teaching large language models to self\-debug\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 8746–8825\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/2460396f2d0d421885997dd1612ac56b-Paper-Conference.pdf)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1)\.
- \[9\]Y\. Dong, X\. Jiang, H\. Liu, Z\. Jin, B\. Gu, M\. Yang, and G\. Li\(2024\)Generalization or memorization: data contamination and trustworthy evaluation for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 12039–12050\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.716),[Link](https://aclanthology.org/2024.findings-acl.716/)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p5.1),[§VII](https://arxiv.org/html/2608.14653#S7.p1.1)\.
- \[10\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal\(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[11\]Y\. Fu, B\. Li, L\. Li, W\. Zhang, and T\. Xie\(2025\)The first prompt counts the most\! an evaluation of large language models on iterative example\-based code generation\.Proceedings of the ACM on Software Engineering2\(ISSTA\),pp\. 1583–1606\.External Links:[Document](https://dx.doi.org/10.1145/3728947),[Link](https://doi.org/10.1145/3728947)Cited by:[§V\-B](https://arxiv.org/html/2608.14653#S5.SS2.p1.1)\.
- \[12\]Y\. Gal and Z\. Ghahramani\(2016\)Dropout as a bayesian approximation: representing model uncertainty in deep learning\.InProceedings of the 33rd International Conference on Machine Learning,pp\. 1050–1059\.External Links:[Link](https://proceedings.mlr.press/v48/gal16.html)Cited by:[1st item](https://arxiv.org/html/2608.14653#S3.I2.i1.p1.1),[§III\-B](https://arxiv.org/html/2608.14653#S3.SS2.p3.1),[§III\-C](https://arxiv.org/html/2608.14653#S3.SS3.p2.1),[§III\-C](https://arxiv.org/html/2608.14653#S3.SS3.p3.1),[§III](https://arxiv.org/html/2608.14653#S3.p1.1)\.
- \[13\]Y\. Gal\(2016\)Uncertainty in deep learning\.Ph\.D\. Thesis,University of Cambridge\.External Links:[Link](https://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/thesis.pdf)Cited by:[1st item](https://arxiv.org/html/2608.14653#S3.I2.i1.p1.1),[2nd item](https://arxiv.org/html/2608.14653#S3.I2.i2.p1.3),[§III\-C](https://arxiv.org/html/2608.14653#S3.SS3.p2.1)\.
- \[14\]L\. Gröninger, B\. Souza, and M\. Pradel\(2025\)ChangeGuard: validating code changes via pairwise learning\-guided execution\.Proceedings of the ACM on Software Engineering2\(FSE\),pp\. 936–956\.External Links:[Document](https://dx.doi.org/10.1145/3715760)Cited by:[§V\-C](https://arxiv.org/html/2608.14653#S5.SS3.p3.1)\.
- \[15\]J\. He, M\. Vero, G\. Krasnopolska, and M\. Vechev\(2024\)Instruction tuning for secure code generation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 18043–18062\.External Links:[Link](https://proceedings.mlr.press/v235/he24k.html)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p7.1)\.
- \[16\]D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. Steinhardt\(2021\)Measuring coding challenge competence with apps\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Vol\.34\.External Links:[Link](https://openreview.net/forum?id=sD93GOzH3i5)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p6.1),[§V\-F](https://arxiv.org/html/2608.14653#S5.SS6.p1.5)\.
- \[17\]X\. Jiang, Y\. Dong, Y\. Tao, H\. Liu, Z\. Jin, and G\. Li\(2025\)ROCODE: integrating backtracking mechanism and program analysis in large language models for code generation\.InProceedings of the IEEE/ACM 47th International Conference on Software Engineering,pp\. 334–346\.External Links:[Document](https://dx.doi.org/10.1109/ICSE55347.2025.00133),[Link](https://doi.org/10.1109/ICSE55347.2025.00133)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8),[§IV](https://arxiv.org/html/2608.14653#S4.p2.1)\.
- \[18\]X\. Jiang, Y\. Dong, L\. Wang, Z\. Fang, Q\. Shang, G\. Li, Z\. Jin, and W\. Jiao\(2024\)Self\-planning code generation with large language models\.ACM Transactions on Software Engineering and Methodology33\(7\),pp\. 182:1–182:30\.External Links:[Document](https://dx.doi.org/10.1145/3672456),[Link](https://doi.org/10.1145/3672456)Cited by:[§V\-B](https://arxiv.org/html/2608.14653#S5.SS2.p1.1)\.
- \[19\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=VD-AYtP0dve)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[20\]H\. Le, H\. Chen, A\. Saha, A\. Gokul, D\. Sahoo, and S\. Joty\(2024\)CodeChain: towards modular code generation through chain of self\-revisions with representative sub\-modules\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1)\.
- \[21\]H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\. H\. Hoi\(2022\)CodeRL: mastering code generation through pretrained models and deep reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 21314–21328\.Cited by:[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[22\]D\. D\. Lewis and W\. A\. Gale\(1994\)A sequential algorithm for training text classifiers\.InProceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3–12\.External Links:[Document](https://dx.doi.org/10.1007/978-1-4471-2099-5%5F1)Cited by:[3rd item](https://arxiv.org/html/2608.14653#S3.I1.i3.p1.1),[4th item](https://arxiv.org/html/2608.14653#S3.I1.i4.p1.1)\.
- \[23\]Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. D\. Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. S\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. Vinyals\(2022\)Competition\-level code generation with alphacode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Document](https://dx.doi.org/10.1126/science.abq1158),[Link](https://doi.org/10.1126/science.abq1158)Cited by:[§II](https://arxiv.org/html/2608.14653#S2.p3.8),[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p5.1)\.
- \[24\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p1.1),[§VII](https://arxiv.org/html/2608.14653#S7.p1.1)\.
- \[25\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark\(2023\)SELF\-refine: iterative refinement with self\-feedback\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.External Links:[Link](https://dl.acm.org/doi/10.5555/3666122.3668141)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1)\.
- \[26\]A\. Ni, S\. Iyer, D\. Radev, V\. Stoyanov, W\. Yih, S\. I\. Wang, and X\. V\. Lin\(2023\)LEVER: learning to verify language\-to\-code generation with execution\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.202,pp\. 26106–26128\.External Links:[Link](https://proceedings.mlr.press/v202/ni23b.html)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[27\]A\. V\. Nikitin, J\. Kossen, Y\. Gal, and P\. Marttinen\(2024\)Kernel language entropy: fine\-grained uncertainty quantification for llms from semantic similarities\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-0283)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[28\]T\. X\. Olausson, J\. P\. Inala, C\. Wang, J\. Gao, and A\. Solar\-Lezama\(2024\)Is self\-repair a silver bullet for code generation?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=y0GJXRungR)Cited by:[§II](https://arxiv.org/html/2608.14653#S2.p2.1)\.
- \[29\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala\(2019\)PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://arxiv.org/abs/1912.01703)Cited by:[§V\-G](https://arxiv.org/html/2608.14653#S5.SS7.p1.1)\.
- \[30\]B\. Settles\(2009\)Active learning literature survey\.Technical reportTechnical Report1648,University of Wisconsin–Madison\.External Links:[Link](https://burrsettles.com/pub/settles.activelearning.pdf)Cited by:[2nd item](https://arxiv.org/html/2608.14653#S3.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2608.14653#S3.I1.i3.p1.1),[4th item](https://arxiv.org/html/2608.14653#S3.I1.i4.p1.1),[5th item](https://arxiv.org/html/2608.14653#S3.I1.i5.p1.3),[§III\-C](https://arxiv.org/html/2608.14653#S3.SS3.p3.1),[§III](https://arxiv.org/html/2608.14653#S3.p1.1)\.
- \[31\]H\. S\. Seung, M\. Opper, and H\. Sompolinsky\(1992\)Query by committee\.InProceedings of the Fifth Annual Workshop on Computational Learning Theory,pp\. 287–294\.External Links:[Document](https://dx.doi.org/10.1145/130385.130417)Cited by:[2nd item](https://arxiv.org/html/2608.14653#S3.I2.i2.p1.3),[§III](https://arxiv.org/html/2608.14653#S3.p1.1)\.
- \[32\]C\. E\. Shannon\(1948\)A mathematical theory of communication\.Bell System Technical Journal27\(3\),pp\. 379–423\.External Links:[Document](https://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x)Cited by:[1st item](https://arxiv.org/html/2608.14653#S3.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2608.14653#S3.I1.i2.p1.1)\.
- \[33\]Z\. Wang, Z\. Zhou, D\. Song, Y\. Huang, S\. Chen, L\. Ma, and T\. Zhang\(2025\)Towards understanding the characteristics of code generation errors made by large language models\.InProceedings of the 47th IEEE/ACM International Conference on Software Engineering,pp\. 2587–2599\.External Links:[Document](https://dx.doi.org/10.1109/ICSE55347.2025.00180)Cited by:[§V\-D](https://arxiv.org/html/2608.14653#S5.SS4.p1.2)\.
- \[34\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6),[Link](https://aclanthology.org/2020.emnlp-demos.6/)Cited by:[§V\-G](https://arxiv.org/html/2608.14653#S5.SS7.p1.1)\.
- \[35\]H\. Zhang, H\. Chen, M\. Chen, and T\. Zhang\(2025\)Active layer\-contrastive decoding reduces hallucination in large language model generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 3028–3046\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.150),[Link](https://aclanthology.org/2025.emnlp-main.150/)Cited by:[§V\-B](https://arxiv.org/html/2608.14653#S5.SS2.p1.1)\.
- \[36\]S\. Zhang, Z\. Chen, Y\. Shen, M\. Ding, J\. B\. Tenenbaum, and C\. Gan\(2023\)Planning with large language models for code generation\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Lr8cOOtYbfL)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p2.1),[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[37\]J\. Zhao, Y\. Sun, C\. Huang, C\. Liu, Y\. Guan, Y\. Zeng, and Y\. Liu\(2025\)Towards secure code generation with llms: a study on common weakness enumeration\.IEEE Transactions on Software Engineering51\(12\),pp\. 3507–3523\.External Links:[Document](https://dx.doi.org/10.1109/TSE.2025.3619281),[Link](https://doi.org/10.1109/TSE.2025.3619281)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p7.1)\.
- \[38\]Q\. Zheng, X\. Xia, X\. Zou, Y\. Dong, S\. Wang, Y\. Xue, Z\. Wang, L\. Shen, A\. Wang, Y\. Li, T\. Su, Z\. Yang, and J\. Tang\(2023\)CodeGeeX: a pre\-trained model for code generation with multilingual benchmarking on humaneval\-x\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5673–5684\.External Links:[Document](https://dx.doi.org/10.1145/3580305.3599790),[Link](https://doi.org/10.1145/3580305.3599790)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p4.1)\.
- \[39\]T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. Yue\(2024\)OpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12834–12859\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[§II](https://arxiv.org/html/2608.14653#S2.p3.8)\.
- \[40\]Z\. Zheng, K\. Ning, Y\. Wang, J\. Zhang, D\. Zheng, M\. Ye, and J\. Chen\(2023\)A survey of large language models for code: evolution, benchmarking, and future trends\.CoRRabs/2311\.10372\.External Links:[Link](https://arxiv.org/abs/2311.10372)Cited by:[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p1.1)\.
- \[41\]T\. Y\. Zhuo, C\. Vu, J\. Chim, H\. Hu, H\. Zhan, W\. Yu, N\. Muennighoff, J\. Kaddour, W\. Li, J\. He, M\. Xu, Z\. Zhang, R\. Widyasari, I\. Paul, S\. Brunner, I\. N\. B\. Yusuf, T\. Hoang, C\. Gong, A\. Zebaze, and P\. Yadav\(2025\)BigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=YrycTjllL0)Cited by:[§I](https://arxiv.org/html/2608.14653#S1.p1.1),[§V\-A](https://arxiv.org/html/2608.14653#S5.SS1.p5.1),[§VII](https://arxiv.org/html/2608.14653#S7.p1.1)\.