Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models

# Efficient Adaptive-Length Infillingfor Diffusion Language Models
Source: [https://arxiv.org/html/2609.02108](https://arxiv.org/html/2609.02108)
## Predict, Don’t Iterate: Efficient Adaptive\-Length Infilling for Diffusion Language Models

Sirui ChenYuanchen BeiLingjie ChenYuchen YanDongqi Fu, Jingrui He, Hanghang TongAffiliation:University of Illinois at Urbana\-Champaign Meta

###### Abstract

Diffusion language models \(DLMs\) have emerged as a promising alternative to the auto\-regressive paradigm\. With bidirectional attention and any\-order generation, DLMs naturally fit infilling tasks, which require generating a middle span conditioned on both the prefix and the suffix\. However, infilling is sensitive to the length of the span, while DLMs require the length to be fixed before generation\. Although prior studies extend DLMs to dynamic lengths, they still suffer from two limitations\. \(i\)Sensitivity to initial length\. These methods require a preset length to initialize the search and are highly sensitive to this initial length, often yielding suboptimal results\. \(ii\)Inference inefficiency\. They either insert length\-changing operations during generation or repeatedly search for an appropriate length using multi\-step denoising confidence, both of which introduce substantial extra forward passes and computational cost\. Therefore, we proposePill\(Probing\-basedInfiLling with preset\-Length\-free decoding\), an efficient infilling method for DLMs that requires no preset initial length and adds far fewer extra forward passes than baselines, substantially reducing inference time\. Experiments show that, across five DLMs spanning different families, architectures, and training recipes on eight infilling benchmarks,Pillimproves over the strongest baseline by\+4\.8\+4\.8average pass rate on code and\+6\.0\+6\.0BLEU\-2 on text, while running1\.82×1\.82\\timesfaster than that baseline\. The code is available at[https://github\.com/Hsu1023/PILL](https://github.com/Hsu1023/PILL)\.

## 1Introduction

Recently, diffusion language models \(DLMs\) have emerged as a promising alternative to auto\-regressive \(AR\) paradigm, showing great potential across various tasks\([Nie et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib1);[Zhu et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib2);[Zhu et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib18);[Ye et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib3)\)\. Unlike AR models that rely on causal attention and decode strictly left to right, DLMs employ bidirectional attention and generate in any order, allowing them to condition on both the left and right context simultaneously\([Hoogeboom et al\., 2021](https://arxiv.org/html/2609.02108#bib.bib26)\)\. This property makes DLMs particularly well suited to infilling, generating a middle span that is coherent with both a given prefix and a given suffix, a setting that AR models are hard to handle\([Bavarian et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib6)\)\.

However, DLMs require the number of masked positions to be fixed before decoding, whereas the correct infilling length is unknown in advance and varies from case to case\([Bie et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib19)\)\. Crucially, infilling quality is highly sensitive to the chosen length: as shown in Figure[1](https://arxiv.org/html/2609.02108#S1.F1)\(a\), the code\-infilling example is solved correctly only when the mask span is given exactly the right number of tokens, while slightly shorter or longer spans produce incorrect completions\. The model must therefore determine a length before it knows what to generate, even though the appropriate length depends on the content itself\. Recent studies tackle this through adaptive\-length decoding\([Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10);[Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12);[Kim et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib25);[Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\), but they still suffer from two limitations\.

![Refer to caption](https://arxiv.org/html/2609.02108v1/teaser2.png)Figure 1:Motivation and Performance ofPill\.\(a\) For DLM infilling tasks, correctness is sensitive to the span lengths\. \(b\) Baselines must preset an initial lengthLLand their final length stays tied to it, whereasPillrequires no preset length\. \(c\)Pillattains the best performance with minimal overhead compared to fixed\-length generation with backbones\. Baselines are reported at initial lengthL=\{4,8,16,32\}L=\\\{4,8,16,32\\\}\.First, existing methods aresensitive to the initial length\. They require a preset length to start the search for a better infilling length\. For example, CAL\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)will be trapped into local minima near the initial length, yielding varying performances, as shown in Fig\.[1](https://arxiv.org/html/2609.02108#S1.F1)\(b\)\. Second, they incursubstantial inference overhead\. Methods either inject explicit one token length\-changing operations\([Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12);[Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10)\), or repeatedly probe denoising confidence over many decoding steps to estimate a length\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\), introducing many extra forward passes and cost\.

To this end, we proposePill, a framework that enables fixed\-length DLMs to perform variable\-length infilling with only two extra forward passes\.Pillconsists of three stages\. \(1\) Instead of searching for a length, we*directly predict*it from the bidirectional hidden state of a single mask token inserted between the prefix and suffix\. \(2\) Since generation is sensitive to length, we expand this estimate into a small set of candidate lengths and decode them*in parallel*at a cost close to decoding one\. \(3\) Since DLMs cannot score a generated sequence by next\-token likelihood, we*select*the final span with a post\-hoc score that favors candidates coherent both internally and with the suffix, using one additional forward pass\. In summary, it is more efficient than baselines: itreplaces their iterative length search, which spends a forward pass per step, with just two extra passes, while the multi\-slot decoding generates all candidates in parallel at almost no extra cost\. As shown in Figure[1](https://arxiv.org/html/2609.02108#S1.F1)\(c\), it achieves the highest quality while adding<10%<10\\%wall\-clock time over vanilla fixed\-length decoding, whereas baselines pay far more for a smaller gain\. Our contributions are three\-fold:

- •A preset\-free infilling framework\. We proposePill, a backbone\-frozen framework that enables DLMs to perform adaptive\-length infilling beyond the fixed lengths with only two extra forward passes, centered on a length probe that predicts target lengths directly from a mask token’s bidirectional hidden state\.
- •Parallel decoding and selection\.We decode a set of length candidates in parallel at near single\-candidate cost via a slot\-wise attention mask, and select the final span with a post\-hoc coherence score in one pass\.
- •Extensive validation\.Across five DLMs and eight code and text benchmarks,Pillsurpasses the strongest baselines by\+4\.8\+4\.8pass rate on code and\+6\.0\+6\.0BLEU\-2 on text while running1\.82×1\.82\\timesfaster\.

## 2Related Work

Diffusion Language Models\.DLMs generate text by iteratively denoising masked or corrupted sequences\. The idea originates from early works such as D3PM\([Austin et al\., 2021a](https://arxiv.org/html/2609.02108#bib.bib13)\)and Diffusion\-LM\([Li et al\., 2022a](https://arxiv.org/html/2609.02108#bib.bib14)\), and is further developed by Masked Diffusion Language Models \(MDLMs\), which improve the modeling objective, sampling, and scalability\([Shi et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib16);[Sahoo et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib15)\)\. Recent models scale DLMs to larger parameters, including research previews such as Gemini\-Diffusion and Seed Diffusion\([Song et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib17)\)and open\-sourced models such as LLaDA\([Nie et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib1)\)and Dream\([Ye et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib3)\), reaching competitive performance across tasks including code generation, instruction following, and reasoning\([Xie et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib4);[Bie et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib19)\)\. Owing to their bidirectional attention, DLMs can condition on both left and right context, which makes them naturally suited to infilling\([Li et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib21)\)\.

Infilling and Adaptive\-Length Decoding\.Infilling has been extensively studied in the AR paradigm, including FIM\-style training\([Bavarian et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib6)\)and code\-specialized models\([Fried et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib22);[Li et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib23);[Roziere et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib24)\), all of which decode the missing span left\-to\-right via causal attention\. While effective, AR models can only access the suffix indirectly through prompt conditioning, motivating bidirectional alternatives such as DLMs that natively condition on both prefix and suffix\([Ye et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib3)\)\. However, DLMs typically require the number of masked positions to be specified before decoding, restricting them to fixed\-length generation\([Lin et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib40)\)\. Recent studies address this by adaptive length decoding\. DAEDAL\([Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10)\)dynamically expands the generation canvas during inference, but only focuses on appending at the end of sequences\. DreamOn\([Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12)\)introduces explicit length\-changing operations into the diffusion process, but requires fine\-tuning the DLM, and potentially compromises general\-purpose capabilities\. FlexMDM\([Kim et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib25)\)and DDOT\([Zhang et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib33)\)also introduces additional fine\-tuning cost\. CAL\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)searches for suitable lengths using early\-step denoising confidence, but incurs extra forward passes and is highly sensitive to the initial length\. In contrast, our approach uses a lightweight probe to directly predict the target length, and a multi\-slot decoding scheme that generates and selects among multiple length candidates in parallel\. This yields adaptive\-length infilling with minimal additional cost and no reliance on a preset initial length\.

## 3Preliminaries

Diffusion Language Models\.Letx=\(x1,…CLOSE,x=\(x\_\{1\},\\dots,OPENxM\)x\_\{M\}\)be a token sequence over vocabulary𝒱\\mathcal\{V\}, augmented with a mask token\[𝙼\]\\mathtt\{\[M\]\}\. DLMs define a forward process that independently replaces eachxix\_\{i\}with\[𝙼\]\\mathtt\{\[M\]\}with probabilityt∈\[0,1\]t\\in\[0,1\], yielding a corrupted sequencex\(t\)x^\{\(t\)\}\. A networkpθ​\(x\(0\)∣x\(t\)\)p\_\{\\theta\}\(x^\{\(0\)\}\\mid x^\{\(t\)\}\)is trained to reconstruct the original tokens at masked positions\([Sahoo et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib15);[Shi et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib16)\)\. Unlike auto\-regressive models, DLMs use bidirectional attention, so every prediction conditions on all other positions on both sides\. At inference, given a prefixPPand a target lengthLL, decoding starts fromx\(K\)=P⊕\[𝙼\]Lx^\{\(K\)\}=P\\oplus\\mathtt\{\[M\]\}^\{L\}, where⊕\\oplusdenotes concatenation, and proceeds forKKsteps\. At each stepk=K,…,1k=K,\\dots,1, the model produces a distribution over𝒱\\mathcal\{V\}at every masked position; a subset of positions with the highest confidence are unmasked, while the rest remain masked for stepk−1k\{\-\}1\. The finalx\(0\)x^\{\(0\)\}contains no masks\. A central limitation is thatLLmust be specified*before*decoding and remains fixed throughout\.

Infilling Tasks for DLMs\.The inference setting above fills a masked region given only a prefixPP\. Infilling extends this setting by additionally fixing a suffixSSas a right\-side condition, which DLMs can exploit directly thanks to their bidirectional attention and any\-order generation\. Given a prefixPPand a suffixSS, the infilling task asks for a middle spanM=\(w1,…,wL∗\)M=\(w\_\{1\},\\dots,w\_\{L^\{\*\}\}\)withwi∈𝒱w\_\{i\}\\in\\mathcal\{V\}, such that the concatenationP⊕M⊕SP\\oplus M\\oplus Smaximizes a task\-specific quality metric𝒬\\mathcal\{Q\}\(e\.g\., pass rate for code, BLEU\-2 for text\)\. The gold lengthL∗L^\{\*\}is unknown at inference time, and𝒬\\mathcal\{Q\}is inaccessible during decoding\. DLMs instantiate the masked region asP⊕\[𝙼\]L0⊕SP\\oplus\\mathtt\{\[M\]\}^\{L\_\{0\}\}\\oplus Sat a preset lengthL0L\_\{0\}and run iterative masked decoding to fill it\.

## 4Method

![Refer to caption](https://arxiv.org/html/2609.02108v1/method.png)Figure 2:Pipeline of the proposed method\.\(a\) Stage I probes the hidden state of a mask token to predict the target lengthL^\\hat\{L\}; \(b\) Stage II decodes the candidate lengths\{L^−r,…,L^\+r\}\\\{\\hat\{L\}\-r,\\dots,\\hat\{L\}\+r\\\}in parallel with a slot\-wise attention mask; \(c\) Stage III selects the final span with post\-hoc score in one forward pass via a well\-designed attention mask\.### 4\.1Overview

We cast the infilling task as two coupled subproblems: \(1\) determining the length of the mask span and \(2\) generating content coherent with both the prefix and the suffix\. The two subproblems are entangled, since the appropriate length depends on what to generate, yet DLMs can only generate with fixed lengths\.Pilldecouples them through a three\-stage design shown in Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\. \(1\) Stage I predicts a target lengthL^\\hat\{L\}directly from hidden states of mask token through a single forward pass with a lightweight MLP probe, avoiding iterative length search\. Since the generation is sensitive to lengths, we expandL^\\hat\{L\}into a small candidate set\{L^−r,⋯,L^\+r\}\\\{\\hat\{L\}\-r,\\cdots,\\hat\{L\}\+r\\\}to enhance predictive reliability\. \(2\) Stage II generates all candidate lengths in parallel at nearly the cost of decoding one, where each candidate length occupies a separate decoding region \(a slot\) and a slot\-wise attention mask isolates the slots so they share prefix/suffix context but do not leak into one another, accelerating generation\. \(3\) Stage III selects the final span from candidates via a post\-hoc score that jointly measures each candidate’s internal coherence and how naturally the suffix continues from it in a single additional forward pass, equipped with a well\-designed attention mask mechanism\. Throughout the pipeline, we only introduce two passes, the probing pass and the scoring pass\. Also, by predicting the length directly rather than searching from a preset value,Pillrequires no initial lengths, yielding consistent performance across datasets\.

### 4\.2Length Probing

The first stage estimates the length of the missing span directly from hidden states, as shown in Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(a\)\. Given a prefixPPand suffixSS, we construct aprobing sequenceby inserting a single mask token between them, denoted asxprobe=P⊕\[M\]⊕Sx\_\{\\mathrm\{probe\}\}=P\\oplus\\texttt\{\[M\]\}\\oplus S, and run a single forward pass of the frozen DLM backbone\. We read the hidden stateh\[𝙼\]∈ℝdh\_\{\\mathtt\{\[M\]\}\}\\in\\mathbb\{R\}^\{d\}of the mask position at a certain layer\. Because the backbone uses bidirectional attention,h\[𝙼\]h\_\{\\mathtt\{\[M\]\}\}aggregates information from both prefix and suffix, making it a natural summary of how much content the missing span is expected to hold\. Rather than using only the mask hidden state, we form the probe representation by concatenating the mean\-pooled hidden states of the last four prefix tokens, the mask hidden state, and the mean\-pooled hidden states of the first four suffix tokensh=\[MeanPool\(hp−4:p−1\);h\[M\];MeanPool\(hs:s\+3\)\]∈ℝ3​dh=\[\\mathrm\{MeanPool\}\(h\_\{p\-4:p\-1\}\);h\_\{\\texttt\{\[M\]\}\};\\mathrm\{MeanPool\}\(h\_\{s:s\+3\}\)\]\\in\\mathbb\{R\}^\{3d\}\. This provides a richer representation than the single confidence scores used by DAEDAL\([Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10)\)and CAL\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)\. Then, a lightweight 3\-layer MLP probefϕ​\(⋅\)f\_\{\\phi\}\(\\cdot\)mapshhto a scalar length estimate

L^=⌊exp⁡\(fϕ​\(h\)\)⌉,\\hat\{L\}=\\big\\lfloor\\exp\\big\(f\_\{\\phi\}\(h\)\\big\)\\big\\rceil,\(1\)where⌊⋅⌉\\lfloor\\cdot\\rceilrounds to the nearest positive integer\. We predictlog⁡L\\log Lrather thanLLdirectly, which guarantees a positive estimate and reduces the influence of long\-tailed length distribution\. More information about the probe can be found in Appendix[A\.1](https://arxiv.org/html/2609.02108#A1.SS1)\. The lightweight probe does not touch the backbone, preserving the DLM ability without finetuning the backbone\. Also, we only introduce one single forward pass in this stage to avoid presetting an initial length, with minimal overhead\. We also validate the probing by comparing predicted lengths against the oracle, indicating that our probe predicts lengths more accurately than the baseline while requiring no preset initial length \(Sec\.[5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px1)\)\.

### 4\.3Multi\-slot Parallel Decoding

Since the DLM generation is highly sensitive to the length of the mask slot, we expand the lengthL^\\hat\{L\}obtained by the probe within a local radiusrrinto\{L^−r,⋯,L^\+r\}\\\{\\hat\{L\}\-r,\\cdots,\\hat\{L\}\+r\\\}to better reduce prediction errors, yieldingN=2​r\+1N=2r\+1candidates, and assign thejj\-th candidate lengthℓj\\ell\_\{j\}to a separate decoding slot, which is a contiguous block of mask tokens that is decoded independently\. But it yields much more decoding cost, therefore we propose amulti\-slot parallel decodingstrategy at a cost close to decoding one slot alone \(Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(b\)\)\.

Sequence Layout\.Since all the candidates share the prefix and suffix, we pack the candidate slots together with a shared context region,P⊕\[𝙼\]L^⊕S⊕\[S1\]⋯\[SN\]P\\oplus\\mathtt\{\[M\]\}^\{\\hat\{L\}\}\\oplus S\\oplus\[S\_\{1\}\]\\cdots\[S\_\{N\}\], where each slotSjS\_\{j\}is initialized withℓj\\ell\_\{j\}mask tokens\. The prefix, middle mask slot and suffix form a shared context, and they attend to each other and are attended by every slotSjS\_\{j\}\. In addition, to avoid interference between mask slotsSjS\_\{j\}, we isolate the slots through a slot\-wise attention mask\. Each slotSjS\_\{j\}attends only to its own tokens and the shared context\{P,\[𝙼\]L^,S\}\\\{P,\\mathtt\{\[M\]\}^\{\\hat\{L\}\},S\\\}, but not to any other mask slotSj′​\(j′≠j\)S\_\{j^\{\\prime\}\}\(j^\{\\prime\}\\neq j\)\. Because each slot sees nearly the same context as in single\-slot generation, its denoising behavior, and thus its output, closely matches that of decoding the slot alone\. This holds only when each slot is given appropriate position ids, which we address below\.

Position IDs\.A subtlety is how to assign position ids to slots of different lengths\. The key principle is that a DLM places each predicted token at the position dictated by its position id\. If the mask span is given more position ids than the content needs, the model treats the extra positions as space still to be filled and under\-generates\. We illustrate this with a concrete example, in which\[TODO\]denotes the span to be filled\.

have 13 books, and you have 23 books\.Wehave\(17\)\[TODO\] books\.The numbers below the text are the position ids assigned by LLaDA\. The intended answer is "36" \(two tokens\)\. When the gap is given exactly two position ids, the model correctly produces "36"\. But when we keep a single mask token yet leave a one\-position gap between it and one side of the context, the model fills only half the answer: leaving the gap on the suffix side makes the mask take position id "\(18\)" while the suffix resumes at "\(20\)", and the model outputs only "3" \(the first token of "36"\); leaving the gap on the prefix side instead yields "6" \(the second token\)\. In each case the model anchors its output to the position ids and leaves the gap unfilled, i\.e\., it under\-generates\. It motivates a no\-gap mask positional id design\. The mask slot should start at the position id immediately after the prefix and end immediately before the suffix, leaving no gaps from them\.

We therefore fix the suffix anchor and*interpolate*each slot’s position ids over a shared gap interval, so that all slots present an identical layout to the suffix regardless of their length\. Let the last prefix token have position idpPp\_\{P\}\. We fix the suffix to start atpS=pP\+L^\+1p\_\{S\}=p\_\{P\}\+\\hat\{L\}\+1, leaving a gap interval\[pP\+1,pS−1\]\[\\,p\_\{P\}\+1,\\ p\_\{S\}\-1\\,\]of widthL^\\hat\{L\}\. For a slotSjS\_\{j\}of lengthℓj\\ell\_\{j\}, itskk\-th token \(k=1,…,ℓjk=1,\\dots,\\ell\_\{j\}\) is assigned

pos⁡\(Sj,k\)=\(pP\+1\)\+\(k−1\)⋅L^−1ℓj−1,\\vskip\-5\.0pt\\mathrm\{pos\}\(S\_\{j\},k\)=\(p\_\{P\}\+1\)\+\(k\-1\)\\cdot\\frac\{\\hat\{L\}\-1\}\{\\ell\_\{j\}\-1\},\(2\)linearly interpolating theℓj\\ell\_\{j\}tokens betweenpP\+1p\_\{P\}\+1andpS−1p\_\{S\}\-1\. Every slot thus spans the same interval and the suffix is anchored atpSp\_\{S\}for all candidates, eliminating the length\-dependent misalignment above\. This design is critical: padding\-based alternatives that left\- or right\-align the slot leave a position\-id gap between the candidate and one side of the context, which leads to under\-generation by the mechanism above\. In contrast, interpolation removes this gap and substantially outperforms both padding variants \(Sec[5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px2)\)\.

Parallel Decoding\.Within each slot, we run the original denoising/unmasking process\. Since the mask slots do not interfere with each other and can be denoised in parallel, we obtainNNinfill candidates\{C1,⋯,CN\}\\\{C\_\{1\},\\cdots,C\_\{N\}\\\}in almost the same time as decoding a single slot\. This also saves KV\-cache computation, since the shared context\{P,\[𝙼\]L^,S\}\\\{P,\\mathtt\{\[M\]\}^\{\\hat\{L\}\},S\\\}remains unchanged throughout decoding\. The middle\[𝙼\]L^\\mathtt\{\[M\]\}^\{\\hat\{L\}\}produces no output, and it merely anchors the shared position\-id interval, and its KV is identical across all slots and steps\. This motivates placing the shared context first in the sequence layout \(P⊕\[𝙼\]L^⊕S⊕\[S1\]⋯\[SN\]P\\oplus\\mathtt\{\[M\]\}^\{\\hat\{L\}\}\\oplus S\\oplus\[S\_\{1\}\]\\cdots\[S\_\{N\}\]\), so that itsKV can be cached and reusedacross decoding steps\. Please see Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(c\) for an illustration\.

### 4\.4Post\-hoc Selection

Stage II returnsNNinfill candidates\{C1,…,CN\}\\\{C\_\{1\},\\dots,C\_\{N\}\\\}of different lengths, and we must commit to one\. AR models can score a sequence by its left\-to\-right next\-token likelihood\. In contrast, DLMs are trained to predict the tokenata masked position rather than thenexttoken, so the distribution at an already\-committed token is not a predictive distribution and cannot serve as a score \(Fig\.[3](https://arxiv.org/html/2609.02108#S4.F3)\)\.

![Refer to caption](https://arxiv.org/html/2609.02108v1/comparison.png)Figure 3:Forward\-pass targets in ARMs and DLMs\.Our goal is to select the candidate that makes thecompletedsequenceP⊕Cj⊕SP\\oplus C\_\{j\}\\oplus Smost coherent as a whole, rather than one that looks good in isolation\. A natural score is the pseudo\-log\-likelihood, which masks each token in turn and reads its log\-likelihood from the rest; however, this requires one forward pass per token and is prohibitively expensive\. Instead, we fix the prefix as a shared condition across all candidates and score the inner part for internal coherence and the suffix for how naturally it continues from the candidate, all in a single forward pass\.

Masked Scoring Probes\.For candidateCjC\_\{j\}we lay out a visible block𝒗=Cj⊕S~=\(v1,…,vℓj\+m\)\\boldsymbol\{v\}=C\_\{j\}\\oplus\\tilde\{S\}=\(v\_\{1\},\\dots,v\_\{\\ell\_\{j\}\+m\}\), theℓj\\ell\_\{j\}committed candidate tokens followed by themmselected suffix tokensS~\\tilde\{S\}\-a small, representative subset of the suffix used to judge how well it continues from the candidate \(introduced later\)\- and a block of*scoring probes*,𝝁=\(μ1,…,μℓj\+m\)\\boldsymbol\{\\mu\}=\(\\mu\_\{1\},\\dots,\\mu\_\{\\ell\_\{j\}\+m\}\), where eachμk\\mu\_\{k\}is a mask token and shares the position id ofvkv\_\{k\}\. We then carefully design an attention mask \(Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(c\)\), enforcing a strictly causal read\. The prefixPPis visible to the entire block\. \(1\) Each*visible*tokenvkv\_\{k\}attends toPPand tov≤kv\_\{\\leq k\}\(inclusive\), so the visible block stays internally causal and no visible representation contains information of a later token\. \(2\) Each*probe*μk\\mu\_\{k\}attends toPPand tov<kv\_\{<k\}only \(strict\), never tovkv\_\{k\}itself, to any later position, or to another probe, to prevent it from accessingv≥kv\_\{\\geq k\}\. Consequently the output atμk\\mu\_\{k\}ispθ\(⋅∣P,v<k\)p\_\{\\theta\}\(\\cdot\\mid P,v\_\{<k\}\), and reading it at the observed token giveslog⁡pθ​\(vk∣P,v<k\)\\log p\_\{\\theta\}\(v\_\{k\}\\mid P,v\_\{<k\}\)with no label leakage; the first probe conditions onPPalone\. The inclusive\-causal visible block is essential: were it bidirectional,v<kv\_\{<k\}would already encodevkv\_\{k\}and leak the token the probe is meant to predict\. Since the suffix tokens follow the full candidate invv, their probes automatically condition on all ofCjC\_\{j\}, exactly the quantity needed to judge continuation\.

Selection Score\.Letckc\_\{k\}ands~k\\tilde\{s\}\_\{k\}be thekk\-th token inCjC\_\{j\}andS~j\\tilde\{S\}\_\{j\}\. Partitioning the per\-position log\-probs ofvvinto its candidate and suffix segments gives an inner score and a suffix score,

sjin=1ℓj​∑k=1ℓjlog⁡pθ​\(ck∣P,c<k\),\\displaystyle s^\{\\text\{in\}\}\_\{j\}=\\frac\{1\}\{\\ell\_\{j\}\}\\sum\_\{k=1\}^\{\\ell\_\{j\}\}\\log p\_\{\\theta\}\\\!\\left\(c\_\{k\}\\mid P,c\_\{<k\}\\right\),sjsuf=1m​∑t=1mlog⁡pθ​\(s~t∣P,Cj,s<t\),\\displaystyle s^\{\\text\{suf\}\}\_\{j\}=\\frac\{1\}\{m\}\\sum\_\{t=1\}^\{m\}\\log p\_\{\\theta\}\\\!\\left\(\\tilde\{s\}\_\{t\}\\mid P,C\_\{j\},\{s\}\_\{<t\}\\right\),and we select the candidate maximizing their convex combination,

j⋆=arg⁡maxj​α​sjin\+\(1−α\)​sjsuf\.j^\{\\star\}=\\arg\\max\_\{j\}\\;\\alpha\\,s^\{\\text\{in\}\}\_\{j\}\+\(1\-\\alpha\)\\,s^\{\\text\{suf\}\}\_\{j\}\.\(3\)The inner score favors spans the model finds internally coherent, while the suffix score favors spans after which the observed suffix becomes likely, penalizing over\- and under\-generated spans that read fluently on their own but break the transition intoSS\. The weightα\\alphabalances local fluency against suffix alignment\. We chooseα=0\.5\\alpha=0\.5in practice\.

Sparse Suffix Selection\.Scoring the entire suffix is unnecessary, as the continuation signal concentrates in a few tokens\. We buildS~\\tilde\{S\}from \(i\) suffix head: the first few suffix tokens, which carry the immediate transition, and \(ii\) important tokens: the suffix tokens that received the highest attention from the infilling region in Stage II, which capture longer\-range dependence \(e\.g\., some variables appear later in the code\)\. These two groups need not be contiguous: the intervening suffix tokens are not scored, but are kept as visible context so that each scored token still conditions on the full true suffix preceding it,log⁡pθ​\(s~t∣P,Cj,s<t\)\\log p\_\{\\theta\}\(\\tilde\{s\}\_\{t\}\\mid P,C\_\{j\},s\_\{<t\}\), with an attention mask\. We cap the considered suffix length at3232and usem∈\[4,8\]m\\in\[4,8\]\(the two groups may overlap\) scored tokens in practice\. For clarity, Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(c\) depicts only the suffix head; the salient tokens follow the same scoring structure and are omitted to save space\.

Single\-pass Scoring\.Finally, theNNcandidate blocks are packed into one sequence and isolated by a block\-wise attention mask \(Fig\.[2](https://arxiv.org/html/2609.02108#S4.F2)\(c\), Global Attention Mask\): the visible tokens and probes of candidatejjattend only to the shared prefix and toCjC\_\{j\}’s own block, never across candidates, so the blocks cannot interfere with each other\. All candidates are therefore scored together in a single additional forward pass\.

## 5Experiments

Table 1:Performance Comparison\. We comparePillwith baselines on eight benchmarks across five models\.### 5\.1Experimental Setup

Models\.We applyPillto five diffusion large language models \(DLMs\): LLaDA\-8B\-Base/Instruct\([Nie et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib1)\), LLaDA\-MoE\-Base\([Zhu et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib2)\), Dream\-7B\-Base\([Ye et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib3)\), and Dream\-Coder\-7B\-Base\([Xie et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib4)\)\. These models cover both general\-purpose and code\-oriented DLMs, spanning different model families \(LLaDA/Dream\), architectures \(dense/MoE\), and training recipes \(base/instruction\-tuned\)\.

Datasets\.We evaluate on code and natural\-language infilling benchmarks, all disjoint from probe training set\. For code, we use HumanEval\-Infilling\([Bavarian et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib6)\)with single\-line \(\-S\) and multi\-line \(\-M\) tasks, construct MBPP\-S/\-M from MBPP\([Austin et al\., 2021b](https://arxiv.org/html/2609.02108#bib.bib7)\)by a similar protocol, and build single\-line Java and C/C\+\+ tasks from MultiPL\-E\([Cassano et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib8)\)\. For text, we construct random contiguous word\-span tasks from WikiText\([Merity et al\., 2016](https://arxiv.org/html/2609.02108#bib.bib9)\)and arXiv abstracts\. We report pass rate for code and BLEU\-2/ROUGE\-L for text\. See Appendix[A\.13](https://arxiv.org/html/2609.02108#A1.SS13)for details\.

Baselines\.We comparePillwith the backbone models \(fixed\-length decoding\) and two dynamic decoding methods, DAEDAL\([Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10)\)and CAL\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)\. Since all baselines require an initial generation length, we follow\([Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12);[Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)and report the average performance overL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\}\. This provides a prior\-free evaluation protocol, and per\-length results are provided in Appendix\. CAL is evaluated using its official implementation and hyperparameters\. We adapt DAEDAL following the implementation in\([Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11)\)to fit the infilling setting\. We also report comparison with DreamOn\([Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12)\)on text tasks\. We also report results onPill\(Oracle\), which uses the same generated candidate set asPillbut replaces the Stage III selector with the ground\-truth task evaluator, providing a upper bound on candidate selection\. All experiments run on a single NVIDIA A40 GPU\.

### 5\.2Main Results

Table[1](https://arxiv.org/html/2609.02108#S5.T1)reports infilling quality across five DLMs and eight code and natural\-language benchmarks\.Pilldelivers the strongest results on nearly every model\-dataset combination, improving the most competitive baseline CAL by\+4\.8\+4\.8pass rate on average across the code benchmarks and by\+6\.0\+6\.0BLEU\-2 points on text\. These gains hold across model families \(LLaDA/Dream\), architectures \(dense/MoE\), training recipes \(base/instruction\-tuned\), and both code and text, indicating thatPillfits different models and tasks\.

![Refer to caption](https://arxiv.org/html/2609.02108v1/ablation.png)Figure 4:Ablation studies on four design choices ofPill\.The baselines further illustrate why adaptive\-length infilling is needed\. DAEDAL, which only appends at the end of the canvas, cannot respect the suffix constraint and degrades even below the fixed\-length backbone on text\. CAL is the strongest baseline but still trailsPillwhile incurring repeated confidence\-search passes and remaining sensitive to the preset initial length \(Fig\.[1](https://arxiv.org/html/2609.02108#S1.F1)\(b\)\), whereasPillneeds no preset length at all\. Finally, the oracle upper boundPill\(Oracle\) shows that the candidate set almost always contains a strong completion, confirming that the multi\-slot stage adequately covers the correct length\. The remaining gap to the oracle indicates that the principal headroom lies in post\-hoc selection rather than generation\.

### 5\.3Efficiency Analysis

Beyond the WikiText results in Fig\.[1](https://arxiv.org/html/2609.02108#S1.F1)\(c\), Table[2](https://arxiv.org/html/2609.02108#S5.T2)compares quality and cost on HumanEval\-S with LLaDA\-8B\-Base\.Pillattains the highest Pass@1 while adding only 7% wall\-clock time over the fixed\-length backbone, in contrast to CAL \(\+94%\+94\\%\) and DAEDAL \(\+117%\+117\\%\)\. Equivalently,Pillis1\.82×1\.82\\timesfaster than the strongest baseline CAL yet6\.66\.6points more accurate\. One of the reasons for this efficiency is thatPillintroduces only two extra forward passes, the probing pass and the scoring pass, whereas the iterative length search in CAL and DAEDAL requires much more\.

Table 2:Efficiency of LLaDA\-8B\-Base on Human\-S\.
### 5\.4Sensitivity to Design Choices

#### Length probing and candidate set\.

Fig\.[4](https://arxiv.org/html/2609.02108#S5.F4)\(b\) varies the layer from which the probe reads the mask hidden state: Pass@1 increases monotonically with depth, consistent with deeper states aggregating richer bidirectional context from both prefix and suffix\. We therefore readhhfrom the last layer for convenience\. Fig\.[4](https://arxiv.org/html/2609.02108#S5.F4)\(a\) varies the candidate radiusrr\. The oracle upper bound keeps rising withrr, confirming that a larger candidate set is more likely to contain the correct length\.Pill, however, saturates beyondr=2r=2, since candidate set is larger and the ratio of correct ones decreases\. Also, we validate the effectiveness of length probing, as shown in Sec\.[5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px1)\.

#### Post\-hoc selection\.

Fig\.[4](https://arxiv.org/html/2609.02108#S5.F4)\(c\) varies the number of scored suffix tokensmm\. Pass@1 peaks atm=4m=4and declines thereafter, supporting our suffix selection: the continuation signal concentrates in a few tokens near the transition, and scoring more of them only injects noise\. Fig\.[4](https://arxiv.org/html/2609.02108#S5.F4)\(d\) varies the weightα\\alphabetween the inner and suffix scores\. Performance peaks aroundα=0\.5\\alpha=0\.5and drops at both extremes, showing that internal coherence and suffix continuation are complementary and neither alone suffices\.

### 5\.5Validation of Stages

#### Length Probing Effectiveness \(Stage I\)\.

We directly evaluate predicted lengths against the gold lengthL∗L^\{\*\}on Human\-S with LLaDA\-8B\-Base, using MAE and Acc@kk, the fraction of predictions withinkktokens ofL∗L^\{\*\}\. For CAL, we sweep the preset initial lengthl∈\{4,8,16,32\}l\\in\\\{4,8,16,32\\\}, whilePillrequires no initial length\.

As shown in Table[3](https://arxiv.org/html/2609.02108#S5.T3), CAL is highly sensitive to its initialization: its MAE increases from 3\.56 \(l=4l\{=\}4\) to 24\.13 \(l=32l\{=\}32\), while Acc@1 drops from 0\.58 to 0\.02\. In contrast,Pillachieves the lowest MAE \(2\.50\) without tuning\. Although CAL \(l=4l\{=\}4\) has higher Acc@1,Pillachieves substantially higher Acc@3 and Acc@5, which better reflects our setting sincePillexpandsL^\\hat\{L\}into nearby candidate lengths and selects among them\. In particular, Acc@5 reaches 86\.4%, showing that the target length is covered by the local candidate window in most cases\.

Table 3:Length prediction performance \(ll: initial length\)\.
#### Effect of Slot Position\-ID Design \(Stage II\)\.

We compare interpolation with left\- and right\-padding, which leave a position\-ID gap on one side of the context\. Since DLMs commonly use RoPE\([Su et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib31)\), interpolation at non\-integer positions requires no additional tuning\([Chen et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib32);[Zhang et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib33)\)\. Forℓj=1\\ell\_\{j\}=1, we set the position ID topP\+1p\_\{P\}\+1\. As shown in Table[4](https://arxiv.org/html/2609.02108#S5.T4), interpolation substantially improves both Avg Pass@1 \(51\.6 vs\. 28\.2/31\.6\) and Oracle Pass@1 \(81\.9 vs\. 73\.2/74\.2\) over left/right padding\. The gap persists under oracle selection, indicating that padding causes structural generation errors rather than merely poorer candidate selection\.

We further compare against independent single\-slot decoding\. Given the same candidate lengths\{L^−r,…,L^\+r\}\\\{\\hat\{L\}\-r,\\ldots,\\hat\{L\}\+r\\\}produced by PILL, we decode each candidate separately in its own sequence, and oracle\-select the best resulting candidate\. Interpolation nearly matches this upper bound \(81\.9/51\.6 vs\. 83\.6/52\.9\), showing that the shared multi\-slot layout introduces little quality loss\.

Table 4:Effect of slot position\-id design \(LLaDA\-8B\-Base, Human\-S,r=2r\{=\}2\)\.
#### Ablation of Post\-hoc Selection Score \(Stage III\)\.

We ablate the post\-hoc selection score in Table[5](https://arxiv.org/html/2609.02108#S5.T5)\. Random uniformly selects a decoded candidate, while predicted\-length only uses the center candidate atL^\\hat\{L\}without post\-hoc selection\. Inner\-only and suffix\-only correspond toα=1\\alpha=1andα=0\\alpha=0in Eq\. \([3](https://arxiv.org/html/2609.02108#S4.E3)\), respectively, and our default combines both signals withα=0\.5\\alpha=0\.5\. Random selection performs poorly despite the presence of strong candidates, showing that effective post\-hoc selection is necessary\. Using the predicted length alone is substantially stronger, while combining the inner and suffix scores achieves the best performance, confirming that the two signals are complementary\.

Table 5:Ablation of the post\-hoc selection score on LLaDA\-8B\-Base and HumanEval\-S\.

## 6Conclusion

We presentedPill, a backbone\-frozen framework that turns fixed\-length DLMs into adaptive\-length infillers through three lightweight stages: probing the mask hidden state to predict the target length, decoding multiple length candidates in parallel via slot\-wise attention, and selecting among them with a post\-hoc coherence score\. Adding only two forward passes and requiring no preset length,Pilloutperforms prior adaptive\-length baselines across five DLMs on both code and text infilling, running substantially faster than them\.

## Limitations

WhilePillis effective and efficient across a range of DLMs and infilling benchmarks, many practical scenarios involve complex editing settings, such as nested spans and repository\-level code patches, remain interesting directions for future work\. In addition, our experiments are conducted on DLMs up to roughly 7B/8B parameters on a single GPU\. Whether the behavior of the length probe and the post\-hoc selection scales to substantially larger diffusion backbones remains to be verified\.

## Acknowledgement

This work is supported by NSF \(2324770\)\. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred\. The U\.S\. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on\.

## References

- Austinet al\.\(2021a\)J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den BergStructured denoising diffusion models in discrete state\-spaces\.Advances in neural information processing systems34,pp\. 17981–17993\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1)\.
- Austinet al\.\(2021b\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p2.1)\.
- Banet al\.\(2021\)Y\. Ban, Y\. Yan, A\. Banerjee, and J\. HeEe\-net: exploitation\-exploration neural networks in contextual bandits\.arXiv preprint arXiv:2110\.03177\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Banet al\.\(2023\)Y\. Ban, Y\. Yan, A\. Banerjee, and J\. HeNeural exploitation and exploration of contextual bandits\.arXiv preprint arXiv:2305\.03784\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Bavarianet al\.\(2022\)M\. Bavarian, H\. Jun, N\. Tezak, J\. Schulman, C\. McLeavey, J\. Tworek, and M\. ChenEfficient training of language models to fill in the middle\.arXiv preprint arXiv:2207\.14255\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02108#S1.p1.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p2.1)\.
- Beiet al\.\(2026\)Y\. Bei, T\. Wei, X\. Ning, Y\. Zhao, Z\. Liu, X\. Lin, Y\. Zhu, H\. Hamann, J\. He, and H\. TongMem\-gallery: benchmarking multimodal long\-term conversational memory for mllm agents\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 40750–40784\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Bieet al\.\(2025\)T\. Bie, M\. Cao, K\. Chen, L\. Du, M\. Gong, Z\. Gong, Y\. Gu, J\. Hu, Z\. Huang, Z\. Lan,et al\.Llada2\. 0: scaling up diffusion language models to 100b\.arXiv preprint arXiv:2512\.15745\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p2.1),[§2](https://arxiv.org/html/2609.02108#S2.p1.1)\.
- Cassanoet al\.\(2022\)F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney, M\. Yee, Y\. Zi, C\. J\. Anderson, M\. Q\. Feldman,et al\.Multipl\-e: a scalable and extensible approach to benchmarking neural code generation\.arXiv preprint arXiv:2208\.08227\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px6.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p2.1)\.
- Chenet al\.\(2026\)J\. Chen, Y\. Liang, and Z\. LiuDflash: block diffusion for flash speculative decoding\.arXiv preprint arXiv:2602\.06036\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)S\. Chen, S\. Wong, L\. Chen, and Y\. TianExtending context window of large language models via positional interpolation\.arXiv preprint arXiv:2306\.15595\.Cited by:[§5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px2.p1.1)\.
- Chenget al\.\(2026\)X\. Cheng, X\. Yu, C\. Shao, J\. Li, Y\. Xiong, Y\. Qian, J\. Zhu, S\. Ma, X\. Zhang, J\. Ye,et al\.DSpark: confidence\-scheduled speculative decoding with semi\-autoregressive generation\.arXiv preprint arXiv:2607\.05147\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Duet al\.\(2021\)B\. Du, S\. Zhang, Y\. Yan, and H\. TongNew frontiers of multi\-network mining: recent developments and future trend\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,pp\. 4038–4039\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Friedet al\.\(2022\)D\. Fried, A\. Aghajanyan, J\. Lin, S\. Wang, E\. Wallace, F\. Shi, R\. Zhong, W\. Yih, L\. Zettlemoyer, and M\. LewisIncoder: a generative model for code infilling and synthesis\.arXiv preprint arXiv:2204\.05999\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p2.1)\.
- Hoogeboomet al\.\(2021\)E\. Hoogeboom, D\. Nielsen, P\. Jaini, P\. Forré, and M\. WellingArgmax flows and multinomial diffusion: learning categorical distributions\.Advances in neural information processing systems34,pp\. 12454–12465\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p1.1)\.
- Huanget al\.\(2025\)F\. Huang, Y\. Bei, Z\. Yang, J\. Jiang, H\. Chen, Q\. Shen, S\. Wang, F\. Karray, and P\. S\. YuLarge language model simulator for cold\-start recommendation\.InProceedings of the eighteenth ACM international conference on web search and data mining,pp\. 261–270\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Jinget al\.\(2024\)B\. Jing, Y\. Yan, K\. Ding, C\. Park, Y\. Zhu, H\. Liu, and H\. TongSterling: synergistic representation learning on bipartite graphs\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 12976–12984\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Kimet al\.\(2025\)J\. Kim, L\. Cheuk\-Kit, C\. Domingo\-Enrich, Y\. Du, S\. Kakade, T\. Ngotiaoco, S\. Chen, and M\. AlbergoAny\-order flexible length masked diffusion\.arXiv preprint arXiv:2509\.01025\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p2.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1)\.
- Laiet al\.\(2026\)X\. Lai, W\. Xu, Y\. Yang, Q\. Chen, Y\. Xu, L\. Zeng, X\. Li, H\. Sun, H\. Zhu, V\. Zhang,et al\.Minimax sparse attention\.arXiv preprint arXiv:2606\.13392\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)J\. Li, R\. Han, C\. Sun, D\. Sun, R\. Wang, J\. Zeng, Y\. Yan, H\. Tong, and T\. AbdelzaherLarge language model\-guided disentangled belief representation learning on polarized social graphs\.In2024 33rd International Conference on Computer Communications and Networks \(ICCCN\),pp\. 1–9\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Liet al\.\(2025a\)J\. Li, X\. Dong, Y\. Zang, Y\. Cao, J\. Wang, and D\. LinBeyond fixed: training\-free variable\-length denoising for diffusion large language models\.arXiv preprint arXiv:2508\.00819\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.p1.1),[§1](https://arxiv.org/html/2609.02108#S1.p2.1),[§1](https://arxiv.org/html/2609.02108#S1.p3.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§4\.2](https://arxiv.org/html/2609.02108#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p3.1)\.
- Liet al\.\(2023\)R\. Li, L\. B\. Allal, Y\. Zi, N\. Muennighoff, D\. Kocetkov, C\. Mou, M\. Marone, C\. Akiki, J\. Li, J\. Chim,et al\.Starcoder: may the source be with you\!\.arXiv preprint arXiv:2305\.06161\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p2.1)\.
- Liet al\.\(2025b\)T\. Li, M\. Chen, B\. Guo, and Z\. ShenA survey on diffusion language models\.arXiv preprint arXiv:2508\.10875\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1)\.
- Liet al\.\(2022a\)X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. HashimotoDiffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1)\.
- Liet al\.\(2026a\)Y\. Li, F\. Wei, C\. Zhang, and H\. ZhangEagle\-3: scaling up inference acceleration of large language models via training\-time test\.Advances in Neural Information Processing Systems38,pp\. 136737–136756\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Liet al\.\(2022b\)Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago,et al\.Competition\-level code generation with alphacode\.Science378\(6624\),pp\. 1092–1097\.Cited by:[§A\.1](https://arxiv.org/html/2609.02108#A1.SS1.SSS0.Px2.p1.1),[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px5.p1.1)\.
- Liet al\.\(2026b\)Z\. Li, H\. Lin, Y\. Xiao, Z\. Li, X\. Song, Z\. Zheng, Y\. He, H\. Yao, K\. Ding, C\. Yu,et al\.IV\-cot: implicit visual chain\-of\-thought for structure\-aware text\-to\-image generation\.arXiv preprint arXiv:2606\.24849\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2026a\)H\. Lin, X\. Jia, S\. Liu, S\. Xia, W\. Huang, H\. Xu, J\. Li, Y\. Xiao, X\. Xing, Z\. Guo,et al\.Efficient diffusion language models: a comprehensive survey\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1)\.
- Linet al\.\(2026b\)H\. Lin, X\. Jia, H\. Xu, B\. Yao, X\. Guo, Y\. Wu, Z\. Lu, Y\. Wei, Q\. Zhang, and Z\. SunDuQuant\+\+: fine\-grained rotation enhances microscaling fp4 quantization\.arXiv preprint arXiv:2604\.17789\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2025a\)H\. Lin, T\. Wang, Y\. Ge, Y\. Ge, Z\. Lu, Y\. Wei, Q\. Zhang, Z\. Sun, and Y\. ShanToklip: marry visual tokens to clip for multimodal comprehension and generation\.arXiv preprint arXiv:2505\.05422\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2024a\)H\. Lin, H\. Xu, Y\. Wu, J\. Cui, Y\. Zhang, L\. Mou, L\. Song, Z\. Sun, and Y\. WeiDuquant: distributing outliers via dual transformation makes stronger quantized llms\.Advances in Neural Information Processing Systems37,pp\. 87766–87800\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2025b\)H\. Lin, H\. Xu, Y\. Wu, Z\. Guo, R\. Zhang, Z\. Lu, Y\. Wei, Q\. Zhang, and Z\. SunQuantization meets dllms: a systematic study of post\-training quantization for diffusion llms\.arXiv preprint arXiv:2508\.14896\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2026c\)H\. Lin, K\. Zhu, H\. Xu, Y\. Wu, Z\. Lu, Q\. Zhang, and Z\. SunBenchmarking trustworthiness of slms: pre\-trained vs\. compressed\.arXiv preprint arXiv:2608\.11981\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2024b\)X\. Lin, J\. Kang, W\. Cong, and H\. TongBemap: balanced message passing for fair graph neural network\.InLearning on Graphs Conference,pp\. 37–1\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2025c\)X\. Lin, Z\. Liu, Z\. Yang, G\. Li, R\. Qiu, S\. Wang, H\. Liu, H\. Li, S\. Keswani, V\. Pardeshi,et al\.Moralise: a structured benchmark for moral alignment in visual language models\.arXiv preprint arXiv:2505\.14728\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Linet al\.\(2026d\)X\. Lin, Z\. Tang, W\. Cong, M\. Hang, K\. Wang, Y\. Wang, Z\. Zeng, T\. Li, H\. Yoo, Z\. Liu,et al\.Mixture of sequence: theme\-aware mixture\-of\-experts for long\-sequence recommendation\.InProceedings of the ACM Web Conference 2026,pp\. 6469–6480\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)H\. Liu, Z\. Yang, and B\. SuDiffusion lms can approximate optimal infilling lengths implicitly\.arXiv preprint arXiv:2602\.00476\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.p1.1),[§1](https://arxiv.org/html/2609.02108#S1.p2.1),[§1](https://arxiv.org/html/2609.02108#S1.p3.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§4\.2](https://arxiv.org/html/2609.02108#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p3.1)\.
- Liuet al\.\(2025\)L\. Liu, Z\. Wang, D\. Zhou, R\. Wang, Y\. Yan, B\. Xiong, S\. He, and H\. TongFew\-shot knowledge graph completion via transfer knowledge from similar tasks\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 4960–4965\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)R\. Liu, H\. Bai, H\. Lin, Y\. Li, H\. Gao, Z\. Xu, L\. Hou, J\. Yao, and C\. YuanIntactkv: improving large language model quantization by keeping pivot tokens intact\.arXiv preprint arXiv:2403\.01241\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024b\)Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. HuKivi: a tuning\-free asymmetric 2bit quantization for kv cache\.arXiv preprint arXiv:2402\.02750\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Manviet al\.\(2026\)R\. Manvi, J\. Hong, T\. Seyde, M\. Labonne, M\. Lechner, and S\. LevineZero\-overhead introspection for adaptive test\-time compute\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 109705–109723\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Merityet al\.\(2016\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px7.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p2.1)\.
- Nieet al\.\(2026a\)S\. Nie, Q\. Min, S\. Xu, Z\. Huang, Y\. Song, Y\. Shan, Y\. Lin, W\. X\. Zhao, C\. Li, and J\. WenImproved large language diffusion models\.arXiv preprint arXiv:2606\.25331\.Cited by:[§A\.4](https://arxiv.org/html/2609.02108#A1.SS4.p1.1)\.
- Nieet al\.\(2026b\)S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. LiLarge language diffusion models\.Advances in Neural Information Processing Systems38,pp\. 50608–50646\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p1.1),[§2](https://arxiv.org/html/2609.02108#S2.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§A\.1](https://arxiv.org/html/2609.02108#A1.SS1.SSS0.Px2.p1.1),[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px8.p1.1)\.
- Raychevet al\.\(2016\)V\. Raychev, P\. Bielik, and M\. VechevProbabilistic model for code with decision trees\.ACM SIGPLAN Notices51\(10\),pp\. 731–747\.Cited by:[§A\.1](https://arxiv.org/html/2609.02108#A1.SS1.SSS0.Px2.p1.1),[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.SSS0.Px3.p1.1)\.
- Roachet al\.\(2020\)S\. Roach, C\. Ni, A\. Kopylov, T\. Lu, J\. Xu, S\. Zhang, B\. Du, D\. Zhou, J\. Wu, L\. Liu,et al\.Canon: complex analytics of network of networks for modeling adversarial activities\.In2020 IEEE International Conference on Big Data \(Big Data\),pp\. 1634–1643\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Roziereet al\.\(2023\)B\. Roziere, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.Code llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p2.1)\.
- Sahooet al\.\(2024\)S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. KuleshovSimple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1),[§3](https://arxiv.org/html/2609.02108#S3.p1.1)\.
- Shiet al\.\(2024\)J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. TitsiasSimplified and generalized masked diffusion for discrete data\.Advances in neural information processing systems37,pp\. 103131–103167\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1),[§3](https://arxiv.org/html/2609.02108#S3.p1.1)\.
- Songet al\.\(2025\)Y\. Song, Z\. Zhang, C\. Luo, P\. Gao, F\. Xia, H\. Luo, Z\. Li, Y\. Yang, H\. Yu, X\. Qu,et al\.Seed diffusion: a large\-scale diffusion language model with high\-speed inference\.arXiv preprint arXiv:2508\.02193\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px2.p1.1)\.
- Wanget al\.\(2023\)D\. Wang, Y\. Yan, R\. Qiu, Y\. Zhu, K\. Guan, A\. Margenot, and H\. TongNetworked time series imputation via position\-aware graph enhanced variational autoencoders\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 2256–2268\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Wanget al\.\(2018\)R\. Wang, Y\. Yan, J\. Wang, Y\. Jia, Y\. Zhang, W\. Zhang, and X\. WangAcekg: a large\-scale knowledge graph for academic data mining\.InProceedings of the 27th ACM international conference on information and knowledge management,pp\. 1487–1490\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)Z\. Wu, L\. Zheng, Z\. Xie, J\. Ye, J\. Gao, S\. Gong, Y\. Feng, Z\. Li, W\. Bi, G\. Zhou,et al\.DreamOn: diffusion language models for code infilling beyond fixed\-size canvas\.arXiv preprint arXiv:2602\.01326\.Cited by:[§A\.10](https://arxiv.org/html/2609.02108#A1.SS10.SSS0.Px2.p1.1),[§A\.13](https://arxiv.org/html/2609.02108#A1.SS13.p1.1),[§1](https://arxiv.org/html/2609.02108#S1.p2.1),[§1](https://arxiv.org/html/2609.02108#S1.p3.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p3.1)\.
- Xiaet al\.\(2025\)S\. Xia, H\. Lin, Y\. Wu, Y\. Zhou, Z\. Li, Z\. Wan, X\. Xing, Y\. Zheng, X\. Li, C\. Shan,et al\.Medrek: retrieval\-based editing for medical llms with key\-aware prompts\.arXiv preprint arXiv:2510\.13500\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21875–21895\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Xieet al\.\(2025\)Z\. Xie, J\. Ye, L\. Zheng, J\. Gao, J\. Dong, Z\. Wu, X\. Zhao, S\. Gong, X\. Jiang, Z\. Li,et al\.Dream\-coder 7b: an open diffusion language model for code\.arXiv preprint arXiv:2509\.01142\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p1.1)\.
- Xinget al\.\(2026\)X\. Xing, Z\. Liu, S\. Xiao, B\. Gao, Y\. Liang, H\. Lin, X\. Zeng, G\. Li, and J\. ZhangEfficientLLM: unified pruning\-aware pretraining for auto\-designed compact language models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7813–7830\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026\)H\. Xu, S\. Chen, R\. Qiu, Y\. Yan, C\. Luo, M\. X\. Cheng, J\. He, and H\. TongPrune as you generate: online rollout pruning for faster and better rlvr\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13876–13893\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Xuet al\.\(2024\)H\. Xu, Y\. Yan, D\. Wang, Z\. Xu, Z\. Zeng, T\. F\. Abdelzaher, J\. Han, and H\. TongSlog: an inductive spectral graph neural network beyond polynomial filter\.InForty\-first International Conference on Machine Learning,Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2024a\)Y\. Yan, Y\. Chen, H\. Chen, X\. Li, Z\. Xu, Z\. Zeng, L\. Liu, Z\. Liu, and H\. TongThegcn: temporal heterophilic graph convolutional network\.arXiv preprint arXiv:2412\.16435\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2023a\)Y\. Yan, Y\. Chen, H\. Chen, M\. Xu, M\. Das, H\. Yang, and H\. TongFrom trainable negative depth to edge heterophily in graphs\.Advances in Neural Information Processing Systems36,pp\. 70162–70178\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2024b\)Y\. Yan, Y\. Hu, Q\. Zhou, L\. Liu, Z\. Zeng, Y\. Chen, M\. Pan, H\. Chen, M\. Das, and H\. TongPacer: network embedding from positional to structural\.InProceedings of the ACM Web Conference 2024,pp\. 2485–2496\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Yanet al\.\(2024c\)Y\. Yan, Y\. Hu, Q\. Zhou, S\. Wu, D\. Wang, and H\. TongTopological anonymous walk embedding: a new structural node embedding approach\.InProceedings of the 33rd ACM International Conference on Information and Knowledge Management,pp\. 2796–2806\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Yanet al\.\(2023b\)Y\. Yan, B\. Jing, L\. Liu, R\. Wang, J\. Li, T\. Abdelzaher, and H\. TongReconciling competing sampling strategies of network embedding\.Advances in Neural Information Processing Systems36,pp\. 6844–6861\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2025\)Y\. Yan, A\. Kolekar, S\. Genc, W\. Xu, E\. W\. Huang, A\. Srinivasan, M\. Jain, Q\. He, and H\. TongTo answer or not to answer \(taona\): a robust textual graph understanding and question answering approach\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 6360–6376\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2021a\)Y\. Yan, L\. Liu, Y\. Ban, B\. Jing, and H\. TongDynamic knowledge graph alignment\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 4564–4572\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2021b\)Y\. Yan, S\. Zhang, and H\. TongBright: a bridging algorithm for network alignment\.InProceedings of the web conference 2021,pp\. 3907–3917\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanet al\.\(2022\)Y\. Yan, Q\. Zhou, J\. Li, T\. Abdelzaher, and H\. TongDissecting cross\-layer dependency inference on multi\-layered inter\-dependent networks\.InProceedings of the 31st ACM International Conference on Information & Knowledge Management,pp\. 2341–2351\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026a\)L\. Yang, H\. Lin, Y\. Wu, C\. Shan, Z\. Sun, and Q\. GuReshape and rotate: adaptive weight reshaping and fine\-grained rotation for ultra\-low\-bit diffusion transformers quantization\.Neurocomputing,pp\. 133830\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026b\)L\. Yang, H\. Lin, Y\. Wu, Z\. Sun, and Q\. GuDapQ\-dit: distribution\-aware post\-training quantization for efficient generative tasks in diffusion transformers\.InProceedings of the 2026 International Conference on Multimedia Retrieval,pp\. 2371–2380\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yanget al\.\(2024\)X\. Yang, H\. Chen, Y\. Yan, Y\. Tang, Y\. Zhao, E\. Xu, Y\. Cai, and H\. TongSimCE: simplifying cross\-entropy loss for collaborative filtering\.arXiv preprint arXiv:2406\.16170\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yeet al\.\(2025\)J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. KongDream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p1.1),[§2](https://arxiv.org/html/2609.02108#S2.p1.1),[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p1.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zeng, Y\. Yan, Z\. Liu, B\. Jing, R\. Qiu, A\. Azad, and H\. TongPlanetalign: a comprehensive python library for benchmarking network alignment\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 66087–66115\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zeng, Y\. Yan, L\. Ying, R\. Srikant, and H\. TongJoint optimal transport and embedding for network alignment\.InProceedings of the ACM on Web Conference 2025,pp\. 2064–2075\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2026a\)Z\. Zeng, W\. Bao, X\. Lin, R\. Qiu, T\. Wei, X\. Ning, Y\. Yan, C\. Luo, M\. X\. Cheng, J\. He,et al\.Subspace alignment for vision\-language model test\-time adaptation\.arXiv preprint arXiv:2601\.08139\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2024\)Z\. Zeng, B\. Du, S\. Zhang, Y\. Xia, Z\. Liu, and H\. TongHierarchical multi\-marginal optimal transport for network alignment\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 16660–16668\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025a\)Z\. Zeng, M\. Hang, X\. Liu, X\. Liu, X\. Lin, R\. Qiu, T\. Wei, Z\. Liu, S\. Yuan, C\. Yang,et al\.Hierarchical lora moe for efficient ctr model scaling\.arXiv preprint arXiv:2510\.10432\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2025b\)Z\. Zeng, X\. Liu, M\. Hang, X\. Liu, Q\. Zhou, C\. Yang, Y\. Liu, Y\. Ruan, L\. Chen, Y\. Chen,et al\.Interformer: effective heterogeneous interaction learning for click\-through rate prediction\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 6225–6233\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2025c\)Z\. Zeng, R\. Qiu, W\. Bao, T\. Wei, X\. Lin, Y\. Yan, T\. F\. Abdelzaher, J\. Han, and H\. TongPave your own path: graph gradual domain adaptation on fused gromov\-wasserstein geodesics\.arXiv preprint arXiv:2505\.12709\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2026b\)Z\. Zeng, Q\. Yu, X\. Lin, R\. Qiu, X\. Ning, T\. Wei, Y\. Yan, J\. He, and H\. TongHarnessing consistency for robust test\-time llm ensemble\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 3528–3545\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zenget al\.\(2023a\)Z\. Zeng, S\. Zhang, Y\. Xia, and H\. TongParrot: position\-aware regularized optimal transport for network alignment\.InProceedings of the ACM web conference 2023,pp\. 372–382\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Zenget al\.\(2023b\)Z\. Zeng, R\. Zhu, Y\. Xia, H\. Zeng, and H\. TongGenerative graph dictionary learning\.InInternational Conference on Machine Learning,pp\. 40749–40769\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025a\)A\. Zhang, A\. Sivakumar, C\. Tang, and C\. ThomasFlexible\-length text infilling for discrete diffusion models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 31332–31347\.Cited by:[§2](https://arxiv.org/html/2609.02108#S2.p2.1),[§5\.5](https://arxiv.org/html/2609.02108#S5.SS5.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)J\. Zhang, Y\. Hsieh, Z\. Wan, H\. Lin, X\. Wang, Z\. Wang, Y\. Lei, and M\. ZhangQuantvla: scale\-calibrated post\-training quantization for vision\-language\-action models\.arXiv preprint arXiv:2602\.20309\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in neural information processing systems36,pp\. 34661–34710\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, H\. Xu, Z\. Yang, C\. Yue, Z\. Lin, H\. Gao, Z\. Wang, and H\. ZhaoTa\-vla: elucidating the design space of torque\-aware vision\-language\-action models\.arXiv preprint arXiv:2509\.07962\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025c\)Z\. Zhang, C\. Yue, H\. Xu, M\. Liao, X\. Qi, H\. Gao, Z\. Wang, and H\. ZhaoRobochemist: long\-horizon and safety\-compliant robotic chemical experimentation\.arXiv preprint arXiv:2509\.08820\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025a\)F\. Zhu, R\. Wang, S\. Nie, X\. Zhang, C\. Wu, J\. Hu, J\. Zhou, J\. Chen, Y\. Lin, J\. Wen,et al\.Llada 1\.5: variance\-reduced preference optimization for large language diffusion models\.arXiv preprint arXiv:2505\.19223\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p1.1)\.
- Zhuet al\.\(2025b\)F\. Zhu, Z\. You, Y\. Xing, Z\. Huang, L\. Liu, Y\. Zhuang, G\. Lu, K\. Wang, X\. Wang, L\. Wei,et al\.Llada\-moe: a sparse moe diffusion language model\.arXiv preprint arXiv:2509\.24389\.Cited by:[§1](https://arxiv.org/html/2609.02108#S1.p1.1),[§5\.1](https://arxiv.org/html/2609.02108#S5.SS1.p1.1)\.
- Zhuet al\.\(2025c\)R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing,et al\.Scaling latent reasoning via looped language models\.arXiv preprint arXiv:2510\.25741\.Cited by:[§A\.12](https://arxiv.org/html/2609.02108#A1.SS12.SSS0.Px1.p1.1)\.

## Appendix AAppendix

### A\.1Details of the Length Probe

#### Architecture\.

The probe is implemented as a 3\-layer MLP with hidden dimensions of 512 and 128\. We use GeLU as the activation function\.

#### Training Datasets\.

For each model, we train a probe\. We collect 196k training samples for each probe, from two sources: \(1\) a code corpus, including Py150\([Raychev et al\., 2016](https://arxiv.org/html/2609.02108#bib.bib27)\), LeetCode, and CodeContests\([Li et al\., 2022b](https://arxiv.org/html/2609.02108#bib.bib28)\); and \(2\) a text corpus from C4\([Raffel et al\., 2020](https://arxiv.org/html/2609.02108#bib.bib30)\)\. For each sample, we randomly mask a contiguous span from the original code or text sequence, using the remaining left and right contexts as the prefix and suffix, respectively\. The length of the masked span is used as the target length\. All training datasets are distinct from the evaluation datasets and are strictly non\-overlapping\. More details are provided in Sec\.[A\.13](https://arxiv.org/html/2609.02108#A1.SS13)\.

#### Data Construction\.

The probe is trained on auxiliary self\-supervised infilling examples and does not use any evaluation instances or test cases from evaluation datasets\. Each example consists of a prefix, an original masked span, and a suffix, with the token length of the masked span used as supervision\.

For code, we construct examples from the public training splits of Py150, LeetCode, and CodeContests\. We retain contexts of at most 2,048 tokens under the LLaDA tokenizer, mask non\-comment code lines, and require non\-empty prefix and suffix contexts\. For multilingual code, the target span is required to contain at least four tokenizer tokens\. We restrict Py150 examples to top\-level Python functions, extract LeetCode examples from its Python solution field, and use only accepted submissions from CodeContests\. No executable tests from these auxiliary datasets are used\.

For text, we sample examples from the English training split of C4\. We normalize each document, retain documents containing 80\-400 words with sufficient alphabetic content, exclude URL\-heavy pages, and mask a random contiguous span of 2\-8 words while requiring at least 32 characters of both prefix and suffix context\. We apply no additional target\-length filtering\. Since hidden representations differ across backbone models, we train a separate probe for each backbone using the same data\-construction pipeline\.

#### Training Recipe\.

For the length probe, we train a lightweight MLP regressor with a regression objective\. Unless otherwise specified, the probe is optimized with AdamW using a learning rate of1×10−31\\times 10^\{\-3\}, weight decay of1×10−41\\times 10^\{\-4\}, batch size of 16, and dropout rate of0\.10\.1\. The model is trained for up to5050epochs with mean squared error \(MSE\) loss\.

Discussion\.All adaptive\-length methods incur some preparation cost; the key difference lies in its*form*and*when it is paid*\. The PILL probe is trained only*once*per backbone \(about 100 minutes on a single NVIDIA A40 GPU\) and is thereafter reused across all tasks and inputs with no further training, adding merely two forward passes at inference time\. In contrast, CAL is*not*cost\-free either: it re\-initializes its confidence parameters for*each*setting \(about 20 minutes, 2k samples\), and this cost recurs*online*, since its iterative confidence search introduces many extra forward passes at inference \(Table[2](https://arxiv.org/html/2609.02108#S5.T2)\)\. DAEDAL likewise relies on fitted/tuned components rather than operating fully out of the box\. PILL’s cost is therefore a one\-time, amortized expense, whereas the baselines pay repeatedly at inference\. Moreover, the probe’s training cost does*not*stem from overfitting to the evaluation distribution\. As shown in our comparison with DreamOn \(Section[A\.10](https://arxiv.org/html/2609.02108#A1.SS10.SSS0.Px2), Table[17](https://arxiv.org/html/2609.02108#A1.T17)\), the text infilling tasks are out\-of\-distribution for our probe, which is fitted only on code; PILL nonetheless outperforms DreamOn across all of its length configurations on both datasets\. This indicates that the length prediction generalizes beyond its training domain, so the gains arise from the method design rather than from in\-distribution length statistics\.

### A\.2Per\-Stage Overhead at a Fixed Length

To attribute cost to each stage, we decomposePill’s runtime on LLaDA\-8B\-Base \(Human\-S\) against a controlled baseline: a single fixed\-length decoding at the*same*predicted lengthL^\\hat\{L\}\(denoted Basic Dec\.\)\. This isolates the cost added by each stage from any effect of the decoding length itself, and therefore differs from the end\-to\-end comparison in Table[2](https://arxiv.org/html/2609.02108#S5.T2), where the backbone is averaged over preset lengthsL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\}and thus decodes at different lengths\. As shown in Table[6](https://arxiv.org/html/2609.02108#A1.T6), relative to this matched\-length decoding, the probe, multi\-slot decoding, and selection add8\.9%8\.9\\%,3\.4%3\.4\\%, and9\.4%9\.4\\%respectively\. Notably, the multi\-slot stage adds almost nothing despite generating all2​r\+12r\+1candidates, confirming that the slot\-wise parallel design decodes the full candidate set at a cost close to a single slot; the two lightweight probes account for the remainder\.

Table 6:Per\-stage overhead at a fixed length\.
### A\.3Full Results across Initial Lengths

Table[7](https://arxiv.org/html/2609.02108#A1.T7)reports the per\-length results that are averaged into Table[1](https://arxiv.org/html/2609.02108#S5.T1)of the main paper, on three categories of tasks: Python \(Human\-S\), other language \(Java\), and text \(Wikitext\)\. Two patterns stand out\. First, baseline quality is highly sensitive to the preset length and varies non\-monotonically with it: the backbone peaks atL=16L\{=\}16on Human\-S \(56\.24\) but atL=8L\{=\}8on Wikitext \(19\.04\), and even the strongest baseline CAL drops from 70\.86 atL=4L\{=\}4to 50\.82 atL=32L\{=\}32on Human\-S, and from 22\.69 to 5\.35 over the same range on Wikitext\. Crucially, the best\-performing length differs across datasets, so no single preset length transfers, which is precisely the difficulty an adaptive method must resolve\. Second, larger lengths inflate the time cost monotonically while quality does not follow, so the longer configurations pay more for worse results\. In contrast,Pilluses no preset length and a single run already surpasses the best length\-specific result of every baseline, at a time cost close to the cheapest backbone configuration\.

\(a\) Results on Human\-S dataset

\(b\) Results on Java dataset

\(c\) Results on Wikitext dataset

Table 7:Baseline results under different initial lengths\.
### A\.4More Results on Recent DLMs

To further evaluate the generalizability ofPillto stronger DLM backbones, we apply it to iLLaDA\-8B\-Base\([Nie et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib5)\), a recently proposed model that improves upon LLaDA through scaled pre\-training and enhanced fine\-tuning\. We use the samePillpipeline and evaluation protocol without modifying the backbone\. Table[8](https://arxiv.org/html/2609.02108#A1.T8)reports the results on Java and C/C\+\+\.

Table 8:Infilling performance on iLLaDA\-8B\-Base\.Table 9:Inference efficiency on iLLaDA\-8B\-Base under the same evaluation protocol as Sec\.[5\.3](https://arxiv.org/html/2609.02108#S5.SS3)\.Pillconsistently outperforms all baselines on both datasets while requiring 1\.11s per sample, compared with 1\.06s for fixed\-length backbone decoding and 2\.01s for CAL\. Thus,Pilladds only about 5% wall\-clock overhead over the backbone while retaining its efficiency advantage over iterative adaptive\-length decoding\.

### A\.5Extension to Multi\-span Infilling

To evaluatePillbeyond single\-span infilling, we construct a multi\-span infilling setting based on MBPP, where each instance contains multiple disjoint missing spans\.Pillhandles all gaps jointly within a single sequence: a single probing pass predicts the lengths of all missing spans, multi\-slot decoding generates their candidate spans in parallel, and the resulting candidates are scored jointly\.

Table 10:Performance and efficiency on multi\-span infilling constructed from MBPP\.As shown in Table[10](https://arxiv.org/html/2609.02108#A1.T10),Pillimproves the pass rate by 11\.79 points over fixed\-length decoding and by 4\.86 points over CAL\. Meanwhile, it incurs only about 1\.5% wall\-clock overhead over the backbone \(6\.70s vs\. 6\.60s\) and is approximately 2\.6×\\timesfaster than CAL\. These results suggest that the probing and parallel\-decoding design extends naturally to multiple disjoint gaps without incurring iterative search for each span\.

### A\.6Robustness of Length Probing

#### Robustness to structural complexity\.

We first examine whether length probing remains reliable when the dependency between the prefix and suffix becomes structurally more complex\. To control for span length, we restrict the analysis to Java and C/C\+\+ examples with the same gold infill length of 10 tokens\. We parse each complete gold program and measure structural complexity using \(1\) the maximum AST nesting depth within the missing span and \(2\) the number of cross\-boundary identifier reuses, i\.e\., identifiers shared between the missing span and its prefix or suffix\. Examples are divided into low\- and high\-complexity groups at the median\.

Table 11:Length\-probing performance \(%\) under different levels of structural complexity\. MAE is measured in tokens; the remaining columns report the percentage of predictions that exactly match or fall within 1/2 tokens of the gold length\.As shown in Table[11](https://arxiv.org/html/2609.02108#A1.T11), greater structural complexity moderately increases MAE and reduces near\-exact prediction accuracy\. Nevertheless, 81\.4% and 70\.2% of high\-complexity Java and C/C\+\+ examples, respectively, remain within two tokens of the gold length, matching the default candidate radiusr=2r=2\. This suggests that the probe remains effective even in structurally complex, nested, and dependency\-heavy contexts\.

#### Robustness to training data size\.

We further study the sensitivity of the length probe to the amount of auxiliary training data\. We train probes with different numbers of samples and evaluate the resultingPillmodels on Human\-S with LLaDA\-8B\-Base\.

Table 12:Effect of the amount of auxiliary probe\-training data on Human\-S with LLaDA\-8B\-Base\.As shown in Table[12](https://arxiv.org/html/2609.02108#A1.T12), performance improves steadily with more probe\-training data but already surpasses CAL with 10k examples \(67\.96 vs\. 64\.74\)\. The improvement becomes modest beyond 50k samples, suggesting that the probe does not critically depend on the full 100k training set\.

### A\.7Scaling of Multi\-slot Parallel Decoding

We study how the computational cost of multi\-slot decoding scales with the number of candidate lengths\. We vary the candidate radius fromr=0r=0tor=4r=4, corresponding to 1–9 candidates, and profile wall\-clock time and peak GPU memory on Human\-S with LLaDA\-8B\-Base using a single NVIDIA A40 GPU\.

Table 13:Scaling of multi\-slot decoding with the number of candidate lengths on Human\-S using LLaDA\-8B\-Base\. The adaptive setting uses 3/5/7 candidates according to prediction uncertainty\.As shown in Table[13](https://arxiv.org/html/2609.02108#A1.T13), both runtime and memory increase approximately with the number of candidates, since the shared prefix/suffix context is reused while only slot\-specific tokens grow with the candidate set\. From one to nine candidates, runtime increases from 0\.82s to 1\.44s \(1\.76×\\times\) and memory from 21\.3GB to 33\.7GB \(1\.58×\\times\)\. Meanwhile, Pass@1 saturates aroundr=2r=2even though the oracle upper bound continues to increase, indicating that adding more candidate lengths primarily improves coverage rather than final selected performance\. This motivates our default choice ofr=2r=2\.

### A\.8Failure Modes

We analyze where the remaining errors ofPillarise by separating them into probe coverage, candidate generation, and post\-hoc selection failures\. Table[14](https://arxiv.org/html/2609.02108#A1.T14)summarizes the breakdown\.

Table 14:Failure mode analysis ofPillon Human\-S\.Probe coverage is the largest source of error\. Interestingly, in 10\.2% of cases associated with coverage failure, a nearby\-length candidate still produces a correct solution\. Candidate generation failures are relatively rare \(3\.6%\), while selection failures account for 11\.4%, consistent with the remaining gap betweenPillandPill\(Oracle\)\.

### A\.9Potential of Uncertainty\-aware Candidate Expansion

Our defaultPilluses a fixed candidate radiusr=2r=2for all examples\. However, length predictions may have different levels of uncertainty, suggesting that a fixed candidate set may not be optimal for every input\. We therefore first examine whether prediction uncertainty provides a useful signal for dynamically adjusting the candidate radius\.

#### Prediction uncertainty\.

We train five bootstrap length probes and use the standard deviation of their predictions as an uncertainty estimate\. Table[15](https://arxiv.org/html/2609.02108#A1.T15)reports the mean uncertainty grouped by gold span length\.

Table 15:Prediction uncertainty across different gold span lengths\.Prediction uncertainty increases consistently with span length and has a Pearson correlation of0\.320\.32with the gold length\. This suggests that longer or more difficult spans tend to require a wider search neighborhood, while a smaller candidate set may suffice for confident predictions\.

#### Dynamic candidate radius\.

Motivated by this observation, we explore an uncertainty\-aware candidate expansion strategy\. We divide examples into low\-, medium\-, and high\-uncertainty groups according to uncertainty terciles and assign 3, 5, and 7 candidate lengths, respectively\. Compared with the default fixed radiusr=2r=2\(five candidates\), the adaptive strategy improves Pass@1 from 71\.35 to 71\.44\.

Although the improvement is modest on the current benchmark, this result shows that prediction uncertainty can be used to dynamically allocate the candidate budget without degrading performance\. We view uncertainty\-aware candidate expansion as a promising direction for settings with more diverse or longer span\-length distributions\.

### A\.10Additional Comparison Results

#### Comparison with Oracle Best\-Initialized CAL\.

Our main evaluation follows a prior\-free protocol that averages CAL over initial lengthsL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\}, since the optimal initialization is unknown at deployment time and varies across datasets\. To further examine whether our results are driven by this averaging protocol, we comparePillagainst an oracle version of CAL that retrospectively selects its best initial length on each test set\.

Table 16:Comparison with CAL using its best test\-set initialization amongL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\}\. This oracle initialization is selected retrospectively and is not available in practical deployment\.As shown in Table[16](https://arxiv.org/html/2609.02108#A1.T16),Pilloutperforms oracle best\-initialized CAL on four of the six benchmarks, despite requiring no preset initial length\. Moreover, CAL’s optimal initialization varies across datasets, ranging fromL=8L=8toL=32L=32, illustrating the difficulty of choosing a single transferable prior\. On Human\-S and Human\-M, oracle CAL performs better, but this comparison assumes access to test\-set performance when choosing the initialization\. Thus, the results support our main prior\-free evaluation while providing a stronger upper\-bound comparison for CAL\.

#### Comparison with DreamOn\.

DreamOn\([Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12)\)attains variable\-length infilling by fine\-tuning the DLM with explicit length\-changing operations, whereasPillrequires no backbone\-finetuning and leaves the backbone untouched\. We compare on the text infilling tasks, which are out\-of\-distribution for both methods \(Table[17](https://arxiv.org/html/2609.02108#A1.T17)\): DreamOn is trained on code, and for this comparison, our probe is also fitted only on code \(with a separately trained code\-only probe\)\. The result is presented in Table[17](https://arxiv.org/html/2609.02108#A1.T17)\. In this out\-of\-domain setting,Pilloutperforms DreamOn across all of its length configurations \(initial lengthl∈\{1,4,8\}l\\in\\\{1,4,8\\\}\) on both datasets, showing thatPilltransfers effectively across domains without backbone fine\-tuning\. DreamOn’s performance also varies noticeably with the initial length, whereasPillrequires no preset length\. We emphasize that DreamOn operates under a different adaptation regime with task\-specific backbone fine\-tuning, so this comparison is not intended to establish superiority over its specialized fine\-tuned system\.

Table 17:Comparison with DreamOn \(ll: initial length\)\.

### A\.11Quality\-Efficiency Trade\-off

Figure[5](https://arxiv.org/html/2609.02108#A1.F5)visualizes the quality–time trade\-off on Wikitext, with each baseline shown at all four preset lengthsL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\}andPillshown as a single length\-free point\.Pilllies at the upper\-left frontier, attaining the highest BLEU\-2 at a time cost lower than nearly all baseline configurations; every baseline point is strictly dominated\. Sweeping a baseline alongLLtraces a curve that moves rightward \(more time\) without consistently moving upward \(better quality\), illustrating that tuning the preset length cannot recover the operating pointPillreaches without any tuning\.

![Refer to caption](https://arxiv.org/html/2609.02108v1/scatter.png)Figure 5:Quality\-efficiency trade\-off on Wikitext \(BLEU\-2 vs\. wall\-clock time, log\-scaledxx\-axis\)\. Each baseline is annotated with its preset lengthL∈\{4,8,16,32\}L\\in\\\{4,8,16,32\\\};Pillrequires no preset length\.Pillsits at the upper\-left frontier and dominates all baseline configurations\.
### A\.12Additional Related Work

#### Efficient Large Language Models\.

With the rapid development of machine learning\([Xia et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib50);[Ban et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib75);[Roach et al\., 2020](https://arxiv.org/html/2609.02108#bib.bib76);[Yu et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib81);[Zeng et al\., 2025c](https://arxiv.org/html/2609.02108#bib.bib82)\)and foundation models\([Lin et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib49);[Li et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib51)\), recent studies have explored a wide range of applications, including recommendation\([Zeng et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib44);[Zeng et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib43);[Yan et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib72);[Yan et al\., 2024a](https://arxiv.org/html/2609.02108#bib.bib78);[Yang et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib80);[Li et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib79);[Jing et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib71);[Liu et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib84);[Lin et al\., 2026d](https://arxiv.org/html/2609.02108#bib.bib90)\), graph learning\([Wang et al\., 2018](https://arxiv.org/html/2609.02108#bib.bib64);[Wang et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib68);[Du et al\., 2021](https://arxiv.org/html/2609.02108#bib.bib73);[Xu et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib38);[Yu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib57);[Yan et al\., 2021a](https://arxiv.org/html/2609.02108#bib.bib65);[Yan et al\., 2021b](https://arxiv.org/html/2609.02108#bib.bib66);[Lin et al\., 2024b](https://arxiv.org/html/2609.02108#bib.bib89)\), reasoning\([Zeng et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib48);[Zeng et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib55);[Yan et al\., 2023a](https://arxiv.org/html/2609.02108#bib.bib69);[Yan et al\., 2023b](https://arxiv.org/html/2609.02108#bib.bib70)\), and multi\-modality tasks\([Zhang et al\., 2025c](https://arxiv.org/html/2609.02108#bib.bib37);[Zhang et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib36);[Bei et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib20);[Lin et al\., 2025c](https://arxiv.org/html/2609.02108#bib.bib91)\)\. Meanwhile, as foundation models continue to scale, their increasing computational and memory costs have motivated extensive research on efficient large language models\. Existing studies improve efficiency from different perspectives, including small language model learning\([Xing et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib53);[Lin et al\., 2026c](https://arxiv.org/html/2609.02108#bib.bib41);[Yan et al\., 2025](https://arxiv.org/html/2609.02108#bib.bib83)\), model compression and quantization\([Lin et al\., 2024a](https://arxiv.org/html/2609.02108#bib.bib34);[Lin et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib35);[Lin et al\., 2025b](https://arxiv.org/html/2609.02108#bib.bib39);[Yang et al\., 2026b](https://arxiv.org/html/2609.02108#bib.bib52);[Yang et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib54);[Zhang et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib61)\), efficient attention and long\-context modeling\([Xiao et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib60);[Zhang et al\., 2023](https://arxiv.org/html/2609.02108#bib.bib62)\), KV\-cache optimization\([Liu et al\., 2024b](https://arxiv.org/html/2609.02108#bib.bib63);[Liu et al\., 2024a](https://arxiv.org/html/2609.02108#bib.bib56)\), and efficient model architectures\([Lai et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib85);[Zhu et al\., 2025c](https://arxiv.org/html/2609.02108#bib.bib86)\)\. These approaches aim to reduce the computational and memory overhead of large language models while preserving their modeling capabilities\. One can refer to\([Lin et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib40)\)\.

#### Efficient Decoding and Inference\.

Recent advances in artificial intelligence\([Zeng et al\., 2023b](https://arxiv.org/html/2609.02108#bib.bib47);[Zeng et al\., 2024](https://arxiv.org/html/2609.02108#bib.bib46);[Yan et al\., 2024b](https://arxiv.org/html/2609.02108#bib.bib74);[Yan et al\., 2024c](https://arxiv.org/html/2609.02108#bib.bib77);[Zeng et al\., 2023a](https://arxiv.org/html/2609.02108#bib.bib45);[Ban et al\., 2021](https://arxiv.org/html/2609.02108#bib.bib67)\)have led to increasingly capable but computationally intensive generative models[Huang et al\. \(2025\)](https://arxiv.org/html/2609.02108#bib.bib29), making efficient inference an important research problem\. Existing studies have explored accelerated decoding\([Xu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib42)\), parallel decoding\([Cheng et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib87)\), speculative decoding\([Chen et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib58);[Li et al\., 2026a](https://arxiv.org/html/2609.02108#bib.bib59)\), and adaptive inference strategies\([Manvi et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib88)\)to reduce sequential computation and redundant model evaluations\. Such efficiency is particularly important for diffusion language models, which generate sequences through iterative denoising and may require repeated forward passes during inference\. In adaptive\-length infilling, existing approaches further introduce additional computation to determine or adjust the generation length\. In contrast, PILL directly predicts the target length, decodes multiple nearby length candidates in parallel, and selects the final candidate with a single additional scoring pass, enabling adaptive\-length infilling with minimal inference overhead\.

### A\.13Details of Datasets

All datasets and models used in this work are publicly available under licenses permitting research use, and we use them accordingly\. Also, all evaluation results for test dataset are from a single run with deterministic decoding\. Following all prior adaptive\-length infilling methods we compare against\([Li et al\., 2025a](https://arxiv.org/html/2609.02108#bib.bib10);[Liu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib11);[Wu et al\., 2026](https://arxiv.org/html/2609.02108#bib.bib12)\), we report results from a single run with deterministic decoding, which keeps the comparison protocol identical across methods\.

Our use is consistent with these resources’ intended research use, and the infilling benchmarks we derive from them are likewise intended for research only\.

#### HumanEval\.

HumanEval\-Infilling\([Bavarian et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib6)\)is an infilling benchmark derived from the original HumanEval code generation benchmark\. It evaluates whether a model can complete missing code spans given both the left and right contexts\. Following prior work, the benchmark contains several infilling settings, including single\-line, multi\-line, random\-span, and random\-span\-light, which cover different granularities and locations of missing code\.

#### MBPP\.

Mostly Basic Python Problems \(MBPP\)\([Austin et al\., 2021b](https://arxiv.org/html/2609.02108#bib.bib7)\)is a Python program synthesis benchmark consisting of 974 crowd\-sourced programming problems\. Each example contains a natural language task description, a reference Python solution, and test cases for functional correctness\. The problems are designed to be solvable by entry\-level programmers and mainly cover basic programming concepts, standard library usage, and simple algorithmic reasoning\.

#### Py150\.

Py150\([Raychev et al\., 2016](https://arxiv.org/html/2609.02108#bib.bib27)\)is a large\-scale Python corpus collected from GitHub repositories\. The dataset contains 150k Python files, split into 100k training files and 50k evaluation files\. The original release provides parsed Python ASTs as well as source files, and applies filtering steps such as removing duplicate files, excluding repository forks, keeping files that can be parsed, and focusing on permissive licenses\. We use Py150 as a source of natural Python code for constructing length\-probe training samples\.

#### LeetCode\.

The LeetCode dataset111https://huggingface\.co/datasets/greengerong/leetcode\.contains algorithmic programming problems from LeetCode, including problem titles, difficulty levels, descriptions, and reference solutions in multiple programming languages such as Python, Java, C\+\+, and JavaScript\. Compared with MBPP, these examples are generally more algorithmic and closer to interview\-style coding tasks\.

#### CodeContests\.

CodeContests\([Li et al\., 2022b](https://arxiv.org/html/2609.02108#bib.bib28)\)is a competitive programming dataset released by DeepMind and used in the development of AlphaCode\. It contains programming problems collected from multiple online judge platforms, such as Aizu, AtCoder, CodeChef, Codeforces, and HackerEarth\. Each problem includes natural language statements, input\-output test cases, and both correct and incorrect human submissions in multiple programming languages\. We use it as a source of challenging algorithmic code\.

#### MultiPL\-E\.

MultiPL\-E\([Cassano et al\., 2022](https://arxiv.org/html/2609.02108#bib.bib8)\)is a multilingual code generation benchmark that translates unit\-test\-driven Python benchmarks, including HumanEval and MBPP, into 18 additional programming languages\. It is designed to evaluate whether code generation models generalize beyond Python and across diverse programming languages\. In our setting, MultiPL\-E provides multilingual code examples and evaluation\-style prompts for constructing infilling instances\.

#### WikiText\.

WikiText\([Merity et al\., 2016](https://arxiv.org/html/2609.02108#bib.bib9)\)is a language modeling dataset extracted from verified Good and Featured articles on Wikipedia\. Compared with heavily preprocessed corpora such as Penn Treebank, WikiText preserves original casing, punctuation, numbers, and long\-form article structure, making it suitable for evaluating long\-range dependencies in natural language modeling\. We use WikiText as a source of high\-quality natural text for non\-code infilling samples\.

#### C4\.

The Colossal Clean Crawled Corpus \(C4\)\([Raffel et al\., 2020](https://arxiv.org/html/2609.02108#bib.bib30)\)is a large\-scale cleaned web corpus derived from Common Crawl and introduced with the T5 framework\. It contains diverse English web text after a series of filtering and cleaning steps\. We use C4 as a broad\-domain natural language corpus to improve the coverage of text\-side length\-probe training examples\.

#### arXiv\.

The arXiv dataset222https://huggingface\.co/datasets/UniverseTBD/arxiv\-abstracts\-large\.provides metadata and abstracts for scholarly articles across domains such as physics, computer science, statistics, electrical engineering, quantitative biology, and economics\. We use arXiv abstracts as scientific\-domain natural text, which complements the general\-domain text from C4 and WikiText\.

### A\.14Potential Risks

Pillis a backbone\-frozen method for diffusion language models: it does not modify the underlying backbone, and its lightweight probe is trained only on public corpora\. We do not expect any personally identifying information in these resources\. As our method builds on pretrained DLMs, it may inherit biases or factual errors present in these backbones, and the infilled spans it produces should be reviewed before use in sensitive applications\.

All datasets used in our experiments are commonly used, publicly available research datasets\. We rely on their released and curated versions rather than collecting data directly from individuals, and we do not use or intentionally retain personally identifying information\. We additionally inspected the data\-construction pipeline and did not identify fields intended to contain personal identifiers or offensive content\.

### A\.15Declaration of AI Assistance

We used AI tools for language polishing and part of the experimental implementation\. These tools did not contribute to the idea, analysis, results, or scientific claims\.