基于 Seq-Drifting 的一步文本生成

# One-Step Text Generation by Seq-Drifting
Source: [https://arxiv.org/html/2610.12879](https://arxiv.org/html/2610.12879)
###### Abstract

Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high\-quality text\. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass\. We proposeSeq\-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text\. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq\-Drifting builds a drift field whose position\-wise targets adapt to the current generated context\. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences\. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference\. Empirically, Seq\-Drifting achieves a generative perplexity of36\.0136\.01on LM1B in one step and performs competitively in both unconditional and conditional generation\. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open\-ended generation\. These results suggest that drifting over token sequences offers a promising path toward effective one\-step text generation\. Our code is available at[https://github\.com/jyliu\-ai/Seq\-Drifting](https://github.com/jyliu-ai/Seq-Drifting)\.

00footnotetext:1Hong Kong Institute of AI for Science, City University of Hong Kong,2Department of Data Science, City University of Hong Kong,3Shenzhen University,4State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China,5Department of Statistics, University of Oxford\.00footnotetext:\*Correspondence to: Ning Miao \(ningmiao@cityu\.edu\.hk\)Figure 1:Seq\-Driftingachieves lower generative perplexity than prior few\-step distilled DLMs, and is on par with or better than multi\-step DLMs, with a44–11×11\\timesspeedup over the few\-step baselines and a900900–2500×2500\\timesspeedup over the multi\-step ones in wall\-clock time\. \(Gen\-PPL under GPT\-2 Large; dataset: LM1B,T=128T\{=\}128\. Detailed comparison in Figure[3](https://arxiv.org/html/2610.12879#S5.F3)and Table[6](https://arxiv.org/html/2610.12879#A2.T6)\.\)## 1Introduction

Autoregressive \(AR\) language models are the dominant paradigm for text generation and have shown strong capabilities across a wide range of domains\([Radford et al\., 2019](https://arxiv.org/html/2610.12879#bib.bib13);[Brown et al\., 2020](https://arxiv.org/html/2610.12879#bib.bib14)\)\. At inference, however, they generate one token at a time, making sequential decoding a fundamental latency bottleneck that grows with sequence length\. Non\-autoregressive alternatives such as diffusion language models \(DLM\) generate multiple tokens in parallel\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12879#bib.bib5);[Li et al\., 2022](https://arxiv.org/html/2610.12879#bib.bib17);[Hu et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib21);[Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\), but they still rely on multiple refinement steps to produce high\-quality samples\. Distillation methods such asSDTT\([Deschenaux and Gulcehre, 2025](https://arxiv.org/html/2610.12879#bib.bib8)\), Di4C\([Hayakawa et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib9)\), and FMLM\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\)reduce the number of steps, but generation quality remains limited at small sampling budgets\. A key question, then, is whether we can directly learn high\-quality one\-step generation through a single forward pass from noise to text\.

Most generative models can be seen as a map from a noise vector to data\. Because this map is highly multimodal, previous methods often need to decompose it into many small steps\. Drifting\([Deng et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib1)\)offers a different route, learning the map directly as a single forward pass in continuous domains\. Rather than learning a trajectory that is unrolled at inference, a drifting model evolves its pushforward map during training\. As the parameters are updated, each noise sample is carried a little closer toward the data distribution, a movement governed by a drifting field that vanishes once the generated and data distributions match\. At convergence the map can reach a fixed point, so a single forward pass can turn noise into a sample without an iterative sampler at inference\.

Although drifting has shown promising results on images, transferring it to text is far from straightforward\. A text sample is a sequence in which every position contains a token drawn from a finite vocabulary\. Since valid text sequences are composed of discrete token choices, a continuous drift step will generally land between valid token embeddings rather than on one\. The drift may therefore operate in regions containing no valid token\. The space is also semantically non\-smooth, as a single substituted token can overturn the meaning of a sentence or make it incoherent, which means nearness in embedding space does not necessarily imply nearness in meaning\. One might hope to sidestep these issues by drifting a single sentence\-level embedding, where the space is continuous\. However, this only relocates the difficulty, because mapping such a vector back to a faithful token sequence is itself a hard sequence\-decoding problem, and a single vector exposes none of the per\-position targets a drift step needs\.

In this work we introduceSeq\-Drifting, which adapts drifting to sequences of continuous token representations and learns the noise\-to\-text map in a single forward pass\. Our core idea is to provide each position with supervision from a context\-dependent set of plausible tokens\. This support set anchors the drift to actual vocabulary tokens and identifies which choices are compatible with the current generated context\. Seq\-Drifting then attracts each output toward these tokens, allowing different noise inputs to select different valid alternatives without pulling them all toward an average representation\. Because the support tokens depend on the generated context, the supervision couples token choices across positions and guides the generator toward coherent sequences\.

Seq\-Drifting establishes a new framework for one\-step text generation and brings three advantages\. First, it generates a sample in a single forward pass, so its inference cost does not grow with a step budget as in multi\-step diffusion and few\-step distillation, while matching their quality at a fraction of the inference cost\. Second, it addresses the ambiguity of one\-step generation without relying on iterative refinement\. Rather than assigning each noise input a unique target sequence, it allows different noise to naturally settle on their own coherent continuations while preserving multiple generation modes\. Finally, it unifies two ways of learning the one\-step map by constructing support sets either from corpus retrieval or scoring by a frozen language model, which opens the possibility of turning an existing autoregressive language model into a one\-step generator\.

Empirically, we show that Seq\-Drifting combines competitive generation quality with substantially lower inference cost\. For example, in Figure[1](https://arxiv.org/html/2610.12879#S0.F1), it reaches a generative perplexity of36\.0136\.01with an11\.2×11\.2\\timeswall\-clock speedup over the strongest four\-step distilled baseline, even outperforming several DLMs that use10241024sampling steps\. This advantage extends to prefix\-conditioned generation, such as machine translation, summarization, and reasoning, which broadens its applicability beyond language continuation\. Overall, these results demonstrate the feasibility of directly learning a one\-step text generator and point to a new direction for diffusion language models\.

## 2Related Work

Diffusion language models \(DLMs\) provide an iterative non\-autoregressive alternative to the token\-by\-token decoding of autoregressive models\([Radford et al\., 2019](https://arxiv.org/html/2610.12879#bib.bib13);[Brown et al\., 2020](https://arxiv.org/html/2610.12879#bib.bib14)\), and can be broadly categorized into discrete and continuous formulations\([Li et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib22)\)\.

Discrete DLMs operate directly over categorical tokens\. D3PM\([Austin et al\., 2021](https://arxiv.org/html/2610.12879#bib.bib12)\)provides a general framework for discrete corruption, while MDLM\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12879#bib.bib5)\)uses masking to learn token recovery from partially observed sequences\. LLaDA\([Nie et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib16)\)further demonstrates masked diffusion at the scale of large language models\. Unlike standard irreversible unmasking, the uniform\-state formulation of Duo\([Sahoo et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib4)\)allows previously generated tokens to be revised\. Block diffusion methods\([Arriola et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib15);[Arriola et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib23)\)instead combine parallel generation within blocks with autoregressive conditioning across blocks\. However, parallel per\-position predictions do not by themselves capture the joint uncertainty of tokens updated simultaneously, making inter\-token dependencies particularly important when sampling is compressed into large jumps\.

Continuous DLMs instead model language generation in a continuous representation space\. Diffusion\-LM\([Li et al\., 2022](https://arxiv.org/html/2610.12879#bib.bib17)\)and DiffuSeq\([Gong et al\., 2023](https://arxiv.org/html/2610.12879#bib.bib18)\)operate on token embeddings, whereas SSD\-LM\([Han et al\., 2023](https://arxiv.org/html/2610.12879#bib.bib19)\)and TESS\([Mahabadi et al\., 2024](https://arxiv.org/html/2610.12879#bib.bib20)\)use simplex\-based representations\. Beyond the state space, an important distinction is how discrete supervision constrains the trajectory\. FLM\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\)learns continuous denoising over one\-hot encodings with token\-level cross\-entropy, and LangFlow\([Chen et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib24)\)connects embedding\-space diffusion to Flow Matching through Bregman divergence\. ELF\([Hu et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib21)\)instead reserves token\-level supervision for the final decoding step, allowing the preceding dynamics to remain in continuous embedding space\. These advances improve continuous generation, but an effective trajectory still requires tens or even hundreds of sampling steps at inference\.

Few\-step methods address this problem by learning to traverse or replace an existing sampling trajectory\. SDTT\([Deschenaux and Gulcehre, 2025](https://arxiv.org/html/2610.12879#bib.bib8)\)compresses denoising through self\-distillation, Di4C\([Hayakawa et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib9)\)explicitly addresses the dimensional correlations lost in factorized reverse transitions, and FMLM\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\)distills a continuous flow into a map between distant timepoints\. Drifting\([Deng et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib1)\)provides a different training formulation, evolving the generator’s outputs through attraction and repulsion while retaining a single forward pass at inference\. Seq\-Drifting extends this formulation to token sequences\. Context\-dependent, per\-position supervision supplies plausible token targets for its current outputs\. The support sets can draw on corpus retrieval or a frozen language model, while the objective directly trains the one\-step map without reproducing a diffusion or flow trajectory\.

## 3Preliminaries

We consider a one\-step generatorGθG\_\{\\theta\}that maps a noise inputϵ∼pϵ\\epsilon\\sim p\_\{\\epsilon\}to an output𝒙=Gθ​\(ϵ\)\\boldsymbol\{x\}=G\_\{\\theta\}\(\\epsilon\)with distributionqθq\_\{\\theta\}, and training seeksqθ≈pdataq\_\{\\theta\}\\approx p\_\{\\mathrm\{data\}\}\. Since the parameters are updated step by step, the same noiseϵ\\epsilonproduces a slightly different output after each step\. The displacement

Δ​𝒙i=Gθi\+1​\(ϵ\)−Gθi​\(ϵ\)\\Delta\\boldsymbol\{x\}\_\{i\}\\;=\\;G\_\{\\theta\_\{i\+1\}\}\(\\epsilon\)\-G\_\{\\theta\_\{i\}\}\(\\epsilon\)\(1\)comes entirely from the parameter update, and is called the*drift*of𝒙\\boldsymbol\{x\}at iterationii\. A*drifting field*Vp,qV\_\{p,q\}\([Deng et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib1)\)prescribes this drift sample by sample, withppfixed to the targetpdatap\_\{\\mathrm\{data\}\}andqqto the current modelqθiq\_\{\\theta\_\{i\}\},

𝒙i\+1=𝒙i\+Vpdata,qθi​\(𝒙i\)\.\\boldsymbol\{x\}\_\{i\+1\}=\\boldsymbol\{x\}\_\{i\}\+V\_\{p\_\{\\mathrm\{data\}\},\\,q\_\{\\theta\_\{i\}\}\}\(\\boldsymbol\{x\}\_\{i\}\)\.\(2\)The field vanishes when the two distributions agree,Vp,p=𝟎V\_\{p,p\}=\\boldsymbol\{0\}, soqθ=pdataq\_\{\\theta\}=p\_\{\\mathrm\{data\}\}is a fixed point of training\. Regressing each output onto its drifted target with a stop gradient turns this into the loss,

ℒ⁡\(θ\)=𝔼ϵ​‖Gθ​\(ϵ\)−sg⁡\(Gθ​\(ϵ\)\+V⁡\(Gθ​\(ϵ\)\)\)‖2,\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{\\epsilon\}\\big\\\|\\,G\_\{\\theta\}\(\\epsilon\)\-\\operatorname\{sg\}\\\!\\big\(G\_\{\\theta\}\(\\epsilon\)\+V\(G\_\{\\theta\}\(\\epsilon\)\)\\big\)\\big\\\|^\{2\},\(3\)whose value equals𝔼ϵ​‖V⁡\(Gθ​\(ϵ\)\)‖2\\mathbb\{E\}\_\{\\epsilon\}\\\|V\(G\_\{\\theta\}\(\\epsilon\)\)\\\|^\{2\}, so minimizing it drives the drift toward zero\.

A concrete field comes from a kernel mean\-shift\. For a sample𝒙\\boldsymbol\{x\}, a positive set𝒫\\mathcal\{P\}drawn frompdatap\_\{\\mathrm\{data\}\}attracts and a negative set𝒩\\mathcal\{N\}drawn fromqθq\_\{\\theta\}repels,

V⁡\(𝒙\)=∑𝒚∈𝒫k~​\(𝒙,𝒚\)​\(𝒚−𝒙\)⏟attraction−∑𝒚∈𝒩k~​\(𝒙,𝒚\)​\(𝒚−𝒙\)⏟repulsion,V\(\\boldsymbol\{x\}\)=\\underbrace\{\\sum\_\{\\boldsymbol\{y\}\\in\\mathcal\{P\}\}\\tilde\{k\}\(\\boldsymbol\{x\},\\boldsymbol\{y\}\)\\,\(\\boldsymbol\{y\}\-\\boldsymbol\{x\}\)\}\_\{\\text\{attraction\}\}\\;\-\\;\\underbrace\{\\sum\_\{\\boldsymbol\{y\}\\in\\mathcal\{N\}\}\\tilde\{k\}\(\\boldsymbol\{x\},\\boldsymbol\{y\}\)\\,\(\\boldsymbol\{y\}\-\\boldsymbol\{x\}\)\}\_\{\\text\{repulsion\}\},\(4\)wherek~\\tilde\{k\}is a similarity kernel normalized over each set\. The attraction moves𝒙\\boldsymbol\{x\}toward the data it already resembles and the repulsion moves it off the other samples, and the two cancel once the generated and data distributions agree\.

#### Defining the drift target\.

The field in Eq\. \([4](https://arxiv.org/html/2610.12879#S3.E4)\) specifies how a generated sample should move during training\. Positive samples provide the destinations for attraction, while other generated samples provide the reference points for repulsion\. The similarity kernel determines which samples contribute most strongly to each force\. Combining the forces gives a displacement, which defines the stop\-gradient target𝒙\+V⁡\(𝒙\)\\boldsymbol\{x\}\+V\(\\boldsymbol\{x\}\)in Eq\. \([3](https://arxiv.org/html/2610.12879#S3.E3)\)\. The generator learns to approach this target through parameter updates\. Adapting drifting to a new domain therefore requires suitable positive and negative samples, a geometry for comparing them, and a rule for converting the field into a training target\.

#### Difficulties in applying drifting to text\.

A text sequence is composed of discrete token choices, which creates three difficulties for drifting\. First, valid outputs lie in a discrete and semantically non\-smooth space\. A continuous displacement can change the decoded token, but proximity between embeddings does not guarantee that the new token fits the surrounding sequence\. Second, the tokens that form a coherent continuation depend on the preceding sequence\. However, a training example supplies only the continuation observed after its own context, which may differ substantially from the generator’s current context\. Using that observed continuation directly as a drift target can therefore yield a weak or misaligned update\. Third, under cosine decoding, radial separation can increase Euclidean distances without changing token identity\. Unconstrained repulsion can therefore spend its force on growing norms while producing no additional token\-level diversity\.

![Refer to caption](https://arxiv.org/html/2610.12879v1/figure_drifting.png)Figure 2:Seq\-Drifting\.In one forward pass the generatorGθG\_\{\\theta\}maps a noise vectorϵ∼𝒩⁡\(0,IC\)\\epsilon\\\!\\sim\\\!\\mathcal\{N\}\(0,I\_\{C\}\)\(optionally with a prefixp1,…,pMp\_\{1\},\\dots,p\_\{M\}\) to token embeddings𝒉1,…,𝒉T\\boldsymbol\{h\}\_\{1\},\\dots,\\boldsymbol\{h\}\_\{T\}, each projected onto the unit sphere as𝒆t=𝒉t/‖𝒉t‖∈SD−1\\boldsymbol\{e\}\_\{t\}=\\boldsymbol\{h\}\_\{t\}/\\\|\\boldsymbol\{h\}\_\{t\}\\\|\\in S^\{D\-1\}\. Supervision is applied*per position*\. A positive set𝒮t\\mathcal\{S\}\_\{t\}of plausible tokens attracts𝒆t\\boldsymbol\{e\}\_\{t\}, while the negative setsNtoutN\_\{t\}^\{\\mathrm\{out\}\}\(other samples\) andNtinN\_\{t\}^\{\\mathrm\{in\}\}\(other positions of the same sample\) repel it, weighted by a similarity kernelφ\\varphiand strengthsλin/out\\lambda\_\{\\mathrm\{in\}/\\mathrm\{out\}\}\. The two combine into the drift force𝑭t\\boldsymbol\{F\}\_\{t\}; a single geodesic step along𝑭t\\boldsymbol\{F\}\_\{t\}gives the stop\-gradient targetg​o​a​ltgoal\_\{t\}, onto which the generator is regressed,ℒ=𝔼​‖𝒆t−sg⁡\(g​o​a​lt\)‖2\\mathcal\{L\}=\\mathbb\{E\}\\\|\\boldsymbol\{e\}\_\{t\}\-\\mathrm\{sg\}\(goal\_\{t\}\)\\\|^\{2\}\.

## 4Seq\-Drifting

To address the difficulties above, Seq\-Drifting represents a sequence with per\-position token representations and provides each position with a context\-dependent set of plausible tokens\. These representations and targets are placed in a shared spherical space, where the resulting drift remains anchored to valid vocabulary while the noise\-to\-text map is learned directly\. Figure[2](https://arxiv.org/html/2610.12879#S3.F2)summarizes the framework\. We first describe the one\-step generator and its representation space, then introduce support\-set construction, the attractive and repulsive forces, and the training objective\.

### 4\.1Spherical one\-step generator

We generate a sequence of continuous representations, each one corresponding to a token\. This allows the generator to produce all positions jointly while exposing each token choice to local drift supervision\. Specifically, our generatorGθG\_\{\\theta\}maps a shared noise vectorϵ∈ℝC\\epsilon\\in\\mathbb\{R\}^\{C\}toTTtoken representations\{𝒉t\}t=1T\\\{\\boldsymbol\{h\}\_\{t\}\\\}\_\{t=1\}^\{T\}, with𝒉t∈ℝD\\boldsymbol\{h\}\_\{t\}\\in\\mathbb\{R\}^\{D\}, in one forward pass\. Architectural details are given in Appendix[A\.1](https://arxiv.org/html/2610.12879#A1.SS1)\. Then, we recover tokens by cosine similarity to a frozen vocabulary embedding table\. Since this decoding rule depends only on direction, we normalize both the output vectors𝒉t\\boldsymbol\{h\}\_\{t\}and the vocabulary embeddings𝒘v\\boldsymbol\{w\}\_\{v\}, i\.e\.,𝒆t=𝒉t/‖𝒉t‖∈SD−1\\boldsymbol\{e\}\_\{t\}=\\boldsymbol\{h\}\_\{t\}/\\\|\\boldsymbol\{h\}\_\{t\}\\\|\\in S^\{D\-1\}and‖𝒘v‖=1\\\|\\boldsymbol\{w\}\_\{v\}\\\|=1\. Attraction, repulsion, and token recovery then depend on the same directional geometry\. The decoded token istokt=arg⁡maxv​𝒆t⊤​𝒘v\\mathrm\{tok\}\_\{t\}=\\arg\\max\_\{v\}\\boldsymbol\{e\}\_\{t\}^\{\\top\}\\boldsymbol\{w\}\_\{v\}, with non\-text tokens excluded\. This operation produces the final text at inference, requiring no separate vocabulary softmax\.

### 4\.2Context\-dependent token support

The main challenge in selecting positives is to provide useful guidance for the sequence the generator currently produces\. A reference sentence may differ from that sequence at many positions, so its tokens need not be suitable targets for the corresponding positions\. We instead construct positive targets at each position, asking which tokens are plausible given the generated context\.

For positiontt, we collect the positive targets into a support set𝒮t\\mathcal\{S\}\_\{t\}\(i\.e\.,𝒫\\mathcal\{P\}in §[3](https://arxiv.org/html/2610.12879#S3)\), with each candidate represented by its unit embedding\. The set supplies concrete token targets without prescribing a single continuation\. It separates contextual plausibility from the choice made by an individual sample\. The support set determines which tokens are eligible for attraction, while the local kernel introduced next determines how strongly each candidate attracts the current output\.

There are two natural ways to form𝒮t\\mathcal\{S\}\_\{t\}\. The*off\-policy*route reads it from the corpus, matching each generation against real sequences and taking the tokens they place at positiontt, so the candidates come from observed continuations\. The*on\-policy*route queries a frozen language model \(hereafter, the*support model*\) on the generator’s current prefixes to propose contextually plausible next\-token candidates, where one forward pass over the generated sequence returns a next\-token distributionptp\_\{t\}at every position at once\.𝒮t\\mathcal\{S\}\_\{t\}collects its most probable tokens \(the topkk\), so the target adapts to wherever the current generator sits\. The same construction also applies to a closed\-source support model, which can be queried multiple times to sample continuations from the generated context and form𝒮t\\mathcal\{S\}\_\{t\}\. We empirically find the on\-policy support the stronger of the two \(§[5\.4](https://arxiv.org/html/2610.12879#S5.SS4)\) and adopt it by default\. The support model is queried only during training to construct the support set𝒮t\\mathcal\{S\}\_\{t\}, and is not required at inference\. The resulting supervision is set\-valued and context\-dependent rather than a fixed per\-position target\.

### 4\.3Angular mode\-seeking attraction

The attraction pulls each position toward the supported token nearest its own embedding\. We measure closeness by angular distanced⁡\(𝒆t,𝒘\)=1−𝒆t⊤​𝒘d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\)=1\-\\boldsymbol\{e\}\_\{t\}^\{\\top\}\\boldsymbol\{w\}and form a low\-temperature similarity kernelφ\\varphi, normalized over the support set,

φ⁡\(𝒘,𝒆t\)∝exp⁡\(−d⁡\(𝒆t,𝒘\)st​τ\),𝒖t=∑𝒘∈𝒮tφ⁡\(𝒘,𝒆t\)​\(𝒘−𝒆t\),st=std𝒘∈𝒮t⁡d⁡\(𝒆t,𝒘\),\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\\;\\propto\\;\\exp\\\!\\Big\(\\\!\-\\frac\{d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\)\}\{s\_\{t\}\\,\\tau\}\\Big\),\\qquad\\boldsymbol\{u\}\_\{t\}=\\\!\\\!\\sum\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\}\\\!\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\\,\(\\boldsymbol\{w\}\-\\boldsymbol\{e\}\_\{t\}\),\\qquad s\_\{t\}=\\operatorname\{std\}\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\}d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\),\(5\)wherests\_\{t\}is the standard deviation ofd⁡\(𝒆t,𝒘\)d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\)for candidates at positionttandτ\\tauis a temperature\. Normalizing bysts\_\{t\}accounts for each position’s candidate spread, whileτ<1\\tau<1sharpens the kernel around nearby candidates\. It is therefore mode\-seeking and concentrates the weight on the near supported token, so different noise inputs commit to different tokens rather than to the blurred centroid of𝒮t\\mathcal\{S\}\_\{t\}\. The mean\-shift then gives the attractive force𝒖t=∑𝒘∈𝒮tφ⁡\(𝒘,𝒆t\)​\(𝒘−𝒆t\)\\boldsymbol\{u\}\_\{t\}=\\sum\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\}\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{w\}\-\\boldsymbol\{e\}\_\{t\}\)\.

### 4\.4Per\-position angular repulsion

The repulsion stops distinct noise inputs from collapsing to the same output\. It acts per position against two negative sets, i\.e\.,𝒩≜𝒩t=\{Ntout,Ntin\}\\mathcal\{N\}\\triangleq\\mathcal\{N\}\_\{t\}=\\\{N\_\{t\}^\{\\mathrm\{out\}\},N\_\{t\}^\{\\mathrm\{in\}\}\\\}\. The cross\-sample setNtoutN\_\{t\}^\{\\mathrm\{out\}\}contains the position\-ttrepresentations from the other samples in the batch\. The intra\-sequence setNtinN\_\{t\}^\{\\mathrm\{in\}\}contains the representations at the other positions of the same sequence, and pushing away from it suppresses within\-sentence repetition\. Each uses the same angular kernelφ\\varphias the attraction,

𝒓tout=∑𝒏∈Ntoutφ⁡\(𝒏,𝒆t\)​\(𝒆t−𝒏\),𝒓tin=∑𝒏∈Ntinφ⁡\(𝒏,𝒆t\)​\(𝒆t−𝒏\)\.\\boldsymbol\{r\}^\{\\mathrm\{out\}\}\_\{t\}=\\\!\\\!\\sum\_\{\\boldsymbol\{n\}\\in N\_\{t\}^\{\\mathrm\{out\}\}\}\\\!\\\!\\varphi\(\\boldsymbol\{n\},\\boldsymbol\{e\}\_\{t\}\)\\,\(\\boldsymbol\{e\}\_\{t\}\-\\boldsymbol\{n\}\),\\qquad\\boldsymbol\{r\}^\{\\mathrm\{in\}\}\_\{t\}=\\\!\\\!\\sum\_\{\\boldsymbol\{n\}\\in N\_\{t\}^\{\\mathrm\{in\}\}\}\\\!\\\!\\varphi\(\\boldsymbol\{n\},\\boldsymbol\{e\}\_\{t\}\)\\,\(\\boldsymbol\{e\}\_\{t\}\-\\boldsymbol\{n\}\)\.\(6\)Both are then rescaled to the magnitude of the attraction before the update weights them byλout\\lambda\_\{\\mathrm\{out\}\}andλin\\lambda\_\{\\mathrm\{in\}\}\. Restricting the cross\-sample setNtoutN\_\{t\}^\{\\mathrm\{out\}\}to the samples that share a context is studied in §[5\.4](https://arxiv.org/html/2610.12879#S5.SS4)\.

On the sphere, repulsion separates outputs by changing their directions rather than increasing their norms\. This prevents radial growth and aligns repulsion with cosine decoding\. Together with the bounded geodesic update in §[4\.5](https://arxiv.org/html/2610.12879#S4.SS5), it defines the displacement of each training target\.

### 4\.5Geodesic update and training

The update sums the attraction and the two repulsions into one drift force,𝑭t=𝒖t\+λout​𝒓tout\+λin​𝒓tin\\boldsymbol\{F\}\_\{t\}=\\boldsymbol\{u\}\_\{t\}\+\\lambda\_\{\\mathrm\{out\}\}\\,\\boldsymbol\{r\}^\{\\mathrm\{out\}\}\_\{t\}\+\\lambda\_\{\\mathrm\{in\}\}\\,\\boldsymbol\{r\}^\{\\mathrm\{in\}\}\_\{t\}, projects it onto the tangent plane at𝒆t\\boldsymbol\{e\}\_\{t\}to drop the radial part,𝑭t′=𝑭t−\(𝑭t⊤​𝒆t\)​𝒆t\\boldsymbol\{F\}^\{\\prime\}\_\{t\}=\\boldsymbol\{F\}\_\{t\}\-\(\\boldsymbol\{F\}\_\{t\}^\{\\top\}\\boldsymbol\{e\}\_\{t\}\)\\,\\boldsymbol\{e\}\_\{t\}, and applies it as one step along the great circle by the exponential map,

g​o​a​lt=cos⁡\(α\)​𝒆t\+sin⁡\(α\)​𝑭t′‖𝑭t′‖,α=min⁡\(‖𝑭t′‖,αmax\)\.goal\_\{t\}=\\cos\(\\alpha\)\\,\\boldsymbol\{e\}\_\{t\}\+\\sin\(\\alpha\)\\,\\frac\{\\boldsymbol\{F\}^\{\\prime\}\_\{t\}\}\{\\\|\\boldsymbol\{F\}^\{\\prime\}\_\{t\}\\\|\},\\qquad\\alpha=\\min\\\!\\big\(\\\|\\boldsymbol\{F\}^\{\\prime\}\_\{t\}\\\|,\\ \\alpha\_\{\\max\}\\big\)\.\(7\)Here we use the norm of𝑭t′\\boldsymbol\{F\}^\{\\prime\}\_\{t\}as the arc length of the geodesic step, capped byαmax\\alpha\_\{\\max\}to prevent overshooting\. Treatingg​o​a​ltgoal\_\{t\}as a fixed target during differentiation, the loss of Eq\. \([3](https://arxiv.org/html/2610.12879#S3.E3)\) becomes the squared distance𝔼​‖𝒆t−sg⁡\(g​o​a​lt\)‖2\\mathbb\{E\}\\,\\\|\\boldsymbol\{e\}\_\{t\}\-\\operatorname\{sg\}\(goal\_\{t\}\)\\\|^\{2\}, which is bounded because both points lie on the sphere\. Appendix[A\.2](https://arxiv.org/html/2610.12879#A1.SS2)analyzes the conditions under which the field vanishes as the temperature approaches zero\. Appendix[A\.3](https://arxiv.org/html/2610.12879#A1.SS3)provides the training pseudocode for the on\-policy support\-set implementation\.

#### Conditional generation\.

For conditional generation, we additionally provide the generator with the conditioning token embeddings, which the output positions access through cross\-attention\. The support model also receives the conditioning sequence followed by the generated continuation to provide token support sets\. To stabilize learning this conditional map from scratch, we first train with an unconditional warm\-up\. During this phase, the context is filled with a constant padding token \(e\.g\.,<\|endoftext\|\>for GPT\-2\) under normal attention, which keeps the conditioning path active while the generator learns fluent text\. We then replace the constant context with the real prefix, allowing the generator to learn context dependence from this fluent initialization\.

## 5Experiments

### 5\.1Setup

#### Baselines\.

Our main comparison is against the*few\-step distilled models*that compress a multi\-step sampler into a handful of steps, including FMLM\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\), Duo\+DCD\([Sahoo et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib4)\), MDLM\+SDTT\([Deschenaux and Gulcehre, 2025](https://arxiv.org/html/2610.12879#bib.bib8)\), and Duo/MDLM\+Di4C\([Hayakawa et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib9)\)\. We report them at11,22, and44steps\. We also compare against*multi\-step models*that generate through multiple \(e\.g\., 1024\) denoising steps, namely FLM\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\), Duo\([Sahoo et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib4)\), MDLM\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12879#bib.bib5)\), CANDI\([Pynadath et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib6)\), RDLM\([Jo and Hwang, 2026](https://arxiv.org/html/2610.12879#bib.bib7)\), LangFlow\([Chen et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib24)\), and ELF\([Hu et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib21)\)\.

#### Data and protocol\.

We follow the evaluation protocol of[Lee et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib2)and report at two output lengths\. In the length\-128128setup the models are trained on One Billion Word \(LM1B\)\([Chelba et al\., 2013](https://arxiv.org/html/2610.12879#bib.bib3)\), and in the length\-10241024setup on OpenWebText\([Gokaslan et al\., 2019](https://arxiv.org/html/2610.12879#bib.bib10)\)using the corresponding benchmark preprocessing\. We draw10241024samples per configuration\. For each method we report the number of forward passes per sample \(Step\), the generative perplexity of the samples under GPT\-2 Large \(Gen\-PPL↓\\downarrow\), and their unigram entropy \(Entropy\)\.

#### Implementation details\.

We consider two implementations of on\-policy support\-set construction\. ForSeq\-DriftingS, we first train a GPT\-2 from*scratch*on the same corpus as the baselines for10510^\{5\}steps, then freeze it and use it as the support model\. This setting uses no additional text beyond the baselines\. ForSeq\-DriftingP, we directly use a*public*pretrained GPT\-2 checkpoint to construct the support sets in the same way\. The support set usesk=64k=64, the attraction temperature isτ=0\.3\\tau=0\.3, and the cross\-sample and intra\-sequence repulsions useλout=5\\lambda\_\{\\mathrm\{out\}\}=5andλin=100\\lambda\_\{\\mathrm\{in\}\}=100\. The geodesic step is capped atαmax=1\.5708\\alpha\_\{\\max\}=1\.5708radians\. We train with AdamW \(learning rate2×10−42\\times 10^\{\-4\}, weight decay00, cosine schedule\) on44H200 GPUs\. Unconditional models are trained for2×1052\\times 10^\{5\}steps\. Conditional models use a2×1052\\times 10^\{5\}\-step unconditional warm\-up, followed by1×1051\\times 10^\{5\}conditional steps, keeping an exponential moving average of the weights with decay0\.9990\.999\. Therefore, the total training step of Seq\-Drifting is shorter than the 1\.0–1\.5M\-step recipes usually used by the baselines\. The effective batch is20482048at length128128and128128at length10241024, the longer sequence being reduced to fit memory\.

![Refer to caption](https://arxiv.org/html/2610.12879v1/time_ppl_readable_v2.png)

Figure 3:Unconditional generation\.*Left:*Gen\-PPL \(under GPT\-2 Large\) and unigram entropy for the few\-step distilled models at11/22/44steps and Seq\-Drifting at a single pass, on LM1B \(T=128T\{=\}128\) and OpenWebText \(T=1024T\{=\}1024\); multi\-step baselines are in Table[6](https://arxiv.org/html/2610.12879#A2.T6)\.*Right:*Gen\-PPL vs\. sampling steps and wall\-clock running time per sample, showing Seq\-Drifting achieves lower Gen\-PPL in a single forward pass at a fraction of the cost\.

### 5\.2Unconditional generation

Figure[3](https://arxiv.org/html/2610.12879#S5.F3)compares the few\-step distilled models with Seq\-Drifting at a single pass\. The full multi\-step results are reported in Table[6](https://arxiv.org/html/2610.12879#A2.T6)in Appendix[B\.3](https://arxiv.org/html/2610.12879#A2.SS3)\. Seq\-DriftingSproduces competitive unconditional samples\. On LM1B, it outperforms the strongest four\-step baseline and even beat several10241024\-step baselines, which directly reflect the effectiveness of our drifting framework\. Seq\-DriftingPfurther reduces Gen\-PPL relative to FMLM by69\.8%69\.8\\%on LM1B and24\.2%24\.2\\%on OpenWebText\. The entropy and Self\-BLEU results in Appendix[B\.1](https://arxiv.org/html/2610.12879#A2.SS1)show that they preserve generation diversity\. To verify that the comparison is not tied to the GPT\-2 Large evaluator, Appendix[B\.4](https://arxiv.org/html/2610.12879#A2.SS4)recomputes Gen\-PPL with Pythia\-1B\([Biderman et al\., 2023](https://arxiv.org/html/2610.12879#bib.bib33)\)and OPT\-1\.3B\([Zhang et al\., 2022](https://arxiv.org/html/2610.12879#bib.bib34)\)\. The resulting comparisons remain consistent with the main table, showing that the observed gains are not specific to the evaluator\.

Table 1:Conditional generation\.A6464\-token prefix followed by a continuation\.The running\-time in Figure[3](https://arxiv.org/html/2610.12879#S5.F3)\(right\) shows that Seq\-Drifting has the lowest single\-sample latency compared with the baselines\. Relative to four\-step FMLM, which reaches a comparable quality level, Seq\-Drifting is11\.2×11\.2\\timesfaster on LM1B and10\.1×10\.1\\timesfaster on OpenWebText\. A broader comparison of sampling time across multi\-step baselines is provided in Appendix[B\.2](https://arxiv.org/html/2610.12879#A2.SS2)\. Qualitative samples are shown in Appendix[B\.9](https://arxiv.org/html/2610.12879#A2.SS9)\.

### 5\.3Conditional generation

Table[1](https://arxiv.org/html/2610.12879#S5.T1)summarizes two conditional\-generation settings, each with a 64\-token context\. On LM1B the model generates 64 output tokens\. On OpenWebText it generates 512 output tokens so that the context and output remain within GPT\-2’s 1024\-token context window\. Both Seq\-DriftingSand Seq\-DriftingPoutperform all few\-step distilled baselines in both settings\. Compared with unconditional generation, the gap between Seq\-DriftingSand Seq\-DriftingPnarrows, and Seq\-DriftingSis even stronger on the512512\-token output setting\. One possible explanation is that the observed prefix narrows the plausible continuations, reducing ambiguity in the support sets and making the conditional map easier to learn\. Overall, the results suggest that Seq\-Drifting can preserve both fluency and conditional relevance in conditional generation\.

### 5\.4Ablations

We report ablations in both the unconditional \(128128\) and conditional \(64→6464\{\\to\}64\) settings in Table[2](https://arxiv.org/html/2610.12879#S5.T2)\. The geometry ablation shows that the spherical update is essential\. Replacing it with a Euclidean update allows repulsion to increase the embedding norm and causes training to diverge\. The positive\-selection ablation highlights that support construction should match the task\. For open\-ended generation, fixed corpus\-based targets do not follow the generator’s evolving contexts\. When the attraction becomes sharp, this mismatch can produce oscillatory training and rising generative perplexity and entropy \(Appendix[B\.8](https://arxiv.org/html/2610.12879#A2.SS8)\)\. For tasks with explicit target answers, dataset\-derived supervision can remain appropriate, as in our reasoning experiments in §[5\.6](https://arxiv.org/html/2610.12879#S5.SS6)\. For repulsion, the cross\-sample term is not enough on its own\. An intra\-sequence term that pushes each position off the other positions of the same generation is also necessary, and dropping it \(*w/o intra*\) collapses the output into within\-sentence token repetition\. In the conditional setting, whether the cross\-sample repulsion is restricted to samples that share a prefix makes little difference\. Finally, unconditional warm\-up stabilizes from\-scratch conditional training by allowing the generator to learn fluency before it must also model prefix dependence\.

Table 2:Ablation studyon unconditional \(128128\) and conditional \(64→6464\{\\to\}64\) generation\.Figure 4:Hyperparameter sweepsof the repulsion strengthλout\\lambda\_\{\\mathrm\{out\}\}, the anti\-repetition weightλin\\lambda\_\{\\mathrm\{in\}\}, the attraction temperatureτ\\tau, and the support sizekk\. Setting: unconditional128128\-token generation \(solid\), conditional64→6464\{\\to\}64generation \(dashed\)\.
### 5\.5Hyperparameters

Figure[4](https://arxiv.org/html/2610.12879#S5.F4)further sweeps parametersλout,λin,τ\\lambda\_\{\\mathrm\{out\}\},\\lambda\_\{\\mathrm\{in\}\},\\tau, andkk\. We observe that the first three parameters show the same broad pattern\. Gen\-PPL first improves and then worsens as the parameter moves away from a favorable middle range, revealing a balance between a target that is too weak and one that is too sharp or too disruptive\.

λout\\lambda\_\{\\mathrm\{out\}\}has a larger effect thanλin\\lambda\_\{\\mathrm\{in\}\}\. When repulsion between different noise samples is too weak, the generator is encouraged to learn an average token direction rather than preserve distinct modes\.λin\\lambda\_\{\\mathrm\{in\}\}mainly controls repetition within a sequence and therefore provides a narrower correction\. Forτ\\tau, making the temperature too small makes the target at each position nearly unique, discarding the useful ambiguity in the support tokens\. For the support size,k=4k=4collapses the output \(entropy below3\.53\.5\)\. Increasing the support size generally restores diversity and improves Gen\-PPL\.

### 5\.6Downstream task evaluation

#### Translation and summarization\.

Following[Hu et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib21), we evaluate Seq\-Drifting on WMT14 \(German–English translation\)\([Bojar et al\., 2014](https://arxiv.org/html/2610.12879#bib.bib25)\)and XSum \(abstractive summarization\)\([Narayan et al\., 2018](https://arxiv.org/html/2610.12879#bib.bib26)\)\. Experimental details are provided in Appendix[B\.5](https://arxiv.org/html/2610.12879#A2.SS5)\. Table[3](https://arxiv.org/html/2610.12879#S5.T3)shows that Seq\-Drifting outperforms the few\-step baselines on translation and on most summarization metrics, although a gap remains relative to the multi\-step results\. We hypothesize that this gap partly reflects a mismatch between the support\-set candidates used during training and the single reference used for evaluation\. When evaluated against results sampled from the support model, Seq\-Drifting obtains substantially higher scores\. Scaling the support model further improves one\-step generation quality \(Figure[5](https://arxiv.org/html/2610.12879#S5.F5)\)\. This observation motivates future work on using increasingly capable language models to construct support sets, with the broader goal of turning existing LLMs into efficient one\-step generators\.

ModelStepWMT14XSumBLEU↑\\uparrowROUGE\-1↑\\uparrowROUGE\-2↑\\uparrowROUGE\-L↑\\uparrowAR–25\.2†30\.5±0\.13\\pm 0\.13†10\.2±0\.11\\pm 0\.11†24\.4±0\.12\\pm 0\.12†MDLM4<<1019\.0±0\.06\\pm 0\.062\.4±0\.01\\pm 0\.0113\.2±0\.04\\pm 0\.046418\.4†33\.4±0\.11\\pm 0\.11†11\.6±0\.10\\pm 0\.10†25\.8±0\.10\\pm 0\.10†Duo4<<1024\.4±0\.06\\pm 0\.065\.6±0\.01\\pm 0\.0118\.1±0\.04\\pm 0\.04100021\.3†31\.4±0\.12\\pm 0\.12†10\.1±0\.10\\pm 0\.10†25\.0±0\.12\\pm 0\.12†ELF111\.523\.7±0\.11\\pm 0\.115\.7±0\.08\\pm 0\.0819\.4±0\.10\\pm 0\.106426\.4†36\.0±0\.13\\pm 0\.13†12\.2±0\.11\\pm 0\.11†27\.8±0\.12\\pm 0\.12†Seq\-Drifting \(vs\. dataset\)118\.624\.7±0\.36\\pm 0\.365\.3±0\.24\\pm 0\.2420\.1±0\.32\\pm 0\.32Seq\-Drifting \(vs\. AR\)130\.833\.2±0\.44\\pm 0\.4411\.6±0\.41\\pm 0\.4128\.7±0\.44\\pm 0\.44

Table 3:Performance on WMT14 De–En and XSum\. The two Seq\-Drifting rows evaluate results on the dataset ground truth and samples from our support model, respectively\.†\\daggerdenotes results taken from[Hu et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib21)\.Figure 5:Support model scaling\. S, M, and L denote small, medium, and large GPT\-2\.

#### Reasoning\.

We further evaluate Seq\-Drifting on math reasoning \(SVAMP\([Patel et al\., 2021](https://arxiv.org/html/2610.12879#bib.bib28)\)and ASDiv\([Miao et al\., 2020](https://arxiv.org/html/2610.12879#bib.bib35)\)\) and logical reasoning \(ProofWriter\([Tafjord et al\., 2021](https://arxiv.org/html/2610.12879#bib.bib27)\)\)\. We scale Seq\-Drifting and all the baselines to0\.50\.5B\. The implementation and training details are provided in Appendix[B\.6](https://arxiv.org/html/2610.12879#A2.SS6)\. Despite using a single forward pass, Seq\-Drifting approaches the accuracy of existing latent\-reasoning method Coconut\([Hao et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib29)\)on the mathematical benchmarks and remains competitive with AR on ProofWriter, while achieving substantially lower per\-sample latency\. These results show that a drifting field can also be learned from supervision with explicit target answers, extending the one\-step framework beyond open\-ended generation\.

Table 4:Performance on SVAMP, ASDiv, and ProofWriter\.

## 6Conclusion

In this paper, we presented Seq\-Drifting, a framework for directly learning one\-step text generation by drifting sequences of continuous token representations\. Context\-dependent support sets provide multiple plausible targets at each position, while mode\-seeking attraction, repulsion, and spherical geometry help preserve coherent and diverse outputs\. Across unconditional generation, conditional continuation, translation, summarization, and reasoning, Seq\-Drifting provides a competitive one\-step operating point and outperforms the evaluated few\-step baselines at a fraction of their inference cost\. The remaining gap to the strongest multi\-step systems motivates future work on stronger support models, richer semantic supervision, and improved token recovery\.

### AI Use Statement

In this work, we used generative AI tools to edit the manuscript for readability and clarity\. All AI\-assisted text was reviewed and revised by the authors\. The authors take full responsibility for the final content of this work, including all claims, analyses, and conclusions\.

## References

- Arriolaet al\.\(2025\)M\. Arriola, A\. Gokaslan, J\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. Sahoo, and V\. KuleshovBlock diffusion: interpolating between autoregressive and diffusion language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 50726–50753\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p2.1)\.
- Arriolaet al\.\(2026\)M\. Arriola, Y\. Schiff, H\. Phung, A\. Gokaslan, and V\. KuleshovEncoder\-decoder diffusion language models for efficient training and inference\.Advances in Neural Information Processing Systems38,pp\. 15612–15644\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p2.1)\.
- Austinet al\.\(2021\)J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den BergStructured denoising diffusion models in discrete state\-spaces\.Advances in neural information processing systems34,pp\. 17981–17993\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p2.1)\.
- Bidermanet al\.\(2023\)S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.Pythia: a suite for analyzing large language models across training and scaling\.InInternational conference on machine learning,pp\. 2397–2430\.Cited by:[§5\.2](https://arxiv.org/html/2610.12879#S5.SS2.p1.1)\.
- Bojaret al\.\(2014\)O\. Bojar, C\. Buck, C\. Federmann, B\. Haddow, P\. Koehn, J\. Leveling, C\. Monz, P\. Pecina, M\. Post, H\. Saint\-Amand,et al\.Findings of the 2014 workshop on statistical machine translation\.InProceedings of the ninth workshop on statistical machine translation,pp\. 12–58\.Cited by:[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p1.1)\.
- Chelbaet al\.\(2013\)C\. Chelba, T\. Mikolov, M\. Schuster, Q\. Ge, T\. Brants, P\. Koehn, and T\. RobinsonOne billion word benchmark for measuring progress in statistical language modeling\.arXiv preprint arXiv:1312\.3005\.Cited by:[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2026\)Y\. Chen, C\. Liang, H\. Sui, R\. Guo, C\. Cheng, J\. You, and G\. LiuLangflow: continuous diffusion rivals discrete in language modeling\.arXiv preprint arXiv:2604\.11748\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p3.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Denget al\.\(2026\)M\. Deng, H\. Li, T\. Li, Y\. Du, and K\. HeGenerative modeling via drifting\.External Links:2602\.04770,[Link](https://arxiv.org/abs/2602.04770)Cited by:[§A\.2](https://arxiv.org/html/2610.12879#A1.SS2.p1.1),[§1](https://arxiv.org/html/2610.12879#S1.p2.1),[§2](https://arxiv.org/html/2610.12879#S2.p4.1),[§3](https://arxiv.org/html/2610.12879#S3.p1.2)\.
- Denget al\.\(2023\)Y\. Deng, K\. Prasad, R\. Fernandez, P\. Smolensky, V\. Chaudhary, and S\. ShieberImplicit chain of thought reasoning via knowledge distillation\.arXiv preprint arXiv:2311\.01460\.Cited by:[§B\.6](https://arxiv.org/html/2610.12879#A2.SS6.p1.1)\.
- Deschenaux and Gulcehre \(2025\)J\. Deschenaux and C\. GulcehreBeyond autoregression: fast llms via self\-distillation through time\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 5007–5045\.Cited by:[§1](https://arxiv.org/html/2610.12879#S1.p1.1.1),[§2](https://arxiv.org/html/2610.12879#S2.p4.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Gokaslanet al\.\(2019\)A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. TellexOpenWebText corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px2.p1.1)\.
- Gonget al\.\(2023\)S\. Gong, M\. Li, J\. Feng, Z\. Wu, and L\. KongDiffuseq: sequence to sequence text generation with diffusion models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p3.1)\.
- Hanet al\.\(2023\)X\. Han, S\. Kumar, and Y\. TsvetkovSsd\-lm: semi\-autoregressive simplex\-based diffusion language model for text generation and modular control\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11575–11596\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p3.1)\.
- Haoet al\.\(2025\)S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. E\. Weston, and Y\. TianTraining large language models to reason in a continuous latent space\.InSecond Conference on Language Modeling,Cited by:[§B\.6](https://arxiv.org/html/2610.12879#A2.SS6.p1.1),[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px2.p1.1)\.
- Hayakawaet al\.\(2025\)S\. Hayakawa, Y\. Takida, M\. Imaizumi, H\. Wakaki, and Y\. MitsufujiDistillation of discrete diffusion through dimensional correlations\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p4.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)K\. Hu, L\. Qiu, Y\. Lu, H\. Zhao, T\. Li, Y\. Kim, J\. Andreas, and K\. HeElf: embedded language flows\.arXiv preprint arXiv:2605\.10938\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p3.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1),[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.12879#S5.T3.fig1)\.
- Jo and Hwang \(2026\)J\. Jo and S\. J\. HwangContinuous diffusion model for language modeling\.Advances in Neural Information Processing Systems38,pp\. 97394–97430\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Leeet al\.\(2026\)C\. Lee, J\. Yoo, M\. Agarwal, S\. Shah, J\. Huang, A\. Raghunathan, S\. Hong, N\. M\. Boffi, and J\. KimOne\-step language modeling via continuous denoising\.arXiv e\-prints,pp\. arXiv–2602\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p3.1),[§2](https://arxiv.org/html/2610.12879#S2.p4.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)T\. Li, M\. Chen, B\. Guo, and Z\. ShenA survey on diffusion language models\.arXiv preprint arXiv:2508\.10875\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p1.1)\.
- Liet al\.\(2022\)X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. HashimotoDiffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p3.1)\.
- Lin \(2004\)C\. LinRouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§B\.5](https://arxiv.org/html/2610.12879#A2.SS5.p1.1)\.
- Mahabadiet al\.\(2024\)R\. K\. Mahabadi, H\. Ivison, J\. Tae, J\. Henderson, I\. Beltagy, M\. E\. Peters, and A\. CohanTess: text\-to\-text self\-conditioned simplex diffusion\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2347–2361\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p3.1)\.
- Miaoet al\.\(2020\)S\. Miao, C\. Liang, and K\. SuA diverse corpus for evaluating and developing english math word problem solvers\.InProceedings of the 58th annual meeting of the Association for Computational Linguistics,pp\. 975–984\.Cited by:[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px2.p1.1)\.
- Narayanet al\.\(2018\)S\. Narayan, S\. B\. Cohen, and M\. LapataDon’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 1797–1807\.Cited by:[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px1.p1.1)\.
- Nieet al\.\(2025\)S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. LiLarge language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[§2](https://arxiv.org/html/2610.12879#S2.p2.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§B\.5](https://arxiv.org/html/2610.12879#A2.SS5.p1.1)\.
- Patelet al\.\(2021\)A\. Patel, S\. Bhattamishra, and N\. GoyalAre nlp models really able to solve simple math word problems?\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 2080–2094\.Cited by:[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px2.p1.1)\.
- Pynadathet al\.\(2026\)P\. Pynadath, J\. Shi, and R\. ZhangCANDI: hybrid discrete\-continuous diffusion models\.InForty\-third International Conference on Machine Learning,Cited by:[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p1.1)\.
- Sahooet al\.\(2024\)S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. KuleshovSimple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.12879#S1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Sahooet al\.\(2025\)S\. S\. Sahoo, J\. Deschenaux, A\. Gokaslan, G\. Wang, J\. Chiu, and V\. KuleshovThe diffusion duality\.Proceedings of machine learning research267,pp\. 52584\.Cited by:[§B\.3](https://arxiv.org/html/2610.12879#A2.SS3.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.12879#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.12879#S5.SS1.SSS0.Px1.p1.1)\.
- Tafjordet al\.\(2021\)O\. Tafjord, B\. Dalvi, and P\. ClarkProofWriter: generating implications, proofs, and abductive statements over natural language\.InFindings of the association for computational linguistics: ACL\-IJCNLP 2021,pp\. 3621–3634\.Cited by:[§5\.6](https://arxiv.org/html/2610.12879#S5.SS6.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2022\)S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§5\.2](https://arxiv.org/html/2610.12879#S5.SS2.p1.1)\.
- Zhuet al\.\(2018\)Y\. Zhu, S\. Lu, L\. Zheng, J\. Guo, W\. Zhang, J\. Wang, and Y\. YuTexygen: a benchmarking platform for text generation models\.InThe 41st international ACM SIGIR conference on research & development in information retrieval,pp\. 1097–1100\.Cited by:[§B\.1](https://arxiv.org/html/2610.12879#A2.SS1.p1.1)\.

## Appendix AMethod details and analysis

### A\.1Generator architecture

A shared noise vectorϵ∈ℝC\\epsilon\\in\\mathbb\{R\}^\{C\}is projected toT⋅DT\\\!\\cdot\\\!Ddimensions, reshaped to\(T,D\)\(T,D\), and combined with learned positional embeddings\. Our generatorGθG\_\{\\theta\}is a1212\-layer pre\-LN Transformer with width768768,1212attention heads, and an FFN width of30723072\. The noise dimension isC=128C=128\. We use QK\-norm on the per\-head queries and keys and a LayerScale gate on each residual branch as stabilization mechanisms\. For conditional generation, conditioning token embeddings are supplied as a prefix and the output positions access them through ungated cross\-attention\.

### A\.2A conditional fixed\-point analysis

We analyze conditions under which matching the generated and data distributions makes the idealized drift field zero\. Forqθ=pdataq\_\{\\theta\}=p\_\{\\mathrm\{data\}\}to be a fixed point of training, the driftVV, and hence the loss of Eq\. \([3](https://arxiv.org/html/2610.12879#S3.E3)\), must be zero there\. Anti\-symmetryVp,q=−Vq,pV\_\{p,q\}=\-V\_\{q,p\}is one way to guarantee this and is how the plain mean\-shift field of[Deng et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib1)does it, but it is a*sufficient*condition, not a necessary one\. Seq\-Drifting’s per\-position field is not anti\-symmetric, so instead of appealing to that argument we checkV=𝟎V=\\boldsymbol\{0\}atqθ=pdataq\_\{\\theta\}=p\_\{\\mathrm\{data\}\}directly\.

#### The field\.

At positiontt, with unit embedding𝒆t\\boldsymbol\{e\}\_\{t\}, the drift of §[4](https://arxiv.org/html/2610.12879#S4)is

𝑭t=∑𝒘∈𝒮tφ⁡\(𝒘,𝒆t\)​\(𝒘−𝒆t\)⏟attraction​𝒖t−λout​∑𝒏∈Ntoutφ⁡\(𝒏,𝒆t\)​\(𝒏−𝒆t\)−λin​∑𝒏∈Ntinφ⁡\(𝒏,𝒆t\)​\(𝒏−𝒆t\),\\boldsymbol\{F\}\_\{t\}=\\underbrace\{\\sum\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\}\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{w\}\-\\boldsymbol\{e\}\_\{t\}\)\}\_\{\\text\{attraction \}\\boldsymbol\{u\}\_\{t\}\}\-\\lambda\_\{\\mathrm\{out\}\}\\\!\\\!\\sum\_\{\\boldsymbol\{n\}\\in N\_\{t\}^\{\\mathrm\{out\}\}\}\\\!\\\!\\varphi\(\\boldsymbol\{n\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{n\}\-\\boldsymbol\{e\}\_\{t\}\)\-\\lambda\_\{\\mathrm\{in\}\}\\\!\\\!\\sum\_\{\\boldsymbol\{n\}\\in N\_\{t\}^\{\\mathrm\{in\}\}\}\\\!\\\!\\varphi\(\\boldsymbol\{n\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{n\}\-\\boldsymbol\{e\}\_\{t\}\),\(8\)withφ\(𝒂,𝒃\)∝exp\(−d\(𝒂,𝒃\)/\(sτ\)\)\\varphi\(\\boldsymbol\{a\},\\boldsymbol\{b\}\)\\propto\\exp\\\!\\big\(\\\!\-d\(\\boldsymbol\{a\},\\boldsymbol\{b\}\)/\(s\\,\\tau\)\\big\)andd⁡\(𝒂,𝒃\)=1−𝒂⊤​𝒃d\(\\boldsymbol\{a\},\\boldsymbol\{b\}\)=1\-\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{b\}the angular kernel, normalized over each set\. The attraction pulls𝒆t\\boldsymbol\{e\}\_\{t\}toward the support set𝒮t\\mathcal\{S\}\_\{t\}of plausible tokens, and the two repulsions push it off the other generationsNtoutN\_\{t\}^\{\\mathrm\{out\}\}and the other positions of its own sequenceNtinN\_\{t\}^\{\\mathrm\{in\}\}\.

#### Attraction and rescaled repulsion at matching distributions\.

Atqθ=pdataq\_\{\\theta\}=p\_\{\\mathrm\{data\}\}the generator reproduces the data\. Suppose each output embedding coincides with a real token embedding,𝒆t=𝒘⋆\\boldsymbol\{e\}\_\{t\}=\\boldsymbol\{w\}^\{\\star\}with𝒘⋆∈𝒮t\\boldsymbol\{w\}^\{\\star\}\\in\\mathcal\{S\}\_\{t\}\. Importantly, this does not depend on any particular support model\. In the mode\-seeking limitτ→0\\tau\\\!\\to\\\!0, a generation already sitting on a supported token has nothing to be pulled toward, so the attraction is zero,

𝒖t=∑𝒘∈𝒮tφ⁡\(𝒘,𝒆t\)​\(𝒘−𝒆t\)⟶φ⁡\(𝒘⋆,𝒆t\)​\(𝒘⋆−𝒆t\)=𝟎,\\boldsymbol\{u\}\_\{t\}=\\sum\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\}\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{w\}\-\\boldsymbol\{e\}\_\{t\}\)\\;\\longrightarrow\\;\\varphi\(\\boldsymbol\{w\}^\{\\star\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{w\}^\{\\star\}\-\\boldsymbol\{e\}\_\{t\}\)=\\boldsymbol\{0\},\(9\)The repulsions then vanish with it, because each is rescaled to the attraction magnitude before entering𝑭t\\boldsymbol\{F\}\_\{t\}, with coefficientarms/rrmsa\_\{\\mathrm\{rms\}\}/r\_\{\\mathrm\{rms\}\}andarms=RMS⁡\(𝒖t\)a\_\{\\mathrm\{rms\}\}=\\mathrm\{RMS\}\(\\boldsymbol\{u\}\_\{t\}\), so the repulsion magnitude is∝arms→0\\propto a\_\{\\mathrm\{rms\}\}\\to 0\. Under these conditions,𝑭t\\boldsymbol\{F\}\_\{t\}is zero at every position andqθ=pdataq\_\{\\theta\}=p\_\{\\mathrm\{data\}\}is a fixed point of the idealized field\. The attraction\-relative rescaling is what lets a strong anti\-collapse repulsion coexist with this fixed point, because the repulsion is rescaled by the attraction magnitude and therefore switches off with it\.

In practice, the temperature is positive and the decoded token distribution can match the data distribution even when generated representations slightly differ from their corresponding vocabulary embeddings\. We now quantify how such discrepancies affect the drift\. Let𝒘t⋆=𝒘tokt\\boldsymbol\{w\}\_\{t\}^\{\\star\}=\\boldsymbol\{w\}\_\{\\mathrm\{tok\}\_\{t\}\}and consider generated sequences for which𝒘t⋆∈𝒮t\\boldsymbol\{w\}\_\{t\}^\{\\star\}\\in\\mathcal\{S\}\_\{t\}is the unique nearest candidate\. Since the kernel weights sum to one, the attraction decomposes exactly as

𝒖t=𝒘t⋆−𝒆t\+∑𝒘∈𝒮t∖\{𝒘t⋆\}φ⁡\(𝒘,𝒆t\)​\(𝒘−𝒘t⋆\)\.\\boldsymbol\{u\}\_\{t\}=\\boldsymbol\{w\}\_\{t\}^\{\\star\}\-\\boldsymbol\{e\}\_\{t\}\+\\sum\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\\setminus\\\{\\boldsymbol\{w\}\_\{t\}^\{\\star\}\\\}\}\\varphi\(\\boldsymbol\{w\},\\boldsymbol\{e\}\_\{t\}\)\(\\boldsymbol\{w\}\-\\boldsymbol\{w\}\_\{t\}^\{\\star\}\)\.\(10\)The first term is the representation discrepancy, and the second is the finite\-temperature correction\. For a support set with at least two candidates, define

Δt\\displaystyle\\Delta\_\{t\}=min𝒘∈𝒮t∖\{𝒘t⋆\}⁡\[d⁡\(𝒆t,𝒘\)−d⁡\(𝒆t,𝒘t⋆\)\]\>0,\\displaystyle=\\min\_\{\\boldsymbol\{w\}\\in\\mathcal\{S\}\_\{t\}\\setminus\\\{\\boldsymbol\{w\}\_\{t\}^\{\\star\}\\\}\}\\big\[d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\)\-d\(\\boldsymbol\{e\}\_\{t\},\\boldsymbol\{w\}\_\{t\}^\{\\star\}\)\\big\]\>0,\(11\)bt\\displaystyle b\_\{t\}=\(\|𝒮t\|−1\)​exp⁡\(−Δtst​τ\)\.\\displaystyle=\(\|\\mathcal\{S\}\_\{t\}\|\-1\)\\exp\\\!\\left\(\-\\frac\{\\Delta\_\{t\}\}\{s\_\{t\}\\tau\}\\right\)\.Each other candidate has at mostexp\(−Δt/\(stτ\)\)\\exp\(\-\\Delta\_\{t\}/\(s\_\{t\}\\tau\)\)times the nearest candidate’s kernel weight, giving1−φ⁡\(𝒘t⋆,𝒆t\)≤bt/\(1\+bt\)1\-\\varphi\(\\boldsymbol\{w\}\_\{t\}^\{\\star\},\\boldsymbol\{e\}\_\{t\}\)\\leq b\_\{t\}/\(1\+b\_\{t\}\)\. All vocabulary embeddings have unit norm, so their pairwise distances are at most two\. It follows that

‖𝒖t−\(𝒘t⋆−𝒆t\)‖2≤2​\[1−φ⁡\(𝒘t⋆,𝒆t\)\]≤2​bt1\+bt\.\\big\\\|\\boldsymbol\{u\}\_\{t\}\-\(\\boldsymbol\{w\}\_\{t\}^\{\\star\}\-\\boldsymbol\{e\}\_\{t\}\)\\big\\\|\_\{2\}\\leq 2\\big\[1\-\\varphi\(\\boldsymbol\{w\}\_\{t\}^\{\\star\},\\boldsymbol\{e\}\_\{t\}\)\\big\]\\leq\\frac\{2b\_\{t\}\}\{1\+b\_\{t\}\}\.\(12\)For singleton supports, the sole candidate has weight one and we setbt=0b\_\{t\}=0\. Define‖𝒂‖RMS=\(𝔼​‖𝒂t‖22\)1/2\\\|\\boldsymbol\{a\}\\\|\_\{\\mathrm\{RMS\}\}=\(\\mathbb\{E\}\\\|\\boldsymbol\{a\}\_\{t\}\\\|\_\{2\}^\{2\}\)^\{1/2\}, averaging over generated samples and positions, and write

δ=\(𝔼​‖𝒆t−𝒘t⋆‖22\)1/2,ρτ=\[𝔼​\(2​bt1\+bt\)2\]1/2\.\\delta=\\big\(\\mathbb\{E\}\\\|\\boldsymbol\{e\}\_\{t\}\-\\boldsymbol\{w\}\_\{t\}^\{\\star\}\\\|\_\{2\}^\{2\}\\big\)^\{1/2\},\\qquad\\rho\_\{\\tau\}=\\left\[\\mathbb\{E\}\\left\(\\frac\{2b\_\{t\}\}\{1\+b\_\{t\}\}\\right\)^\{2\}\\right\]^\{1/2\}\.\(13\)TheL2L^\{2\}triangle inequality gives‖𝒖‖RMS≤δ\+ρτ\\\|\\boldsymbol\{u\}\\\|\_\{\\mathrm\{RMS\}\}\\leq\\delta\+\\rho\_\{\\tau\}\. Under the attraction\-relative rescaling above, each rescaled repulsion has RMS no greater than that of the attraction, using the same normalization groups and leaving zero repulsion at zero\. Tangent projection cannot increase the force norm, and the geodesic displacementVt=g​o​a​lt−𝒆tV\_\{t\}=goal\_\{t\}\-\\boldsymbol\{e\}\_\{t\}satisfies‖Vt‖2=2​\|sin⁡\(αt/2\)\|≤αt≤‖𝑭t′‖2\\\|V\_\{t\}\\\|\_\{2\}=2\|\\sin\(\\alpha\_\{t\}/2\)\|\\leq\\alpha\_\{t\}\\leq\\\|\\boldsymbol\{F\}^\{\\prime\}\_\{t\}\\\|\_\{2\}, whereαt\\alpha\_\{t\}is the step angle at positionttand zero tangent force gives zero displacement\. Applying the triangle inequality to the combined force therefore yields

‖V‖RMS≤‖𝑭‖RMS≤\(1\+λout\+λin\)​\(δ\+ρτ\)\.\\\|V\\\|\_\{\\mathrm\{RMS\}\}\\leq\\\|\\boldsymbol\{F\}\\\|\_\{\\mathrm\{RMS\}\}\\leq\(1\+\\lambda\_\{\\mathrm\{out\}\}\+\\lambda\_\{\\mathrm\{in\}\}\)\(\\delta\+\\rho\_\{\\tau\}\)\.\(14\)The loss averaged over samples and positions is bounded by the square of this expression\. For a fixed distribution of representations satisfying the support and uniqueness conditions above,ρτ→0\\rho\_\{\\tau\}\\to 0asτ→0\\tau\\to 0by dominated convergence, sincebt→0b\_\{t\}\\to 0and2​bt/\(1\+bt\)≤22b\_\{t\}/\(1\+b\_\{t\}\)\\leq 2\. At positive temperature, the bound remains small when the representation discrepancy and the temperature correction are small enough for the chosen repulsion weights\. Thus, when continuous representations do not coincide exactly, matching the decoded distribution can ensure that the drift is bounded\.

### A\.3Seq\-Drifting algorithm

The pseudocode in Figure[6](https://arxiv.org/html/2610.12879#A1.F6)summarizes Seq\-Drifting training with on\-policy support sets constructed by a frozen support model\. Each iteration generates token representations, builds context\-dependent support sets, and combines attraction and repulsion to form supervision targets on the sphere\. The generator is then updated by regressing toward these fixed targets, with optional prefixes enabling conditional generation\.

Algorithm 1: Seq\-Drifting training\.

Input:generatorGθG\_\{\\theta\}, frozen support modelpψp\_\{\\psi\}, normalized token table\{𝒘v\}v∈𝒱\\\{\\boldsymbol\{w\}\_\{v\}\\\}\_\{v\\in\\mathcal\{V\}\}, batch sizeBB, support sizekk,τ,λout,λin,αmax\\tau,\\lambda\_\{\\mathrm\{out\}\},\\lambda\_\{\\mathrm\{in\}\},\\alpha\_\{\\max\}\.

repeat Sample noiseϵ1,…,ϵB\\epsilon\_\{1\},\\ldots,\\epsilon\_\{B\}and optional prefixes𝒑1,…,𝒑B\\boldsymbol\{p\}\_\{1\},\\ldots,\\boldsymbol\{p\}\_\{B\}\.

Compute one\-step outputs𝒉b,1:T=Gθ\(ϵb,𝒑b\)\\boldsymbol\{h\}\_\{b,1:T\}=G\_\{\\theta\}\(\\epsilon\_\{b\},\\boldsymbol\{p\}\_\{b\}\)and normalize𝒆b,t=𝒉b,t/‖𝒉b,t‖\\boldsymbol\{e\}\_\{b,t\}=\\boldsymbol\{h\}\_\{b,t\}/\\\|\\boldsymbol\{h\}\_\{b,t\}\\\|\.

Decode provisional tokensx^b,t=arg⁡maxv​𝒆b,t⊤​𝒘v\\hat\{x\}\_\{b,t\}=\\arg\\max\_\{v\}\\boldsymbol\{e\}\_\{b,t\}^\{\\top\}\\boldsymbol\{w\}\_\{v\}\.

Run one causal forward pass ofpψp\_\{\\psi\}on each generated context and form𝒮b,t\\mathcal\{S\}\_\{b,t\}from the top\-kkplausible next tokens\.

foreach samplebband positionttdo Compute angular attraction𝒖b,t\\boldsymbol\{u\}\_\{b,t\}from𝒮b,t\\mathcal\{S\}\_\{b,t\}\.

Compute cross\-sample and intra\-sequence repulsions𝒓b,tout\\boldsymbol\{r\}^\{\\mathrm\{out\}\}\_\{b,t\}and𝒓b,tin\\boldsymbol\{r\}^\{\\mathrm\{in\}\}\_\{b,t\}\.

Rescale each repulsion to the attraction magnitude:𝒓~b,tj=\(arms/rrmsj\)​𝒓b,tj\\widetilde\{\\boldsymbol\{r\}\}^\{j\}\_\{b,t\}=\(a\_\{\\mathrm\{rms\}\}/r\_\{\\mathrm\{rms\}\}^\{j\}\)\\boldsymbol\{r\}^\{j\}\_\{b,t\}forj∈\{out,in\}j\\in\\\{\\mathrm\{out\},\\mathrm\{in\}\\\}, where the RMS values are computed over the batch and positions; set𝒓~j=𝟎\\widetilde\{\\boldsymbol\{r\}\}^\{j\}=\\boldsymbol\{0\}whenrrmsj=0r\_\{\\mathrm\{rms\}\}^\{j\}=0\.

Set𝑭b,t=𝒖b,t\+λout​𝒓~b,tout\+λin​𝒓~b,tin\\boldsymbol\{F\}\_\{b,t\}=\\boldsymbol\{u\}\_\{b,t\}\+\\lambda\_\{\\mathrm\{out\}\}\\widetilde\{\\boldsymbol\{r\}\}^\{\\mathrm\{out\}\}\_\{b,t\}\+\\lambda\_\{\\mathrm\{in\}\}\\widetilde\{\\boldsymbol\{r\}\}^\{\\mathrm\{in\}\}\_\{b,t\}\.

Project𝑭b,t\\boldsymbol\{F\}\_\{b,t\}to the tangent plane and map it tog​o​a​lb,tgoal\_\{b,t\}by Eq\. \([7](https://arxiv.org/html/2610.12879#S4.E7)\)\.

end for Updateθ\\thetawith∇θ1B​T​∑b,t‖𝒆b,t−sg⁡\(g​o​a​lb,t\)‖22\\nabla\_\{\\theta\}\\frac\{1\}\{BT\}\\sum\_\{b,t\}\\\|\\boldsymbol\{e\}\_\{b,t\}\-\\operatorname\{sg\}\(goal\_\{b,t\}\)\\\|\_\{2\}^\{2\}\.

untilconvergence

Figure 6:Seq\-Drifting training pseudocode\.

## Appendix BAdditional results

### B\.1Self\-BLEU

Table[5](https://arxiv.org/html/2610.12879#A2.T5)reports Self\-BLEU\([Zhu et al\., 2018](https://arxiv.org/html/2610.12879#bib.bib11)\)for the few\-step distilled baselines, at each of their step counts \(11,22,44\), and for Seq\-Drifting at a single step\. Self\-BLEU measures the n\-gram overlap among a model’s own generations\. A fully mode\-collapsed generator that emits the same sample every time attains the maximum of1\.01\.0\.

Table 5:Self\-BLEU across the step budget\.Per\-dataset Self\-BLEU for the few\-step distilled models and Seq\-Drifting; a fully mode\-collapsed generator scores1\.01\.0\(bottom row\)\.
### B\.2Sampling time across step budgets

Figure[7](https://arxiv.org/html/2610.12879#A2.F7)reports seconds per sample as a function of the sampling budget for the multi\-step baselines\. The horizontal dashed line marks the one\-step latency of Seq\-Drifting\. Sampling time grows with the number of denoising steps, while Seq\-Drifting remains constant at one step\.

Figure 7:Sampling time across step budgets\.Seconds per sample on LM1B \(T=128T=128\) and OpenWebText \(T=1024T=1024\)\. The x\-axis is logarithmic in the number of sampling steps and the y\-axis is logarithmic in seconds per sample\. The dashed line marks Seq\-Drifting\.
### B\.3Multi\-step baselines

Table[6](https://arxiv.org/html/2610.12879#A2.T6)gives the full multi\-step baselines \(FLM, Duo, MDLM, CANDI, RDLM, LangFlow, ELF\) at their11/88/10241024step budgets on both datasets, with Seq\-Drifting included for reference\. Table[7](https://arxiv.org/html/2610.12879#A2.T7)gives the multi\-step baselines on the*conditional*task at the same step budgets\.

#### Multi\-step baseline implementations\.

For FLM, we use the authors’ released checkpoints for LM1B and OpenWebText\([Lee et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib2)\)\. For Duo, MDLM, and CANDI on LM1B, we use the reproduction checkpoints distributed by[Lee et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib2)\. On OpenWebText, we use the released Duo and MDLM weights\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12879#bib.bib5);[Sahoo et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib4)\)\. For CANDI on OpenWebText, we report the published result from[Lee et al\. \(2026\)](https://arxiv.org/html/2610.12879#bib.bib2), while no public conditional OpenWebText checkpoint is available\. For RDLM, we use the released LM1B checkpoint and its accompanying SDE configuration\([Jo and Hwang, 2026](https://arxiv.org/html/2610.12879#bib.bib7)\)\. We do not report OpenWebText results because no corresponding public checkpoint is available\. For LangFlow, we use the authors’ released checkpoints\([Chen et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib24)\)\. For ELF, we use the released OpenWebText results and checkpoints\([Hu et al\., 2026](https://arxiv.org/html/2610.12879#bib.bib21)\); no corresponding LM1B checkpoint is available\.

Table 6:Unconditional generation: multi\-step modelsacross the step budget, with Seq\-Drifting at a single pass\. LM1B \(T=128T\{=\}128\) and OpenWebText \(T=1024T\{=\}1024\)\. A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting\.Table 7:Conditional generation: multi\-step modelsacross the step budget, with Seq\-Drifting at a single pass\. A6464\-token prefix, then a continuation \(64→6464\{\\to\}64on LM1B,64→51264\{\\to\}512on OWT\)\. A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting\.

### B\.4Alternative Gen\-PPL evaluators

The main comparison reports Gen\-PPL under GPT\-2 Large\. To test whether the results depend on that evaluator, we recompute unconditional Gen\-PPL with Pythia\-1B and OPT\-1\.3B in Table[8](https://arxiv.org/html/2610.12879#A2.T8)\. All other sampling settings are unchanged\.

Table 8:Unconditional Gen\-PPL under alternative evaluators\.Gen\-PPL computed with Pythia\-1B and OPT\-1\.3B for LM1B \(T=128T\{=\}128\) and OpenWebText \(T=1024T\{=\}1024\)\.
### B\.5Translation and summarization setup

For WMT14, the input contains a 64\-token source sequence and the model generates a 64\-token target sequence\. For XSum, the input contains up to 1024 source tokens and the model generates a 64\-token summary\. We report corpus BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2610.12879#bib.bib31)\)for WMT14 and ROUGE\-1, ROUGE\-2, and ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2610.12879#bib.bib32)\)for XSum\.

For Seq\-Drifting, we first train an autoregressive model from scratch on each task’s training split\. In the small support\-model setting, this model is a 12\-layer pre\-LN Transformer with hidden width768768, 12 attention heads, FFN width30723072, and the same tokenizer and context lengths used by the corresponding experiment\. It uses a causal attention mask and is trained with the standard next\-token cross\-entropy objective\. We then train Seq\-Drifting using the frozen autoregressive model’s token distributions as its on\-policy support signal, without initializing from a pretrained language model\. At evaluation time, we compare generated outputs against both the dataset ground\-truth references and samples generated by the autoregressive model, using the same task metrics in both cases\.

### B\.6Reasoning implementation details

For mathematical reasoning, we train on GSM8K\-Aug\([Deng et al\., 2023](https://arxiv.org/html/2610.12879#bib.bib30)\)and evaluate out\-of\-domain generalization on SVAMP and ASDiv\. Ground\-truth solution sequences define the support set for the attraction objective\. For logical reasoning, we use the binary \(true/false\) subset of ProofWriter, training on the combined D0–D5 training split and evaluating on the corresponding test split; unknown\-labeled examples are excluded\. Seq\-Drifting uses a0\.50\.5B\-parameter backbone with the Qwen2\.5 architecture\. We compare it with autoregressive \(AR\), non\-autoregressive \(NAR\), and latent\-reasoning baseline Coconut\([Hao et al\., 2025](https://arxiv.org/html/2610.12879#bib.bib29)\)\. We report answer accuracy and wall\-clock time per sample for each benchmark in Table[4](https://arxiv.org/html/2610.12879#S5.T4)\.

- •Coconutperforms latent reasoning by feeding continuous thought representations back into the model’s hidden states instead of emitting explicit chain\-of\-thought tokens\.

### B\.7Training dynamics

Figure[8](https://arxiv.org/html/2610.12879#A2.F8)tracks Gen\-PPL and unigram entropy over training for the conditional runs\. The first200200k steps are the unconditional warm\-up phase, during which the context is filled with a constant token under normal attention\. The drift reaches coherent English and drives Gen\-PPL down by orders of magnitude; real prefix conditioning then turns on and perplexity keeps falling\.

![Refer to caption](https://arxiv.org/html/2610.12879v1/training_512.png)

![Refer to caption](https://arxiv.org/html/2610.12879v1/training_512_dom.png)

Figure 8:Training dynamics of the conditional runs\.Gen\-PPL \(log scale, left axis\) and unigram entropy \(right axis\) versus training step, for Seq\-DriftingP64→6464\{\\to\}64\(top left\), Seq\-DriftingS64→6464\{\\to\}64\(top right\), Seq\-DriftingP64→51264\{\\to\}512\(bottom left\), and Seq\-DriftingS64→51264\{\\to\}512\(bottom right\)\. All panels show training through300300k steps for both output lengths\. The dashed line marks the warm\-up→\\toconditional switch at200200k steps\.
### B\.8Off\-policy positives

The*off\-policy*ablation of §[5\.4](https://arxiv.org/html/2610.12879#S5.SS4)constructs support sets from corpus retrieval instead of language\-model predictions\. We precompute a memory bank of the tokennn\-gram embeddings of a set of real sequences, and at each generated position we take as the support set𝒮t\\mathcal\{S\}\_\{t\}the tokens that real sequences place at the position where they best match the current generation, ranked by the position\-aligned min\-nn\-gram distance of §[4\.2](https://arxiv.org/html/2610.12879#S4.SS2)\. Everything downstream, the angular attraction, the two repulsions, and the geodesic update, is unchanged\. The one difference from the on\-policy signal is that these targets are*fixed*by the data and do not follow the generator\.

#### Observed training behavior\.

Figure[9](https://arxiv.org/html/2610.12879#A2.F9)tracks this run at length128128\. During the temperature warm\-up, over which the attraction temperatureτ\\tauanneals from1\.01\.0to0\.10\.1across the first1010k steps, Gen\-PPL falls as the soft, mode\-covering attraction pulls the batch toward realnn\-grams\. Onceτ\\taureaches its sharp steady value at1010k, each generation commits hard to its single nearest realnn\-gram\. Because those targets are fixed and do not adapt, the sharp commitment lets the anti\-collapse repulsion over\-scatter the batch faster than the fixed attraction can pull it back, which sets up a limit cycle\. The repulsion over\-scatters the batch, so entropy and Gen\-PPL spike\. The samples are then far apart, the repulsion weakens, and the attraction pulls them back toward real tokens, so entropy and Gen\-PPL fall\. Once re\-consolidated the repulsion strengthens again and the cycle repeats, which is the sawtooth in Figure[9](https://arxiv.org/html/2610.12879#A2.F9)\. The cycle does not close, each swing ends higher than the last, so Gen\-PPL and entropy trend upward overall and the run moves toward higher\-entropy text with reduced global coherence\. The on\-policy support model avoids this this because it scores the generation’s own prefix, so its targets follow the generator and the attraction stays balanced against the repulsion\.

Figure 9:Off\-policy training dynamics\(unconditional, length128128\)\. Gen\-PPL \(log scale, left axis\) and unigram entropy \(right axis\) versus training step for the off\-policy positives ablation\. The dashed line marks the end of the temperature warm\-up at1010k steps, whereτ\\taureaches0\.10\.1and the run begins to oscillate and degrade\.

### B\.9Qualitative samples

The following examples are produced in a single forward pass and truncated for display\.

Seq\-DriftingPentropy: 4\.30 Gen\-PPL: 36\.01It will provide an excellent overview of a wide variety of topics: "Innovations & Challenges of the Future," "A History of Technology" and, the latest developments and advancements that the technology has in general and the development and future capabilities in specific applications\. This is one of our largest conferences ever organized for industry\-leading attendees in North American technology industry, which will give you more insight as you go to work with technology\.About the Author:Klaus K\. Krieg was awarded his doctor, …

Seq\-DriftingSentropy: 4\.21 Gen\-PPL: 93\.81A spokeswoman at the Brown’s Downing Street offices said: "There it would not happen here at home, will be able to make up his mind whether that’s right another very me\. i guess he has been very, but if he did have been opportunity, if he has been very very me\. the so far as well \.\.\. and this time around there, if he did\. i think terms people who knows about what happened on afternoon, …

Seq\-Drifting WMTSource:Wir sind stolz auf unsere Leistung, aber wir wollen jedes Spiel gewinnen\.Reference:We are proud of our performance, but we want to win every game\.Prediction:We are proud of our performance, but we want to win every game\.

Seq\-Drifting XSumSource:Police said officers from the North West counter terrorism unit searched an address on Peakdale Avenue, Crumpsall, Manchester on Friday\. The suspect, 26, was arrested the following day on suspicion of offences under the Terrorism Act\. Police would not comment on the nature of the alleged offence but said it believed it caused "no threat" to the community\.Reference:A man is being held in Manchester on suspicion of terrorism offences\.Prediction:A man has been arrested on suspicion of terrorism offences

Seq\-Drifting Math ReasoningQuestion:A group of students are preparing for a water balloon fight\. Each student needs 5 balloons\. If there are 12 students, how many balloons are needed in total?Prediction:<<5\*12=60\>\> \#\#\#\# 60

Seq\-Drifting ProofWriterQuestion:Theory: Bob is kind\. Bob is nice\. Bob is white\. Charlie is kind\. Charlie is white\. Erin is red\. Erin is rough\. Harry is blue\. Harry is kind\. Harry is red\. Blue people are nice\. All white people are red\. If someone is white and blue then they are nice\. All rough people are red\. If someone is smart then they are blue\. If someone is kind then they are red\. If someone is nice then they are rough\. If someone is red then they are smart\. If someone is red and rough then they are nice\.Query:Bob is not nice\. Give the proof in order and finish with \#\#\#\# true or \#\#\#\# false\.Prediction:Fact: Bob is nice\. \#\#\#\# false