Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance

# Boosting Diversity in Text Diffusion Models via Entropy‑Based Guidance
Source: [https://arxiv.org/html/2608.00024](https://arxiv.org/html/2608.00024)
## Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy‑Based Guidance

###### Abstract

Although diffusion models have revolutionized continuous domains like image synthesis through high quality generations and controllable guidance mechanisms, bringing this controllability to the discrete, sequential nature of text remains an open challenge\. Meanwhile, current sampling strategies and guidance methods adjust token likelihoods without capturing the broader semantic landscape, leading to a suboptimal balance between fidelity and diversity\. In this work, we introduce a novel training\-free Semantic\-Aware Kernel Entropy \(SAKE\) guidance method\. Our method computes the order\-2 Rényi entropy over a kernel Gram matrix that captures both cross\-token semantic interactions and relative token positions\. By linearizing this objective in the embedding space, we derive a tractable guidance signal that dynamically adjusts the sampling distribution—flattening it to encourage exploration during redundancy and sharpening it for fidelity when diverse\. Empirical experiments demonstrate that our approach achieves a superior Pareto frontier between fidelity and diversity, and improves multi\-sample performance on reasoning\-intensive tasks, such as code and mathematics generation, compared to temperature scaling and discrete guidance baselines\.

Machine Learning, ICML

## 1Introduction

Text generation has long been dominated by autoregressive \(AR\) models\(Brownet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib47); Liuet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib46); Yanget al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib27)\), which decompose the joint probability of a sequence into a product of conditional next\-token probabilities\. While highly effective, AR generation is inherently constrained by serial latency and a lack of bidirectional context during the decoding process\. Recently, Diffusion Language Models \(DLMs\) have emerged as a compelling non\-autoregressive alternative, offering a paradigm shift toward parallel decoding\(Austinet al\.,[2021](https://arxiv.org/html/2608.00024#bib.bib18); Nieet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib21); Yeet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib22)\)\. The core mechanism of modern DLMs is a diffusion process operating directly within the discrete categorical token space\. This involves a forward corruption process that gradually transitions valid tokens into noise, paired with a generative model trained to reverse this trajectory\. Unlike autoregressive approaches, this framework enables simultaneous and iterative refinement of all tokens\. Consequently, DLMs leverage global bidirectional context throughout generation and enable flexible trade\-offs between computational cost and sample quality\.

The success of continuous diffusion models in image and audio synthesis is largely driven by their controllability through inference\-time guidance\(Dhariwal and Nichol,[2021](https://arxiv.org/html/2608.00024#bib.bib4); Ho and Salimans,[2022](https://arxiv.org/html/2608.00024#bib.bib5); Bansalet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib35)\)\. Such guidance modifies the reverse diffusion dynamics to sample from desired conditional distributions without retraining\. However, extending these paradigms to the discrete, sequential domain of text generation presents fundamental challenges\. In continuous spaces, guidance usually operates by modifying the score function \(the gradient of the data log\-likelihood\) via a differentiable energy function\. In contrast, in the discrete domain, gradients are undefined, and the score function is based on a categorical posterior distribution\. Modifying this distribution requires re\-normalization, which is computationally intractable due to the need to calculate a partition function that sums overVLV^\{L\}possible sequences, whereLLis the sequence length andVVthe vocabulary size\. Although recent methods reduce this exponential cost to linear by assuming token independence\(Sahooet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib19)\), such factorizations ignore the complex cross\-token correlations that are essential for coherent semantic formulation\.

Besides computational intractability, discrete text diffusion suffers from a lack of semantic structure in the logit space, in contrast to the continuous diffusion that exploits the intrinsic semantic geometry of the continuous feature space\. Because discrete models estimate transition probabilities over categorical distributions, the resulting logits encode model confidence rather than semantic similarity\. Consequently, existing DLM sampling and guidance strategies \(e\.g\., temperature scaling, D\-CFG\(Sahooet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib19)\)\) are limited to manipulating token likelihoods without awareness of the broader semantic landscape\. To induce diversity, these methods rely on non\-semantic distribution flattening via high\-temperature scaling rather than navigating the textual manifold\. This fundamentally limits the fidelity\-diversity trade\-off, as diversity is obtained through random noise injection into the probability mass function rather than through structured semantic exploration\.

Diversity is often viewed as a trade\-off against quality, but we argue that in complex reasoning, diversity in chain\-of\-thoughts \(CoT\) is a prerequisite for robustness rather than a mere source of variety\. Prior work such as Self\-Consistency\(Wanget al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib23)\)and Tree\-of\-Thoughts\(Yaoet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib24)\)shows that generating multiple distinct reasoning paths increases the probability of recovering the correct answer\. Diversity prevents the model’s probability distribution from collapsing onto a single, potentially erroneous chain of thought\. When exploration is restricted to a narrow cluster of high\-likelihood tokens, the model risks reinforcing its initial biases, leading to premature convergence on flawed reasoning\. Consistent with this, our empirical results show that diversity\-guided chains of thought significantly improves complex reasoning task pass rates, validating the utility of exploration in deterministic tasks\.

In this paper, we study the guidance for DLMs in the discrete, sequential setting\. Given a trained diffusion language model, we guide sampling to maximize task success in a way that maintains cross\-token interactions, preserving diversity and semantic validity\. Our contributions are threefold\.

- •We propose a general computationally efficient*diversity guidance*framework for discrete text diffusion\. This framework overcomes the limitations of prior token\-independent assumptions by integrating sequence\-level semantic information\. Building on this framework, we propose*Semantic\-Aware Kernel Entropy*\(SAKE\), a tractable objective that models sequence semantic diversity, and explicitly maximize Rényi entropy of the sequence\.
- •We demonstrate that SAKE functions as an*adaptive distribution modulator*\. Unlike static temperature scaling, our method dynamically adjusts the generation probability: it flattens the distribution to encourage exploration during mode collapse and sharpens it to ensure coherence when intrinsic diversity is already high\.
- •Empirical evaluations on both synthetic settings and standard LLM benchmarks confirm that our approach yields a superior trade\-off between quality and diversity compared to discrete guidance baselines and temperature\-based sampling\. Notably, applying SAKE to LLaDA\-8B\-base yields significant performance gains, improving HumanEval pass@32 \(41\.1→55\.841\.1\\rightarrow 55\.8\), MBPP pass@32 \(48\.2→56\.148\.2\\rightarrow 56\.1\), and GSM8K self\-consistency \(71\.5→75\.171\.5\\rightarrow 75\.1\)\.

## 2Related Works

### 2\.1Diffusion Models for Text Generation

Early text diffusion operated in continuous embedding spaces\(Liet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib16); Yuanet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib17)\), but the performance is limited compared to AR baselines due to a challenge mapping denoised continuous vector back to a coherent discrete token\. To address the embedding bottleneck, recent research has shifted toward discrete diffusion processes that operate directly on the categorical token space\. Foundational work D3PM\(Austinet al\.,[2021](https://arxiv.org/html/2608.00024#bib.bib18)\)formalized diffusion over discrete states using transition matrices, defining noise processes such as uniform corruption or corruption to a specific mask token \(absorbing state\)\. Building on this, modern Diffusion Language Models \(DLMs\) have scaled significantly\(Louet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib3); Sahooet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib19); Ouet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib20); Nieet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib21); Yeet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib22)\), and demonstrated that DLMs can match or exceed AR models in perplexity and zero\-shot generation while enabling arbitrary infilling and iterative refinement\.

### 2\.2Guidance Mechanisms for Diffusion Models

Continuous score\-based diffusion models\(Sohl\-Dicksteinet al\.,[2015](https://arxiv.org/html/2608.00024#bib.bib11); Hoet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib8); Songet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib10); Song and Ermon,[2019](https://arxiv.org/html/2608.00024#bib.bib9)\)has shown remarkable abilities in various generation tasks, including images\(Rombachet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib29); Dhariwal and Nichol,[2021](https://arxiv.org/html/2608.00024#bib.bib4); Hoet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib30)\)and videos\(Heet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib31); Blattmannet al\.,[2023a](https://arxiv.org/html/2608.00024#bib.bib32),[b](https://arxiv.org/html/2608.00024#bib.bib33)\)\. Conditional generation is achieved by guidance, modifying the learned score function during inference\(Dhariwal and Nichol,[2021](https://arxiv.org/html/2608.00024#bib.bib4); Nicholet al\.,[2021](https://arxiv.org/html/2608.00024#bib.bib34); Bansalet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib35); Ho and Salimans,[2022](https://arxiv.org/html/2608.00024#bib.bib5); Corsoet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib36); Askari Hemmatet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib37); Jalaliet al\.,[2025a](https://arxiv.org/html/2608.00024#bib.bib28); Saniet al\.,[2026](https://arxiv.org/html/2608.00024#bib.bib65); Jalaliet al\.,[2026](https://arxiv.org/html/2608.00024#bib.bib52)\)\. As mentioned earlier, transferring these techniques to discrete diffusion is mathematically non\-trivial\. To make guidance tractable,Schiffet al\.\([2024](https://arxiv.org/html/2608.00024#bib.bib1)\)proposed Discrete Classifier\-Free Guidance \(D\-CFG\) and Discrete Classifier\-Based Guidance \(D\-CBG\), which rely on an independence assumption that assumes token transitions are independent\. Although this reduces the normalization cost from exponential to linear with sequence length, the independence assumption ignores the cross\-token correlations that are vital for semantic formulation\.

### 2\.3Diversity in Generation Tasks

Beyond generation quality, diversity has increasingly become a critical value within the research community\. Various metrics have been proposed to quantify diversity in both image\(Sajjadiet al\.,[2018](https://arxiv.org/html/2608.00024#bib.bib39); Jalaliet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib6); Ospanovet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib53); Ospanov and Farnia,[2025](https://arxiv.org/html/2608.00024#bib.bib60); Ospanovet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib61)\)and text\(Zhuet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib38)\)domains\. In auto\-regressive text generation, several methods extending beyond temperature scaling have been developed to enhance output variety\(Vijayakumaret al\.,[2016](https://arxiv.org/html/2608.00024#bib.bib43); Holtzmanet al\.,[2019](https://arxiv.org/html/2608.00024#bib.bib44); Nguyenet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib45)\)\. Also, the entropy\-based novelty of sample generation has been studied in\(Zhanget al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib62),[2025](https://arxiv.org/html/2608.00024#bib.bib63); Lotfianet al\.,[2026](https://arxiv.org/html/2608.00024#bib.bib68)\), and the related works\(Huet al\.,[2025a](https://arxiv.org/html/2608.00024#bib.bib56); Rezaeiet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib58); Huet al\.,[2025b](https://arxiv.org/html/2608.00024#bib.bib57); Jafari and Farnia,[2026](https://arxiv.org/html/2608.00024#bib.bib67); Nia and Farnia,[2026](https://arxiv.org/html/2608.00024#bib.bib66)\)study online diversity\-aware evaluation of generative models\. The role and comparison of embeddings for diversity evaluation has been analyzed in\(Steinet al\.,[2023](https://arxiv.org/html/2608.00024#bib.bib69); Jalaliet al\.,[2025b](https://arxiv.org/html/2608.00024#bib.bib54); Wuet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib49); Gonget al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib55); Wu and Farnia,[2026](https://arxiv.org/html/2608.00024#bib.bib50); Wuet al\.,[2026](https://arxiv.org/html/2608.00024#bib.bib51)\)\. Meanwhile, diffusion models have demonstrated remarkable capability in controlling diversity for image generation, due to the flexible inference mechanisms\. However, while training\-free guidance techniques have proven effective for continuous image diffusion\(Sadatet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib41); Corsoet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib36); Kirchhofet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib42); Jalaliet al\.,[2025a](https://arxiv.org/html/2608.00024#bib.bib28)\), the diversity guidance in discrete diffusion models for text generation is unexplored\.

## 3Preliminaries

Continuous Diffusion Generative Models\.Diffusion Models \(DMs\)\(Hoet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib8); Song and Ermon,[2019](https://arxiv.org/html/2608.00024#bib.bib9); Songet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib10)\)are generative models designed to sample from a target data distributionpdata​\(𝐱0\)p\_\{\\text\{data\}\}\(\\mathbf\{x\}\_\{0\}\)by reversing a predefined forward noising process\(Sohl\-Dicksteinet al\.,[2015](https://arxiv.org/html/2608.00024#bib.bib11)\)\. In the forward diffusion process, a data sample𝐱0\\mathbf\{x\}\_\{0\}is progressively perturbed with Gaussian noise over a continuous time intervalt∈\[0,T\]t\\in\[0,T\]\. This process is mathematically described as adding noise to obtain a noisy state𝐱t=αt​𝐱0\+1−αt​ϵt\\mathbf\{x\}\_\{t\}=\\sqrt\{\\alpha\_\{t\}\}\\mathbf\{x\}\_\{0\}\+\\sqrt\{1\-\\alpha\_\{t\}\}\\boldsymbol\{\\epsilon\}\_\{t\}, whereϵt∼𝒩​\(𝟎,𝐈\)\\boldsymbol\{\\epsilon\}\_\{t\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)represents standard Gaussian noise, andαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\]is a monotonically decreasing schedule controlling the noise level\. DMs\(Hoet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib8)\)train a neural networkϵθ:𝒳×\[T\]↦𝒳\\boldsymbol\{\\epsilon\}\_\{\\theta\}:\\mathcal\{X\}\\times\[T\]\\mapsto\\mathcal\{X\}to predict the noiseϵt\\boldsymbol\{\\epsilon\}\_\{t\}at each time steptt\. This objective implicitly learns thescore functionof the marginal distributionpt​\(𝐱t\)p\_\{t\}\(\\mathbf\{x\}\_\{t\}\)\(Song and Ermon,[2019](https://arxiv.org/html/2608.00024#bib.bib9); Songet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib10)\):

minθ⁡𝔼𝐱t,ϵt,t​\[‖ϵθ​\(𝐱t,t\)\+1−αt​∇𝐱tlog⁡pt​\(𝐱t\)⏟Score Function‖22\],\\min\_\{\\theta\}\\mathbb\{E\}\_\{\\mathbf\{x\}\_\{t\},\\boldsymbol\{\\epsilon\}\_\{t\},t\}\\left\[\\left\\\|\\boldsymbol\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)\+\\sqrt\{1\-\\alpha\_\{t\}\}\\underbrace\{\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log p\_\{t\}\(\\mathbf\{x\}\_\{t\}\)\}\_\{\\text\{Score Function\}\}\\right\\\|\_\{2\}^\{2\}\\right\],\(1\)
During inference, samples are generated by solving the reverse\-time SDE fromt=Tt=Ttot=0t=0\. Crucially, there exists a corresponding deterministic process known as theprobability flow ODE\(PF\-ODE\), whose trajectories share the same marginal distributions\{pt​\(𝐱t\)\}t∈\[0,T\]\\\{p\_\{t\}\(\\mathbf\{x\}\_\{t\}\)\\\}\_\{t\\in\[0,T\]\}as the SDE\(Songet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib10); Lipmanet al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib14)\)\.

For conditional generation tasks like text\-to\-image synthesis, the objective of diffusion models is to learn the conditional distributionp​\(𝐱0∣y\)p\(\\mathbf\{x\}\_\{0\}\\mid y\)given a conditionyy\. Following the score matching framework\(Song and Ermon,[2019](https://arxiv.org/html/2608.00024#bib.bib9); Songet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib10)\), the corresponding conditional score can be expressed as:

∇𝐱tlog⁡pt​\(𝐱t\|y\)⏟Conditional Score=∇𝐱tlog⁡pt​\(𝐱t\)⏟Unconditional Score\+∇𝐱tlog⁡pt​\(y\|𝐱t\)⏟Guidance\.\\underbrace\{\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log p\_\{t\}\(\\mathbf\{x\}\_\{t\}\|y\)\}\_\{\\text\{Conditional Score\}\}=\\underbrace\{\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log p\_\{t\}\(\\mathbf\{x\}\_\{t\}\)\}\_\{\\text\{Unconditional Score\}\}\+\\underbrace\{\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log p\_\{t\}\(y\|\\mathbf\{x\}\_\{t\}\)\}\_\{\\text\{Guidance\}\}\.\(2\)
Discrete Diffusion Process\.The forward diffusion process progressively corrupts discrete data, typically toward an absorbing state \(e\.g\., a mask token\) or a high\-entropy \(often uniform\) noise distribution over the vocabulary\(Austinet al\.,[2021](https://arxiv.org/html/2608.00024#bib.bib18)\)\. In principle, this process could be modeled as a time\-inhomogeneous continuous\-time Markov chain on a finite state space𝒳\\mathcal\{X\}\. The process is governed by a rate matrixQt∈ℝ\|𝒳\|×\|𝒳\|Q\_\{t\}\\in\\mathbb\{R\}^\{\|\\mathcal\{X\}\|\\times\|\\mathcal\{X\}\|\}for each timett, where the off\-diagonal entries are non\-negative and columns sum to zero\. The evolution of the probability mass function \(PMF\)pt∈ℝ\|𝒳\|p\_\{t\}\\in\\mathbb\{R\}^\{\|\\mathcal\{X\}\|\}over the states is described by the Kolmogorov forward equation:

dd​t​pt=Qt​pt\.\\frac\{d\}\{dt\}p\_\{t\}=Q\_\{t\}p\_\{t\}\.\(3\)The corresponding reverse process, which generates data by reversing the corruption, is also a Markov process\. Its exact transition ratesQ¯t\\overline\{Q\}\_\{t\}are usually defined using the forward rates and the*concrete score*ratio\(Menget al\.,[2022](https://arxiv.org/html/2608.00024#bib.bib2)\),rt​\(y∣x\)=pt​\(y\)/pt​\(x\)r\_\{t\}\(y\\mid x\)=p\_\{t\}\(y\)/p\_\{t\}\(x\):

Q¯t​\(y,x\)=Qt​\(x,y\)​rt​\(y∣x\),\\overline\{Q\}\_\{t\}\(y,x\)=Q\_\{t\}\(x,y\)r\_\{t\}\(y\\mid x\),\(4\)with the diagonal entries defined asQ¯t​\(x,x\)=−∑y≠xQ¯t​\(y,x\)\\overline\{Q\}\_\{t\}\(x,x\)=\-\\sum\_\{y\\neq x\}\\overline\{Q\}\_\{t\}\(y,x\)\. As the true score ratiortr\_\{t\}is intractable, in practice it is approximated with a neural networksθ​\(x,t\)​\[y\]≈rt​\(y∣x\)s\_\{\\theta\}\(x,t\)\[y\]\\approx r\_\{t\}\(y\\mid x\)\. This yields a learned, normalized reverse transition probability kernel fory≠xy\\neq x:

Pt​\(y∣x\)=Qt​\(x,y\)​sθ​\(x,t\)​\[y\]∑z≠xQt​\(x,z\)​sθ​\(x,t\)​\[z\]\.P\_\{t\}\(y\\mid x\)=\\frac\{Q\_\{t\}\(x,y\)s\_\{\\theta\}\(x,t\)\[y\]\}\{\\sum\_\{z\\neq x\}Q\_\{t\}\(x,z\)s\_\{\\theta\}\(x,t\)\[z\]\}\.\(5\)

## 4Methodology

### 4\.1Unified Framework for Text Diffusion Guidance

Let𝒱\\mathcal\{V\}denote the vocabulary, and the base model’s logits denoted as𝐳∈ℝ\|𝒱\|\\mathbf\{z\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}, inducing a probability distributionPbase​\(y\)=softmax​\(𝐳\)yP\_\{\\mathrm\{base\}\}\(y\)=\\mathrm\{softmax\}\(\\mathbf\{z\}\)\_\{y\}over tokensy∈𝒱y\\in\\mathcal\{V\}\.

We define a generalized class of guided distributionPγP\_\{\\gamma\}, parameterized by a guidance signal vectorψ∈ℝ\|𝒱\|\\mathbf\{\\psi\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}and a guidance scalarγ∈ℝ\\gamma\\in\\mathbb\{R\}, as

Pγ​\(y\)=1Z​\(γ\)​Pbase​\(y\)⋅exp⁡\(γ⋅ψy\),P\_\{\\gamma\}\(y\)=\\frac\{1\}\{Z\(\\gamma\)\}P\_\{\\mathrm\{base\}\}\(y\)\\cdot\\exp\\left\(\\gamma\\cdot\\mathbf\{\\psi\}\_\{y\}\\right\),\(6\)whereZ​\(γ\)Z\(\\gamma\)is the partition function\. This formulation encapsulates a broad class of sampling strategies\. To characterize the behavior of these strategies, we analyze the local sensitivity of the Shannon entropyH​\[Pγ\]H\[P\_\{\\gamma\}\]with respect to the guidance scalarγ∈ℝ\\gamma\\in\\mathbb\{R\}\.

###### Proposition 4\.1\(Entropy Dynamics of Text Diffusion Guidance\)\.

The first\-order change in the Shannon entropy of the guided distributionPγP\_\{\\gamma\}at the limit of zero guidance is governed by the negative covariance between the base log\-probabilities and the guidance signal:

∇γH​\[Pγ\]\|γ=0=−CovY∼Pbase​\[log⁡Pbase​\(Y\),ψY\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{Y\\sim P\_\{\\mathrm\{base\}\}\}\\big\[\\log P\_\{\\mathrm\{base\}\}\(Y\),\\mathbf\{\\psi\}\_\{Y\}\\big\]\.\(7\)Sincelog⁡Pbase​\(y\)=𝐳y−log⁡Z​\(0\)\\log P\_\{\\mathrm\{base\}\}\(y\)=\\mathbf\{z\}\_\{y\}\-\\log Z\(0\), and covariance is translation invariant under constant shifts \(such aslog⁡Z​\(0\)\\log Z\(0\)\), the gradient simplifies to:

∇γH​\[Pγ\]\|γ=0=−CovPbase​\[𝐳,ψ\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[\\mathbf\{z\},\\mathbf\{\\psi\}\]\.\(8\)

###### Proof\.

We defer the proof to the Appendix ∎

This proposition establishes that guidance acts as an entropy reducer if and only if the signalψ\\mathbf\{\\psi\}is positively correlated with the base model’s logits𝐳\\mathbf\{z\}\.

### 4\.2Analysis of Existing Training\-free Guidance Mechanisms

We now apply Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1)to characterize the behavior of existing strategies\.

Temperature Scaling\.Temperature scaling is a commonly used sampling strategy that rescales the base logits by a temperature parameterτ\\tau\. For0<τ<10<\\tau<1, standard temperature sampling is equivalent to setting the scalar strengthγ=1τ−1\\gamma=\\frac\{1\}\{\\tau\}\-1and defining the guidance signalψ\\mathbf\{\\psi\}for temperature scaling as the logits themselves, i\.e\.,ψtemp=𝐳\\mathbf\{\\psi\}\_\{\\mathrm\{temp\}\}=\\mathbf\{z\}\. Substituting this into Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1)yields:

∇γH\|γ=0=−VarPbase​\[𝐳\]≤0\.\\left\.\\nabla\_\{\\gamma\}H\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Var\}\_\{P\_\{\\mathrm\{base\}\}\}\[\\mathbf\{z\}\]\\leq 0\.\(9\)Since variance is strictly non\-negative, temperature scaling monotonically reduces entropy\. More broadly, alternative sampling strategies such as Top\-kkand Nucleus sampling explicitly truncate the probability distribution, which inevitably constrains the diversity of generated outputs\.

Discrete Classifier\-Free Guidance \(D\-CFG\)\.D\-CFG extrapolates between an unconditional estimate𝐳∅\\mathbf\{z\}\_\{\\emptyset\}and a conditional estimate𝐳c\\mathbf\{z\}\_\{c\}\. Treating the unconditional distribution as the basePbaseP\_\{\\mathrm\{base\}\}, the guidance signal is given by the residual vectorψcfg=𝐳c−𝐳∅\\mathbf\{\\psi\}\_\{\\mathrm\{cfg\}\}=\\mathbf\{z\}\_\{c\}\-\\mathbf\{z\}\_\{\\emptyset\}\. Substituting this into Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1)and exploiting the linearity of covariance, the entropy gradient decomposes into two competing terms:

∇γH\|γ=0=VarP∅​\[𝐳∅\]−CovP∅​\[𝐳∅,𝐳c\]\.\\left\.\\nabla\_\{\\gamma\}H\\right\|\_\{\\gamma=0\}=\\mathrm\{Var\}\_\{P\_\{\\emptyset\}\}\[\\mathbf\{z\}\_\{\\emptyset\}\]\-\\mathrm\{Cov\}\_\{P\_\{\\emptyset\}\}\[\\mathbf\{z\}\_\{\\emptyset\},\\mathbf\{z\}\_\{c\}\]\.\(10\)D\-CFG typically acts as an entropy reducer by sharpening the distribution around the mode, it increases entropy only in specific cases of misalignment between the conditional and prior models \(characterized by low covariance between𝐳∅\\mathbf\{z\}\_\{\\emptyset\}and𝐳c\\mathbf\{z\}\_\{c\}\)\.

In summary, existing control mechanisms are less suited for enhancing meaningful diversity\. Although temperature scaling can increase entropy, it does so at the cost of injecting indiscriminate noise, whereas truncation methods are explicitly designed to suppress the distribution tail\. Similarly, D\-CFG lacks a consistent mechanism to promote variation\. Furthermore, these standard methods operate exclusively in the logit space, which primarily reflects model confidence rather than semantic content\.

The sharpening behavior of standard methods, combined with the lack of semantic awareness in the logit space, fundamentally limits diversity in text generation tasks\. To overcome these limitations, we propose SAKE guidance in the following subsection, a mechanism that leverages the semantic information inherent in the sequence to actively promote diverse generation\. SAKE also supports bi\-directional entropy modulation, which allows the model to dynamically switch between exploration and exploitation\.

### 4\.3Our Method: Semantic\-Aware Kernel Entropy Guidance

In this subsection, we proceed in three steps\. First, we formalize the general discrete diversity signal used for guidance\. Second, we resolve the intractability of the discrete formulation via a computationally efficient embedding\-space linearization\. Finally, we introduce a semantic\-aware diversity function that improve sequence spectral diversity by maximizing the order\-2 Rényi entropy\.

#### 4\.3\.1Discrete Diversity Guidance

General Discrete Diversity Guidance Signal\.We first define the general discrete diversity signal for DLMs as:

ψdiv​\(y\)=𝒟​\(y;𝐱−i\)−𝒟​\(xi;𝐱−i\)\\mathbf\{\\psi\}\_\{\\mathrm\{div\}\}\(y\)=\\mathcal\{D\}\\big\(y;\\mathbf\{x\}\_\{\-i\}\\big\)\-\\mathcal\{D\}\\big\(x\_\{i\};\\mathbf\{x\}\_\{\-i\}\\big\)\(11\)where𝒟:ℕL→ℝ\\mathcal\{D\}:\\mathbb\{N\}^\{L\}\\rightarrow\\mathbb\{R\}is the discrete diversity function,LLis the sequence length,i≤Li\\leq Lis the token index,𝐱−i\\mathbf\{x\}\_\{\-i\}is the sequence except token at indexii\. In this case,ψdiv​\(y\)\\mathbf\{\\psi\}\_\{\\mathrm\{div\}\}\(y\)measures the diversity ”gap” \(or potential difference\) between a proposal for the next statey∈𝒱y\\in\\mathcal\{V\}and the current statexix\_\{i\}\. Therefore, a token that brings more diversity gain will be more likely generated after guidance\. In combination with Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1), the sign of the covariance now becomes indefinite\. Unlike static sharpening methods, diversity guidance creates a dynamic feedback loop:

- •Exploration \(entropy increase\):If the base prediction𝐳\(i\)\\mathbf\{z\}^\{\(i\)\}is semantically redundant \(highly similar to other elements𝐳\(j\)\\mathbf\{z\}^\{\(j\)\}\), the signalψdiv\(i\)\\mathbf\{\\psi\}\_\{\\mathrm\{div\}\}^\{\(i\)\}will oppose the direction of𝐳\(i\)\\mathbf\{z\}^\{\(i\)\}\. This yields a negative covariance, increasing entropy and encouraging the model to explore\.
- •Exploitation \(entropy decrease\):If the base prediction is already diverse, the difference term vanishes, allowing the natural confidence of the model to dominate\.

Algorithm 1Diversity Guidance for Discrete Text Diffusion0:Pretrained DLM

fθf\_\{\\theta\}, Prompt

𝐜\\mathbf\{c\}, Number of steps

TT, Guidance scale

γ\\gamma, Bandwidths

σ,σattn\\sigma,\\sigma\_\{\\text\{attn\}\}, Token embedding matrix

E∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}, masking schedule

N​\(t\)N\(t\)\.

1:Initialize:

𝐱0=\[𝐜,\[M\],…,\[M\]\]\\mathbf\{x\}\_\{0\}=\[\\mathbf\{c\},\\text\{\[M\]\},\\dots,\\text\{\[M\]\}\]
2:for

t=1t=1to

TTdo

3:Logits

𝐙t,Embedding​𝐇t=fθ​\(𝐱t−1\)\\mathbf\{Z\}\_\{t\},\\text\{Embedding \}\\mathbf\{H\}\_\{t\}=f\_\{\\theta\}\(\\mathbf\{x\}\_\{t\-1\}\)
4:Compute semantic kernel

𝐊\\mathbf\{K\}where

𝐊i​j=exp⁡\(−‖𝐡i−𝐡j‖2/2​σ2\)\\mathbf\{K\}\_\{ij\}=\\exp\(\-\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}/2\\sigma^\{2\}\)
5:Compute attention kernel

𝐊attn\\mathbf\{K\}\_\{\\text\{attn\}\}where

𝐊attn,i​j=exp⁡\(−‖i−j‖2/2​σattn2\)\\mathbf\{K\}\_\{\\text\{attn\},ij\}=\\exp\(\-\\\|i\-j\\\|^\{2\}/2\\sigma\_\{\\text\{attn\}\}^\{2\}\)
6:Initialize

G=𝟎d×LG=\\mathbf\{0\}^\{d\\times L\}
7:for

i=1i=1to

LLdo

8:

𝐠i=∑j≠i\(𝐊attn,i​j⋅𝐊i​j\)2⋅\(𝐡i−𝐡j\)\\mathbf\{g\}\_\{i\}=\\sum\_\{j\\neq i\}\(\\mathbf\{K\}\_\{\\text\{attn\},ij\}\\cdot\\mathbf\{K\}\_\{ij\}\)^\{2\}\\cdot\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)
9:

G​\[:,i\]=2σ2​𝐠iG\[:\\,,\\,i\]=\\frac\{2\}\{\\sigma^\{2\}\}\\mathbf\{g\}\_\{i\}
10:endfor

11:Guidance Signal

𝚿div=E​G\\mathbf\{\\Psi\}\_\{\\text\{div\}\}=E\\,G
12:

𝐙~t=𝐙t\+γ⋅𝚿div\\tilde\{\\mathbf\{Z\}\}\_\{t\}=\\mathbf\{Z\}\_\{t\}\+\\gamma\\cdot\\mathbf\{\\Psi\}\_\{\\text\{div\}\}
13:Sample

𝐱t\\mathbf\{x\}\_\{t\}based on

N​\(t\)N\(t\)and guided logits

𝐙~t\\tilde\{\\mathbf\{Z\}\}\_\{t\}
14:endfor

However, considering the size ofψdiv\\mathbf\{\\psi\}\_\{\\mathrm\{div\}\}is\|𝒱\|\|\\mathcal\{V\}\|, and current DLMs usually adopt a huge vocabulary size in practice\. Guiding a sequence of lengthLLwill lead toL×\|𝒱\|L\\times\|\\mathcal\{V\}\|evaluations of the diversity function, which is unacceptable for real\-time DLM inferences\. We will first show how to efficiently calculate the diversity signal\.

#### 4\.3\.2Efficient Compute via Embedding\-Space Linearization\.

To avoid the brute\-force evaluation of the discrete diversity gap, we linearize the diversity function within the continuous embedding space\. This allows us to replace the discrete difference with a gradient projection\.

Let𝐡y\\mathbf\{h\}\_\{y\}and𝐡xi\\mathbf\{h\}\_\{x\_\{i\}\}denote the continuous embedding vectors for a candidate tokenyyand the current tokenxix\_\{i\}, respectively\. Let𝐇x−i\\mathbf\{H\}\_\{x\_\{\-i\}\}be the matrix of token embeddings for a sequence except tokenii\. Assuming continous diversity function𝒟~\\tilde\{\\mathcal\{D\}\}is differentiable with respect to the embedding of theii\-th token, the diversity potential for candidateyycan be approximated as:

ψ~div\(i\)​\(y\)\\displaystyle\\tilde\{\\psi\}^\{\(i\)\}\_\{\\mathrm\{div\}\}\(y\)≜𝒟~​\(𝐡y;𝐇x−i\)−𝒟~​\(𝐡xi;𝐇x−i\)\\displaystyle\\triangleq\\tilde\{\\mathcal\{D\}\}\\big\(\\mathbf\{h\}\_\{y\};\\mathbf\{H\}\_\{x\_\{\-i\}\}\\big\)\-\\tilde\{\\mathcal\{D\}\}\\big\(\\mathbf\{h\}\_\{x\_\{i\}\};\\mathbf\{H\}\_\{x\_\{\-i\}\}\\big\)\(12\)≈⟨∇𝐡xi𝒟~​\(𝐡xi;𝐇x−i\),𝐡y−𝐡xi⟩,\\displaystyle\\approx\\left\\langle\\nabla\_\{\\mathbf\{h\}\_\{x\_\{i\}\}\}\\tilde\{\\mathcal\{D\}\}\(\\mathbf\{h\}\_\{x\_\{i\}\};\\mathbf\{H\}\_\{x\_\{\-i\}\}\),\\,\\mathbf\{h\}\_\{y\}\-\\mathbf\{h\}\_\{x\_\{i\}\}\\right\\rangle,\(13\)since𝐡xiT​∇𝐡xi𝒟~​\(𝐡xi;𝐇x−i\)\\mathbf\{h\}\_\{x\_\{i\}\}^\{T\}\\nabla\_\{\\mathbf\{h\}\_\{x\_\{i\}\}\}\\tilde\{\\mathcal\{D\}\}\(\\mathbf\{h\}\_\{x\_\{i\}\};\\mathbf\{H\}\_\{x\_\{\-i\}\}\)is a fixed term for allyyand softmax is shift\-invariant, we may ignore this term and the final guidance signal can be approximated as:

ψ~div\(i\)≈E​∇𝐡xi𝒟~​\(𝐡xi;𝐇x−i\),\\tilde\{\\psi\}^\{\(i\)\}\_\{\\mathrm\{div\}\}\\approx E\\,\\nabla\_\{\\mathbf\{h\}\_\{x\_\{i\}\}\}\\tilde\{\\mathcal\{D\}\}\(\\mathbf\{h\}\_\{x\_\{i\}\};\\mathbf\{H\}\_\{x\_\{\-i\}\}\),\(14\)whereE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}is the vocabulary embedding matrix containing all candidate tokens\. This approximation reduces the computation fromL×\|𝒱\|L\\times\|\\mathcal\{V\}\|evaluations of discrete diversity function𝒟\\mathcal\{D\}to a single backward pass of a continuous diversity function𝒟~\\tilde\{\\mathcal\{D\}\}\. This formulation establishes a generic, computationally efficient framework for diversity guidance, applicable to any differentiable diversity metric𝒟~\\tilde\{\\mathcal\{D\}\}without incurring the latency costs of brute\-force search\.

Eq\.[13](https://arxiv.org/html/2608.00024#S4.E13)relies on a first\-order Taylor expansion, assuming a smooth continuous diversity functionD~\\tilde\{D\}and bounded embedding distances‖𝐡𝐲−𝐡𝐱𝐢‖\\\|\\mathbf\{h\_\{y\}\}\-\\mathbf\{h\_\{x\_\{i\}\}\}\\\|\. These assumptions hold in modern LLMs \(like LLaDA\) because LayerNorm/RMSNorm constrains embeddings, and Gaussian kernel in ourD~\\tilde\{D\}is smooth\.

A theoretical limitation of this linear approximation is that, unlike the true bounded RBF kernel, it does not plateau\. Under large guidance scales \(γ\\gamma\), an outlier token lying far along the gradient could receive an overestimated diversity score\. Fortunately, this edge case is controllable\. By restricting the guidance signal to a ’trust region’, simply applying a mild Top\-p or Top\-k filter to the base logits before guidance, we ensure diversification only occurs among tokens that are semantically plausible\.

#### 4\.3\.3SAKE: Semantic\-Aware Kernel Entropy Guidance

Having established the efficient diversity guidance framework, we now instantiate the continuous objective𝒟~\\tilde\{\\mathcal\{D\}\}to specifically target semantic redundancy\. We propose maximizing the Rényi entropy of the kernel Gram matrix to promote*sequence*spectral diversity grounded in information\-theoretic principles\.

We used two types of kernel\. One is the*semantic*kernel, which measures the semantic similarity between tokens\. The other one is the*attention*kernels, which measures the weights that tokens attend to each other in sequences\. In our implementation, we leverage the relative positional information of tokens to determine attention weights\.

###### Proposition 4\.3\(SAKE Guidance Signal\)\.

Let𝐇=\[𝐡1,…,𝐡L\]∈ℝd×L\\mathbf\{H\}=\[\\mathbf\{h\}\_\{1\},\\dots,\\mathbf\{h\}\_\{L\}\]\\in\\mathbb\{R\}^\{d\\times L\}be the matrix of token embeddings for a sequence of lengthLL\. Consider the Gaussian RBF kernelκ​\(𝐡i,𝐡j\)=exp⁡\(−‖𝐡i−𝐡j‖2/2​σ2\)\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)=\\exp\(\-\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}/2\\sigma^\{2\}\)with bandwidthσ\>0\\sigma\>0, and let𝐊∈ℝL×L\\mathbf\{K\}\\in\\mathbb\{R\}^\{L\\times L\}be the kernel Gram matrix with𝐊i​j=κ​\(𝐡i,𝐡j\)\\mathbf\{K\}\_\{ij\}=\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)\. Similarly,𝐊attn,\(i,j\)=exp⁡\(−‖i−j‖2/2​σattn2\)\\mathbf\{K\}\_\{\\text\{attn\},\(i,j\)\}=\\exp\(\-\\\|i\-j\\\|^\{2\}/2\\sigma\_\{\\text\{attn\}\}^\{2\}\)\.

![Refer to caption](https://arxiv.org/html/2608.00024v1/comparison_configs_circle.png)Figure 1:Comparison of sampling behaviors for a 2D 8\-Gaussian\-mixture target\.Top row: baseline unguided reverse process at different temperaturesT∈\{1,5,10,20\}T\\in\\\{1,5,10,20\\\}, illustrating a coverage\-fidelity trade\-off \(higherTTcovers more modes but yields more diffuse, lower\-quality samples\)\.Bottom row: Our SAKE with guidance strengthsγ∈\{0\.1,0\.2,0\.5,1\.0\}\\gamma\\in\\\{0\.1,0\.2,0\.5,1\.0\\\}, which improves mode coverage while maintaining better precision\. Rightmost column shows ground\-truth samples from the 8\-Gaussian mixture\.The normalized Hadamard product of kernel Gram matrices serves as a valid quantum density matrix\. The order\-2 Rényi entropy of this representation is:

H2​\(1L​𝐊⊙𝐊attn\)=−log⁡\(1L2​‖𝐊⊙𝐊attn‖F2\)\.H\_\{2\}\(\\frac\{1\}\{L\}\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\)=\-\\log\\left\(\\frac\{1\}\{L^\{2\}\}\\\|\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\\\|\_\{F\}^\{2\}\\right\)\.\(15\)To increaseH2H\_\{2\}by adjusting theii\-th token embedding𝐡i\\mathbf\{h\}\_\{i\}, we define the continuous diversity function:

𝒟~κ​\(𝐡i;𝐇−i\)≜−∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2\.\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}\(\\mathbf\{h\}\_\{i\};\\mathbf\{H\}\_\{\-i\}\)\\triangleq\-\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\.\(16\)where𝐇−i\\mathbf\{H\}\_\{\-i\}denotes the embeddings of all tokens except theii\-th\. The gradient of this function with respect to𝐡i\\mathbf\{h\}\_\{i\}is:

∇𝐡i𝒟~κ=2σ2​∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\)\.\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}=\\frac\{2\}\{\\sigma^\{2\}\}\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\.\(17\)

###### Proof\.

We defer the proof to the Appendix ∎

The gradient∇𝐡i𝒟~κ\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}has an intuitive geometric interpretation as a weighted sum of repulsive forces, where each contextual token𝐡j\\mathbf\{h\}\_\{j\}pushes𝐡i\\mathbf\{h\}\_\{i\}away with strength proportional toκattn​\(i,j\)2​κ​\(𝐡i,𝐡j\)2\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\. By substituting∇𝐡i𝒟~κ\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}into the first\-order approximation from Eq\.[14](https://arxiv.org/html/2608.00024#S4.E14), we obtain the computationally efficient SAKE guidance signal:

ψ~div\(i\)≈E​∇𝐡i𝒟~κ=2​Eσ2​∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\),\\tilde\{\\psi\}^\{\(i\)\}\_\{\\mathrm\{div\}\}\\approx E\\,\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}=\\frac\{2E\}\{\\sigma^\{2\}\}\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\),\(18\)whereE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}is the vocabulary embedding matrix\. This formulation directly connects kernel\-based order\-2 Rényi entropy maximization to practical sequence generation with efficient spectral diversity guidance, which has linear sequence length complexity\.

Algorithm[1](https://arxiv.org/html/2608.00024#alg1)shows the pseudocode of our proposed SAKE method\. The algorithm takes as input a pretrained DLMfθf\_\{\\theta\}, a prompt𝐜\\mathbf\{c\}, the number of diffusion stepsTT, guidance scaleγ\\gamma, kernel bandwidths, an embedding matrixEE, and a masking schedule\. Line 1 first initializes a sequence with the prompt followed by masked tokens\. At each steptt, the model produces logits and token embeddings in line 3, from which semantic and attention kernels are computed to capture token similarity and positional relevance in lines 4\-5\. Lines 7–11 then compute the gradient of the diversity function, which is projected onto the vocabulary to form the diversity guidance signal in line 12\. In line 13, the signal is added to the base logits to obtain entropy\-modulated logits, which are then used to sample the next sequence state according to the diffusion schedule in line 14\. AfterTTsteps, the procedure terminates and outputs the final sequencexTx\_\{T\}\.

## 5Numerical Results

### 5\.1Gaussian Synthesis

![Refer to caption](https://arxiv.org/html/2608.00024v1/gaussian_arith_distinct_2x2.png)Figure 2:Pareto frontier comparisons across three generative tasks, highlighting the diversity–quality trade\-off for our diversity guidance and baseline methods\.Top\-left:Gaussian mixture synthesis, plotting coverage vs\. density\.Top\-right:arithmetic series generation, plotting uniqueness \(unique%\) vs\. validity\.Bottom:Two tasks: story continuation and brainstorm generation, plotting distinct\-2 vs\. perplexity \(log scale, lower is better\)\. In each panel, frontier markers summarize the best\-achievable quality at a given diversity level\. Our SAKE method \(blue circles\) shifts the frontier outward, improving diversity at comparable quality\.To compare temperature scaling with our diversity guidance, we use a 2D Gaussian mixture model for precise evaluation \(Figure[4](https://arxiv.org/html/2608.00024#A4.F4)\)\. The baseline \(top\) shows that increasing temperatureTTimproves mode coverage but significantly sacrifices fidelity, resulting in diffuse samples\. In contrast, our method \(bottom\) with varying strengthγ\\gammaachieves broad mode coverage while maintaining tight alignment with the target components\. This confirms that our approach effectively prevents mode collapse without the quality degradation inherent to high\-temperature sampling\. The detailed analysis is provided in Appendix[B](https://arxiv.org/html/2608.00024#A2)\.

### 5\.2Diversity\-Quality Pareto Frontier Experiments

We analyze the Diversity\-Quality Pareto frontier to evaluate the trade\-off between generation quality and variety\. Figure[2](https://arxiv.org/html/2608.00024#S5.F2)demonstrates that*diversity\-guided*sampling consistently yields superior frontiers compared to standard*temperature sampling*\. While temperature scaling forces a strict trade\-off—sacrificing quality for diversity—our method shifts the frontier outward, maintaining higher quality at comparable diversity levels across all tasks\.

Gaussian Mixture Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), top\-left\): Using Coverage \(diversity\) and Density \(quality\) metrics\(Naeemet al\.,[2020](https://arxiv.org/html/2608.00024#bib.bib25)\), we find that increasing temperatureTTexpands coverage but causes a precipitous drop in density\. Conversely, sweeping guidance strengthγ\\gammashifts the frontier outward, achieving higher coverage for equivalent density levels compared to the baseline\.

Arithmetic Series Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), top\-right\): In this controlled setting, we measure*Validity*\(correctness\) against*Uniqueness*\. High\-temperature sampling improves uniqueness only at the cost of a sharp drop in validity\. Diversity guidance significantly extends the Pareto frontier, enabling high uniqueness with minimal loss in validity\.

Story Continuation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), bottom\-left\): We analyze the trade\-off between Perplexity \(quality\) and Distinct\-2 \(diversity\)\(Liet al\.,[2016](https://arxiv.org/html/2608.00024#bib.bib26)\)using LLaDA\(Nieet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib21)\)\. Results indicate that aggressive temperature sampling incurs steep perplexity penalties for marginal diversity gains\. Diversity guidance achieves a more favorable balance, outperforming baselines \(including CFG\) particularly in high\-diversity regimes\.

Brainstorm Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), bottom\-right\): This task, requiring multiple creative proposals per prompt, presents a challenge for baselines\. Temperature sampling suffers rapid quality degradation at high diversity, while D\-CFG remains overly conservative\. In contrast, diversity guidance establishes a superior frontier, maintaining low perplexity even as Distinct\-2 increases\. Qualitative examples \(Table[1](https://arxiv.org/html/2608.00024#S5.T1)\) further confirm that while baselines collapse into repetitive syntactic templates, SAKE successfully diversifies semantic content without sacrificing coherence\.

Collectively, these results suggest that SAKE is a more effective strategy for traversing the quality\-diversity manifold than relying solely on stochastic temperature scaling\. The detailed analysis is provided in Appendix[C](https://arxiv.org/html/2608.00024#A3)\.

Table 1:Qualitative comparison between SAKE and the baseline under identical temperatureTTand prompt\.
### 5\.3Diversity\-Guided Chain\-of\-Thought in Complex Reasoning Tasks

![Refer to caption](https://arxiv.org/html/2608.00024v1/humaneval_passk.png)Figure 3:Pass@k rates on HumanEval at low \(T=0\.2T=0\.2\) and high \(T=0\.7T=0\.7\) temperatures\. AtT=0\.2T=0\.2, standard Temperature Sampling and D\-CFG exhibit signs of mode collapse, while our SAKE method shows a steady improvement fromk=1k=1tok=32k=32\. Notably, our method atT=0\.2T=0\.2achieves a Pass@32 score comparable to the baseline atT=0\.7T=0\.7, proving it can extract diverse, high\-quality solutions without relying on high temperature\.Table 2:Performance comparison on code generation and mathematical reasoning benchmarks\.Benchmark\(Metric\)\#ShotsBase ModelD\-CFGSAKEHumanEval\(Pass@1\)032\.933\.732\.0HumanEval\(Pass@32\)041\.141\.255\.8MBPP\(Pass@1\)340\.237\.339\.0MBPP\(Pass@32\)348\.245\.956\.1GSM8K SC\(SM\)571\.573\.275\.1GSM8K SC\(FE\)568\.869\.773\.5Table 3:Generation speed \(Tokens/sec\) with standard deviation \(±σ\\pm\\sigma\)\.Table[2](https://arxiv.org/html/2608.00024#S5.T2)summarizes performance on HumanEval, MBPP, and GSM8K using LLaDA\-8B\. We compare the unguided base model, D\-CFG, and our SAKE using standard stochastic sampling\. We report Pass@32 \(n=32n\{=\}32\) for code tasks and self\-consistency \(n=5n\{=\}5\) for GSM8K\. To ensure a fair comparison, we perform a hyperparameter search for both the baseline methods and ours via grid search\.

Our method boosts performance in settings requiring diverse exploration\. On HumanEval, while Pass@1 remains competitive, Pass@32 improves substantially to 55\.8, outperforming both the base model \(41\.1\) and D\-CFG \(41\.2\)\. Similarly, on MBPP, Pass@32 rises to 56\.1 \(vs\. base 48\.2\)\. This indicates our method generates higher*effective*diversity, producing a candidate set with a higher probability of correctness rather than merely increasing surface variability\.

Figure[3](https://arxiv.org/html/2608.00024#S5.F3)highlights the mitigation of mode collapse\. At low temperature \(T=0\.2T\{=\}0\.2\), the baseline saturates quickly \(0\.33→0\.410\.33\\rightarrow 0\.41\), whereas our method forces exploration, improving Pass@32 to 0\.56\. Notably, our method atT=0\.2T\{=\}0\.2matches the baseline’s performance atT=0\.7T\{=\}0\.7, demonstrating the ability to extract latent knowledge without the quality degradation risks associated with high\-temperature sampling\. For GSM8K \(5\-shot\), our method achieves 75\.1 \(strict\) and 73\.5 \(flexible\), surpassing both the base model \(71\.5/68\.8\) and D\-CFG\. These results confirm that diversity guidance benefits downstream selection mechanisms \(Pass@kk, majority voting\) by reallocating probability mass toward distinct, plausible reasoning paths\.

### 5\.4Computation Efficiency

As shown in Eq\.[18](https://arxiv.org/html/2608.00024#S4.E18), the SAKE diversity signal possesses linear sequence length complexity and can be efficiently calculated\. We empirically validate this theoretical efficiency in Table[3](https://arxiv.org/html/2608.00024#S5.T3), which reports the generation speed \(tokens/sec, batch size equals 1\) across varying prompt lengths for the LLaDA model in a single GPU\.

The results demonstrate that SAKE introduces negligible inference latency\. On average, our method maintains a generation speed of44\.0944\.09tokens/sec, retaining approximately93%93\\%of the unguided baseline’s throughput \(47\.4147\.41tokens/sec\)\. In contrast, D\-CFG suffers from a substantial computational penalty, averaging only31\.8331\.83tokens/sec—a reduction of nearly33%33\\%compared to the Baseline\.

This efficiency gap becomes more significant at longer context windows\. At a prompt length of 1024, SAKE remains highly efficient \(21\.5121\.51tokens/sec\) compared to the Baseline \(23\.1923\.19tokens/sec\)\. Conversely, D\-CFG experiences a sharp decline to11\.9811\.98tokens/sec, effectively halving the generation speed of the base model\. These findings confirm that SAKE provides a lightweight guidance mechanism that scales effectively to longer sequences\.

## 6Limitations and Discussion

Similar to many other training\-free guidance methods, our method introduces additional inference\-time computation through the evaluation of the semantic kernel and attention kernel\. As with guidance\-based decoding more broadly, this creates a trade\-off between controllability and latency\. Although the added cost is substantially smaller than that of guidance baselines, and our embedding\-space linearization keeps the complexity linear in sequence length, the method nevertheless incurs non\-negligible overhead relative to unguided decoding\. Empirically, the throughput reduction is modest \(approximately 7% as shown in Table[3](https://arxiv.org/html/2608.00024#S5.T3)\), suggesting that the approach remains practical in many deployment settings\. However, this overhead may still limit applicability in real\-time scenarios with extremely strict latency constraints, where even small inference slowdowns can be undesirable\.

A second limitation concerns the theoretical characterization of guidance\-induced distribution shift\. While Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1)analyzes the effect of guidance on the output distribution through changes in Shannon entropy, this does not yet provide a complete account of how the guidance strength hyperparameterγ\\gammashapes broader properties of the decoded distribution\. In particular, althoughγ\\gammacan be selected effectively in practice through validation or modest hyperparameter search, a more precise mathematical relationship between guidance strength and target distribution shift remains underdeveloped\. A further theoretical treatment of this relationship would improve interpretability and could help reduce empirical tuning\.

## 7Conclusions

We presented a general framework for inference\-time diversity guidance in discrete diffusion language models, named*Semantic\-Aware Kernel Entropy*\(SAKE\)\. By maximizing Rényi entropy within the semantic space, SAKE overcomes the limitations of token\-independent assumptions and static temperature scaling\. Our method serves as an adaptive modulator, dynamically balancing exploration and coherence to improve algorithmic robustness\. Empirical results on code and math reasoning benchmarks confirm that SAKE boosts performance by improving diversity without retraining\.

## Acknowledgment

This work is partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14210725, and is partially supported by CUHK Direct Research Grant with CUHK Project No\. 4055164\. The work is also supported by a grant under 1\+1\+1 CUHK\-CUHK\(SZ\)\-GDSTC Joint Collaboration Fund\. Also, the authors would like to thank the anonymous reviewers and metareviewer for their constructive suggestions and insightful feedback\.

## Impact Statement

This research contributes to the advancement of Machine Learning methodologies\. While we acknowledge the broader societal implications inherent to AI technologies, we do not foresee any immediate negative consequences or specific ethical concerns arising directly from this work that necessitate distinct discussion\.

## References

- M\. Arriola, A\. Gokaslan, J\. T\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. S\. Sahoo, and V\. Kuleshov \(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.External Links:2503\.09573,[Link](https://arxiv.org/abs/2503.09573)Cited by:[Appendix D](https://arxiv.org/html/2608.00024#A4.SS0.SSS0.Px3.p2.7)\.
- R\. Askari Hemmat, M\. Hall, A\. Sun, C\. Ross, M\. Drozdzal, and A\. Romero\-Soriano \(2024\)Improving geo\-diversity of generated images with contextualized vendi score guidance\.InEuropean Conference on Computer Vision,pp\. 213–229\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den Berg \(2021\)Structured denoising diffusion models in discrete state\-spaces\.Advances in neural information processing systems34,pp\. 17981–17993\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1),[§3](https://arxiv.org/html/2608.00024#S3.p4.4)\.
- A\. Bansal, H\. Chu, A\. Schwarzschild, S\. Sengupta, M\. Goldblum, J\. Geiping, and T\. Goldstein \(2023\)Universal guidance for diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 843–852\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p2.3),[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- A\. Blattmann, T\. Dockhorn, S\. Kulal, D\. Mendelevitch, M\. Kilian, D\. Lorenz, Y\. Levi, Z\. English, V\. Voleti, A\. Letts,et al\.\(2023a\)Stable video diffusion: scaling latent video diffusion models to large datasets\.arXiv preprint arXiv:2311\.15127\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- A\. Blattmann, R\. Rombach, H\. Ling, T\. Dockhorn, S\. W\. Kim, S\. Fidler, and K\. Kreis \(2023b\)Align your latents: high\-resolution video synthesis with latent diffusion models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 22563–22575\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p1.1)\.
- G\. Corso, Y\. Xu, V\. D\. Bortoli, R\. Barzilay, and T\. S\. Jaakkola \(2024\)Particle Guidance: non\-I\.I\.D\. diverse sampling with diffusion models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- P\. Dhariwal and A\. Nichol \(2021\)Diffusion models beat gans on image synthesis\.Advances in neural information processing systems34,pp\. 8780–8794\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p2.3),[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- S\. Gong, Y\. Jiang, Q\. Dou, and F\. Farnia \(2025\)Kernel\-based unsupervised embedding alignment for enhanced visual representation in vision\-language models\.InInternational Conference on Machine Learning,pp\. 19912–19931\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- Y\. He, T\. Yang, Y\. Zhang, Y\. Shan, and Q\. Chen \(2022\)Latent video diffusion models for high\-fidelity long video generation\.arXiv preprint arXiv:2211\.13221\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.00024#S3.p1.10)\.
- J\. Ho, C\. Saharia, W\. Chan, D\. J\. Fleet, M\. Norouzi, and T\. Salimans \(2022\)Cascaded diffusion models for high fidelity image generation\.Journal of Machine Learning Research23\(47\),pp\. 1–33\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- J\. Ho and T\. Salimans \(2022\)Classifier\-free diffusion guidance\.arXiv preprint arXiv:2207\.12598\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p2.3),[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi \(2019\)The curious case of neural text degeneration\.arXiv preprint arXiv:1904\.09751\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- X\. Hu, H\. Leung, and F\. Farnia \(2025a\)A multi\-armed bandit approach to online selection and evaluation of generative models\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 1864–1872\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- X\. Hu, H\. Leung, and F\. Farnia \(2025b\)PAK\-ucb contextual bandit: an online learning approach to prompt\-aware selection of generative models and llms\.InProceedings of the 26th International Conference on Machine Learning \(ICML\),Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- D\. Jafari and F\. Farnia \(2026\)DAK\-UCB: diversity\-aware prompt routing for LLMs and generative models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nnN2TKlS5C)Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- M\. Jalali, H\. Lei, A\. Gohari, and F\. Farnia \(2025a\)SPARKE: scalable prompt\-aware diversity and novelty guidance in diffusion models via rke score\.Advances in Neural Information Processing Systems\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- M\. Jalali, C\. T\. Li, and F\. Farnia \(2023\)An information\-theoretic evaluation of generative models in learning multi\-modal distributions\.Advances in Neural Information Processing Systems36,pp\. 9931–9943\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- M\. Jalali, B\. D\. Nia, and F\. Farnia \(2025b\)Towards an explainable comparison and alignment of feature embeddings\.InInternational Conference on Machine Learning,pp\. 26757–26796\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- M\. Jalali, A\. Ospanov, A\. Gohari, and F\. Farnia \(2026\)Conditional vendi score: prompt\-aware diversity evaluation for generative AI models and LLMs\.InThe 29th International Conference on Artificial Intelligence and Statistics,External Links:[Link](https://openreview.net/forum?id=iDrZToIsyd)Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- M\. Kirchhof, J\. Thornton, L\. Béthune, P\. Ablin, E\. Ndiaye, and M\. Cuturi \(2025\)Shielded diffusion: generating novel and diverse images using sparse repellency\.External Links:2410\.06025,[Link](https://arxiv.org/abs/2410.06025)Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- J\. Li, M\. Galley, C\. Brockett, J\. Gao, and W\. B\. Dolan \(2016\)A diversity\-promoting objective function for neural conversation models\.InProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 110–119\.Cited by:[Appendix C](https://arxiv.org/html/2608.00024#A3.p5.1),[§5\.2](https://arxiv.org/html/2608.00024#S5.SS2.p4.1)\.
- X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto \(2022\)Diffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le \(2022\)Flow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§3](https://arxiv.org/html/2608.00024#S3.p2.3)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p1.1)\.
- M\. Lotfian, M\. Jalali, and F\. Farnia \(2026\)PromptSplit: revealing prompt\-level disagreement in generative models\.InICML 2026 Workshop on Combining Theory and Benchmarks: Towards A Virtuous Cycle to Understand and Guarantee Foundation Model Performance,External Links:[Link](https://openreview.net/forum?id=sUThOs5ADa)Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. Lou, C\. Meng, and S\. Ermon \(2023\)Discrete diffusion modeling by estimating the ratios of the data distribution\.arXiv preprint arXiv:2310\.16834\.Cited by:[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- C\. Meng, K\. Choi, J\. Song, and S\. Ermon \(2022\)Concrete score matching: generalized score matching for discrete data\.Advances in Neural Information Processing Systems35,pp\. 34532–34545\.Cited by:[§3](https://arxiv.org/html/2608.00024#S3.p4.6)\.
- M\. F\. Naeem, S\. J\. Oh, Y\. Uh, Y\. Choi, and J\. Yoo \(2020\)Reliable fidelity and diversity metrics for generative models\.InInternational conference on machine learning,pp\. 7176–7185\.Cited by:[Appendix C](https://arxiv.org/html/2608.00024#A3.p3.4),[§5\.2](https://arxiv.org/html/2608.00024#S5.SS2.p2.2)\.
- M\. N\. Nguyen, A\. Baker, C\. Neo, A\. Roush, A\. Kirsch, and R\. Shwartz\-Ziv \(2024\)Turning up the heat: min\-p sampling for creative and coherent llm outputs\.arXiv preprint arXiv:2407\.01082\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- B\. D\. Nia and F\. Farnia \(2026\)Mixture\-greedy for online generative model selection: is UCB necessary in diversity\-aware multi\-armed bandits?\.InICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning,External Links:[Link](https://openreview.net/forum?id=8h8nkxR1ac)Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. Nichol, P\. Dhariwal, A\. Ramesh, P\. Shyam, P\. Mishkin, B\. McGrew, I\. Sutskever, and M\. Chen \(2021\)GLIDE: towards photorealistic image generation and editing with text\-guided diffusion models\.arXiv preprint arXiv:2112\.10741\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[Appendix C](https://arxiv.org/html/2608.00024#A3.p5.1),[§1](https://arxiv.org/html/2608.00024#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1),[§5\.2](https://arxiv.org/html/2608.00024#S5.SS2.p4.1)\.
- A\. Ospanov and F\. Farnia \(2025\)Do vendi scores converge with finite samples? truncated vendi score for finite\-sample convergence guarantees\.InThe 41st Conference on Uncertainty in Artificial Intelligence,Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. Ospanov, M\. Jalali, and F\. Farnia \(2025\)Scendi score: prompt\-aware diversity evaluation via schur complement of clip embeddings\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 16927–16937\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. Ospanov, J\. Zhang, M\. Jalali, X\. Cao, A\. Bogdanov, and F\. Farnia \(2024\)Towards a scalable reference\-free evaluation of generative models\.Advances in Neural Information Processing Systems37,pp\. 120892–120927\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li \(2024\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.arXiv preprint arXiv:2406\.03736\.Cited by:[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- P\. Rezaei, F\. Farnia, and C\. T\. Li \(2025\)Be more diverse than the most diverse: optimal mixtures of generative models via mixture\-UCB bandit algorithms\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2Chkk5Ye2s)Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer \(2022\)High\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 10684–10695\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- S\. Sadat, J\. Buhmann, D\. Bradley, O\. Hilliges, and R\. M\. Weber \(2024\)CADS: unleashing the diversity of diffusion models through condition\-annealed sampling\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. Chiu, A\. Rush, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[Appendix D](https://arxiv.org/html/2608.00024#A4.SS0.SSS0.Px3.p2.7),[§1](https://arxiv.org/html/2608.00024#S1.p2.3),[§1](https://arxiv.org/html/2608.00024#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- M\. S\. Sajjadi, O\. Bachem, M\. Lucic, O\. Bousquet, and S\. Gelly \(2018\)Assessing generative models via precision and recall\.Advances in neural information processing systems31\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- M\. M\. Sani, N\. Jamali, M\. Jalali, and F\. Farnia \(2026\)MMD guidance: training\-free distribution adaptation for diffusion models via maximum mean discrepancy guidance\.InForty\-third International Conference on Machine Learning,Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- Y\. Schiff, S\. S\. Sahoo, H\. Phung, G\. Wang, S\. Boshar, H\. Dalla\-torre, B\. P\. de Almeida, A\. Rush, T\. Pierrot, and V\. Kuleshov \(2024\)Simple guidance mechanisms for discrete diffusion models\.arXiv preprint arXiv:2412\.10193\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1)\.
- J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. Ganguli \(2015\)Deep unsupervised learning using nonequilibrium thermodynamics\.InInternational conference on machine learning,pp\. 2256–2265\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.00024#S3.p1.10)\.
- Y\. Song and S\. Ermon \(2019\)Generative modeling by estimating gradients of the data distribution\.Advances in neural information processing systems32\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.00024#S3.p1.10),[§3](https://arxiv.org/html/2608.00024#S3.p3.2)\.
- Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole \(2020\)Score\-based generative modeling through stochastic differential equations\.arXiv preprint arXiv:2011\.13456\.Cited by:[§2\.2](https://arxiv.org/html/2608.00024#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.00024#S3.p1.10),[§3](https://arxiv.org/html/2608.00024#S3.p2.3),[§3](https://arxiv.org/html/2608.00024#S3.p3.2)\.
- G\. Stein, J\. Cresswell, R\. Hosseinzadeh, Y\. Sui, B\. Ross, V\. Villecroze, Z\. Liu, A\. L\. Caterini, E\. Taylor, and G\. Loaiza\-Ganem \(2023\)Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models\.Advances in Neural Information Processing Systems36,pp\. 3732–3784\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. K\. Vijayakumar, M\. Cogswell, R\. R\. Selvaraju, Q\. Sun, S\. Lee, D\. Crandall, and D\. Batra \(2016\)Diverse beam search: decoding diverse solutions from neural sequence models\.arXiv preprint arXiv:1610\.02424\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p4.1)\.
- Y\. Wu and F\. Farnia \(2026\)The maximum von neumann entropy principle: theory and applications in machine learning\.InIEEE International Symposium on Information Theory \(ISIT\),Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- Y\. Wu, M\. Jalali, and F\. Farnia \(2026\)KODA: contrastive representation comparison and alignment for vision\-language foundation models\.InInternational Conference on Machine Learning,Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- Y\. Wu, J\. Zhang, and F\. Farnia \(2025\)When kernels multiply, clusters unify: fusing embeddings with the kronecker product\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix C](https://arxiv.org/html/2608.00024#A3.p5.1),[§1](https://arxiv.org/html/2608.00024#S1.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.Advances in neural information processing systems36,pp\. 11809–11822\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p4.1)\.
- J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong \(2025\)Dream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§1](https://arxiv.org/html/2608.00024#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- H\. Yuan, Z\. Yuan, C\. Tan, F\. Huang, and S\. Huang \(2022\)Seqdiffuseq: text diffusion with encoder\-decoder transformers\.arXiv preprint arXiv:2212\.10325\.Cited by:[§2\.1](https://arxiv.org/html/2608.00024#S2.SS1.p1.1)\.
- J\. Zhang, M\. Jalali, C\. T\. Li, and F\. Farnia \(2025\)Unveiling differences in generative models: a scalable differential clustering approach\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- J\. Zhang, C\. T\. Li, and F\. Farnia \(2024\)An interpretable evaluation of entropy\-based novelty of generative models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 59148–59172\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.
- Y\. Zhu, H\. Zhang, B\. Wu, J\. Li, Z\. Zheng, P\. Zhao, L\. Chen, and Y\. Bian \(2025\)Measuring diversity in synthetic datasets\.arXiv preprint arXiv:2502\.08512\.Cited by:[§2\.3](https://arxiv.org/html/2608.00024#S2.SS3.p1.1)\.

## Appendix AProofs

### Proof of Proposition[4\.1](https://arxiv.org/html/2608.00024#S4.Thmtheorem1)

###### Proposition A\.1\(Entropy Dynamics of Text Diffusion Guidance\)\.

The first\-order change in the Shannon entropy of the guided distributionPγP\_\{\\gamma\}at the limit of zero guidance is governed by the negative covariance between the base log\-probabilities and the guidance signal:

∇γH​\[Pγ\]\|γ=0=−CovY∼Pbase​\[log⁡Pbase​\(Y\),ψY\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{Y\\sim P\_\{\\mathrm\{base\}\}\}\[\\log P\_\{\\mathrm\{base\}\}\(Y\),\\psi\_\{Y\}\]\.\(19\)Furthermore, sincelog⁡Pbase​\(y\)=zy−log⁡Z​\(0\)\\log P\_\{\\mathrm\{base\}\}\(y\)=z\_\{y\}\-\\log Z\(0\), this simplifies to:

∇γH​\[Pγ\]\|γ=0=−CovPbase​\[𝐳,ψ\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[\\mathbf\{z\},\\psi\]\.\(20\)

###### Proof\.

LetPbase​\(y\)P\_\{\\mathrm\{base\}\}\(y\)be the probability of a candidate tokeny∈𝒱y\\in\\mathcal\{V\}from the vocabulary𝒱\\mathcal\{V\}under the unguided base model, defined via logitszyz\_\{y\}:

Pbase​\(y\)=ezyZ​\(0\),P\_\{\\mathrm\{base\}\}\(y\)=\\frac\{e^\{z\_\{y\}\}\}\{Z\(0\)\},whereZ​\(0\)=∑y′∈𝒱ezy′Z\(0\)=\\sum\_\{y^\{\\prime\}\\in\\mathcal\{V\}\}e^\{z\_\{y^\{\\prime\}\}\}\. Note thatlog⁡Pbase​\(y\)=zy−log⁡Z​\(0\)\\log P\_\{\\mathrm\{base\}\}\(y\)=z\_\{y\}\-\\log Z\(0\)\.

The guided distributionPγP\_\{\\gamma\}is defined by tilting the base distribution with a guidance signalψy\\psi\_\{y\}scaled by a parameterγ\\gamma:

Pγ​\(y\)=Pbase​\(y\)​eγ​ψyZ​\(γ\),P\_\{\\gamma\}\(y\)=\\frac\{P\_\{\\mathrm\{base\}\}\(y\)e^\{\\gamma\\psi\_\{y\}\}\}\{Z\(\\gamma\)\},where the partition function isZ​\(γ\)=∑y′∈𝒱Pbase​\(y′\)​eγ​ψy′=𝔼Y∼Pbase​\[eγ​ψY\]Z\(\\gamma\)=\\sum\_\{y^\{\\prime\}\\in\\mathcal\{V\}\}P\_\{\\mathrm\{base\}\}\(y^\{\\prime\}\)e^\{\\gamma\\psi\_\{y^\{\\prime\}\}\}=\\mathbb\{E\}\_\{Y\\sim P\_\{\\mathrm\{base\}\}\}\[e^\{\\gamma\\psi\_\{Y\}\}\]\.

The Shannon entropy of the guided distribution is:

H​\[Pγ\]=−∑yPγ​\(y\)​log⁡Pγ​\(y\)\.H\[P\_\{\\gamma\}\]=\-\\sum\_\{y\}P\_\{\\gamma\}\(y\)\\log P\_\{\\gamma\}\(y\)\.
We compute the gradient of the entropy with respect toγ\\gamma:

∇γH​\[Pγ\]=∇γ\(−∑yPγ​\(y\)​log⁡Pγ​\(y\)\)\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=\\nabla\_\{\\gamma\}\\left\(\-\\sum\_\{y\}P\_\{\\gamma\}\(y\)\\log P\_\{\\gamma\}\(y\)\\right\)\.Using the product rule:

∇γH​\[Pγ\]=−∑y\(\(∇γPγ​\(y\)\)​log⁡Pγ​\(y\)\+Pγ​\(y\)​∇γPγ​\(y\)Pγ​\(y\)\)\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=\-\\sum\_\{y\}\\left\(\(\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)\)\\log P\_\{\\gamma\}\(y\)\+P\_\{\\gamma\}\(y\)\\frac\{\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)\}\{P\_\{\\gamma\}\(y\)\}\\right\)\.This simplifies to:

∇γH​\[Pγ\]=−∑y\(∇γPγ​\(y\)\)​log⁡Pγ​\(y\)−∑y∇γPγ​\(y\)\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=\-\\sum\_\{y\}\(\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)\)\\log P\_\{\\gamma\}\(y\)\-\\sum\_\{y\}\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)\.Since∑yPγ​\(y\)=1\\sum\_\{y\}P\_\{\\gamma\}\(y\)=1is constant, its derivative is zero \(∑y∇γPγ​\(y\)=0\\sum\_\{y\}\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)=0\)\. Thus:

∇γH​\[Pγ\]=−∑y\(∇γPγ​\(y\)\)​log⁡Pγ​\(y\)\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=\-\\sum\_\{y\}\(\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)\)\\log P\_\{\\gamma\}\(y\)\.\(21\)
The log\-probability is given by:

log⁡Pγ​\(y\)=log⁡Pbase​\(y\)\+γ​ψy−log⁡Z​\(γ\)\.\\log P\_\{\\gamma\}\(y\)=\\log P\_\{\\mathrm\{base\}\}\(y\)\+\\gamma\\psi\_\{y\}\-\\log Z\(\\gamma\)\.Differentiating with respect toγ\\gamma:

∇γlog⁡Pγ​\(y\)=ψy−∇γlog⁡Z​\(γ\)\.\\nabla\_\{\\gamma\}\\log P\_\{\\gamma\}\(y\)=\\psi\_\{y\}\-\\nabla\_\{\\gamma\}\\log Z\(\\gamma\)\.We compute∇γlog⁡Z​\(γ\)\\nabla\_\{\\gamma\}\\log Z\(\\gamma\):

∇γlog⁡Z​\(γ\)=1Z​\(γ\)​∇γ​∑y′∈𝒱Pbase​\(y′\)​eγ​ψy′=∑y′∈𝒱Pbase​\(y′\)​eγ​ψy′Z​\(γ\)​ψy′=𝔼Pγ​\[ψ\]\.\\nabla\_\{\\gamma\}\\log Z\(\\gamma\)=\\frac\{1\}\{Z\(\\gamma\)\}\\nabla\_\{\\gamma\}\\sum\_\{y^\{\\prime\}\\in\\mathcal\{V\}\}P\_\{\\mathrm\{base\}\}\(y^\{\\prime\}\)e^\{\\gamma\\psi\_\{y^\{\\prime\}\}\}=\\sum\_\{y^\{\\prime\}\\in\\mathcal\{V\}\}\\frac\{P\_\{\\mathrm\{base\}\}\(y^\{\\prime\}\)e^\{\\gamma\\psi\_\{y^\{\\prime\}\}\}\}\{Z\(\\gamma\)\}\\psi\_\{y^\{\\prime\}\}=\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\.Substituting this back, we get:

∇γlog⁡Pγ​\(y\)=ψy−𝔼Pγ​\[ψ\]\.\\nabla\_\{\\gamma\}\\log P\_\{\\gamma\}\(y\)=\\psi\_\{y\}\-\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\.
Using the identity∇γP=P​\(∇γlog⁡P\)\\nabla\_\{\\gamma\}P=P\(\\nabla\_\{\\gamma\}\\log P\):

∇γPγ​\(y\)=Pγ​\(y\)​\(ψy−𝔼Pγ​\[ψ\]\)\.\\nabla\_\{\\gamma\}P\_\{\\gamma\}\(y\)=P\_\{\\gamma\}\(y\)\(\\psi\_\{y\}\-\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\)\.
Substituting the result into Equation \([21](https://arxiv.org/html/2608.00024#A1.E21)\):

∇γH​\[Pγ\]\\displaystyle\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=−∑y\[Pγ​\(y\)​\(ψy−𝔼Pγ​\[ψ\]\)\]​log⁡Pγ​\(y\)\\displaystyle=\-\\sum\_\{y\}\\left\[P\_\{\\gamma\}\(y\)\(\\psi\_\{y\}\-\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\)\\right\]\\log P\_\{\\gamma\}\(y\)=−∑yPγ​\(y\)​ψy​log⁡Pγ​\(y\)\+𝔼Pγ​\[ψ\]​∑yPγ​\(y\)​log⁡Pγ​\(y\)\\displaystyle=\-\\sum\_\{y\}P\_\{\\gamma\}\(y\)\\psi\_\{y\}\\log P\_\{\\gamma\}\(y\)\+\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\\sum\_\{y\}P\_\{\\gamma\}\(y\)\\log P\_\{\\gamma\}\(y\)=−𝔼Pγ​\[ψY​log⁡Pγ​\(Y\)\]\+𝔼Pγ​\[ψ\]​𝔼Pγ​\[log⁡Pγ​\(Y\)\]\.\\displaystyle=\-\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\_\{Y\}\\log P\_\{\\gamma\}\(Y\)\]\+\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\psi\]\\mathbb\{E\}\_\{P\_\{\\gamma\}\}\[\\log P\_\{\\gamma\}\(Y\)\]\.Using the definition of covariance,Cov​\[A,B\]=𝔼​\[A​B\]−𝔼​\[A\]​𝔼​\[B\]\\mathrm\{Cov\}\[A,B\]=\\mathbb\{E\}\[AB\]\-\\mathbb\{E\}\[A\]\\mathbb\{E\}\[B\], we obtain:

∇γH​\[Pγ\]=−CovY∼Pγ​\[log⁡Pγ​\(Y\),ψY\]\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]=\-\\mathrm\{Cov\}\_\{Y\\sim P\_\{\\gamma\}\}\[\\log P\_\{\\gamma\}\(Y\),\\psi\_\{Y\}\]\.
Taking the limit asγ→0\\gamma\\to 0, we havePγ→PbaseP\_\{\\gamma\}\\to P\_\{\\mathrm\{base\}\}\. Thus:

∇γH​\[Pγ\]\|γ=0=−CovY∼Pbase​\[log⁡Pbase​\(Y\),ψY\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{Y\\sim P\_\{\\mathrm\{base\}\}\}\[\\log P\_\{\\mathrm\{base\}\}\(Y\),\\psi\_\{Y\}\]\.This proves Equation \(7\)\.

Recall thatlog⁡Pbase​\(y\)=zy−log⁡Z​\(0\)\\log P\_\{\\mathrm\{base\}\}\(y\)=z\_\{y\}\-\\log Z\(0\)\. Substituting this into the covariance:

CovPbase​\[log⁡Pbase​\(Y\),ψY\]=CovPbase​\[zY−log⁡Z​\(0\),ψY\]\.\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[\\log P\_\{\\mathrm\{base\}\}\(Y\),\\psi\_\{Y\}\]=\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[z\_\{Y\}\-\\log Z\(0\),\\psi\_\{Y\}\]\.Since covariance is invariant under constant shifts \(i\.e\.,Cov​\[X\+c,Y\]=Cov​\[X,Y\]\\mathrm\{Cov\}\[X\+c,Y\]=\\mathrm\{Cov\}\[X,Y\]\) andlog⁡Z​\(0\)\\log Z\(0\)is constant with respect toYY:

CovPbase​\[zY−log⁡Z​\(0\),ψY\]=CovPbase​\[zY,ψY\]\.\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[z\_\{Y\}\-\\log Z\(0\),\\psi\_\{Y\}\]=\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[z\_\{Y\},\\psi\_\{Y\}\]\.Therefore:

∇γH​\[Pγ\]\|γ=0=−CovPbase​\[𝐳,ψ\]\.\\left\.\\nabla\_\{\\gamma\}H\[P\_\{\\gamma\}\]\\right\|\_\{\\gamma=0\}=\-\\mathrm\{Cov\}\_\{P\_\{\\mathrm\{base\}\}\}\[\\mathbf\{z\},\\psi\]\.This proves Equation \(8\)\. ∎

### Proof of Proposition[4\.3](https://arxiv.org/html/2608.00024#S4.Thmtheorem3)

###### Proposition A\.2\(SAKE Guidance Signal\)\.

Let𝐇=\[𝐡1,…,𝐡L\]∈ℝd×L\\mathbf\{H\}=\[\\mathbf\{h\}\_\{1\},\\dots,\\mathbf\{h\}\_\{L\}\]\\in\\mathbb\{R\}^\{d\\times L\}be the matrix of token embeddings for a sequence of lengthLL\. Consider the Gaussian RBF kernelκ​\(𝐡i,𝐡j\)=exp⁡\(−‖𝐡i−𝐡j‖2/2​σ2\)\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)=\\exp\(\-\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}/2\\sigma^\{2\}\)with bandwidthσ\>0\\sigma\>0, and let𝐊∈ℝL×L\\mathbf\{K\}\\in\\mathbb\{R\}^\{L\\times L\}be the kernel Gram matrix with𝐊i​j=κ​\(𝐡i,𝐡j\)\\mathbf\{K\}\_\{ij\}=\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)\. Similarly,𝐊attn,\(i,j\)=exp⁡\(−‖i−j‖2/2​σattn2\)\\mathbf\{K\}\_\{\\text\{attn\},\(i,j\)\}=\\exp\(\-\\\|i\-j\\\|^\{2\}/2\\sigma\_\{\\text\{attn\}\}^\{2\}\)\.

The normalized Hadamard product of kernel Gram matrices serves as a valid quantum density matrix\. The order\-2 Rényi entropy of this representation is:

H2​\(1L​𝐊⊙𝐊attn\)=−log⁡\(1L2​‖𝐊⊙𝐊attn‖F2\)\.H\_\{2\}\\left\(\\frac\{1\}\{L\}\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\\right\)=\-\\log\\left\(\\frac\{1\}\{L^\{2\}\}\\\|\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\\\|\_\{F\}^\{2\}\\right\)\.\(22\)To increaseH2H\_\{2\}by adjusting theii\-th token embedding𝐡i\\mathbf\{h\}\_\{i\}, we define the continuous diversity function:

𝒟~κ​\(𝐡i;𝐇−i\)≜−∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2\.\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}\(\\mathbf\{h\}\_\{i\};\\mathbf\{H\}\_\{\-i\}\)\\triangleq\-\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\.\(23\)where𝐇−i\\mathbf\{H\}\_\{\-i\}denotes the embeddings of all tokens except theii\-th\. The gradient of this function with respect to𝐡i\\mathbf\{h\}\_\{i\}is:

∇𝐡i𝒟~κ=2σ2​∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\)\.\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}=\\frac\{2\}\{\\sigma^\{2\}\}\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\.\(24\)

We verify the proposition in three parts: \(1\) the validity of the density matrix, \(2\) the derivation of the Rényi entropy, and \(3\) the gradient of the diversity function\.

Let𝐌=𝐊⊙𝐊attn\\mathbf\{M\}=\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\. We claim thatρ=1L​𝐌\\rho=\\frac\{1\}\{L\}\\mathbf\{M\}is a valid density matrix\. A matrixρ\\rhois a density matrix if it is Hermitian, Positive Semi\-Definite \(PSD\), and has unit trace\.

1. 1\.Hermitian Property:Both𝐊\\mathbf\{K\}and𝐊attn\\mathbf\{K\}\_\{\\text\{attn\}\}are real symmetric matrices \(since the Gaussian kernelκ​\(x,y\)=κ​\(y,x\)\\kappa\(x,y\)=\\kappa\(y,x\)is symmetric\)\. The Hadamard product of two symmetric matrices is symmetric\. Thus,ρ\\rhois real and symmetric, which implies it is Hermitian \(ρ=ρ†\\rho=\\rho^\{\\dagger\}\)\.
2. 2\.Positive Semi\-Definite \(PSD\):𝐊\\mathbf\{K\}is a Gram matrix generated by a valid kernel \(Gaussian RBF\), so𝐊⪰0\\mathbf\{K\}\\succeq 0\. Similarly,𝐊attn\\mathbf\{K\}\_\{\\text\{attn\}\}is a Gram matrix generated by a Gaussian kernel over indices, so𝐊attn⪰0\\mathbf\{K\}\_\{\\text\{attn\}\}\\succeq 0\. By theSchur Product Theorem, the Hadamard product of two PSD matrices is also PSD\. Therefore,𝐌⪰0\\mathbf\{M\}\\succeq 0andρ⪰0\\rho\\succeq 0\.
3. 3\.Unit Trace:The diagonal elements of a Gaussian kernel Gram matrix are all 1, becauseκ​\(𝐱,𝐱\)=exp⁡\(0\)=1\\kappa\(\\mathbf\{x\},\\mathbf\{x\}\)=\\exp\(0\)=1\. Thus,𝐌i​i=𝐊i​i⋅\(𝐊attn\)i​i=1⋅1=1\\mathbf\{M\}\_\{ii\}=\\mathbf\{K\}\_\{ii\}\\cdot\(\\mathbf\{K\}\_\{\\text\{attn\}\}\)\_\{ii\}=1\\cdot 1=1\. The trace is: Tr​\(ρ\)=Tr​\(1L​𝐌\)=1L​∑i=1L𝐌i​i=1L⋅L=1\.\\text\{Tr\}\(\\rho\)=\\text\{Tr\}\\left\(\\frac\{1\}\{L\}\\mathbf\{M\}\\right\)=\\frac\{1\}\{L\}\\sum\_\{i=1\}^\{L\}\\mathbf\{M\}\_\{ii\}=\\frac\{1\}\{L\}\\cdot L=1\.

Therefore,ρ=1L​𝐊⊙𝐊attn\\rho=\\frac\{1\}\{L\}\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}is a valid density matrix\.

The definition of Order\-2 Rényi entropy for a density matrixρ\\rhois:

H2​\(ρ\)=−log⁡\(Tr​\(ρ2\)\)\.H\_\{2\}\(\\rho\)=\-\\log\(\\text\{Tr\}\(\\rho^\{2\}\)\)\.Substitutingρ=1L​𝐌\\rho=\\frac\{1\}\{L\}\\mathbf\{M\}:

Tr​\(ρ2\)=Tr​\(\(1L​𝐌\)2\)=1L2​Tr​\(𝐌2\)\.\\text\{Tr\}\(\\rho^\{2\}\)=\\text\{Tr\}\\left\(\\left\(\\frac\{1\}\{L\}\\mathbf\{M\}\\right\)^\{2\}\\right\)=\\frac\{1\}\{L^\{2\}\}\\text\{Tr\}\(\\mathbf\{M\}^\{2\}\)\.Since𝐌\\mathbf\{M\}is real and symmetric,𝐌⊤=𝐌\\mathbf\{M\}^\{\\top\}=\\mathbf\{M\}\. The trace of the square of a symmetric matrix is the sum of the squares of its elements, which is the squared Frobenius norm:

Tr​\(𝐌2\)=Tr​\(𝐌⊤​𝐌\)=‖𝐌‖F2\.\\text\{Tr\}\(\\mathbf\{M\}^\{2\}\)=\\text\{Tr\}\(\\mathbf\{M\}^\{\\top\}\\mathbf\{M\}\)=\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\.Substituting this back into the entropy equation:

H2​\(ρ\)=−log⁡\(1L2​‖𝐌‖F2\)=−log⁡\(1L2​‖𝐊⊙𝐊attn‖F2\)\.H\_\{2\}\(\\rho\)=\-\\log\\left\(\\frac\{1\}\{L^\{2\}\}\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\\right\)=\-\\log\\left\(\\frac\{1\}\{L^\{2\}\}\\\|\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\\\|\_\{F\}^\{2\}\\right\)\.This matches Equation[22](https://arxiv.org/html/2608.00024#A1.E22)in the proposition\.

Maximizing the entropyH2​\(ρ\)H\_\{2\}\(\\rho\)is equivalent to minimizing the argument of the logarithm, specifically the squared Frobenius norm‖𝐊⊙𝐊attn‖F2\\\|\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\\\|\_\{F\}^\{2\}\. This norm represents the collision probability or ”purity” of the state\.

To derive a tractable diversity signal for guiding theii\-th token, we decompose the Frobenius norm of the Hadamard product matrix𝐌=𝐊⊙𝐊attn\\mathbf\{M\}=\\mathbf\{K\}\\odot\\mathbf\{K\}\_\{\\text\{attn\}\}\. The elements are𝐌j​k=κ​\(𝐡j,𝐡k\)⋅κattn​\(j,k\)\\mathbf\{M\}\_\{jk\}=\\kappa\(\\mathbf\{h\}\_\{j\},\\mathbf\{h\}\_\{k\}\)\\cdot\\kappa\_\{\\text\{attn\}\}\(j,k\)\.

‖𝐌‖F2\\displaystyle\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}=∑j=1L∑k=1L\(κattn​\(j,k\)⋅κ​\(𝐡j,𝐡k\)\)2\\displaystyle=\\sum\_\{j=1\}^\{L\}\\sum\_\{k=1\}^\{L\}\\left\(\\kappa\_\{\\text\{attn\}\}\(j,k\)\\cdot\\kappa\(\\mathbf\{h\}\_\{j\},\\mathbf\{h\}\_\{k\}\)\\right\)^\{2\}\(25\)=∑j≠i∑k≠i𝐌j​k2⏟independent of​𝐡i\+𝐌i​i2⏟diagonal term\+2​∑j≠i𝐌i​j2⏟cross terms\.\\displaystyle=\\underbrace\{\\sum\_\{j\\neq i\}\\sum\_\{k\\neq i\}\\mathbf\{M\}\_\{jk\}^\{2\}\}\_\{\\text\{independent of \}\\mathbf\{h\}\_\{i\}\}\+\\underbrace\{\\mathbf\{M\}\_\{ii\}^\{2\}\}\_\{\\text\{diagonal term\}\}\+\\underbrace\{2\\sum\_\{j\\neq i\}\\mathbf\{M\}\_\{ij\}^\{2\}\}\_\{\\text\{cross terms\}\}\.\(26\)
We observe that the diagonal term𝐌i​i=κ​\(𝐡i,𝐡i\)​κattn​\(i,i\)=1⋅1=1\\mathbf\{M\}\_\{ii\}=\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{i\}\)\\kappa\_\{\\text\{attn\}\}\(i,i\)=1\\cdot 1=1, so𝐌i​i2=1\\mathbf\{M\}\_\{ii\}^\{2\}=1, which is constant\. The cross terms involve the interaction between tokeniiand all other tokensjj:

2​∑j≠i𝐌i​j2=2​∑j≠i\(κattn​\(i,j\)⋅κ​\(𝐡i,𝐡j\)\)2=2​∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2\.2\\sum\_\{j\\neq i\}\\mathbf\{M\}\_\{ij\}^\{2\}=2\\sum\_\{j\\neq i\}\\left\(\\kappa\_\{\\text\{attn\}\}\(i,j\)\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)\\right\)^\{2\}=2\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\.Minimizing‖𝐌‖F2\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}with respect to𝐡i\\mathbf\{h\}\_\{i\}is therefore equivalent to minimizing the sum of these squared cross\-terms\. This motivates the definition of the continuous diversity function𝒟~κ\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}, which we seek tomaximize\(due to the negative sign\):

𝒟~κ​\(𝐡i;𝐇−i\)≜−∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2\.\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}\(\\mathbf\{h\}\_\{i\};\\mathbf\{H\}\_\{\-i\}\)\\triangleq\-\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\.
We compute the gradient of the function defined in Equation \(2\):

𝒟~κ​\(𝐡i\)=−∑j≠iCi​j⋅κ​\(𝐡i,𝐡j\)2,\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}\(\\mathbf\{h\}\_\{i\}\)=\-\\sum\_\{j\\neq i\}C\_\{ij\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\},whereCi​j=κattn​\(i,j\)2C\_\{ij\}=\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\.

First, recall the definition of the kernelκ​\(𝐡i,𝐡j\)=exp⁡\(−‖𝐡i−𝐡j‖22​σ2\)\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)=\\exp\\left\(\-\\frac\{\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}\}\{2\\sigma^\{2\}\}\\right\)\. The squared kernel is:

κ​\(𝐡i,𝐡j\)2=exp⁡\(−‖𝐡i−𝐡j‖2σ2\)\.\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}=\\exp\\left\(\-\\frac\{\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}\}\{\\sigma^\{2\}\}\\right\)\.Applying the chain rule to find the gradient with respect to𝐡i\\mathbf\{h\}\_\{i\}:

∇𝐡i\(κ​\(𝐡i,𝐡j\)2\)\\displaystyle\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\left\(\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\\right\)=∇𝐡iexp⁡\(−‖𝐡i−𝐡j‖2σ2\)\\displaystyle=\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\exp\\left\(\-\\frac\{\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}\}\{\\sigma^\{2\}\}\\right\)=exp⁡\(−‖𝐡i−𝐡j‖2σ2\)⋅∇𝐡i\(−‖𝐡i−𝐡j‖2σ2\)\\displaystyle=\\exp\\left\(\-\\frac\{\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}\}\{\\sigma^\{2\}\}\\right\)\\cdot\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\left\(\-\\frac\{\\\|\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\\\|^\{2\}\}\{\\sigma^\{2\}\}\\right\)=κ​\(𝐡i,𝐡j\)2⋅\(−1σ2⋅2​\(𝐡i−𝐡j\)\)\\displaystyle=\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\\cdot\\left\(\-\\frac\{1\}\{\\sigma^\{2\}\}\\cdot 2\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\\right\)=−2σ2​κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\)\.\\displaystyle=\-\\frac\{2\}\{\\sigma^\{2\}\}\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\.Substituting this into the gradient of the diversity function:

∇𝐡i𝒟~κ\\displaystyle\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\tilde\{\\mathcal\{D\}\}\_\{\\kappa\}=−∑j≠iCi​j⋅∇𝐡i\(κ​\(𝐡i,𝐡j\)2\)\\displaystyle=\-\\sum\_\{j\\neq i\}C\_\{ij\}\\cdot\\nabla\_\{\\mathbf\{h\}\_\{i\}\}\\left\(\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\\right\)=−∑j≠iCi​j⋅\(−2σ2​κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\)\)\\displaystyle=\-\\sum\_\{j\\neq i\}C\_\{ij\}\\cdot\\left\(\-\\frac\{2\}\{\\sigma^\{2\}\}\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\\right\)=2σ2​∑j≠iκattn​\(i,j\)2⋅κ​\(𝐡i,𝐡j\)2​\(𝐡i−𝐡j\)\.\\displaystyle=\\frac\{2\}\{\\sigma^\{2\}\}\\sum\_\{j\\neq i\}\\kappa\_\{\\text\{attn\}\}\(i,j\)^\{2\}\\cdot\\kappa\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}\)^\{2\}\(\\mathbf\{h\}\_\{i\}\-\\mathbf\{h\}\_\{j\}\)\.
∎

## Appendix BDetailed Analysis of Gaussian Synthesis

To demonstrate the distinct behaviors of temperature scaling versus our proposed diversity guidance, we first examine a controlled environment with a known ground truth\. Specifically, we use a simple discrete diffusion model to approximate a 2D Gaussian mixture distribution, a setting where high\-probability modes are explicitly defined\. This allows for a precise evaluation of generation quality and diversity based on mode coverage\.

Figure[4](https://arxiv.org/html/2608.00024#A4.F4)illustrates the synthesis of a 8\-component Gaussian mixture target\. The top row presents an unguided baseline where the key hyperparameter is the sampling temperatureT∈\{1,5,10,20\}T\\in\\\{1,5,10,20\\\}, which modulates the stochasticity of the reverse dynamics\. LowTTtends to yield overly concentrated samples with limited mode coverage, while higherTTincreases exploration and covers more mixture components\. However, this improved coverage comes at a clear cost in sample quality\. WhenTTis large, the particle clouds become substantially more diffuse and less well\-aligned with individual Gaussian components, indicating degraded fidelity despite broader support\. The bottom row applies our proposed diversity guidance term with strengthγ∈\{0\.1,0\.2,0\.5,1\.0\}\\gamma\\in\\\{0\.1,0\.2,0\.5,1\.0\\\}, showing progressively improved mode coverage asγ\\gammaincreases\. Particles distribute across the mixture components more evenly and visually better match the target support\. These results suggest that while temperature scaling forces a trade\-off between concentration and exploration, our proposed method effectively regularizes against mode collapse, enabling broad coverage without relying on high\-temperature sampling\.

## Appendix CDetailed Analysis of Diversity\-Quality Trade\-off

We extend our evaluation to broader settings by analyzing the Diversity\-Quality Pareto frontier\. The Pareto frontier reveals the optimal trade\-off boundary between two competing objectives\. In this context, an outward shift of the frontier indicates that a method achieves higher diversity for a given quality budget \(or vice versa\)\.

Figure[2](https://arxiv.org/html/2608.00024#S5.F2)demonstrates that*diversity\-guided*sampling consistently yields superior Pareto frontiers compared to standard*temperature sampling*across three distinct generative tasks\. In all settings, we observe that while temperature sampling forces a strict trade\-off that sacrificing quality for diversity, diversity guidance mitigates this degradation, maintaining higher sample quality at comparable levels of diversity\.

Gaussian Mixture Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), top\-left\): We evaluate the trade\-off between Coverage \(diversity metric\) and Density \(quality metric\) proposed byNaeemet al\.\([2020](https://arxiv.org/html/2608.00024#bib.bib25)\), computed usingk=5k\{=\}5nearest neighbors and a bandwidth ofσ=3\\sigma\{=\}3over 2,000 samples\. The generation process involves 500 steps with 2,000 particles initialized from a single point\. While varying temperatureT∈\[1,30\]T\\in\[1,30\]expands coverage, it causes a precipitous decline in density\. Conversely, sweeping the guidance strengthγ∈\[0\.01,2\.0\]\\gamma\\in\[0\.01,2\.0\]shifts the frontier outward, achieving higher coverage for equivalent density levels\.

Arithmetic Series Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), top\-right\): We select this task as a controlled setting where quality and diversity metrics are mathematically precise and unambiguous\. The objective is to generate number sequences that follow a strict arithmetic progression\. We measure*Validity*\(the fraction of generated sequences satisfying the arithmetic constraint\) against*Uniqueness*\(the fraction of unique sequences among valid samples\)\. Results are averaged over 5 runs of 1,000 samples each\. We observe that high\-temperature sampling improves uniqueness only at the cost of a sharp drop in validity\. In contrast, diversity guidance significantly extends the Pareto frontier, allowing for high uniqueness with minimal loss in validity\.

Story Continuation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), bottom\-left\): We analyze the trade\-off between Perplexity \(quality metric; lower is better\) and Distinct\-2 \(diversity metric\) proposed byLiet al\.\([2016](https://arxiv.org/html/2608.00024#bib.bib26)\)\. We employ a discrete diffusion language model LLaDA\(Nieet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib21)\)for generation and Qwen3\-4B\(Yanget al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib27)\)as the external evaluator\. The results indicate that aggressive temperature sampling incurs a steep perplexity penalty for marginal gains in diversity\. Diversity guidance attains a more favorable balance, achieving higher Distinct\-2 scores at lower perplexity levels\. While Classifier\-Free Guidance \(CFG\) offers a competitive baseline in specific regimes, diversity guidance maintains robust performance in high\-diversity ranges\.

Brainstorm Generation \(Fig\.[2](https://arxiv.org/html/2608.00024#S5.F2), bottom\-right\): We design this experiment to require the model to generate multiple proposals for a single prompt within one generation\. For example, “Generate 5 prompts for an image of dog\. Be creative\.” We use the same generative model and evaluator as in the story continuation task\. We observe that this task presents a challenging trade\-off for baseline methods\. As shown by the red curve, temperature sampling suffers from a rapid degradation in quality \(increasing perplexity\) as it attempts to increase diversity beyond a Distinct\-2 score of 0\.25\. Similarly, D\-CFG \(green diamonds\) remains clustered in a conservative region, failing to explore the high\-diversity distinctness required for brainstorming\. In contrast, diversity guidance establishes a superior Pareto frontier in the high\-diversity region, maintaining low perplexity \(high quality\) even as the Distinct\-2 metric increases\. The method effectively decouples diversity from significant quality loss, allowing the model to generate varied, creative ideas without sacrificing semantic coherence\. A case study comparing the baseline and our SAKE method under the same temperature setting further illustrates this behavior\. As shown in Table[1](https://arxiv.org/html/2608.00024#S5.T1), the baseline collapses into repetitive syntactic templates, recycling identical phrases across items\. Conversely, SAKE successfully diversifies the semantic content, varying subjects \(breeds\), actions, and settings while maintaining coherence\.

## Appendix DAdditional Numerical Results

##### LLaDA on HumanEval with Nucleus Sampling \(top\-p\)\.

We further examine the effect of nucleus sampling on code\-generation performance by sweepingtop\-pvalues for the base LLaDA model on HumanEval\. Table[8](https://arxiv.org/html/2608.00024#A4.T8)reports the correspondingpass@32results\. Performance improves substantially astop\-pincreases from 0\.5 to 0\.7, rising from 35\.4 to 40\.9\. Beyond this range, performance remains relatively stable, with values between 39\.6 and 41\.2 acrosstop\-psettings from 0\.8 to 1\.0\. The best result is obtained attop\-p= 0\.95, which achieves apass@32of 41\.2\. This observation suggests that moderate\-to\-high sampling diversity is beneficial for HumanEval, while overly restrictive sampling degrades performance\. Overall, these results support the use of the standard HumanEval setting oftop\-p= 0\.95 in our experiments\.

##### Hyperparameter sensitivity\.

We further examine the sensitivity of our method to the two main bandwidth hyperparameters: the semantic bandwidthσ\\sigmaand the attention bandwidthσattn\\sigma\_\{\\text\{attn\}\}\. Overall, the results indicate that performance is stable across broad hyperparameter ranges, suggesting that the method does not require fine\-grained tuning\.

##### Semantic bandwidthσ\\sigma\.

The semantic bandwidthσ\\sigmadetermines the similarity scale at which differences between token representations are treated as meaningful\. Smaller values ofσ\\sigmamake the method more sensitive to local semantic variation, while larger values impose a broader notion of diversity\. Intuitively, smallerσ\\sigmavalues may be better suited for discouraging near\-duplicate generations or repetitive loops, whereas larger values can encourage diversity at the level of broader semantic themes\.

Tables[4](https://arxiv.org/html/2608.00024#A4.T4)and[5](https://arxiv.org/html/2608.00024#A4.T5)report the sensitivity of generation perplexity \(Gen PPL\) toσ\\sigmafor MDLM\(Sahooet al\.,[2024](https://arxiv.org/html/2608.00024#bib.bib19)\)and BD3LM\(Arriolaet al\.,[2025](https://arxiv.org/html/2608.00024#bib.bib48)\), respectively\. For MDLM, the best performance is achieved aroundσ=20\\sigma=20, but results remain close to optimal throughout the range\[5,30\]\[5,30\]\. For BD3LM, the lowest Gen PPL is obtained nearσ=80\\sigma=80, with similarly stable performance over the broader interval\[40,120\]\[40,120\]\. In both cases, performance degrades only whenσ\\sigmabecomes substantially larger than these ranges\. These findings suggest that the semantic bandwidth is robust to moderate misspecification, and that selectingσ\\sigmawithin a broad reasonable interval is sufficient in practice\.

Table 4:Sensitivity to semantic bandwidthσ\\sigmafor MDLM\. Lower Gen PPL is better\.Table 5:Sensitivity to semantic bandwidthσ\\sigmafor BD3LM\. Lower Gen PPL is better\.
##### Attention bandwidthσattn\\sigma\_\{\\text\{attn\}\}\.

The attention bandwidthσattn\\sigma\_\{\\text\{attn\}\}controls the effective positional receptive field by modulating how quickly attention decays with token distance\. Larger values permit stronger influence from more distant tokens, whereas smaller values concentrate attention more locally\.

A useful interpretation ofσattn\\sigma\_\{\\text\{attn\}\}can be obtained by defining a small thresholdϵ\\epsilonat which attention is considered to have effectively vanished\. For token distancedd, this yields

exp⁡\(−d22​σattn2\)=ϵ⟹d=σattn​2​ln⁡\(1ϵ\)\.\\exp\\left\(\-\\frac\{d^\{2\}\}\{2\\sigma\_\{\\text\{attn\}\}^\{2\}\}\\right\)=\\epsilon\\quad\\Longrightarrow\\quad d=\\sigma\_\{\\text\{attn\}\}\\sqrt\{2\\ln\\left\(\\frac\{1\}\{\\epsilon\}\\right\)\}\.\(27\)For example, whenϵ=0\.01\\epsilon=0\.01, the corresponding vanishing distance is approximatelydv≈3\.03​σattnd\_\{v\}\\approx 3\.03\\,\\sigma\_\{\\text\{attn\}\}\. This provides a simple initialization heuristic: one may chooseσattn\\sigma\_\{\\text\{attn\}\}by specifying a desired effective context range, thereby substantially reducing the need for exhaustive tuning\.

Tables[6](https://arxiv.org/html/2608.00024#A4.T6)and[7](https://arxiv.org/html/2608.00024#A4.T7)summarize the empirical sensitivity of Gen PPL toσattn\\sigma\_\{\\text\{attn\}\}\. For BD3LM, performance improves substantially asσattn\\sigma\_\{\\text\{attn\}\}increases from very small values, and remains near\-optimal across a broad range from 10 to 80\. For MDLM, the best values are obtained aroundσattn=20\\sigma\_\{\\text\{attn\}\}=20to 40, but performance remains stable over a much wider interval extending up to 300\. Taken together, these results indicate thatσattn\\sigma\_\{\\text\{attn\}\}is also not highly sensitive, and that the vanishing\-distance heuristic provides a practical and interpretable guideline for setting this parameter\.

Table 6:Sensitivity to attention bandwidthσattn\\sigma\_\{\\text\{attn\}\}for BD3LM\. Lower Gen PPL is better\.Table 7:Sensitivity to attention bandwidthσattn\\sigma\_\{\\text\{attn\}\}for MDLM\. Lower Gen PPL is better\.Table 8:Effect of nucleus sampling \(top\-p\) on HumanEval for the base LLaDA model\.![Refer to caption](https://arxiv.org/html/2608.00024v1/comparison_configs_grid.png)Figure 4:Comparison of sampling behaviors for a 2D 9\-Gaussian\-mixture target\.Top row: baseline unguided reverse process at different temperaturesT∈\{1,5,10,20\}T\\in\\\{1,5,10,20\\\}, illustrating a coverage\-fidelity trade\-off \(higherTTcovers more modes but yields more diffuse, lower\-quality samples\)\.Bottom row: Diversity guidance with strengthsγ∈\{0\.1,0\.2,0\.5,1\.0\}\\gamma\\in\\\{0\.1,0\.2,0\.5,1\.0\\\}, which improves mode coverage while maintaining better precision\. Rightmost column shows ground\-truth samples\.Table 9:Comparison of pass@k results across HumanEval and MBPP benchmarks, with varying temperatures\.