Do Large Language Models Capture the Diversity in their Training Data?

# Do Large Language Models Capture the Diversityin their Training Data?
Source: [https://arxiv.org/html/2609.02275](https://arxiv.org/html/2609.02275)
Youqi Wu††thanks:Department of Computer Science and Engineering, The Chinese University of Hong Kong, yqwu24@cse\.cuhk\.edu\.hkFarzan Farnia††thanks:Department of Computer Science and Engineering, The Chinese University of Hong Kong, farnia@cse\.cuhk\.edu\.hk

###### Abstract

Large language models are trained to model conditional distributions over text, yet it remains inadequately understood whether they capture the full diversity of plausible outputs present in their training data\. We study this question through an information\-theoretic lens by comparing the*conditional entropy*of model\-generated outputs with that of the corresponding training data\. Given paired input\-output samples, we use conditional entropy and its matrix\-based analogue based on*von Neumann entropy*to measure output variability beyond what is explained by the conditioning input, without requiring multiple reference outputs for the same prompt\. Across LLM families with publicly available training data, including OLMo, Pythia, and GPT\-Neo, we consistently find that model\-generated outputs exhibit lower conditional entropy than their training data, across different model scales, sequence lengths, and decoding strategies\. We observe a similar conditional diversity gap beyond language modeling, including class\-conditioned ImageNet generators and text\-conditioned models trained on MS\-COCO\. To address this gap, we propose a post\-hoc correction mechanism that generates multiple outputs for each input and reweights them through a matrix\-entropy projection, increasing conditional diversity while remaining close to the original model distribution\. We prove the concavity of the matrix\-based conditional entropy functional, which makes the resulting entropy\-constrained projection a convex optimization problem, and develop a scalable mirror\-descent algorithm for its implementation\. Our results reveal a systematic conditional diversity gap between modern generative models and their training data, and provide an information\-theoretic framework for measuring and mitigating this gap\.

## 1Introduction

Large language models are increasingly used as conditional distribution models: given a prompt, instruction, context, or document, they are expected to produce not only a single plausible completion, but samples from a rich set of valid outputs\. The same viewpoint underlies many other conditional generative systems, including text\-to\-image models, image captioning systems, and class\-conditioned image generators\[[1](https://arxiv.org/html/2609.02275#bib.bib39),[2](https://arxiv.org/html/2609.02275#bib.bib40),[3](https://arxiv.org/html/2609.02275#bib.bib2),[4](https://arxiv.org/html/2609.02275#bib.bib3)\]\. In these settings, the conditioning variable rarely determines a unique output\. A prompt may admit multiple correct phrasings, reasoning paths, levels of detail, or stylistic choices; an image may admit several faithful captions; and a class label may correspond to a broad range of visual instances\. A central question is therefore whether modern conditional generative models capture the full range of outputs supported by the data distribution\.

Most existing evaluations do not directly answer this question\. Standard language\-modeling metrics such as likelihood or perplexity measure average predictive fit, while many generation benchmarks focus on accuracy, alignment, and semantic consistency\. For image and multimodal generation, metrics such as CLIPScore\[[5](https://arxiv.org/html/2609.02275#bib.bib8)\], MAUVE\[[6](https://arxiv.org/html/2609.02275#bib.bib12)\], and conditional Fréchet\-type distances\[[7](https://arxiv.org/html/2609.02275#bib.bib11)\]provide valuable tools for comparing generated and reference samples\. However, a model could still perform well under such metrics while still concentrating its probability mass on a narrower subset of valid outputs\. This is important for LLMs and prompt\-guided generation, where the goal is not merely to produce one acceptable answer, but to represent the conditional variability present in natural text data\.

A direct measurement of*conditional*output variability is statistically challenging\. In many datasets, we observe paired samples\(X,Y\)\(X,Y\), whereXXis the conditioned variable andYYis the corresponding output, but we do not observe many independent human outputs for the exact same condition\. Thus, estimating conditional entropy separately for each fixed prompt is usually infeasible\. To address this, we study conditional entropy through paired input\-output samples\. At the classical level, Shannon conditional entropy measures the uncertainty inYYthat remains after observingXX\. At the kernel level, we use a matrix\-based analog built from a product kernel on pairs\(X,Y\)\(X,Y\)and subtract the entropy of the conditioning variable\. This gives a way to measure output variability beyond the variability already explained by the inputs, without requiring repeated reference outputs for every condition\.

Using this viewpoint, we find a consistent conditional output\-range gap in modern generative models\. Across LLM families with publicly accessible training data, including OLMo\[[8](https://arxiv.org/html/2609.02275#bib.bib41)\], Pythia\[[9](https://arxiv.org/html/2609.02275#bib.bib42)\], and GPT\-Neo\[[10](https://arxiv.org/html/2609.02275#bib.bib43)\], we empirically evaluate that model\-generated continuations exhibit lower conditional entropy than the corresponding training data\. We observe the same phenomenon beyond text, including class\-conditioned ImageNet generation and text\-conditioned image models trained on MS\-COCO\. These results suggest that the issue is not specific to one architecture or training approach\. Rather, conditional generative models appear to represent a narrowed version of the conditional distribution present in their data, as their conditional entropy underestimates that of the training data\.

We formalize this phenomenon using conditional entropy and matrix\-based conditional entropy, which extends the recent diversity bias analysis in\[[11](https://arxiv.org/html/2609.02275#bib.bib1)\]from unconditional generative models to conditional generative AI and LLM models\. The resulting score compares the entropy of the joint input\-output distribution with the entropy of the input marginal, thereby isolating the part of the variability attributable to the conditional output law\. This formulation will be natural for LLMs and general conditional generation systems, because it evaluates paired samples directly, rather than requiring multiple ground\-truth responses for each prompt\. It also connects the empirical measurement problem to matrix\-based quantum entropy functionals\.

On the technical side, we prove that the proposed matrix\-based conditional entropy is a concave functional of the joint input\-output distribution\. This structural property has two consequences\. It yields finite\-sample monotonicity and underestimation properties analogous to the classical behavior of the classical Shannon entropy\. More importantly, the concavity of the matrix\-based quantum entropy implies that projecting a generated conditional distribution toward a higher conditional\-entropy region can be formulated as a convex optimization problem\.

Following the analysis, we propose a post\-hoc sample\-space reweighting method for LLMs and conditional generators\. Given promptsx1,…,xNx\_\{1\},\\ldots,x\_\{N\}, we generate and consider multiple outputsyi,1,…,yi,my\_\{i,1\},\\ldots,y\_\{i,m\}for each promptxix\_\{i\}and jointly optimize weights over these generated candidates so as to improve the overall conditional variability\. The key constraint is that the total weight assigned to each prompt remains fixed, so the input marginal is preserved and only the conditional distribution over generated outputs is adjusted\. The projection then finds the closest reweighted empirical distribution whose matrix\-based conditional entropy exceeds a target level\. In this way, the method does not retrain the model or synthesize new samples; instead, it asks whether the model has already generated valid but underweighted outputs, and redistributes probability mass toward a broader conditional empirical distribution\.

For scalability, we develop an efficient mirror\-descent implementation over a product of simplices, one simplex for the candidates generated from each condition\. This extends exponentiated\-gradient reweighting from the unconditional setting to conditional generation while exactly preserving the prompt marginal\. Since the joint input\-output feature space is a tensor\-product space and can be prohibitively high\-dimensional, we implement the method using sketched joint features\. For finite\-dimensional embeddings, we use random projections of Kronecker\-product features; for shift\-invariant kernels, we use direct joint random Fourier features for the product kernel\. These approximations avoid explicitly forming the full tensor\-product representation and reduce the cost of each optimization step to depend on the sketch dimension rather than the raw product dimension\. The contributions of this work can be summarized as:

- •We formulate the problem of whether LLMs and conditional generative models capture the full range of valid training outputs using conditional entropy and its matrix\-based counterpart\.
- •We empirically demonstrate a consistent conditional output\-range gap across LLMs with publicly available training data, class\-conditioned ImageNet and MS\-COCO generators\.
- •We derive concavity and finite\-sample properties of the applied matrix\-based conditional entropy functional, and show that entropy\-constrained reweighting can be formulated as a convex program\.
- •We develop a scalable post\-hoc reweighting algorithm based on product\-simplex mirror descent and sketched joint features for adjusting conditional output distributions\.

## 2Preliminaries

We consider a conditional generation problem with conditioning variableX∈𝒳X\\in\\mathcal\{X\}and output variableY∈𝒴Y\\in\\mathcal\{Y\}\. The data distribution is denoted byPX​YP\_\{XY\}, with marginalPXP\_\{X\}and conditional lawPY\|XP\_\{Y\\,\|\\,X\}\. A conditional generative model specifies a conditional distributionQY\|XQ\_\{Y\\,\|\\,X\}, which together with the same input marginal induces the joint model distributionQX​Y=PX​QY\|XQ\_\{XY\}=P\_\{X\}Q\_\{Y\\,\|\\,X\}\.

LetkX:𝒳×𝒳→ℝk\_\{X\}:\\mathcal\{X\}\\times\\mathcal\{X\}\\to\\mathbb\{R\}andkY:𝒴×𝒴→ℝk\_\{Y\}:\\mathcal\{Y\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}be positive semidefinite kernels\. Throughout the paper, we use normalized kernels which satisfykX​\(x,x\)=1k\_\{X\}\(x,x\)=1andkY​\(y,y\)=1k\_\{Y\}\(y,y\)=1for everyx∈𝒳x\\in\\mathcal\{X\}andy∈𝒴y\\in\\mathcal\{Y\}\.111Any kernel functionkkcan be normalized by replacingk⁡\(z,z′\)k\(z,z^\{\\prime\}\)withk~​\(z,z′\)=k⁡\(z,z′\)/k⁡\(z,z\)​k​\(z′,z′\)\\widetilde\{k\}\(z,z^\{\\prime\}\)=k\(z,z^\{\\prime\}\)/\\sqrt\{k\(z,z\)k\(z^\{\\prime\},z^\{\\prime\}\)\}\.For paired samples, we use the product kernel

kX​Y​\(\[x,y\],\[x′,y′\]\)=kX​\(x,x′\)⋅kY​\(y,y′\)\.k\_\{XY\}\\big\(\[x,y\],\[x^\{\\prime\},y^\{\\prime\}\]\\big\)=k\_\{X\}\(x,x^\{\\prime\}\)\\cdot k\_\{Y\}\(y,y^\{\\prime\}\)\.Given samples\(x1,y1\),…,\(xn,yn\)\(x\_\{1\},y\_\{1\}\),\\ldots,\(x\_\{n\},y\_\{n\}\), letKXK\_\{X\}andKYK\_\{Y\}be the corresponding Gram matrices\. The product kernel has Gram matrix where⊙\\odotdenotes the Hadamard product:KX​Y=KX⊙KYK\_\{XY\}=K\_\{X\}\\odot K\_\{Y\}

For a positive semidefinite matrixAAwith unit\-traceTr⁡\(A\)=1\\operatorname\{Tr\}\(A\)=1, its von Neumann entropy is defined as

HvN​\(A\)=−Tr⁡\(A​log⁡A\)H\_\{\\mathrm\{vN\}\}\(A\)=\-\\operatorname\{Tr\}\\bigl\(A\\log A\\bigr\)Following the matrix\-based entropy framework of\[[12](https://arxiv.org/html/2609.02275#bib.bib26),[13](https://arxiv.org/html/2609.02275#bib.bib6)\], we measure conditional output uncertainty by the difference between the joint input–output entropy and the input entropy:

HvN​\(Y\|X;P^n\):=HvN​\(1n​KX​Y\)−HvN​\(1n​KX\)\.H\_\{\\mathrm\{vN\}\}\\bigl\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\\bigr\):=H\_\{\\mathrm\{vN\}\}\\bigl\(\\frac\{1\}\{n\}K\_\{XY\}\\bigr\)\-H\_\{\\mathrm\{vN\}\}\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\bigr\)\.We note that the exponential of the above function is the same as the*Conditional Vendi*score, which is introduced and analyzed by Jalali et al\.\[[13](https://arxiv.org/html/2609.02275#bib.bib6)\]\. This quantity is the kernel analogue of conditional entropy: it measures the uncertainty contributed by the output after accounting for the entropy already present in the conditioning variable\.

We also use the order\-22matrix entropyH2​\(A\)=−log⁡Tr⁡\(A2\)H\_\{2\}\(A\)=\-\\log\\operatorname\{Tr\}\(A^\{2\}\), which gives the empirical conditional order\-22entropy as follows where∥⋅∥F\\\|\\cdot\\\|\_\{F\}denotes the Frobenius norm:

H2​\(Y\|X;P^n\):=H2​\(1n​KX​Y\)−H2​\(1n​KX\)=log⁡‖KX‖F2‖KX⊙KY‖F2\.H\_\{2\}\\bigl\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\\bigr\):=H\_\{2\}\\bigl\(\\frac\{1\}\{n\}K\_\{XY\}\\bigr\)\-H\_\{2\}\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\bigr\)=\\log\\frac\{\\\|K\_\{X\}\\\|\_\{F\}^\{2\}\}\{\\\|K\_\{X\}\\odot K\_\{Y\}\\\|\_\{F\}^\{2\}\}\.This order\-22version is useful as a computationally simpler counterpart to von Neumann entropy, as formulated and framed as the*Conditional RKE*score in\[[13](https://arxiv.org/html/2609.02275#bib.bib6)\]\. Population\-level operator definitions and additional details are deferred to Appendix[B](https://arxiv.org/html/2609.02275#A2)\.

## 3Conditional output\-range gaps in open language models

We first evaluate whether open language models match the conditional output range of their training data\. We focus on OLMo, Pythia, and GPT\-Neo, for which the training corpora are publicly available or reconstructable\. Each example is represented as a paired sample\(X,Y\)\(X,Y\), whereXXis a prefix withlp∈\{3,4,5\}l\_\{p\}\\in\\\{3,4,5\\\}tokens andYYis the corresponding continuation withlc∈\{2,3,4\}l\_\{c\}\\in\\\{2,3,4\\\}tokens\. For the training baseline,YYis the continuation observed in the corpus; for model samples,YYis generated from the same prefix using greedy decoding, nucleus sampling, or ancestral sampling\. We then compare the paired training and model distributions using conditional von Neumann entropy \(VNE\), conditional order\-22matrix entropy, and lexical Distinct\-22and Distinct\-33scores\.

In all numerical experiments and tables, we report theexponentiated formsexp⁡\(HvN\)\\exp\(H\_\{\\mathrm\{vN\}\}\)andexp⁡\(H2\)\\exp\(H\_\{2\}\), so that the reported values can be interpreted as effective numbers of conditionally distinguishable outputs under the corresponding entropy measures\. Table[1](https://arxiv.org/html/2609.02275#S3.T1)reports the results\. The gap is defined asΔ=Metrictrain−Metricmodel\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{model\}\}; therefore, positive values indicate that model\-generated continuations have a smaller measured conditional output range than the training data\. The pattern is consistent across all three model families and all decoding methods: the reported gap is positive\.

The lexical Distinct\-nnscores provide a complementary surface\-level check\. They also show positive gaps in every setting, with the largest deficits again occurring under greedy decoding\. However, Distinct\-nndoes not explicitly account for the conditioning prefix and only measures n\-gram variation in the generated continuations\. We therefore use it as supporting evidence, while the main comparison is based on conditional entropy over paired prefix–continuation samples\. The confidence intervals, computed over five independent evaluations, are small relative to the observed gaps, indicating that the effect is stable rather than a fluctuation of a single sample draw\.

These experiments motivate two questions\. First, what structural property makes conditional von Neumann entropy a principled analogue of Shannon conditional entropy? Second, if a model has already generated multiple candidate outputs for each input, can we reweight those candidates to obtain a higher\-entropy conditional empirical distribution while remaining close to the original model samples? The next section answers the first question through a concavity result\. The following section uses this result to formulate post\-hoc conditional reweighting as a convex projection problem\.

Table 1:Diversity metrics across models and generation strategies with sample size 20K\. For each model, we report conditional VNE, conditional RKE, and Distinct\-2 as mean±\\pmstandard deviation\.ModelGroupExp\.Cond\.VNE𝚫\\bm\{\\Delta\}VNEExp\.Cond\.RKE𝚫\\bm\{\\Delta\}RKEDistinct\-2𝚫\\bm\{\\Delta\}D\-2OLMoTraining Data338\.58±0\.69338\.58\{\\scriptsize\\pm 0\.69\}–67\.43±0\.6367\.43\{\\scriptsize\\pm 0\.63\}–0\.709±0\.0040\.709\{\\scriptsize\\pm 0\.004\}–Greedy Decoding218\.88±0\.34218\.88\{\\scriptsize\\pm 0\.34\}119\.7046\.17±0\.1946\.17\{\\scriptsize\\pm 0\.19\}21\.260\.276±0\.0010\.276\{\\scriptsize\\pm 0\.001\}0\.433Nucleus Sampling287\.31±0\.98287\.31\{\\scriptsize\\pm 0\.98\}51\.2755\.08±0\.2755\.08\{\\scriptsize\\pm 0\.27\}12\.350\.516±0\.0030\.516\{\\scriptsize\\pm 0\.003\}0\.193Ancestral Sampling297\.31±0\.78297\.31\{\\scriptsize\\pm 0\.78\}41\.2756\.81±0\.2256\.81\{\\scriptsize\\pm 0\.22\}10\.620\.558±0\.0030\.558\{\\scriptsize\\pm 0\.003\}0\.151PythiaTraining Data262\.55±0\.47262\.55\{\\scriptsize\\pm 0\.47\}–53\.44±0\.3853\.44\{\\scriptsize\\pm 0\.38\}–0\.701±0\.0060\.701\{\\scriptsize\\pm 0\.006\}–Greedy Decoding176\.39±0\.27176\.39\{\\scriptsize\\pm 0\.27\}86\.1640\.98±0\.6440\.98\{\\scriptsize\\pm 0\.64\}12\.460\.266±0\.0010\.266\{\\scriptsize\\pm 0\.001\}0\.435Nucleus Sampling232\.24±0\.78232\.24\{\\scriptsize\\pm 0\.78\}30\.3147\.51±0\.2747\.51\{\\scriptsize\\pm 0\.27\}5\.930\.525±0\.0050\.525\{\\scriptsize\\pm 0\.005\}0\.176Ancestral Sampling241\.79±0\.23241\.79\{\\scriptsize\\pm 0\.23\}20\.7649\.00±0\.4249\.00\{\\scriptsize\\pm 0\.42\}4\.440\.564±0\.0060\.564\{\\scriptsize\\pm 0\.006\}0\.137GPT\-NeoTraining Data260\.01±0\.52260\.01\{\\scriptsize\\pm 0\.52\}–52\.98±0\.2352\.98\{\\scriptsize\\pm 0\.23\}–0\.701±0\.0080\.701\{\\scriptsize\\pm 0\.008\}–Greedy Decoding179\.36±0\.88179\.36\{\\scriptsize\\pm 0\.88\}80\.6543\.34±0\.9343\.34\{\\scriptsize\\pm 0\.93\}9\.640\.284±0\.0030\.284\{\\scriptsize\\pm 0\.003\}0\.417Nucleus Sampling229\.62±0\.42229\.62\{\\scriptsize\\pm 0\.42\}30\.3948\.48±0\.6748\.48\{\\scriptsize\\pm 0\.67\}4\.500\.526±0\.0020\.526\{\\scriptsize\\pm 0\.002\}0\.175Ancestral Sampling238\.50±0\.61238\.50\{\\scriptsize\\pm 0\.61\}21\.5149\.59±0\.3249\.59\{\\scriptsize\\pm 0\.32\}3\.390\.570±0\.0020\.570\{\\scriptsize\\pm 0\.002\}0\.131

## 4Theoretical Features of Kernel\-induced Conditional von Neumann Entropy

We next establish the structural property needed for our proposed optimization of conditional von Neumann entropy\. For Shannon entropy, we note that conditional entropy is well\-known to be concave in the joint distributionPX​YP\_\{XY\}\. In this section, we prove that the kernel matrix\-based \(quantum\) conditional entropy defined in\[[12](https://arxiv.org/html/2609.02275#bib.bib26)\]used in our formulation is also a concave functional of the joint distribution\. This property gives a convex\-analytic justification for the projection method developed in Section[5](https://arxiv.org/html/2609.02275#S5)\. We begin by reviewing the corresponding known fact for the classical Shannon entropy\.

###### Proposition 1\(Concavity of classical conditional Shannon entropy\)\.

For finite alphabets𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}, the Shannon conditional entropyPX​Y⟼H⁡\(Y\|X\)P\_\{XY\}\\longmapsto H\(Y\|X\)is concave in the joint distributionPX​YP\_\{XY\}\.

###### Proof\.

This proposition is known in the literature\. For completeness, we give a proof in the Appendix\. ∎

Next, we highlight that the concavity of the kernel\-induced matrix\-based \(quantum\) conditional entropy in\[[12](https://arxiv.org/html/2609.02275#bib.bib26)\]does not follow from this result or a similar proof\. This is because unlike the known identity for classical Shannon entropy thatH⁡\(Y\|X\)=𝔼x∼PX​\[H⁡\(Y\|X=x\)\]H\(Y\|X\)=\\mathbb\{E\}\_\{x\\sim P\_\{X\}\}\[H\(Y\|X=x\)\], the kernel\-induced conditional von Neumann entropy in\[[12](https://arxiv.org/html/2609.02275#bib.bib26)\]does not satisfy the same identity\. In the proof of the following proposition, we utilize the strong subadditivity of the quantum entropy to prove the concavity of the quantum conditional entropy functional as defined in\[[12](https://arxiv.org/html/2609.02275#bib.bib26)\]:

###### Proposition 2\(Concavity of kernel\-induced conditional von Neumann entropy\)\.

LetkXk\_\{X\}andkYk\_\{Y\}be normalized positive semidefinite kernels\. Then, the conditional von Neumann entropy defined in\[[12](https://arxiv.org/html/2609.02275#bib.bib26)\]

PX​Y⟼ℋvN​\(Y\|X;PX​Y\)P\_\{XY\}\\longmapsto\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\|X;P\_\{XY\}\)is a concave functional of the joint distributionPX​YP\_\{XY\}\.

###### Proof\.

We provide the proof in the Appendix\. ∎

Our shown Proposition[2](https://arxiv.org/html/2609.02275#Thmproposition2)highlights the key structural property needed by the correction method we propose in the next section\. Since superlevel sets of concave functions are convex sets, for every thresholdρ\\rhothe set

\{Q:ℋvN​\(Y\|X;Q\)≥ρ\}\\bigl\\\{Q:\\\>\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\|X;Q\)\\geq\\rho\\bigr\\\}is a convex set\. Therefore, if we search over empirical distributions supported on a fixed set of generated samples, the constraint that the reweighted distribution has conditional entropy at leastρ\\rhois a convex constraint\. As a result, the conditional von Neumann entropy is suitable not only as a diagnostic, but further as the basis for a tractable post\-hoc reweighting procedure\.

## 5Kernel\-based Entropy\-Constrained Projection for Conditional Generators

The empirical gaps in Section[3](https://arxiv.org/html/2609.02275#S3)motivate a post\-hoc correction problem\. Suppose a conditional generator has already produced multiple candidate outputs for each input\. We ask whether one can reweight these candidates so that the resulting conditional distribution moves toward a higher conditional\-entropy region while remaining close to the original model distribution\. The concavity of conditional von Neumann entropy gives a projection framework for this question\.

Fix an input marginalPXP\_\{X\}\. For a conditional lawRY\|XR\_\{Y\|X\}, define

ℋvN​\(RY\|X,PX\):=ℋvN​\(Y\|X;PX​RY\|X\),\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R\_\{Y\|X\};P\_\{X\}\):=\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\|X;P\_\{X\}R\_\{Y\|X\}\),wherePX​RY\|XP\_\{X\}R\_\{Y\|X\}denotes the joint law induced byPXP\_\{X\}andRY\|XR\_\{Y\|X\}\. For a thresholdρ\\rho, let

𝒞ρ​\(PX\)=\{RY\|X:ℋvN​\(RY\|X,PX\)≥ρ\}\.\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)=\\bigl\\\{R\_\{Y\|X\}:\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R\_\{Y\|X\};P\_\{X\}\)\\geq\\rho\\bigr\\\}\.By Proposition[2](https://arxiv.org/html/2609.02275#Thmproposition2),𝒞ρ​\(PX\)\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)is a convex set\.

We compare conditional laws through divergences lifted to joint laws with the same input marginal\. Given a Bregman divergenceDΦD\_\{\\Phi\}on joint laws, define

DΦ\|PX​\(RY\|X,QY\|X\):=DΦ​\(PX​RY\|X,PX​QY\|X\)\.D\_\{\\Phi\|P\_\{X\}\}\\bigl\(R\_\{Y\|X\},Q\_\{Y\|X\}\\bigr\):=D\_\{\\Phi\}\\bigl\(P\_\{X\}R\_\{Y\|X\},P\_\{X\}Q\_\{Y\|X\}\\bigr\)\.For example, whenDΦD\_\{\\Phi\}is KL divergence, this gives thePXP\_\{X\}\-averaged conditional KL\. WhenDΦD\_\{\\Phi\}is the squared Hilbertian distance between kernel mean embeddings of joint pairs\(X,Y\)\(X,Y\), it gives the conditional MMD with the input marginal held fixed\.

###### Theorem 1\.

Letρ=ℋvN​\(PY\|X,PX\)\\rho=\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(P\_\{Y\|X\};P\_\{X\}\), and letQY\|X⋆Q^\{\\star\}\_\{Y\|X\}be theDΦ\|PXD\_\{\\Phi\|P\_\{X\}\}\-Bregman projection ofQY\|XQ\_\{Y\|X\}onto𝒞ρ​\(PX\)\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)\. Then, we have

DΦ\|PX​\(PY\|X,QY\|X⋆\)≤DΦ\|PX​\(PY\|X,QY\|X\)−DΦ\|PX​\(QY\|X⋆,QY\|X\)D\_\{\\Phi\|P\_\{X\}\}\\bigl\(P\_\{Y\|X\},Q^\{\\star\}\_\{Y\|X\}\\bigr\)\\leq D\_\{\\Phi\|P\_\{X\}\}\\bigl\(P\_\{Y\|X\},Q\_\{Y\|X\}\\bigr\)\\,\-\\,D\_\{\\Phi\|P\_\{X\}\}\\bigl\(Q^\{\\star\}\_\{Y\|X\},Q\_\{Y\|X\}\\bigr\)

The theorem gives a geometric interpretation of the projection step\. If the entropy threshold is chosen so that the data conditional law is feasible, then the projection cannot increase the Bregman discrepancy from the data law\. The second term is the Pythagorean decrease obtained by replacingQY\|XQ\_\{Y\|X\}with its high\-entropy projection\. The proof is provided in Appendix[C](https://arxiv.org/html/2609.02275#A3)\.

We now instantiate this principle on generated samples\. Letx1,…,xNx\_\{1\},\\ldots,x\_\{N\}be input prompts, and letyi,1,…,yi,my\_\{i,1\},\\ldots,y\_\{i,m\}bemmgenerated outputs for promptxix\_\{i\}\. We assign weightspi∈Δmp\_\{i\}\\in\\Delta\_\{m\}to the candidates generated fromxix\_\{i\}and place joint massqi,j=1N​pi,jq\_\{i,j\}=\\tfrac\{1\}\{N\}p\_\{i,j\}on\(xi,yi,j\)\(x\_\{i\},y\_\{i,j\}\)\. Hence each input keeps total mass1/N1/N, and the reweighting modifies only the conditional distribution over generated outputs\.

LetK∈ℝN​m×N​mK\\in\\mathbb\{R\}^\{Nm\\times Nm\}be the product\-kernel matrix over generated pairs,

K\(i,j\),\(i′,j′\)=kX​\(xi,xi′\)​kY​\(yi,j,yi′,j′\)\.K\_\{\(i,j\),\(i^\{\\prime\},j^\{\\prime\}\)\}=k\_\{X\}\(x\_\{i\},x\_\{i^\{\\prime\}\}\)k\_\{Y\}\(y\_\{i,j\},y\_\{i^\{\\prime\},j^\{\\prime\}\}\)\.Letui,j=1/\(N​m\)u\_\{i,j\}=1/\(Nm\)represent the uniform weight distribution \(original baseline\)\. We formulate the finite\-support projection as

minq∈ℝN​m\\displaystyle\\min\_\{q\\in\\mathbb\{R\}^\{Nm\}\}\(q−u\)⊤​K​\(q−u\)\\displaystyle\(q\-u\)^\{\\top\}K\(q\-u\)\(1\)s\.t\.\\displaystyle\\mathrm\{s\.t\.\}ℋvN​\(q\)≥ρ,\\displaystyle\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(q\)\\geq\\rho,qi,j≥0,∑j′=1mqi,j′=1Nfor alli∈\{1,…,N\},j∈\{1,…,m\}\.\\displaystyle q\_\{i,j\}\\geq 0,\\qquad\\sum\_\{j^\{\\prime\}=1\}^\{m\}q\_\{i,j^\{\\prime\}\}=\\frac\{1\}\{N\}\\quad\\text\{for all \}\\\>i\\in\\\{1,\\ldots,N\\\},\\,j\\in\\\{1,\\ldots,m\\\}\.The objective is the finite\-sample conditional MMD between the reweighted empirical distribution and the original uniformly weighted model samples\. The entropy term is the empirical conditional von Neumann entropy of the weighted paired sample\. Since the block constraints fix the empirical input marginal, the input\-entropy term is constant over the feasible set\.

###### Proposition 3\.

The optimization problem equation[1](https://arxiv.org/html/2609.02275#S5.E1)is a convex program, and its objective is the finite\-support instance of the lifted squared\-MMD discrepancyDΦ\|PXD\_\{\\Phi\|P\_\{X\}\}\.

For large sample sets, we solve a covariance\-space approximation of equation[1](https://arxiv.org/html/2609.02275#S5.E1)\. Letz~i,j∈ℝr\\tilde\{z\}\_\{i,j\}\\in\\mathbb\{R\}^\{r\}be normalized sketched features for the pair\(xi,yi,j\)\(x\_\{i\},y\_\{i,j\}\), chosen so that their inner products approximate the product kernel\. Define

C~​\(p\)=1N​∑i=1N∑j=1mpi,j​z~i,j​z~i,j⊤\.\\widetilde\{C\}\(p\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{m\}p\_\{i,j\}\\tilde\{z\}\_\{i,j\}\\tilde\{z\}\_\{i,j\}^\{\\top\}\.We optimize the penalized covariance\-space objective

minp1,…,pN∈ΔmF⁡\(p\):=‖1N​∑i,j\(pi,j−1/m\)​z~i,j‖22−λ​HvN​\(C~​\(p\)\)\.\\min\_\{p\_\{1\},\\ldots,p\_\{N\}\\in\\Delta\_\{m\}\}\\quad F\(p\):=\\Bigl\\\|\\frac\{1\}\{N\}\\sum\_\{i,j\}\(p\_\{i,j\}\-1/m\)\\tilde\{z\}\_\{i,j\}\\Bigr\\\|\_\{2\}^\{2\}\-\\lambda H\_\{\\mathrm\{vN\}\}\(\\widetilde\{C\}\(p\)\)\.\(2\)
###### Proposition 4\.

The optimization problem equation[2](https://arxiv.org/html/2609.02275#S5.E2)is a convex program\. It is the covariance\-space analogue of the kernel projection equation[1](https://arxiv.org/html/2609.02275#S5.E1)under the sketched product\-kernel representation\.

We propose the algorithm*Conditional Entropy Projection by Block Exponentiated Gradient*\(CEP\-BEG\) for solving equation[2](https://arxiv.org/html/2609.02275#S5.E2)\. As described in Algorithm[1](https://arxiv.org/html/2609.02275#alg1), CEP\-BEG performs block mirror descent over the product simplexΔmN\\Delta\_\{m\}^\{N\}, with separated exponentiated\-gradient \(EG\) normalization per input block\. Note that we include this block normalization to preserve the empirical input marginal\. Theorem[2](https://arxiv.org/html/2609.02275#Thmtheorem2)provides a convergence guarantee for the averaged iterate in CEP\-BEG\.

Algorithm 1CEP\-BEG: Conditional Entropy Projection by Block Exponentiated Gradient0:Inputs

\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}, generated outputs

\{yi,j\}j=1m\\\{y\_\{i,j\}\\\}\_\{j=1\}^\{m\}, sketch dimension

rr, entropy weight

λ\\lambda, step size

η\\eta, iterations

TT\.

1:Construct normalized sketched joint features

\{z~i,j∈ℝr\}\\\{\\tilde\{z\}\_\{i,j\}\\in\\mathbb\{R\}^\{r\}\\\}\.

2:Initialize

pi,j\(0\)=1/mp\_\{i,j\}^\{\(0\)\}=1/mfor all

i,ji,j\.

3:for

t=0,…,T−1t=0,\\ldots,T\-1do

4:Form the weighted covariance

C~​\(p\(t\)\)\\widetilde\{C\}\(p^\{\(t\)\}\)\.

5:Compute a gradient or subgradient

g\(t\)g^\{\(t\)\}of

FFat

p\(t\)p^\{\(t\)\}\.

6:Update each input block by

pi,j\(t\+1\)=pi,j\(t\)​exp⁡\(−η​gi,j\(t\)\)/\(∑ℓ=1mpi,ℓ\(t\)​exp⁡\(−η​gi,ℓ\(t\)\)\)p\_\{i,j\}^\{\(t\+1\)\}=\{p\_\{i,j\}^\{\(t\)\}\\exp\(\-\\eta g\_\{i,j\}^\{\(t\)\}\)\}/\\bigl\(\{\\sum\_\{\\ell=1\}^\{m\}p\_\{i,\\ell\}^\{\(t\)\}\\exp\(\-\\eta g\_\{i,\\ell\}^\{\(t\)\}\)\}\\bigr\)\.

7:endfor

8:returnWeights

qi,j=pi,j\(T\)/Nq\_\{i,j\}=p\_\{i,j\}^\{\(T\)\}/Non the generated pairs\.

###### Theorem 2\(Convergence of CEP\-BEG\)\.

AssumeFFis convex and that the CEP\-BEG gradients satisfy∑i=1N‖gi\(t\)‖∞2≤G2\\sum\_\{i=1\}^\{N\}\\\|g\_\{i\}^\{\(t\)\}\\\|\_\{\\infty\}^\{2\}\\leq G^\{2\}for allttwheregi\(t\)=\(gi,1\(t\),…,gi,m\(t\)\)g\_\{i\}^\{\(t\)\}=\(g\_\{i,1\}^\{\(t\)\},\\ldots,g\_\{i,m\}^\{\(t\)\}\)\. Letp¯\(T\)=1T​∑t=0T−1p\(t\)\\bar\{p\}^\{\(T\)\}=\\tfrac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}p^\{\(t\)\}\. Then, choosingη=2​N​log⁡m/\(G​T\)\\eta=\\sqrt\{2N\\log m\}/\(G\\sqrt\{T\}\), for any minimizerp⋆p^\{\\star\}of equation[2](https://arxiv.org/html/2609.02275#S5.E2)we have

F⁡\(p¯\(T\)\)−F⁡\(p⋆\)≤G​2​N​log⁡mT\.F\(\\bar\{p\}^\{\(T\)\}\)\-F\(p^\{\\star\}\)\\leq G\\sqrt\{\\frac\{2N\\log m\}\{T\}\}\.

The sketched features can be constructed in two complementary ways\. For finite\-dimensional embeddings, Gaussian random projection of the tensor\-product feature gives a Johnson–Lindenstrauss\-type guarantee on the generated sample set\. For shift\-invariant kernels, direct joint random Fourier features approximate the product kernel without forming tensor products\. With sketch dimensionrr, each CEP\-BEG iteration costsO⁡\(N​m​r2\+r3\)O\(Nmr^\{2\}\+r^\{3\}\)using a full eigendecomposition of ther×rr\\times rcovariance matrix\. The proofs, gradient and feature constructions, and concentration bounds are in Appendix[C](https://arxiv.org/html/2609.02275#A3)\.

## 6Numerical Results

### 6\.1Experimental Setup

Models and datasets\.Section[3](https://arxiv.org/html/2609.02275#S3)studies the conditional output\-range gap for three fully open language models: OLMo\[[8](https://arxiv.org/html/2609.02275#bib.bib41)\], Pythia\[[9](https://arxiv.org/html/2609.02275#bib.bib42)\], and GPT\-Neo\[[10](https://arxiv.org/html/2609.02275#bib.bib43)\]\. Specifically, OLMo is trained on Dolma\[[14](https://arxiv.org/html/2609.02275#bib.bib49)\], a large\-scale open text corpus designed to support transparent language\-model pretraining\. Pythia and GPT\-Neo are trained on The Pile\[[15](https://arxiv.org/html/2609.02275#bib.bib50)\], a diverse collection of English text from multiple domains, including web text, books, academic papers, code, and other curated sources\. Beyond language models, we evaluate two conditional image\-generation settings\. For class\-conditioned ImageNet generation, we follow the DGM\-Eval benchmark\[[16](https://arxiv.org/html/2609.02275#bib.bib44)\]and use generated samples from LDM\[[17](https://arxiv.org/html/2609.02275#bib.bib45)\], ADM\[[18](https://arxiv.org/html/2609.02275#bib.bib46)\], BigGAN\[[19](https://arxiv.org/html/2609.02275#bib.bib47)\], and DiT\-XL\-2\[[20](https://arxiv.org/html/2609.02275#bib.bib48)\], with ImageNet\[[21](https://arxiv.org/html/2609.02275#bib.bib54)\]training images as the real\-data reference\. For text\-conditioned MS\-COCO generation, we evaluate U\-ViT\[[22](https://arxiv.org/html/2609.02275#bib.bib51)\], SDXL\[[23](https://arxiv.org/html/2609.02275#bib.bib52)\], and PixArt\[[24](https://arxiv.org/html/2609.02275#bib.bib53)\], using MS\-COCO 2014\[[25](https://arxiv.org/html/2609.02275#bib.bib55)\]training set as the reference distribution\.

Tasks and evaluation protocol\.We first evaluate conditional diversity gaps across language and image generation tasks, and then study two applications of the proposed conditional diversity objective\. For language modeling, following Section[3](https://arxiv.org/html/2609.02275#S3), we compare training\-corpus continuations with model continuations generated from the same prefixes using greedy decoding, nucleus sampling withp=0\.9p=0\.9, and ancestral sampling\. For image generation, we compare real and generated images under matched conditions, including class labels on ImageNet and captions on MS\-COCO\. We then evaluate the proposed objective in two application settings\. For language models, we apply entropy\-projected reweighting to multiple candidate continuations generated for each prefix, while preserving the prefix marginal\. For text\-to\-image generation, we incorporate the conditional diversity objective into the diffusion sampling process as a sampling\-time guidance term, encouraging higher\-diversity outputs under the same text condition on MS\-COCO dataset\.

Feature representations and kernels\.We extract features using four fixed pretrained models: Qwen3\-Embedding\[[26](https://arxiv.org/html/2609.02275#bib.bib56)\], CLIP\[[27](https://arxiv.org/html/2609.02275#bib.bib57)\], T5\[[28](https://arxiv.org/html/2609.02275#bib.bib58)\], and DINOv2\[[29](https://arxiv.org/html/2609.02275#bib.bib59)\]\. Then we construct kernels on the resulting features and evaluate three kernel families: Gaussian, cosine, and degree\-3 polynomial kernels\. All kernels are normalized before computing conditional VNE and conditional RKE\.

### 6\.2Conditional Diversity Gaps in Language Models

Figure[1](https://arxiv.org/html/2609.02275#S6.F1)reports conditional VNE across sample sizes from55K to2020K for OLMo, Pythia, and GPT\-Neo\. We also observe that conditional VNE increases with sample size for both training and generated continuations\. However, the training\-data curve grows faster than the generated\-sample curves, so the measured diversity gap generally becomes larger as more samples are included\. The gap is not restricted to the approximately11B\-parameter models used in the main experiments: we additionally evaluated OLMo\-3\-7B, Pythia\-2\.8B, and Pythia\-6\.9B, and the conditional output\-range gap remains substantial across all three larger models; the complete sample\-size curves are reported in Appendix[E\.10](https://arxiv.org/html/2609.02275#A5.SS10)\. We also perform a human\-interpretable lexical audit counting distinct semantic categories in training versus generated continuations, which recovers the same ordering as conditional VNE; the details are reported in Appendix[E\.9](https://arxiv.org/html/2609.02275#A5.SS9)\.

### 6\.3Long\-Sequence Conditional Diversity Gaps

To test whether the gap persists beyond the short prefix–continuation lengths in Section[3](https://arxiv.org/html/2609.02275#S3), we extend the experiments to prefix lengths\{8,16,32\}\\\{8,16,32\\\}and continuation lengths\{16,32,64\}\\\{16,32,64\\\}tokens across GPT\-Neo, OLMo, and Pythia, representing each sequence as a concatenation of22\-token segment embeddings; the construction details and a chunk\-size ablation are reported in Appendix[E\.6](https://arxiv.org/html/2609.02275#A5.SS6)\. Table[2](https://arxiv.org/html/2609.02275#S6.T2)reports the exponentiated conditional VNE for the training reference and the three decoding strategies, along with the average gap\. The training data exhibit higher conditional entropy than every decoding strategy in all evaluated configurations, indicating that the conditional diversity shortfall is a robust phenomenon across sequence lengths\.

Table 2:Conditional diversity gaps for long prefix–continuation sequences\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling, using prefix lengths\{8,16,32\}\\\{8,16,32\\\}and continuation lengths\{16,32,64\}\\\{16,32,64\\\}\.ModelPrefixContinuationTrainingGreedyNucleusAncestralAvg\. GapGPT\-Neo881616456\.9456\.9366\.1366\.1414\.5414\.5418\.5418\.557\.257\.2883232511\.4511\.4394\.5394\.5452\.5452\.5458\.4458\.476\.376\.3886464558\.7558\.7416\.7416\.7485\.1485\.1494\.9494\.993\.193\.116161616482\.0482\.0438\.4438\.4456\.9456\.9460\.0460\.030\.230\.216163232520\.4520\.4460\.1460\.1483\.0483\.0488\.2488\.243\.343\.316166464558\.8558\.8479\.5479\.5504\.9504\.9512\.3512\.359\.959\.932321616513\.8513\.8494\.9494\.9502\.6502\.6505\.5505\.512\.812\.832323232536\.6536\.6507\.3507\.3517\.6517\.6519\.9519\.921\.721\.732326464563\.9563\.9519\.3519\.3530\.6530\.6535\.5535\.535\.435\.4OLMo881616462\.9462\.9410\.3410\.3433\.0433\.0439\.4439\.435\.335\.3883232520\.0520\.0456\.5456\.5484\.9484\.9489\.0489\.043\.243\.2886464575\.6575\.6502\.1502\.1534\.0534\.0541\.2541\.249\.849\.816161616487\.8487\.8456\.9456\.9470\.5470\.5474\.0474\.020\.720\.716163232530\.4530\.4488\.0488\.0506\.0506\.0510\.4510\.428\.928\.916166464576\.9576\.9521\.7521\.7545\.4545\.4549\.2549\.238\.138\.132321616524\.0524\.0506\.1506\.1513\.6513\.6516\.0516\.012\.112\.132323232551\.4551\.4524\.7524\.7534\.8534\.8538\.1538\.118\.918\.932326464585\.2585\.2546\.9546\.9561\.1561\.1564\.2564\.227\.827\.8Pythia881616463\.3463\.3374\.9374\.9423\.0423\.0429\.7429\.754\.154\.1883232519\.2519\.2415\.3415\.3468\.2468\.2475\.2475\.266\.366\.3886464568\.2568\.2452\.9452\.9506\.7506\.7509\.5509\.578\.578\.516161616488\.6488\.6449\.1449\.1465\.9465\.9469\.3469\.327\.227\.216163232527\.7527\.7480\.1480\.1496\.6496\.6499\.3499\.335\.735\.716166464567\.7567\.7510\.6510\.6519\.8519\.8523\.4523\.449\.849\.832321616521\.6521\.6503\.4503\.4511\.5511\.5513\.7513\.712\.112\.132323232545\.0545\.0519\.5519\.5528\.3528\.3530\.2530\.219\.019\.032326464572\.8572\.8536\.4536\.4539\.8539\.8542\.7542\.733\.233\.2

### 6\.4Conditional Diversity Gaps Beyond Language Models

Figure[2](https://arxiv.org/html/2609.02275#A5.F2)and Figure[3](https://arxiv.org/html/2609.02275#A5.F3)report conditional VNE across sample sizes from2\.52\.5K to2020K for class\-conditioned ImageNet generation and text\-conditioned MS\-COCO generation, respectively\. In both settings, the real\-data reference consistently achieves higher conditional VNE than the generated samples under matched conditions\. We also observe that the gap generally increases with sample size, as the real\-data conditional VNE grows faster than the generated\-sample conditional VNE\. This trend is consistent with the LLM experiments in Section[3](https://arxiv.org/html/2609.02275#S3)\.

![Refer to caption](https://arxiv.org/html/2609.02275v1/sample_size.png)

Figure 1:Conditional diversity gaps between training data and generated data in language models\.
### 6\.5Entropy\-Projected Reweighting for Conditional Generation

We next evaluate the proposed entropy\-projected reweighting method on language generation\. For each prefix, we generate 10 candidate continuations and reweight them using the conditional entropy projection in Section[5](https://arxiv.org/html/2609.02275#S5)\. Table[3](https://arxiv.org/html/2609.02275#S6.T3)compares the original uniformly weighted candidates, denoted as “w/o Projection”, with the reweighted candidates, denoted as “w/ Projection”\. These results indicate that the proposed projection can increase the measured conditional diversity of generated samples without retraining the model or generating new outputs\.

Table 3:Effect of entropy\-projected reweighting on conditional diversity\. We report exponential of conditional VNE and conditional RKE at sample sizes 10K and 20K, averaged over 5 runs\.ModelSettingSample Size = 10KSample Size = 20KExp\.Cond\.VNEExp\.Cond\.RKEExp\.Cond\.VNEExp\.Cond\.RKEOLMow/o Projection228\.17±2\.54\\pm 2\.54222\.55±4\.28\\pm 4\.28366\.35±2\.01\\pm 2\.01241\.14±4\.27\\pm 4\.27w/ Projection233\.88±1\.89\\pm 1\.89257\.25±6\.37\\pm 6\.37376\.77±1\.90\\pm 1\.90284\.92±5\.34\\pm 5\.34Pythiaw/o Projection230\.55±1\.46\\pm 1\.46217\.22±6\.02\\pm 6\.02371\.72±1\.63\\pm 1\.63236\.24±3\.98\\pm 3\.98w/ Projection236\.94±1\.84\\pm 1\.84254\.40±5\.69\\pm 5\.69382\.74±1\.38\\pm 1\.38285\.05±4\.11\\pm 4\.11GPT\-Neow/o Projection211\.18±1\.92\\pm 1\.92225\.51±4\.85\\pm 4\.85340\.90±1\.48\\pm 1\.48246\.13±4\.26\\pm 4\.26w/ Projection219\.08±0\.70\\pm 0\.70268\.23±6\.24\\pm 6\.24353\.68±2\.20\\pm 2\.20296\.02±3\.01\\pm 3\.01

We also compare the proposed projection with a temperature\-scaling baseline and conduct a qualitative probe of the reweighting behavior; both are reported in Appendices[E\.7](https://arxiv.org/html/2609.02275#A5.SS7)and[E\.8](https://arxiv.org/html/2609.02275#A5.SS8)\. The temperature comparison shows that matching the conditional entropy alone is insufficient, and the probe confirms that the entropy gain comes from de\-peaking valid modes rather than promoting low\-quality outputs\.

### 6\.6Conditional VNE Guidance for Diffusion Sampling

We further evaluate whether the conditional VNE score can be used not only as an evaluation metric, but also as a sampling\-time guidance signal for conditional generation\. We apply conditional VNE guidance to SDXL on MS\-COCO captions\. Detailed implementation is provided in Appendix[E\.4](https://arxiv.org/html/2609.02275#A5.SS4)\. Table[4](https://arxiv.org/html/2609.02275#S6.T4)reports the results\. SDXL with conditional VNE guidance achieves higher conditional VNE than SDXL without guidance across all evaluated sample sizes\.

Table 4:Text\-conditioned image generation on MS\-COCO using SDXL with and without the guidance\. We report exponential of conditional VNE as mean±\\pmstandard deviation over 5 independent runs\.Data / Model10K12\.5K15K17\.5K20KReference MS\-COCO28\.51±0\.0628\.51\\pm 0\.0631\.93±0\.0631\.93\\pm 0\.0635\.03±0\.0335\.03\\pm 0\.0337\.92±0\.0437\.92\\pm 0\.0440\.52±0\.0340\.52\\pm 0\.03SDXL w/o Guidance22\.51±0\.0922\.51\\pm 0\.0924\.76±0\.0724\.76\\pm 0\.0726\.76±0\.0726\.76\\pm 0\.0728\.58±0\.0428\.58\\pm 0\.0430\.25±0\.0430\.25\\pm 0\.04SDXL w/ Guidance23\.75±0\.0923\.75\\pm 0\.0926\.21±0\.0626\.21\\pm 0\.0628\.44±0\.0728\.44\\pm 0\.0730\.43±0\.0430\.43\\pm 0\.0432\.27±0\.0232\.27\\pm 0\.02

### 6\.7Downstream Application: Conditional VNE\-Weighted MBR Decoding

We next evaluate the proposed reweighting framework on a downstream task, model\-based minimum Bayes risk \(MBR\) decoding\[[30](https://arxiv.org/html/2609.02275#bib.bib66)\], where replacing the original candidate weights with the conditional entropy projection yields a conditional\-Vendi\-weighted variant \(CVS\-MBR\) that we compare with Monte Carlo MBR \(MC\-MBR\), model\-based MBR \(MBMBR\), and its length\-normalized variant \(MBMBR\-L\) on three summarization settings; implementation details are provided in Appendix[D\.3](https://arxiv.org/html/2609.02275#A4.SS3)\. Table[5](https://arxiv.org/html/2609.02275#S6.T5)reports conditional VNE together with ROUGE\-L, BERTScore, Distinct\-22, and Self\-BLEU\-22\. CVS\-MBR achieves the highest conditional VNE and Distinct\-22, as well as the lowest Self\-BLEU\-22, across all three settings, while a paired bootstrap shows no statistically significant degradation in ROUGE\-L or BERTScore relative to MBMBR\-L, demonstrating a diversity\-oriented quality–diversity trade\-off\.

Table 5:Conditional VNE\-weighted MBR decoding on three summarization settings\. We report conditional VNE, ROUGE\-L, BERTScore, Distinct\-22, and Self\-BLEU\-22\. Arrows indicate the desirable direction for each metric\. CVS\-MBR denotes the proposed conditional\-Vendi\-weighted variant\.SettingMethodCond\.VNE↑\\uparrowROUGE\-L↑\\uparrowBERTScore↑\\uparrowDistinct\-2↑\\uparrowSelf\-BLEU\-2↓\\downarrowXSumBART\-large\-XSumMC\-MBR27\.8827\.880\.3350\.3350\.9150\.9150\.9360\.9360\.2090\.209MBMBR4\.934\.930\.3460\.3460\.9180\.9180\.9290\.9290\.2120\.212MBMBR\-L26\.4426\.440\.3500\.3500\.9170\.9170\.9380\.9380\.2070\.207CVS\-MBR28\.3328\.330\.3310\.3310\.9150\.9150\.9450\.9450\.1970\.197CNN/DailyMailBART\-large\-CNNMC\-MBR33\.0033\.000\.2680\.2680\.8790\.8790\.9440\.9440\.1860\.186MBMBR3\.353\.350\.2730\.2730\.8800\.8800\.9450\.9450\.1840\.184MBMBR\-L28\.8028\.800\.2660\.2660\.8790\.8790\.9480\.9480\.1740\.174CVS\-MBR33\.3433\.340\.2720\.2720\.8790\.8790\.9510\.9510\.1650\.165CNN/DailyMailMistral\-7B\-InstructMC\-MBR12\.2012\.200\.2560\.2560\.8800\.8800\.9190\.9190\.2330\.233MBMBR4\.624\.620\.2540\.2540\.8810\.8810\.9240\.9240\.2290\.229MBMBR\-L13\.1513\.150\.2610\.2610\.8810\.8810\.9200\.9200\.2340\.234CVS\-MBR14\.0914\.090\.2610\.2610\.8810\.8810\.9260\.9260\.2240\.224

Ablation Studies\.To examine the robustness of the conditional diversity gaps, we conduct ablation studies under different experimental choices\. Specifically, we vary the evaluation sample size, the kernel function, and the embedding model\. For the language\-model experiments, we further vary the prefix–continuation construction by considering prefix token lengths in\{3,4,5\}\\\{3,4,5\\\}and continuation token lengths in\{2,3,4\}\\\{2,3,4\\\}\. For the long\-sequence experiments, we additionally perform a chunk\-size ablation over chunk sizes\{1,2,4\}\\\{1,2,4\\\}used in the segment concatenation representation\. For the sample\-size analysis, we draw subsamples from 2\.5k to 20k with different random seeds out of 30k samples\. Full results are provided in the Appendix\.

## 7Conclusion and Limitations

We studied whether LLMs and conditional generative models capture the full range of valid outputs present in their training distributions\. Using conditional entropy and its matrix\-based von Neumann analogue, we observed a consistent conditional output\-range gap across open language models and conditional image\-generation settings\. We specifically showed the concavity of kernel\-induced conditional von Neumann entropy, supporting both its use as a diagnostic and the convexity of entropy\-constrained projection\. Building on this structure, we proposed a post\-hoc reweighting method that preserves the input marginal while redistributing mass across generated candidates, with an efficient product\-simplex mirror\-descent implementation using sketched joint features\.

The proposed entropy\-based view complements standard evaluations of factuality, alignment, and semantic correctness\. Our direct training\-data comparison focuses on models for which the relevant training data are publicly available or reconstructable; for many widely used LLMs and generative AI models, the training data are not publicly accessible, making the analysis more difficult but potentially valuable\. These considerations suggest natural extensions to closed\-source systems and alternative feature choices\.

## References

- \[1\]\(2018\)Improving language understanding by generative pre\-training\.External Links:[Link](https://openai.com/)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p1.1)\.
- \[2\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p1.1)\.
- \[3\]R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer\(2022\)High\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10684–10695\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52688.2022.01042),[Link](https://openaccess.thecvf.com/content/CVPR2022/html/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.html)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p1.1)\.
- \[4\]J\. Li, D\. Li, S\. Savarese, and S\. C\. H\. Hoi\(2023\)BLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 19730–19742\.External Links:[Link](https://proceedings.mlr.press/v202/li23q.html)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p1.1)\.
- \[5\]J\. Hessel, A\. Holtzman, M\. Forbes, R\. L\. Bras, and Y\. Choi\(2021\)CLIPScore: a reference\-free evaluation metric for image captioning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 7514–7528\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.595),[Link](https://aclanthology.org/2021.emnlp-main.595/)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1),[§1](https://arxiv.org/html/2609.02275#S1.p2.1)\.
- \[6\]K\. Pillutla, S\. Swayamdipta, R\. Zellers, J\. Thickstun, S\. Welleck, Y\. Choi, and Z\. Harchaoui\(2021\)MAUVE: measuring the gap between neural text and human text using divergence frontiers\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 4816–4828\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1),[§1](https://arxiv.org/html/2609.02275#S1.p2.1)\.
- \[7\]J\. Koo, J\. Hernandez, M\. Haji\-Ali, Z\. Yang, and V\. Ordóñez\(2025\)Evaluating text\-to\-image synthesis with a conditional fréchet distance\.CoRRabs/2503\.21721\.External Links:[Link](https://arxiv.org/abs/2503.21721)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1),[§1](https://arxiv.org/html/2609.02275#S1.p2.1)\.
- \[8\]D\. Groeneveld, I\. Beltagy, E\. Walsh, A\. Bhagia, R\. Kinney, O\. Tafjord, A\. Jha, H\. Ivison, I\. Magnusson, Y\. Wang, S\. Arora, D\. Atkinson, R\. Authur, K\. Chandu, A\. Cohan, J\. Dumas, Y\. Elazar, Y\. Gu, J\. Hessel, T\. Khot, W\. Merrill, J\. Morrison, N\. Muennighoff, A\. Naik, C\. Nam, M\. Peters, V\. Pyatkin, A\. Ravichander, D\. Schwenk, S\. Shah, W\. Smith, E\. Strubell, N\. Subramani, M\. Wortsman, P\. Dasigi, N\. Lambert, K\. Richardson, L\. Zettlemoyer, J\. Dodge, K\. Lo, L\. Soldaini, N\. Smith, and H\. Hajishirzi\(2024\)OLMo: accelerating the science of language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15789–15809\.External Links:[Link](https://aclanthology.org/2024.acl-long.841/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.841)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p4.1),[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[9\]S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. Van Der Wal\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 2397–2430\.External Links:[Link](https://proceedings.mlr.press/v202/biderman23a.html)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p4.1),[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[10\]GPT\-Neo: Large scale autoregressive language modeling with meshtensorflowExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.5551208),[Link](https://doi.org/10.5281/zenodo.5551208)Cited by:[§1](https://arxiv.org/html/2609.02275#S1.p4.1),[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[11\]F\. Farnia, M\. Jalali, and A\. Ospanov\(2026\)Exposing diversity bias in deep generative models: statistical origins and correction of diversity error\.InICLR 2026 2nd Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy,External Links:[Link](https://openreview.net/forum?id=cYFrRU16nq)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1),[§1](https://arxiv.org/html/2609.02275#S1.p5.1)\.
- \[12\]L\. G\. S\. Giraldo, M\. Rao, and J\. C\. Principe\(2015\)Measures of entropy from data using infinitely divisible kernels\.IEEE Transactions on Information Theory61\(1\),pp\. 535–548\.External Links:[Document](https://dx.doi.org/10.1109/TIT.2014.2370058),[Link](https://doi.org/10.1109/TIT.2014.2370058)Cited by:[§2](https://arxiv.org/html/2609.02275#S2.p3.2),[§4](https://arxiv.org/html/2609.02275#S4.p1.1),[§4](https://arxiv.org/html/2609.02275#S4.p3.1),[Proposition 2](https://arxiv.org/html/2609.02275#Thmproposition2.p1.1.1)\.
- \[13\]M\. Jalali, A\. Ospanov, A\. Gohari, and F\. Farnia\(2026\)Conditional vendi score: prompt\-aware diversity evaluation for text\-guided generative ai models\.InProceedings of The 29th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research\. PMLR,pp\. 02–05\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1),[§2](https://arxiv.org/html/2609.02275#S2.p3.2),[§2](https://arxiv.org/html/2609.02275#S2.p3.3),[§2](https://arxiv.org/html/2609.02275#S2.p4.2)\.
- \[14\]L\. Soldaini, R\. Kinney, A\. Bhagia, D\. Schwenk, D\. Atkinson, R\. Authur, B\. Bogin, K\. Chandu, J\. Dumas, Y\. Elazar, V\. Hofmann, A\. H\. Jha, S\. Kumar, L\. Lucy, X\. Lyu, N\. Lambert, I\. Magnusson, J\. Morrison, N\. Muennighoff, A\. Naik, C\. Nam, M\. E\. Peters, A\. Ravichander, K\. Richardson, Z\. Shen, E\. Strubell, N\. Subramani, O\. Tafjord, P\. Walsh, L\. Zettlemoyer, N\. A\. Smith, H\. Hajishirzi, I\. Beltagy, D\. Groeneveld, J\. Dodge, and K\. Lo\(2024\)Dolma: an open corpus of three trillion tokens for language model pretraining research\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15725–15788\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[15\]L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy\(2020\)The Pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[16\]G\. Stein, J\. Cresswell, R\. Hosseinzadeh, Y\. Sui, B\. Ross, V\. Villecroze, Z\. Liu, A\. L\. Caterini, E\. Taylor, and G\. Loaiza\-Ganem\(2023\)Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[17\]R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer\(2022\)High\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10684–10695\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[18\]P\. Dhariwal and A\. Q\. Nichol\(2021\)Diffusion models beat gans on image synthesis\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 8780–8794\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[19\]A\. Brock, J\. Donahue, and K\. Simonyan\(2019\)Large scale gan training for high fidelity natural image synthesis\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[20\]W\. Peebles and S\. Xie\(2023\)Scalable diffusion models with transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 4195–4205\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[21\]J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei\(2009\)ImageNet: a large\-scale hierarchical image database\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 248–255\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2009.5206848)Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[22\]F\. Bao, S\. Nie, K\. Xue, Y\. Cao, C\. Li, H\. Su, and J\. Zhu\(2023\)All are worth words: a ViT backbone for diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22669–22679\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[23\]D\. Podell, Z\. English, K\. Lacey, A\. Blattmann, T\. Dockhorn, J\. Müller, J\. Penna, and R\. Rombach\(2024\)SDXL: improving latent diffusion models for high\-resolution image synthesis\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[24\]J\. Chen, C\. Ge, E\. Xie, Y\. Wu, L\. Yao, X\. Ren, Z\. Wang, P\. Luo, H\. Lu, and Z\. Li\(2024\)PIXART\-Σ\\Sigma: weak\-to\-strong training of diffusion transformer for 4k text\-to\-image generation\.InEuropean Conference on Computer Vision,pp\. 74–91\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[25\]T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. Zitnick\(2014\)Microsoft COCO: common objects in context\.InEuropean Conference on Computer Vision,pp\. 740–755\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p1.1)\.
- \[26\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou\(2025\)Qwen3 Embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p3.1)\.
- \[27\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p3.1)\.
- \[28\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p3.1)\.
- \[29\]M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jégou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. Bojanowski\(2024\)DINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research\.Cited by:[§6\.1](https://arxiv.org/html/2609.02275#S6.SS1.p3.1)\.
- \[30\]Y\. Jinnai, Y\. Yasui, K\. Koyama, H\. Shimizu, and T\. Nishikawa\(2024\)Model\-based minimum Bayes risk decoding for text generation\.InProceedings of the International Conference on Machine Learning,Cited by:[§D\.3](https://arxiv.org/html/2609.02275#A4.SS3.p5.1),[§6\.7](https://arxiv.org/html/2609.02275#S6.SS7.p1.1)\.
- \[31\]Y\. Hu, B\. Liu, J\. Kasai, Y\. Wang, M\. Ostendorf, R\. Krishna, and N\. A\. Smith\(2023\)TIFA: accurate and interpretable text\-to\-image faithfulness evaluation with question answering\.InProceedings of the IEEE/CVF International Conference on Computer Vision,External Links:[Document](https://dx.doi.org/10.1109/ICCV51070.2023.01866),[Link](https://openaccess.thecvf.com/content/ICCV2023/html/Hu_TIFA_Accurate_and_Interpretable_Text-to-Image_Faithfulness_Evaluation_with_Question_Answering_ICCV_2023_paper.html)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[32\]J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. Dong\(2023\)ImageReward: learning and evaluating human preferences for text\-to\-image generation\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/33646ef0ed554145eab65f6250fab0c9-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[33\]J\. Li, M\. Galley, C\. Brockett, J\. Gao, and B\. Dolan\(2016\)A diversity\-promoting objective function for neural conversation models\.InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 110–119\.External Links:[Document](https://dx.doi.org/10.18653/v1/N16-1014),[Link](https://aclanthology.org/N16-1014/)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[34\]Y\. Zhu, S\. Lu, L\. Zheng, J\. Guo, W\. Zhang, J\. Wang, and Y\. Yu\(2018\)Texygen: a benchmarking platform for text generation models\.InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval,pp\. 1097–1100\.External Links:[Document](https://dx.doi.org/10.1145/3209978.3210080),[Link](https://doi.org/10.1145/3209978.3210080)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[35\]K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu\(2002\)BLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.External Links:[Document](https://dx.doi.org/10.3115/1073083.1073135),[Link](https://aclanthology.org/P02-1040/)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[36\]R\. Vedantam, C\. L\. Zitnick, and D\. Parikh\(2015\)CIDEr: consensus\-based image description evaluation\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 4566–4575\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2015.7299087),[Link](https://openaccess.thecvf.com/content_cvpr_2015/html/Vedantam_CIDEr_Consensus-Based_Image_2015_CVPR_paper.html)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[37\]P\. Anderson, B\. Fernando, M\. Johnson, and S\. Gould\(2016\)SPICE: semantic propositional image caption evaluation\.InComputer Vision – ECCV 2016,Lecture Notes in Computer Science, Vol\.9909,pp\. 382–398\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-46454-1%5F24),[Link](https://link.springer.com/chapter/10.1007/978-3-319-46454-1_24)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[38\]T\. Unterthiner, S\. van Steenkiste, K\. Kurach, R\. Marinier, M\. Michalski, and S\. Gelly\(2019\)FVD: a new metric for video generation\.InICLR 2019 Workshop on Deep Generative Models for Highly Structured Data,External Links:[Link](https://openreview.net/forum?id=rylgEULtdN)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[39\]J\. Z\. Wu, G\. Fang, H\. Wu, X\. Wang, Y\. Ge, X\. Cun, D\. J\. Zhang, J\. Liu, Y\. Gu, R\. Zhao, W\. Lin, W\. Hsu, Y\. Shan, and M\. Z\. Shou\(2024\)Towards a better metric for text\-to\-video generation\.CoRRabs/2401\.07781\.External Links:[Link](https://arxiv.org/abs/2401.07781)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[40\]Y\. Liu, L\. Li, S\. Ren, R\. Gao, S\. Li, S\. Chen, X\. Sun, and L\. Hou\(2023\)FETV: a benchmark for fine\-grained evaluation of open\-domain text\-to\-video generation\.CoRRabs/2311\.01813\.External Links:[Link](https://arxiv.org/abs/2311.01813)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[41\]A\. Ospanov, M\. Jalali, and F\. Farnia\(2025\)Scendi score: prompt\-aware diversity evaluation via schur complement of clip embeddings\.In2025 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 16927–16937\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[42\]M\. Jalali, H\. Lei, A\. Gohari, and F\. Farnia\(2025\)SPARKE: scalable prompt\-aware diversity guidance in diffusion models via RKE score\.CoRRabs/2506\.10173\.External Links:[Link](https://arxiv.org/abs/2506.10173)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p1.1)\.
- \[43\]M\. Dombrowski, W\. Zhang, S\. Cechnicka, H\. Reynaud, and B\. Kainz\(2025\)Image generation diversity issues and how to tame them\.CoRRabs/2411\.16171\.External Links:[Link](https://arxiv.org/abs/2411.16171)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[44\]L\. Yun, C\. An, Z\. Wang, L\. Peng, and J\. Shang\(2025\)The price of format: diversity collapse in LLMs\.CoRRabs/2505\.18949\.External Links:[Link](https://arxiv.org/abs/2505.18949)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[45\]Q\. Wang and A\. B\. Chan\(2019\)Describing like humans: on diversity in image captioning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 4195–4203\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2019.00432)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[46\]N\. Carlini, F\. Tramèr, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, Ú\. Erlingsson, A\. Oprea, and C\. Raffel\(2021\)Extracting training data from large language models\.In30th USENIX Security Symposium \(USENIX Security 21\),pp\. 2633–2650\.External Links:[Link](https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[47\]N\. Carlini, J\. Hayes, M\. Nasr, M\. Jagielski, V\. Sehwag, F\. Tramèr, B\. Balle, D\. Ippolito, and E\. Wallace\(2023\)Extracting training data from diffusion models\.CoRRabs/2301\.13188\.External Links:[Link](https://arxiv.org/abs/2301.13188)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[48\]X\. Gu, Y\. Wang, Y\. Li, X\. Li, T\. Salimans, C\. Zhang, and Y\. Song\(2024\)On memorization in diffusion models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=9nT8ouPui8)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p2.1)\.
- \[49\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p3.1)\.
- \[50\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal\(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p3.1)\.
- \[51\]Z\. Li, C\. Chen, T\. Xu, Z\. Qin, J\. Xiao, Z\. Luo, and R\. Sun\(2025\)Preserving diversity in supervised fine\-tuning of large language models\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p3.1)\.
- \[52\]H\. Shen\(2026\)On entropy control in llm\-rl algorithms\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p3.1)\.
- \[53\]A\. Nikitin, J\. Kossen, Y\. Gal, and P\. Marttinen\(2024\)Kernel language entropy: fine\-grained uncertainty quantification for llms from semantic similarities\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[54\]D\. Friedman and A\. B\. Dieng\(2023\)The vendi score: a diversity evaluation metric for machine learning\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=g97OHbQyk1)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[55\]A\. Ospanov, J\. Zhang, M\. Jalali, X\. Cao, A\. Bogdanov, and F\. Farnia\(2024\)Towards a scalable reference\-free evaluation of generative models\.Advances in Neural Information Processing Systems37,pp\. 120892–120927\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[56\]M\. Jalali, C\. T\. Li, and F\. Farnia\(2023\)An information\-theoretic evaluation of generative models in learning multi\-modal distributions\.Advances in Neural Information Processing Systems36,pp\. 9931–9943\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[57\]L\. Wei, Z\. Tan, C\. Li, J\. Wang, and W\. Huang\(2024\)Diff\-erank: a novel rank\-based metric for evaluating large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[58\]M\. R\. Arefin, G\. Subbaraj, N\. Gontier, Y\. LeCun, I\. Rish, R\. Shwartz\-Ziv, and C\. Pal\(2025\)Seq\-vcr: preventing collapse in intermediate transformer representations for enhanced reasoning\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p4.1)\.
- \[59\]G\. Pereyra, G\. Tucker, J\. Chorowski, Ł\. Kaiser, and G\. Hinton\(2017\)Regularizing neural networks by penalizing confident output distributions\.CoRRabs/1701\.06548\.External Links:[Link](https://arxiv.org/abs/1701.06548)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p5.1)\.
- \[60\]C\. Szegedy, V\. Vanhoucke, S\. Ioffe, J\. Shlens, and Z\. Wojna\(2016\)Rethinking the inception architecture for computer vision\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2818–2826\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.308),[Link](https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p5.1)\.
- \[61\]J\. C\. Peterson, R\. M\. Battleday, T\. L\. Griffiths, and O\. Russakovsky\(2019\)Human uncertainty makes classification more robust\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 9616–9625\.External Links:[Document](https://dx.doi.org/10.1109/ICCV.2019.00971),[Link](https://openaccess.thecvf.com/content_ICCV_2019/html/Peterson_Human_Uncertainty_Makes_Classification_More_Robust_ICCV_2019_paper.html)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p5.1)\.
- \[62\]E\. T\. Anzaku, L\. Obongo, P\. Bogolin, and H\. Cholakkal\(2024\)Re\-assessing imagenet: how aligned is its single\-label assumption with its multi\-label nature?\.CoRRabs/2412\.18409\.External Links:[Link](https://arxiv.org/abs/2412.18409)Cited by:[Appendix A](https://arxiv.org/html/2609.02275#A1.p5.1)\.
- \[63\]E\. H\. Lieb and M\. B\. Ruskai\(1973\)Proof of the strong subadditivity of quantum\-mechanical entropy\.Physical Review Letters30\(10\),pp\. 434–436\.External Links:[Document](https://dx.doi.org/10.1103/PhysRevLett.30.434)Cited by:[§C\.2](https://arxiv.org/html/2609.02275#A3.SS2.p6.1.1)\.
- \[64\]A\. Katz, A\. Jaiswal, M\. Nesca, and E\. Milios\(2023\)Identifying risk factors associated with lower back pain in electronic medical record free text: deep learning approach using clinical note annotations\.JMIR Medical Informatics\.Cited by:[§D\.1](https://arxiv.org/html/2609.02275#A4.SS1.p4.1)\.
- \[65\]A\. Jaiswal and E\. Milios\(2023\)Breaking the token barrier: chunking and convolution for efficient long text classification with BERT\.arXiv preprint arXiv:2304\.11547\.Cited by:[§D\.1](https://arxiv.org/html/2609.02275#A4.SS1.p4.1)\.
- \[66\]M\. Günther, I\. Mohr, D\. J\. Williams, B\. Wang, and H\. Xiao\(2024\)Late chunking: contextual chunk embeddings using long\-context embedding models\.arXiv preprint arXiv:2409\.03753\.Cited by:[§D\.1](https://arxiv.org/html/2609.02275#A4.SS1.p4.1)\.
- \[67\]S\. Zhang, Y\. Liang, M\. Gong, D\. Jiang, and N\. Duan\(2022\)Multi\-view document representation learning for open\-domain dense retrieval\.InProceedings of the ACM Web Conference,Cited by:[§D\.1](https://arxiv.org/html/2609.02275#A4.SS1.p4.1)\.
- \[68\]K\. Santhanam, O\. Khattab, J\. Saad\-Falcon, C\. Potts, and M\. Zaharia\(2022\)ColBERTv2: effective and efficient retrieval via lightweight late interaction\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[§D\.1](https://arxiv.org/html/2609.02275#A4.SS1.p4.1)\.
- \[69\]T\. Le Bronnec, A\. Yang, J\. Renault, G\. Biau, and E\. Vincent\(2024\)Exploring precision and recall to assess the quality and diversity of LLMs\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,Cited by:[§E\.7](https://arxiv.org/html/2609.02275#A5.SS7.p2.1),[Table 8](https://arxiv.org/html/2609.02275#A5.T8),[Table 8](https://arxiv.org/html/2609.02275#A5.T8.4)\.

## Appendix ARelated Work

Evaluation of conditional generative AI\.Evaluation in conditional generation is shaped primarily by fidelity, alignment, and preference\-based criteria rather than by intrinsic variety under fixed conditions\. In text\-to\-image generation, representative examples include CLIPScore\[[5](https://arxiv.org/html/2609.02275#bib.bib8)\], TIFA\[[31](https://arxiv.org/html/2609.02275#bib.bib9)\], ImageReward\[[32](https://arxiv.org/html/2609.02275#bib.bib10)\], and more recent conditional fidelity metrics such as cFreD\[[7](https://arxiv.org/html/2609.02275#bib.bib11)\]\. In open\-ended text generation, MAUVE\[[6](https://arxiv.org/html/2609.02275#bib.bib12)\]has become a prominent distributional metric, while lexical diversity diagnostics such as distinct\-nn\[[33](https://arxiv.org/html/2609.02275#bib.bib13)\]and Self\-BLEU\[[34](https://arxiv.org/html/2609.02275#bib.bib14)\]remain common\. In image captioning, standard evaluation still centers on BLEU\[[35](https://arxiv.org/html/2609.02275#bib.bib15)\], CIDEr\[[36](https://arxiv.org/html/2609.02275#bib.bib16)\], and SPICE\[[37](https://arxiv.org/html/2609.02275#bib.bib17)\]\. In text\-to\-video generation, common practice combines video\-quality metrics such as FVD\[[38](https://arxiv.org/html/2609.02275#bib.bib18)\]with newer alignment\-oriented metrics such as T2VScore\[[39](https://arxiv.org/html/2609.02275#bib.bib19)\]and evaluation suites such as FETV\[[40](https://arxiv.org/html/2609.02275#bib.bib20)\]\. Also, the Scendi score\[[41](https://arxiv.org/html/2609.02275#bib.bib67)\]and Conditional Vendi/RKE scores\[[13](https://arxiv.org/html/2609.02275#bib.bib6),[42](https://arxiv.org/html/2609.02275#bib.bib25)\]provide conditional diversity scores for prompt\-aware diversity quantification\. These works have substantially advanced benchmarking for conditional generation, but they are not designed to isolate the variety that a model contributes under a fixed condition\. Our work complements this literature by focusing specifically on conditional variety as a first\-class evaluation target\.

Variety shortfall, novelty, and memorization in modern generative AI\.A growing empirical literature suggests that diversity shortfall is a real phenomenon in modern generative systems\. Specifically, Farnia et al\.\[[11](https://arxiv.org/html/2609.02275#bib.bib1)\]conduct a diversity gap analysis for unconditional generative models, numerically showing a von Neumann entropy gap between standard generative models and their training data, which we extend to the conditional generation case by analyzing the conditional von Neumann entropy function\. In image generation, Dombrowski et al\.\[[43](https://arxiv.org/html/2609.02275#bib.bib4)\]report that current state\-of\-the\-art models cover only a limited fraction of the diversity present in the training distribution\. In LLMs, Yun et al\.\[[44](https://arxiv.org/html/2609.02275#bib.bib5)\]identify*diversity collapse*, where formatting and instruction structure drive semantically similar outputs even under decoding regimes intended to encourage variety\. In image captioning, Wang and Chan\[[45](https://arxiv.org/html/2609.02275#bib.bib21)\]show that models optimized for standard captioning metrics often produce generic captions and remain far below human performance in diversity\. Related concerns arise in the memorization and novelty literature\. Carlini et al\.\[[46](https://arxiv.org/html/2609.02275#bib.bib22)\]demonstrate that large language models can regurgitate memorized training data, while Carlini et al\.\[[47](https://arxiv.org/html/2609.02275#bib.bib23)\]show analogous extraction phenomena for diffusion models\. More recent empirical studies also investigate memorization behavior in diffusion models directly\[[48](https://arxiv.org/html/2609.02275#bib.bib24)\]\. Our work differs in that it uses the conditional kernel entropy to analyze a shortfall relative to the empirical training distribution\.

Entropy\-based analysis of LLMs\.Entropy has recently become a useful tool for studying uncertainty, reliability, and training dynamics in LLMs\. Semantic entropy\[[49](https://arxiv.org/html/2609.02275#bib.bib31),[50](https://arxiv.org/html/2609.02275#bib.bib32)\]estimates uncertainty over meanings rather than surface forms and has been used for hallucination detection and reliability assessment\. Other work uses entropy directly in training or fine\-tuning: Li et al\.\[[51](https://arxiv.org/html/2609.02275#bib.bib36)\]connect diversity preservation in supervised fine\-tuning to entropy\-regularized objectives, while Shen\[[52](https://arxiv.org/html/2609.02275#bib.bib37)\]studies entropy control in LLM reinforcement learning\. These works use entropy mainly for uncertainty estimation, hallucination detection, or training\-time regularization\. Our focus is different: we compare the conditional entropy of model\-generated outputs against the corresponding training distribution and study the resulting conditional output\-range gap\.

Matrix\-entropy methods for LLMs\.Closely related to our use of matrix\-based entropy are recent works applying von Neumann or spectral entropy to LLM outputs and representations\. Nikitin et al\.\[[53](https://arxiv.org/html/2609.02275#bib.bib33)\]propose Kernel Language Entropy \(KLE\), a method for uncertainty quantification that builds positive semidefinite unit\-trace kernels over LLM outputs and measures semantic uncertainty using von Neumann entropy\. Vendi score\[[54](https://arxiv.org/html/2609.02275#bib.bib7),[55](https://arxiv.org/html/2609.02275#bib.bib68)\]uses von Neumann entropy of the kernel matrices for evaluating the variety of the generated data, which has been similarly formulated for order\-2 Rényi entropy as the RKE score\[[56](https://arxiv.org/html/2609.02275#bib.bib69)\]\. Wei et al\.\[[57](https://arxiv.org/html/2609.02275#bib.bib34)\]use matrix\-entropy\-inspired representation spectra for LLM evaluation, showing that rank/entropy\-based representation measures correlate with model scale and standard performance indicators\. Arefin et al\.\[[58](https://arxiv.org/html/2609.02275#bib.bib35)\]use matrix\-based entropy to diagnose representation collapse in intermediate Transformer layers and propose a regularizer to increase representation richness for reasoning\. These works demonstrate that spectral entropy is informative for LLM uncertainty, representation quality, and training behavior\. However, they primarily analyze unconditional output sets or internal representations\. On the other hand, our work studies*conditional*entropy and compares model outputs directly to their training data\.

Entropy regularization and predictive uncertainty\.Finally, our work is related in spirit to entropy\-promoting regularization in supervised learning\. Confidence\-penalty regularization\[[59](https://arxiv.org/html/2609.02275#bib.bib27)\]and label smoothing\[[60](https://arxiv.org/html/2609.02275#bib.bib28)\]both discourage overly sharp predictive distributions and have been shown to improve deep models across tasks\. More recent work has emphasized that real\-world supervised datasets may contain substantial ambiguity even when only a single hard label is observed, as reflected for example in human label distributions\[[61](https://arxiv.org/html/2609.02275#bib.bib29)\]and critiques of the single\-label assumption in ImageNet\-style benchmarks\[[62](https://arxiv.org/html/2609.02275#bib.bib30)\]\. Although these works are not about generative diversity evaluation, they share a common theme with our analysis: finite datasets can under\-represent the uncertainty or variability of the underlying task, and entropy\-promoting mechanisms can partially compensate for this mismatch\.

## Appendix BExtended Preliminaries

This appendix records the population\-level operator definitions corresponding to the empirical quantities in Section[2](https://arxiv.org/html/2609.02275#S2)\. Throughout,kXk\_\{X\}andkYk\_\{Y\}are normalized positive semidefinite kernels with feature maps

ϕX:𝒳→ℋX,ϕY:𝒴→ℋY,\\phi\_\{X\}:\\mathcal\{X\}\\to\\mathcal\{H\}\_\{X\},\\qquad\\phi\_\{Y\}:\\mathcal\{Y\}\\to\\mathcal\{H\}\_\{Y\},so thatkX​\(x,x′\)=⟨ϕX​\(x\),ϕX​\(x′\)⟩ℋXk\_\{X\}\(x,x^\{\\prime\}\)=\\langle\\phi\_\{X\}\(x\),\\phi\_\{X\}\(x^\{\\prime\}\)\\rangle\_\{\\mathcal\{H\}\_\{X\}\},kY​\(y,y′\)=⟨ϕY​\(y\),ϕY​\(y′\)⟩ℋYk\_\{Y\}\(y,y^\{\\prime\}\)=\\langle\\phi\_\{Y\}\(y\),\\phi\_\{Y\}\(y^\{\\prime\}\)\\rangle\_\{\\mathcal\{H\}\_\{Y\}\}, and

‖ϕX​\(x\)‖ℋX2=1,‖ϕY​\(y\)‖ℋY2=1\.\\\|\\phi\_\{X\}\(x\)\\\|\_\{\\mathcal\{H\}\_\{X\}\}^\{2\}=1,\\qquad\\\|\\phi\_\{Y\}\(y\)\\\|\_\{\\mathcal\{H\}\_\{Y\}\}^\{2\}=1\.The product kernel on paired samples is induced by the tensor\-product feature map

ψ⁡\(x,y\)=ϕX​\(x\)⊗ϕY​\(y\),\\psi\(x,y\)=\\phi\_\{X\}\(x\)\\otimes\\phi\_\{Y\}\(y\),since

⟨ψ⁡\(x,y\),ψ⁡\(x′,y′\)⟩ℋX⊗ℋY=kX​\(x,x′\)​kY​\(y,y′\)\.\\langle\\psi\(x,y\),\\psi\(x^\{\\prime\},y^\{\\prime\}\)\\rangle\_\{\\mathcal\{H\}\_\{X\}\\otimes\\mathcal\{H\}\_\{Y\}\}=k\_\{X\}\(x,x^\{\\prime\}\)k\_\{Y\}\(y,y^\{\\prime\}\)\.
For a joint distributionPX​YP\_\{XY\}, define the joint and marginal covariance operators

CX​Y​\(P\)=𝔼\(X,Y\)∼P​\[ψ⁡\(X,Y\)​ψ​\(X,Y\)∗\],CX​\(PX\)=𝔼X∼PX​\[ϕX​\(X\)​ϕX​\(X\)∗\]\.C\_\{XY\}\(P\)=\\mathbb\{E\}\_\{\(X,Y\)\\sim P\}\\bigl\[\\psi\(X,Y\)\\psi\(X,Y\)^\{\\ast\}\\bigr\],\\qquad C\_\{X\}\(P\_\{X\}\)=\\mathbb\{E\}\_\{X\\sim P\_\{X\}\}\\bigl\[\\phi\_\{X\}\(X\)\\phi\_\{X\}\(X\)^\{\\ast\}\\bigr\]\.The normalization of the kernels implies

Tr⁡\(CX​Y​\(P\)\)=1,Tr⁡\(CX​\(PX\)\)=1,\\operatorname\{Tr\}\(C\_\{XY\}\(P\)\)=1,\\qquad\\operatorname\{Tr\}\(C\_\{X\}\(P\_\{X\}\)\)=1,so these are trace\-one positive semidefinite operators whenever the expectations are well defined\. The population conditional von Neumann entropy is

ℋvN​\(Y∣X;P\):=HvN​\(CX​Y​\(P\)\)−HvN​\(CX​\(PX\)\)\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\mid X;P\):=H\_\{\\mathrm\{vN\}\}\(C\_\{XY\}\(P\)\)\-H\_\{\\mathrm\{vN\}\}\(C\_\{X\}\(P\_\{X\}\)\)\.This is the population operator analogue of the empirical quantity

HvN​\(1n​KX⊙KY\)−HvN​\(1n​KX\)\.H\_\{\\mathrm\{vN\}\}\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\odot K\_\{Y\}\\bigr\)\-H\_\{\\mathrm\{vN\}\}\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\bigr\)\.Indeed, for the empirical distributionP^n=1n​∑i=1nδ\(xi,yi\)\\widehat\{P\}\_\{n\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\delta\_\{\(x\_\{i\},y\_\{i\}\)\}, the nonzero eigenvalues of the empirical covariance operator

CX​Y​\(P^n\)=1n​∑i=1nψ⁡\(xi,yi\)​ψ​\(xi,yi\)∗C\_\{XY\}\(\\widehat\{P\}\_\{n\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\psi\(x\_\{i\},y\_\{i\}\)\\psi\(x\_\{i\},y\_\{i\}\)^\{\\ast\}coincide with the nonzero eigenvalues of1n​KX​Y\\frac\{1\}\{n\}K\_\{XY\}, whereKX​Y=KX⊙KYK\_\{XY\}=K\_\{X\}\\odot K\_\{Y\}\. Similarly, the nonzero eigenvalues ofCX​\(P^n,X\)C\_\{X\}\(\\widehat\{P\}\_\{n,X\}\)coincide with those of1n​KX\\frac\{1\}\{n\}K\_\{X\}\. Thus the Gram\-matrix formula in the main text is exactly the finite\-sample representation of the operator definition\.

The order\-22version is defined at the population level by

ℋ2​\(Y∣X;P\):=H2​\(CX​Y​\(P\)\)−H2​\(CX​\(PX\)\),H2​\(A\):=−log⁡Tr⁡\(A2\)\.\\mathcal\{H\}\_\{2\}\(Y\\mid X;P\):=H\_\{2\}\(C\_\{XY\}\(P\)\)\-H\_\{2\}\(C\_\{X\}\(P\_\{X\}\)\),\\qquad H\_\{2\}\(A\):=\-\\log\\operatorname\{Tr\}\(A^\{2\}\)\.For empirical samples, this becomes

H2​\(1n​KX​Y\)−H2​\(1n​KX\)=log⁡‖KX‖F2‖KX⊙KY‖F2\.H\_\{2\}\\bigl\(\\frac\{1\}\{n\}K\_\{XY\}\\bigr\)\-H\_\{2\}\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\bigr\)=\\log\\frac\{\\\|K\_\{X\}\\\|\_\{F\}^\{2\}\}\{\\\|K\_\{X\}\\odot K\_\{Y\}\\\|\_\{F\}^\{2\}\}\.The equality follows from

Tr⁡\(\(1n​KX\)2\)=1n2​‖KX‖F2,Tr⁡\(\(1n​KX​Y\)2\)=1n2​‖KX⊙KY‖F2\.\\operatorname\{Tr\}\\bigl\(\\bigl\(\\frac\{1\}\{n\}K\_\{X\}\\bigr\)^\{2\}\\bigr\)=\\frac\{1\}\{n^\{2\}\}\\\|K\_\{X\}\\\|\_\{F\}^\{2\},\\qquad\\operatorname\{Tr\}\\bigl\(\\bigl\(\\frac\{1\}\{n\}K\_\{XY\}\\bigr\)^\{2\}\\bigr\)=\\frac\{1\}\{n^\{2\}\}\\\|K\_\{X\}\\odot K\_\{Y\}\\\|\_\{F\}^\{2\}\.
The normalized\-kernel assumption is used throughout to ensure that the covariance operators have unit trace and hence can be interpreted as density operators for von Neumann entropy\. If a kernelkksatisfiesk⁡\(z,z\)\>0k\(z,z\)\>0, the normalized kernel

k~​\(z,z′\)=k⁡\(z,z′\)k⁡\(z,z\)​k​\(z′,z′\)\\widetilde\{k\}\(z,z^\{\\prime\}\)=\\frac\{k\(z,z^\{\\prime\}\)\}\{\\sqrt\{k\(z,z\)k\(z^\{\\prime\},z^\{\\prime\}\)\}\}is positive semidefinite and satisfiesk~​\(z,z\)=1\\widetilde\{k\}\(z,z\)=1\. In infinite\-dimensional settings, the definitions above are understood for trace\-class covariance operators with finite entropy; all empirical Gram\-matrix quantities used in the experiments are finite\-dimensional and therefore well defined\.

## Appendix CProofs and Additional Theoretical Results

### C\.1Proof of Proposition[1](https://arxiv.org/html/2609.02275#Thmproposition1)

###### Proof\.

LetPX​Y\(0\)P^\{\(0\)\}\_\{XY\}andPX​Y\(1\)P^\{\(1\)\}\_\{XY\}be two joint distributions on finite alphabets𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}, and let

PX​Yλ=\(1−λ\)​PX​Y\(0\)\+λ​PX​Y\(1\),λ∈\[0,1\]\.P^\{\\lambda\}\_\{XY\}=\(1\-\\lambda\)P^\{\(0\)\}\_\{XY\}\+\\lambda P^\{\(1\)\}\_\{XY\},\\qquad\\lambda\\in\[0,1\]\.We use the convention0​log⁡0=00\\log 0=0\. For any joint lawPX​YP\_\{XY\},

H\(Y\|X;P\)=−∑x,yPX​Y\(x,y\)logPX​Y​\(x,y\)PX​\(x\)\.H\(Y\\,\|\\,X;P\)=\-\\sum\_\{x,y\}P\_\{XY\}\(x,y\)\\log\\frac\{P\_\{XY\}\(x,y\)\}\{P\_\{X\}\(x\)\}\.Fixx∈𝒳x\\in\\mathcal\{X\}\. Applying the log\-sum inequality to the two collections

ay=\(1−λ\)​PX​Y\(0\)​\(x,y\),by=λ​PX​Y\(1\)​\(x,y\)a\_\{y\}=\(1\-\\lambda\)P^\{\(0\)\}\_\{XY\}\(x,y\),\\qquad b\_\{y\}=\\lambda P^\{\(1\)\}\_\{XY\}\(x,y\)gives

∑yPX​Yλ​\(x,y\)​log⁡PX​Yλ​\(x,y\)PXλ​\(x\)≤\(1−λ\)​∑yPX​Y\(0\)​\(x,y\)​log⁡PX​Y\(0\)​\(x,y\)PX\(0\)​\(x\)\+λ​∑yPX​Y\(1\)​\(x,y\)​log⁡PX​Y\(1\)​\(x,y\)PX\(1\)​\(x\)\.\\sum\_\{y\}P^\{\\lambda\}\_\{XY\}\(x,y\)\\log\\frac\{P^\{\\lambda\}\_\{XY\}\(x,y\)\}\{P^\{\\lambda\}\_\{X\}\(x\)\}\\leq\(1\-\\lambda\)\\sum\_\{y\}P^\{\(0\)\}\_\{XY\}\(x,y\)\\log\\frac\{P^\{\(0\)\}\_\{XY\}\(x,y\)\}\{P^\{\(0\)\}\_\{X\}\(x\)\}\+\\lambda\\sum\_\{y\}P^\{\(1\)\}\_\{XY\}\(x,y\)\\log\\frac\{P^\{\(1\)\}\_\{XY\}\(x,y\)\}\{P^\{\(1\)\}\_\{X\}\(x\)\}\.Multiplying by−1\-1and summing overxxyields

H⁡\(Y\|X;Pλ\)≥\(1−λ\)​H​\(Y\|X;P\(0\)\)\+λ​H​\(Y\|X;P\(1\)\),H\(Y\\,\|\\,X;P^\{\\lambda\}\)\\geq\(1\-\\lambda\)H\(Y\\,\|\\,X;P^\{\(0\)\}\)\+\\lambda H\(Y\\,\|\\,X;P^\{\(1\)\}\),which proves concavity\. ∎

### C\.2Proof of Proposition[2](https://arxiv.org/html/2609.02275#Thmproposition2)

We first make explicit the operator representation used in the proof\. LetkXk\_\{X\}andkYk\_\{Y\}be normalized positive semidefinite kernels with feature maps

ϕX:𝒳→ℋX,ϕY:𝒴→ℋY,\\phi\_\{X\}:\\mathcal\{X\}\\to\\mathcal\{H\}\_\{X\},\\qquad\\phi\_\{Y\}:\\mathcal\{Y\}\\to\\mathcal\{H\}\_\{Y\},satisfying

‖ϕX​\(x\)‖ℋX2=kX​\(x,x\)=1,‖ϕY​\(y\)‖ℋY2=kY​\(y,y\)=1\.\\\|\\phi\_\{X\}\(x\)\\\|\_\{\\mathcal\{H\}\_\{X\}\}^\{2\}=k\_\{X\}\(x,x\)=1,\\qquad\\\|\\phi\_\{Y\}\(y\)\\\|\_\{\\mathcal\{H\}\_\{Y\}\}^\{2\}=k\_\{Y\}\(y,y\)=1\.For a joint distributionPX​YP\_\{XY\}, define

CX​Y​\(P\)=𝔼\(X,Y\)∼P​\[\(ϕX​\(X\)⊗ϕY​\(Y\)\)​\(ϕX​\(X\)⊗ϕY​\(Y\)\)∗\],C\_\{XY\}\(P\)=\\mathbb\{E\}\_\{\(X,Y\)\\sim P\}\\Bigl\[\(\\phi\_\{X\}\(X\)\\otimes\\phi\_\{Y\}\(Y\)\)\(\\phi\_\{X\}\(X\)\\otimes\\phi\_\{Y\}\(Y\)\)^\{\\ast\}\\Bigr\],and

CX​\(PX\)=𝔼X∼PX​\[ϕX​\(X\)​ϕX​\(X\)∗\]\.C\_\{X\}\(P\_\{X\}\)=\\mathbb\{E\}\_\{X\\sim P\_\{X\}\}\\Bigl\[\\phi\_\{X\}\(X\)\\phi\_\{X\}\(X\)^\{\\ast\}\\Bigr\]\.The normalized\-kernel assumption gives

Tr⁡\(CX​Y​\(P\)\)=1,Tr⁡\(CX​\(PX\)\)=1\.\\operatorname\{Tr\}\(C\_\{XY\}\(P\)\)=1,\\qquad\\operatorname\{Tr\}\(C\_\{X\}\(P\_\{X\}\)\)=1\.Thus, these covariance operators are density operators whenever the relevant entropies are finite\. The conditional von Neumann entropy used in the paper is

ℋvN​\(Y\|X;P\)=HvN​\(CX​Y​\(P\)\)−HvN​\(CX​\(PX\)\)\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P\)=H\_\{\\mathrm\{vN\}\}\(C\_\{XY\}\(P\)\)\-H\_\{\\mathrm\{vN\}\}\(C\_\{X\}\(P\_\{X\}\)\)\.For empirical Gram matrices and finite\-dimensional sketched features, all operators below are finite\-dimensional\. For general kernels, the same argument applies under the standard trace\-class and finite\-entropy conditions for the corresponding density operators\.

###### Proof\.

Let

ρX​Y​\(P\):=CX​Y​\(P\)\.\\rho\_\{XY\}\(P\):=C\_\{XY\}\(P\)\.We first identify the correct marginal operator\. For a rank\-one tensor term,

\(ϕX​\(x\)⊗ϕY​\(y\)\)​\(ϕX​\(x\)⊗ϕY​\(y\)\)∗,\(\\phi\_\{X\}\(x\)\\otimes\\phi\_\{Y\}\(y\)\)\(\\phi\_\{X\}\(x\)\\otimes\\phi\_\{Y\}\(y\)\)^\{\\ast\},taking the partial trace overℋY\\mathcal\{H\}\_\{Y\}gives

TrY⁡\[\(ϕX​\(x\)⊗ϕY​\(y\)\)​\(ϕX​\(x\)⊗ϕY​\(y\)\)∗\]=‖ϕY​\(y\)‖ℋY2​ϕX​\(x\)​ϕX​\(x\)∗\.\\operatorname\{Tr\}\_\{Y\}\\Bigl\[\(\\phi\_\{X\}\(x\)\\otimes\\phi\_\{Y\}\(y\)\)\(\\phi\_\{X\}\(x\)\\otimes\\phi\_\{Y\}\(y\)\)^\{\\ast\}\\Bigr\]=\\\|\\phi\_\{Y\}\(y\)\\\|\_\{\\mathcal\{H\}\_\{Y\}\}^\{2\}\\phi\_\{X\}\(x\)\\phi\_\{X\}\(x\)^\{\\ast\}\.SincekY​\(y,y\)=1k\_\{Y\}\(y,y\)=1, this equalsϕX​\(x\)​ϕX​\(x\)∗\\phi\_\{X\}\(x\)\\phi\_\{X\}\(x\)^\{\\ast\}\. Taking expectation over\(X,Y\)∼P\(X,Y\)\\sim Pyields

TrY⁡ρX​Y​\(P\)=CX​\(PX\)\.\\operatorname\{Tr\}\_\{Y\}\\rho\_\{XY\}\(P\)=C\_\{X\}\(P\_\{X\}\)\.Therefore, we have

ℋvN​\(Y\|X;P\)=HvN​\(ρX​Y​\(P\)\)−HvN​\(TrY⁡ρX​Y​\(P\)\)\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P\)=H\_\{\\mathrm\{vN\}\}\(\\rho\_\{XY\}\(P\)\)\-H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho\_\{XY\}\(P\)\)\.
Now take two joint distributionsPX​Y\(0\)P^\{\(0\)\}\_\{XY\}andPX​Y\(1\)P^\{\(1\)\}\_\{XY\}, and define

PX​Yλ=\(1−λ\)​PX​Y\(0\)\+λ​PX​Y\(1\),λ∈\[0,1\]\.P^\{\\lambda\}\_\{XY\}=\(1\-\\lambda\)P^\{\(0\)\}\_\{XY\}\+\\lambda P^\{\(1\)\}\_\{XY\},\\qquad\\lambda\\in\[0,1\]\.The covariance map is affine, so

ρX​Y​\(Pλ\)=\(1−λ\)​ρX​Y​\(P\(0\)\)\+λ​ρX​Y​\(P\(1\)\)\.\\rho\_\{XY\}\(P^\{\\lambda\}\)=\(1\-\\lambda\)\\rho\_\{XY\}\(P^\{\(0\)\}\)\+\\lambda\\rho\_\{XY\}\(P^\{\(1\)\}\)\.We set the following:

ρX​Y\(0\):=ρX​Y​\(P\(0\)\),ρX​Y\(1\):=ρX​Y​\(P\(1\)\),ρX​Yλ:=\(1−λ\)​ρX​Y\(0\)\+λ​ρX​Y\(1\)\.\\rho^\{\(0\)\}\_\{XY\}:=\\rho\_\{XY\}\(P^\{\(0\)\}\),\\qquad\\rho^\{\(1\)\}\_\{XY\}:=\\rho\_\{XY\}\(P^\{\(1\)\}\),\\qquad\\rho^\{\\lambda\}\_\{XY\}:=\(1\-\\lambda\)\\rho^\{\(0\)\}\_\{XY\}\+\\lambda\\rho^\{\(1\)\}\_\{XY\}\.
We consider an auxiliary two\-dimensional classical registerRRwith orthonormal basis vectorse0,e1e\_\{0\},e\_\{1\}, and define the block\-diagonal density operator

ΩR​X​Y=\(1−λ\)​\(e0​e0∗\)⊗ρX​Y\(0\)\+λ⁡\(e1​e1∗\)⊗ρX​Y\(1\)\.\\Omega\_\{RXY\}=\(1\-\\lambda\)\(e\_\{0\}e\_\{0\}^\{\\ast\}\)\\otimes\\rho^\{\(0\)\}\_\{XY\}\+\\lambda\(e\_\{1\}e\_\{1\}^\{\\ast\}\)\\otimes\\rho^\{\(1\)\}\_\{XY\}\.ItsX​YXYmarginal isΩX​Y=ρX​Yλ\\Omega\_\{XY\}=\\rho^\{\\lambda\}\_\{XY\}\. Hence,

HvN​\(Y\|X\)Ω=HvN​\(ρX​Yλ\)−HvN​\(TrY⁡ρX​Yλ\)=ℋvN​\(Y\|X;Pλ\)\.H\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X\)\_\{\\Omega\}=H\_\{\\mathrm\{vN\}\}\(\\rho^\{\\lambda\}\_\{XY\}\)\-H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho^\{\\lambda\}\_\{XY\}\)=\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\\lambda\}\)\.
We next compute the conditional entropy after additionally conditioning on the classical registerRR\. SinceΩR​X​Y\\Omega\_\{RXY\}is block diagonal,

HvN​\(ΩR​X​Y\)=h⁡\(λ\)\+\(1−λ\)​HvN​\(ρX​Y\(0\)\)\+λ​HvN​\(ρX​Y\(1\)\),H\_\{\\mathrm\{vN\}\}\(\\Omega\_\{RXY\}\)=h\(\\lambda\)\+\(1\-\\lambda\)H\_\{\\mathrm\{vN\}\}\(\\rho^\{\(0\)\}\_\{XY\}\)\+\\lambda H\_\{\\mathrm\{vN\}\}\(\\rho^\{\(1\)\}\_\{XY\}\),where

h⁡\(λ\)=−\(1−λ\)​log⁡\(1−λ\)−λ​log⁡λ\.h\(\\lambda\)=\-\(1\-\\lambda\)\\log\(1\-\\lambda\)\-\\lambda\\log\\lambda\.Similarly,

ΩR​X=\(1−λ\)​\(e0​e0∗\)⊗TrY⁡ρX​Y\(0\)\+λ⁡\(e1​e1∗\)⊗TrY⁡ρX​Y\(1\),\\Omega\_\{RX\}=\(1\-\\lambda\)\(e\_\{0\}e\_\{0\}^\{\\ast\}\)\\otimes\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(0\)\}\_\{XY\}\+\\lambda\(e\_\{1\}e\_\{1\}^\{\\ast\}\)\\otimes\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(1\)\}\_\{XY\},and therefore

HvN​\(ΩR​X\)=h⁡\(λ\)\+\(1−λ\)​HvN​\(TrY⁡ρX​Y\(0\)\)\+λ​HvN​\(TrY⁡ρX​Y\(1\)\)\.H\_\{\\mathrm\{vN\}\}\(\\Omega\_\{RX\}\)=h\(\\lambda\)\+\(1\-\\lambda\)H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(0\)\}\_\{XY\}\)\+\\lambda H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(1\)\}\_\{XY\}\)\.Subtracting the last two equations results in

HvN​\(Y\|X​R\)Ω=\(1−λ\)​\[HvN​\(ρX​Y\(0\)\)−HvN​\(TrY⁡ρX​Y\(0\)\)\]\+λ⁡\[HvN​\(ρX​Y\(1\)\)−HvN​\(TrY⁡ρX​Y\(1\)\)\]\.H\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,XR\)\_\{\\Omega\}=\(1\-\\lambda\)\\Bigl\[H\_\{\\mathrm\{vN\}\}\(\\rho^\{\(0\)\}\_\{XY\}\)\-H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(0\)\}\_\{XY\}\)\\Bigr\]\+\\lambda\\Bigl\[H\_\{\\mathrm\{vN\}\}\(\\rho^\{\(1\)\}\_\{XY\}\)\-H\_\{\\mathrm\{vN\}\}\(\\operatorname\{Tr\}\_\{Y\}\\rho^\{\(1\)\}\_\{XY\}\)\\Bigr\]\.Equivalently,

HvN​\(Y\|X​R\)Ω=\(1−λ\)​ℋvN​\(Y\|X;P\(0\)\)\+λ​ℋvN​\(Y\|X;P\(1\)\)\.H\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,XR\)\_\{\\Omega\}=\(1\-\\lambda\)\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\(0\)\}\)\+\\lambda\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\(1\)\}\)\.
Using the strong subadditivity of von Neumann entropy\[[63](https://arxiv.org/html/2609.02275#bib.bib38)\]implies the monotonicity of quantum conditional entropy under additional conditioning:

HvN​\(Y\|X\)Ω≥HvN​\(Y\|X​R\)Ω\.H\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X\)\_\{\\Omega\}\\geq H\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,XR\)\_\{\\Omega\}\.Combining the identities above gives

ℋvN​\(Y\|X;Pλ\)≥\(1−λ\)​ℋvN​\(Y\|X;P\(0\)\)\+λ​ℋvN​\(Y\|X;P\(1\)\),\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\\lambda\}\)\\geq\(1\-\\lambda\)\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\(0\)\}\)\+\\lambda\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P^\{\(1\)\}\),which finishes the concavity proof for the kernel\-induced matrix\-based conditional entropy\. ∎

### C\.3Additional Theoretical Results on Finite\-Sample Conditional Entropy

###### Corollary 1\(Finite\-sample underestimation and monotonicity for Shannon conditional entropy\)\.

LetP^n\\widehat\{P\}\_\{n\}be the empirical joint distribution ofnni\.i\.d\. samples from a distributionPX​YP\_\{XY\}on finite alphabets\. Then

𝔼⁡\[H⁡\(Y\|X;P^n\)\]≤H⁡\(Y\|X;P\)\.\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\leq H\(Y\\,\|\\,X;P\)\.Moreover, for all integersm≤nm\\leq n,

𝔼⁡\[H⁡\(Y\|X;P^m\)\]≤𝔼⁡\[H⁡\(Y\|X;P^n\)\]\.\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{m\}\)\]\\leq\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\.

###### Proof\.

The first inequality follows from Jensen’s inequality and Proposition[1](https://arxiv.org/html/2609.02275#Thmproposition1):

𝔼⁡\[H⁡\(Y\|X;P^n\)\]≤H⁡\(Y\|X;𝔼⁡\[P^n\]\)=H⁡\(Y\|X;P\)\.\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\leq H\(Y\\,\|\\,X;\\mathbb\{E\}\[\\widehat\{P\}\_\{n\}\]\)=H\(Y\\,\|\\,X;P\)\.
For monotonicity, letZ1,…,ZnZ\_\{1\},\\ldots,Z\_\{n\}be the paired samples, whereZi=\(Xi,Yi\)Z\_\{i\}=\(X\_\{i\},Y\_\{i\}\)\. Conditional onZ1,…,ZnZ\_\{1\},\\ldots,Z\_\{n\}, draw a subsetS⊆\{1,…,n\}S\\subseteq\\\{1,\\ldots,n\\\}of sizemmuniformly without replacement, and letP^S\\widehat\{P\}\_\{S\}be the empirical distribution of the selected samples\. Then

𝔼\[P^S\|Z1,…,Zn\]=P^n\.\\mathbb\{E\}\[\\widehat\{P\}\_\{S\}\\,\|\\,Z\_\{1\},\\ldots,Z\_\{n\}\]=\\widehat\{P\}\_\{n\}\.The concavity results in the following:

H\(Y\|X;P^n\)=H\(Y\|X;𝔼\[P^S\|Z1,…,Zn\]\)≥𝔼\[H\(Y\|X;P^S\)\|Z1,…,Zn\]\.H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)=H\\Bigl\(Y\\,\|\\,X;\\mathbb\{E\}\[\\widehat\{P\}\_\{S\}\\,\|\\,Z\_\{1\},\\ldots,Z\_\{n\}\]\\Bigr\)\\geq\\mathbb\{E\}\\Bigl\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{S\}\)\\,\|\\,Z\_\{1\},\\ldots,Z\_\{n\}\\Bigr\]\.Taking expectations gives

𝔼⁡\[H⁡\(Y\|X;P^n\)\]≥𝔼⁡\[H⁡\(Y\|X;P^S\)\]\.\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\geq\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{S\}\)\]\.The unconditional law ofP^S\\widehat\{P\}\_\{S\}is the same as the empirical distribution ofmmi\.i\.d\. samples fromPX​YP\_\{XY\}\. Therefore

𝔼⁡\[H⁡\(Y\|X;P^n\)\]≥𝔼⁡\[H⁡\(Y\|X;P^m\)\]\.\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\geq\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{m\}\)\]\.∎

###### Theorem 3\(First\-order bias of empirical Shannon conditional entropy\)\.

AssumeXXandYYtake values in finite alphabets, and define

𝒮X​Y=\{\(x,y\):PX​Y​\(x,y\)\>0\},𝒮X=\{x:PX​\(x\)\>0\}\.\\mathcal\{S\}\_\{XY\}=\\\{\(x,y\):P\_\{XY\}\(x,y\)\>0\\\},\\qquad\\mathcal\{S\}\_\{X\}=\\\{x:P\_\{X\}\(x\)\>0\\\}\.Using natural logarithms,

H⁡\(Y\|X;P\)−𝔼⁡\[H⁡\(Y\|X;P^n\)\]=\|𝒮X​Y\|−\|𝒮X\|2​n\+O⁡\(1n2\)\.H\(Y\\,\|\\,X;P\)\-\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]=\\frac\{\|\\mathcal\{S\}\_\{XY\}\|\-\|\\mathcal\{S\}\_\{X\}\|\}\{2n\}\+O\\Bigl\(\\frac\{1\}\{n^\{2\}\}\\Bigr\)\.

###### Proof\.

For a discrete distributionPPwith support sizessand strictly positive probabilities on its support, the Miller–Basharin expansion gives

H⁡\(P\)−𝔼⁡\[H⁡\(P^n\)\]=s−12​n\+O⁡\(1n2\)\.H\(P\)\-\\mathbb\{E\}\[H\(\\widehat\{P\}\_\{n\}\)\]=\\frac\{s\-1\}\{2n\}\+O\\Bigl\(\\frac\{1\}\{n^\{2\}\}\\Bigr\)\.Applying this expansion toPX​YP\_\{XY\}gives

H⁡\(X,Y,P\)−𝔼⁡\[H⁡\(X,Y,P^n\)\]=\|𝒮X​Y\|−12​n\+O⁡\(1n2\)\.H\(X,Y;P\)\-\\mathbb\{E\}\[H\(X,Y;\\widehat\{P\}\_\{n\}\)\]=\\frac\{\|\\mathcal\{S\}\_\{XY\}\|\-1\}\{2n\}\+O\\Bigl\(\\frac\{1\}\{n^\{2\}\}\\Bigr\)\.Applying the same expansion to the marginalPXP\_\{X\}gives

H⁡\(X,PX\)−𝔼⁡\[H⁡\(X,P^n,X\)\]=\|𝒮X\|−12​n\+O⁡\(1n2\)\.H\(X;P\_\{X\}\)\-\\mathbb\{E\}\[H\(X;\\widehat\{P\}\_\{n,X\}\)\]=\\frac\{\|\\mathcal\{S\}\_\{X\}\|\-1\}\{2n\}\+O\\Bigl\(\\frac\{1\}\{n^\{2\}\}\\Bigr\)\.Since

H⁡\(Y\|X;P\)=H⁡\(X,Y,P\)−H⁡\(X,PX\)H\(Y\\,\|\\,X;P\)=H\(X,Y;P\)\-H\(X;P\_\{X\}\)and

H⁡\(Y\|X;P^n\)=H⁡\(X,Y,P^n\)−H⁡\(X,P^n,X\),H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)=H\(X,Y;\\widehat\{P\}\_\{n\}\)\-H\(X;\\widehat\{P\}\_\{n,X\}\),subtracting the marginal expansion from the joint expansion yields

H⁡\(Y\|X;P\)−𝔼⁡\[H⁡\(Y\|X;P^n\)\]=\|𝒮X​Y\|−\|𝒮X\|2​n\+O⁡\(1n2\)\.H\(Y\\,\|\\,X;P\)\-\\mathbb\{E\}\[H\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]=\\frac\{\|\\mathcal\{S\}\_\{XY\}\|\-\|\\mathcal\{S\}\_\{X\}\|\}\{2n\}\+O\\Bigl\(\\frac\{1\}\{n^\{2\}\}\\Bigr\)\.∎

###### Corollary 2\(Finite\-sample underestimation and monotonicity for conditional von Neumann entropy\)\.

LetP^n\\widehat\{P\}\_\{n\}be the empirical distribution ofnni\.i\.d\. paired samples fromPX​YP\_\{XY\}\. Assume the relevant conditional von Neumann entropies are finite\. Then

𝔼⁡\[ℋvN​\(Y\|X;P^n\)\]≤ℋvN​\(Y\|X;P\)\.\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\leq\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P\)\.Moreover, for all integersm≤nm\\leq n,

𝔼⁡\[ℋvN​\(Y\|X;P^m\)\]≤𝔼⁡\[ℋvN​\(Y\|X;P^n\)\]\.\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{m\}\)\]\\leq\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\.

###### Proof\.

The first inequality follows from Jensen’s inequality and Proposition[2](https://arxiv.org/html/2609.02275#Thmproposition2):

𝔼⁡\[ℋvN​\(Y\|X;P^n\)\]≤ℋvN​\(Y\|X;𝔼⁡\[P^n\]\)=ℋvN​\(Y\|X;P\)\.\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\]\\leq\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\mathbb\{E\}\[\\widehat\{P\}\_\{n\}\]\)=\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;P\)\.
For monotonicity, condition onnnpaired samplesZ1,…,ZnZ\_\{1\},\\ldots,Z\_\{n\}and draw a subsetSSof sizemmuniformly without replacement\. As above,

𝔼\[P^S\|Z1,…,Zn\]=P^n\.\\mathbb\{E\}\[\\widehat\{P\}\_\{S\}\\,\|\\,Z\_\{1\},\\ldots,Z\_\{n\}\]=\\widehat\{P\}\_\{n\}\.By concavity,

ℋvN\(Y\|X;P^n\)≥𝔼\[ℋvN\(Y\|X;P^S\)\|Z1,…,Zn\]\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{n\}\)\\geq\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(Y\\,\|\\,X;\\widehat\{P\}\_\{S\}\)\\,\|\\,Z\_\{1\},\\ldots,Z\_\{n\}\]\.Taking expectations and using the fact thatP^S\\widehat\{P\}\_\{S\}has the same law as an empirical distribution formed frommmi\.i\.d\. samples proves the result\. ∎

### C\.4Additional Theoretical Results on Lifted Bregman Divergences

Fix an input marginalPXP\_\{X\}\. The map

RY\|X⟼PX​RY\|XR\_\{Y\\,\|\\,X\}\\longmapsto P\_\{X\}R\_\{Y\\,\|\\,X\}embeds a conditional law into a joint law whose input marginal is fixed\. Given a Bregman divergenceDΦD\_\{\\Phi\}on a convex domain of joint laws, define

DΦ\|PX​\(RY\|X,QY\|X\)=DΦ​\(PX​RY\|X,PX​QY\|X\)\.D\_\{\\Phi\\,\|\\,P\_\{X\}\}\(R\_\{Y\\,\|\\,X\},Q\_\{Y\\,\|\\,X\}\)=D\_\{\\Phi\}\(P\_\{X\}R\_\{Y\\,\|\\,X\},P\_\{X\}Q\_\{Y\\,\|\\,X\}\)\.
IfDΦD\_\{\\Phi\}is the KL divergence, then

DΦ\|PX\(RY\|X,QY\|X\)=𝔼X∼PX\[KL\(RY\|X\(⋅\|X\)∥QY\|X\(⋅\|X\)\)\],D\_\{\\Phi\\,\|\\,P\_\{X\}\}\(R\_\{Y\\,\|\\,X\},Q\_\{Y\\,\|\\,X\}\)=\\mathbb\{E\}\_\{X\\sim P\_\{X\}\}\\Bigl\[\\mathrm\{KL\}\\bigl\(R\_\{Y\\,\|\\,X\}\(\\cdot\\,\|\\,X\)\\,\\\|\\,Q\_\{Y\\,\|\\,X\}\(\\cdot\\,\|\\,X\)\\bigr\)\\Bigr\],wheneverPX​RY\|XP\_\{X\}R\_\{Y\\,\|\\,X\}is absolutely continuous with respect toPX​QY\|XP\_\{X\}Q\_\{Y\\,\|\\,X\}\.

Ifψ⁡\(x,y\)\\psi\(x,y\)is a feature map on joint pairs and

μPX​R=𝔼X∼PX,Y∼R\(⋅\|X\)\[ψ\(X,Y\)\]\\mu\_\{P\_\{X\}R\}=\\mathbb\{E\}\_\{X\\sim P\_\{X\},\\,Y\\sim R\(\\cdot\\,\|\\,X\)\}\[\\psi\(X,Y\)\]is the kernel mean embedding ofPX​RY\|XP\_\{X\}R\_\{Y\\,\|\\,X\}, then

DMMD\|PX​\(RY\|X,QY\|X\)=‖μPX​R−μPX​Q‖ℋ2\.D\_\{\\mathrm\{MMD\}\\,\|\\,P\_\{X\}\}\(R\_\{Y\\,\|\\,X\},Q\_\{Y\\,\|\\,X\}\)=\\\|\\mu\_\{P\_\{X\}R\}\-\\mu\_\{P\_\{X\}Q\}\\\|\_\{\\mathcal\{H\}\}^\{2\}\.This is the Bregman divergence generated byΦ⁡\(μ\)=‖μ‖ℋ2\\Phi\(\\mu\)=\\\|\\mu\\\|\_\{\\mathcal\{H\}\}^\{2\}on the Hilbert space of mean embeddings, since

DΦ​\(μ,ν\)=‖μ−ν‖ℋ2\.D\_\{\\Phi\}\(\\mu,\\nu\)=\\\|\\mu\-\\nu\\\|\_\{\\mathcal\{H\}\}^\{2\}\.

### C\.5Proof of Theorem[1](https://arxiv.org/html/2609.02275#Thmtheorem1)

Forρ∈ℝ\\rho\\in\\mathbb\{R\}, we consider the definition:

𝒞ρ​\(PX\)=\{RY\|X:ℋvN​\(RY\|X,PX\)≥ρ\}\.\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)=\\Bigl\\\{R\_\{Y\\,\|\\,X\}:\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R\_\{Y\\,\|\\,X\};P\_\{X\}\)\\geq\\rho\\Bigr\\\}\.
###### Lemma 1\(Convexity of conditional entropy superlevel sets\)\.

The set𝒞ρ​\(PX\)\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)is convex\.

###### Proof\.

LetRY\|X\(0\),RY\|X\(1\)∈𝒞ρ​\(PX\)R^\{\(0\)\}\_\{Y\\,\|\\,X\},R^\{\(1\)\}\_\{Y\\,\|\\,X\}\\in\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)andλ∈\[0,1\]\\lambda\\in\[0,1\]\. Define

RY\|Xλ=\(1−λ\)​RY\|X\(0\)\+λ​RY\|X\(1\)\.R^\{\\lambda\}\_\{Y\\,\|\\,X\}=\(1\-\\lambda\)R^\{\(0\)\}\_\{Y\\,\|\\,X\}\+\\lambda R^\{\(1\)\}\_\{Y\\,\|\\,X\}\.Because the input marginal is fixed,

PX​RY\|Xλ=\(1−λ\)​PX​RY\|X\(0\)\+λ​PX​RY\|X\(1\)\.P\_\{X\}R^\{\\lambda\}\_\{Y\\,\|\\,X\}=\(1\-\\lambda\)P\_\{X\}R^\{\(0\)\}\_\{Y\\,\|\\,X\}\+\\lambda P\_\{X\}R^\{\(1\)\}\_\{Y\\,\|\\,X\}\.By Proposition[2](https://arxiv.org/html/2609.02275#Thmproposition2),

ℋvN​\(RY\|Xλ,PX\)≥\(1−λ\)​ℋvN​\(RY\|X\(0\),PX\)\+λ​ℋvN​\(RY\|X\(1\),PX\)≥ρ\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R^\{\\lambda\}\_\{Y\\,\|\\,X\};P\_\{X\}\)\\geq\(1\-\\lambda\)\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R^\{\(0\)\}\_\{Y\\,\|\\,X\};P\_\{X\}\)\+\\lambda\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(R^\{\(1\)\}\_\{Y\\,\|\\,X\};P\_\{X\}\)\\geq\\rho\.ThusRY\|Xλ∈𝒞ρ​\(PX\)R^\{\\lambda\}\_\{Y\\,\|\\,X\}\\in\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)\. ∎

We also use the following lemma on Bregman divergences\. LetΩ\\Omegabe a convex domain, letΦ:Ω→ℝ\\Phi:\\Omega\\to\\mathbb\{R\}be differentiable and strictly convex, and define

DΦ​\(A,B\)=Φ⁡\(A\)−Φ⁡\(B\)−⟨∇Φ​\(B\),A−B⟩\.D\_\{\\Phi\}\(A,B\)=\\Phi\(A\)\-\\Phi\(B\)\-\\langle\\nabla\\Phi\(B\),A\-B\\rangle\.
###### Lemma 2\.

Let𝒞⊆Ω\\mathcal\{C\}\\subseteq\\Omegabe nonempty and convex\. Suppose

Q⋆∈arg⁡minR∈𝒞​DΦ​\(R,Q\)\.Q^\{\\star\}\\in\\underset\{R\\in\\mathcal\{C\}\}\{\\arg\\\!\\min\}\\\>D\_\{\\Phi\}\(R,Q\)\.Then, for everyP∈𝒞P\\in\\mathcal\{C\},

DΦ​\(P,Q⋆\)\+DΦ​\(Q⋆,Q\)≤DΦ​\(P,Q\)\.D\_\{\\Phi\}\(P,Q^\{\\star\}\)\+D\_\{\\Phi\}\(Q^\{\\star\},Q\)\\leq D\_\{\\Phi\}\(P,Q\)\.

###### Proof\.

The first\-order optimality condition for the projection gives

⟨∇Φ​\(Q⋆\)−∇Φ​\(Q\),P−Q⋆⟩≥0,P∈𝒞\.\\langle\\nabla\\Phi\(Q^\{\\star\}\)\-\\nabla\\Phi\(Q\),P\-Q^\{\\star\}\\rangle\\geq 0,\\qquad P\\in\\mathcal\{C\}\.The three\-point identity for Bregman divergences gives

DΦ​\(P,Q\)=DΦ​\(P,Q⋆\)\+DΦ​\(Q⋆,Q\)\+⟨∇Φ​\(Q⋆\)−∇Φ​\(Q\),P−Q⋆⟩\.D\_\{\\Phi\}\(P,Q\)=D\_\{\\Phi\}\(P,Q^\{\\star\}\)\+D\_\{\\Phi\}\(Q^\{\\star\},Q\)\+\\langle\\nabla\\Phi\(Q^\{\\star\}\)\-\\nabla\\Phi\(Q\),P\-Q^\{\\star\}\\rangle\.Combining the two displays yields the inequality\. ∎

###### Proof of Theorem[1](https://arxiv.org/html/2609.02275#Thmtheorem1)\.

The choice

ρ=ℋvN​\(PY\|X,PX\)\\rho=\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(P\_\{Y\\,\|\\,X\};P\_\{X\}\)impliesPY\|X∈𝒞ρ​\(PX\)P\_\{Y\\,\|\\,X\}\\in\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)\. By Lemma[1](https://arxiv.org/html/2609.02275#Thmlemma1),𝒞ρ​\(PX\)\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)is convex\. Applying Lemma[2](https://arxiv.org/html/2609.02275#Thmlemma2)to the lifted joint lawsPX​PY\|X,PX​QY\|X,PX​QY\|X⋆P\_\{X\}P\_\{Y\\,\|\\,X\},\\,P\_\{X\}Q\_\{Y\\,\|\\,X\},\\,P\_\{X\}Q^\{\\star\}\_\{Y\\,\|\\,X\}over the convex set\{PX​RY\|X:RY\|X∈𝒞ρ​\(PX\)\}\\\{P\_\{X\}R\_\{Y\\,\|\\,X\}:R\_\{Y\\,\|\\,X\}\\in\\mathcal\{C\}\_\{\\rho\}\(P\_\{X\}\)\\\}results in

DΦ\|PX​\(PY\|X,QY\|X⋆\)\+DΦ\|PX​\(QY\|X⋆,QY\|X\)≤DΦ\|PX​\(PY\|X,QY\|X\)\.D\_\{\\Phi\\,\|\\,P\_\{X\}\}\(P\_\{Y\\,\|\\,X\},Q^\{\\star\}\_\{Y\\,\|\\,X\}\)\+D\_\{\\Phi\\,\|\\,P\_\{X\}\}\(Q^\{\\star\}\_\{Y\\,\|\\,X\},Q\_\{Y\\,\|\\,X\}\)\\leq D\_\{\\Phi\\,\|\\,P\_\{X\}\}\(P\_\{Y\\,\|\\,X\},Q\_\{Y\\,\|\\,X\}\)\.Therefore, the proof is complete\. ∎

### C\.6Additional Theoretical Results on Finite\-Support Conditional MMD Projection

Letx1,…,xNx\_\{1\},\\ldots,x\_\{N\}be input prompts, and letyi,1,…,yi,my\_\{i,1\},\\ldots,y\_\{i,m\}be generated candidates forxix\_\{i\}\. LetM=N​mM=Nmand index pairs bya=\(i,j\)a=\(i,j\):

sa=\(xi,yi,j\)\.s\_\{a\}=\(x\_\{i\},y\_\{i,j\}\)\.Letq∈ℝMq\\in\\mathbb\{R\}^\{M\}be a distribution over the generated pairs and letu∈ℝMu\\in\\mathbb\{R\}^\{M\}be the uniform baseline,

ui,j=1N​m\.u\_\{i,j\}=\\frac\{1\}\{Nm\}\.The conditional block constraints are

qi,j≥0,∑j=1mqi,j=1N,i=1,…,N\.q\_\{i,j\}\\geq 0,\\qquad\\sum\_\{j=1\}^\{m\}q\_\{i,j\}=\\frac\{1\}\{N\},\\qquad i=1,\\ldots,N\.Equivalently,qi,j=pi,j/Nq\_\{i,j\}=p\_\{i,j\}/Nwithpi∈Δmp\_\{i\}\\in\\Delta\_\{m\}\.

Let

k⁡\(\(x,y\),\(x′,y′\)\)=kX​\(x,x′\)​kY​\(y,y′\)k\(\(x,y\),\(x^\{\\prime\},y^\{\\prime\}\)\)=k\_\{X\}\(x,x^\{\\prime\}\)k\_\{Y\}\(y,y^\{\\prime\}\)be the product kernel, and letK∈ℝM×MK\\in\\mathbb\{R\}^\{M\\times M\}be the Gram matrix over the generated pairs\. Then the squared MMD between the reweighted distributionqqand the uniform baselineuuis

‖∑a\(qa−ua\)​ψ​\(sa\)‖ℋ2=\(q−u\)⊤​K​\(q−u\)\.\\Bigl\\\|\\sum\_\{a\}\(q\_\{a\}\-u\_\{a\}\)\\psi\(s\_\{a\}\)\\Bigr\\\|\_\{\\mathcal\{H\}\}^\{2\}=\(q\-u\)^\{\\top\}K\(q\-u\)\.Thus, the objective in equation[1](https://arxiv.org/html/2609.02275#S5.E1)is the finite\-support instance of the lifted conditional MMD with empirical input marginal\.

The weighted joint covariance operator is

CX​Y​\(q\)=∑aqa​ψ​\(sa\)​ψ​\(sa\)∗\.C\_\{XY\}\(q\)=\\sum\_\{a\}q\_\{a\}\\psi\(s\_\{a\}\)\\psi\(s\_\{a\}\)^\{\\ast\}\.Because the product kernel is normalized and∑aqa=1\\sum\_\{a\}q\_\{a\}=1, this operator has trace one\. The empirical input covariance is

CX=1N​∑i=1NϕX​\(xi\)​ϕX​\(xi\)∗\.C\_\{X\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\phi\_\{X\}\(x\_\{i\}\)\\phi\_\{X\}\(x\_\{i\}\)^\{\\ast\}\.The block constraints makeCXC\_\{X\}independent ofqq, so

ℋvN​\(q\)=HvN​\(CX​Y​\(q\)\)−HvN​\(CX\)\.\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(q\)=H\_\{\\mathrm\{vN\}\}\(C\_\{XY\}\(q\)\)\-H\_\{\\mathrm\{vN\}\}\(C\_\{X\}\)\.
The nonzero eigenvalues ofCX​Y​\(q\)C\_\{XY\}\(q\)coincide with the nonzero eigenvalues of the weighted Gram matrix

Kq=diag⁡\(q\)​K​diag⁡\(q\)\.K\_\{q\}=\\operatorname\{diag\}\(\\sqrt\{q\}\)\\,K\\,\\operatorname\{diag\}\(\\sqrt\{q\}\)\.Indeed, writingCX​Y​\(q\)=A​A∗C\_\{XY\}\(q\)=AA^\{\\ast\}with columnsqa​ψ​\(sa\)\\sqrt\{q\_\{a\}\}\\psi\(s\_\{a\}\)givesKq=A∗​AK\_\{q\}=A^\{\\ast\}A, andA​A∗AA^\{\\ast\}andA∗​AA^\{\\ast\}Ahave the same nonzero spectrum\.

### C\.7Proof of Proposition[3](https://arxiv.org/html/2609.02275#Thmproposition3)

###### Proof\.

The objective\(q−u\)⊤​K​\(q−u\)\(q\-u\)^\{\\top\}K\(q\-u\)is convex becauseKKis positive semidefinite\. The nonnegativity and block\-marginal constraints are linear\. The mapq↦CX​Y​\(q\)q\\mapsto C\_\{XY\}\(q\)is affine, andHvNH\_\{\\mathrm\{vN\}\}is concave on positive trace\-one operators\. SinceCXC\_\{X\}is fixed over the feasible set,q↦ℋvN​\(q\)q\\mapsto\\mathcal\{H\}\_\{\\mathrm\{vN\}\}\(q\)is concave\. Hence, the entropy superlevel constraint is convex, and the program is convex\. ∎

### C\.8Proof of Proposition[4](https://arxiv.org/html/2609.02275#Thmproposition4)

###### Proof\.

The squared norm term is convex because it is the squared norm of an affine function ofpp\. The mapp↦C~​\(p\)p\\mapsto\\widetilde\{C\}\(p\)is affine, and von Neumann entropy is concave on trace\-one positive semidefinite matrices\. Thereforep↦−HvN​\(C~​\(p\)\)p\\mapsto\-H\_\{\\mathrm\{vN\}\}\(\\widetilde\{C\}\(p\)\)is convex\. The constraintspi∈Δmp\_\{i\}\\in\\Delta\_\{m\}are linear, so the problem is convex\. ∎

### C\.9Proof of Theorem[2](https://arxiv.org/html/2609.02275#Thmtheorem2)

CEP\-BEG uses the product\-simplex negative\-entropy mirror map

Ψ⁡\(p\)=∑i=1N∑j=1mpi,j​log⁡pi,j\.\\Psi\(p\)=\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{m\}p\_\{i,j\}\\log p\_\{i,j\}\.The associated Bregman divergence is

DΨ​\(p,p′\)=∑i=1N∑j=1mpi,j​log⁡pi,jpi,j′\.D\_\{\\Psi\}\(p,p^\{\\prime\}\)=\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{m\}p\_\{i,j\}\\log\\frac\{p\_\{i,j\}\}\{p^\{\\prime\}\_\{i,j\}\}\.The mirror step is

p\(t\+1\)=arg⁡minp1,…,pN∈Δm​\{⟨g\(t\),p⟩\+1η​DΨ​\(p,p\(t\)\)\},p^\{\(t\+1\)\}=\\underset\{p\_\{1\},\\ldots,p\_\{N\}\\in\\Delta\_\{m\}\}\{\\arg\\\!\\min\}\\\>\\Bigl\\\{\\langle g^\{\(t\)\},p\\rangle\+\\frac\{1\}\{\\eta\}D\_\{\\Psi\}\(p,p^\{\(t\)\}\)\\Bigr\\\},which yields the block exponentiated\-gradient update in Algorithm[1](https://arxiv.org/html/2609.02275#alg1)\.

###### Proof\.

For each blockii, the negative\-entropy mirror descent inequality gives

⟨gi\(t\),pi\(t\)−pi⟩≤DKL​\(pi,pi\(t\)\)−DKL​\(pi,pi\(t\+1\)\)η\+η2​‖gi\(t\)‖∞2\.\\langle g\_\{i\}^\{\(t\)\},p\_\{i\}^\{\(t\)\}\-p\_\{i\}\\rangle\\leq\\frac\{D\_\{\\mathrm\{KL\}\}\(p\_\{i\},p\_\{i\}^\{\(t\)\}\)\-D\_\{\\mathrm\{KL\}\}\(p\_\{i\},p\_\{i\}^\{\(t\+1\)\}\)\}\{\\eta\}\+\\frac\{\\eta\}\{2\}\\\|g\_\{i\}^\{\(t\)\}\\\|\_\{\\infty\}^\{2\}\.Summing overi=1,…,Ni=1,\\ldots,Nyields

⟨g\(t\),p\(t\)−p⟩≤DΨ​\(p,p\(t\)\)−DΨ​\(p,p\(t\+1\)\)η\+η2​∑i=1N‖gi\(t\)‖∞2\.\\langle g^\{\(t\)\},p^\{\(t\)\}\-p\\rangle\\leq\\frac\{D\_\{\\Psi\}\(p,p^\{\(t\)\}\)\-D\_\{\\Psi\}\(p,p^\{\(t\+1\)\}\)\}\{\\eta\}\+\\frac\{\\eta\}\{2\}\\sum\_\{i=1\}^\{N\}\\\|g\_\{i\}^\{\(t\)\}\\\|\_\{\\infty\}^\{2\}\.Takingp=p⋆p=p^\{\\star\}and using convexity ofFF,

F⁡\(p\(t\)\)−F⁡\(p⋆\)≤⟨g\(t\),p\(t\)−p⋆⟩\.F\(p^\{\(t\)\}\)\-F\(p^\{\\star\}\)\\leq\\langle g^\{\(t\)\},p^\{\(t\)\}\-p^\{\\star\}\\rangle\.Summing overt=0,…,T−1t=0,\\ldots,T\-1gives

∑t=0T−1\(F⁡\(p\(t\)\)−F⁡\(p⋆\)\)≤DΨ​\(p⋆,p\(0\)\)η\+η​T​G22\.\\sum\_\{t=0\}^\{T\-1\}\\big\(F\(p^\{\(t\)\}\)\-F\(p^\{\\star\}\)\\big\)\\leq\\frac\{D\_\{\\Psi\}\(p^\{\\star\},p^\{\(0\)\}\)\}\{\\eta\}\+\\frac\{\\eta TG^\{2\}\}\{2\}\.Sincepi,j\(0\)=1/mp^\{\(0\)\}\_\{i,j\}=1/mandpi⋆∈Δmp\_\{i\}^\{\\star\}\\in\\Delta\_\{m\},

DΨ​\(p⋆,p\(0\)\)=∑i=1N∑j=1mpi,j⋆​log⁡\(m​pi,j⋆\)≤N​log⁡m\.D\_\{\\Psi\}\(p^\{\\star\},p^\{\(0\)\}\)=\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{m\}p^\{\\star\}\_\{i,j\}\\log\(mp^\{\\star\}\_\{i,j\}\)\\leq N\\log m\.Dividing byTTand using Jensen’s inequality for the convex functionFF,

F⁡\(p¯\(T\)\)≤1T​∑t=0T−1F⁡\(p\(t\)\),F\(\\bar\{p\}^\{\(T\)\}\)\\leq\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}F\(p^\{\(t\)\}\),we obtain

F⁡\(p¯\(T\)\)−F⁡\(p⋆\)≤N​log⁡mη​T\+η​G22\.F\(\\bar\{p\}^\{\(T\)\}\)\-F\(p^\{\\star\}\)\\leq\\frac\{N\\log m\}\{\\eta T\}\+\\frac\{\\eta G^\{2\}\}\{2\}\.The stated choice ofη=2​N​log⁡m/\(G​T\)\\eta=\\sqrt\{2N\\log m\}/\(G\\sqrt\{T\}\)minimizes this upper bound which becomes the following and finishes the proof:

F⁡\(p¯\(T\)\)−F⁡\(p⋆\)≤G​2​N​log⁡mT\.F\(\\bar\{p\}^\{\(T\)\}\)\-F\(p^\{\\star\}\)\\leq G\\sqrt\{\\frac\{2N\\log m\}\{T\}\}\.The proof is hence complete\. ∎

For the regularized entropy implementation, a concrete gradient bound follows under the assumption of a spectral floor\. If‖z~i,j‖2=1\\\|\\tilde\{z\}\_\{i,j\}\\\|\_\{2\}=1and the eigenvalues of the matrices used inside the logarithm are bounded below byτ\>0\\tau\>0, then

‖gi\(t\)‖∞≤4\+λ⁡\(1\+\|log⁡τ\|\)Nfor all​i,t\.\\\|g\_\{i\}^\{\(t\)\}\\\|\_\{\\infty\}\\leq\\frac\{4\+\\lambda\(1\+\|\\log\\tau\|\)\}\{N\}\\quad\\text\{for all \}i,t\.Consequently, one may take

G=4\+λ⁡\(1\+\|log⁡τ\|\)N\.G=\\frac\{4\+\\lambda\(1\+\|\\log\\tau\|\)\}\{\\sqrt\{N\}\}\.The bound follows from‖r~​\(p\)‖2≤2\\\|\\tilde\{r\}\(p\)\\\|\_\{2\}\\leq 2and

‖log⁡C~​\(p\)\+I‖op≤1\+\|log⁡τ\|\\\|\\log\\widetilde\{C\}\(p\)\+I\\\|\_\{\\mathrm\{op\}\}\\leq 1\+\|\\log\\tau\|on the spectral interval\[τ,1\]\[\\tau,1\]\.

## Appendix DImplementation Details

### D\.1LLM Data Construction and Sample Generation

We construct paired prefix–continuation samples from the corresponding open training corpora of the evaluated language models\. For OLMo, we use the Dolma corpus; for Pythia and GPT\-Neo, we use The Pile\. Each example is represented as a pair\(X,Y\)\(X,Y\), whereXXis a short prefix used as the conditioning input andYYis the following continuation\. The same prefix set is used for both the training\-data reference and the model\-generated continuations, so the input marginal is matched across groups\.

For OLMo, we use theallenai/OLMo\-1B\-hfcheckpoint and its associated tokenizer\. The tokenizer padding token is set to the end\-of\-sequence token when no padding token is provided, and left padding is used for batched autoregressive generation\. The model is loaded with automatic device mapping and half precision\. To construct the training reference for OLMo, we use the Hugging Face Dolma dataset\. We load the downloaded Dolma files with the Hugging Facedatasetslibrary and process each document into paragraph\-level candidates\. We split documents by blank lines, normalize whitespace, and keep only paragraphs with at least500500characters and at least33sentence\-ending marks\. From each document, we uniformly sample the candidate paragraphs\. We then tokenize each candidate paragraph and keep the first paragraph that contains at least the required number of tokens\. Unless otherwise specified, we use a prefix length of33tokens and a continuation length of22tokens\. The prefix tokens are used as the conditionXX, and the immediately following tokens are used as the training continuationYtrainY\_\{\\mathrm\{train\}\}\. To examine the effect of token\-length choices, we additionally conduct an ablation over all combinations of prefix lengths in\{3,4,5\}\\\{3,4,5\\\}and continuation lengths in\{2,3,4\}\\\{2,3,4\\\}\. We also conduct a sample\-size ablation by drawing subsamples of different sizes under multiple random seeds and reporting the resulting variation across runs\. Documents without a valid paragraph of sufficient length are skipped\.

For each prefix, we generate model continuations using three decoding strategies\. Greedy decoding selects the most likely token at each step\. Nucleus sampling samples from the smallest token set whose cumulative probability mass exceedsp=0\.9p=0\.9\. Ancestral sampling samples from the full next\-token distribution, implemented by settingtop\_p=1\.0\. All strategies use the same maximum number of newly generated tokens as the training continuation length\. We store the generated data in JSON format with fields for the example id, prefix, training continuation, greedy decoding output, nucleus sampling output, and ancestral sampling output\. In the main experiments, we sample up to3030K prefix–continuation pairs and evaluate conditional diversity at different sample sizes using subsets of these generated records\. For Pythia and GPT\-Neo, we follow the same construction using The Pile as the training\-data reference\. We use the Hugging Face checkpointsEleutherAI/pythia\-1bandEleutherAI/gpt\-neo\-1\.3B, together with their corresponding tokenizers\. For each model, we tokenize documents from The Pile, extract matched prefix–continuation pairs, and generate continuations from the same prefix set under greedy decoding, nucleus sampling with the samep=0\.9p=0\.9, and ancestral sampling\. This ensures that the comparison between training and generated groups is always performed under a shared empirical input marginal\.

Long\-sequence construction\.For the long\-sequence experiments in Section[6\.3](https://arxiv.org/html/2609.02275#S6.SS3), we extend the prefix length to\{8,16,32\}\\\{8,16,32\\\}tokens and the continuation length to\{16,32,64\}\\\{16,32,64\\\}tokens\. To represent a long sequence while preserving local information, we split each sequence into22\-token segments, compute the Qwen3\-Embedding of each segment independently, and concatenate the resulting embeddings into a single feature vector\. The same chunking construction has been widely used in the long\-text literature\[[64](https://arxiv.org/html/2609.02275#bib.bib60),[65](https://arxiv.org/html/2609.02275#bib.bib61),[66](https://arxiv.org/html/2609.02275#bib.bib62),[67](https://arxiv.org/html/2609.02275#bib.bib63),[68](https://arxiv.org/html/2609.02275#bib.bib64)\]\. The same prefix set is used for the training\-data reference and the model\-generated continuations\. We additionally perform a chunk\-size ablation over chunk sizes\{1,2,4\}\\\{1,2,4\\\}in Appendix[E\.6](https://arxiv.org/html/2609.02275#A5.SS6)\. The entropy computation depends primarily on the number of samples and the representation dimension rather than directly on the number of tokens; longer sequences mainly increase the cost of constructing their representations, after which the same kernel\-based estimator and optimization procedure apply\.

In Table[2](https://arxiv.org/html/2609.02275#S6.T2)we observe two consistent trends in the gap\. First, for a fixed prefix length, the gap generally grows with the continuation length, because a longer continuation leaves more room for valid variation that the model under\-covers\. Second, for a fixed continuation length, the gap generally shrinks as the prefix length increases, because stronger conditioning reduces the range of plausible continuations\. Nevertheless, no configuration eliminates the gap\.

### D\.2Image Data Construction and Sample Generation

We construct paired condition–image samples for two conditional image\-generation settings: class\-conditioned ImageNet generation and text\-conditioned MS\-COCO generation\. Each example is represented as a pair\(X,Y\)\(X,Y\), whereXXis the conditioning variable andYYis the corresponding image\. For ImageNet,XXis the class label andYYis the image\. For MS\-COCO,XXis the text caption andYYis the image\. In both settings, we compare generated images with real\-data references under matched conditions\.

For the class\-conditioned ImageNet experiments, we follow the DGM\-Eval benchmark and use the generated image samples provided for several class\-conditioned generators, including ADM, LDM, BigGAN, and DiT\-XL\-2\. The real\-data reference is constructed from ImageNet training images with their corresponding class labels\. For each generated sample, we use its class label as the condition and the generated image as the output\. We then form paired samples\(Xi,Yi\)\(X\_\{i\},Y\_\{i\}\)for both the real ImageNet reference and each generator\. Conditional diversity is evaluated by comparing the real and generated paired distributions under the same class\-conditioned protocol\.

For the text\-conditioned MS\-COCO experiments, we use captions sampled from the MS\-COCO dataset as conditioning inputs\. Given a captionXiX\_\{i\}, we generate an imageYiY\_\{i\}using each text\-to\-image model\. We evaluate U\-ViT\-S/2 and its corresponding Deep variant, as well as SDXL\-Base\-1\.0 and PixArt\-Σ\\Sigma\. For the real\-data reference, we use the MS\-COCO image associated with the sampled caption\. This gives paired caption–image samples for the real MS\-COCO reference and for each generated model group\. The same caption set is used when comparing the reference and generated samples, so the empirical input marginal over captions is matched\.

For all image\-generation experiments, conditional VNE and conditional RKE are computed from paired condition–image samples using the product kernel between condition features and image features\. In the ImageNet setting, the condition feature is derived from the class label; in the MS\-COCO setting, the condition feature is derived from the caption\. Unless otherwise specified, we use the gaussian kernel with bandwidth selection following the literature\.

For sample\-size analysis, we evaluate conditional diversity over subsets of different sizes\. In the ImageNet and MS\-COCO gap experiments, we report the exponential of conditional VNE over sample sizes ranging from2\.52\.5K to2020K\. For each sample size, we draw subsamples under multiple random seeds and report the mean and standard deviation across evaluations\. This protocol is applied consistently to the real\-data reference and to each generated model group\.

### D\.3Evaluation Settings and Hyperparameters

For the LLM diversity\-gap experiments, the default text representation is Qwen3\-Embedding\. We compute embeddings for the prefix and continuation texts separately, construct a product kernel over the paired prefix–continuation samples, and evaluate conditional VNE and conditional RKE on the resulting normalized kernel matrices\. Unless otherwise stated, we use the Gaussian kernel with bandwidth selected by the median heuristic on the corresponding feature distances\. We also report ablations with T5 embeddings, cosine kernels, and degree\-3 polynomial kernels withγ=1\\gamma=1in Tables[13](https://arxiv.org/html/2609.02275#A5.T13),[14](https://arxiv.org/html/2609.02275#A5.T14), and[15](https://arxiv.org/html/2609.02275#A5.T15)\.

For the image experiments, image features are extracted using the default visual encoder DINOv2 and condition text features are extracted by CLIP model\. We use the same normalized product\-kernel construction for all conditional VNE and conditional RKE evaluations, so differences in the reported scores reflect the conditional output distributions under matched input conditions\.

For entropy\-projected reweighting, each prefix is associated with1010generated candidate continuations\. The optimization is performed over a product of simplices, with one simplex for the candidates associated with each prefix, so the empirical prefix marginal is fixed throughout optimization\. We initialize the weights uniformly, use the conditional entropy objective described in Section[5](https://arxiv.org/html/2609.02275#S5), and report the reweighted empirical distribution using the same evaluation pipeline as the unweighted generated samples\. For scalability, the joint product features are computed through the sketched representation described in the method section\.

For conditional VNE guidance in diffusion sampling, we use SDXL on MS\-COCO captions and set the guidance scale toη=0\.03\\eta=0\.03\. The guidance step is applied during denoising after the standard SDXL update\. All guided and unguided comparisons use the same caption set and the same evaluation features, and conditional VNE is reported as mean±\\pmstandard deviation over independent subsamples\. All experiments were run on 2 RTX\-4090 GPUs\.

For the MBR decoding experiments in Section[6\.7](https://arxiv.org/html/2609.02275#S6.SS7), we follow the candidate\-generation protocol of Jinnai et al\.\[[30](https://arxiv.org/html/2609.02275#bib.bib66)\], using nucleus sampling withp=0\.9p=0\.9for a controlled comparison\. We evaluate XSum with BART\-large\-XSum, CNN/DailyMail with BART\-large\-CNN, and CNN/DailyMail with Mistral\-7B\-Instruct\-v0\.1 using the CNN/DM prompt from their Appendix G\. For each of5050test inputs, we compare Monte Carlo MBR, model\-based MBR with and without length normalization, and the proposed conditional\-Vendi\-weighted variant \(CVS\-MBR\)\. All methods operate on the same candidate pool and utility function and differ only in the reference weighting used in the MBR expectation\.

The methods compared in Table[5](https://arxiv.org/html/2609.02275#S6.T5)represent different operating points\. MBMBR and MBMBR\-L use model\-probability weights and are mainly designed to improve expected generation quality; accordingly, they achieve higher ROUGE\-L and BERTScore, with length normalization mitigating sequence\-length bias\. In contrast, CVS\-MBR explicitly reduces concentration in the reference distribution and therefore achieves the highest conditional VNE and Distinct\-22, as well as the lowest Self\-BLEU\-22, across all three settings\. To examine the quality–diversity trade\-off, we performed a paired bootstrap over the5050test inputs with20002000resamples, comparing CVS\-MBR against MBMBR\-L\. The95%95\\%confidence intervals for the ROUGE\-L and BERTScore differences contain zero in all three settings, indicating no statistically significant degradation in quality while CVS\-MBR consistently attains the highest conditional VNE\.

## Appendix EAdditional Numerical Results

### E\.1Additional LLM Diversity Gap Results

Tables[17](https://arxiv.org/html/2609.02275#A5.T17)and[18](https://arxiv.org/html/2609.02275#A5.T18)provide the full LLM diversity\-gap results at sample sizes1010K and2020K\. Across OLMo, Pythia, and GPT\-Neo, the training continuations consistently achieve higher conditional VNE and conditional RKE than model\-generated continuations from the same prefixes\. The ordering among decoding strategies is also stable: greedy decoding has the largest gap, nucleus sampling reduces the gap, and ancestral sampling is usually the closest to the training reference\. For example, at2020K samples, the conditional VNE gap for OLMo is119\.70119\.70under greedy decoding,51\.2751\.27under nucleus sampling, and41\.2741\.27under ancestral sampling\.

The lexical diversity metrics show the same qualitative pattern\. Distinct\-2 and Distinct\-3 are substantially lower for greedy decoding than for the training continuations, while stochastic decoding recovers much of the surface\-level n\-gram diversity\. However, even when Distinct scores become relatively close to the training reference, the conditional VNE and conditional RKE gaps remain positive\. This indicates that the proposed kernel\-based conditional diversity metrics capture conditional output\-range differences that are not fully explained by local n\-gram variety\.

Comparing1010K and2020K samples further shows that the measured conditional diversity gap becomes more pronounced as the evaluation set grows\. The training reference gains conditional VNE faster than the generated groups, which is consistent with the sample\-size curves in Figure[1](https://arxiv.org/html/2609.02275#S6.F1)\. This behavior suggests that the training data continues to reveal additional valid conditional continuations as more examples are included, whereas the generated samples cover a narrower conditional range\.

### E\.2Additional Image Diversity Gap Results

Figure[2](https://arxiv.org/html/2609.02275#A5.F2)and Table[20](https://arxiv.org/html/2609.02275#A5.T20)report the ImageNet class\-conditioned results\. The ImageNet reference has the highest conditional VNE at every sample size\. Among the evaluated generators, DiT\-XL\-2 and LDM are closest to the reference, while ADM and BigGAN exhibit much larger gaps\. At2020K samples, ImageNet reaches54\.1554\.15, compared with49\.9549\.95for DiT\-XL\-2,47\.8647\.86for LDM,22\.4722\.47for ADM, and17\.0417\.04for BigGAN\. Thus, the strongest diffusion and transformer\-based generators approach the reference more closely, but they still do not fully match the conditional diversity of real images under the same class labels\.

Figure[3](https://arxiv.org/html/2609.02275#A5.F3)and Table[21](https://arxiv.org/html/2609.02275#A5.T21)show the text\-conditioned MS\-COCO results\. The MS\-COCO reference again achieves the highest conditional VNE across all sample sizes\. SDXL is the strongest among the evaluated generated groups in this experiment, followed closely by U\-ViT Deep S/2 and U\-ViT S/2, while PixArt\-Σ\\Sigmahas the largest gap\. At2020K samples, the reference reaches40\.5240\.52, whereas SDXL reaches30\.2530\.25, U\-ViT Deep S/2 reaches29\.9429\.94, U\-ViT S/2 reaches29\.5029\.50, and PixArt\-Σ\\Sigmareaches25\.2525\.25\.

The image results mirror the LLM results: the real\-data reference grows faster with sample size than the generated groups, so the diversity gap becomes clearer as more samples are evaluated\. Because the comparisons are made under matched class labels or captions, the gap reflects differences in conditional output variability rather than a mismatch in the empirical input marginal\.

### E\.3Additional Reweighting Results

The entropy\-projected reweighting results in Table[3](https://arxiv.org/html/2609.02275#S6.T3)show that conditional diversity can be increased post hoc by redistributing probability mass over already generated candidates\. The reweighting improves conditional VNE for all three LLMs at both evaluated sample sizes\. At2020K samples, conditional VNE increases from366\.35366\.35to376\.77376\.77for OLMo, from371\.72371\.72to382\.74382\.74for Pythia, and from340\.90340\.90to353\.68353\.68for GPT\-Neo\.

The effect is especially strong for conditional RKE\. At2020K samples, RKE increases from241\.14241\.14to284\.92284\.92for OLMo, from236\.24236\.24to285\.05285\.05for Pythia, and from246\.13246\.13to296\.02296\.02for GPT\-Neo\. These gains are obtained without retraining the model and without adding new generated continuations\. Since the projection preserves the total mass assigned to each prefix, the improvement comes from changing the conditional distribution over available continuations rather than altering the input distribution\.

These results support the interpretation that generated candidate sets often contain underweighted alternatives\. The projection can recover part of this latent conditional diversity by moving mass toward candidates that increase the joint condition–output entropy while remaining within the fixed candidate support\.

### E\.4Additional Conditional VNE Guidance Results

We further evaluate whether the conditional VNE score can be used not only as an evaluation metric, but also as a sampling\-time guidance signal for conditional generation\. The goal is to encourage higher conditional diversity during sampling while keeping the text condition fixed\.

Letc\(n\)c^\{\(n\)\}denote the caption for thenn\-th generated image, and letzt\(n\)z\_\{t\}^\{\(n\)\}be its latent variable at diffusion steptt\. Given previously generated caption\-image pairs\{\(c\(i\),z\(i\)\)\}i=1n−1\\\{\(c^\{\(i\)\},z^\{\(i\)\}\)\\\}\_\{i=1\}^\{n\-1\}and the current pair\(c\(n\),zt\(n\)\)\(c^\{\(n\)\},z\_\{t\}^\{\(n\)\}\), we construct a caption kernelKCK\_\{C\}and a latent\-image kernelKZK\_\{Z\}\. The conditional VNE guidance score is

H^vN​\(Z∣C\)=HvN​\(1n​\(KC⊙KZ\)\)−HvN​\(1n​KC\),\\widehat\{H\}\_\{\\mathrm\{vN\}\}\(Z\\mid C\)=H\_\{\\mathrm\{vN\}\}\\\!\\left\(\\frac\{1\}\{n\}\(K\_\{C\}\\odot K\_\{Z\}\)\\right\)\-H\_\{\\mathrm\{vN\}\}\\\!\\left\(\\frac\{1\}\{n\}K\_\{C\}\\right\),where⊙\\odotdenotes the Hadamard product\. Since the caption kernelKCK\_\{C\}is fixed during the latent update, the guidance acts through the gradient of the joint conditional term with respect to the current latent variable\. After the standard SDXL denoising update, we apply an additional guidance step

zt−1\(n\)←zt−1\(n\)\+η​∇zt−1\(n\)H^vN​\(Z∣C\),z\_\{t\-1\}^\{\(n\)\}\\leftarrow z\_\{t\-1\}^\{\(n\)\}\+\\eta\\nabla\_\{z\_\{t\-1\}^\{\(n\)\}\}\\widehat\{H\}\_\{\\mathrm\{vN\}\}\(Z\\mid C\),whereη\\etais the guidance scale\. For the experiment, we use captions from the MS\-COCO dataset and generate3030K images with SDXL, both with and without the proposed guidance, and the guidance scale we used is0\.030\.03\. We then evaluate conditional VNE at sample sizes from1010K to2020K reported in Table[4](https://arxiv.org/html/2609.02275#S6.T4)\. SDXL with conditional VNE guidance consistently obtains higher conditional VNE than SDXL without guidance at every evaluated sample size\. For example, at2020K samples, the conditional VNE increases from30\.2530\.25without guidance to32\.2732\.27with guidance\. The guided samples still remain below the reference MS\-COCO data, which reaches40\.5240\.52at the same sample size, but the guidance reduces the measured gap between generated and reference samples\.

### E\.5Ablation Study Analysis

Tables[13](https://arxiv.org/html/2609.02275#A5.T13),[14](https://arxiv.org/html/2609.02275#A5.T14), and[15](https://arxiv.org/html/2609.02275#A5.T15)examine robustness to embedding and kernel choices\. The absolute scale of conditional VNE and conditional RKE changes substantially across feature maps and kernels, as expected, but the qualitative conclusion is unchanged: the training reference has the highest conditional diversity, greedy decoding has the largest gap, and stochastic decoding reduces but does not eliminate the gap\. This consistency indicates that the observed conditional diversity gap is not an artifact of a single embedding model or kernel family\.

Table[13](https://arxiv.org/html/2609.02275#A5.T13)repeats the LLM analysis with T5 embeddings\. Even under this alternative text representation, all generated groups remain below the corresponding training reference\. At2020K samples, the conditional VNE gaps under ancestral sampling are10\.1110\.11for OLMo,6\.946\.94for Pythia, and5\.635\.63for GPT\-Neo, while greedy decoding has much larger gaps\. Tables[14](https://arxiv.org/html/2609.02275#A5.T14)and[15](https://arxiv.org/html/2609.02275#A5.T15)show the same ranking under degree\-3 polynomial and cosine kernels, respectively, although the numerical scale is compressed relative to the default Gaussian\-kernel evaluation\.

Table[16](https://arxiv.org/html/2609.02275#A5.T16)studies the effect of increasing the prefix length to44tokens with22continuation tokens\. The training reference still has higher conditional VNE and conditional RKE than all generated groups for all three models\. The gaps are generally smaller than in the default33\-token\-prefix setting, which is consistent with stronger conditioning reducing the range of plausible short continuations\. Nevertheless, the decoding\-order pattern remains unchanged: greedy decoding is least diverse, followed by nucleus sampling and ancestral sampling\.

Table[19](https://arxiv.org/html/2609.02275#A5.T19)reports the full prefix–continuation length ablation for conditional VNE\. Increasing continuation length tends to increase the absolute conditional VNE for both training and generated continuations because the output space becomes larger\. The average gap remains positive in every tested configuration, confirming that the conditional diversity gap persists across the evaluated tokenization choices\. The gap is often larger for shorter prefixes and longer continuations, where the condition leaves more uncertainty about valid outputs and the model must cover a broader conditional continuation space\.

### E\.6Chunk\-Size Ablation for Long\-Sequence Representations

The long\-sequence experiments in Section[6\.3](https://arxiv.org/html/2609.02275#S6.SS3)represent each sequence as a concatenation of22\-token segment embeddings\. To examine the sensitivity of the reported gap to this choice, we vary the chunk size over\{1,2,4\}\\\{1,2,4\\\}tokens per segment on GPT\-Neo\. Since chunk size=2=2corresponds to the configuration already reported in Table[2](https://arxiv.org/html/2609.02275#S6.T2), here we report the two remaining cases: Tables[6](https://arxiv.org/html/2609.02275#A5.T6)and[7](https://arxiv.org/html/2609.02275#A5.T7)show the exponentiated conditional VNE for chunk sizes11and44, respectively\. Across all three chunk sizes, the training data consistently achieve higher conditional VNE than the generated continuations under every decoding strategy and every prefix–continuation configuration\. The absolute scale of the conditional VNE changes with the chunk size, but the qualitative ordering training\>\>ancestral\>\>nucleus\>\>greedy and the positivity of the gap are preserved\. This confirms that the observed long\-sequence conditional diversity gap is not an artifact of the specific chunk size used in the main long\-sequence analysis\.

Table 6:Chunk\-size ablation for long\-sequence conditional VNE on GPT\-Neo with chunk size=1=1\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling, using prefix lengths\{8,16,32\}\\\{8,16,32\\\}and continuation lengths\{16,32,64\}\\\{16,32,64\\\}\.PrefixContinuationTrainingGreedyNucleusAncestral881616437\.3437\.3337\.3337\.3400\.7400\.7410\.2410\.2883232474\.5474\.5351\.8351\.8425\.2425\.2436\.9436\.9886464503\.7503\.7365\.3365\.3446\.2446\.2462\.3462\.316161616431\.3431\.3380\.3380\.3409\.3409\.3415\.4415\.416163232455\.6455\.6389\.7389\.7422\.9422\.9432\.9432\.916166464477\.9477\.9400\.8400\.8435\.4435\.4447\.5447\.532321616432\.9432\.9406\.6406\.6420\.3420\.3424\.9424\.932323232446\.3446\.3409\.3409\.3427\.8427\.8433\.7433\.732326464462\.1462\.1413\.6413\.6434\.4434\.4442\.3442\.3Table 7:Chunk\-size ablation for long\-sequence conditional VNE on GPT\-Neo with chunk size=4=4\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling, using prefix lengths\{8,16,32\}\\\{8,16,32\\\}and continuation lengths\{16,32,64\}\\\{16,32,64\\\}\.PrefixContinuationTrainingGreedyNucleusAncestral881616454\.7454\.7382\.5382\.5430\.5430\.5433\.4433\.4883232515\.0515\.0415\.2415\.2479\.8479\.8482\.0482\.0886464572\.7572\.7435\.4435\.4518\.1518\.1525\.0525\.016161616455\.0455\.0428\.2428\.2443\.1443\.1443\.1443\.116163232496\.6496\.6454\.4454\.4475\.9475\.9477\.6477\.616166464541\.3541\.3474\.1474\.1503\.7503\.7507\.4507\.432321616459\.8459\.8450\.8450\.8455\.8455\.8456\.5456\.532323232484\.1484\.1468\.0468\.0475\.5475\.5475\.4475\.432326464514\.8514\.8483\.0483\.0493\.9493\.9495\.1495\.1
### E\.7Comparison with Temperature Scaling

A natural question is whether the same diversity gain can be achieved by simply raising the sampling temperature\. Theorem[1](https://arxiv.org/html/2609.02275#Thmtheorem1)does not necessarily apply to temperature scaling, because changing the temperature need not produce the closest distribution satisfying the entropy constraint\. To examine this baseline empirically, we repeated the OLMo experiment using ancestral sampling with different temperatures\. Increasing the temperature raises conditional entropy but does not eliminate the gap within the evaluated range; even at a relatively high temperature, the conditional VNE remains below the training\-data value\. We then searched for a temperature that matches the training\-data conditional entropy and found that approximatelyT≈1\.9T\\approx 1\.9achieves this target\. However, matching entropy alone did not match the underlying data distribution\.

Table[8](https://arxiv.org/html/2609.02275#A5.T8)reports two independent quality\-oriented measures: precision\[[69](https://arxiv.org/html/2609.02275#bib.bib65)\], measuring generated probability mass within the estimated support of the training continuations, and external conditional negative log likelihood, computed using an independent language model conditioned on the same prefix\. AlthoughT≈1\.9T\\approx 1\.9approximately matches the reference entropy, it yields worse precision and worse external NLL than the training data\. In contrast, the proposed projection achieves substantially higher precision and lower external NLL\. These results support the distinction motivated by Theorem[1](https://arxiv.org/html/2609.02275#Thmtheorem1): simply matching entropy is insufficient, and the adjustment should be performed through the closest\-distribution projection rather than an arbitrary entropy increase\.

Table 8:Comparison of temperature scaling and the proposed entropy projection on OLMo\. Precision measures generated probability mass within the estimated support of the training continuations\[[69](https://arxiv.org/html/2609.02275#bib.bib65)\]\. External conditional NLL is computed using an independent language model conditioned on the same prefix\. The temperatureT≈1\.9T\\approx 1\.9is tuned to approximately match the training\-data conditional entropy\.MethodPrecision↑\\uparrowExt\. Cond\. NLL↓\\downarrowTraining data1\.0001\.0004\.6794\.679TemperatureT≈1\.9T\\approx 1\.90\.9300\.9304\.9384\.938Entropy projection \(ours\)0\.9790\.9793\.7133\.713
### E\.8Qualitative Probe of the Reweighting Behavior

To inspect how the projection redistributes probability mass, we conduct a controlled probe\. We take two prompts, “A vibrant city in Northern America is \_\_\_” and “A renowned celebrity is \_\_\_”, sample10001000continuations for each, and reweight them using the conditional entropy projection\. The per\-output mass shift reveals a clean de\-peaking pattern: weight leaves the over\-represented dominant modes and moves to equally valid but under\-represented alternatives\. Table[9](https://arxiv.org/html/2609.02275#A5.T9)reports the before/after weights for the top dominant modes and selected up\-weighted tail candidates\. Quantitatively, the discrete entropy and the corresponding effective number of distinct continuations both increase: for the city probe, the discrete entropy rises from2\.5872\.587to2\.8532\.853and the effective number of distinct continuations rises from13\.2913\.29to17\.3417\.34; for the celebrity probe, the entropy rises from3\.9793\.979to4\.3984\.398and the effective number rises from53\.4653\.46to81\.2581\.25\. These examples make the practical effect of the projection transparent: it reduces excessive concentration on dominant generated answers and increases the representation of plausible alternatives already present in the candidate pool, rather than promoting low\-quality or implausible outputs\.

Table 9:Qualitative probe of the entropy projection on two prompts\. We report the probability mass before and after projection for the top down\-weighted dominant modes and selected up\-weighted tail candidates\. “Before” and “After” denote the mass assigned by the original uniformly weighted candidates and the reweighted candidates, respectively\.PromptDown\-weighted \(dominant modes\): Before→\\toAfterUp\-weighted \(plausible tail\): Before→\\toAfter“A vibrant city in Northern America is \_\_\_”Toronto:0\.206→0\.1770\.206\\to 0\.177 Vancouver:0\.198→0\.1370\.198\\to 0\.137 New York:0\.096→0\.0680\.096\\to 0\.068Miami:0\.018→0\.0460\.018\\to 0\.046 Denver:0\.020→0\.0350\.020\\to 0\.035 Philadelphia:0\.010→0\.0220\.010\\to 0\.022“A renowned celebrity is \_\_\_”Oprah Winfrey:0\.134→0\.0640\.134\\to 0\.064 Madonna:0\.068→0\.0360\.068\\to 0\.036 Michael Jackson:0\.052→0\.0200\.052\\to 0\.020 Tom Cruise:0\.032→0\.0200\.032\\to 0\.020Jackie Chan:0\.010→0\.0190\.010\\to 0\.019 Kate Middleton:0\.007→0\.0130\.007\\to 0\.013 Queen Victoria:0\.004→0\.0150\.004\\to 0\.015 J\. K\. Rowling:0\.002→0\.0120\.002\\to 0\.012

### E\.9Lexical Audit of Conditional Diversity

To examine whether the observed conditional\-entropy ordering also reflects in a directly human\-interpretable form, we perform a word\-level audit using the same OLMo setting as the main text with20,00020\{,\}000samples\. We count exact, case\-insensitive occurrences from three fixed and enumerable semantic lexicons: countries \(192192terms\), sports \(8080terms\), and occupations \(118118terms\)\. For each category, we normalize the term counts into a categorical distribution and report the number of distinct terms observed as well as the effective number of equally probable lexical choices2H2^\{H\}, whereHHis the Shannon entropy of the normalized distribution in bits\.

Table[10](https://arxiv.org/html/2609.02275#A5.T10)reports the results\. Importantly, all three independent lexical audits recover the same ordering as conditional VNE: training\>\>ancestral\>\>nucleus\>\>greedy\. The effect is also concrete at the level of frequencies\. For example, in country mentions, greedy decoding assigns20\.0%20\.0\\%of its mass to “UK” and19\.8%19\.8\\%to “United States”, compared with only8\.4%8\.4\\%and4\.3%4\.3\\%in the training continuations\. This indicates that the conditional VNE gap corresponds to a measurable shortfall in the semantic variety of generated continuations, not merely a change in the kernel\-derived score\.

Table 10:Lexical audit of conditional diversity on OLMo at sample size2020K\. For each semantic category, we report the number of distinct terms observed and the effective number of equally probable lexical choices2H2^\{H\}, whereHHis the Shannon entropy of the normalized term\-frequency distribution in bits\. The ordering across all three categories matches the conditional VNE ordering: training\>\>ancestral\>\>nucleus\>\>greedy\.GroupCond\.VNECountries: distinctCountries:2H2^\{H\}Sports: distinctSports:2H2^\{H\}Occupations: distinctOccupations:2H2^\{H\}Training data414\.2414\.2959554\.454\.4353522\.422\.4757549\.949\.9Greedy369\.4369\.4828226\.926\.9202013\.613\.6666635\.935\.9Nucleus390\.8390\.8828237\.337\.3252517\.917\.9696942\.642\.6Ancestral394\.9394\.9838341\.941\.9242418\.418\.4686844\.744\.7

### E\.10Conditional Diversity Gaps in Larger Language Models

The main experiments evaluate approximately11B\-parameter models\. To examine whether the observed gap is restricted to smaller or earlier models, we additionally evaluate OLMo\-3\-7B and extend the Pythia experiments to2\.82\.8B and6\.96\.9B parameters under the same setting\. Tables[11](https://arxiv.org/html/2609.02275#A5.T11)and[12](https://arxiv.org/html/2609.02275#A5.T12)report the exponentiated conditional VNE across sample sizes from2\.52\.5K to2020K\.

For OLMo\-3\-7B, the relative reductions at2020K are approximately33\.0%33\.0\\%,16\.2%16\.2\\%, and12\.0%12\.0\\%under greedy, nucleus, and ancestral decoding, respectively\. For Pythia\-2\.8B and Pythia\-6\.9B, the corresponding gaps also remain substantial\. These results indicate that the conditional output\-range gap is not restricted to the earlier approximately11B\-parameter models and does not disappear merely by increasing model size\.

Table 11:Conditional diversity scores for OLMo\-3\-7B across sample sizes\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling\.Group2\.5K5K7\.5K10K12\.5K15K17\.5K20KTraining data86\.9286\.92138\.84138\.84183\.04183\.04220\.30220\.30254\.52254\.52284\.20284\.20312\.61312\.61339\.17339\.17Greedy71\.9271\.92107\.87107\.87135\.95135\.95158\.50158\.50178\.47178\.47196\.05196\.05212\.69212\.69227\.40227\.40Nucleus79\.5579\.55125\.16125\.16161\.26161\.26191\.71191\.71217\.79217\.79241\.40241\.40264\.24264\.24284\.58284\.58Ancestral81\.8081\.80128\.47128\.47166\.06166\.06198\.25198\.25226\.70226\.70252\.94252\.94276\.80276\.80298\.40298\.40

Table 12:Conditional diversity scores for Pythia\-2\.8B and Pythia\-6\.9B across sample sizes\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling\.ModelGroup2\.5K5K7\.5K10K12\.5K15K17\.5K20KPythia\-2\.8BTraining data72\.6072\.60114\.80114\.80149\.39149\.39177\.89177\.89203\.14203\.14225\.89225\.89247\.28247\.28265\.87265\.87Greedy58\.4958\.4986\.6986\.69108\.65108\.65125\.62125\.62140\.05140\.05152\.45152\.45163\.21163\.21172\.54172\.54Nucleus66\.0766\.07101\.37101\.37131\.11131\.11155\.42155\.42176\.17176\.17194\.62194\.62211\.69211\.69226\.57226\.57Ancestral68\.9868\.98105\.77105\.77136\.45136\.45161\.83161\.83182\.24182\.24201\.95201\.95220\.13220\.13236\.08236\.08Pythia\-6\.9BTraining data72\.6072\.60114\.80114\.80149\.39149\.39177\.89177\.89203\.14203\.14225\.89225\.89247\.28247\.28265\.87265\.87Greedy58\.1858\.1886\.9586\.95109\.39109\.39126\.75126\.75141\.22141\.22153\.96153\.96164\.73164\.73174\.13174\.13Nucleus67\.9467\.94105\.16105\.16135\.18135\.18159\.59159\.59179\.85179\.85199\.01199\.01215\.33215\.33230\.26230\.26Ancestral68\.7968\.79106\.71106\.71138\.51138\.51164\.17164\.17185\.90185\.90206\.12206\.12224\.19224\.19239\.50239\.50

Table 13:Ablation study with T5 embeddings for kernel\-based conditional diversity metrics at sample size 20K\. The standard deviations of conditional VNE and conditional RKE are below0\.010\.01across 5 independent runs\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE49\.74–Exp\.Cond\.RKE4\.25–Distinct\-20\.769–Distinct\-30\.964–Greedy DecodingExp\.Cond\.VNE28\.2221\.52Exp\.Cond\.RKE3\.340\.91Distinct\-20\.3370\.432Distinct\-30\.5380\.426Nucleus SamplingExp\.Cond\.VNE37\.9411\.81Exp\.Cond\.RKE3\.740\.51Distinct\-20\.5880\.181Distinct\-30\.8890\.075Ancestral SamplingExp\.Cond\.VNE39\.6410\.11Exp\.Cond\.RKE3\.810\.44Distinct\-20\.6270\.142Distinct\-30\.9170\.047PythiaTraining DataExp\.Cond\.VNE43\.06–Exp\.Cond\.RKE4\.15–Distinct\-20\.755–Distinct\-30\.930–Greedy DecodingExp\.Cond\.VNE26\.3616\.70Exp\.Cond\.RKE3\.430\.72Distinct\-20\.3250\.430Distinct\-30\.4970\.433Nucleus SamplingExp\.Cond\.VNE34\.318\.75Exp\.Cond\.RKE3\.760\.39Distinct\-20\.5970\.158Distinct\-30\.8660\.064Ancestral SamplingExp\.Cond\.VNE36\.126\.94Exp\.Cond\.RKE3\.830\.32Distinct\-20\.6320\.123Distinct\-30\.8900\.040GPT\-NeoTraining DataExp\.Cond\.VNE43\.07–Exp\.Cond\.RKE4\.15–Distinct\-20\.755–Distinct\-30\.929–Greedy DecodingExp\.Cond\.VNE27\.1814\.49Exp\.Cond\.RKE3\.510\.64Distinct\-20\.3440\.411Distinct\-30\.5100\.419Nucleus SamplingExp\.Cond\.VNE34\.217\.46Exp\.Cond\.RKE3\.780\.37Distinct\-20\.5990\.156Distinct\-30\.8560\.073Ancestral SamplingExp\.Cond\.VNE36\.045\.63Exp\.Cond\.RKE3\.850\.30Distinct\-20\.6380\.117Distinct\-30\.8820\.047Table 14:Ablation study with a degree\-3 polynomial kernel for kernel\-based conditional diversity metrics at sample size 20K\. The standard deviations of conditional VNE and conditional RKE are below0\.010\.01across 5 independent runs\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE8\.71–Exp\.Cond\.RKE2\.54–Distinct\-20\.769–Distinct\-30\.964–Greedy DecodingExp\.Cond\.VNE6\.282\.43Exp\.Cond\.RKE2\.410\.13Distinct\-20\.3370\.432Distinct\-30\.5380\.426Nucleus SamplingExp\.Cond\.VNE7\.561\.15Exp\.Cond\.RKE2\.460\.08Distinct\-20\.5880\.181Distinct\-30\.8890\.075Ancestral SamplingExp\.Cond\.VNE7\.681\.03Exp\.Cond\.RKE2\.470\.07Distinct\-20\.6270\.142Distinct\-30\.9170\.047PythiaTraining DataExp\.Cond\.VNE7\.64–Exp\.Cond\.RKE2\.50–Distinct\-20\.755–Distinct\-30\.930–Greedy DecodingExp\.Cond\.VNE5\.821\.82Exp\.Cond\.RKE2\.260\.24Distinct\-20\.3250\.430Distinct\-30\.4970\.433Nucleus SamplingExp\.Cond\.VNE7\.140\.50Exp\.Cond\.RKE2\.360\.14Distinct\-20\.5970\.158Distinct\-30\.8660\.064Ancestral SamplingExp\.Cond\.VNE7\.210\.43Exp\.Cond\.RKE2\.370\.13Distinct\-20\.6320\.123Distinct\-30\.8900\.040GPT\-NeoTraining DataExp\.Cond\.VNE7\.64–Exp\.Cond\.RKE2\.50–Distinct\-20\.755–Distinct\-30\.929–Greedy DecodingExp\.Cond\.VNE5\.721\.92Exp\.Cond\.RKE2\.240\.26Distinct\-20\.3440\.411Distinct\-30\.5100\.419Nucleus SamplingExp\.Cond\.VNE7\.010\.63Exp\.Cond\.RKE2\.330\.17Distinct\-20\.5990\.156Distinct\-30\.8560\.073Ancestral SamplingExp\.Cond\.VNE7\.050\.59Exp\.Cond\.RKE2\.340\.16Distinct\-20\.6380\.117Distinct\-30\.8820\.047Table 15:Ablation study with a cosine kernel for kernel\-based conditional diversity metrics at sample size 20K\. The standard deviations of conditional VNE and conditional RKE are below0\.010\.01across 5 independent runs\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE2\.72–Exp\.Cond\.RKE1\.35–Distinct\-20\.769–Distinct\-30\.964–Greedy DecodingExp\.Cond\.VNE2\.150\.57Exp\.Cond\.RKE1\.240\.11Distinct\-20\.3370\.432Distinct\-30\.5380\.426Nucleus SamplingExp\.Cond\.VNE2\.380\.34Exp\.Cond\.RKE1\.280\.07Distinct\-20\.5880\.181Distinct\-30\.8890\.075Ancestral SamplingExp\.Cond\.VNE2\.430\.29Exp\.Cond\.RKE1\.290\.06Distinct\-20\.6270\.142Distinct\-30\.9170\.047PythiaTraining DataExp\.Cond\.VNE2\.67–Exp\.Cond\.RKE1\.34–Distinct\-20\.755–Distinct\-30\.930–Greedy DecodingExp\.Cond\.VNE2\.190\.47Exp\.Cond\.RKE1\.250\.09Distinct\-20\.3250\.430Distinct\-30\.4970\.433Nucleus SamplingExp\.Cond\.VNE2\.430\.24Exp\.Cond\.RKE1\.290\.05Distinct\-20\.5970\.158Distinct\-30\.8660\.064Ancestral SamplingExp\.Cond\.VNE2\.460\.20Exp\.Cond\.RKE1\.300\.04Distinct\-20\.6320\.123Distinct\-30\.8900\.040GPT\-NeoTraining DataExp\.Cond\.VNE2\.67–Exp\.Cond\.RKE1\.34–Distinct\-20\.755–Distinct\-30\.929–Greedy DecodingExp\.Cond\.VNE2\.270\.41Exp\.Cond\.RKE1\.270\.07Distinct\-20\.3440\.411Distinct\-30\.5100\.419Nucleus SamplingExp\.Cond\.VNE2\.460\.22Exp\.Cond\.RKE1\.290\.05Distinct\-20\.5990\.156Distinct\-30\.8560\.073Ancestral SamplingExp\.Cond\.VNE2\.500\.17Exp\.Cond\.RKE1\.300\.04Distinct\-20\.6380\.117Distinct\-30\.8820\.047Table 16:Ablation study on prefix token length, with prefix token length set to 4 at sample size 20K\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE310\.48–Exp\.Cond\.RKE100\.06–Distinct\-20\.709–Distinct\-30\.947–Greedy DecodingExp\.Cond\.VNE245\.7264\.77Exp\.Cond\.RKE72\.0827\.98Distinct\-20\.3230\.386Distinct\-30\.5450\.401Nucleus SamplingExp\.Cond\.VNE281\.2829\.20Exp\.Cond\.RKE83\.6716\.39Distinct\-20\.5340\.175Distinct\-30\.8580\.089Ancestral SamplingExp\.Cond\.VNE287\.6322\.85Exp\.Cond\.RKE86\.2913\.77Distinct\-20\.5720\.137Distinct\-30\.8920\.055PythiaTraining DataExp\.Cond\.VNE240\.47–Exp\.Cond\.RKE84\.79–Distinct\-20\.708–Distinct\-30\.918–Greedy DecodingExp\.Cond\.VNE193\.5846\.88Exp\.Cond\.RKE67\.2817\.51Distinct\-20\.3100\.398Distinct\-30\.5040\.414Nucleus SamplingExp\.Cond\.VNE223\.8616\.61Exp\.Cond\.RKE76\.508\.29Distinct\-20\.5370\.171Distinct\-30\.8340\.084Ancestral SamplingExp\.Cond\.VNE229\.1111\.36Exp\.Cond\.RKE78\.606\.19Distinct\-20\.5780\.130Distinct\-30\.8650\.053GPT\-NeoTraining DataExp\.Cond\.VNE240\.47–Exp\.Cond\.RKE84\.79–Distinct\-20\.709–Distinct\-30\.917–Greedy DecodingExp\.Cond\.VNE194\.9845\.49Exp\.Cond\.RKE70\.0114\.78Distinct\-20\.3300\.379Distinct\-30\.5100\.407Nucleus SamplingExp\.Cond\.VNE223\.7416\.73Exp\.Cond\.RKE77\.727\.07Distinct\-20\.5470\.162Distinct\-30\.8310\.086Ancestral SamplingExp\.Cond\.VNE229\.4511\.02Exp\.Cond\.RKE79\.695\.10Distinct\-20\.5820\.127Distinct\-30\.8630\.055Table 17:Ablation study on sample size for diversity metrics across models and generation strategies at sample size 10K\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE221\.31–Exp\.Cond\.RKE67\.33–Distinct\-20\.769–Distinct\-30\.964–Greedy DecodingExp\.Cond\.VNE154\.9766\.34Exp\.Cond\.RKE46\.1921\.14Distinct\-20\.3370\.432Distinct\-30\.5380\.426Nucleus SamplingExp\.Cond\.VNE193\.5827\.73Exp\.Cond\.RKE55\.4411\.89Distinct\-20\.5880\.181Distinct\-30\.8890\.075Ancestral SamplingExp\.Cond\.VNE198\.5122\.80Exp\.Cond\.RKE56\.9210\.41Distinct\-20\.6270\.142Distinct\-30\.9170\.047PythiaTraining DataExp\.Cond\.VNE176\.51–Exp\.Cond\.RKE52\.89–Distinct\-20\.755–Distinct\-30\.930–Greedy DecodingExp\.Cond\.VNE127\.4649\.05Exp\.Cond\.RKE40\.3712\.52Distinct\-20\.3250\.430Distinct\-30\.4970\.433Nucleus SamplingExp\.Cond\.VNE159\.3617\.15Exp\.Cond\.RKE46\.846\.05Distinct\-20\.5970\.158Distinct\-30\.8660\.064Ancestral SamplingExp\.Cond\.VNE165\.1011\.41Exp\.Cond\.RKE48\.274\.62Distinct\-20\.6320\.123Distinct\-30\.8900\.040GPT\-NeoTraining DataExp\.Cond\.VNE175\.98–Exp\.Cond\.RKE52\.60–Distinct\-20\.755–Distinct\-30\.929–Greedy DecodingExp\.Cond\.VNE130\.1145\.87Exp\.Cond\.RKE43\.039\.57Distinct\-20\.3440\.411Distinct\-30\.5100\.419Nucleus SamplingExp\.Cond\.VNE159\.3516\.63Exp\.Cond\.RKE47\.934\.67Distinct\-20\.5990\.156Distinct\-30\.8560\.073Ancestral SamplingExp\.Cond\.VNE163\.3912\.59Exp\.Cond\.RKE48\.713\.89Distinct\-20\.6380\.117Distinct\-30\.8820\.047Table 18:Ablation study on sample size for diversity metrics across models and generation strategies at sample size 20K\. The gap is computed asΔ=Metrictrain−Metricgroup\\Delta=\\mathrm\{Metric\}\_\{\\mathrm\{train\}\}\-\\mathrm\{Metric\}\_\{\\mathrm\{group\}\}\. Positive gaps indicate lower diversity than the training data\.ModelGroupMetricValue𝚫\\bm\{\\Delta\}OLMoTraining DataExp\.Cond\.VNE338\.58–Exp\.Cond\.RKE67\.43–Distinct\-20\.709–Distinct\-30\.946–Greedy DecodingExp\.Cond\.VNE218\.88119\.70Exp\.Cond\.RKE46\.1721\.26Distinct\-20\.2760\.433Distinct\-30\.4650\.481Nucleus SamplingExp\.Cond\.VNE287\.3151\.27Exp\.Cond\.RKE55\.0812\.35Distinct\-20\.5160\.193Distinct\-30\.8470\.099Ancestral SamplingExp\.Cond\.VNE297\.3141\.27Exp\.Cond\.RKE56\.8110\.62Distinct\-20\.5580\.151Distinct\-30\.8830\.063PythiaTraining DataExp\.Cond\.VNE262\.55–Exp\.Cond\.RKE53\.44–Distinct\-20\.701–Distinct\-30\.913–Greedy DecodingExp\.Cond\.VNE176\.3986\.16Exp\.Cond\.RKE40\.9812\.46Distinct\-20\.2660\.435Distinct\-30\.4230\.490Nucleus SamplingExp\.Cond\.VNE232\.2430\.31Exp\.Cond\.RKE47\.515\.93Distinct\-20\.5250\.176Distinct\-30\.8260\.087Ancestral SamplingExp\.Cond\.VNE241\.7920\.76Exp\.Cond\.RKE49\.004\.44Distinct\-20\.5640\.137Distinct\-30\.8570\.056GPT\-NeoTraining DataExp\.Cond\.VNE260\.01–Exp\.Cond\.RKE52\.98–Distinct\-20\.701–Distinct\-30\.912–Greedy DecodingExp\.Cond\.VNE179\.3680\.65Exp\.Cond\.RKE43\.349\.64Distinct\-20\.2840\.417Distinct\-30\.4370\.475Nucleus SamplingExp\.Cond\.VNE229\.6230\.39Exp\.Cond\.RKE48\.484\.50Distinct\-20\.5260\.175Distinct\-30\.8120\.100Ancestral SamplingExp\.Cond\.VNE238\.5021\.51Exp\.Cond\.RKE49\.593\.39Distinct\-20\.5700\.131Distinct\-30\.8480\.064Table 19:Full prefix and continuation length ablation for LLM conditional VNE\. We report the exponential of conditional VNE for the training reference and generated continuations under greedy decoding, nucleus sampling, and ancestral sampling\. The average gap is computed across the three decoding strategies\.ModelPrefixContinuationTrainingGreedyNucleusAncestralAvg\. GapOLMo3322338\.58218\.88287\.31297\.3070\.753333617\.63404\.37564\.52581\.67100\.783344741\.13517\.42716\.87726\.3587\.584422310\.48245\.72281\.27287\.6338\.944433470\.69388\.95450\.11455\.5239\.164444532\.24463\.86525\.02529\.5126\.11Pythia3322262\.55176\.39232\.24241\.7945\.743333492\.00301\.65439\.49456\.9192\.653344604\.57397\.22575\.02589\.2984\.064422240\.46193\.58223\.85229\.1024\.954433370\.43285\.74345\.38352\.8842\.434444423\.95342\.08408\.47414\.8035\.50GPT\-Neo3322262\.55179\.36229\.62238\.4946\.733333491\.53301\.45432\.98452\.0996\.023344607\.21391\.38561\.90579\.9796\.134422240\.47194\.97223\.74229\.4524\.424433371\.87284\.75341\.55352\.0645\.754444426\.57339\.42404\.99412\.1941\.04

![Refer to caption](https://arxiv.org/html/2609.02275v1/imagenet_seperate.png)

Figure 2:Conditional diversity gap on class\-conditioned ImageNet generation\. We plot conditional VNE as a function of sample size for the ImageNet training\-data reference and generated samples from ADM, DiT\-XL\-2, LDM, and BigGAN under matched class labels\. Across all evaluated sample sizes, the reference ImageNet data achieves higher conditional VNE than the generated samples, and the gap generally increases as the sample size grows\.![Refer to caption](https://arxiv.org/html/2609.02275v1/coco_seperate.png)

Figure 3:Conditional diversity gap on text\-conditioned MS\-COCO generation\. We plot conditional VNE as a function of sample size for the MS\-COCO real\-data reference and generated samples from SDXL, PixArt, U\-ViT, and Deep U\-ViT under matched text captions\. Across all evaluated sample sizes, the real\-data reference achieves higher conditional VNE than the generated samples, and the gap generally becomes larger with increasing sample size\.Table 20:Conditional diversity scores for class\-conditioned ImageNet generation across sample sizes\. Rows correspond to fixed sample sizes, and columns correspond to the ImageNet reference data and evaluated generative models\. We report the exponential of conditional VNE as mean±\\pmstandard deviation over 5 independent runs\.Sample SizeImageNet LDM ADM BigGAN DiT\-XL\-2 2\.5K17\.53±0\.0917\.53\\pm 0\.0916\.66±0\.1116\.66\\pm 0\.115\.78±0\.025\.78\\pm 0\.025\.32±0\.035\.32\\pm 0\.0316\.96±0\.0616\.96\\pm 0\.065K26\.41±0\.1226\.41\\pm 0\.1224\.50±0\.0824\.50\\pm 0\.089\.20±0\.049\.20\\pm 0\.048\.02±0\.058\.02\\pm 0\.0525\.14±0\.1025\.14\\pm 0\.107\.5K33\.01±0\.1033\.01\\pm 0\.1030\.27±0\.1930\.27\\pm 0\.1912\.07±0\.0312\.07\\pm 0\.0310\.14±0\.0610\.14\\pm 0\.0631\.25±0\.1731\.25\\pm 0\.1710K38\.57±0\.0838\.57\\pm 0\.0834\.90±0\.1934\.90\\pm 0\.1914\.56±0\.0414\.56\\pm 0\.0411\.89±0\.0511\.89\\pm 0\.0536\.18±0\.2236\.18\\pm 0\.2212\.5K43\.15±0\.1843\.15\\pm 0\.1838\.75±0\.1838\.75\\pm 0\.1816\.80±0\.0416\.80\\pm 0\.0413\.40±0\.0513\.40\\pm 0\.0540\.24±0\.2040\.24\\pm 0\.2015K47\.23±0\.2347\.23\\pm 0\.2342\.14±0\.1742\.14\\pm 0\.1718\.84±0\.0618\.84\\pm 0\.0614\.75±0\.0514\.75\\pm 0\.0543\.86±0\.1943\.86\\pm 0\.1917\.5K50\.63±0\.1750\.63\\pm 0\.1745\.13±0\.1745\.13\\pm 0\.1720\.72±0\.0420\.72\\pm 0\.0415\.95±0\.0415\.95\\pm 0\.0447\.08±0\.2447\.08\\pm 0\.2420K54\.15±0\.1454\.15\\pm 0\.1447\.86±0\.1747\.86\\pm 0\.1722\.47±0\.0322\.47\\pm 0\.0317\.04±0\.0217\.04\\pm 0\.0249\.95±0\.2149\.95\\pm 0\.21

Table 21:Conditional diversity scores for text\-conditioned MS\-COCO generation across sample sizes\. Rows correspond to fixed sample sizes, and columns correspond to the MS\-COCO reference data and evaluated text\-to\-image generative models\. We report the exponential of conditional VNE as mean±\\pmstandard deviation over independent runs\.Sample SizeMS\-COCO U\-ViT Deep S/2 U\-ViT S/2 SDXL PixArt\-α\\alpha 2\.5K13\.88±0\.0713\.88\\pm 0\.0711\.63±0\.0611\.63\\pm 0\.0611\.51±0\.0411\.51\\pm 0\.0412\.02±0\.0212\.02\\pm 0\.0210\.87±0\.0410\.87\\pm 0\.045K19\.91±0\.0419\.91\\pm 0\.0416\.07±0\.0616\.07\\pm 0\.0615\.89±0\.0915\.89\\pm 0\.0916\.49±0\.0316\.49\\pm 0\.0314\.54±0\.0514\.54\\pm 0\.057\.5K24\.62±0\.0524\.62\\pm 0\.0519\.33±0\.0719\.33\\pm 0\.0719\.09±0\.0919\.09\\pm 0\.0919\.75±0\.1419\.75\\pm 0\.1417\.15±0\.0317\.15\\pm 0\.0310K28\.51±0\.0628\.51\\pm 0\.0622\.03±0\.0522\.03\\pm 0\.0521\.73±0\.0521\.73\\pm 0\.0522\.51±0\.0922\.51\\pm 0\.0919\.30±0\.0119\.30\\pm 0\.0112\.5K31\.93±0\.0631\.93\\pm 0\.0624\.36±0\.0524\.36\\pm 0\.0524\.03±0\.0624\.03\\pm 0\.0624\.76±0\.0724\.76\\pm 0\.0721\.06±0\.0321\.06\\pm 0\.0315K35\.03±0\.0335\.03\\pm 0\.0326\.38±0\.0626\.38\\pm 0\.0626\.00±0\.0626\.00\\pm 0\.0626\.76±0\.0726\.76\\pm 0\.0722\.62±0\.0222\.62\\pm 0\.0217\.5K37\.92±0\.0437\.92\\pm 0\.0428\.24±0\.0528\.24\\pm 0\.0527\.84±0\.0527\.84\\pm 0\.0528\.58±0\.0428\.58\\pm 0\.0424\.01±0\.0024\.01\\pm 0\.0020K40\.52±0\.0340\.52\\pm 0\.0329\.94±0\.0329\.94\\pm 0\.0329\.50±0\.0429\.50\\pm 0\.0430\.25±0\.0430\.25\\pm 0\.0425\.25±0\.0125\.25\\pm 0\.01