双重预算约束下的稀疏文本扩散：上下文连通性与专家激活

# Sparse Text Diffusion under Dual Budgets:Context Connectivity and Expert Activation
Source: [https://arxiv.org/html/2610.41487](https://arxiv.org/html/2610.41487)
Chadbourne DavisAffiliation:Suffolk UniversityPaul RajAffiliation:Texas A&M University\-Commerce

###### Abstract

Long\-context diffusion repeatedly pays for both token interaction and feed\-forward computation\. We formulate these costs as two separate budgets: the edges available to an attention graph and the experts activated for each token\. This view connects local, dilated, and global attention with conditional expert computation without assuming that sparsity alone preserves document\-level information\. We specify normalized masked attention, distinguish denoising depth from sampling iterations, and derive graph and compute bounds\. A reported sparse\-expert configuration, denoted DualBudget, is analyzed across scientific summarization, multi\-hop question answering, dialogue, and sentence\-pair prediction\. All supplied table measurements are retained\. The length study shows gains over the listed baselines through 16K, together with measurable degradation as length increases\. Ablations reveal mixed changes across the three reported scores, while the cost record indicates a 20% reduction relative to DiffuSeq under an unspecified timing convention\. The analysis separates these observations from unmeasured claims about training speed, absorbing states, global coherence, and deployment memory\.

## 1Introduction

Denoising diffusion and score\-based modeling provide iterative generative principles\[[1](https://arxiv.org/html/2610.41487#bib.bib1),[2](https://arxiv.org/html/2610.41487#bib.bib2)\]\. DiffuSeq formulates sequence\-to\-sequence generation\[[3](https://arxiv.org/html/2610.41487#bib.bib3)\], with Transformer attention providing interactions within each denoiser\[[4](https://arxiv.org/html/2610.41487#bib.bib4)\]\. Longer sequences increase the cost of every update: dense query–key interactions grow quadratically, while expanded feed\-forward capacity remains expensive even under restricted attention\.

These costs motivate joint accounting of*context connectivity*and*expert activation*\. Longformer combines local and task\-motivated global attention\[[5](https://arxiv.org/html/2610.41487#bib.bib5)\]\. Sparse expert layers select which parameterized functions process each token\[[6](https://arxiv.org/html/2610.41487#bib.bib6),[7](https://arxiv.org/html/2610.41487#bib.bib7)\]\. The former controls information access; the latter controls conditional computation\. Reducing one need not reduce the other: a short neighborhood can feed an expensive dense sublayer, while sparse experts can consume representations built through dense attention\.

Related mechanisms already coexist\. DrDiff combines expert scheduling, hierarchical sparse attention, and accelerated sampling for long\-text diffusion\[[8](https://arxiv.org/html/2610.41487#bib.bib8)\]; DiffuSeq\-v2 introduces soft absorbing states and continuous\-space solvers\[[9](https://arxiv.org/html/2610.41487#bib.bib9)\]\. We formulate common budgets to analyze the reported sparse\-expert comparison, rather than claim these component families originate here\. The formulation exposes separate requirements: attention must preserve useful paths, routing must specify activated experts and dispatch costs, and sampling must identify its state and prediction convention\.

At the task level, agents also allocate scarce reasoning capacity\. Markets of Minds evaluates expert assistance through marginal benefit and opportunity cost under shared budgets\[[10](https://arxiv.org/html/2610.41487#bib.bib10)\]\. This motivates an agent\-facing interpretation of the two budgets: a planner can assign priorities to document spans, while the denoiser determines which token connections and expert activations realize them\. Span\-level coordination defines an extension interface; the experiments below concern the token\-level denoiser\.

We develop this perspective in three stages\. First, we formalize a sparse attention graph and a selected\-expert layer with explicit normalization\. Second, we derive interaction and active\-compute bounds and explain their connectivity and memory limits\. Third, we analyze the supplied measurements, retaining aggregate scores, length sweeps, ablations, and cost values as separate records\. The resulting account identifies which efficiency and quality conclusions follow from those records and which require additional instrumentation\.

![Refer to caption](https://arxiv.org/html/2610.41487v1/dual_budget_denoiser.png)Figure 1:One denoising evaluation and its two computational budgets\. \(a\) The denoiser predicts a clean state for the sampling update\. \(b\) Sparse token interactions\. \(c\) Selected expert computation\. Highlighting is schematic and specifies no measured activation count\.
## 2Related Work

### 2\.1Diffusion Representations and Sampling

Diffusion\-LM denoises continuous word representations, while SSD\-LM generates text semi\-autoregressively in simplex space\[[11](https://arxiv.org/html/2610.41487#bib.bib11),[12](https://arxiv.org/html/2610.41487#bib.bib12)\]\. Structured discrete transitions\[[13](https://arxiv.org/html/2610.41487#bib.bib13)\], score\-entropy estimation in SEDD\[[14](https://arxiv.org/html/2610.41487#bib.bib14)\], and masked objectives in MDLM\[[15](https://arxiv.org/html/2610.41487#bib.bib15)\]offer different treatments of discrete language\. LLaDA extends masked diffusion to large\-scale language modeling\[[16](https://arxiv.org/html/2610.41487#bib.bib16)\]; soft\-masked corruption provides another training design\[[17](https://arxiv.org/html/2610.41487#bib.bib17)\]\. These choices concern representations and objectives, without fixing attention connectivity or expert dispatch\.

Counterfactual adaptation in CF\-VLM and semantic caption refinement\[[18](https://arxiv.org/html/2610.41487#bib.bib18),[19](https://arxiv.org/html/2610.41487#bib.bib19)\]operate at other levels\. Autoregressive diffusion explores prediction decomposition\[[20](https://arxiv.org/html/2610.41487#bib.bib20)\]; DPM\-Solver\+\+ specifies accelerated numerical integration\[[21](https://arxiv.org/html/2610.41487#bib.bib21)\]\. Failure\-driven workflow refinement concerns procedural organization\[[22](https://arxiv.org/html/2610.41487#bib.bib22)\]\. Their distinct roles motivate separating corruption, computation, and scheduling costs\.

### 2\.2Expert Routing and Execution

GShard and GLaM scale conditional capacity\[[23](https://arxiv.org/html/2610.41487#bib.bib23),[24](https://arxiv.org/html/2610.41487#bib.bib24)\], while ST\-MoE examines stability and transfer\[[25](https://arxiv.org/html/2610.41487#bib.bib25)\]\. KABB coordinates agent expertise\[[26](https://arxiv.org/html/2610.41487#bib.bib26)\]\. Expert\-choice routing changes capacity assignment\[[27](https://arxiv.org/html/2610.41487#bib.bib27)\]; MegaBlocks addresses dynamic dispatch through block\-sparse execution\[[28](https://arxiv.org/html/2610.41487#bib.bib28)\]\. GAM\-Agent studies visual\-reasoning collaboration\[[29](https://arxiv.org/html/2610.41487#bib.bib29)\]\. DeepSpeed\-MoE targets system execution\[[30](https://arxiv.org/html/2610.41487#bib.bib30)\], whereas MAT\-Agent concerns training coordination\[[31](https://arxiv.org/html/2610.41487#bib.bib31)\]\.

Parameter\-efficient expert architectures\[[32](https://arxiv.org/html/2610.41487#bib.bib32)\], Mixtral\[[33](https://arxiv.org/html/2610.41487#bib.bib33)\], and DeepSeekMoE\[[34](https://arxiv.org/html/2610.41487#bib.bib34)\]illustrate adaptation and specialization choices\. OSC aligns collaborator knowledge and Agora exchanges uncertainty\[[35](https://arxiv.org/html/2610.41487#bib.bib35),[36](https://arxiv.org/html/2610.41487#bib.bib36)\]\. Token dispatch, expert residency, and agent assistance consequently require different resource accounting\.

### 2\.3Context Mechanisms and Evaluation Scope

Sparse Transformers, BigBird, and ETC construct restricted interactions\[[37](https://arxiv.org/html/2610.41487#bib.bib37),[38](https://arxiv.org/html/2610.41487#bib.bib38),[39](https://arxiv.org/html/2610.41487#bib.bib39)\]\. Reformer uses hashing\[[40](https://arxiv.org/html/2610.41487#bib.bib40)\]; Linformer and Performer use low\-rank and feature\-based approximations\[[41](https://arxiv.org/html/2610.41487#bib.bib41),[42](https://arxiv.org/html/2610.41487#bib.bib42)\]\. FlashAttention improves IO efficiency for exact attention\[[43](https://arxiv.org/html/2610.41487#bib.bib43)\]\. These mechanisms should be distinguished from changing the explicit mask alone\.

HTC\-VLM and image\-based video encoders alter visual\-token processing\[[44](https://arxiv.org/html/2610.41487#bib.bib44),[45](https://arxiv.org/html/2610.41487#bib.bib45)\]\. DepthSSC adapts geometric representations\[[46](https://arxiv.org/html/2610.41487#bib.bib46)\]; Tri\-MARF and MM\-CoT address annotation and visual reasoning\[[47](https://arxiv.org/html/2610.41487#bib.bib47),[48](https://arxiv.org/html/2610.41487#bib.bib48)\]\. The NTIRE restoration challenge\[[49](https://arxiv.org/html/2610.41487#bib.bib49)\], ActPhysCause video report\[[50](https://arxiv.org/html/2610.41487#bib.bib50)\], and deepfake scaling study\[[51](https://arxiv.org/html/2610.41487#bib.bib51)\]provide cross\-domain evaluation contexts\. Their metrics and budgets do not substitute for text\-diffusion measurements\.

### 2\.4Task\-Level Agent Coordination

ReAct interleaves reasoning and actions\[[52](https://arxiv.org/html/2610.41487#bib.bib52)\]; AutoGen supports multi\-agent conversations\[[53](https://arxiv.org/html/2610.41487#bib.bib53)\]\. CAMEL studies role\-based cooperation\[[54](https://arxiv.org/html/2610.41487#bib.bib54)\], while multiagent debate refines responses through repeated exchanges\[[55](https://arxiv.org/html/2610.41487#bib.bib55)\]\. These supply context for a task\-level coordinator whose requests would consume denoiser resources\. That extension is distinct from the evaluated token\-level configuration\.

### 2\.5Closest Diffusion Comparison

DrDiff overlaps directly with sparse expert denoising, but is absent from the supplied score tables\. Its routing, masks, and solver are not matched here, so no numerical superiority is claimed\. The present analysis makes graph accounting, prediction conventions, and the available comparison’s empirical scope explicit\.

## 3Dual\-Budget Denoising

### 3\.1States, Time, and Network Depth

Lety=\(y1,…,yn\)y=\(y\_\{1\},\\ldots,y\_\{n\}\)be target tokens,ccthe conditioning input, andz0=Emb⁡\(y\)∈ℝn×hz\_\{0\}=\\operatorname\{Emb\}\(y\)\\in\\mathbb\{R\}^\{n\\times h\}their clean embeddings\. Diffusion timettindexes corruption and reconstruction; layer indexℓ\\ellindexes computation inside a denoiser\. These indices are independent\. A model withLLlayers evaluated forSSsampling updates incursSSdenoiser evaluations, not a singleLL\-step diffusion trajectory\.

For a Gaussian marginal, write

zt=α¯t​z0\+1−α¯t​ϵ,ϵ∼𝒩⁡\(0,I\)\.z\_\{t\}=\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,z\_\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\epsilon,\\quad\\epsilon\\sim\\mathcal\{N\}\(0,I\)\.\(1\)The denoiser receives the noisy target, conditioning information, and time\. We use clean\-state prediction notation

z^0=fθ​\(zt,t,c\)\.\\hat\{z\}\_\{0\}=f\_\{\\theta\}\(z\_\{t\},t,c\)\.\(2\)This notation fixes the object being predicted; it does not identify an unreported checkpoint parameterization\. Predicting noise or the previous state requires its corresponding conversion and training objective\.

### 3\.2The Context Budget as an Attention Graph

At a layer, letQ,K,VQ,K,Vbe projected token representations\. For query positionii, define an allowed key set𝒩i\\mathcal\{N\}\_\{i\}\. Local or dilated neighborhoods are generated from an offset set𝒲\\mathcal\{W\}of at mostwwoffsets and a strideδ\\delta:

𝒲i=\{i\+δ​r:r∈𝒲\}∩\{1,…,n\}\.\\mathcal\{W\}\_\{i\}=\\\{i\+\\delta r:r\\in\\mathcal\{W\}\\\}\\cap\\\{1,\\ldots,n\\\}\.\(3\)Let𝒢\\mathcal\{G\}containggdesignated global positions\. The combined neighborhood is

𝒩i=\{\{1,…,n\},i∈𝒢,𝒲i∪𝒢,i∉𝒢\.\\mathcal\{N\}\_\{i\}=\\begin\{cases\}\\\{1,\\ldots,n\\\},&i\\in\\mathcal\{G\},\\\\ \\mathcal\{W\}\_\{i\}\\cup\\mathcal\{G\},&i\\notin\\mathcal\{G\}\.\\end\{cases\}\(4\)The graph edge\(i,j\)\(i,j\)denotes queryiiattending to keyjj\. Global queries read all positions, while ordinary queries can read global keys\. A causal task would additionally intersect the allowed set with its causal mask; the full bidirectional pattern is not automatically valid for every prediction interface\.

Attention is normalized over the entire allowed key set:

ai​j\\displaystyle a\_\{ij\}=exp⁡\(qi⊤​kj/dk\)∑u∈𝒩iexp⁡\(qi⊤​ku/dk\),\\displaystyle=\\frac\{\\exp\(q\_\{i\}^\{\\top\}k\_\{j\}/\\sqrt\{d\_\{k\}\}\)\}\{\\sum\_\{u\\in\\mathcal\{N\}\_\{i\}\}\\exp\(q\_\{i\}^\{\\top\}k\_\{u\}/\\sqrt\{d\_\{k\}\}\)\},\(5\)oi\\displaystyle o\_\{i\}=∑j∈𝒩iai​j​vj\.\\displaystyle=\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}\}a\_\{ij\}v\_\{j\}\.This definition avoids applying a scalar softmax independently to a single query–key pair\. It also separates the mask from the learned attention values: an allowed edge can receive little weight, and a high\-value excluded edge remains unavailable\.

![Refer to caption](https://arxiv.org/html/2610.41487v1/attention_graphs.png)Figure 2:Constructed attention masks illustrating local and dilated access with global positions\. Rows are queries and columns are keys\. These are protocol illustrations, not learned masks or benchmark measurements\.
### 3\.3The Expert Budget as Conditional Computation

Letuiu\_\{i\}be the representation entering an expert layer andE1,…,EME\_\{1\},\\ldots,E\_\{M\}its expert functions\. A router produces probabilities

pi=softmax⁡\(Wr​ui\),𝒮i⊆\{1,…,M\}\.p\_\{i\}=\\operatorname\{softmax\}\(W\_\{r\}u\_\{i\}\),\\qquad\\mathcal\{S\}\_\{i\}\\subseteq\\\{1,\\ldots,M\\\}\.\(6\)The selected set𝒮i\\mathcal\{S\}\_\{i\}defines which experts execute\. Its size is bounded by an activation budgetkk\. A general selected\-expert output is

MoE⁡\(ui\)=∑e∈𝒮iπi​e​Ee​\(ui\),\|𝒮i\|≤k\.\\operatorname\{MoE\}\(u\_\{i\}\)=\\sum\_\{e\\in\\mathcal\{S\}\_\{i\}\}\\pi\_\{ie\}E\_\{e\}\(u\_\{i\}\),\\quad\|\\mathcal\{S\}\_\{i\}\|\\leq k\.\(7\)The weightsπi​e\\pi\_\{ie\}may use selected router probabilities with or without renormalization\. The choice affects the numerical function and must be recorded\. A dense softmax over all experts does not establish sparse execution unless a selection or dispatch rule is also specified\.

Token selection is distinct from expert residency\. An implementation may execute onlykkexperts for one token while storing allMMexperts and their optimizer state\. Capacity limits, dropped or rerouted tokens, load balancing, and dispatch communication can further change both output quality and wall\-clock cost\. We keep these implementation quantities explicit rather than infer them from the word “sparse\.”

### 3\.4Training and the Sampling Interface

Under clean\-state prediction, a reconstruction objective can be written

ℒrec=𝔼t,ϵ,y,c​∥z0−fθ​\(zt,t,c\)∥F2\+R⁡\(z0\)\.\\mathcal\{L\}\_\{\\rm rec\}=\\mathbb\{E\}\_\{t,\\epsilon,y,c\}\\lVert z\_\{0\}\-f\_\{\\theta\}\(z\_\{t\},t,c\)\\rVert\_\{F\}^\{2\}\+R\(z\_\{0\}\)\.\(8\)HereRRdenotes the latent regularization described in the source formulation\. Its functional form and coefficient are not specified by the score tables\. Token position must not be conflated with diffusion time in the reconstruction target\.

Inference requires a sampler compatible with the trained prediction convention\. We write its interface as

zt′=Ψ⁡\(zt,fθ​\(zt,t,c\),t,t′\),t′<t\.z\_\{t^\{\\prime\}\}=\\Psi\\bigl\(z\_\{t\},f\_\{\\theta\}\(z\_\{t\},t,c\);t,t^\{\\prime\}\\bigr\),\\quad t^\{\\prime\}<t\.\(9\)For a continuous formulation with velocityvθv\_\{\\theta\}, the associated integral iszt=zs\+∫stvθ​\(zτ,τ,c\)​𝑑τz\_\{t\}=z\_\{s\}\+\\int\_\{s\}^\{t\}v\_\{\\theta\}\(z\_\{\\tau\},\\tau,c\)\\,d\\tau\. A denoiser output cannot simply be exponentiated and substituted for this velocity\. DDIM changes the sampling path while retaining the training objective\[[56](https://arxiv.org/html/2610.41487#bib.bib56)\]; Euler and DPM\-Solver\+\+ use different numerical updates\. Because the record mentions both without resolving the implementation, the reported comparison is not assigned to one particular solver here\.

Absorbing or soft\-masked corruption is a separate design choice\. DiffuSeq\-v2 already introduces a soft absorbing state\[[9](https://arxiv.org/html/2610.41487#bib.bib9)\]\. The supplied comparison does not specify its transition schedule or isolate its contribution\. We therefore retain it as an implementation question rather than attribute a measured stabilization benefit to an undefined mechanism\.

## 4Cost and Connectivity Analysis

### 4\.1Counting Token Interactions

The context budget is the number of allowed attention edges,Bctx=∑i\|𝒩i\|B\_\{\\rm ctx\}=\\sum\_\{i\}\|\\mathcal\{N\}\_\{i\}\|\. Under Eq\. \([4](https://arxiv.org/html/2610.41487#S3.E4)\),

Bctx≤\(n−g\)​\(w\+g\)\+g​n\.B\_\{\\rm ctx\}\\leq\(n\-g\)\(w\+g\)\+gn\.\(10\)Each of then−gn\-gordinary queries reads at mostw\+gw\+gkeys, while each global query readsnn\. Boundary clipping and overlaps can reduce the actual count\. Thus attention\-score and value\-aggregation work scales asO⁡\(n⁡\(w\+g\)​h\)O\(n\(w\+g\)h\)for fixed\-width projections, rather thanO⁡\(n2​h\)O\(n^\{2\}h\)for dense pairwise interactions\. Query, key, and value projections still require their own matrix operations\.

Linear scaling innnrequireswwandggto remain bounded\. If all tokens become global, the dense pattern returns\. If window size grows with document length, the resulting complexity must include that growth\. Dilation changes the spacing of selected positions, rather than their count, so it can expand geometric span without a proportional increase in allowed keys\. This is a count statement, not a guarantee that all useful information remains reachable\.

### 4\.2Reachability and Global Bottlenecks

For symmetric local windows with radiusrrand fixed strideδ\\delta, one layer moves information at mostδ​r\\delta rpositions along an available local edge\. AcrossLLlayers, a path’s displacement is bounded byL​δ​rL\\delta r\. A large displacement bound does not imply that every intermediate position is reachable\. For example, repeating stride two without other connections can preserve parity\-separated components\.

Global nodes change this argument\. With both global\-query and global\-key edges, information from any position can reach a global position in one layer and another ordinary position in the next\. The graph therefore admits short paths\. The transmitted representation still has finite width and learned attention weights, so short paths do not establish lossless document storage or guaranteed factual coherence\. Position\-sensitive context use remains an empirical concern\[[57](https://arxiv.org/html/2610.41487#bib.bib57)\]; a small graph diameter alone does not resolve it\.

The toy masks in Figure[2](https://arxiv.org/html/2610.41487#S3.F2)illustrate these structural differences\. They are constructed examples, not learned masks or benchmark observations\. Their role is to make explicit the connectivity assumptions behind an asymptotic claim\. A measured implementation should separately record mask degree, selected global positions, and the distribution of attention mass\.

### 4\.3Combining Context and Expert Budgets

Define the expert\-activation budget asBexp=∑i\|𝒮i\|≤n​kB\_\{\\rm exp\}=\\sum\_\{i\}\|\\mathcal\{S\}\_\{i\}\|\\leq nk\. If one expert has hidden widthhfh\_\{f\}, its feed\-forward matrix work is proportional toh​hfhh\_\{f\}\. AcrossLLlayers andSSdenoiser evaluations, a useful decomposition is

Ctotal=SL\[\\displaystyle C\_\{\\rm total\}=SL\\bigl\[O⁡\(n​h2\)\+O⁡\(Bctx​h\)\\displaystyle O\(nh^\{2\}\)\+O\(B\_\{\\rm ctx\}h\)\(11\)\+O\(Bexphhf\)\+Croute\]\.\\displaystyle\+O\(B\_\{\\rm exp\}hh\_\{f\}\)\+C\_\{\\rm route\}\\bigr\]\.The terms represent projection work, sparse interactions, selected experts, and routing/dispatch overhead\. A conventional dense router can itself require work proportional ton​h​MnhM\. Communication and capacity padding depend on the execution system\. Omitting these terms can make an activation\-count reduction look like a wall\-clock guarantee that it does not provide\.

This decomposition also separates training corruption steps from inference evaluations\. A training horizon of 2048 possible times does not imply that every inference sample uses 2048 denoiser calls\. Conversely, reducing calls changes cost only through the implemented schedule and solver\. Claims about fewer sampling steps should report the actual number of function evaluations and any quality change under a matched configuration\.

### 4\.4Memory and the Meaning of Efficiency

Sparse masks reduce the score entries that need to be materialized by a compatible sparse kernel\. They do not remove token representations, projections, expert parameters, gradients, or optimizer state\. A dense implementation that masks a full score matrix may retain quadratic memory despite a sparse logical graph\. Training and inference also have different residency requirements\.

Practical efficiency therefore needs several measurements: peak allocated memory, stored parameter and optimizer state, attention workspace, dispatch buffers, denoiser evaluations, latency, and throughput\. These quantities should be reported at matched batch size, precision, sequence length, and hardware\. The available cost table supports a narrower within\-record comparison; it does not provide a complete decomposition of Eq\. \([11](https://arxiv.org/html/2610.41487#S4.E11)\)\.

Figure 3:Reported inference cost and 2\-gram novelty\. Coordinates reproduce the three source rows\. The cost unit, normalization, and uncertainty are unspecified, so the plot supports an ordering within this record rather than a general latency claim\.

## 5Empirical Evaluation

### 5\.1Tasks, Metrics, and Implementation Scope

Scientific summarization uses the arXiv dataset associated with discourse\-aware long\-document summarization\[[58](https://arxiv.org/html/2610.41487#bib.bib58)\]\. HotpotQA measures answer and supporting\-fact prediction\[[59](https://arxiv.org/html/2610.41487#bib.bib59)\]\. Commonsense\-focused dialogue supplies response\-generation examples\[[60](https://arxiv.org/html/2610.41487#bib.bib60)\]\. QQP concerns sentence\-pair decisions; the cited bilateral matching model is background for sentence matching\[[61](https://arxiv.org/html/2610.41487#bib.bib61)\]\. Its accuracy is not a direct test of generated paraphrase quality\.

ROUGE\[[62](https://arxiv.org/html/2610.41487#bib.bib62)\], BLEU\[[63](https://arxiv.org/html/2610.41487#bib.bib63)\], and BERTScore\[[64](https://arxiv.org/html/2610.41487#bib.bib64)\]capture different similarities\. The dialogue table retains fractional scores, while the summarization table reports ROUGE values on a larger numeric scale\. We preserve those scales and avoid combining them into one overall quality score\. None of these automatic measures alone establishes factual consistency, global discourse coherence, or causal reasoning\.

The reported architecture has 12 layers and 12 attention heads, an A100 execution setting, a square\-root noise schedule, and a 2048\-step training horizon\. The full ablation setting uses window size 512\. The expert count, activation count, model width, global\-token selection, dilation schedule, optimizer, data splits, and sampler implementation are unspecified\. These omissions limit exact reproduction and attribution to the two budgets\.

The comparison labels are retained at their documented scope\. Longformer is a long\-context architecture with a generative LED variant\[[5](https://arxiv.org/html/2610.41487#bib.bib5)\]; its decoder and checkpoint must be specified for a generation comparison\. The table label DiffuSeq does not establish whether it uses the original model or its accelerated successor\. GPT\-4 appears in the source background\[[65](https://arxiv.org/html/2610.41487#bib.bib65)\], but no numerical GPT\-4 row is available, so it is not described as an evaluated baseline\.

Table 1:Scientific summarization: aggregate ROUGE comparison\. This record is separate from the length sweep\.Table 2:Aggregate HotpotQA answer and supporting\-fact scores\.Table 3:Dialogue scores under the source evaluation\. Values retain their original fractional scale\.Table 4:QQP accuracy under the recorded protocol; this is not a direct measure of generated paraphrase quality\.
### 5\.2Aggregate Task Results

The sparse\-expert row is denoted DualBudget throughout the analysis\. This descriptive label identifies the recorded sparse\-expert configuration throughout the comparisons\. On aggregate arXiv summarization, it records 44\.41/18\.73/39\.89 for R1/R2/RL\. Relative to Longformer, the differences are 2\.97/1\.21/1\.19 points; relative to DiffuSeq they are 5\.29/2\.30/2\.01\. These are metric\-specific improvements, rather than a single claim of a uniform 3–5 point gain\.

For aggregate HotpotQA, answer EM/F1 reaches 72\.88/85\.42 and support EM/F1 reaches 66\.69/90\.40\. Against Longformer, the four differences are 1\.67, 3\.00, 1\.58, and 0\.90 points\. Stronger supporting\-fact scores are consistent with improved evidence selection under the recorded task, but do not isolate an attention\-mask mechanism\. The models may differ in other training or architectural choices\.

The dialogue row records 0\.049 BLEU, 0\.233 ROUGE\-L, and 0\.628 BERTScore\. QQP accuracy is 95\.3, compared with 92\.3 and 91\.7 for the two listed baselines\. These results show breadth across the supplied records, but the tasks differ in target length and output structure\. Sentence\-pair accuracy and short dialogue response quality do not independently demonstrate long\-document generation\.

### 5\.3Length Sensitivity through 16K

The arXiv length sweep is separate from the aggregate table\. At 8K, DualBudget records 46\.85/19\.72/41\.35; at 16K, it records 43\.60/18\.11/39\.75\. The corresponding decreases are 3\.25, 1\.61, and 1\.60 points\. The model retains higher scores than the listed baselines at each length, while its own scores still decline\. We therefore describe relative resilience over the tested range rather than length\-invariant generation quality\.

Table 5:Scientific summarization across reported context lengths\. All three ROUGE metrics are retained\.Figure 4:Two measured trajectories from the supplied length sweeps: summarization R1 and HotpotQA answer F1\. Points reproduce table values; connecting segments guide the eye and do not introduce measurements between tested lengths\.HotpotQA exhibits a more mixed pattern\. Answer EM changes from 74\.10 at 8K to 72\.88 at 16K, while answer F1 changes from 86\.00 to 85\.42\. Answer F1 at 12K is 85\.30, so the final increase of 0\.12 demonstrates that not every metric is monotonic across the three lengths\. Support EM and F1 decrease from 68\.00/91\.20 to 66\.69/90\.40\. Each trajectory should be read separately\.

Table 6:HotpotQA scores at 8K, 12K, and 16K\. Each pair is exact match / F1\.Long\-context in\-context learning has its own task\-dependent failure modes\[[66](https://arxiv.org/html/2610.41487#bib.bib66)\]\. The present sweep does not establish a universal 8K boundary for other language models\. Nor does the label “16K” resolve whether length refers to the input, target, or combined representation\. That boundary, truncation policy, output length, and batch construction must accompany a reproducible scaling comparison\.

### 5\.4Ablating Attention, Steps, and Window Size

The six configurations in Table[7](https://arxiv.org/html/2610.41487#S5.T7)preserve the original ablation values\. Removing sparse attention changes the three scores by−1\.89/−0\.74/−1\.48\-1\.89/\-0\.74/\-1\.48relative to the full configuration\. Because the dense and sparse kernels, capacities, and training controls are not supplied, this comparison does not independently prove that fewer attention edges cause better quality or faster execution\.

Table 7:Six reported ablation configurations\. A/B/C preserve the three numeric columns because the source prose’s metric labels conflict with the matching summarization baseline\. No uncertainty estimates are available\.Reducing the horizon from 2048 to 1024 changes the scores by−1\.30/−0\.70/−0\.63\-1\.30/\-0\.70/\-0\.63\. Increasing it to 4096 changes them by\+0\.30/−0\.18/\+0\.31\+0\.30/\-0\.18/\+0\.31\. The second score becomes lower, so more steps are not uniformly better across the reported triplet\. Window 256 changes the scores by−0\.61/−0\.51/−0\.24\-0\.61/\-0\.51/\-0\.24, whereas window 1024 changes them by−0\.01/−0\.07/\+0\.03\-0\.01/\-0\.07/\+0\.03\. The latter is close to the full setting but does not improve every score\.

The ablation prose calls the triplet BLEU/ROUGE/BERTScore, although its baseline exactly matches the aggregate summarization ROUGE triplet\. We label the columns A/B/C to retain this unresolved metric identity\. This ambiguity does not alter the numerical differences above, but it limits a metric\-specific interpretation\. Seed\-level uncertainty and a clarified evaluator are needed before small changes are treated as reliable improvements\.

### 5\.5Reported Cost and Novelty

The inference\-cost record lists 0\.80 for DualBudget, 1\.00 for DiffuSeq, and 1\.40 for Longformer\. Under one common timing or normalization convention, 0\.80 is 20% below 1\.00 and approximately 42\.9% below 1\.40\. The record therefore ranks DualBudget as the lowest\-cost entry\. It does not support the original prose ordering that called Longformer fastest\.

Table 8:Reported inference cost and 2\-gram novelty\. The source table does not define the timing unit or normalization\.The same rows report 2\-gram novelty of 0\.90, 0\.75, and 0\.60\. Their differences are 0\.15 and 0\.30 in the recorded scale\. Higher novelty need not mean greater factuality, useful diversity, or better content coverage; the definition of the reference set and denominator is also needed\. A low\-cost, high\-novelty point should therefore be assessed together with task scores and generation fidelity\.

The source table omits the timing unit, batch size, precision, and evaluation schedule\. Although an accompanying plot uses a seconds\-per\-sample label, the underlying timing record is unavailable\. We retain the values as reported cost, rather than assert a deployment latency or hardware\-independent speedup\. No training\-time measurements are provided to substantiate the separate 15% training reduction claim\.

## 6Discussion and Limitations

### 6\.1What the Dual\-Budget View Explains

Context edges control information access, while selected experts control conditional transformation\. A disconnected graph cannot be repaired merely by increasing expert count, and global connectivity does not ensure informative expert inputs\. The reported scores are compatible with a useful combination, but architecture size, training exposure, conditioning, and decoding remain alternative explanations\. The ablations do not separately vary expert activation or the global\-token budget\. Matched parameter budgets, identical data, measured dispatch, and controlled interventions would isolate these effects\.

### 6\.2Quality beyond Automatic Overlap

A scientific summary can match reference phrasing while omitting a limitation or introducing an unsupported causal relation\. Answer F1 can improve without better reasoning traces, and novel bi\-grams can express useful reformulations or errors\. Evaluation should therefore retain support, contradiction, coverage, and discourse judgments alongside overlap scores\.

The supplied qualitative asset contains one summary example about tensor models and emergent geometry\. It illustrates differences in specificity, but cannot establish an error frequency\. A systematic study would retain source documents, generated texts, evaluator versions, and blinded annotations for sampled outputs\.

### 6\.3Connection to Agent Coordination

KABB, GAM\-Agent, and Markets of Minds\[[26](https://arxiv.org/html/2610.41487#bib.bib26),[29](https://arxiv.org/html/2610.41487#bib.bib29),[10](https://arxiv.org/html/2610.41487#bib.bib10)\]motivate an extension in which task\-level agents select document spans, negotiate shared capacity, or request specialized reasoning\. A coordinator could assign a span priority and a budget, while the denoiser exposes context\-edge and expert\-activation costs\. This connects deliberation about where computation is useful to execution inside each denoising step\. No agent coordinator is evaluated in the reported tables\.

### 6\.4Reproducible Evaluation

Replication requires four linked descriptions: the model’s widths, experts, activation rule, capacity factor, and masks; the diffusion corruption, prediction target, sampler, and time grid; the data splits, source/target lengths, and truncation; and the measurement software, hardware, precision, batch size, evaluator settings, and seeds\. Absorbing\-state kernels and gate normalization must be included if used\.

At each length, an instrumented evaluation should report attention edges, executed expert assignments, dispatch overhead, peak memory, denoiser calls, latency, and throughput\. Quality should be compared at matched sampling cost as well as matched update count\. Fewer active experts can leave parameter residency unchanged; communication can dominate after attention becomes sparse; additional global positions can erode the linear\-cost regime\. These measurements would connect the theoretical budgets to practical behavior\. They are not additional results contained in the current record\.

## 7Conclusion

Sparse text diffusion can be analyzed through separate budgets for context edges and expert activations\. The formulation clarifies attention normalization, connectivity, conditional compute, and sampler compatibility\. The supplied configuration leads the listed task comparisons through 16K, while retaining length\-dependent degradation and mixed ablation effects\. Its reported inference\-cost ordering is favorable under an unresolved measurement convention\. These observations motivate a matched, instrumented evaluation of both budgets, with explicit corruption, routing, and data definitions\.

## References

- \[1\]J\. Ho, A\. Jain and P\. Abbeel,“Denoising Diffusion Probabilistic Models,” 2020,arXiv:2006\.11239\.
- \[2\]Y\. Song et al\.,“Score\-Based Generative Modeling through Stochastic Differential Equations,” 2020,arXiv:2011\.13456\.
- \[3\]S\. Gong et al\.,“DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models,” 2022,arXiv:2210\.08933\.
- \[4\]A\. Vaswani et al\.,“Attention is all you need,”Advances in neural information processing systems, vol\. 30, 2017\.
- \[5\]I\. Beltagy, M\. E\. Peters and A\. Cohan,“Longformer: The long\-document transformer,”arXiv preprint arXiv:2004\.05150, 2020\.
- \[6\]N\. Shazeer et al\.,“Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer,”CoRR, vol\. abs/1701\.06538, 2017\.
- \[7\]W\. Fedus, B\. Zoph and N\. Shazeer,“Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol\. 23, no\. 120, pp\. 1–39, 2022\.
- \[8\]J\. Zhang et al\.,“DrDiff: Dynamic routing diffusion with hierarchical attention for breaking the efficiency\-quality trade\-off,”inProc\. EMNLP, Nov\. 2025, pp\. 9320–9340\.
- \[9\]S\. Gong et al\.,“DiffuSeq\-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models,” 2023,arXiv:2310\.05793\.
- \[10\]J\. Zhang et al\.,“Markets of Minds: A 51\-Billion\-State Economy of Intelligence,” 2026,arXiv:2610\.32190\.
- \[11\]X\. L\. Li et al\.,“Diffusion\-LM Improves Controllable Text Generation,” 2022,arXiv:2205\.14217\.
- \[12\]X\. Han, S\. Kumar and Y\. Tsvetkov,“SSD\-LM: Semi\-autoregressive Simplex\-based Diffusion Language Model for Text Generation and Modular Control,” 2022,arXiv:2210\.17432\.
- \[13\]J\. Austin et al\.,“Structured denoising diffusion models in discrete state\-spaces,”Advances in Neural Information Processing Systems, vol\. 34, pp\. 17981–17993, 2021\.
- \[14\]A\. Lou, C\. Meng and S\. Ermon,“Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution,” 2023,arXiv:2310\.16834\.
- \[15\]S\. S\. Sahoo et al\.,“Simple and Effective Masked Diffusion Language Models,” 2024,arXiv:2406\.07524\.
- \[16\]S\. Nie et al\.,“Large Language Diffusion Models,” 2025,arXiv:2502\.09992\.
- \[17\]J\. Chen et al\.,“A cheaper and better diffusion language model with soft\-masked noise,”inProc\. EMNLP, Dec\. 2023, pp\. 4765–4775\.
- \[18\]J\. Zhang et al\.,“CF\-VLM:CounterFactual Vision\-Language Fine\-tuning,” 2025,arXiv:2506\.17267\.
- \[19\]J\. Zhang et al\.,“Top\-Down Semantic Refinement for Image Captioning,” 2025,arXiv:2510\.22391\.
- \[20\]E\. Hoogeboom et al\.,“Autoregressive diffusion models,”arXiv preprint arXiv:2110\.02037, 2021\.
- \[21\]C\. Lu et al\.,“Dpm\-solver\+\+: Fast solver for guided sampling of diffusion probabilistic models,”arXiv preprint arXiv:2211\.01095, 2022\.
- \[22\]J\. Zhang et al\.,“Failure\-Driven Workflow Refinement,” 2025,arXiv:2510\.10035\.
- \[23\]D\. Lepikhin et al\.,“Gshard: Scaling giant models with conditional computation and automatic sharding,”arXiv preprint arXiv:2006\.16668, 2020\.
- \[24\]N\. Du et al\.,“Glam: Efficient scaling of language models with mixture\-of\-experts,”inInternational Conference on Machine Learning\. PMLR, 2022, pp\. 5547–5569\.
- \[25\]B\. Zoph et al\.,“ST\-MoE: Designing Stable and Transferable Sparse Expert Models,” 2022,arXiv:2202\.08906\.
- \[26\]J\. Zhang et al\.,“KABB: Knowledge\-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi\-Agent Systems,” 2025,arXiv:2502\.07350\.
- \[27\]Y\. Zhou et al\.,“Mixture\-of\-experts with expert choice routing,”Advances in Neural Information Processing Systems, vol\. 35, pp\. 7103–7114, 2022\.
- \[28\]T\. Gale et al\.,“MegaBlocks: Efficient Sparse Training with Mixture\-of\-Experts,” 2022,arXiv:2211\.15841\.
- \[29\]J\. Zhang et al\.,“GAM\-Agent: Game\-theoretic and uncertainty\-aware collaboration for complex visual reasoning,”inAdvances in Neural Information Processing Systems, 2025, vol\. 38\.
- \[30\]S\. Rajbhandari et al\.,“Deepspeed\-moe: Advancing mixture\-of\-experts inference and training to power next\-generation ai scale,”inInternational conference on machine learning\. PMLR, 2022, pp\. 18332–18346\.
- \[31\]J\. Zhang et al\.,“MAT\-Agent: Adaptive Multi\-Agent Training Optimization,” 2025,arXiv:2510\.17845\.
- \[32\]Z\.\-F\. Gao et al\.,“Parameter\-efficient mixture\-of\-experts architecture for pre\-trained language models,”arXiv preprint arXiv:2203\.01104, 2022\.
- \[33\]A\. Q\. Jiang et al\.,“Mixtral of Experts,” 2024,arXiv:2401\.04088\.
- \[34\]D\. Dai et al\.,“DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture\-of\-Experts Language Models,” 2024,arXiv:2401\.06066\.
- \[35\]J\. Zhang et al\.,“OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi\-Agent LLM Collaboration,” 2025,arXiv:2509\.04876\.
- \[36\]J\. Zhang et al\.,“Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi\-Agent Bandit Systems,” 2026,arXiv:2601\.18735\.
- \[37\]R\. Child et al\.,“Generating Long Sequences with Sparse Transformers,” 2019,arXiv:1904\.10509\.
- \[38\]M\. Zaheer et al\.,“Big bird: Transformers for longer sequences,”Advances in neural information processing systems, vol\. 33, pp\. 17283–17297, 2020\.
- \[39\]J\. Ainslie et al\.,“ETC: Encoding long and structured inputs in transformers,”inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\), Nov\. 2020, pp\. 268–284\.
- \[40\]N\. Kitaev, Ł\. Kaiser and A\. Levskaya,“Reformer: The Efficient Transformer,” 2020,arXiv:2001\.04451\.
- \[41\]S\. Wang et al\.,“Linformer: Self\-Attention with Linear Complexity,” 2020,arXiv:2006\.04768\.
- \[42\]K\. Choromanski et al\.,“Rethinking Attention with Performers,” 2020,arXiv:2009\.14794\.
- \[43\]T\. Dao et al\.,“FlashAttention: Fast and Memory\-Efficient Exact Attention with IO\-Awareness,” 2022,arXiv:2205\.14135\.
- \[44\]J\. Zhang et al\.,“Hybrid Token Compression for Vision\-Language Models,” 2025,arXiv:2512\.08240\.
- \[45\]J\. Zhang et al\.,“Video Encoders Built on Image Representations,” 2026,arXiv:2610\.06616\.
- \[46\]J\. Yao et al\.,“DepthSSC: Monocular 3D Semantic Scene Completion via Depth\-Spatial Alignment and Voxel Adaptation,” 2023,arXiv:2311\.17084\.
- \[47\]J\. Zhang et al\.,“Tri\-MARF: A tri\-modal multi\-agent responsive framework for comprehensive 3d object annotation,”inThe Thirty\-ninth Annual Conference on Neural Information Processing Systems, 2025\.
- \[48\]J\. Zhang et al\.,“MM\-CoT: A Benchmark for Probing Visual Chain\-of\-Thought Reasoning in Multimodal Models,” 2025,arXiv:2512\.08228\.
- \[49\]Z\. Chen et al\.,“The Fourth Challenge on Image Super\-Resolution \(×\\times4\) at NTIRE 2026: Benchmark Results and Method Overview,” 2026,arXiv:2604\.14558\.
- \[50\]J\. Zhang et al\.,“The LoViF 2026 ActPhysCause challenge: A baseline report on action\-conditioned video world models,” ECCV Workshop on Low\-Level Vision Frontiers \(LoViF\), challenge report, 2026\.
- \[51\]W\. Wang et al\.,“Scaling Laws for Deepfake Detection,” 2025,arXiv:2510\.16320\.
- \[52\]S\. Yao et al\.,“ReAct: Synergizing Reasoning and Acting in Language Models,” 2022,arXiv:2210\.03629\.
- \[53\]Q\. Wu et al\.,“AutoGen: Enabling Next\-Gen LLM Applications via Multi\-Agent Conversation,” 2023,arXiv:2308\.08155\.
- \[54\]G\. Li et al\.,“CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society,” 2023,arXiv:2303\.17760\.
- \[55\]Y\. Du et al\.,“Improving Factuality and Reasoning in Language Models through Multiagent Debate,” 2023,arXiv:2305\.14325\.
- \[56\]J\. Song, C\. Meng and S\. Ermon,“Denoising Diffusion Implicit Models,” 2020,arXiv:2010\.02502\.
- \[57\]N\. F\. Liu et al\.,“Lost in the Middle: How Language Models Use Long Contexts,” 2023,arXiv:2307\.03172\.
- \[58\]A\. Cohan et al\.,“A discourse\-aware attention model for abstractive summarization of long documents,”inProc\. NAACL, June 2018, pp\. 615–621\.
- \[59\]Z\. Yang et al\.,“HotpotQA: A dataset for diverse, explainable multi\-hop question answering,”inProc\. EMNLP, Oct\.\-Nov\. 2018, pp\. 2369–2380\.
- \[60\]P\. Zhou et al\.,“Commonsense\-focused dialogues for response generation: An empirical study,”inProc\. SIGDIAL, July 2021, pp\. 121–132\.
- \[61\]Z\. Wang, W\. Hamza and R\. Florian,“Bilateral multi\-perspective matching for natural language sentences,”arXiv preprint arXiv:1702\.03814, 2017\.
- \[62\]C\.\-Y\. Lin,“Rouge: A package for automatic evaluation of summaries,”inText summarization branches out, 2004, pp\. 74–81\.
- \[63\]K\. Papineni et al\.,“Bleu: a method for automatic evaluation of machine translation,”inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp\. 311–318\.
- \[64\]T\. Zhang et al\.,“Bertscore: Evaluating text generation with bert,”arXiv preprint arXiv:1904\.09675, 2019\.
- \[65\]J\. Achiam et al\.,“Gpt\-4 technical report,”arXiv preprint arXiv:2303\.08774, 2023\.
- \[66\]T\. Li et al\.,“Long\-context llms struggle with long in\-context learning,”arXiv preprint arXiv:2404\.02060, 2024\.