NVIDIA might just have solved the biggest tradeoff in LLMs.

Every LLM makes you pick between speed and quality. Autoregressive models like GPT give you quality, but they’re slow, because they generate one token at a time.

Diffusion models do the opposite. They generate whole blocks in parallel, so they’re fast, but that speed has always come at the cost of quality.

Before we get to NVIDIA’s fix, it’s worth understanding why diffusion falls behind in the first place.

A diffusion LLM starts with a block of masked tokens and unmasks them over a few passes. On every pass, it reads all the text written so far and predicts the masked tokens in that block together.

So the same network is doing two very different jobs at once. It has to understand the context that’s already there while also guessing at tokens that don’t exist yet.

That’s one set of weights pulled in two directions, so it never gets good at either.

TwoTower fixes this by not forcing one network to do both. It clones a pretrained autoregressive model into two towers.

The first is a frozen context tower that reads everything written so far, exactly like the original did. It’s never trained, so the model’s intelligence stays perfectly intact.

The second is a trainable denoiser tower that fills in each new block of tokens in parallel. To stay on track, it constantly cross-attends back to the context tower.

Here’s where it gets clever. The two connect layer by layer, so denoiser layer 5 talks to context layer 5, and so on.

That gives the denoiser the backbone’s full multi-scale understanding, not just a final summary of it.

The results:

↳ 2.42x higher generation throughput ↳ Keeps 98.7% of the original model’s quality ↳ Built on a 30B hybrid Mamba-Transformer MoE backbone ↳ Trained on just ~2.1T tokens, a fraction of the 25T used to pretrain the original

That last number is the real story. Nothing here was trained from scratch.

It’s an adaptation you bolt onto an autoregressive model you already have, and it turns slow one-token-at-a-time decoding into fast parallel block generation, cheaply.

So if you’re shipping anything latency-sensitive, this is a blueprint worth studying. You no longer have to choose between the model you trust and the speed you need.

Weights, code, and recipe are all open-sourced.

Link to the paper: https://arxiv.org/pdf/2606.26493

That said, if you want to go deeper, I wrote a detailed article that explains LLM inference from first principles, step by step.

The article is quoted below.

Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context

Source: https://arxiv.org/html/2606.26493 John Kamalu∗Roger WaleffeMostofa PatwaryMohammad ShoeybiBryan Catanzaro

Abstract

AbstractDiffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: afrozenAR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B—an open-weight 30B hybrid Mamba-Transformer MoE model—and trained on∼2.1{\sim}2.1T tokens, Nemotron-Labs-TwoTower retains98.7%of the autoregressive baseline’s quality while offering2.42×\timeshigher wall-clock generation throughput. We release the code and model weights in theNemotron-Labs-TwoTower collection.

**footnotetext:Core contributor.\\dagger\\daggerfootnotetext:Project lead. Contact:[email protected]## 1Introduction

Autoregressive (AR) language models are the predominant paradigm in text generation(radford2019language;grattafiori2024llama;liu2024deepseek). The one-token-at-a-time decoding process, however, creates a throughput bottleneck. Discrete diffusion language models take a different approach: they generate tokens in parallel and refine them iteratively, offering higher throughput and finer-grained controllability(austin2021structured;sahoo2024simple;nie2025large;arriola2025block). In this paradigm, existing models typically use a single decoder for two distinct roles at every denoising step, representing the clean tokens and denoising the corrupted tokens. This entanglement pulls the same set of weights in different directions, limiting their capacity to excel at either.

arriola2025encoderobserved that these two roles can be handled by separate modules, proposing an encoder-decoder architecture where the encoder represents clean tokens and a lightweight decoder iteratively denoises each block. Their largest experiment, at 1.7B scale, trains both modules with tied weights. It remains to be seen whether fully decoupling the two roles holds at larger scales, particularly on modern hybrid architectures that combine Mamba, attention, and mixture-of-experts (MoE) layers.

We present TwoTower and instantiate it as Nemotron-Labs-TwoTower on Nemotron-3-Nano-30B-A3B(blakeman2025nvidia), an open-weight hybrid model that interleaves Mamba-2, attention, and mixture-of-experts layers. We take two copies of the pretrained network and assign them complementary roles: the AR context tower is keptfrozenas a causal model over clean tokens, preserving the backbone’s autoregressive capability; the diffusion denoiser tower is trained via a mask-diffusion objective with bidirectional self-attention within each noisy block and causal cross-attention to past clean context.

(a)Two-tower architecture: a frozen AR context tower conditions a diffusion denoiser tower over each noisy block. Accuracy vs Throughput(Relative to Baseline AR)Throughput (×\timesbaseline)Accuracy (% of baseline)

(b)Quality–throughput curve across confidence thresholds and denoising budgets, normalized to the AR baseline.

Figure 1:Overview of Nemotron-Labs-TwoTower. (a) The frozen AR context tower runs causally over the prompt and committed tokens, preserving the pretrained backbone and maintaining reusable KV cache and Mamba states. The trainable diffusion denoiser tower iteratively resolves each noisy block using layer-aligned context attention and context-seeded Mamba states. (b) Quality–throughput trade-off relative to the one-token-at-a-time AR baseline.The towers connect layer-by-layer. Each denoiser layer cross-attends to the corresponding layer of the context tower, giving the denoiser multi-scale access to the backbone’s representations. This is in contrast to prior approaches that broadcast only the last hidden state. The denoiser is further modulated by the diffusion timestep via adaptive layer normalization(peebles2023scalable), which is common in image diffusion transformers but less so in masked diffusion language models(ou2024your).

Trained on∼2.1{\sim}2.1T tokens—a fraction of the total 25T tokens used to pretrain the backbone—Nemotron-Labs-TwoTower preserves most of the autoregressive baseline’s quality while delivering substantially higher throughput on a 30B hybrid MoE architecture. At the default operating point, Nemotron-Labs-TwoTower commits multiple tokens per refinement step early in decoding, helping explain the observed wall-clock speedup over one-token-at-a-time AR decoding. We release the training recipe, code, and model weights in theNemotron-Labs-TwoTower collection.

2Method

In this section we describe the two-tower architecture (Section2.1), the masked diffusion formulation (Section2.2), and the training recipe (Section2.3).

2.1Two-Tower Architecture

TwoTower is a general approach that can be applied to any pretrained autoregressive language model. In this work we instantiate it on Nemotron-3-Nano-30B-A3B(blakeman2025nvidia), which consists of 52 layers: 23 Mamba-2 layers, 6 self-attention layers, and 23 mixture-of-experts (MoE) layers.

We create two copies of this network and assign them complementary roles (see Figure1(a)). Given a prompt, theAR context toweracts as a causal autoregressive model over the prompt and previously committed tokens, producing per-layer KV pairs and final Mamba states. Generation then proceeds block by block: each new block is initialized withSS[MASK]tokens and iteratively refined by thediffusion denoiser toweroverTTdenoising steps. At every layer of the denoiser, attention layers cross-attend to the corresponding context-tower KV cache, while Mamba-2 layers seed their initial state from the corresponding context-tower Mamba state. Once all tokens in a block are clean, the block is committed; the context tower processes the committed tokens causally to update its caches, and generation continues with the next block. Algorithm1outlines this block-wise autoregressive generation process.

Algorithm 1TwoTower block-wise generation.0:Prompt

𝐱prompt\mathbf{x}_{\text{prompt}}, block size

SS, denoising steps

TT 1:

(KV,States)←ContextTower​(𝐱prompt)(\text{KV},\text{States})\leftarrow\text{ContextTower}(\mathbf{x}_{\text{prompt}})⊳\trianglerightBuild AR context

2:foreach block

b=1,2,…b=1,2,\dotsdo

3:

𝐳b←([MASK],…,[MASK])\mathbf{z}^{b}\leftarrow(\texttt{[MASK]},\dots,\texttt{[MASK]}) 4:for

t=1,…,Tt=1,\dots,Tdo

5:

𝐳b←SampleStep​(𝐳b;DenoiserTower​(𝐳b,t,KV,States))\mathbf{z}^{b}\leftarrow\textsc{SampleStep}\bigl(\mathbf{z}^{b};\,\text{DenoiserTower}(\mathbf{z}^{b},t,\text{KV},\text{States})\bigr) 6:endfor

7:

𝐱b←𝐳b\mathbf{x}_{b}\leftarrow\mathbf{z}^{b}⊳\trianglerightCommit block

8:

(KV,States)←ContextTower​(𝐱b,KV,States)(\text{KV},\text{States})\leftarrow\text{ContextTower}(\mathbf{x}_{b},\text{KV},\text{States})⊳\trianglerightUpdate AR context

9:endfor

At inference time, TwoTower introduces a fixed memory footprint for the context tower weights, while keeping a single prefix cache. Only the context tower maintains KV and Mamba states across blocks, updating them as blocks are committed, so the sequence-length-dependent cache memory scales like the AR baseline.

We keep the context tower body unchanged. Its final vocabulary projection/LM head is optional: it can be omitted in the default diffusion-generation path, where the context tower only needs to produce states, and retained when the context tower is used for speculative decoding,verification, likelihood evaluation, or AR scoring. In contrast, we make a few architectural modifications to the denoiser tower to adapt it for diffusion training and inference. We describe each modification below, and provide ablations in Section3.2.

Bidirectional Attention.Within the block under refinement, the denoiser relaxes the causal mask: noisy tokens attend bidirectionally to other noisy tokens while remaining causal with respect to past clean blocks. This adds no parameters and, for standard dense attention kernels, does not change per-layer FLOPs. At denoiser layerii, queries from the noisy block attend over a concatenated key/value sequence of the context tower’s layer-iiKV for past blocks0,…,b−10,\dots,b{-}1and the denoiser’s own layer-iiKV for blockbb:

Attn​(𝐐b(i),[𝐊<bctx,(i);𝐊bden,(i)],[𝐕<bctx,(i);𝐕bden,(i)]).\text{Attn}\big(\mathbf{Q}_{b}^{(i)},\;[\mathbf{K}_{<b}^{\text{ctx},(i)};\,\mathbf{K}_{b}^{\text{den},(i)}],\;[\mathbf{V}_{<b}^{\text{ctx},(i)};\,\mathbf{V}_{b}^{\text{den},(i)}]\big).(1)The cross-attention islayer-aligned: denoiser layeriiattends to context layerii. Because both towers initialize from the same pretrained checkpoint, same-index layers operate at comparable representation levels, making layer-aligned cross-attention a natural pairing.

Bidirectional Mamba.We also test a parameter-free bidirectional Mamba-2 variant by running the pretrained Mamba-2 weights left-to-right and right-to-left from zero states, then averaging the outputs. This roughly doubles per-layer state-space model (SSM) FLOPs; ablations show only marginal quality gains, so the final design keeps Mamba causal.

Time Conditioning.We condition the denoiser on the timestepttusing adaLN-single(peebles2023scalable;chen2023pixart). A global MLP mapsttto shared scale, shift, and gate parameters, with per-layer learned embeddings for layer-specific modulation. On the 30B backbone, this adds only 1.5M parameters; we replicate these small modules on each tensor-parallel rank rather than sharding them, avoiding extra communication cost. Time conditioning improves sampling quality (Section3.2).

Expert Routing.We use the backbone’s existing MoE routing mechanism, including sequence-level load balancing. Tokens are noise-aware through adaLN modulation. We make no explicit routing choices and allow experts to learn specialization from the denoising objective.

2.2Block Diffusion Language Modeling

We train the denoiser under the masked diffusion framework(sahoo2024simple;shi2024simplified;ou2024your), applied to each autoregressive block. Let𝐱b=(xb1,…,xbS)\mathbf{x}_{b}=(x_{b}^{1},\dots,x_{b}^{S})denote thebb-th block ofSStokens, and let𝐜<b=(KV<b,Statesb−1)\mathbf{c}_{<b}=(\text{KV}_{<b},\text{States}_{b-1})denote the context-tower attention KV cache over previous clean blocks and the per-layer Mamba boundary states after blockb−1b{-}1. We model the sequence distribution as block-autoregressive:

log⁡pθ​(𝐱)=∑b=1Blog⁡pθ​(𝐱b∣𝐱<b),\log p_{\theta}(\mathbf{x})=\sum_{b=1}^{B}\log p_{\theta}(\mathbf{x}_{b}\mid\mathbf{x}_{<b}),(2)where each conditional block distribution is parameterized by a masked diffusion process conditioned on𝐜<b\mathbf{c}_{<b}.

For diffusion timet∈(0,1]t\in(0,1], the forward process corrupts a clean block𝐱b\mathbf{x}_{b}into a noisy block𝐳tb\mathbf{z}_{t}^{b}by independently replacing each token with[MASK]. The scheduleαt\alpha_{t}is the probability that a clean token remains unmasked at noise leveltt, so1−αt1-\alpha_{t}is the masking probability; in our experiments we use the linear scheduleαt=1−t\alpha_{t}=1-t. Usingδy\delta_{y}to denote a point mass at tokenyyand𝐦\mathbf{m}for the[MASK]point mass, the per-token marginal is

q​(𝐳tb,ℓ∣xbℓ)=Cat​(𝐳tb,ℓ;αt​δxbℓ+(1−αt)​𝐦),q(\mathbf{z}_{t}^{b,\ell}\mid x_{b}^{\ell})=\text{Cat}\!\left(\mathbf{z}_{t}^{b,\ell};\,\alpha_{t}\delta_{x_{b}^{\ell}}+(1-\alpha_{t})\mathbf{m}\right),(3)whereαt\alpha_{t}decreases from nearly11to0asttincreases. Smallttcorresponds to lightly corrupted blocks, while largettapproaches a fully masked block.

The denoiser predicts the clean token at masked positions, conditioned on the noisy block, the diffusion time, and the context cache. The masked diffusion ELBO motivates a time-weighted negative log-likelihood over masked tokens(sahoo2024simple;shi2024simplified); for the linear scheduleαt=1−t\alpha_{t}=1-t, the theoretical weight simplifies to1/t1/t. In training, we omit this importance weight for stability and optimize the mean negative log-likelihood over masked positions:

ℒMD=𝔼t,𝐳t​[1|ℳt|​∑(b,ℓ)∈ℳt−log⁡pθ​(xbℓ∣𝐳tb,t,𝐜<b)].\mathcal{L}_{\mathrm{MD}}=\mathbb{E}_{t,\mathbf{z}_{t}}\left[\frac{1}{|\mathcal{M}_{t}|}\sum_{(b,\ell)\in\mathcal{M}_{t}}-\log p_{\theta}\!\left(x_{b}^{\ell}\mid\mathbf{z}_{t}^{b},t,\mathbf{c}_{<b}\right)\right].(4)Hereℳt={(b,ℓ):ztb,ℓ=[MASK]}\mathcal{M}_{t}=\{(b,\ell):z_{t}^{b,\ell}=\texttt{[MASK]}\}is the set of masked token positions.

Sampling.Algorithm1describes the outer block-autoregressive loop, while Algorithm2summarizes the single-block sampler used in our main results. The sampler receives the prefix caches(KV<b,Statesb−1)(\text{KV}_{<b},\text{States}_{b-1})from the context tower and keeps them fixed while refining blockbb; the caches are advanced only after the block is committed. We also consider the standard predict-and-noise sampler used in block diffusion models(arriola2025block;arriola2025encoder), which predicts a clean block and then re-masks low-confidence positions according to the noise schedule. Our main results use a confidence-unmasking variant with thresholdγ\gamma: at each step, the denoiser predicts all currently masked tokens in parallel, predictions aboveγ\gammaare committed, and the remaining uncertain positions stay masked for later refinement. This makes the number of committed tokens adaptive to the model’s confidence while ensuring the block is completed withinTTsteps.

Algorithm 2Single-block confidence unmasking.0:Prefix caches

(KV<b,Statesb−1)(\text{KV}_{<b},\text{States}_{b-1}), block size

SS, steps

TT, confidence threshold

γ\gamma 1:

𝐳←([MASK],…,[MASK])\mathbf{z}\leftarrow(\texttt{[MASK]},\dots,\texttt{[MASK]}) 2:for

r=1,…,Tr=1,\dots,Tdo

3:

τ←|{ℓ:zℓ=[MASK]}|/S\tau\leftarrow|\{\ell:z^{\ell}=\texttt{[MASK]}\}|/S 4:

pθ←DenoiserTower​(𝐳,τ,KV<b,Statesb−1)p_{\theta}\leftarrow\text{DenoiserTower}(\mathbf{z},\tau,\text{KV}_{<b},\text{States}_{b-1}) 5:

𝐳←ConfidenceSample​(𝐳,pθ,γ,T−r+1)\mathbf{z}\leftarrow\textsc{ConfidenceSample}(\mathbf{z},p_{\theta},\gamma,T-r+1) 6:endfor

7:return

𝐳\mathbf{z}

2.3Training

We train only the denoiser tower, while using the context tower as a frozen AR representation model. Training follows the two-stage curriculum of the Nemotron-3-Nano backbone: an initial broad-coverage phase followed by a higher-quality refinement phase.

Initialization and data.Phase 1 initializes both towers from the same pretrained Nemotron-3-Nano-30B-A3B base checkpoint and trains the denoiser on a subset of the Nemotron-3-Nano phase-1 blend, which emphasizes data diversity and broad coverage. Phase 2 continues the denoiser from Phase 1, and switches to the Nemotron-3-Nano phase-2 blend, which upweights higher-quality and STEM-focused sources. Across both phases, we train on roughly∼1.4{\sim}1.4T tokens.

Optimization.We use BF16 precision, AdamW(loshchilov2019decoupledweightdecayregularization), and a Warmup-Stable-Decay learning rate schedule(hu2024minicpmunveilingpotentialsmall)with peak learning rate1×10−41\text{\times}{10}^{-4}and final learning rate1×10−61\text{\times}{10}^{-6}. The optimizer and learning-rate schedule are reset at phase boundaries. Our final configuration uses block sizeS=16S{=}16; in the current implementation, the Mamba chunk size is matched toSSso the existing Mamba kernel exposes states at diffusion block boundaries.

Implementation.We implemented the training loop in Megatron-LM111https://github.com/nvidia/megatron-lm(megatron-lm). For each clean sequence, the frozen context tower runs once with no-grad under its standard causal AR mask. Its attention layers produce layer-aligned KV caches, while its Mamba-2 layers expose the recurrent conv and SSM states after every clean block. These block-boundary states are the states a causal Mamba layer would carry forward after consuming each block. The denoiser starts from the same clean sequence, applies the mask-diffusion corruption, and processes all noisy blocks in one forward pass: attention layers attend to past clean context blocks and bidirectionally within the current noisy block, while Mamba layers fold blocks into the batch dimension. Block 0 starts from a zero Mamba state; blockbbstarts from the context-tower Mamba state after blockb−1b{-}1. This batched formulation exposes the denoiser to diverse block-level corruption patterns within each optimizer step, without increasing the underlying sequence batch.

3Results

We evaluate Nemotron-Labs-TwoTower against the autoregressive Nemotron-3-Nano-30B-A3B baseline used to initialize both towers. We report base-model benchmarks spanning general knowledge, code, math, commonsense, and multilingual tasks, using checkpoints after long-context pretraining and before instruction tuning, RL, or alignment. All evaluations use BF16 precision on 2×\timesH100 GPUs. Throughput is measured from the wall-clock time to produce the final answer under the same inference setup, and reported as speedup relative to the AR baseline.

3.1Quality–Throughput Trade-off

Nemotron-Labs-TwoTower preserves98.7%of the AR baseline’s aggregate benchmark quality while improving wall-clock generation throughput by2.42×\times. This operating point lies on the quality–throughput curve in Figure1(b); Figure2expands the same point into category-level accuracy and relative throughput. The result shows that pretrained AR representations can support a trainable diffusion denoiser while preserving broad benchmark quality and yielding faster generation. We use confidence unmasking (Algorithm2) with thresholdγ=0.8\gamma{=}0.8and block sizeS=16S{=}16. Varying the confidence thresholdγ\gammaand number of denoising stepsTTtraces the broader curve. Higher-confidence settings recover slightly more quality, while lower thresholds accept more tokens per step and reach throughput beyond3×3\timeswith larger quality loss. We report theγ=0.8\gamma{=}0.8point as the best balance we found between preserving the backbone’s capability and realizing decoding-speed gains.

Figure 2:Category-level comparison between the Nemotron-3-Nano-30B-A3B autoregressive baseline and Nemotron-Labs-TwoTower. Left: average accuracy by benchmark category. Right: relative wall-clock throughput for generative evaluations. Nemotron-Labs-TwoTower preserves most of the AR baseline’s quality while improving relative generation throughput to2.42×\mathbf{2.42\times}.The benchmark task breakdown is shown in Figure2. General knowledge remains within about one point of the AR baseline, code and math show modest degradation, and commonsense and multilingual performance are recovered or improved. The Pareto curve and task breakdown indicate that the two-tower diffusion model preserves most of the pretrained model’s behavior. Notably, this quality recovery comes from adaptation rather than re-pretraining. Nemotron-Labs-TwoTower starts from an off-the-shelf AR checkpoint and trains the denoiser on only∼2.1{\sim}2.1T tokens, a fraction of the 25T tokens used to pretrain the backbone.

3.2Design Choices

Table1tracks the main design choices on an internal Nemotron-3-Nano-30B-A3B base checkpoint. The ablations compare bidirectional attention with causal Mamba, bidirectional Mamba, adaLN time conditioning, and continued training on the phase-2 blend. We also study whether the pretrained AR representation should remain fixed during denoiser adaptation (Table2).

Table 1:Design choices for Nemotron-Labs-TwoTower using block sizeS=32S{=}32on an internal Nemotron-3-Nano-30B-A3B base checkpoint. We start with bidirectional attention, causal Mamba, and the phase-1 data blend, then add bidirectional Mamba, adaLN time conditioning, and continued training on the phase-2 data blend.Bidirectional Mamba.A right-to-left Mamba scan leaves generation essentially unchanged (72.94 to 72.96) while reducing code and math (68.64 to 68.05 and 80.57 to 79.78). Since this variant roughly doubles denoiser SSM compute, we keep Mamba causal and rely on bidirectional attention within the noisy block for denoising context.

Time conditioning.AdaLN time conditioning improves generation, code, and math from 72.94, 68.64, and 80.57 to 74.12, 69.61, and 81.30, with only 1.5M added parameters. The gain shows that the denoiser benefits from explicit access to the current noise level.

Data curriculum.Continuing the time-conditioned denoiser on the phase-2 blend further improves generation, code, and math to 75.11/71.51/82.08. This mirrors the backbone’s own curriculum. Broad phase-1 coverage is useful for initial adaptation, while the higher-quality phase-2 blend improves downstream generation, code, and math.

Tower Decoupling.The decoupled two-tower design gives the smallest quality drop, while continued backbone AR training leads to lower accuracy on generation and math. Tying the towers, i.e., sharing weights between the towers, under a joint AR+diffusion loss is substantially worse under both AR and diffusion decoding modes. These results suggest keeping the context tower frozen and adapting a separate denoiser.

Table 2:Tower-decoupling ablation after∼167\sim 167B phase-1 tokens, reported as relative accuracy change from Nemotron-3-Nano-30B-A3B. Nemotron-Labs-TwoTower keeps the context tower frozen and trains a separate denoiser. Alternative rows continue backbone AR training or tie the two towers under a joint AR+diffusion objective, evaluated with AR or diffusion decoding as indicated.

3.3Training Block Size

Training block size controls how many tokens the denoiser resolves before the context tower is updated. Larger blocks expose more parallelism but require denoising a longer span from a fixed prefix; smaller blocks stay closer to autoregressive conditioning at lower speedup. Table3reports block-size ablations on the internal Nemotron-3-Nano-30B-A3B checkpoint.

Table 3:Training block-size ablations on an internal Nemotron-3-Nano-30B-A3B checkpoint. The first three rows sweep phase-1 training block size; the next three rows continue training on the phase-2 blend. Training and inference use the same block size. Throughput is reported as wall-clock speedup over the AR baseline for each ablation run.The trend is consistent. Reducing the training block size improves quality. In phase 1, moving fromS=128S{=}128toS=32S{=}32improves generation, code, and math from 69.59/63.56/75.73 to 72.94/68.64/80.57. After phase-2 continuation,S=16S{=}16reaches 77.10/74.56/85.45, whileS=8S{=}8gives only marginal additional generation and math quality at lower throughput. We useS=16S{=}16as the default training block size because it captures most of the quality gain from smaller blocks while retaining a clear speed advantage over the AR baseline.

3.4Sampling Block Size

We next separate training block size from sampling block size. Table4fixes the phase-2 checkpoint trained atS=16S{=}16and varies only the block size used during sampling.

Table 4:Sampling block-size sensitivity. We fix the Nemotron-Labs-TwoTower checkpoint trained at block sizeS=16S{=}16on the phase-2 blend and vary only the sampling block size.The results are asymmetric. Sampling with blocks larger than the training block size is poorly matched to the denoiser, especially on generation-heavy tasks. HumanEval falls from 76.40 atS=16S{=}16to 19.85 atS=64S{=}64, while GSM8K and MATH-500 fall to 2.20. Sampling with smaller blocks is more robust: theS=8S{=}8setting slightly improves MMLU, GSM8K, MATH-500, and multilingual accuracy relative toS=16S{=}16. However, smaller blocks require more frequent context-tower updates, which lowers throughput as shown in Table3. We therefore useS=16S{=}16as the default block size, since it preserves most of the quality benefit of smaller blocks while retaining higher throughput.

3.5Released Checkpoint

For the released Nemotron-Labs-TwoTower checkpoint, we initialize from the final Nemotron-3-Nano-30B-A3B base model and train the denoiser in three stages: phase-1 adaptation at block sizeS=32S{=}32, phase-2 continuation atS=32S{=}32, and a final phase-2 continuation atS=16S{=}16. The first two stages use the larger block size for training efficiency, while the last stage adapts the denoiser to the default block size used for sampling. The quality–throughput curve in Figure1(b)and the category-level results in Figure2are reported from this released checkpoint.

3.6Sampling Dynamics

Figure 3:Sampling dynamics for the released Nemotron-Labs-TwoTower checkpoint (γ=0.8\gamma{=}0.8,S=16S{=}16). Statistics are computed from generation traces over 100 problems per task and include only blocks that contribute to the extracted answer; AppendixAshows the full task set and unfiltered traces.(a)Percentage of answer-producing blocks that complete at each diffusion step for GSM8K and MMLU-Pro. Most blocks finish in the first few steps, with MMLU-Pro showing a broader tail.(b)Distribution of token commitments by position and diffusion step for GSM8K (log scale; zero counts are mapped to one for visualization). Earlier positions are usually committed earlier, producing an autoregressive upper-left triangular pattern.We study the sampling behavior of the released Nemotron-Labs-TwoTower checkpoint under the default confidence-unmasking setup (γ=0.8\gamma{=}0.8,S=16S{=}16). Here a block denotes the group ofSStokens refined jointly before being committed to the context tower. We sample 100 problems from each multi-token generative benchmark and record the diffusion trace for every answer-producing block; Figure3shows representative task views, and AppendixAextends the analysis across the full task set.

Block completion is adaptive. In Figure3a, both GSM8K and MMLU-Pro complete many blocks in the first few diffusion steps, but MMLU-Pro retains more mass at intermediate and late steps. AppendixAshows the same pattern across tasks: MGSM and MBPP-Sanitized often finish blocks immediately (Figure6), while code and reasoning tasks sustain longer tails.

Figure 4:Average committed tokens per diffusion step. The first diffusion step commits the most tokens, after which the count drops as the sampler focuses on the remaining low-confidence positions.Commitment counts are strongly front-loaded in time. Figure4shows that the first diffusion step commits the largest number of tokens on average, after which the count drops quickly. Autoregressive decoding commits exactly one token per step; in contrast, Nemotron-Labs-TwoTower commits multiple tokens per step early in refinement, helping explain how iterative block refinement can still yield wall-clock gains despite taking multiple denoising steps. This behavior is consistent with Algorithm2: each step predicts all masked positions in parallel, commits the high-confidence subset, and leaves the harder residual positions for later refinement.

Within a block, commitments also follow a left-to-right ordering. Figure3b shows that earlier positions are typically committed earlier, while later positions are more likely to survive to later diffusion steps, producing an upper-left triangular pattern. AppendixAshows that this behavior is not unique to GSM8K and appears across tasks with varying strength (Figure7). A plausible explanation is a strong left-to-right inductive bias inherited from the pretrained backbone: the context tower is fully causal, and the denoiser combines bidirectional block attention with the same unidirectional Mamba structure; since the backbone contains 23 Mamba layers and only 6 attention layers, this bias may dominate the resulting sampling behavior.

4Conclusion

We presented TwoTower, a two-tower diffusion language modeling approach that adapts a pretrained autoregressive backbone into a block-wise diffusion generator. The context tower preserves the backbone’s causal representation; the denoiser refines noisy blocks with bidirectional block attention, layer-aligned context attention, and context-seeded Mamba states. This separation lets the model reuse the structure learned during autoregressive pretraining while converting token-by-token decoding into iterative block refinement.

On the 30B hybrid backbone, Nemotron-Labs-TwoTower preserves most of the AR baseline’s quality while delivering higher wall-clock generation throughput. The ablations show bidirectional attention and time conditioning improve denoising, causal Mamba is preferable to bidirectional Mamba, and block size/confidence control the quality–throughput trade-off. The released checkpoint starts from an off-the-shelf AR model and trains the denoiser on a fraction of the backbone pretraining budget. At inference time, Nemotron-Labs-TwoTower keeps the context-tower weights resident alongside the diffusion denoiser, increasing the fixed model-weight memory footprint while maintaining a single persistent prefix cache, so the sequence-length-dependent cache memory scales like the AR baseline.

The results show that masked diffusion can serve as a practical decoding adaptation for large pretrained AR models, including hybrid MoE backbones. We release weights, training code, and recipe in theNemotron-Labs-TwoTower collection. In future updates, we plan to add post-trained Nemotron-Labs-TwoTower models to the same collection.

Acknowledgments

We thank Jared Casper, Yonggan Fu, Abhinav Garg, Jiantao Jiao, Ante Jukic, Mikail Khona, Markus Kliegl, Karsten Kreis, Pavlo Molchanov, Rauf Nasretdinov, Keshav Santhanam, Kevin Shih, and David Tarjan for helpful discussions.

References

Appendix AAdditional Sampling Figures

Unlike Figure3, these visualizations (and Figure4) include all generated blocks from Section3.6, including those beyond the extracted answer. They provide a complete view of the sampler, and are not restricted to answer-producing blocks alone.

Figure 5:Average diffusion step at which blockbbcompletes. Plots terminate at different block indices because different tasks generate different numbers of blocks.Figure 6:Percentage of generated blocks that complete at each diffusion step across the full task set. Most blocks finish in the first few steps, with task-dependent completion tails.Figure 7:Distribution of token commitments by position and diffusion step across all generated blocks. Earlier positions are typically committed earlier, producing a left-to-right upper-left triangular pattern with task-dependent strength.