So, i wanted to test TwoTowers from Nvidia but it was a bit big. So i quantized it. but the speed is very slow. looks like vllm and llamacpp are still working on supporting block-diffusion.

So the hobbit is here… for whenever twotowers is supported in these inference engines.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B…

thetom-ai/Nemotron-TheHobbit-30B-A3B · Hugging Face

Source: https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#nemotron-thehobbit-30b-a3bNemotron-TheHobbit-30B-A3B

The Two Towers, shrunk to fit in one.🧙

NVIDIA’sNemotron-Labs-TwoTower-30B-A3B-Baseis a block-diffusionLM whose two-tower design officially needs2× 80 GB (H100/A100)— ~59 GB per GPU in BF16 — to run its diffusion mode. This is that model put on aConfig-I diet: a4-bitTQ4\_1Squantization that squeezesboth towers into ~38 GB, so the full ARandmask-diffusion paths run on asingle GPU. A little Hobbit that walks into Mordor where the Two Towers needed a fellowship.

Verified end-to-end on one NVIDIA DGX Spark (GB10, 128 GB unified).

⚠️Research artifact — correct but slow.Runs in eager PyTorch with kernel-free torch fallbacks (no fusedmamba\_ssm/causal\_conv1d), doing two datacenter-GPUs’ work serially on one card:~2–4 tok/son a GB10. Shared so the community can test it on real hardware and speed it up — seeHelp wanted.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#what-config-i-meansWhat “Config-I” means

Config-IisTurboQuant’sTQ4\_1Sweight format: aWalsh–Hadamard (RHT) rotationapplied per 32-element block, then4-bit 16-level Lloyd-Maxquantization withdual half-block scales(~5 bits/weight). The rotation is the trick — itGaussianizesevery weight distribution, so one fixed 4-bit codebook is near-optimal foreverytensor type at once. That’s what lets it survive theMamba-2 state-space recurrence, where naïve int8 collapses into garbage.

Measured per-weight reconstructioncosine ≈ 0.997,uniformacross Mamba-2, attention, and the 128-expert MoE tensors. SSM scalar params (A\_log,dt, conv1d,D), norms, router gates, the PixArt-style adaLN time-conditioning tables, and embeddings are kept at BF16. Both AR and diffusion produce coherent, mutually-consistent text.

SizeBF16 (NVIDIA reference)118 GB~59 GB × 2 GPUsConfig-ITQ4\_1S38 GB1 GPU

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#the-model-underneathThe model underneath

A two-towerblock-diffusionLM: a context tower + a time-conditioned denoiser tower, each aNemotron-H hybrid(Mamba-2 SSM + attention + 128-expert MoE, top-6, ~3 B active). It generates a block of tokens in parallel via iterative masked denoising instead of one token at a time.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#running-itRunning it

pip install torch transformers accelerate safetensors einops numpy
python run_configi.py     # loads both towers on cuda:0, runs an AR + diffusion demo

run\_configi\.pyis self-contained: it injects torch stand-ins for themamba\_ssm/causal\_conv1dkernels the denoiser imports (soyou do not need mamba-ssm, which won’t build on many GPUs), swaps the quantized Linears for an on-the-fly Config-I dequant layer, and applies a sync-free MoE. See the script for theload\(\)/generate\_ar/generate\_mask\_diffusionAPI.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#performance–why-its-slowPerformance & why it’s slow

NVIDIA’s throughput comes from 2× 80 GB with fusedmamba\_ssm/causal\_conv1dTriton kernels. Here, on one GPU, the model is bottlenecked by the absence of fused kernels and by per-forward 4-bit dequantization. Speedups already applied —torch\.compile’d dequant, async-free MoE(the stock loop did18 k CPU↔GPU syncs/token), andbf16 tensor-coredequant +autocast(the stock path fell to fp32 SIMT matmuls) — together **9×** (0.24 → ~2 tok/s diffusion). Datacenter throughput needs fused kernels and/or the 2-GPU reference setup.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#engine-support-status-july-2026–%F0%9F%9A%A7-in-progressEngine support status (July 2026 — 🚧 in progress)

Fast native inference for the two-towerdiffusionpath doesnotexist in llama.cpp or vLLM yet — the base model is only days old. As of this writing there isno TwoTower-specific support in either engine(searched both repos; zero “two-tower” PRs/issues). Current state:

• vLLM— no two-tower diffusion support. Itdoessupport theNemotronH AR backbone(NemotronHForCausalLM, PRs#47088,#47440), sovllm serveruns theautoregressivepath,notthe diffusion two-tower. Generic block-diffusion plumbing is landing viaDiffusionGemma(#45982,#47462,#47464) — the infrastructure a TwoTower port would build on.
• llama.cpp— no two-tower / cross-attention-denoiser path. Block-diffusion is being added forDiffusionGemma(#24427) andLLaDA 2.0(#17454). Any “GGUF quants for llama.cpp/Ollama/LM Studio” you see forNemotronrefer to theAR backbone(Nemotron-3-Nano),notthis diffusion model.

**vLLM and llama.cpp integration to support this is still in progress — will need to revisit.**Until a fast engine path lands, NVIDIA’s 2× 80 GB reference (fusedmamba\_ssm/causal\_conv1dkernels) is the throughput path, and this repo’srun\_configi\.pyis the single-GPU (eager, ~2–4 tok/s) route. Whoever ports TwoTower to a fast engine will most likely build on the DiffusionGemma / LLaDA block-diffusion work landing now.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#help-wantedHelp wanted

• Test on 2× 80 GB(or a single H200/B200) — report AR-vs-diffusion quality + throughput.
• fp8 weight cache— dequant once to fp8-resident (~63 GB, still 1 GPU) to kill per-forward dequant (est. ~10–15 tok/s).
• Fused kernels / CUDA graphsfor the two-tower diffusion path.
• Retention benchmarks(MMLU / GSM8K / HumanEval, AR vs diffusion) vs the BF16 reference.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#reproductionReproduction

quantize\_twotower\_gpu\.py(included) quantizes the BF16 checkpoint to Config-I on-GPU in ~7 min.

https://huggingface.co/thetom-ai/Nemotron-TheHobbit-30B-A3B#attributionAttribution

• Base model:© NVIDIA,Nemotron-Labs-TwoTower-30B-A3B-Base, under the NVIDIA Open Model License. This is an independent community quantization, not affiliated with or endorsed by NVIDIA.
• Quantization:TurboQuantConfig-I /tq4\_1s(RHT-rotated 4-bit).
• Torch fallback kernels, sync-free MoE, and single-GPU loader added for this release.