ShimQuant enables running Nemotron-3.5-Lightning on 16 GB GPUs with a 11.77 GiB quantized file, providing a usable option below previous 18 GiB limits.
TL;DR: Every public low-bit GGUF of this model is secretly ~4.70 bpw. Shim the rows to 256 and it becomes a real 3.07 bpw / 11.77 GiB file that runs 262K context on 16GB. Needs patched llama.cpp — not LM Studio or Ollama. In the AtomicChat HuggingFace repo it says "There is currently no good 16 GB option for this model, from anyone." That was true, I wanted to figure out why, and it's a quantizer problem, not a model problem. k-quants and i-quants need the row width to divide by 256. Nemotron's don't, so about 99% of its parameters can't legally take one. llama-quantize swaps in a 32 block type instead and keeps the filename you asked for, which is why every low bit quant of this model comes out around 4.70 bpw regardless of its label. If you saw my census post yesterday, same bug, Nemotron is just the worst case I found. Smallest usable build anyone shipped was ~18 GiB. ShimQuant shims each affected row out to the next multiple of 256 so the low bit types actually apply, then slices the activations back at inference. That gets it to 3.07 bpw, 11.77 GiB, 262,144 context on a 16 GB card. So far I've measured it two ways, KL divergence against a Q8_0 reference and HumanEval. Against stock IQ2_M it's 6.2 GiB smaller and less divergent. On HumanEval it ties AtomicChat's 19.65 GB build at 91.5% while being 7 GB smaller. More benchmarks are running, I'll update the card as they land. It does not beat stock IQ3_XXS on divergence. That one is 6.2 GiB bigger and three times closer to Q8. So the claim isn't that this is the best file, it's that below ~18 GiB the stock quantizer gives you nothing usable for this model and this is usable in that gap. The catch This will not load in stock llama.cpp, LM Studio, Ollama, or anything unpatched. It needs the ShimQuant patch. It fails right away instead of corrupting quietly: check_tensor_dims: tensor 'blk.0.ssm_in.weight' has wrong shape; expected 2688, 10304, got 2816, 10304 If you don't want to build a patched llama.cpp then this file isn't for you. But it's the only usable option under 18 GiB, so if you're on a 16 GB card and want to run Nemotron it's this or nothing. Model: https://huggingface.co/BoldingBuilds/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ShimQuant-GGUF Patch: https://github.com/JoshBolding/shimquant Census across 25 repos and 443 quants: https://github.com/JoshBolding/ggufaudit
NVIDIA released Nemotron 3.5 Lightning 30B-A3B-NVFP4, a hybrid MoE LLM with 3B active parameters, up to 1M context, and speculative decoding support for efficient single-GPU inference.
A new ternary quantized version of Qwen3.6 27B, called Bonsai 27B, allows running the model on 12GB GPUs with 10x less memory and 95% of original performance, making it accessible for local deployment.
NVIDIA releases Nemotron 3.5 Lightning model, with 30B total parameters and only 3B active, supports 1M token context, commercially usable, local deployment friendly, output speed up to 4x faster.
A quantized version of Qwen3.6 27B using a pure Q4_K_M method fits entirely in 16 GB VRAM, achieving up to 40 tok/s token generation speed with MTP, and significantly reducing model size compared to other GGUF variants.
NVIDIA released Nemotron 3.5 Lightning, a 30B-parameter MoE model with only 3B active parameters, optimized for agent execution tasks. It claims faster, cheaper tool calls and agent execution while staying fully open-source under OpenMDW-1.1.