What is the biggest dense model that would fit into 128 GB RAM (at MXFP4)?
Summary
Discusses the largest dense model that can be loaded in 128 GB RAM using MXFP4 quantization.
Similar Articles
NVFP4 kv cache quantization on sm120 will make 32GB VRAM systems very capable
NVFP4 KV cache quantization on sm120 significantly improves memory efficiency for large language models, enabling 32GB VRAM systems to achieve ~60 tok/sec inference at 196k context size with Qwen3.6-27B.
It's unbelievable! I used the mmap function in llama.cpp to fit Qwen3.8-Flash-Next IQ3_XSS into 16G+64G RAM, and the speed still reached 26t/s.
A user successfully used the mmap function in llama.cpp to fit the Qwen3.8-Flash-Next IQ3_XSS model into 16GB+64GB RAM, achieving a speed of 26 tokens per second, which outperforms a larger non-MOE 30B model.
Is there a GLM 5.3 Flash Antirez/DS4 GGUF targeted at 192 GB RAM?
The author asks whether a GLM 5.3 Flash Antirez/DS4 GGUF model around 192 GB RAM exists and seeks advice on creating such a model.
Maxing out 64GB of RAM - Qwen3.5 122B A10B at UD-Q2_K_XL w/ MTP fully replaced Qwen3 Next 80B at UD-Q4_K_XL for me
A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
High VRAM local coding model — still Qwen 3.6 27B?
The user discusses their experience with Qwen 3.6 27B for local coding tasks and asks for recommendations for larger models (100B+) suitable for systems with 224GB of VRAM.