Tag
Testing a 1.56TB Mixture-of-Experts model on a 6GB RTX 4050 laptop, requiring patched memory streaming with NVMe to achieve 0.106 tokens/s decode speed.
A user runs Kimi K3, a 2.8T-parameter open-weight MoE model, on 80 RTX 5090 GPUs using only GDDR7 and Ethernet—no HBM—achieving 20 tok/s single stream, the first frontier model to run on consumer hardware.
A user questions why bartowski claims DeepSeek-V4-Flash is in MXFP4 format when the original model page lists BF16, F32, and other tensor types, not MXFP4.
Discusses the largest dense model that can be loaded in 128 GB RAM using MXFP4 quantization.
A tweet thread introduces a visualizer for micro-scaling/block quant formats like NVFP4 and MXFP4, explaining how these low-precision floats work and their use in LLM inference to reduce memory bandwidth demands.
A technical comparison between nvfp4 and mxfp4 formats, highlighting that nvfp4 uses an additional tensor-wise scale factor to overcome fp4's range limit, allowing more precision in block-wise scale factors.
DealignAI releases CRACK-abliterated and MXFP4/MXFP8 quantized versions of Qwen3.6-27B and 35B models, preserving MTP for faster speculative decoding on Apple Silicon.
This paper decomposes MXFP4 quantization error into three additive components—scale bias, deadzone truncation, and grid noise—and proposes targeted corrections that recover BF16 accuracy to within 0.7 pp on Qwen2.5-3B and 3.0 pp on Qwen3-30B-A3B-Base for LLM reinforcement learning post-training.