Tag
ISTA-DASLab released GSQ-RCO non-uniform GGUF quantizations of DeepSeek-V4.1-Flash-1017 (763.2B MoE) at 1.75-2.25 bits per weight, assigning per-tensor quantization types via gradient-based search so the model can run on a single 12GB GPU with llama.cpp.
Introduces Variable Bit-width Quantization (VBQ), a training-time method where each group of 64 weights learns its own bit-width (1,2,4,8) via Gumbel-Softmax relaxation. VBQ discovers a heterogeneous allocation that yields a 'bigger-but-smaller' regime, e.g., a 131M parameter model at 1.82 mean bits beats a 55M FP16 model while using less storage, and a 1.46B model matches a 593M FP16 with ~3.7x less storage.