Tag
Quantization-Aware Healing is a method that recovers compressed 4-bit language models by distilling directly from the original uncompressed model, offering faster and more stable performance than Quantization-Aware Training.
Nunchaku, a 4-bit diffusion inference engine based on SVDQuant, is now natively integrated into Hugging Face Diffusers, enabling fast and memory-efficient loading of quantized diffusion models with a simple from_pretrained() call.
Nota AI releases a 4-bit quantized version of Upstage's Solar Open2 250B MoE model, using proprietary NVFP4 quantization that requires NVIDIA Blackwell GPUs.
A new wave of large language models including GLM 4.5, Qwen 3.5, MiniMax M2.7, Deepseek V4 Flash, Xiaomi MiMo 2.5, StepFun 3.7 Flash, and Tencent Hy3 can now be run locally on a dual DGX Spark setup with 250GB usable memory at 4-bit quantization, costing approximately $7,000–$8,000.
Cohere releases North Mini Code, a 30B-A3B open-weights model with 4-bit quantization for code generation and agentic coding tasks, supporting 256K context.
This paper adapts paired binary sample-size calculations to 4-bit quantization benchmarks, providing a conservative minimum detectable effect (MDE) bound that helps benchmark designers determine reliability before running experiments. A pilot audit shows that much of the observed variance across small subsamples is binomial sampling noise, not true model unreliability.
Gemma 4 is a 4-bit quantized model optimized for Apple Silicon, enabling fast local inference on Mac devices, reducing reliance on cloud computing.