[Paper] EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

The EdgeRazor method uses an entropy-guided distillation process to better translate a teacher model's logit probability distributions into the student model's low-bit / mixed-precision hidden-layer features, without attempting to preserve the teacher model's parameter structures. This is more computationally expensive than existing quantization methods, but much less so than QAT, and yields better results. The student model preserves more of the teacher model's competence at extremely low parameter precision (the authors demonstrate 1.88 bits per parameter). Since it's not a different internal representation like traditional quantization, inference implementations like llama.cpp do not need to be modified to take advantage of it. Hopefully this means more-useful high-parameter/low-memory models in our future, so we can eke more competent inference out of our consumer-grade GPUs. The paper: https://arxiv.org/abs/2605.04062 The authors' code: https://github.com/zhangsq-nju/EdgeRazor The authors applied their technique to a few models and uploaded them to Huggingface: https://huggingface.co/collections/zhangsq-nju/edgerazor-nbit Unfortunately since EdgeRazor is somewhat compute-intensive, their example models are all pretty tiny: MobileLLM, Qwen3-0.6B, Qwen3-1.7B, and Qwen2.5-Omni-7B