New tensor type layouts for my GGUF uploads

Reddit r/LocalLLaMA 09/10/26, 07:05 PM Tools

gguf quantization tensor-layouts model-optimization huggingface-blog upload-format

Summary

The author shares a blog post about new tensor type layouts for GGUF uploads, detailing research that shows improved model quantization performance.

Hey all, long time no post. Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here: https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :) https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png If anyone has any questions let me know!

Original Article

Similar Articles

Gemma 4 12B Q3: +8.55% Coding Performance From Tensor-Level Quantization Allocation

Reddit r/LocalLLaMA

A developer created a task-aware GGUF quantization pipeline that uses tensor-level bit allocation to improve Gemma 4 12B Q3 coding performance by 8.55% over a hand-tuned imatrix while increasing model size by only 0.119%.

@aisearchio: GLM 5.2 GGUF is already here! 8-bit is ~half the size of the full model. Smaller versions coming soon https://huggingfa…

X AI KOLs Timeline

GLM 5.2 GGUF quantized model is released, with 8-bit version half the size of the full model; smaller versions are coming soon.

moar QAT stuff and hairy ticks

Reddit r/LocalLLaMA

The author releases improved GGUF quantized versions of Gemma 4 models (12B and 31B) using a more accurate quantization-aware training process that achieves lower KLD and higher same-top percentage than stock quantizations.

@antirez: Uploading a new HF imatrix GGUF for 2 bits: same name, different content with fixed down layer of shared experts (there…

X AI KOLs Following

A corrected 2-bit GGUF model file has been uploaded to Hugging Face after fixing a bug in the imatrix computation, leading to improved logits recall and reduced error.

google/gemma-4-12B-it-qat-q4_0-gguf

Hugging Face Models Trending

Google DeepMind releases Gemma 4 models optimized with Quantization-Aware Training (QAT) in multiple formats including GGUF, enabling high quality with reduced memory requirements.