@ggerganov: Run GGUF models directly with transformers. This work brings ggml's Metal kernels to the transformers ecosystem, increa…
Summary
This work integrates ggml's Metal kernels into the transformers library, allowing GGUF models to run directly with improved compatibility and performance for fast local inference on Mac.
View Cached Full Text
Cached at: 09/24/26, 12:18 AM
Run GGUF models directly with transformers. This work brings ggml’s Metal kernels to the transformers ecosystem, increasing compatibility and performance. More info below
Yes. There is already initial support for MXFP4/NVFP4 activations in the ggml CUDA backend (only when the hardware supports it natively). FP8 is in planning/discussion phase.
@ggerganov GGUF checkpoints running in transformers with ggml Metal kernels is a real product win — same local weights, more ways to run them on Mac.
What that still does not settle: fit vs feel are different checks. Desk math for a dense ~27B Q4_K_M (estimate, not a bench): ~16.4 GB file → ~21.3 GB with 8k context + runtime (+~30%). Same “fits” on Mac Studio M5 Max 128 GB (~28.1 tok/s) and RTX 5090 (~82.1 tok/s) — bandwidth decides how it feels.
Want the two-number math — does it fit, and at how many tok/s? Learn to calculate it here:
Similar Articles
@julien_c: transformers x @ggml_org collab
Transformers and ggml_org collaborate to integrate GGML kernels into the transformers library, enabling GGUF file loading at performance levels comparable to llama.cpp.
GGUFs in transformers natively!
Hugging Face announces native support for GGUF files in the transformers library, allowing easier use of quantized models with PyTorch tooling and performance comparable to llama.cpp.
Transformers now runs llama.cpp quants
Hugging Face's transformers library now supports GGUF models from llama.cpp, enabling efficient local inference on consumer hardware through familiar APIs.
@UnslothAI: We made GLM-5.3-Flash run 3.3x faster locally! Local GGUF inference is now 1.6–3.4× faster with optimized decoding and …
Unsloth AI announces optimizations for GLM-5.3-Flash, enabling 1.6–3.4× faster local GGUF inference with multi-token prediction and hardware requirements for running models locally.
@no_stp_on_snek: turboquant+ is now a swappable backend in LocalAI alongside tinygrad and sglang. if you're running GGUF models and want…
turboquant+ backend added to LocalAI, enabling longer context for GGUF models without hardware upgrade.