gguf

A trimmed English-only GGUF version of Kimi K3 (IQ2-XXS) reduces model size from 711GB to 478GB by removing multi-language components, with early tests suggesting it may match or outperform the standard 2-bit version on coding tasks.

A user shares their llama.cpp server configuration for running Qwen 3.6 27B on an RTX 5090, achieving 80-100 t/s, and asks the community for alternative settings and tips.

A discussion question asking about the theoretically optimal quantization bit-width for LLMs under a fixed memory budget, referencing recent 3-bit/2-bit results and scaling-law work from 2025-2026.

A detailed report on quantizing LiquidAI's LFM2.5-2.6B model with various GGUF and KV cache quantizations, showing it fits on an 8GB Raspberry Pi with minimal degradation, but warning against Q4_K_M.

Unsloth releases new GGUF quantizations of Kimi K3, ranging from 466GB to 649GB, enabling efficient deployment of the large model.

NVIDIA's entire speech stack—ASR, TTS, and codec—is now quantized to GGUF and runs locally on-device via NeMo-Speech.cpp, with new model releases for Magpie-TTS, Nemotron Speech Streaming, and Parakeet.

A user reports that quantising the f16 MTP draft model to Q4_K for Gemma 4 31b gives roughly 10% faster decode (65 to 72 TPS) on dual 3090s compared to the default Q4_0, while Q2_K performs worse.

Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.

Unsloth AI announces DSpark, enabling DeepSeek-V4-Flash GGUF models to run ~1.4–2× faster locally, reaching 120 tokens/s with no accuracy change.

A developer reports that Gemma 4 multimodal features broke in newer llama.cpp builds when using Unsloth's GGUF models, due to an incompatible mmproj file. Switching to ggml-org's official models fixed the issue, highlighting a recurring compatibility concern between third-party quantizers and llama.cpp updates.

Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.

HuggingModels announces the release of Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-GGUF, a fine-tuned and quantized version of the Qwen model.

jabbatheduck released a GGUF quantized version of the REAP expert-pruned DeepSeek-V4-Flash checkpoint, aggressively compressed for memory-constrained inference on consumer GPUs while preserving router and attention precision.

Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.

Qwen3-TTS voice cloning has been merged into mainline llama.cpp, enabling local text-to-speech with voice cloning from short reference audio via the llama-tts binary, supporting multiple languages. Limitations remain, including only the Base model and no server endpoint yet.

A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.

Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.

A user reports that DeepSeek V4 Flash, running as a 2-bit quantized GGUF on dual RTX 3080s, is the first local model to score 100% on a real-world SQL benchmark, matching frontier models like Opus 4.7 and GPT-5.5.

Hugging Face page for DeepSeek-V4-Flash-0731-GGUF, a GGUF quantized version of DeepSeek V4 Flash with an updated template supporting reasoning levels, ready for use with llama.cpp, vLLM, Ollama, and other local inference tools.

atomic.chat released 14 quantized GGUF builds of DeepSeek V4 Flash 0731, from lossless BF16 to 1-bit. They recommend AD-IQ2_M for 128GB hardware, which matches the original's token choice 83.6% of the time.