gguf

A detailed benchmark comparing 16 quantizations of Qwen3.6 27B across GGUF, NVFP4, AWQ, AutoRound, and FP8 formats, measuring KL divergence from the unquantized reference. Weight-only GGUF quants generally offer the best quality-size tradeoffs, while vLLM quants vary substantially.

Unsloth releases a GGUF-quantized version of Meta's Muse Glimmer 30B model, designed for local agentic tasks with multimodal input, tool use, and multi-step reasoning.

A skeptical take on the BigBang-v1 finetune of Qwen 3.5, noting suspicious benchmark claims and potential test contamination despite Bartowski's GGUF conversion.

Meta Superintelligence Lab releases Muse Glimmer 30B in GGUF quantized formats for local inference, a distilled causal language model with a perception encoder for autonomous agentic tasks, tool use, multimodal input, and failure recovery on consumer hardware.

A trimmed English-only GGUF version of Kimi K3 (IQ2-XXS) reduces model size from 711GB to 478GB by removing multi-language components, with early tests suggesting it may match or outperform the standard 2-bit version on coding tasks.

A user shares their llama.cpp server configuration for running Qwen 3.6 27B on an RTX 5090, achieving 80-100 t/s, and asks the community for alternative settings and tips.

A discussion question asking about the theoretically optimal quantization bit-width for LLMs under a fixed memory budget, referencing recent 3-bit/2-bit results and scaling-law work from 2025-2026.

A detailed report on quantizing LiquidAI's LFM2.5-2.6B model with various GGUF and KV cache quantizations, showing it fits on an 8GB Raspberry Pi with minimal degradation, but warning against Q4_K_M.

Unsloth releases new GGUF quantizations of Kimi K3, ranging from 466GB to 649GB, enabling efficient deployment of the large model.

NVIDIA's entire speech stack—ASR, TTS, and codec—is now quantized to GGUF and runs locally on-device via NeMo-Speech.cpp, with new model releases for Magpie-TTS, Nemotron Speech Streaming, and Parakeet.

A user reports that quantising the f16 MTP draft model to Q4_K for Gemma 4 31b gives roughly 10% faster decode (65 to 72 TPS) on dual 3090s compared to the default Q4_0, while Q2_K performs worse.

Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.

Unsloth AI announces DSpark, enabling DeepSeek-V4-Flash GGUF models to run ~1.4–2× faster locally, reaching 120 tokens/s with no accuracy change.

A developer reports that Gemma 4 multimodal features broke in newer llama.cpp builds when using Unsloth's GGUF models, due to an incompatible mmproj file. Switching to ggml-org's official models fixed the issue, highlighting a recurring compatibility concern between third-party quantizers and llama.cpp updates.

Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.

HuggingModels announces the release of Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-GGUF, a fine-tuned and quantized version of the Qwen model.

jabbatheduck released a GGUF quantized version of the REAP expert-pruned DeepSeek-V4-Flash checkpoint, aggressively compressed for memory-constrained inference on consumer GPUs while preserving router and attention precision.

Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.

Qwen3-TTS voice cloning has been merged into mainline llama.cpp, enabling local text-to-speech with voice cloning from short reference audio via the llama-tts binary, supporting multiple languages. Limitations remain, including only the Base model and no server endpoint yet.

A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.