gguf

Tencent's Hy3 295B model now available in 1-bit and 4-bit GGUF formats, achieving 2.2x faster local inference compared to cloud API while maintaining quality, as demonstrated by running on 4x RTX 5090 with 128GB VRAM.

The unsloth/inkling-GGUF page provides the quantized GGUF version of Inkling, a 975B-parameter multimodal MoE model (41B active) from Thinking Machines, designed for text, image, and audio inputs with open weights and support for local deployment via libraries like Unsloth, SGLang, and vLLM.

Empero AI releases new GGUF quantizations for Qwythos-9B-Claude-Mythos-5-1M with a fixed chat template, supporting 1M context and function calling, dominating base Qwen3.5-9B in benchmarks.

MawForge presents a memory-bounded approach for serving large Mixture-of-Experts (MoE) language models on constrained unified-memory machines by storing the full model on disk and materializing expert tensors into a bounded cache on demand. Experiments on a MacBook Pro M5 Pro demonstrate effective serving of 34GB and 25GB models within a 24GB memory envelope, with analysis of cache size trade-offs and speculative decoding results.

GnLOLot releases GGUF quantizations of the MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking model, a 1B parameter thinking model fine-tuned on Fable 5 data with improved tool/function calling compared to V1, designed for local deployment via llama.cpp and compatible runtimes.

Poolside releases GGUF quantizations of the Laguna S 2.1 AI model, including a DFlash speculative decoding draft model, enabling efficient local inference with llama.cpp.

LuffyTheFox released a GGUF quantized version of the Qwen3.6-35B-A3B model, denoised with the Genesis algorithm to reduce tensor noise and improve instruction following, based on an uncensored aggressive variant merged with Hermes finetune data.

LuffyTheFox releases Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V5-GGUF, a modified version of Qwen model using the Genesis post-training algorithm to repair signal purity and reduce noise via custom SVD on GGUF-format tensors.

A new uncensored GGUF model based on Qwen3.6-35B-A3B with technical improvements using data reconstruction and noise suppression techniques. Features reduced refusal rates and enhanced performance via a novel repair method.

An interactive Jacobian-lens visualizer and live steerer for GGUF models running on llama.cpp, enabling real-time model interpretability and control.

GGUF quantizations of Tencent's Hy3 295B MoE model and NVIDIA's audio-capable 30B MoE model are released with detailed benchmarks, imatrix calibration, and full reproducibility data.

The author shares surprising results from systematic tests on how different quantization levels (e.g., Q4_K_M, Q5_K_M) affect model capabilities separately, showing that math accuracy degrades more than knowledge tasks, and calls for more rigorous testing on context decay across quant levels.

Qwythos-9B-v2-GGUF is the GGUF quantization of the improved Qwythos-9B-v2 model, featuring fixed looping behavior, restored MTP head, and preserved reasoning capabilities.

Qwen-3.5-Opus-GLM-27B is a 27B parameter GGUF merge combining the strengths of Qwen's reasoning, Opus's creativity, and GLM's efficiency, designed for high-performance local AI without cloud dependency.

Step 3.7 Flash is now available as an IQ4_XS GGUF quantized model, enabling efficient local inference with preserve_thinking support via llama.cpp, vLLM, Ollama, and other tools.

The Hy3 model is now supported in llama.cpp via a pull request, with GGUF quantizations available. Early testing shows coherent output from Q2_K at 10-11 t/s on high-end hardware.

A cheat sheet listing top AI models optimized for 32GB VRAM using GGUF Q4 quantization, with direct download links from HuggingFace. Includes models from Qwen, DeepSeek, Llama, and Mistral families, with tips on quantization and context settings.

Qualcomm launched GenieX, an SDK for running LLMs on Windows laptops, achieving 20 tok/s on Gemma 4 26B and supporting llama.cpp with GGUF models.

TensorSharp is an open-source .NET library for running LLM inference locally, supporting GGUF models and offering a CLI, web chatbot, and OpenAI-compatible APIs with multiple backend options (CUDA, Metal, CPU).

Prism ML releases Bonsai-27B-gguf, a 27-billion parameter language model with binary (1.125-bit) weights, achieving a ~14x size reduction while retaining ~90% of FP16 reasoning performance. It runs on consumer hardware with high throughput.