Tag
Announcement that GGUF quantizations of MiniMax H3 are available, with the Q2 version being only 8.49 GB for lower-end GPUs.
DeepSeek-V4-Flash-0731 is released in GGUF format at 2.88 bits per weight (95 GiB), making the 284B-parameter MoE runnable on 128 GB machines. Includes a detailed operating guide and a required TurboQuant llama.cpp fork.
Hugging Face repository providing GGUF quantizations of MiniMax-H3 models for use with ComfyUI, including directory structure and links to required VAEs.
A GitHub patch allows running DeepSeek V4 Flash in LM Studio on 128GB Macs by sideloading antirez's llama.cpp fork, working around struct-layout drift, decoding splits, and code-signing issues.
Brian Roemmele reports that DeepSeek V4 Flash (304B, 1M context) now runs locally on Apple Silicon via the ds4 engine, sharing GGUF quantized builds with a fresh imatrix. The Hugging Face repo provides installation instructions and notes that these files are ds4-specific, not for llama.cpp.
A model card for Huihui-DeepSeek-V4-Flash-0731-abliterated-GGUF, an abliterated (uncensored) GGUF quantized variant of DeepSeek-V4-Flash, designed for local use with llama.cpp and ds4.
This Hugging Face model card presents LiquidAI's LFM2.5-2.6B model in GGUF quantized format, with instructions for running it locally via llama.cpp, vLLM, Ollama, and other tools.
Unsloth releases an IQ3 GGUF quantization of DeepSeek-V4-Flash-0731, enabling local inference via llama.cpp, Ollama, LM Studio, and other tools.
DeepSeek-V4-Flash-0731 is shown running as an unsloth GGUF quant on a single 40GB A100, with 17.7 tok/s and 6 experts loaded into VRAM, enabling a full agentic coding loop.
Unsloth teases the upcoming release of DeepSeek V4 Flash GGUF quantized model on Hugging Face.
This is a highly experimental GGUF version of the 2.8T-parameter Kimi K3 MoE model, with 55% of experts pruned and quantized to ~2.15 bpw (319 GiB). It requires a specific llama.cpp PR and custom patches to run, and includes detailed instructions for usage.
Recent versions of llama.cpp now automatically load MTP tensors for draft-mtp architectures, even if speculative decoding is not enabled, potentially increasing VRAM usage for users with bundled MTP blocks in their GGUF files.
User shares first home-lab results running Kimi K3 on 768GB DDR5 and 2x5090 using a llama.cpp fork and Q2_K quant, reporting prefill speeds of 50-70 tps and decoding tps that increases over time.
Kimi K3 is a massive 2.9 trillion parameter mixture-of-experts model with 104B active parameters, 1M context length, and native MXFP4 training, now available in GGUF quantizations ranging from 540GB to smaller sizes, though requiring substantial hardware to run.
User successfully runs the Kimi-k3 model using llama.cpp on high-end hardware, achieving low tokens per second (0.41 prompt eval, 0.23 generation).
A user notices that the Q4_K_M quantized version of Laguna S 2.1 increased from 68GB to 96GB, likely due to using more FP16 layers, and discusses potential issues with quantization and context looping.
Recent llama.cpp commits broke preserve_thinking behavior for older DeepSeek V4 gguf chat templates, causing issues in coding agent contexts. The fix is to override the gguf template with a new one using --chat-template-file.
Kimi K3 text-only model is now supported in llama.cpp, enabling local inference of this open-source LLM using the C++ inference engine.
The article details the current support status for the Ling-3.0-flash model weights across inference engines: SGLang commits to day-0 integration, vLLM awaits open weights, and llama.ccp lacks conversion for the Bailing MoE variant. It notes that the release pattern involves a free API window followed by open-sourcing, as seen with Ling-2.6-flash.
The article evaluates how different quantizations of the Qwen3.6-27B model affect output quality using KL divergence and top-1 token accuracy, as well as visual examples like SVG drawings.