Tag
A quantized GGUF version of the Kimi K3 model is now available, optimized for running on RTX 3090 GPUs.
Demonstrates achieving 256k context on a 16GB RTX 4080 Super using Ternary Bonsai 27B Q2_0 model with llama.cpp, achieving up to 141 tok/s generation speed.
A 140 GB IQ2_XXS REAP quantized version of GLM 5.2 for coding has been created, and the author is looking for testers.
Ornith-1.0-35B Q3_K_M is a 3-bit quantized version of a 35B parameter model, requiring about 17 GB VRAM, with KLD checking against BF16 to ensure fidelity.
NVIDIA's nvfp4 quantized version of Qwen 3.6 35B is recommended over the Unsloth variant, offering better performance. The model is available on HuggingFace for use in AI applications.
Unsloth released Gemma 4 QAT MTP assistant models as GGUF files on Hugging Face, available in q8_0 and larger quantization formats.
The author released a pure Rust, CPU-only inference implementation of the LFM2.5-8B-A1B model (4-bit Q4KM quantization), achieving a decode speed of approximately 37 tokens/s and memory usage around 7GB. The goal is to make LLMs runnable on cheap VPS or older machines. The implementation is open source and published as a cargo crate.
Jackrong/Qwopus3.6-27B-v2-MTP-GGUF is a quantized GGUF version of a 27B parameter language model, hosted on Hugging Face with instructions for use with various libraries and tools.