gguf

User shares optimization benchmarks for DeepSeek-V4-Flash (Q2_K) running on an RTX 5090 using a fork of llama.cpp, achieving 21.3 tokens/s generation and 1 million context size.

A GGUF quantized version of MiniCPM5-1B-Claude-Opus-Fable5-Thinking model is released on Hugging Face, with usage instructions for llama.cpp, vLLM, and Ollama.

A user asks about Ubuntu's compatibility for local AI tools like vLLM, llama.cpp, and ComfyUI when switching to Linux.

Another open-source tool on GitHub, Shimmy, is a single 5MB file written in Rust that provides fast and stable local inference with a full OpenAI-compatible API, targeting Ollama's pain points. It starts in under 100ms and uses about 50MB of memory.

GGUF quantizations of DeepSeek V4 Flash in 2-bit, 3-bit, and 4-bit precisions, made available on Hugging Face for local inference with tools like llama.cpp and Ollama.

ThinkingCap-Qwen3.6-27B is a fine-tuned version of Qwen3.6-27B that uses 50% fewer thinking tokens on average while maintaining answer quality. This repository provides GGUF quantizations for local inference with llama.cpp.

Bartowski has released a GGUF quantized version of DeepSeek-V4-Flash, inviting comparison with Antirez's version.

Sharing a new recipe for dflash speculative decoding in llama.cpp, achieving ~70 TPS on a single RTX 3090 using Qwen3.6-27B GGUF with a draft model.

Kyle Hessling announces the release of Qwopus-3.6-35B-A3B-MTP-Coder, a fast MOE model specialized for coding, available in GGUF format.

An update on the Ornith-1.0-35B GGUF model introduces a native MTP speculative-decode graft for faster inference on a single GPU, achieving ~1.3-1.35x decode speedup while maintaining near-identical token distribution. Benchmark numbers for throughput, TTFT, and long-context performance across multiple quants are provided.

Qwythos 9B is a new open-source, uncensored reasoning model based on Qwen3.5, offering GGUF quantizations, 1 million token context, vision, and function calling, with significant performance improvements over the base model.

MiaAI-Lab provides a simple Bash start/stop script for running Qwen3.6 27B/35B GGUF models via llama-server, optimized for speed and coding performance.

A quantized GGUF version of the abliterated GLM-5.2 model is released on Hugging Face, enabling local inference with various tools like Transformers, llama.cpp, and vLLM.

The author claims Qwen3-VL-2B is the only viable vision-language model for JSON extraction on low-end hardware, outperforming larger models like Qwen3-VL-4B, yet it is absent from major benchmarks.

Meet Gemma 4 12B Agentic Fable5, a locally-run GGUF model designed for coding, terminal tasks, and agentic workflows, with 206k downloads.

Ornith-1.0-9B is a new 9B parameter AI model optimized for 8-12GB GPUs, achieving strong performance on agentic coding benchmarks, matching or surpassing models 2-3x its size.

DeepReinforce AI releases Ornith-1.0, a self-improving open-source model family for agentic coding, including a 35B MoE variant that achieves state-of-the-art performance on coding benchmarks and runs efficiently on single GPUs like the 5090.

A new uncensored GGUF quantized version of the Qwythos-9B-Claude-Mythos-5-1M model, created using abliteration, is released on Hugging Face.

A user tested the unsloth quantized GLM-5.2 model on a high-end consumer-like system with dual RTX 5090, achieving 12 tokens per second.

deepreinforce-ai releases Ornith-1.0-35B-GGUF, a state-of-the-art open-source coding agent model that uses self-improving reinforcement learning to jointly optimize scaffold and solution generation, achieving SOTA performance on coding benchmarks.