Tag
Cloud AI services like ChatGPT, Claude, and Grok are down, highlighting the reliability of local AI tools such as llama.cpp.
This is the GGUF version of the K2-Horizon-MoVA-36B-A4B model, a Mixture-of-Experts AI model with 36B total parameters and 4B active per token, optimized for use with llama.cpp. It demonstrates frontier-class performance on agentic and reasoning benchmarks, competing with larger and closed models.
Demonstrates running the quantized Qwen3.6-35B model on a low-end laptop to create a Zelda-like RPG in 24 minutes, showing that GPU-poor scenarios are manageable in 2026.
User benchmarks and details a GPU-resident expert cache PR in llama.cpp that boosts decode speed for Qwen3.8-Flash-Next on a dual RTX 3090 system from 17 to 25-29 tokens per second.
The tweet shares GGUF builds of Tencent's Hy4-preview model, including quantized versions and instructions for running them with a patched llama.cpp.
A user questions the relevance of MLX on Apple M5 chips after llama.cpp updates for Metal improve GGUF model prefill performance, making MLX potentially redundant for certain tasks.
Vision support for the DeepSeek-V4-Flash-Vision-Exp model has been merged into the llama.cpp framework, enhancing its capabilities for multimodal AI inference.
Showcases the performance of the Qwen3.8-27B AI model running on a 12GB RTX 3080 Ti GPU, achieving 47 tokens per second at 128K context using llama.cpp with specific quantization settings.
The user shares benchmark results showing vLLM's significantly faster prefill performance compared to llama.cpp and other engines, and questions the technical reasons behind this speed difference.
Vellium v1.1.0 introduces live voice chat with local STT/TTS using Whisper and TeraTTSv2, and simplifies llama.cpp setup for easier integration with local AI models.
A user demonstrated that the Qwen 3.8 27b AI model can generate a fully self-contained Super Mario clone game in a single prompt using LLAMA.cpp, completing the task in 117 minutes on a Windows PC and Macbook M5 Air.
MTP has been released for the Qwen3.8-Flash-Next-GGUF model, providing detailed instructions on integration with inference tools like llama.cpp, vLLM, and Ollama for deployment.
A user is considering upgrading from a 5060ti 16GB to either 1x32GB V100 or 2x16GB V100 for better performance with the Qwen 3.8 model using llama.cpp, and asking for other options in a similar price range.
The article details custom optimizations for running the Qwen3.8-Flash-Next AI model on Mac M1 Max hardware, including SSD streaming, custom quantizations, and a sparse attention mechanism to improve performance.
This article reports on running the Qwen3.8-Flash-Next model on a MacBook Pro M5 Max, benchmarking speed versus context depth over 100 turns, with insights into performance and issues like role confusion at long contexts.
ShimQuant enables running Nemotron-3.5-Lightning on 16 GB GPUs with a 11.77 GiB quantized file, providing a usable option below previous 18 GiB limits.
A call for contributions to open pull requests in llama.cpp aimed at optimizing CPU, RAM, disk, and hybrid inference for faster performance, targeting completion by year-end.
An adaptive KV cache streaming fork of llama.cpp enables running the Qwen 3.8 27B model with 262K context on a 16GB RTX 5070 Ti GPU, achieving ~25 tok/s by efficiently managing memory between RAM and VRAM.
This article presents official 1-bit quantization builds for the Hy4 preview model, offering GGUF files with reduced sizes and instructions for running on patched llama.cpp.
Benchmark results for running Qwen3.8-Flash-Next with MTP on Strix Halo hardware using the Vulkan backend in llama.cpp, including performance metrics and observations on output quality.