Llama.cpp's auto fit works much better than I expected
Summary
Llama.cpp's new --fit flag enables running models larger than VRAM with surprisingly high token/s, breaking the old VRAM-only limitation.
Similar Articles
llama: use f16 mask for FA to save VRAM by am17an · Pull Request #23764 · ggml-org/llama.cpp
This pull request for the llama.cpp inference engine implements using f16 mask for Flash Attention to reduce VRAM usage.
@Sumanth_077: Stop guessing which models fit in your VRAM! llmfit is a CLI tool that auto-detects your hardware and ranks 206 models …
llmfit is an open-source CLI tool that detects your hardware and ranks over 200 LLMs by which ones will actually run on your system, automatically choosing the best quantization that fits.
After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
The article shares an optimal llama.cpp configuration for running the Qwen 3.8 27B model on 16GB VRAM with 73k context, demonstrating its performance in agentic coding workflows through a real-world software engineering project.
First time running local models
A user shares their experience running the ik_llama model locally, praising its speed despite having only 12GB of VRAM.
A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM
A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.