Got an old slow low vram GPU laying around? Might be worth it to use for Just Vision mmproj llama.cpp
Summary
The article suggests using an old GPU to offload the mmproj component in llama.cpp, which improves speed for multimodal processing without affecting inference, as an alternative to the slow --no-mmproj-offload option.
Similar Articles
llama.cpp - how to free up even more space on your GPU
A thread sharing practical tips for freeing up GPU memory in llama.cpp, such as offloading mmproj to CPU and adjusting KV cache types, while discussing parameters like --cache-type-k/v and --spec-draft-n-max.
A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM
A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.
Running Qwen 3.8 next on 16vram+32ram - A useful/fun post for the gpu poors
A Reddit user shares how they successfully ran the Qwen 3.8 Next MoE model on a system with 16GB VRAM and 32GB RAM using aggressive quantization and specific llama.cpp settings, achieving usable performance for large models on limited hardware.
@analogalok: my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 …
User runs Gemma 4 31B dense model on 8GB VRAM gaming laptop at ~3 tokens/sec using llama.cpp with MTP speculative decoding, demonstrating feasibility of running a 31B dense model on consumer hardware and proposing agentic workflows where a fast MoE model routes to this slower dense model for hard tasks.
Experts first llama.cpp
A developer created an experimental fork of llama.cpp that offloads only used experts instead of entire layers to VRAM, achieving speed improvements for MoE models on GPUs with limited VRAM like the RTX 2060 12GB. The author is asking for testers to validate performance on other Nvidia GPUs.