First time running local models
Summary
A user shares their experience running the ik_llama model locally, praising its speed despite having only 12GB of VRAM.
Similar Articles
Local LLM CPU users... How long is it taking you to do anything?
A discussion about the performance of running large language models locally on CPU, especially with large context sizes, and the challenges of VRAM constraints.
Local agent workspace on a 4GB laptop GPU (RTX 3050 Ti): the tok/s and where a small model struggles once it has to call tools, build artifacts, and RAG
The author benchmarks local Qwen models of various sizes on a 4GB RTX 3050 Ti laptop GPU within the Bike4Mind workspace, finding the 2B model at Q4_K_M quantization is the sweet spot for fitting in VRAM, achieving 96 tok/s. Smaller models struggle with tool selection, artifact generation requiring multiple models, and RAG embeddings causing model swap overhead.
@analogalok: my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 …
User runs Gemma 4 31B dense model on 8GB VRAM gaming laptop at ~3 tokens/sec using llama.cpp with MTP speculative decoding, demonstrating feasibility of running a 31B dense model on consumer hardware and proposing agentic workflows where a fast MoE model routes to this slower dense model for hard tasks.
@theemozilla: We're working on making the local model experience better in Hermes, what are the best local models at each weight clas…
The user asks for recommendations on the best local AI models for different VRAM classes (8-16GB, 24-32GB, 128GB), mentioning Gemma4, Qwen, and DeepSeek variants, as they work on improving local model support in Hermes.
Llama.cpp's auto fit works much better than I expected
Llama.cpp's new --fit flag enables running models larger than VRAM with surprisingly high token/s, breaking the old VRAM-only limitation.