Tag
A draft model for Qwen 3.8 27B is highly effective on 16 GB GPUs, achieving around 60 tokens per second on an RX 9070 XT and offering better VRAM efficiency than built-in MTP.
The article suggests using an old GPU to offload the mmproj component in llama.cpp, which improves speed for multimodal processing without affecting inference, as an alternative to the slow --no-mmproj-offload option.
The author shares their experience running a 30B parameter model with EXL3 quantization on a 12GB VRAM GPU, achieving efficient performance and speed for coding and agent tasks.
Release of a quantized version of the Qwen3.8-27B-DFlash2 model optimized for NVIDIA Blackwell GPUs, achieving similar decode speed and output quality with 2.6x less VRAM usage and extended context length.
The article presents a benchmark tool for evaluating local LLM configurations, focusing on VRAM usage, performance metrics, and hardware optimization to assist developers in optimizing setups.
A discussion thread sharing configurations and tips for running AI models like Qwen3.8-27B on 16 GB VRAM Windows systems, focusing on memory optimization techniques.
A user shares their setup using Ling 3.0 Tiny as an auxiliary model for Hermes (Qwen 3.8 27B) to handle simple tasks like context compression and summarization, improving speed and efficiency without quality loss.
The article shares an optimal llama.cpp configuration for running the Qwen 3.8 27B model on 16GB VRAM with 73k context, demonstrating its performance in agentic coding workflows through a real-world software engineering project.
User seeks community advice on reducing VRAM usage and freeing OS RAM when serving DeepSeek-V4-Flash-0731 on two DGX Spark machines with vLLM, sharing detailed configuration and memory measurements.
BeeLlama.cpp v0.4.1 introduces KVarN, KV precision tail, and improved KV cache support, with benchmarks showing that using tail 1024 allows kvarn5 and q6_0 to match q8_0 precision while using less VRAM.
An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.
Unsloth improved Qwen3.6-27B Q6_K context length from 23K to 64K on a single 32GB card by accurately counting the draft model's KV cache in bytes instead of using a flat VRAM cushion.
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
A user shares optimized settings for running Qwen3.6 27B (Q8_0) on a dual GPU setup (RTX 4090 + RTX 3090) with llama.cpp, achieving 75-100 t/s and 1500 pp with 250k context.
A guide on optimizing VRAM usage on an AMD 7900XTX to run a 27B Qwen model with Q6K quantization and 131k context by compiling llama.cpp with OpenBLAS and CUDA_FA_ALL_QUANTS, and using kvcache quantization at q5_0/q4_0.
Continuous batching has been added to TRL for GRPO, improving speed and VRAM usage without needing vLLM. The tweet explains how it works and when to use it.
This thread recommends AI models optimized for different VRAM levels, highlighting VibeThinker-3B for its strong reasoning performance at 3B parameters, along with other models for coding and general use.
A thread sharing practical tips for freeing up GPU memory in llama.cpp, such as offloading mmproj to CPU and adjusting KV cache types, while discussing parameters like --cache-type-k/v and --spec-draft-n-max.
Testing shows that default pipeline parallelism in llama.cpp wastes VRAM with no speed benefit; compiling with GGML_SCHED_MAX_COPIES=1 saves significant VRAM while maintaining identical inference speed.
Alok demonstrates running Gemma 4 26B MoE on 8GB VRAM using Unsloth's QAT quant and the -cmoe flag in llama.cpp, achieving 20 tokens/sec with 250k context, marking a major milestone for budget local AI.