llama-cpp

A local experiment comparing Qwen 35B-A3B MoE and Qwen 27B dense on coding-maintenance tasks, finding the MoE model ~3.9× faster with a smaller quality gap than expected.

A user shares their llama.cpp server configuration for running Qwen 3.6 27B on an RTX 5090, achieving 80-100 t/s, and asks the community for alternative settings and tips.

A llama.cpp PR proposes switching quantized-KV decode from the VEC to TILE SYCL kernel on Intel Battlemage, reportedly delivering up to ~169% faster decode at 118K context. The PR is open with caveats and limited independent verification.

A llama.cpp pull request adds an x86 VNNI implementation for Q2_0 × Q8_0 dot products, delivering 3.0–3.6x CPU-only speedups on Bonsai models with bit-for-bit kernel correctness and 99.2% token agreement.

A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.

Qwen3-TTS voice cloning has been merged into mainline llama.cpp, enabling local text-to-speech with voice cloning from short reference audio via the llama-tts binary, supporting multiple languages. Limitations remain, including only the Base model and no server endpoint yet.

Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.

A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.

DeepSeek-V4-Flash-0731 is released in GGUF format at 2.88 bits per weight (95 GiB), making the 284B-parameter MoE runnable on 128 GB machines. Includes a detailed operating guide and a required TurboQuant llama.cpp fork.

A GitHub patch allows running DeepSeek V4 Flash in LM Studio on 128GB Macs by sideloading antirez's llama.cpp fork, working around struct-layout drift, decoding splits, and code-signing issues.

llama.cpp has added support for MTP/DSpark on DeepSeek V4 Flash, enabling local inference with the popular C/C++ LLM runtime.

A PSA warning DeepSeek-V4-Flash-0731 users that system role messages are hoisted to the top, ruining prompt caching and proximity, and advising the use of latest_reminder instead.

This Hugging Face model card presents LiquidAI's LFM2.5-2.6B model in GGUF quantized format, with instructions for running it locally via llama.cpp, vLLM, Ollama, and other tools.

Unsloth releases an IQ3 GGUF quantization of DeepSeek-V4-Flash-0731, enabling local inference via llama.cpp, Ollama, LM Studio, and other tools.

Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.

Benchmark results for MindControl, a sampler-level guided reasoning budget for llama.cpp, show multi-stage signaling reduces token consumption without accuracy loss, and sometimes improves it on easier tasks.

A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.

A detailed comparison of three local AI runtimes—Ollama, LM Studio, and llama.cpp—helping practitioners choose the right one for their workflow based on interface, API compatibility, quantization control, and more.