llama-cpp

Unsloth announces that Qwen3.8 can now run locally, shrinking the 2.4T-parameter model from 4.9TB to 397GB via Dynamic 1-bit quantization, with a guide and GGUF release.

User tested Nemotron 3.5 Lightning locally with llama.cpp and quants, finding good speed and agentic tool-calling but below-expectation coding output for its size.

The author quantizes DeepSeek V4 0731, fixing FP8 downconversion issues that skew baselines, and benchmarks 38 quant files on 8× RTX 5090 to show GPU-dependent results and file-size-based comparisons.

A hobbyist describes building a low-power llama.cpp server using an Intel N100 motherboard and a refurbished RTX 5060 Ti, sharing performance numbers, power consumption, and model choices.

A user reports that in Llama-CPP with parallel sub-agents, decode performance is great but a single agent's prefill (e.g., processing a web search) stalls all other agents, and asks for tuning suggestions.

A user shares local testing of Muse Glimmer (Q4 quant via Unsloth) on llama.cpp with OpenCode, noting it performs below Qwen3.6 27B but had reliable tool calls.

Benchmarks unsloth's Muse Glimmer 30B on an RTX 5090 with speculative decoding, achieving up to 253 t/s using a DFlash draft model and a GPU-based argmax PR, though the PR is still a draft.

DiffusionGemma technical report released on arXiv, with ongoing work on llama.cpp pull requests to enable faster local inference on limited VRAM.

User reports that Muse Glimmer, a 30B model, fits on a single RTX 3090 with full 256k context using Q4_K_XL quantization and DFlash, achieving 64-124 tok/s and perfect long-context retrieval, unlike comparable models.

A user reports successfully running a 1M-token context on a single RTX 3090 using a Qwen-based 35B A3B model (17GB VRAM) with KVarN 4-bit KV-cache quantization in a BeeLlama.cpp fork, extracting 7 needles from different parts of the text.

Meta Superintelligence Lab releases Muse Glimmer 30B in GGUF quantized formats for local inference, a distilled causal language model with a perception encoder for autonomous agentic tasks, tool use, multimodal input, and failure recovery on consumer hardware.

Presents VectraYX-Vision-1B, a sub-2B Spanish/LATAM cybersecurity vision-language model coupling a frozen SigLIP encoder with a Spanish decoder via an MLP, yet reports near-zero visual grounding despite functional pipelines, with open-source weights and remediation plans.

A local experiment comparing Qwen 35B-A3B MoE and Qwen 27B dense on coding-maintenance tasks, finding the MoE model ~3.9× faster with a smaller quality gap than expected.

A user shares their llama.cpp server configuration for running Qwen 3.6 27B on an RTX 5090, achieving 80-100 t/s, and asks the community for alternative settings and tips.

A llama.cpp PR proposes switching quantized-KV decode from the VEC to TILE SYCL kernel on Intel Battlemage, reportedly delivering up to ~169% faster decode at 118K context. The PR is open with caveats and limited independent verification.

A llama.cpp pull request adds an x86 VNNI implementation for Q2_0 × Q8_0 dot products, delivering 3.0–3.6x CPU-only speedups on Bonsai models with bit-for-bit kernel correctness and 99.2% token agreement.

A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.

Qwen3-TTS voice cloning has been merged into mainline llama.cpp, enabling local text-to-speech with voice cloning from short reference audio via the llama-tts binary, supporting multiple languages. Limitations remain, including only the Base model and no server endpoint yet.