llama-cpp

A pull request to llama.cpp adds Nemotron 3.5 Transcribe support in the mtmd (multimodal) module, extending the library's audio transcription capabilities.

An experiment analyzes why Qwen3.5-0.8B gets stuck repeating 'Wait' by using an interpretability lens on the 'Wait' token, showing doubt-related features dominating near stopping points, and applies a tiny weight edit (0.016% of weights) that converts 'Wrong/Nope/Hmm' into 'Verification/Completion/Result' while sharply improving MATH-500 performance; code and data are released.

A pull request to llama.cpp adds support for the Clover GPU/accelerator backend, extending the library's hardware compatibility for LLM inference.

Mistral announces 'Le Chonk' (Mistral Large 4), a massive 1T-parameter model, highlighting llama.cpp support and community-friendly features on Hugging Face.

Basalt (a fork of Strata) lands new PRs setting a new benchmark record with IQ3_S Qwen3.8 Flash Next on a single RTX 5080 with 64GB DDR5, reaching 5500 tok/s prompt processing and up to 275 tok/s peak generation during a live opencode coding session.

erusev 与 llama.cpp 团队合作开发了 Llama，一款可在 Mac 菜单栏运行开源模型的小型应用，并发布了 35 秒演示视频。

Liquid AI 发布 LFM3-1.7B 的 GGUF 版本，支持在 CPU/GPU 上本地推理，兼容 llama.cpp、Ollama、LM Studio 等工具，并提供经过量化感知蒸馏（QAD）的 Q4_0 检查点以恢复量化损失的精度。

A user extracted the pure DFlash draft part from Ling-3.0-tiny-UBTree (dropping the ubtree_selector tensors) so it runs on stock mainline llama.cpp, reporting +8% speedup for prose and +61% for Python code generation, with the converted GGUF and conversion procedure published on Hugging Face.

A llama.cpp pull request discussion about improving strata decode performance on an AMD R9700 GPU, referencing the ggml/llama.cpp C/C++ LLM inference project.

Opposable 推出可在 Android 手机本地运行的 GUI 智能体：使用 4-bit 量化的 GUI-Owl 1.5 2B 模型（基于 Qwen3-VL，Alibaba Mobile-Agent 团队出品），通过 llama.cpp 在设备上直接读屏并操作任意应用，无需服务器。通过裁剪状态栏、丢弃平坦背景图像 token 等优化，Mac M2 CPU 上单步耗时从 10.6 秒降至 7.3 秒，且在 36 屏测试集上精度不降。

AbliteratedOutTheBox 是一个 llama.cpp 分支，可在推理时对任意加载的模型实时执行 abliteration（去除对齐/拒绝机制），无需微调、无需额外文件，也不占用额外显存。

A hands-on benchmark of 10 AI coding agents run headless against a local Qwen3.5-4B model across five coding tasks, finding that Pi, goose, and Crush finished fastest while Codex CLI failed entirely due to API incompatibility with local setups.

A user reports GPU hangs with NVIDIA Xid 8 on an RTX 5090 Laptop GPU under Linux when running local LLMs, pointing to a llama.cpp issue where disabling CUDA graphs (GGML_CUDA_DISABLE_GRAPHS=1) serves as a performance-sapping workaround with no fix yet.

ISTA-DASLab 发布/更新了 Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF 模型仓库，提供可在 llama.cpp、vLLM、Ollama、LM Studio 等本地环境中运行的量化编码模型及部署说明。

A user loaded the 560B-parameter Ling 3.1 Flash (25B active) MoE model across four DGX Sparks using llama.cpp RPC with Atomic Chat's Q4 GGUF build, since no official FP4 vLLM build was available. Performance comparisons against Ling 3.0 Flash 124B are still pending.

A community guide on running Qwen 3.8 Flash Next with a full FP8 n-gram table (PLE) instead of a heavily quantized one, using ple_fp8_pack.py to pack safetensors into a GGUF sidecar, yielding noticeably better output quality with no speed penalty.

A user shares benchmarks showing that raising the --adapt-every parameter from 1 to 6 in Strata boosts token generation speed by 9-55% across multiple Qwen 3.8 Flash Next prompts, with the largest gains on long-context code descriptions.

A beginner-friendly Reddit guide sharing 10 things you can build with the open-source Qwen 3.6 27B model via llama.cpp, including custom apps, chat frontends, ReAct loops, MCP tools, sub-agents, and pipeline runners.

一位 Reddit 用户发布了 PS5LM 的分支，使 llama.cpp 能在越狱 PS5 上原生运行并利用 GPU 和更多内存，从而在本地运行 Qwen3.8-27B 等大型模型，速度超过 100 tokens/s。

腾讯混元团队成员公开了 Hy4 Preview（770B 参数 MoE）的极致量化方案，通过 STQ1_0 编码与混合精度分配将约 1.5TB 权重压缩至 214 GiB，并在 llama.cpp 中实现接近 IQ1_M 的推理速度。