Tag
Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.
A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.
DeepSeek-V4-Flash-0731 is released in GGUF format at 2.88 bits per weight (95 GiB), making the 284B-parameter MoE runnable on 128 GB machines. Includes a detailed operating guide and a required TurboQuant llama.cpp fork.
A GitHub patch allows running DeepSeek V4 Flash in LM Studio on 128GB Macs by sideloading antirez's llama.cpp fork, working around struct-layout drift, decoding splits, and code-signing issues.
llama.cpp has added support for MTP/DSpark on DeepSeek V4 Flash, enabling local inference with the popular C/C++ LLM runtime.
A PSA warning DeepSeek-V4-Flash-0731 users that system role messages are hoisted to the top, ruining prompt caching and proximity, and advising the use of latest_reminder instead.
This Hugging Face model card presents LiquidAI's LFM2.5-2.6B model in GGUF quantized format, with instructions for running it locally via llama.cpp, vLLM, Ollama, and other tools.
Unsloth releases an IQ3 GGUF quantization of DeepSeek-V4-Flash-0731, enabling local inference via llama.cpp, Ollama, LM Studio, and other tools.
Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.
Benchmark results for MindControl, a sampler-level guided reasoning budget for llama.cpp, show multi-stage signaling reduces token consumption without accuracy loss, and sometimes improves it on easier tasks.
A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.
A detailed comparison of three local AI runtimes—Ollama, LM Studio, and llama.cpp—helping practitioners choose the right one for their workflow based on interface, API compatibility, quantization control, and more.
Recent versions of llama.cpp now automatically load MTP tensors for draft-mtp architectures, even if speculative decoding is not enabled, potentially increasing VRAM usage for users with bundled MTP blocks in their GGUF files.
The author presents a method using GBNF grammars in llama.cpp to constrain local AI model outputs, ensuring valid JSON. A compiler is built to convert tool schemas into grammar rules, with per-turn narrowing for relevant tools.
A developer built a GBNF grammar compiler in Rust for llama.cpp that enforces tool-calling JSON schemas, making small models (like 8B parameters) reliably call tools. The system narrows grammar to only matched tools per turn, improving reliability.
User successfully runs the Kimi-k3 model using llama.cpp on high-end hardware, achieving low tokens per second (0.41 prompt eval, 0.23 generation).
A user notices that the Q4_K_M quantized version of Laguna S 2.1 increased from 68GB to 96GB, likely due to using more FP16 layers, and discusses potential issues with quantization and context looping.
Recent llama.cpp commits broke preserve_thinking behavior for older DeepSeek V4 gguf chat templates, causing issues in coding agent contexts. The fix is to override the gguf template with a new one using --chat-template-file.
User reports improved tokens per second (tps) with ThinkingCap-Qwen3.6-27B compared to Qwen3.5-27B, with no quality loss, recommending it as a daily driver until the next Qwen release.
Benchmarks of Gemma 4 and Qwen 3.6 MoE models on AMD 6800H iGPU using llama.cpp Vulkan backend show that MoE models and lower-bit quantization (Q4_0) provide the best performance, while Q8_0 is too slow for large models.