Tag
Benchmarks of Gemma 4 and Qwen 3.6 MoE models on AMD 6800H iGPU using llama.cpp Vulkan backend show that MoE models and lower-bit quantization (Q4_0) provide the best performance, while Q8_0 is too slow for large models.
Kimi K3 text-only model is now supported in llama.cpp, enabling local inference of this open-source LLM using the C++ inference engine.
The article evaluates how different quantizations of the Qwen3.6-27B model affect output quality using KL divergence and top-1 token accuracy, as well as visual examples like SVG drawings.
A developer built llmux, an open-source tool to simplify managing model profiles across different engines like vLLM and llama.cpp, allowing one-click switching with Docker and NVIDIA GPU support.
A significant update to llama.cpp requires all previously generated GGUF files to be regenerated, indicating a major breaking change to the model format.
llama.cpp now fully supports the Model Context Protocol (MCP) for all protocols, enabling agentic chat and tool integration directly in its WebUI without external dependencies.
Slipstream streams MoE expert weights from SSD instead of RAM, enabling large coding models (35B–480B) on 36 GB MacBooks. Benchmarks show ~13–19 tok/s for 35B models and ~2.8 tok/s for 118B, with honest reporting of failed approaches.
Introduces a fully local AI girlfriend project made by a Bilibili developer, integrating four models: Silero VAD, Whisper, llama.cpp, and Qwen3-TTS, all packed into 15G VRAM with hot-swapping capability.
An analysis of native MTP speculative decoding in llama.cpp shows significant speedups (1.4x-2.2x) for dense models like Qwen3.6-27B, but underwhelming results on MoE architectures, where gains are minimal due to already low per-step overhead.
CachyLLama is a fork of llama.cpp that adds a persistent SSD-backed KV cache and multi-tier caching, dramatically reducing prompt reprocessing time for long local-agent sessions on slower hardware.
Tested the updated Gemma 4 locally using llama.cpp on an M5 Pro, achieving 60 tokens/s for coding tasks with OpenCode; good for backend but poor for UI/UX.
The article reports that infinite thinking loops in Laguna S 2.1 AI model are likely caused by quantization artifacts. Switching to an MoE-aware APEX quant (e.g., Myric/Laguna-S-2.1-APEX-GGUF) and using default sampling settings (temp 0.7, top_p 0.95, top_k 20) resolved the looping for most cases. Additionally, framing prompts around tool calls can prevent overthinking.
Apple M5 silicon supports INT8 activations for matrix multiplication, but inference backends like MLX and Llama.cpp currently use 16-bit; custom w8a8 kernels achieve up to 1.4x speedup on Gemma4 prefill tasks.
Laguna S-2.1 model has been updated with a fix for yarn_attn_factor (corrected to 1.0) and an improved chat template that fixes broken thinking, preserves thinking, and enables tool calling. Users are advised to use the updated GGUF from the official repo.
Describes running the GLM-5.2 model with 4-bit quantization at 12.2 tokens per second on a cluster of 16 AMD MI50 GPUs using llama.cpp's RPC, achieving coherent long-form generation at 10.7k context.
This pull request adds support for Laguna XS.2 & M.1 hardware in llama.cpp, expanding compatibility.
A detailed strategy using VRAM as disk cache with unified memory and llama.cpp settings achieves 340 pp/s and 9.6 tg/s for Kimi K2.7 on a single DGX Spark.
Pi 0.81.0 adds support for llama.cpp, enabling local LLM inference within the Pi coding agent harness.
A detailed guide on running the 27B Bonsai model on hardware with only 8GB VRAM using a 1-bit quantized version and the PrismML fork of llama.cpp, including exact server commands and configuration.
A new GStreamer element integrates llama.cpp to enable local LLM-based text translation and transformation in multimedia pipelines, allowing real-time subtitle generation and rephrasing.