Tag
Recent llama.cpp commits broke preserve_thinking behavior for older DeepSeek V4 gguf chat templates, causing issues in coding agent contexts. The fix is to override the gguf template with a new one using --chat-template-file.
Kimi K3 text-only model is now supported in llama.cpp, enabling local inference of this open-source LLM using the C++ inference engine.
The article details the current support status for the Ling-3.0-flash model weights across inference engines: SGLang commits to day-0 integration, vLLM awaits open weights, and llama.ccp lacks conversion for the Bailing MoE variant. It notes that the release pattern involves a free API window followed by open-sourcing, as seen with Ling-2.6-flash.
The article evaluates how different quantizations of the Qwen3.6-27B model affect output quality using KL divergence and top-1 token accuracy, as well as visual examples like SVG drawings.
A quick evaluation of coding performance across four GGUF variants of the Qwen 3.6-35B model.
A significant update to llama.cpp requires all previously generated GGUF files to be regenerated, indicating a major breaking change to the model format.
Release 0.4 of audio.cpp adds C++/GGML inference for Higgs Audio v3 TTS 4B (10x real-time) and Fish Audio S2 Pro, with full GGUF loading and Q8 speed/VRAM gains.
The article reports that infinite thinking loops in Laguna S 2.1 AI model are likely caused by quantization artifacts. Switching to an MoE-aware APEX quant (e.g., Myric/Laguna-S-2.1-APEX-GGUF) and using default sampling settings (temp 0.7, top_p 0.95, top_k 20) resolved the looping for most cases. Additionally, framing prompts around tool calls can prevent overthinking.
Laguna S-2.1 model has been updated with a fix for yarn_attn_factor (corrected to 1.0) and an improved chat template that fixes broken thinking, preserves thinking, and enables tool calling. Users are advised to use the updated GGUF from the official repo.
Laguna-S-2.1, a quantized AI model, runs on a 6-year-old gaming PC with an RTX 3080, achieving 10 t/s decode and using 8.3 GB VRAM and 52.2 GB host RAM.
Unsloth released GGUF quantizations of the Laguna S 2.1 Mixture-of-Experts model, a 118B parameter coding model with 8B active parameters, 1M context window, and agentic capabilities. The quantized versions enable efficient local deployment.
A curated list of recommended AI models (e.g., Qwen3.5, Gemma 4, Mistral Medium) for systems with up to 256GB unified RAM, covering creative, coding, agentic, and general chat use cases.
QuixiAI releases embeddinggemma.c, a fast cross-platform embedding inference engine written in C, supporting multiple backends (CPU, Metal, CUDA, ROCm, SYCL) and Matryoshka embeddings with a standard HTTP API.
Qwythos-9B is a 9B-parameter local LLM with 1M-token context and native tool calling, available in GGUF format for Ollama, llama.cpp, and LM Studio. It significantly outperforms its base model Qwen3.5-9B on key benchmarks.
A community fine-tuned and merged version of Qwen 3.5 9B that claims to exceed critical benchmarks of larger models, available in GGUF format with multi-token prediction support.
DavidAU releases Qwen3.6-27B-Fable-Fusion-711, a multi-stage fine-tune of Qwen 3.6 27B that claims to exceed 700 ARC-C benchmark, surpassing base models and matching closed-source models, available in GGUF format for consumer hardware.
A quantized GGUF version of the Kimi K3 model is now available, optimized for running on RTX 3090 GPUs.
Thinking Machines' Inkling, a 975B parameter model running locally on a Mac Studio via llama.cpp, listened to a doctor's visit audio and accurately diagnosed heart failure from subtle cues in small talk, demonstrating advanced medical reasoning without leaving the machine.
A modified GGUF version of Bonsai-27b with refusals removed has been released on Hugging Face.
A compact, importance-calibrated ROCmFP4 quantization of Qwen 3.5 122B model for high-memory AMD systems, achieving improved quality (14% lower KLD) and performance (28.45 tok/s). Requires ROCmFPX runtime; not compatible with stock llama.cpp.