llama-cpp

A new GStreamer element integrates llama.cpp to enable local LLM-based text translation and transformation in multimedia pipelines, allowing real-time subtitle generation and rephrasing.

A personal review of using the Qwen 3.8 model for agentic coding, praising its ability to handle complex tasks like integrating llama.cpp with Godot while noting issues with looping.

Thinking Machines' Inkling, a 975B parameter model running locally on a Mac Studio via llama.cpp, listened to a doctor's visit audio and accurately diagnosed heart failure from subtle cues in small talk, demonstrating advanced medical reasoning without leaving the machine.

Demonstrates achieving 256k context on a 16GB RTX 4080 Super using Ternary Bonsai 27B Q2_0 model with llama.cpp, achieving up to 141 tok/s generation speed.

Tencent Hunyuan team releases quantized versions for the 295B-parameter Hy3 large model. The 1-bit version compresses weights to 85.5GB, enabling deployment on a single 96GB inference GPU with ~50% faster decoding. The open-source GGUF format is compatible with the llama.cpp ecosystem.

A user shares achieving 80 tok/s on a Qwen3.6 35B model with Q4_K_M quantization and 100k context on a single Nvidia P40 using TheTom's TurboQuant fork of llama.cpp, highlighting various optimizations.

Updates on Bonsai-27B and Ternary-Bonsai-27B models, detailing upstream merge status in llama.cpp across CPU, Metal, CUDA, Vulkan backends, and discussing model limitations and roadmap.

User tests PrismML's new Bonsai 27B model on Nvidia DGX Spark, reporting benchmark speeds and issues with llama.cpp build, while PrismML announces the model as the first 27B-class model to run on a phone.

A blog post describing an efficient method to pack ternary numbers into 8-bit bytes using SIMD-friendly unpacking, achieving 1.6 bits per trit, with applications in LLM weight quantization like BitNet b1.58.

An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.

Celebrating the 10000th release of llama.cpp, a tool for running LLMs locally.

A milestone release of llama.cpp, the open-source library for running large language models locally on consumer hardware, bringing improvements in performance or new features.

A tweet challenges the claim that Macs are useless for AI without CUDA, presenting benchmarks showing a Mac Mini achieves 56 t/s generation on a 30B model, arguing it's fine for local inference while DGX excels at training and prefill.

AngelSlim/Hy3-GGUF is a toolkit for quantizing and deploying Hy3 large language models on llama.cpp, featuring MTP self-speculative decoding and a thinking/tool-call parser for efficient inference.

A three-line fix corrects a precision bug in llama.cpp's CUDA code for Nvidia P100 GPUs (sm_60), dramatically improving output quality without performance loss. The patch extends an existing exemption to the P100, which has fast fp16 hardware that was incorrectly using lower precision.

An interactive Jacobian-lens visualizer and live steerer for GGUF models running on llama.cpp, enabling real-time model interpretability and control.

An analysis of PCIe transfer performance when running llama.cpp with dual GPUs using pipeline and tensor parallelism.

llama.cpp b9966 introduces a fix for the -sm tensor mode that caches regex patterns, eliminating 29 recompilations per tensor per token on the decode thread, resulting in significantly reduced CPU overhead.

A curated collection of LLM inference recipes and performance benchmarks for popular hardware configurations, including NVIDIA RTX Pro 6000 Blackwell, H100, AMD Strix Halo, and RTX 5090, with detailed batch size, quantization, and throughput data.

Demonstrates achieving 20GB VRAM and 448GB/s bandwidth for around $100 using two NVIDIA P102-100 cards, running a llama.cpp server with a Qwen model and supporting 3 concurrent users with large context.