Tag
Hugging Face announces native support for GGUF files in the transformers library, allowing easier use of quantized models with PyTorch tooling and performance comparable to llama.cpp.
Quantized versions of the K2-Horizon AI model series are now available for download on Hugging Face, supporting various sizes from 0.9B to 36B parameters. However, the llama.cpp support is still in progress, requiring a fork for use.
The article compares GSQ and ByteShape quantizations of the Qwen 3.8 27B model on an RTX 3060, revealing that ByteShape's quant underperformed despite claims of high similarity to the original model.
Dynamic Quantiser is an open-source tool that enables users to create custom and high-quality dynamic quantizations of LLM GGUF files, minimizing cosine deviation for better accuracy compared to standard quants.
This article describes a fork of llama.cpp called focus-llama that implements Declarative Attention from a recent paper, allowing models to declare needed context chunks during inference to optimize KV cache usage and reduce decode time.
A user upgraded from a single Nvidia GTX-1080Ti to a dual GPU setup with an AMD Radeon MI50, achieving significant speed improvements in running local large language models, with the added GPU costing less than $140.
The author compared the performance of Qwen3.8 27B IQ3_XXS and Bonsai Ternary PQ2 on limited VRAM, finding that Qwen is faster and uses fewer tokens, while Bonsai has a smaller file size but longer generation times.
The user has implemented nvfp4 KV cache support for the Qwen3.8 model on a heterogeneous GPU setup using custom CUDA kernels and quantization to optimize performance.
The author optimized the Qwen3.8-Flash-Next model to run on a 64GB Mac using expert streaming and other techniques, achieving ~27 tok/s by publishing a checkpoint and a llama.cpp fork.
Ternary-Bonsai-2-27B is a 27B parameter AI model compressed into 2-bit ternary format, running on llama.cpp with CUDA and Metal support, facilitating lightweight on-device AI deployment.
This article presents a multi-hour investigation into speeding up local inference for Qwen MoE models using llama.cpp, including experimental results, patches, benchmarks, and reproduction guides, with some workloads showing substantial gains.
An unofficial fork of llama.cpp introduces a persistent expert pool for MoE models, optimized to reduce expert re-copies over PCIe on 16GB AMD gfx906 GPUs, thereby improving decode throughput for large context lengths.
ByteShape releases full ShapeLearn quantized versions of the Qwen 3.8 27B model in GGUF format, with benchmarking showing improvements in quality-speed frontier and support for speculative decoding.
A GitHub fork of llama.cpp optimized for dual AMD 7900 XTX GPUs, significantly improving decode speed for the Qwen 3.8 Q8 model to 82 tokens per second.
Bonsai 2 27B is a compressed AI model that achieves near-lossless performance in a 9x smaller footprint, with setup instructions provided for use with Prism's llama.cpp fork.
A talk about llama.cpp speculative decoding methods (MTP, dflash, dspark) at the dotAI conference will have its replay available soon.
Release of Ternary-Bonsai-2-27B-gguf, a 27B-class language model using ternary weights for extreme compression (5.9 GB) while retaining 98.2% of FP16 intelligence, optimized for efficient inference on laptops and single GPUs.
This pull request enables CUDA graph for MTP draft in llama.cpp to improve performance of LLM inference on GPUs.
This post provides a guide for setting up agentic systems, emphasizing the importance of foundational tools like Linux, Tailscale, and specific engines before selecting an AI model. It outlines eight key components and habits for effective local and cloud deployment.
The article presents a custom fork of llama.cpp optimized for NVIDIA Ampere architecture GPUs like the 3090, achieving over 90 tokens per second for 27B parameter models with large context windows, making local inference faster than API speeds.