Tag
This article tracks updates on various low-bit AI models, including Bonsai, BitCPM, and others, with performance improvements and compatibility updates for backends like llama.cpp.
Ling-3.0 (BailingMoE3) is now officially supported in llama.cpp, with benchmarks on Intel Arc B580 showing efficient local inference including 128K context in 12GB VRAM.
This article presents performance results and optimization tips for using llama.cpp with RPC to run AI models across a 5070 Ti and 1080 Ti GPUs over ethernet, emphasizing trade-offs between prefill and token generation speeds.
A user created a simple game with just two prompts using the Qwen 3.8 AI model in Q4 quantization on llama.cpp with dual RTX 3060 GPUs, demonstrating impressive AI-assisted coding despite RAM limitations.
The article details an experiment achieving 50 tokens per second inference with Qwen3.8-27B at 256K context on a 24GB GPU using Multi-Token Prediction and custom optimizations.
The article shares an optimal llama.cpp configuration for running the Qwen 3.8 27B model on 16GB VRAM with 73k context, demonstrating its performance in agentic coding workflows through a real-world software engineering project.
Koboldcpp v1.119 has been released as an easy-to-use AI text-generation software supporting GGML and GGUF models, with features including image generation, speech-to-text, and multimodal capabilities.
This is a quantized version of the Qwen3.8-27B AI model using IQ4_XS quantization, optimized for 16GB RAM systems, with instructions for local deployment using various tools like llama.cpp and Ollama.
A user demonstrates running the Qwen 3.8 27b model with llama-server to generate an HTML simulation of an aquarium panel failure, including detailed token usage and model parameters.
This is a GGUF conversion of the uncensored Qwen3.8-27B model, an abliterated version with safety alignment removed for research purposes, running on llama.cpp with various quantizations and vision support.
A showcase of a personal local AI inference build featuring Intel Arc B140 GPUs and custom hardware, running llama.cpp with SYCL back-end on Ubuntu.
An open-source Windows application for managing llama.cpp runtimes, providing a visual interface to run multiple AI models simultaneously with monitoring and control features.
This article describes the release of a quantized GGUF version of the Qwen3.8-27B AI model, optimized for efficient local inference on hardware with limited VRAM.
This article describes the unofficial quantization of the AI9Stars G9v3-39A5B MoE model to GGUF format and the creation of a llama.cpp fork to enable local inference with BF16 and Q4_K_M variants.
llama.cpp server build 10423 adds an experimental --tools-runtime option that lets the webui run tool shell commands inside rootless sandboxed containers (e.g., podman:alpine or docker:alpine), downloading and instantiating the image automatically.
A quantized GGUF release of the Qwen3.8-27B model with reduced refusal behavior, retaining multi-token prediction and offering various quantization options for use with llama.cpp.
The author fine-tuned Qwen2.5-Coder-1.5B on 125k natural-language/command pairs to generate shell commands, runs locally on a laptop CPU in ~1 second, and published the weights and code.
An experiment running the Qwen3.8-2.4T-A95B MoE model locally on dual consumer GPUs (RTX 5090 + 5060 Ti) with llama.cpp, achieving ~0.8 tok/s with MTP speculative decoding enabled.
User reports running Qwen 3.6 35B A3B-Q8_0 gguf on a Radeon 7600 with llama.cpp and ROCm, achieving 21 tokens per second after VRAM overclocking, with a note about a display-related performance bug.
Unsloth announces that Qwen3.8 can now run locally, shrinking the 2.4T-parameter model from 4.9TB to 397GB via Dynamic 1-bit quantization, with a guide and GGUF release.