Tag
Benchmark results for Inkling-2-Small GGUF quants (Unsloth UD-Q4_K_XL and Bartowski Q4_K_M) running on a Mac Studio M3 Ultra via llama.cpp, with Unsloth reaching ~19 tok/s decode and Bartowski ~27 tok/s decode at 1000-token prompt depth.
Manning Books is offering a free ebook, 'Local AI and Small Language Models', featuring chapters on small language models, CPU inference with GGUF and llama.cpp, quantization, LoRA/DoRA specialization, and building private local AI apps with Ollama's Python library.
Mistral Large 4 (codenamed Le Chonk) is released on Hugging Face Hub — a multimodal, multilingual Mixture-of-Experts model with 1T total/49B active parameters and a 1M-token context window, featuring day-0 support in transformers, vLLM, and llama.cpp along with quantized GGUF versions.
Liquid AI 发布 LFM3-Omni-2B 的 GGUF 版本,支持在 CPU/GPU 上本地推理。该模型可理解文本、图像、视频和音频,并以文本和语音输出,专为设备端使用打造。
A Hugging Face release packaging HauhauCS's uncensored Qwen3.8-27B weights with DavidAU's Turbo-Brilliance 'AI Construct' chat template modes into a single GGUF, enabling role-based uncensored responses in llama.cpp, LM Studio, and Ollama.
一位用户分享了对新发布的 LiquidAI LFM2.5-24B-A2B 模型的实测体验,称其在 llama.cpp + ROCm 上可达约 95 tps,Q8 量化占 30GB 显存并支持 128k 上下文,在 Pi Agent 工作流中表现稳定可靠。
作者更新了将稠密模型转换为混合专家(MoE)模型的项目,发布了 Qwen2.5-1.5B-MoE-A0.9B 及 0.5B/1.5B 的 GGUF 版本。结果显示 GPU 和 CPU 推理速度明显提升(约 32-41%),但困惑度上升约 40%,作者称其为小模型上的概念验证。
A llama.cpp pull request (#29184) by am17an fuses shared experts into the MMVQ (matrix-matrix vector quantization) CUDA kernel, optimizing inference for mixture-of-experts models on GPUs.
A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.
The author benchmarks a local coding agent setup—Qwen3.8-27B Q4_K_M on a 24GB RTX 3090 via llama.cpp with OpenCode—reporting throughput figures, four bounded Python coding tasks (three passing checks), and a reasoning-budget failure where the model exhausted its 8K output allowance without producing a patch.
The author raises concerns about the common practice of creating hard forks of llamacpp optimized for specific architectures without upstream contributions, questioning the reasons and noting a disconnect from traditional open-source norms.
A user recommends a GitHub project for faster inference of Qwen 3.8 Flash Next on Strix Halo, showing benchmark speeds that significantly outperform llama.cpp.
The user switched from Windows to Linux and replaced llamacpp with vllm, achieving a 30-50% speed improvement in AI inference.
Vision support for the Minimax-M3 model has been merged into the llama.cpp project, enabling multimodal inference for this model locally.
Laguna-S-2.1, a quantized AI model, runs on a 6-year-old gaming PC with an RTX 3080, achieving 10 t/s decode and using 8.3 GB VRAM and 52.2 GB host RAM.
A new PR for llama.cpp boosts prompt processing on ROCm by ~15% and fixes a bug making Q2_K quantization 28x faster.
Comprehensive benchmarks of llama.cpp's speculative decoding methods on Qwen 3.6 27B show n-gram stacking on DFlash achieves up to 6x speedup on iterative coding tasks, with ngram-mod providing most of the gain and zero VRAM cost.
Benchmark results for Qwen3.6-35B-A3B-UD-Q8_K_XL on DGX Spark using llama.cpp script by Mia, showing fast token generation times across various context lengths.
A user asks about Ubuntu's compatibility for local AI tools like vLLM, llama.cpp, and ComfyUI when switching to Linux.
A user reports that the Qwen3.6-27B model performs better and more reliably with llama.cpp than with vLLM, citing tool call errors and 'lobotomized' behavior in vLLM despite extensive configuration.