llamacpp

Tag

Cards List
#llamacpp

Inkling-2-Small Unsloth Q4 on M3 Ultra: 19 tok/s decode

Reddit r/LocalLLaMA ↗ · 9h ago

Benchmark results for Inkling-2-Small GGUF quants (Unsloth UD-Q4_K_XL and Bartowski Q4_K_M) running on a Mac Studio M3 Ultra via llama.cpp, with Unsloth reaching ~19 tok/s decode and Bartowski ~27 tok/s decode at 1000-token prompt depth.

0 favorites 0 likes
#llamacpp

@ManningBooks: Want to run AI locally? Explore small language models, CPU inference, quantization, and private AI apps with our free e…

X AI KOLs Timeline ↗ · 2026-11-02 Cached

Manning Books is offering a free ebook, 'Local AI and Small Language Models', featuring chapters on small language models, CPU inference with GGUF and llama.cpp, quantization, LoRA/DoRA specialization, and building private local AI apps with Ollama's Python library.

0 favorites 0 likes
#llamacpp

@mervenoyann: hey, Le Chonk by @MistralAI is now on Hub! to celebrate, we ship day-0 support in transformers and llama.cpp, as well a…

X AI KOLs Following ↗ · 2026-10-29 Cached

Mistral Large 4 (codenamed Le Chonk) is released on Hugging Face Hub — a multimodal, multilingual Mixture-of-Experts model with 1T total/49B active parameters and a 1M-token context window, featuring day-0 support in transformers, vLLM, and llama.cpp along with quantized GGUF versions.

0 favorites 0 likes
#llamacpp

LiquidAI/LFM3-Omni-2B-GGUF

Reddit r/LocalLLaMA ↗ · 2026-10-27 Cached

Liquid AI 发布 LFM3-Omni-2B 的 GGUF 版本,支持在 CPU/GPU 上本地推理。该模型可理解文本、图像、视频和音频,并以文本和语音输出,专为设备端使用打造。

0 favorites 0 likes
#llamacpp

Qwen3.8-27B-Defiance-Uncensored-MTP-GGUF

Reddit r/LocalLLaMA ↗ · 2026-10-17 Cached

A Hugging Face release packaging HauhauCS's uncensored Qwen3.8-27B weights with DavidAU's Turbo-Brilliance 'AI Construct' chat template modes into a single GGUF, enabling role-based uncensored responses in llama.cpp, LM Studio, and Ollama.

0 favorites 0 likes
#llamacpp

Very impressed with LFM2.5-24-A2B in Pi Agent

Reddit r/LocalLLaMA ↗ · 2026-10-15

一位用户分享了对新发布的 LiquidAI LFM2.5-24B-A2B 模型的实测体验,称其在 llama.cpp + ROCm 上可达约 95 tps,Q8 量化占 30GB 显存并支持 128k 上下文,在 Pi Agent 工作流中表现稳定可靠。

0 favorites 0 likes
#llamacpp

Update: Converting dense models into Mixture-of-Experts

Reddit r/LocalLLaMA ↗ · 2026-10-13

作者更新了将稠密模型转换为混合专家(MoE)模型的项目,发布了 Qwen2.5-1.5B-MoE-A0.9B 及 0.5B/1.5B 的 GGUF 版本。结果显示 GPU 和 CPU 推理速度明显提升(约 32-41%),但困惑度上升约 40%,作者称其为小模型上的概念验证。

0 favorites 0 likes
#llamacpp

CUDA: fuse shared experts into MMVQ by am17an · Pull Request #29184 · ggml-org/llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-10-02 Cached

A llama.cpp pull request (#29184) by am17an fuses shared experts into the MMVQ (matrix-matrix vector quantization) CUDA kernel, optimizing inference for mixture-of-experts models on GPUs.

0 favorites 0 likes
#llamacpp

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

Reddit r/LocalLLaMA ↗ · 2026-10-02

A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.

0 favorites 0 likes
#llamacpp

Qwen3.8-27B Q4_K_M on one RTX 3090 + OpenCode: throughput, four coding tasks, and a reasoning-budget failure

Reddit r/LocalLLaMA ↗ · 2026-10-01

The author benchmarks a local coding agent setup—Qwen3.8-27B Q4_K_M on a 24GB RTX 3090 via llama.cpp with OpenCode—reporting throughput figures, four bounded Python coding tasks (three passing checks), and a reasoning-budget failure where the model exhausted its 8K output allowance without producing a patch.

0 favorites 0 likes
#llamacpp

I am concerned about all these disparate hard forks that target specific architectures instead of opening a PR against upstream

Reddit r/LocalLLaMA ↗ · 2026-09-28

The author raises concerns about the common practice of creating hard forks of llamacpp optimized for specific architectures without upstream contributions, questioning the reasons and noting a disconnect from traditional open-source norms.

0 favorites 0 likes
#llamacpp

If you are running Qwen 3.8 Flash Next on Strix Halo, use this software for inference. It's so much faster than llama.cpp especially at high context.

Reddit r/LocalLLaMA ↗ · 2026-09-28

A user recommends a GitHub project for faster inference of Qwen 3.8 Flash Next on Strix Halo, showing benchmark speeds that significantly outperform llama.cpp.

0 favorites 0 likes
#llamacpp

i finally switched from windows to linux and got a 30-50% boost in speed.

Reddit r/LocalLLaMA ↗ · 2026-08-23

The user switched from Windows to Linux and replaced llamacpp with vllm, achieving a 30-50% speed improvement in AI inference.

0 favorites 0 likes
#llamacpp

Vision Support for Minimax-M3 has been merged into llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-07-26 Cached

Vision support for the Minimax-M3 model has been merged into the llama.cpp project, enabling multimodal inference for this model locally.

0 favorites 0 likes
#llamacpp

Laguna-S-2.1 runs on my 6 years old gaming PC!

Reddit r/LocalLLaMA ↗ · 2026-07-22

Laguna-S-2.1, a quantized AI model, runs on a 6-year-old gaming PC with an RTX 3080, achieving 10 t/s decode and using 8.3 GB VRAM and 52.2 GB host RAM.

0 favorites 0 likes
#llamacpp

There's a new PR for llamacpp claiming to boost prompt processing with rocm by around 15%, also fixes a bug which makes Q2_K 28x faster

Reddit r/LocalLLaMA ↗ · 2026-07-21 Cached

A new PR for llama.cpp boosts prompt processing on ROCm by ~15% and fixes a bug making Q2_K quantization 28x faster.

0 favorites 0 likes
#llamacpp

I tested all llama.cpp's speculative decoding methods on Qwen 3.6 27B: MTP ~2.7x, DFlash ~3.7x, n-gram stack ~6x on real coding. Local AI win. My findings on RTX 6000 PRO.

Reddit r/LocalLLaMA ↗ · 2026-07-16

Comprehensive benchmarks of llama.cpp's speculative decoding methods on Qwen 3.6 27B show n-gram stacking on DFlash achieves up to 6x speedup on iterative coding tasks, with ngram-mod providing most of the gain and zero VRAM cost.

0 favorites 0 likes
#llamacpp

@ivanfioravanti: DGX Spark Context Benchmark on Qwen3.6-35B-A3B-UD-Q8_K_XL llamacpp script released by Mia. It's fast! Time to test qual…

X AI KOLs Timeline ↗ · 2026-07-06 Cached

Benchmark results for Qwen3.6-35B-A3B-UD-Q8_K_XL on DGX Spark using llama.cpp script by Mia, showing fast token generation times across various context lengths.

0 favorites 0 likes
#llamacpp

I’m switching to Linux, is Ubuntu the most compatible with local AI?

Reddit r/LocalLLaMA ↗ · 2026-07-02

A user asks about Ubuntu's compatibility for local AI tools like vLLM, llama.cpp, and ComfyUI when switching to Linux.

0 favorites 0 likes
#llamacpp

Qwen3.6 27B more dumb in vLLM compared to llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-06-24

A user reports that the Qwen3.6-27B model performs better and more reliably with llama.cpp than with vLLM, citing tool call errors and 'lobotomized' behavior in vLLM despite extensive configuration.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback