Tag
K2 Horizon 7B, a compact AI model, achieves impressive benchmark scores rivaling larger models and shows solid performance in initial testing for tasks like compiling llama.cpp for CUDA.
Otis is an open-source AI agent that provides a minimal, privacy-focused experience for running local and hosted open-weight models, automatically recommending and downloading models based on hardware.
This pull request adds support for the Maple 20B-A1B ternary Mixture of Experts architecture for CPU inference in the llama.cpp framework.
A Reddit user shares how they successfully ran the Qwen 3.8 Next MoE model on a system with 16GB VRAM and 32GB RAM using aggressive quantization and specific llama.cpp settings, achieving usable performance for large models on limited hardware.
An experiment using the Qwen3.8-Flash-Next-IQ4_XS model on llama.cpp to generate SVG art from a prompt, showing impressive results with some preview discrepancies in OpenWebUi.
The author shares their experience building a high-VRAM AI PC with multiple GPUs to run local models for software engineering, detailing solutions to hardware challenges like PSU limitations and GPU mounting.
ThreadShelf is an open-source tool that allows users to locally archive, search, and continue chats from OpenRouter, LM Studio, and AI Studio, with semantic search and llama.cpp integration.
The user describes struggles with context, compaction, and memory management for local AI models using pi.dev plugins and seeks suggestions for solutions that handle varying model context windows and VRAM limitations.
The user describes a hardware configuration with dual RTX 3090 GPUs and an AMD EPYC CPU for running the Qwen3-Flash-Next model using llama.cpp, achieving 38 tokens per second in single-stream inference, and seeks advice on whether to add a third GPU or upgrade the CPU to improve performance, especially for running multiple parallel agents.
This pull request adds missing AMD GCN MMQ configuration to ggml-cuda for HIP, enhancing prefill performance for RDNA2 GPUs such as MI50 and MI60 in the llama.cpp inference library.
This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.
Google launches llama.app, a wrapper for llama.cpp that serves as a local server for running AI models.
A benchmarking report on running multiple AI agents concurrently using llama.cpp on 2× RTX 4090 GPUs, revealing performance limits and optimal configurations for Qwen models.
The article discusses alternatives to the official llama.cpp for optimized performance on Strix Halo hardware, with specific forks achieving significantly higher throughput for AI inference tasks.
The author advocates for ExLlamaV3, an underrated local AI inference tool that outperforms llama.cpp on NVIDIA hardware, with recent updates like CPU MoE offload enhancing its performance.
A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.
A developer built a custom branch of llama.cpp that implements expert expansion for Mixture-of-Experts models, tested it on Metal, and is seeking cross-platform feedback.
A GitHub pull request adds support for the Tencent Hy 4 (hy_v4) preview architecture to the llama.cpp C/C++ inference library, enabling local inference for this new model.
Unsloth AI announces optimizations for GLM-5.3-Flash, enabling 1.6–3.4× faster local GGUF inference with multi-token prediction and hardware requirements for running models locally.
The article details the merging of Multi-Token Prediction (MTP) support into ik_llama.cpp, which significantly enhances token generation speed for the Qwen3.8-Flash-Next model, with benchmarks showing up to double the speed on high-end hardware.