Tag
The user fixed crashes when running large AI models on an AMD GPU in headless Linux by disabling runtime power management, preventing model weights from being dumped into limited RAM and causing OOM errors.
The llama.cpp project has launched a new official documentation site at llama.app, offering comprehensive guides for local LLM inference using llama cli and llama server.
llama.cpp now supports MTP (Multi-Token Prediction) for GLM-Air models, enhancing inference capabilities for large language models in C/C++.
This article benchmarks various engines for running the Qwen 3.8 27B model on macOS, comparing their speed and performance in agentic coding tasks, and recommends MTPLX or llama.cpp with MTP for best results.
The article discusses an AMD-specific branch of llama.cpp that significantly boosts prompt processing speed for AMD users, with up to 2x faster performance on dense models using ROCm/Hip, though with some trade-offs in other metrics.
The author conducted experiments to run DeepSeek-V4-Flash-0731 with 4-bit quantizations on a 128GB RAM system, using optimizations like memory mlocking and prompt processing strategies to achieve acceptable inference speeds.
A fork of llama.cpp has been created and optimized for AMD GFX906 GPUs, improving performance on Mi50, Mi60, Radeon VII, and GCN HIP for machine learning and LLM applications.
A user forked the Ninfer inference engine to run on CMP170HX GPUs, doubling performance for the Qwen3.6-35B model with llama.cpp and sharing the configuration details.
This article provides a detailed installation guide for running the Qwen3.8-27B-Unleashed model on an RTX 3060 12GB card, covering version selection, download issues, and startup configuration, and summarizes solutions for common errors.
Llama.cpp, a popular open-source tool for running LLaMA models, has released version 0.2.0 with changelog and pre-built binaries available on GitHub.
An implementation of DSpark PC Tree in a llama.cpp fork shows performance gains up to 29.5% faster inference on Qwen 3.0 models based on benchmark results.
The benchmark compares DFlash2 and MTP techniques in llama.cpp, showing that DFlash2 offers around 20% faster token generation but reduces available context by 38%.
Liquid AI releases DSpark draft model checkpoints for the LFM2.5 family, enabling up to 3.2x faster inference on GPUs and devices with minimal quality trade-off, and with day-one support for open-source tools like llama.cpp and SGLang.
This draft pull request introduces AVX2 optimizations to speed up large batch size prompt processing for IQ models in llama.cpp, an open-source tool for efficient LLM inference.
This article presents the GGUF build of the Mach-1-Additive-35B AI model, optimized for local inference on consumer hardware with a custom llama.cpp fork, and highlights strong benchmark performance.
The author benchmarked llama.cpp flags on a hybrid GPU setup with an RTX 4090 laptop and AMD XTX 7900 eGPU, achieving 70% faster generation, 40% faster prefill, and discovering a bug related to MTP in multi-GPU configurations.
The user proposes an automatic caching mechanism for long-context sessions in llama.cpp to avoid repeated prefills after restarts, enhancing usability on slower hardware.
The post details the quantization of Ornith 1.5 9B and 35B-A3B AI models using Atomic Dynamic methods, providing benchmarks against stock quantizations and sharing Hugging Face collections.
The article demonstrates running the Ling-3.0-tiny AI model on an NVIDIA Orin Nano Super 8GB device with IQ4_NL quantization, achieving 33 tok/s decode speed and full 128K context, showcasing practical edge AI deployment.
The user tested DFlash2 on the Qwen3.8 27B model, reporting improved inference speeds for code generation but with increased memory usage compared to MTP.