Tag
Observer, an open-source app that lets local LLMs monitor your screen and send notifications, now supports no-install in-browser model downloads (Qwen3.5-0.8b via transformers.js) plus a desktop llama.cpp build. The update makes local AI screen monitoring accessible in about 10 seconds without any setup.
The llama.cpp team and Hugging Face launched Llama (llama.app), a tiny open-source menu bar/system tray app for Mac and Windows that runs open models locally with an OpenAI-compatible API at localhost:9931/v1.
Infermeld is an experimental open-source Linux companion kit (v0.1.0) for running one GGUF model across mixed AMD and NVIDIA GPUs using llama.cpp, packaging device selection, build instructions, preflight checks, and a thermal guard. Tested on an RX 6000 XT + RTX 3080 with Qwen3.6-35B-A3B, with the maintainer seeking other hardware combos for reproduction.
A user reports that a newly hyped inference engine degrades into a repetitive 'doom loop' when running an IQ3_S quantized model, while the same model runs coherently under llama.cpp — suggesting the new engine may have issues with small quantized open-weights models.
awesome-jev 的定期巡检新增 3 条条目（555 → 558）：llama.cpp 新增 System One 决策模型运行时支持、Databricks 推出 ai_decide AI Function 作为结构化决策 API，以及一个基于 Jev 官方文档的实际 RAG 测试案例。
llama.cpp PR #29818 adds a new /v1/systemone API endpoint to the llama-server, supporting the models laya, julia-1, lev, openjev, and kev, submitted by ngxson.
Janus 是一个单文件 Go 二进制工具，通过 Vulkan 在 AMD/Intel/NVIDIA GPU 或 CPU 上运行 GGUF 模型，并提供 OpenAI 兼容 API，无需 Python、Docker 或 Ollama。
The article argues that specialized, one-off inference engines will outperform general ones like llama.cpp due to optimization for specific models and hardware, predicting their rise as the norm in AI inference.
The author proposes optimizing llama.cpp for specific models by stripping unnecessary code and using AI to automate the process, aiming for 2x+ performance improvements.
A user shares their experience running the Qwen3.8-Flash-Next model on a Mac M4Pro, highlighting faster performance with quantization and achieving 131K context size using llama.cpp.
The author shares their experience using Tesla P100 GPUs for local AI inference, finding them cost-effective and performant with optimizations via llama.cpp, despite initial advice from frontier models.
An experiment applies logit penalties for specific overthinking tokens to Qwen models using llama.cpp, improving accuracy on the MATH-500 benchmark across various quantizations, with notable accuracy gains and reduced reasoning tokens.
A blog post detailing performance optimizations in llama.cpp that make prompt lookup drafting up to 42x faster and reduce memory usage by 2.6x, based on techniques from Daniel Lemire and Martin Ankerl.
OrcaRouter released OrcaSAQ-2-Cyber-27B-Uncensored-GGUF, a quantized 27B reasoning model that compresses an uncensored Qwen3.8-27B from 54.7 GB BF16 to 15.7 GB with only +0.80% perplexity degradation, 94.4% Top-1 agreement, and 262K context for local deployment via llama.cpp, Ollama, and LM Studio.
Mica v0.1 4B is an AI model that plays a real Minecraft server, making decisions without generating tokens, and successfully crafted an iron pickaxe from empty inventory in 23 steps with a latency of 90-150 ms per decision.
The author transferred pretrained PLE n-gram memory from Qwen3.8-Flash-Next to Qwen3.5-0.8B, achieving a 5.05% reduction in validation perplexity without fine-tuning the backbone. Key findings include the effectiveness of dynamic gating and reader training for memory injection.
This update to llama.cpp introduces a Vulkan-based implementation of int8 cooperative matrix multiplication for AMD RDNA3 and RDNA4 GPUs, enhancing inference performance on these hardware platforms.
A pull request to add support for the Ling 3.0 VL model in llama.cpp, enhancing its capability to handle vision-language models.
A user shares their local AI setup using two BC-250 ex-mining APUs to run the Qwen3.6-35B-A3B model with llama.cpp, achieving 60 tok/s and 64k context for under $300.
Bonsai-Llama-Jev is an open-source, vision-enabled typed-decision inference system that runs locally with low VRAM and high accuracy, outperforming other systems in a diverse benchmark.