llm-inference

A paper introduces galahad-kv, a memory layer that persists LLM key-value states to encrypted local NVMe disk and reloads them byte-exact instead of recomputing, demonstrating a 50-million-token window on a single H100 with Gemma 4 models that is 2.8–4.3x faster and 8.8–12.3x less energy-hungry than recompute.

A pull request to llama.cpp adds support for the K2 Horizon dense model and MoVA architecture, extending local LLM inference coverage in the popular C/C++ inference framework.

A publishing company details how it cut daily AI agent API costs from $5.86 to $1.21 (a 79% reduction) by reducing agent wake-ups and switching from timer-based heartbeats to on-demand execution, while sharing lessons on caching, cost-per-result measurement, and moving agents off a laptop onto a server.

An in-depth explainer covering parallelism strategies for LLM inference—data, tensor, pipeline, context, and expert parallelism—explaining how to split inference across GPUs without losing performance to communication overhead. It walks through prefill vs. decode phases, memory and communication bottlenecks, and how to choose a defensible layout given model, hardware, and workload constraints.

DLoop 提出一种循环式投机解码方法，在验证前自适应地进行多次起草阶段，通过循环感知训练保持草稿模型可靠性，在 EAGLE-3、DFlash 等方法上实现 5% 到 41% 的墙钟加速且保持无损解码。

DFlash 2, a lightweight block-diffusion model for flash speculative decoding from z-lab/haoailab, enabling efficient high-quality parallel drafting across a wide range of open LLM families with checkpoints released on Hugging Face.

llama.cpp v0.6.0 获得正式发布，新增对 Qwen4Exp 的 MTP（Multi-Token Prediction）推测解码支持以及大量其他改进。

A developer shares performance numbers and optimization ideas for streaming Qwen Flash Next Q4 MoE experts from SSD on a 64 GB Mac mini M5, achieving 17.5 tok/s decode with the GPU idle 27% of the time waiting on expert loads, and asks for ideas to improve expert prefetch accuracy and hit rates.

A dual-DGX-Spark user reports that a new GLM 5.3 Flash recipe delivers a 50-90% decode speed boost, making it faster than DeepSeek v4.0 Flash while offering better reasoning quality on local hardware.

A community-curated list of budget and legacy GPUs (NVIDIA GTX/Maxwell/Pascal, AMD Radeon and Instinct MI-series) that can run LLM inference through the Vulkan backend on Linux, bypassing the latest CUDA/ROCm support requirements. The AMD Instinct MI50 16GB at around $150 is highlighted as the top value pick.

作者将 llama.cpp 的 server 分支为 AgrillaMoE，针对 Qwen3.6-35B-A3B MoE 模型在 16GB 显存 GPU 上优化，配合 2-bit 量化可达 ~57-60 tok/s，并支持 OpenAI/Anthropic API 兼容接口；其 MoE 扩展补丁在运行时增加每 token 激活专家数（20 vs 8），无需重训即可在 GPQA-Diamond 上提升约 2.5 分。

An open-source project implements Qwen3.5 (9B/27B INT4) LLM inference in VHDL for FPGA fabric, running on cheap ex-mining SQRL FK33 and Jungle Cat boards (XCVU35P with HBM2), reaching ~2-3 tok/s generation at 75 MHz with projections scaling up via tensor parallelism and higher clocks. The repo (llm.vhdl, MIT licensed) also explores theoretical ASIC throughput at TSMC N3.

Developer forked the Strata LLM inference engine and used Claude Code to heavily modify it for an IBM AC922 POWER9 + Tesla V100 system, boosting Qwen3.8-FN UD-Q4_K_XL performance from 15 tk/s decode to ~113 tk/s decode and 7,357 tk/s prefill via memory management, NVLink optimization, and expert caching improvements.

A Reddit user's project turns an iPhone into a secondary GPU for a 24GB MacBook using a single 10 Gb/s USB-C cable, running the local Qwen 27B model and boosting long-text processing speed by over 40%.

A curated list of 10 open-source GitHub repos for building serious AI agent stacks, covering inference engines (SIE, vLLM, SGLang), agent frameworks (PydanticAI, DSPy, CrewAI, LlamaIndex), model gateways (LiteLLM), vector search (Qdrant), and self-hosted UIs (Open WebUI).

A personal project presents PULSAR-ASM, a minimal LLM inference engine written entirely in flat x86-64 assembly (FASM) that runs Gemma-2B in FP16 with a 5.2 KB binary footprint, zero C/C++ runtime dependencies, and ~4.5-4.7 tokens/s on an older quad-core CPU — framed as a first-principles exploration for bare-metal and microcontroller LLM inference.

This paper shows that models adapting on their own generated text during inference destabilize over long (128K-token) horizons, and proposes a causal decomposition plus a 'Settlement' validation mechanism that tests candidate updates on independent text before retaining them, substantially reducing damage while preserving genuine adaptation.

A tweet highlights a free ~5% tokens-per-second performance boost on Blackwell GPUs by enabling CustomAllReduce on SM120 when using tensor parallelism 4, joking that closed labs hate this trick.

PyTorch Foundation Ambassador Abdulsalam Bande will present a poster on ERRC (Entropy-Reinvested Residual Correction) at PyTorch Conference North America 2026 in San Jose, compressing inter-GPU communication to boost multi-GPU LLM inference efficiency, alongside the broader conference program covering the open-source AI stack.

该论文提出用短多项式（degree-3/4 BF16）替代 LLM 中原生的 sigmoid、tanh 和 SiLU 超越函数求值，在 NVIDIA GB200 上将完整训练步吞吐提升最高 8.0%，同时保持相近的训练损失和模型精度。