Tag
作者基于一个月的重度使用体验,认为 DeepSeek 4.1 Flash 在编程任务上已与 Claude Opus 等前沿模型难以区分,但成本低几个数量级,业界却未因此震动。文章指出 KV 缓存缩减约 437 倍使全天会话成本不到一美元,预示便宜且'足够好'的中国模型正在改变开发者的工作方式。
A paper introduces galahad-kv, a memory layer that persists LLM key-value states to encrypted local NVMe disk and reloads them byte-exact instead of recomputing, demonstrating a 50-million-token window on a single H100 with Gemma 4 models that is 2.8–4.3x faster and 8.8–12.3x less energy-hungry than recompute.
An in-depth explainer covering parallelism strategies for LLM inference—data, tensor, pipeline, context, and expert parallelism—explaining how to split inference across GPUs without losing performance to communication overhead. It walks through prefill vs. decode phases, memory and communication bottlenecks, and how to choose a defensible layout given model, hardware, and workload constraints.
A technical explainer on how vLLM's chunked prefill prevents long prompts from stalling concurrent decoding requests, covering latency tradeoffs, mixed batching, and the `--max-num-batched-tokens` knob in vLLM V1.
Blockway released Agens Volundr 32B Preview, an Apache-2.0 32B dense model on a hybrid architecture where only 18 of 72 layers keep a KV cache, aiming for efficient long-context inference on consumer/local GPUs. The team reports competitive benchmarks versus Qwen3.8-27B on coding and math but weaker agentic performance, with a custom sglang build required for now.
The paper introduces a marginal attention space for language models, showing that token-wise reduction yields a text-intrinsic signal conserved across 60+ LLMs while head-wise reduction forms a model-private signature, enabling offline per-head KV-cache eviction budgets that perform competitively with per-document methods.
An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.
WakeKV is a reactive, reversible KV-cache residency policy that dynamically demotes cooling attention heads to a recoverable CPU reservoir, outperforming frozen classification and destructive eviction baselines (SnapKV, R-KV, ReasonAlloc) on long-context serving, with a FlexiCache/vLLM implementation confirming throughput gains on Mistral-7B.
A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.
Researchers introduce KV-streams, an approach that compacts agentic RL rollouts by directly pruning the live KV cache instead of re-prefilling, nearly doubling RL training speed on SWE-bench Verified while keeping training and generation token distributions nearly identical.
ANEMLL-FORGE reports on applying int8 V-compression to the KV-cache of Qwen 3.8-27B during prefill on Apple's ANE M6, with compression and dequantization performed on ANE to yield memory I/O savings.
A user reports solid results running a $5400 eBay 8x V100 server for local LLM inference, using nvfp4 NVIDIA checkpoints via a heavily optimized vLLM fork (1CatAI/1Cat-vLLM) that unpacks to fp16 on the fly, achieving over 200 tok/s on a 27B model with TP=4 and strong prefill throughput.
An in-depth explainer on how LLM inference really works: the prefill/decode loop, KV caching, how file encodings and quantization labels (e.g. NVFP4) determine which kernels a GPU actually runs, demonstrated with Qwen 3.8 27B on an RTX 5090 — same model, same label, two very different kernel paths and performance.
An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.
Peking University researchers introduce NAMOH, an architecture-native sparse attention mechanism that activates K of H heads per token, so head routing jointly determines active parameters and available context. It enables parameter scaling to directly support efficient long-context scaling, outperforming fully activated models with the same total parameters while remaining compatible with GQA and existing sparse attention methods.
A user shares a workaround enabling vLLM tensor parallelism with tp=6 on six GPUs by padding model architecture dimensions with zeros until divisible, achieving higher KV cache utilization for Qwen 27B inference on Radeon 7900 XTX GPUs.
Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.
The author describes their agent workflow using asyncio.gather to batch vLLM API calls with shared prefixes and asks for best practices to optimize this process, proposing a pre-caching strategy for efficient decoding.
TORQUE 提出一种在旋转前后联合优化保留哪些坐标以高精度的量化框架,在固定比特预算下证明 top-k 保留可最小化误差上界,并在 KV 缓存压缩、激活压缩和近邻检索上取得更好的精度-存储权衡。
QuantMLA 提出一种功能对齐的双路径量化框架,对 MLA 的内容路径与 RoPE 路径进行低比特 KV 缓存压缩,首次实现 INT4 联合缓存且精度几乎无损,并通过自定义低比特注意力内核在 128K 上下文下获得 3.59 倍压缩和 5.168 倍吞吐提升。