kv-cache

Tag

Cards List
#kv-cache

Why isn't the industry freaking out about DeepSeek 4.1 Flash?

Hacker News Top ↗ · 2026-10-08 Cached

作者基于一个月的重度使用体验,认为 DeepSeek 4.1 Flash 在编程任务上已与 Claude Opus 等前沿模型难以区分,但成本低几个数量级,业界却未因此震动。文章指出 KV 缓存缩减约 437 倍使全天会话成本不到一美元,预示便宜且'足够好'的中国模型正在改变开发者的工作方式。

0 favorites 0 likes
#kv-cache

Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

Hugging Face Daily Papers ↗ · 2026-10-07 Cached

A paper introduces galahad-kv, a memory layer that persists LLM key-value states to encrypted local NVMe disk and reloads them byte-exact instead of recomputing, demonstrating a 50-million-token window on a single H100 with Gemma 4 models that is 2.8–4.3x faster and 8.8–12.3x less energy-hungry than recompute.

0 favorites 0 likes
#kv-cache

@_avichawla: https://x.com/_avichawla/status/2107427400779026843

X AI KOLs Timeline ↗ · 2026-10-06 Cached

An in-depth explainer covering parallelism strategies for LLM inference—data, tensor, pipeline, context, and expert parallelism—explaining how to split inference across GPUs without losing performance to communication overhead. It walks through prefill vs. decode phases, memory and communication bottlenecks, and how to choose a defensible layout given model, hardware, and workload constraints.

0 favorites 0 likes
#kv-cache

@akshay_pachaar: How vLLM stops one long prompt from stalling others: (chunked prefill, clearly explained) Every LLM request has two pha…

X AI KOLs Timeline ↗ · 2026-10-05 Cached

A technical explainer on how vLLM's chunked prefill prevents long prompts from stalling concurrent decoding requests, covering latency tradeoffs, mixed batching, and the `--max-num-batched-tokens` knob in vLLM V1.

0 favorites 0 likes
#kv-cache

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

Reddit r/LocalLLaMA ↗ · 2026-10-05

Blockway released Agens Volundr 32B Preview, an Apache-2.0 32B dense model on a hybrid architecture where only 18 of 72 layers keep a KV cache, aiming for efficient long-context inference on consumer/local GPUs. The team reports competitive benchmarks versus Qwen3.8-27B on coding and math but weaker agentic performance, with a custom sglang build required for now.

0 favorites 0 likes
#kv-cache

Emergent Structure in the Marginal Attention Space of Language Models

arXiv cs.CL ↗ · 2026-10-05 Cached

The paper introduces a marginal attention space for language models, showing that token-wise reduction yields a text-intrinsic signal conserved across 60+ LLMs while head-wise reduction forms a model-private signature, enabling offline per-head KV-cache eviction budgets that perform competitively with per-document methods.

0 favorites 0 likes
#kv-cache

The Surprising Effectiveness of Shared Memory in Looped Transformers

arXiv cs.LG ↗ · 2026-10-05 Cached

An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.

0 favorites 0 likes
#kv-cache

WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds

arXiv cs.CL ↗ · 2026-10-05 Cached

WakeKV is a reactive, reversible KV-cache residency policy that dynamically demotes cooling attention heads to a recoverable CPU reservoir, outperforming frozen classification and destructive eviction baselines (SnapKV, R-KV, ReasonAlloc) on long-context serving, with a FlexiCache/vLLM implementation confirming throughput gains on Mistral-7B.

0 favorites 0 likes
#kv-cache

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

Reddit r/LocalLLaMA ↗ · 2026-10-02

A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.

0 favorites 0 likes
#kv-cache

@_avichawla: An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usua…

X AI KOLs Timeline ↗ · 2026-10-02 Cached

Researchers introduce KV-streams, an approach that compacts agentic RL rollouts by directly pruning the live KV cache instead of re-prefilling, nearly doubling RL training speed on SWE-bench Verified while keeping training and generation token distributions nearly identical.

0 favorites 0 likes
#kv-cache

@anemll: Effect of V-compression (int8) in KV-cache on prefill Qwen 3.8-27B, ANE M6 ANEMLL-FORGE. Compression (Quant-dequant) is…

X AI KOLs Following ↗ · 2026-10-02 Cached

ANEMLL-FORGE reports on applying int8 V-compression to the KV-cache of Qwen 3.8-27B during prefill on Apple's ANE M6, with compression and dequantization performed on ANE to yield memory I/O savings.

0 favorites 0 likes
#kv-cache

$5400 eBay 8x V100 server cranks on flash-next

Reddit r/LocalLLaMA ↗ · 2026-10-01

A user reports solid results running a $5400 eBay 8x V100 server for local LLM inference, using nvfp4 NVIDIA checkpoints via a heavily optimized vLLM fork (1CatAI/1Cat-vLLM) that unpacks to fp16 on the fly, achieving over 200 tok/s on a 27B model with TP=4 and strong prefill throughput.

0 favorites 0 likes
#kv-cache

@TheAhmadOsman: You do not pick a model and a GPU and call it done. You pick a file encoding, and a kernel path. The GPU follows those.…

X AI KOLs Timeline ↗ · 2026-10-01 Cached

An in-depth explainer on how LLM inference really works: the prefill/decode loop, KV caching, how file encodings and quantization labels (e.g. NVFP4) determine which kernels a GPU actually runs, demonstrated with Qwen 3.8 27B on an RTX 5090 — same model, same label, two very different kernel paths and performance.

0 favorites 0 likes
#kv-cache

@_avichawla: The LLM scheduling loop, clearly explained: (bookmark this) Whenever a user sends a prompt to an LLM, the resulting inf…

X AI KOLs Timeline ↗ · 2026-10-01 Cached

An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.

0 favorites 0 likes
#kv-cache

Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head

arXiv cs.CL ↗ · 2026-10-01 Cached

Peking University researchers introduce NAMOH, an architecture-native sparse attention mechanism that activates K of H heads per token, so head routing jointly determines active parameters and available context. It enables parameter scaling to directly support efficient long-context scaling, outperforming fully activated models with the same total parameters while remaining compatible with GQA and existing sparse attention methods.

0 favorites 0 likes
#kv-cache

tp=6 can work on vLLM, with padding

Reddit r/LocalLLaMA ↗ · 2026-10-01

A user shares a workaround enabling vLLM tensor parallelism with tp=6 on six GPUs by padding model architecture dimensions with zeros until divisible, achieving higher KV cache utilization for Qwen 27B inference on Radeon 7900 XTX GPUs.

0 favorites 0 likes
#kv-cache

@ekzhang1: So I would like to share my analysis of PD disagg that I've run by Horace and many folks at Thinky You may have heard t…

X AI KOLs Timeline ↗ · 2026-09-30 Cached

Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.

0 favorites 0 likes
#kv-cache

Best practice for processing batch vLLM api calls with shared prefix?

Reddit r/LocalLLaMA ↗ · 2026-09-30

The author describes their agent workflow using asyncio.gather to batch vLLM API calls with shared prefixes and asks for best practices to optimize this process, proposing a pre-caching strategy for efficient decoding.

0 favorites 0 likes
#kv-cache

TORQUE: Optimizing What (not) to Quantize Before and After Rotation

arXiv cs.LG ↗ · 2026-09-30 Cached

TORQUE 提出一种在旋转前后联合优化保留哪些坐标以高精度的量化框架,在固定比特预算下证明 top-k 保留可最小化误差上界,并在 KV 缓存压缩、激活压缩和近邻检索上取得更好的精度-存储权衡。

0 favorites 0 likes
#kv-cache

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

QuantMLA 提出一种功能对齐的双路径量化框架,对 MLA 的内容路径与 RoPE 路径进行低比特 KV 缓存压缩,首次实现 INT4 联合缓存且精度几乎无损,并通过自定义低比特注意力内核在 128K 上下文下获得 3.59 倍压缩和 5.168 倍吞吐提升。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback