kv-cache

Tag

Cards List
#kv-cache

@dair_ai: We love papers that go deeper down the stack. This is a good example that shifts from agentic context engineering to ag…

X AI KOLs Timeline ↗ · 2026-10-21 Cached

A paper proposing 'cache engineering', letting LLM agents directly manipulate their KV cache via drop and repos operations without invalidating the trajectory, co-designed with the AKV serving engine, achieving 27.7% cost reduction, 1.2x throughput, and 7.1% further savings via recursive self-improvement of cache policies.

0 favorites 0 likes
#kv-cache

@swyx: i retired from the business of making single-stock calls a decade ago but trust me when i say that the stock market sti…

X AI KOLs Following ↗ · 2026-10-21 Cached

swyx argues the market hasn't priced in that every GPU sold gained ~30% in lifetime value, citing Latent.Space's breakdown of DeepSeek's production numbers: 155T tokens/day with high cache hit rates and ~80% GPU cost margins.

0 favorites 0 likes
#kv-cache

@PyTorch: Simon Mo (@simon_mo_) of @inferact presents the state of @vllm_project at #PyTorchCon 2026, highlighting optimizations …

X AI KOLs Following ↗ · 2026-10-20 Cached

Simon Mo of Inferact presented the state of the vLLM project at PyTorchCon 2026, highlighting KV cache management and kernel optimizations across TPU, AMD, and NVIDIA hardware to address LLM performance and scalability challenges.

0 favorites 0 likes
#kv-cache

@PyTorch: When GPU memory fills up, @vllm_project offloads the KV cache to CPU or storage. @_llm_d_ tracks where every block live…

X AI KOLs Following ↗ · 2026-10-20 Cached

PyTorch Conference North America in San Jose (Oct 20-21) will feature a poster by Bongwoo Bak of Moreh on KV cache block placement, covering vLLM offloading strategies between GPU, CPU, and shared storage. The post also promotes conference registration tiers and pricing.

0 favorites 0 likes
#kv-cache

Cache Engineering for Agents

Hugging Face Daily Papers ↗ · 2026-10-20 Cached

提出 AKV 的 cache engineering 方案,让 LLM agent 在不修改轨迹的前提下直接操作 KV cache(drop/repos),并配合协同设计的 serving 引擎,在降低成本与延迟的同时提升基准分数。

0 favorites 0 likes
#kv-cache

Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM (11 minute read)

TLDR AI ↗ · 2026-10-20 Cached

AWS introduces a push-based NIXL connector in vLLM that moves KV cache transfer coordination off the critical path for disaggregated prefill-decode inference, delivering up to 3x better TTFT and 30% better TPOT on EC2 P5en instances with EFA.

0 favorites 0 likes
#kv-cache

We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.

Reddit r/LocalLLaMA ↗ · 2026-10-19

Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎(MIT 开源),通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache,使 262K 长上下文能在 16 GB GPU 预算内运行,并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。

0 favorites 0 likes
#kv-cache

Nemotron 3.5 Bolt is okayish for small multiuser setups

Reddit r/LocalLLaMA ↗ · 2026-10-19

A hands-on review of Nemotron 3.5 Bolt, noting its efficient KV cache, 2-bit/NVFP4 quantization, and Mamba2 hybrid attention that make it well-suited for small multi-user setups on limited hardware, despite only mediocre intelligence by current standards.

0 favorites 0 likes
#kv-cache

Why harnesses matter for energy (and token) efficiency

Hacker News Top ↗ · 2026-10-16 Cached

Neuralwatt 分析一个月云端流量发现,agent harness 如何构建 prompt 决定了大量 GPU 计算量:缓存命中率差异导致未缓存输入在 1.6% 到 11.1% 之间,一次缓存失效大致会让请求能耗翻倍,且这些问题多数可通过避免编辑已发送内容来修复。

0 favorites 0 likes
#kv-cache

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

arXiv cs.CL ↗ · 2026-10-16 Cached

A survey paper that reframes long-context compression as a memory-management problem for LLMs, unifying prompt compression, KV-cache management, retrieval augmentation, and agent memory into a single information-lifecycle framework with a budgeted cost model.

0 favorites 0 likes
#kv-cache

RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

arXiv cs.LG ↗ · 2026-10-16 Cached

RoLaQ提出了针对多头潜注意力(MLA)模型的2位KV缓存联合旋转与范围校准的离线量化方法,在DeepSeek-V2-Lite上实现3.96倍BF16 KV容量和2.51倍解码吞吐提升。

0 favorites 0 likes
#kv-cache

DLite: History-free Speculative Decoding for Agentic AI

Hugging Face Daily Papers ↗ · 2026-10-16 Cached

DLite提出一种无历史上下文的投机解码框架,通过复用目标模型的隐状态进行局部预测并省去历史KV缓存维护,在agentic工作负载下将接受长度提升至2-6,较SOTA方法DSpark提升最高70%。

0 favorites 0 likes
#kv-cache

I tested Qwen3.8 27B KLD for 21 Models at 9 different KV quants that fit on 16GB VRAM (189 benchmarks).

Reddit r/LocalLLaMA ↗ · 2026-10-15

A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.

0 favorites 0 likes
#kv-cache

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

arXiv cs.CL ↗ · 2026-10-15 Cached

Titan-Llama 提出一种后训练方法,通过分层记忆架构和测试时参数更新,将预训练注意力模型改造为具备外部神经记忆模块的长上下文高效推理模型,在 QuALITY 和 LongHealth 上显著提升准确率并降低推理延迟与显存占用。

0 favorites 0 likes
#kv-cache

Joint Subspace Sparse Attention for Efficient Long-Context Decoding

arXiv cs.CL ↗ · 2026-10-15 Cached

论文提出联合子空间稀疏注意力(JSSA),通过学习共享低维查询-键子空间并设计免gather的索引注意力核,在128K上下文下实现最高6.43倍解码吞吐提升,同时保持接近全KV的精度。

0 favorites 0 likes
#kv-cache

First Passage Attention

Hugging Face Daily Papers ↗ · 2026-10-15 Cached

提出 First Passage Attention(FPA),用路径极值替代单调衰减累积,从而在压缩 KV 缓存的同时实现完美的长程检索;在 110M 语言模型上以约 25% 缓存达到 8 倍训练长度的 passkey 检索。

0 favorites 0 likes
#kv-cache

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Hugging Face Daily Papers ↗ · 2026-10-15 Cached

该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈,提出按并行 rank 分片 MLA KV 存储、异步加载等优化,将 p99 延迟从 83.32 秒降至 0.67 秒,并显著提升预填充吞吐。

0 favorites 0 likes
#kv-cache

Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM (15 minute read)

TLDR AI ↗ · 2026-10-15 Cached

vLLM 推出基于推送(push-based)的 NIXL KV 缓存传输机制,将传输协调与准备工作移出关键路径,在分离式预填充/解码推理中最高可将 TTFT 改善 3 倍、TPOT 改善 30%。该功能在 AWS Trainium/EFA 集群上开发验证后,作为加速器无关特性贡献给 vLLM。

0 favorites 0 likes
#kv-cache

Sabi-Noema: A capability and efficiency focused fine-tune of Qwen 3.8 27B

Reddit r/LocalLLaMA ↗ · 2026-10-14

发布了 Qwen 3.8 27B 的两个后训练版本 Sabi-Noema 与 Sabi-Noema-MLA,在代码、智能体和工具调用能力上提升,同时将回答冗长度和推理 token 消耗减少约一半,并通过无训练的 MLA 注意力转换将 KV cache 最多压缩 8.4 倍。

0 favorites 0 likes
#kv-cache

PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference

Hugging Face Daily Papers ↗ · 2026-10-14 Cached

提出PN-Mix,一种面向RoPE解码器LLM的混合精度键缓存量化方法,通过查询-键对范数识别显著通道对,在约2.26位平均精度下达到或超越现有方法在2.32–2.39位下的精度,且可优雅降级至1.5位。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback