Tag
KVCache Meetup Korea 2026, co-hosted by PyTorchKR and XCENA, featured talks from XCENA, SK hynix, FuriosaAI, and LMCache on CXL memory pooling, GPU/HBM roofline analysis, batch scheduling on RNGD, and process-isolated KV Cache serving.
RidgeFill 通过用闭式解岭回归预测深层输入、跳过大部分上层计算来加速 LLM prefill，在 Qwen3.8 系列上实现最高 1.6x 的 TTFT 提速，且几乎不损失检索精度。
A discussion of NVIDIA's TriAttention research on KV cache compression in vLLM, arguing that real gains come from freeing GPU memory blocks rather than evicting tokens, with reported 2.5× faster decoding and 10.7× less KV cache memory.
This paper introduces bidirectional KV-cache patching as a method to trace how fine-tuned behaviors are instantiated in language models, showing that behaviors like responding in German or emergent misalignment can be transferred between a LoRA fine-tuned model and its base model via KV cache reuse, with effects concentrating on pivotal tokens such as system-prompt and chat-template positions.
The paper introduces Agentic Soft-Token Compression (ASTC), a training-free method that uses an LLM's native recurrent dynamics (GDN-attention) to compress long-horizon agent context into soft tokens, matching or exceeding uncompressed accuracy with up to 50.68% fewer Attention Pairs.
A developer shares open-source vLLM patches and Triton kernels that run 8 coding agents with 250K-token contexts concurrently on two RTX 3090s by keeping the full int4 KV cache in system RAM and fetching only ~1% of tokens per attention step via a rank-64 key sketch, reporting ~545-790 tok/s total and 31/32 recall at 250K.
A developer patched Strata to speed up DeepSeek V4.1 agent sessions 20× on consumer hardware by adding KV-splice prompt caching and resident-only expert routing during prompt reads, running a 1M-context model fully offline on a gaming GPU.
A developer diagnoses why identical browser-agent runs on a local 27B model ranged from 40 to 261 seconds, eventually finding the culprit was a requirement-checking rule layer forcing unnecessary page opens rather than the model itself.
GAAC is a lossless KV cache compression and serving system that exploits BF16 exponent redundancy with group-adaptive codes and a fused attention kernel, achieving ~1.4x KV compression and 1.04-1.27x speedups on NVIDIA DGX Spark and Apple M4 without approximating KV values.
A detailed benchmark shows QEngine 1.3.1 (compiled for RTX 5090s) is the fastest Qwen3.8-Flash-Next engine for single-stream decode, ~1.4-1.9x faster than Strata and ~3x faster than tuned vLLM, with a KV-in-RAM mode that dramatically improves warm returns to idle conversations.
The author argues that two common coding-agent complaints—burning through weekly usage quotas and ignoring style guides to write defensive TypeScript—stem from one attention bug: pasting 'DON'T' code examples into prompts loads those token sequences into the KV-cache, where they act as semantic attractors and prime the model to emit exactly the banned pattern, triggering long, expensive correction loops. They propose a 4-part positive override structure for prompts and passing call-graph blast radii to subagents via Graphviz strict digraphs instead of prose or Mermaid.
Part 6 of an open-source Inference Engineering repo is released, covering KV cache memory allocation, growth, release, and capacity-aware scheduling using real model inference and DynamicCache.
文章清晰解释了前缀感知的 LLM 路由机制：通过按块哈希提示词并维护块到 worker 的索引，让请求路由到持有共享前缀 KV 缓存的 GPU，实测在 8 个 vLLM Pod（16 张 H100）上将 p90 首 token 延迟从 92.5 秒降至 0.54 秒。
DeepSeek open-sourced DeepInfer, its inference stack that reuses KV cache from SSDs via 3FS to deliver first tokens up to 26x faster, enabled by MLA's 93% KV cache compression.
WAKMan is a workload-aware KV cache manager for LLM servers that automatically selects its eviction policy from live traffic, improving hit rate over vLLM's LRU by 5 percentage points on average across mixed single-turn, multi-turn, and agentic request workloads.
This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.
提出了 RAQ-MLA，一种针对 Multi-Head Latent Attention 缓存的角色感知量化方法，无需训练即可将 MLA 缓存压缩至约 2.5 比特，在四个 MLA 模型上推理与编码任务几乎无损。
TraceQuery 是一个轻量级稀疏解码框架，通过全去噪轨迹监督预测需要重新计算的少量位置，为扩散大语言模型（dLLM）推理提供最优的质量-计算权衡。
Cloudflare disclosed a tenant isolation flaw in its GLM-5.3 Workers AI model where cross-request KV cache exposure leaked other users' data in 155 long-context responses, caused by a SGLang misconfiguration and bug, now remediated with hardened deployments.
提出 Full-Duplex Long-Horizon Benchmark（FD-LHB），发现现有全双工语音模型在长时间对话中会因上下文耗尽或重复、静音、乱码而崩溃，并通过 Initialization-Anchor Pinning 和 Duplex Repetition Penalty 两种免重训练修复方法显著缓解对话退化。