Tag
KVCache Meetup Korea 2026, co-hosted by PyTorchKR and XCENA, featured talks from XCENA, SK hynix, FuriosaAI, and LMCache on CXL memory pooling, GPU/HBM roofline analysis, batch scheduling on RNGD, and process-isolated KV Cache serving.
RidgeFill 通过用闭式解岭回归预测深层输入、跳过大部分上层计算来加速 LLM prefill，在 Qwen3.8 系列上实现最高 1.6x 的 TTFT 提速，且几乎不损失检索精度。
Hierarchos Native, a Python-free Rust+Vulkan backend for LLM training, fine-tuning and inference, received a major update with 58% lower forward latency on Intel HD 520, RDNA3 qualification passing 166/166 runs, and a GPT-2 numerical parity fix.
Berth is a lightweight Go-based GPU resource manager that lets CUDA workloads like llama.cpp and ComfyUI share VRAM via leases, priorities, and a queue, with model parking, CUDA checkpointing, and an OpenAI-compatible gateway. It is currently in alpha (v0.4.1) with Linux NVIDIA support and experimental AMD/Windows support.
A developer patched Strata to speed up DeepSeek V4.1 agent sessions 20× on consumer hardware by adding KV-splice prompt caching and resident-only expert routing during prompt reads, running a 1M-context model fully offline on a gaming GPU.
A GitHub repo shared by a user shows a notable inference speedup for running Qwen3.8 27B locally on the AMD Radeon RX 7900 XTX GPU.
A detailed write-up of migrating a multi-GPU LLM inference rig from a Core Ultra 7 265KF to an EPYC 7443 platform, covering BMC fan fixes, RAM overclocking, BIOS/PCIe tuning, and llama.cpp tweaks — yielding up to 4.2× faster decode for MoE models and big bandwidth/IOPS gains.
作者讨论了LLM推理引擎生态日益碎片化的现象，从早期依赖llama.cpp转向各种fork项目（如Niffler、Strata、Avalon、Basalt等），并质疑llama.cpp为何在性能上落后于许多独立开发者的作品。
A user benchmarked LLM inference on rented vast.ai servers with dual AMD EPYC 7642 (ROME) boards, large DRAM, and RTX 3090s, finding that RAM-heavy server builds deliver near-linear scaling up to 8 concurrent streams — hitting 94 tok/s at c=1 and 456 tok/s at c=8 on Qwen 3.8 Flash Next Unsloth Q8.
A technical explainer on why LLM outputs can vary between identical requests: floating-point accumulation order changes with batch size on the GPU, causing tied tokens to flip and completions to diverge even at temperature 0. It highlights batch-invariance flags in vLLM, SGLang, and DeepSeek's DeepInfer, noting the throughput trade-offs for reproducible outputs.
GAAC is a lossless KV cache compression and serving system that exploits BF16 exponent redundancy with group-adaptive codes and a fused attention kernel, achieving ~1.4x KV compression and 1.04-1.27x speedups on NVIDIA DGX Spark and Apple M4 without approximating KV values.
GAUSS 是一个工作负载感知的分析与蒙特卡洛模拟器，可预测 LLM 推理服务中 TTFT 和 TBT 的完整延迟分布及可持续请求速率，支持对未部署的工作负载、路由策略和硬件进行容量规划与尾延迟分析。
A speculative analysis predicting that Qwen4's hybrid-context (HC) architecture will widen the quality gap between Q4 and Q6 quantization, requiring HC-aware quantization strategies and favoring llama.cpp over NVFP4-focused tools on non-Blackwell hardware.
TKintegers 推出基于平衡三进制权重表示 K5 的转换器与 GPU 推理实现，在 NVIDIA L4 上对 Qwen2.5-0.5B-Instruct 实现相对 TorchAO FP8 基线约 3.21× prefill 与 3.10× decode 吞吐提升。
Part 6 of an open-source Inference Engineering repo is released, covering KV cache memory allocation, growth, release, and capacity-aware scheduling using real model inference and DynamicCache.
The author recommends Baseten's "Inference Engineering" book, suggesting readers with some background start with Chapters 5, 6, and 2 to systematically learn about LLM inference techniques.
A technical analysis of optimizing LLM inference using agent fleets, based on DeepSeek 4.1 Flash optimization reaching 2300 tok/s, highlighting that most gains come from configs/graphs rather than new kernels, questioning megakernel approaches, and emphasizing verification and convergence in automated optimization runs.
A curated list of 8 open-source GitHub repos (Ollama, vLLM, LangGraph, Smolagents, LlamaIndex, MCP servers, n8n, Browser Use) that together form a lean, self-hostable AI agent and automation stack for 2026, replacing stacked SaaS subscriptions.
Princeton and Polymathic AI introduce FlashMap, a continuous flow drafter for speculative decoding that generates an entire token block in one forward pass while capturing inter-token dependencies, achieving up to 6.1× lossless acceleration on Qwen3-8B and outperforming the DFlash baseline on every benchmark.
提出 Boa Compressor，一个基于 Mamba 状态空间模型的即插即用在线上下文压缩器，通过单次线性时间编码任意长度上下文，并以蒸馏方式训练以匹配完整上下文下的下一词元分布，在相同压缩率下匹敌或超越离线压缩基线且延迟降低一个数量级。