inference-optimization

NVIDIA highlighted how European retailers are adopting its open models and physical intelligence technologies, with Shopify cutting inference costs by 90% and retailers like zooplus, Vestiaire Collective, and L'Oréal using digital twins and AI for product discovery, image processing, and research.

提出 KVLearn——一种面向分离式 LLM 服务架构的基于学习的 KV 缓存保留框架，通过前缀复用预测、成本感知保留分数和自适应阈值控制，端到端首 token 延迟最多降低 56%，节点间 KV 传输量最多减少 53%。

Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference that supports broad GPU coverage, diverse quantization formats, and outperforms existing implementations like Marlin, FlashInfer, and DeepGEMM.

A paper proposing 'cache engineering', letting LLM agents directly manipulate their KV cache via drop and repos operations without invalidating the trajectory, co-designed with the AKV serving engine, achieving 27.7% cost reduction, 1.2x throughput, and 7.1% further savings via recursive self-improvement of cache policies.

NVIDIA researchers introduce MLIPS, a training-free method that shares a producer layer's broad candidate pool across sparse attention layers, achieving 1.14–1.40× speedups and 14.8–27.4% lower GPU energy on DeepSeek-V4-Flash with minimal accuracy change.

EAGLE-4 发布，作为投机解码加速组件在 Qwen3.8-27B 上达到 961.6 tokens/s，单请求下比 DFlash 2 快 1.28 倍、比 DSpark 快 1.40 倍，通过 draft-and-verify 循环实现约 3 倍加速且输出完全一致。

提出 ALLOT：一种训练自由的块稀疏注意力分配方法，通过细胞探测与轨迹校准在延迟预算内优化视频扩散 Transformer 各层-步的稀疏度，在 HunyuanVideo 上实现 11.8 dB PSNR 提升。

Ujval Kapasi of NVIDIA delivers a PyTorchCon 2026 keynote on what agentic workloads demand from the PyTorch stack, using NVIDIA Vera Rubin as a case study for hardware-software co-design across CPUs, GPUs, memory, and networking, covering optimizations in PyTorch, Triton, and vLLM.

Shopify 在 PyTorchCon 2026 主旨演讲中展示基于 PyTorch 与 vLLM 的持续学习闭环：校准评审模型标记低分输出，批评者诊断失败并通过重放修复，使小规模专用模型在特定任务上超越前沿模型，同时通过提示压缩优化推理。

该论文研究在 MoE 模型 prefill 阶段减少路由专家对短答案问答 F1 的影响，发现大部分损失来自提示词首尾少数敏感 token，并提出将约 4% 的 token 保持全专家以几乎无损地降低 TTFT。

The paper presents Apex, a CPU-GPU collaborative MoE inference system that adaptively assigns GPU-missed experts to either CPU or GPU based on token load, improving prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline.

New IQ_K quantization types (IQ2_K through IQ6_K) are proposed for llama.cpp, showing substantial KLD improvements over existing Q-quants at the same file sizes, though with slightly reduced performance and slower quant creation.

A community guide on running Qwen 3.8 Flash Next with a full FP8 n-gram table (PLE) instead of a heavily quantized one, using ple_fp8_pack.py to pack safetensors into a GGUF sidecar, yielding noticeably better output quality with no speed penalty.

研究人员提出 ORL（On-Policy Residual Learning）与自研推理引擎 dlmrt，使扩散语言模型在单张 RTX 5090 上达到 2,788 tokens/s，8×B200 上达 7,708 tokens/s，端到端吞吐提升最高 6.9 倍且质量不降，模型、训练代码与推理引擎全部开源。

Apple ML Research introduces RSI-Inference Bench, a benchmark and environments built from real-world vLLM and SGLang inference-engineering tasks to test whether AI agents can improve the systems that run AI, reporting a best-agent pass@1 of just 26.7% after spending $100K in API tokens and 50K GPU hours.

Zakaria 介绍 overshoot：一款优化 AI 团队推理栈的软件，号称可将 GPU 成本减半且无质量损失，用户累计已节省超 500 万美元。Jerry Liu 转发祝贺其发布。

该论文论证冻结的短上下文草稿模型无需重新训练即可用于超长上下文的推测解码，通过限制注意力缓存和重置旋转位置编码两种推理时机制，使其速度优势在长达两百万 token 的上下文中保持稳定。

RoLaQ提出了针对多头潜注意力（MLA）模型的2位KV缓存联合旋转与范围校准的离线量化方法，在DeepSeek-V2-Lite上实现3.96倍BF16 KV容量和2.51倍解码吞吐提升。

Drama3 模型现已支持在 audio.cpp 中运行，在 RTX 5090 上全速运行时最快可达实时速度的 5 倍，或在低于 8 GB 显存下以约 2 倍实时速度运行，仍需进一步优化与修正。

DLite提出一种无历史上下文的投机解码框架，通过复用目标模型的隐状态进行局部预测并省去历史KV缓存维护，在agentic工作负载下将接受长度提升至2-6，较SOTA方法DSpark提升最高70%。