Tag
A new Cohere paper shows speculative decoding works well on fine-grained MoE models even at batch size 1, since adjacent tokens share 20%–45% of experts (vs 4%–12% expected), making verification far cheaper than prior work predicted across 10 MoE models tested on H100s.
A discussion of NVIDIA's TriAttention research on KV cache compression in vLLM, arguing that real gains come from freeing GPU memory blocks rather than evicting tokens, with reported 2.5× faster decoding and 10.7× less KV cache memory.
GAAC is a lossless KV cache compression and serving system that exploits BF16 exponent redundancy with group-adaptive codes and a fused attention kernel, achieving ~1.4x KV compression and 1.04-1.27x speedups on NVIDIA DGX Spark and Apple M4 without approximating KV values.
SVPino 介绍了一款名为 Mischief 1 的图像到视频生成模型，据称比其开源基础模型快 5 倍、推理成本低 80%，非常适合低成本快速迭代创意。
TKintegers 推出基于平衡三进制权重表示 K5 的转换器与 GPU 推理实现，在 NVIDIA L4 上对 Qwen2.5-0.5B-Instruct 实现相对 TorchAO FP8 基线约 3.21× prefill 与 3.10× decode 吞吐提升。
文章清晰解释了前缀感知的 LLM 路由机制：通过按块哈希提示词并维护块到 worker 的索引，让请求路由到持有共享前缀 KV 缓存的 GPU，实测在 8 个 vLLM Pod（16 张 H100）上将 p90 首 token 延迟从 92.5 秒降至 0.54 秒。
llama.cpp has submitted a performance PR for the small Gemma 4 models E2B and E4B that makes layers reusing KV across layers compute only the last token during prefill. On an M4 Pro (Metal), this increases E2B’s prompt processing speed from about 1614 tok/s to 4603 tok/s (roughly 2.9×), while keeping the output identical.
Basalt (a fork of Strata) lands new PRs setting a new benchmark record with IQ3_S Qwen3.8 Flash Next on a single RTX 5080 with 64GB DDR5, reaching 5500 tok/s prompt processing and up to 275 tok/s peak generation during a live opencode coding session.
一篇技术博客深入解析 Meta 于 10 月 15 日发布的开源模型 Muse Spark Open（816B 总参数、42B 激活参数的 MoE 架构），并介绍 Daniel Gross 发起的 10 万美元 Open Kernel Challenge，邀请开发者在 H200 上优化其推理内核与 SGLang 服务性能。
Decagon 介绍了 AutoResearch 系统：由 AI agents 自动诊断推理瓶颈、在生产流量回放环境中验证优化（量化、调度、引擎补丁、投机解码等），报告端到端吞吐提升最高达 2.5x，且生产变更仍需人工审批。
A hobbyist details two days of tuning a single Tesla P40 24GB (PCIe x1) to run the 35B-A3B Ornith 1.5 MoE model at full 262k context, achieving ~60 t/s short-prompt and ~30 t/s at 100k depth without dropping below Q4 weights or q8 KV, via ECC disabling, int8 dp4a attention, and a trained MTP draft head check.
WAKMan is a workload-aware KV cache manager for LLM servers that automatically selects its eviction policy from live traffic, improving hit rate over vLLM's LRU by 5 percentage points on average across mixed single-turn, multi-turn, and agentic request workloads.
NVIDIA 的这篇技术博客介绍了在 DGX Spark 上加速 Ling-3.0-flash MoE 模型推理的三种优化技术：Humming MXFP4 MoE 内核、LM head 的在线 FP8 量化，以及基于 NEXTN 的多 token 预测投机解码，并分析了组合配置下的集成问题。
TAO 提出一个工具感知的联合调度框架，用于基于 LLM 的智能体服务，通过动态规划在容量约束下选择和放置会话，相比最优基线最高降低 26.8% 延迟、36.3% TTFT 并提升 29.5% 解码吞吐。
The paper shows that MoE channel selectors need no offline calibration data when using top-k selection instead of calibrated thresholds, and introduces a weight-derived reader (ternary sketch + low-rank map) that retains 98.1% accuracy at 17.8% of routed-expert reads while speeding up MoE blocks 1.60-2.69x over optimized vLLM.
Google Gemma 声称利用 Gemini 4 Argon 在 A100 上通过 vLLM 实现了 2.85 倍推理加速，gemma-4-E2B-it 吞吐量从约 37,900 提升至 108,000+ tokens/sec，且保持 100% token parity、无质量下降。
The discussion points out that DeepSeek's open-source DeepInfer is an inference framework deeply customized for DeepSeek 4.1F. Its KV cache is extremely optimized at just 890 bytes per token, meaning task scheduling never needs to account for VRAM capacity. In contrast, the Kimi series models have KV cache overhead 15–30x higher, resulting in poor inference performance and user experience.
An MIT Technology Review sponsored article arguing that enterprise AI's competitive advantage now lies in the serving harness — architecture, memory management, scheduling, and cost optimization — rather than in model choice itself. It urges leaders to design production systems around workload constraints like latency, concurrency, and GPU utilization.
A pull request to llama.cpp refactors Intel XMX matrix engine support, delivering massive improvements to speculative decoding performance on Intel hardware.
A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.