Tag
A new Cohere paper shows speculative decoding works well on fine-grained MoE models even at batch size 1, since adjacent tokens share 20%–45% of experts (vs 4%–12% expected), making verification far cheaper than prior work predicted across 10 MoE models tested on H100s.
Meta's new TextSeal2 paper shows that blending two secret keys into one virtual key plus noise makes hidden watermarks in AI-generated text easier to detect, even with speculative decoding, reaching 89% detection at 0.1% false-positive rate versus TextSeal's 71%.
TraceSpec is a framework linking LLM agent trajectory evaluation with adaptive draft training for speculative decoding, introducing a TraceSpec-Traj corpus and improving replay throughput by 12.5% to 22.6% over its DSpark base across multiple model sizes.
The paper compares continuous diffusion language models (LangFlow) and speculative drafters (DSpark) as standalone generators under matched training budgets, finding LangFlow superior on perplexity and MAUVE at longer sequence lengths while DSpark-RNN leads at length 16, highlighting iterative full-sequence refinement as a key advantage.
JetBrains released Mellum2.1-12B-A2.5B-Thinking-MTP, a 12B MoE (2.5B active) thinking coding model with a multi-token prediction head for speculative decoding in vLLM. The update focuses mainly on reinforcement learning post-training in real repository environments to improve agentic software engineering tasks.
Kapitsa 0.2.26, an open-source Apple Silicon inference engine, publishes M5 Max benchmarks for Qwen, Nemotron, Flint and Clover models, featuring its KFlash best-first tree verification and SSD offloading that runs a 180B MoE at 61 tok/s on a 48GB MacBook Pro.
Announcement recommending users upgrade from the MTP head to the DFlash 2 draft model for Qwen3.8-Flash-Next speculative decoding in llama.cpp, claiming ~17% average speedup, with Q4_0 quantization recommended as it matches Q8_0 acceptance/speed while using 287 MiB less memory. The card also details runtime memory trade-offs between the MTP head and the DFlash 2 drafter, noting the smaller on-disk drafter costs about the same at runtime due to its larger compute buffer.
TryMirai open-sourced uzu, an Apple Silicon inference engine, featuring Weaver + DFlash speculative decoding that makes a 9B LLM (Qwen3.5) 4x faster locally and 2.6x faster than standard speculative decoding.
Decagon 介绍了 AutoResearch 系统：由 AI agents 自动诊断推理瓶颈、在生产流量回放环境中验证优化（量化、调度、引擎补丁、投机解码等），报告端到端吞吐提升最高达 2.5x，且生产变更仍需人工审批。
A community post laying out feature requests for Google's upcoming Gemma 5, including looped transformers for better reasoning, DeepSeek's Engram memory, cheaper KV cache via Compressed Sparse Attention, a potential ~120B MoE variant, drafter improvements, and keeping Gemma well-rounded beyond coding benchmarks.
Magnitude is an open-source Rust LLM inference engine that auto-tunes GPU kernels on your local hardware, claiming 2.7x faster decode and 1.6x faster prefill than Ollama on a MacBook, with one-click integration into agents like Claude Code and Codex.
Princeton and Polymathic AI introduce FlashMap, a continuous flow drafter for speculative decoding that generates an entire token block in one forward pass while capturing inter-token dependencies, achieving up to 6.1× lossless acceleration on Qwen3-8B and outperforming the DFlash baseline on every benchmark.
该论文提出 Packed-SS，一种自投机解码调度方案，通过让扩散起草器在验证当前草稿时提前一个调用构建下一草稿，实现单次模型调用完成验证与起草，输入规模随投机宽度线性增长，并在 Nemotron 3B/8B 与 SetDLM 1.7B 上取得更高吞吐量。
NVIDIA 的这篇技术博客介绍了在 DGX Spark 上加速 Ling-3.0-flash MoE 模型推理的三种优化技术：Humming MXFP4 MoE 内核、LM head 的在线 FP8 量化，以及基于 NEXTN 的多 token 预测投机解码，并分析了组合配置下的集成问题。
Soft-MTP 提出一种 KL 梯度恢复技巧，去除多token预测训练中的噪声残差梯度，在无额外开销下改进困惑度、投机解码与推理能力。
该论文研究了扩散大语言模型作为投机解码草稿模型时的两个关键参数——扩散努力程度与接受准则宽松度——的联合变化，发现二者协同调整能够超越单步扩散草稿方法的速度-质量帕累托前沿。
SPECMEM提出了一种在部署期间持续维护的工具调用记忆系统，通过在线写回已验证的调用记录并配合LRU淘汰和副作用门控机制，在多会话多用户场景下实现最高1.74倍的墙钟加速，同时避免了不可逆操作的错误。
A pull request to llama.cpp refactors Intel XMX matrix engine support, delivering massive improvements to speculative decoding performance on Intel hardware.
对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎（basalt、capy-flash、ninfer-ext）进行全 262k 上下文基准测试，basalt 在所有上下文长度和两种解码模式下均领先（prefill 8.1k-9.2k tok/s，greedy decode 176-265 tok/s），解码排名与 MTP 草稿接受率一致。
A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.