Tag
A CUDA megakernel engine for Qwen3.8-27B achieves 1.4-1.9x faster inference than llama.cpp on a single RTX 3090 (140 tok/s for code generation) with no measurable accuracy loss, verified via KL divergence and perplexity benchmarks against the reference implementation.
提醒用户尝试 llama.cpp 中新合并的概率式 MTP（多 token 预测）功能，在散文生成场景下解码速度提升约 14%，最优 draft-n-max / draft-p-min 参数与贪心采样基本一致。
该论文研究无分词器的字节级语言模型，发现配合 token-superposition 训练与哈希嵌入后，扁平 Transformer 随规模扩大能持续超越子词模型，并能自发学习出类似分词器的局部文本抽象，据此可将投机解码的接受率提升 3.4 倍。
A user reports strong single-request decode speeds running Qwen3.8-Flash-Next in Unsloth's UD-Q4_K_XL quant on Strata with an RTX PRO 6000, showing roughly 2x faster throughput than vLLM on the same GPU across prose, counting, coding, and reasoning tasks.
Benchmark thread showing Uzu, an open-source inference engine from Mirai, running Qwen3.5-9B at 92.1 tokens/s on a 16GB M5 MacBook Pro — about 4x faster than llama.cpp (22 tok/s) and MLX (25.1 tok/s) — by using tree-based speculative decoding with a Weaver drafter to break the memory-bandwidth ceiling of ordinary local decoding.
LemonSeed Studio 0.5.8 brings on-device LLM inference to iPadOS via an eGPU (AMD R9700 in a Thunderbolt enclosure), running Qwen3.8-27B Q4 at ~159 tok/s with DFlash2 speculative decoding across iPadOS, macOS and Linux using the unified LemonSeed Engine.
一篇技术博客深入解释了投机解码为什么快：并非因为总FLOPs减少（实际上反而增加），而是因为它改变了GPU的工作类型——在低批量场景下把原本空闲的算力从内存瓶颈（memory-bound）转为计算瓶颈（compute-bound），用单序列的'深度'换取了批量解码的'广度'。
文章介绍开源推理引擎 Uzu（Mirai 出品），通过最多提前草拟 16 个 token 的推测解码与为 Apple M5 优化的 kernel，在 M5 芯片上运行 4-bit Qwen3.5 9B 达到 92-117 tokens/s，远超 llama.cpp 与 MLX 的 22-25 tokens/s，并解释了 batch-one 解码的内存带宽瓶颈原理。
Lithos 开源了 lithos-metal，一个为 Apple Silicon 生成 Metal megakernels 的推理引擎，配合 DSpark 投机解码可在单台 M5 Max 上以每用户 200+ tokens/s 的峰值速度运行 Qwen3.8-27B，支持一行命令接入各类 coding agent。
This paper proposes Trajectory-Local Adaptive Retrieval (TLAR), a speculative decoding method that retrieves matched continuations from the model's own generation history and combines them with model-generated drafts in a shared candidate tree, showing improved token acceptance and end-to-end throughput without changing the target model's output distribution.
DScout 通过将扩散草稿器的训练与推理分解为前缀接受度和延迟权衡，提升了推测解码的吞吐量，在 Qwen3-4B 和 Qwen3-8B 上分别实现 3.84 倍和 4.14 倍的平均加速。
Hugging Face's Merve Huckan shares slides/materials from her dev conference talk on the local AI ecosystem, covering llama.cpp, prefill vs decode, memory types, and speculative decoding. She invites attribution when using the resources.
Georgi Gerganov recommends upgrading to DFlash speculative decoding for faster Qwen3.8-27B inference in llama.cpp, requiring the latest llama.cpp v0.6.0 release with the ggml-org/Qwen3.8-27B-GGUF model.
A cafe-llama.cpp update unlocks Qwen's Multi-Token Prediction potential for speculative decoding, roughly doubling decoding speed (~25 to 50+ tok/s on Qwen3.8-Flash-Next, ~40 to 100 tok/s on Qwen3.6-35B Q8) with just the --spec-draft-n-max 4 flag.
SpecFold 论文提出一种算法-系统协同设计，通过识别并折叠多分支投机验证中的计算冗余（token级残差门控与折叠注意力/FFN复用），在扩散大语言模型上实现最高1.64x吞吐提升（相比Spiffy），且保持任务性能相当。
DLoop 提出一种循环式投机解码方法，在验证前自适应地进行多次起草阶段，通过循环感知训练保持草稿模型可靠性，在 EAGLE-3、DFlash 等方法上实现 5% 到 41% 的墙钟加速且保持无损解码。
DFlash 2, a lightweight block-diffusion model for flash speculative decoding from z-lab/haoailab, enabling efficient high-quality parallel drafting across a wide range of open LLM families with checkpoints released on Hugging Face.
llama.cpp v0.6.0 获得正式发布，新增对 Qwen4Exp 的 MTP（Multi-Token Prediction）推测解码支持以及大量其他改进。
A hands-on benchmark comparing four local LLM inference engines (oMLX 0.7.0, Rapid-MLX 0.15.5, Splash 1.2.1, MTPLX 2.12.2) on a 36 GB M3 Max MacBook Pro, finding Rapid-MLX fastest on Qwen3.6-35B-A3B (~110 tok/s) and Splash fastest on dense Qwen3.8-27B (~31.7 tok/s via speculative decoding).
This paper measures the per-token 'sufficient compute' of LLMs using a Mixture-of-Agents panel of fifteen models across Qwen, OLMo, and R1-distilled families, finding a 0.5B agent reproduces 92-95% of reference tokens while the costliest 10% of tokens account for 64-80% of FLOPs. The resulting per-token compute map improves model routing latency from 7.59s to 5.12s and enables speculative drafting with 32.6% fewer draft tokens than fixed-window baselines.