Tag
mlx-dspark v0.10.0 adds support for Qwen3.8-27B on Apple Silicon, providing up to 3x faster inference through speculative decoding with lossless verification.
NInfer adds day-0 support for the Qwen3.8-27B model, achieving around 200 tokens per second on a single RTX 5090 with speculative decoding, and includes engine improvements like concurrent requests and kernel optimizations.
FlashDrive is an algorithm-system co-design framework that cuts the inference latency of vision-language-action models for autonomous driving by 4.7× (from 717 ms to 151 ms on a single GPU) using streaming KV-cache reuse, non-autoregressive diffusion drafting, and adaptive step caching, with negligible accuracy loss.
This paper introduces Decoupled Contrastive Decoding (DCD), which uses an expert-aligned lightweight proposer for speculative decoding while keeping the contrastive signal only in verification, achieving speedups over vanilla contrastive decoding without degrading output distribution.
An experiment running the Qwen3.8-2.4T-A95B MoE model locally on dual consumer GPUs (RTX 5090 + 5060 Ti) with llama.cpp, achieving ~0.8 tok/s with MTP speculative decoding enabled.
Qwen3.8-2.4T-A95B by Alibaba Qwen and Alibaba Cloud is now available on Modal, served with a custom DFlash speculator trained on tool-call-heavy data and a full 1M context window.
A developer reports running Meta's Muse Glimmer 30B up to ~3.3x faster on Apple Silicon using speculative decoding in mlx-dspark, with byte-identical output and no quality tradeoff.
Presents LibraSpec, a training-free, plug-and-play algorithm that dynamically selects speculative decoding lengths via marginal-gain-driven optimization, achieving consistent speedups across multiple models and benchmarks.
Benchmarks unsloth's Muse Glimmer 30B on an RTX 5090 with speculative decoding, achieving up to 253 t/s using a DFlash draft model and a GPU-based argmax PR, though the PR is still a draft.
Meta releases Muse Glimmer, a 30B open-weight multimodal model optimized for local agent workflows, with permissive Apache 2.0 licensing, 4-bit quantization support, speculative decoding, and broad ecosystem integrations.
A discussion on why speculative decoding matured in 2026 for LLM inference, citing Uber's early use, Apple and DeepMind papers, and Tri Dao's research, with observations on adoption in frameworks and local deployments.
This post describes two configuration flags that increase the official Ling-3.0-flash INT4 inference speed from 20.8 to 38.7 tok/s on a single DGX Spark, while warning about the need for a specific vLLM fork and noting tradeoffs with long-context performance.
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
A developer is building Octane, a speculative decoding runtime for local LLM inference on consumer hardware, aiming for 2-3x speedup with exact output quality. Currently in active development with paged KV cache, continuous batching, and batched attention implemented.
This preprint introduces DBLast, a dependent block drafter for stochastic speculative decoding, using a low-rank latent mixture over token positions and an acceptance-oriented training objective to improve accepted draft length in higher-entropy decoding regimes. Experiments with Qwen3-4B and Qwen3-8B show consistent improvements over independent block sampling.
LG AI Research presents K-EXAONE 2.0, a 750B-parameter MoE foundation model upcycled from K-EXAONE, supporting 256K context and six languages, with self-speculative decoding for efficient inference.
Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.
AcceptMoE is a verifier-side expert selection method for speculative decoding of MoE LLMs, which reduces expert-weight traffic and improves throughput by up to 2.06x under expert offloading with minimal accuracy loss.
This paper proposes speculative correction, a training-free draft-then-refine decoding strategy for diffusion language models, showing quality-latency improvements using LLaDA2.1 models.
Technical write-up on running DeepSeek-V4-Flash-0731 with full 1M context on a single RTX 5090 + 256GB DDR5 desktop using vLLM with CPU/RAM offloading, achieving ~800 tps prefill and ~15 tps decode.