inference-speedup

TreeGraft introduces a multi-drafter framework for tree-based speculative decoding, optimizing draft tree quality with adaptive scheduling to achieve significant inference speedups over single-drafter methods.

NVIDIA demonstrates how quantization-aware distillation (QAD) using NVIDIA Model Optimizer improves the Nemotron 3.5 Lightning model, reducing memory usage and increasing throughput while preserving accuracy for agentic benchmarks.

LiquidAI releases draft models for speculative decoding to accelerate their LFM2.5 models, achieving up to 2× faster inference on H100 and Apple silicon without quality degradation.

DFlash 2 is a block-diffusion draft model for speculative decoding that improves inference speed for the Qwen3.8-27B language model, offering higher acceptance length and throughput in benchmarks.

The paper introduces Mobius-v0, an architecture that decouples knowledge storage from reasoning to enhance efficiency, showing a 7B model achieves comparable performance with 62.6% training data and Intern-S2-Mobius delivers 4x inference speedup.

FVAttn is a training-free sparse attention system that uses runtime load balancing to improve distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism for video generation, achieving up to 4.41x attention speedup and 2.11x inference speedup over FlashAttention on step-distilled Wan2.2 I2V while maintaining competitive video quality.

Explores predicting which MoE experts will be used for the next token to improve CPU/GPU offloading, achieving 30->150-200 tg/s speedup, and questions the feasibility of implementation.

mlx-dspark brings DeepSeek's DSpark and z-lab's DFlash speculative decoding drafters to Apple Silicon via MLX, enabling lossless speedup (~1.4–1.6×, up to 2× on code/math) and an OpenAI-compatible API for local inference.

Ornith 35B shows 30-40% token generation speedup when paired with Qwen3.6 35B DFlash speculative model in llama-server, achieving 80% acceptance rate on mixed code and text, though prompt processing suffers.

The paper introduces the context-ready transformer, a recurrent architecture that pre-contextualizes tokens before the transformer block, achieving significant inference speedups (e.g., 1.7x on A100) while matching or exceeding standard transformer performance with fewer layers.

MeshFlow introduces an equivariant optimal-transport flow matching model for direct triangle mesh generation, achieving state-of-the-art quality while providing approximately 18x inference speedup over autoregressive methods.

This paper proposes dMoE, a block-level mixture-of-experts framework for diffusion large language models that aggregates token-level expert distributions into block-level routing, reducing activated experts and memory usage while maintaining performance.

Domino is a speculative decoding framework that decouples causal dependency modeling from autoregressive drafting, using a parallel backbone and lightweight causal refinement head to achieve up to 5.49× end-to-end speedup on Qwen3 models.

SEATS is a training-free, stage-adaptive token selection method that reduces computational overhead in omni-modal LLMs by progressively pruning redundant visual and audio tokens, achieving a 9.3x FLOPs reduction and 4.8x prefill speedup while preserving 96.3% performance.

Graft is a training-free framework that enhances speculative decoding by combining pruning and retrieval to improve acceptance rates and inference speed, achieving up to 5.41x speedup on short-context benchmarks and up to 21.8% improvement over EAGLE-3 on Qwen3-235B.

ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.

This paper introduces FeF-DLLM, a discrete diffusion language model that eliminates factorization errors by using exact prefix-conditioned factorization and accelerates inference via speculative decoding, achieving significant improvements in accuracy and speed on benchmarks such as GSM8K and MATH.

Orthrus is a dual-architecture framework that combines autoregressive LLMs with diffusion models for fast parallel token generation while maintaining exact inference fidelity via shared KV caches and consensus mechanisms, achieving up to 7.8x speedup.

DFlash introduces a block diffusion method for flash speculative decoding to enhance inference speed in large language models.