Tag
This paper proposes a hardware-software co-design framework for N:M sparse vision Transformer inference, achieving over 2.2× latency speedup on GPUs while maintaining accuracy through a novel CUDA kernel (MD-SpMM) and a deployment-aware sparsity search.
Proposes Progressive Tree Drafting (PTD), a training-free, model-agnostic speculative decoding method that uses progressive tree structures and stepwise pruning to enable parallel generation of multiple draft paths, achieving up to 2x speedup on various benchmarks.
This paper introduces a self-supervised early exit method for LLMs, allowing computation to stop early at intermediate layers when confidence is high, thereby reducing inference cost. It also presents Dynamic Self-Speculative Decoding (DSSD) which achieves higher token acceptance than existing baselines.
A new implementation of the fastest speculative decoding for Qwen models is available, with support in sglang via a Hugging Face fork and upcoming support in the uzu engine.
Novita releases Kimi-K2.6 DSpark, a speculator model that uses speculative decoding to accelerate inference of the base Kimi-K2.6 model, achieving higher accepted token lengths.
HunyuanOCR-1.5 is a lightweight end-to-end OCR vision-language model that improves efficiency via DFlash (6.37x inference speedup) and capability via Agentic Data Flow, achieving top-tier performance on document parsing, OCR, and multilingual tasks.
DSpark is a speculative decoding framework that combines semi-autoregressive draft generation with confidence-scheduled verification to accelerate LLM inference and improve throughput in high-concurrency settings.
Introduces Flex-Forcing, a unified training and inference framework that allows video diffusion models to operate under both bidirectional and autoregressive regimes via a flexible chunking mechanism over temporal and denoising steps, achieving better video quality, long-video stability, and faster inference.
Introduces Multi-Token Residual Prediction (MRP), a technique that accelerates diffusion language model inference by predicting residuals between adjacent denoising steps, achieving up to 1.56x speedup in SGLang and recovering up to +16 accuracy points in aggressive decoding settings.
A multi-agent system with over 100 agents collaborates for a week to accelerate Gemma 4 on vLLM by 5x, demonstrating emergent self-organized communication and resource pooling.
This paper proposes Dynamic-dLLM, a training-free framework that accelerates diffusion large language models by dynamically allocating cache-update budgets and calibrating decoding thresholds, achieving over 3x speedup on models like LLaDA and Dream while maintaining performance.
详细介绍了针对语音克隆模型的W4A4 CUDA内核优化，通过INT4量化和融合LoRA，实现了比FP16快2.6倍的推理速度。
On Friday, we released six new state-of-the-art drafters for accelerated inference, along with a blog post on speculative decoding and a roofline model tool to estimate speedups.
Charles Frye announces the co-release with Z Lab of six new DFlash speculators for Alibaba Qwen 3.x models, achieving over 1k output tokens per second for Qwen 3.5 122B-A10B on a B200.
Modal and Z Lab release six new DFlash speculative decoding draft models for Qwen 3.x, achieving over 1000 tokens per second on a B200 and arguing that speculative decoding is the most impactful inference optimization.
GLM-5.2 adopts MTP (Multi-Token Prediction) technology to accelerate inference and fixes a training-inference discrepancy in GLM-5.1's MTP that caused KV cache mixing issues.
Microsoft Research introduces Next-Latent Prediction (NextLat), a self-supervised method that trains transformers to predict their own next latent state, enabling compact world models for reasoning and planning and achieving up to 3.3x faster inference via self-speculative decoding.
Tensordyne announced a breakthrough inference system using logarithmic math in hardware, claiming 17x more tokens per watt and 13x higher throughput than NVIDIA Blackwell, achieved by replacing complex multiplication with simple addition in log space.
DFlash, a block-diffusion drafter with KV injection, is now running at frontier scale, achieving up to 4.3x greater throughput over baseline, integrated with Modal and SGLang for Qwen 397B.
This paper proposes Prefilling-dLLM, a training-free framework that partitions the prefix into chunks and caches KV representations, achieving state-of-the-art quality and up to 28x speedup for long-context inference in diffusion language models.