speedup

SpecLA proposes a speculative decoding runtime tailored for stateful linear-attention models, achieving up to 1.70x end-to-end speedup over autoregressive decoding on an NVIDIA H100 with a GDN-1.3B target.

The FastVideo team releases new FP4 attention kernels for B300, achieving up to 1.69x speedup over FlashAttention 4.

Unsloth releases quantized Qwen3.6 models using NVFP4 format, achieving 2.5x faster inference speeds.

Two Hong Kong students achieved a 5x speedup by adding another loop outside the original automated research framework, without needing a better model or more compute. It is considered one of the most useful papers for ordinary Agent developers.

Blog post by Sankalp detailing how he used Codex to achieve a 232x faster GPU kernel for QR decomposition in GPU Mode's contest, outlining his auto-research methodology.

A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.

Fable achieves top rank on KernelBench-Mega by writing a highly efficient CUDA megakernel with an 18.71X speedup, signaling progress toward recursive self-improvement in AI R&D.

The author used Codex to optimize DeepSeek V4 Flash 8-bit MLX on oMLX, achieving approximately 1.6x prefill and 3x decode speedup.

Ollama announces that Gemma 4 is now nearly 90% faster on Apple Silicon using MLX, thanks to improved multi-token prediction enabled by default, with automatic tuning to avoid slowdown.

DeepSeek released a paper and MIT-licensed open-source implementation of speculative decoding (DSpark) that speeds up LLM responses by up to 80% by using a small 'guess' model and a large 'check' model, achieving both speed and accuracy without tradeoffs.

JetSpec introduces parallel tree drafting for speculative decoding, achieving up to 9.64x end-to-end speedup on LLM inference while maintaining lossless accuracy, with throughput reaching ~1000 TPS on a single B200 GPU.

Nvidia claims a 15x speedup in text generation using a diffusion model, generating entire blocks at once.

GLM 5.2 delivers major performance gains on Mac Studio with 512GB RAM, achieving prefill speeds above 100 t/s at high context lengths and enabling 4-bit quantization for contexts over 100k tokens, as detailed in a pull request by the oMLX creator.

Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks, achieving 33x speedup over cuML and 200x over FAISS by eliminating redundant memory reads and writes.

Anima Anandkumar highlights that neural operators, despite simple benchmarks, have achieved massive speedups (10,000–million times) in hard real-world problems like high-resolution AI weather modeling (FourCastNet) and nuclear fusion turbulence, referencing a new paper showing learned solvers become more cost-effective as PDE tasks get harder.

This paper accelerates the NeurASP neurosymbolic AI framework by implementing vectorization, batch processing, and caching, achieving multiple orders of magnitude speedup on larger tasks.

A developer benchmarks Gemma 4 E4B using Google's LiteRT engine against a Q4 GGUF quant, finding ~2.4x speedup in text generation due to multi-token prediction (MTP), but only 1.1x in image captioning. The post provides a Python wrapper for an OpenAI-compatible endpoint, though with limitations like deterministic output and single-session engine.

Atomic Chat's MTP technique speeds up Qwen dense models by 2.5x and MoE models by 25% on 2x RTX 5090 with zero accuracy loss and ~1 GB extra VRAM, using speculative decoding to draft and verify multiple tokens in one pass.

A fork of llama.cpp fixes the --split-mode tensor issue with quantized KV caches, achieving up to 40% speed improvement on dual GPU setups without quality loss.

NousResearch releases Lighthouse Attention, a selection-based hierarchical attention that achieves 1.4-1.7x wall-clock speedup at 98K context and ~17x faster forward/backward pass than standard attention at 512K context on a single B200, validated on 530M-parameter Llama-3 models across 50B tokens.