Tag
A user reports that quantising the f16 MTP draft model to Q4_K for Gemma 4 31b gives roughly 10% faster decode (65 to 72 TPS) on dual 3090s compared to the default Q4_0, while Q2_K performs worse.
Unsloth AI announces DSpark, enabling DeepSeek-V4-Flash GGUF models to run ~1.4–2× faster locally, reaching 120 tokens/s with no accuracy change.
Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.
A discussion questioning whether extremely high decode speeds (1k-10k tok/s) for large models like Qwen 3.5 397B or GLM-5.2 would unlock new use cases or be better spent on loading larger models.
Google Gemma announces that developers can now use the Gemma 4 31B model as the brain for voice AI, enabled by Hugging Face and Cerebras for ultra-fast inference, as part of an open-source cascaded speech-to-speech stack.
Spatially Speculative Decoding (SSD) accelerates autoregressive image models by predicting entire rows in parallel using small helper networks, achieving up to 13.28x speedup while maintaining benchmark performance.
NVIDIA researchers introduced SparDA, a transformer variant that adds a fourth projection (Forecast) to predict next-layer KV blocks, enabling prefetching from CPU memory and reducing selection cost, achieving 1.7x faster decoding and 6.5 point accuracy gain on long reasoning.
NVIDIA's Puzzle-75B-A9B model achieves 132 tokens per second using NVFP4 quantization on three RTX 3090 GPUs, raising discussion about the lack of competition in this model size category.
A user achieved a 29x speedup for DeepSeek V4 Flash inference on 4x RTX 3090 GPUs by optimizing llama.cpp, reducing a 23k prompt from 25 minutes to 53 seconds.
A user benchmarks the newly merged DFlash speculative decoding method in llama.cpp on Qwen 3.6 27B, achieving up to 4.44x speedup at 36K context compared to baseline, with detailed leaderboard and quality tests.
NVIDIA proposes a method to convert any LLM into a faster one by splitting it into two copies: one frozen for context, the other trained to generate multiple tokens in parallel, achieving 2.4x speedup with ~99% quality retention using only 8% of training data.
A user considers buying four Ascend GX10s to run GLM5.2, citing performance numbers like 400-500 tok/s prompt processing and ~15 tok/s output at 128k context, and plans for future open-source models.
Nvidia's optimized Qwen 3.6 27B NVFP4 model achieves 41% faster single-session inference and 23-25% faster concurrent inference on DGX Spark compared to Unsloth's version.
FlexiSLM introduces dynamic frame rate capabilities for speech input and output in spoken language models, outperforming fixed-frame-rate models and enabling controllable inference speed.
Sam Altman announces that a model offering 750 tokens per second will be available for 5.6 SOL in July.
A seed-stage YC startup, Wafer AI, is competing in the inference speed race, with Databricks achieving 392 token/s on GLM-5.2, topping Artificial Analysis.
An experiment with over 100 AI agents collaborating for a week to improve Gemma 4 inference speed in vLLM achieved a 5x speedup, revealing emergent behaviors like self-policing, division of labor, and communal knowledge sharing.
Geometric Action Model repurposes a geometric foundation model for robot policy learning, achieving 85.5% on LIBERO-Plus with 6.9 ms inference, 55× faster than baselines.
MiniMax Sparse Attention (MSA) achieves up to 28.4x reduction in attention compute at 1M tokens by adding a routing branch that selectively chooses key-value blocks for attention, enabling 14.2x faster prefill and 7.6x faster decoding on H800 GPUs while matching full attention benchmark performance.
A collaboration between Modal, SGLang, and Z Lab integrates DFlash speculation into SGLang, achieving up to 4.3x throughput improvement for Alibaba's Qwen 397B-A17B model, advancing open intelligence.