Tag
HyScale 提出了一种面向长上下文训练的学习型稀疏注意力系统，通过 HyTopK（免分数表的精确 top-k 选择）和选择重放（selection replay）技术，在 1M token 序列上将 GLM-5.1 训练速度提升至 Megatron-LM 的 2.07 倍。
在 PyTorch Con 2026 上，Mark Saroufim 探讨了支撑更优优化器的线性代数内核，涵盖 QR、对称特征分解与 Cholesky，并从 GPU MODE 的线性代数竞赛中总结了如何通过暴露并行性、重叠独立阶段和合并更新来减少数据搬运。
Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎（MIT 开源），通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache，使 262K 长上下文能在 16 GB GPU 预算内运行，并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。
A team shares how they built a production voice-agent stack using self-hosted Pipecat, Mai Transcribe 2, a Sarvam conversational LLM, and a Malayalam fine-tuned VoxCPM2 TTS, achieving 20 concurrent low-latency voice streams on a single L4 GPU with an automated WhatsApp post-call handoff.
llama.cpp の CUDA バックエンドにおいて、低ビット型（low-bit types）向けの MMVQ（マトリックス・ベクトル量子化積）演算を高速化する GitHub プルリクエスト (#30517) が公開された。量子化されたモデルの GPU 推論性能の改善を目指すもの。
作者将 adaptive KV streaming 从 CUDA 移植到 ROCm/HIP，并结合 RDNA 4 内核优化，使 Qwen3-27B (IQ4_XS) 能在单张 16GB RX 9070 XT 上以完整 262k 上下文运行，MTP 解码达到 80+ t/s。项目包含大量内核级优化与功耗调优结果，已发布到 GitHub 分支。
UC San Diego 与 Amazon Annapurna Labs 提出 'How to Train Your Little Dragon' 训练方法，通过监督微调与强化学习使一个 8B 小模型学会构建和改进 GPU 优化脚本，在正确率和内核性能上超越多个万亿参数级前沿 LLM。
Basalt, a Blackwell-tuned inference engine forked from Strata for Qwen 3.8 Flash Next, introduces dynamic vision encoding that keeps the image encoder in RAM when idle to save VRAM, uploading it only when an image arrives with near-zero text-inference overhead. The team also shared commit hashes so developers can port the technique to their own engines.
A CUDA megakernel engine for Qwen3.8-27B achieves 1.4-1.9x faster inference than llama.cpp on a single RTX 3090 (140 tok/s for code generation) with no measurable accuracy loss, verified via KL divergence and perplexity benchmarks against the reference implementation.
A developer shares performance numbers and optimization ideas for streaming Qwen Flash Next Q4 MoE experts from SSD on a 64 GB Mac mini M5, achieving 17.5 tok/s decode with the GPU idle 27% of the time waiting on expert loads, and asks for ideas to improve expert prefetch accuracy and hit rates.
DeepSeek released DeepGEMM, an open-source GPU math engine used to train and run its own models, claiming up to 98% token cost savings by squeezing more tokens per second out of existing GPUs.
A community developer released Ninfer 4080, an optimized local inference runtime that runs the Qwen 3.8 27B GSQ model at 100k context on a 16GB RTX 4080, achieving up to ~2720 tok/s prefill and 262 tok/s decode via DFlash2 speculative decoding — significantly faster than general-purpose engines like llama.cpp or vLLM.
The team behind .wave introduces an inference engine using a Wave Persistent Kernel compiler to serve NVIDIA Nemotron 3.5 ASR Streaming at $0.00045/minute, sustaining 4,800 concurrent streams on one H100 with p99 frame completion under 106.3 ms — roughly 20x NVIDIA's published stream capacity and a fraction of competing ASR pricing.
A tweet highlights a free ~5% tokens-per-second performance boost on Blackwell GPUs by enabling CustomAllReduce on SM120 when using tensor parallelism 4, joking that closed labs hate this trick.
A llama.cpp pull request (#29184) by am17an fuses shared experts into the MMVQ (matrix-matrix vector quantization) CUDA kernel, optimizing inference for mixture-of-experts models on GPUs.
GEM-KMeans introduces a spectrally normalized, memory-efficient NLR formulation for K-means clustering that fuses updates into a single matrix-multiplication epilogue, reducing GPU HBM storage to one factor while maintaining clustering accuracy at massive scale.
The author raises concerns about the common practice of creating hard forks of llamacpp optimized for specific architectures without upstream contributions, questioning the reasons and noting a disconnect from traditional open-source norms.
A user shares kernel optimizations for Tesla P100 GPUs to improve performance when running the Qwen 3.8 model with llama.cpp, achieving significant speedups in inference.
The article introduces 1Cat-vLLM, a fork of vLLM optimized for NVIDIA V100 GPUs, and compares its performance with llama.cpp for serving large language models like Qwen3.6-35b.
Augment Code switched its coding-agent backend to Stefano Ermon's Mercury 2.5 diffusion model, achieving 82% latency reduction and 90% cost cut in production. The article highlights the performance advantages of diffusion models and the need for independent AI benchmarking tools.