triton

A developer shares ApexW, an experimental PyTorch optimizer using compact optimizer states and a Triton-based execution path, claiming ~1.5x faster optimizer steps and halved optimizer-state memory on large-tensor workloads — while openly noting regressions on many-small-tensor models and minimal end-to-end gains.

斯坦福 CS336（2026 春）第 6 讲的笔记，讲解如何通过减少 GPU 内存往返来优化内核：对比 softmax 多次遍历与融合行内核的内存流量，介绍 Triton 的块级编程模型，并强调先用 CUDA event 测量基线、再做内核融合与分块优化。

NVIDIA recsys-examples now offers an end-to-end HSTU generative recommender inference workflow on Dynamo-Triton, combining PyTorch AOTI compilation with FlexKV-backed KV caching to achieve up to 5.93x lower latency on an RTX PRO 6000 Blackwell GPU.

Kaiming Cheng of Meta will present KernelAgent, a hardware-guided multi-agent harness that optimizes GPU Triton kernels, at PyTorch Conference North America 2026; it achieved a 2.02x speedup over prior versions and 1.56x over default torch.compile across 100 KernelBench L1 tasks.

PyTorch/Meta blog reveals Jagged Flash Attention built with TLX (Triton Low-level Extensions) on NVIDIA Blackwell B200, using ~3.2K lines of Triton code—3x less than FlashAttention-4—while outperforming FA4 by ~13% forward and ~50% backward on jagged attention shapes used in Meta's Generative Ads Model.

KernelZero introduces a co-evolution framework with Proposer and Coder models to enhance GPU kernel generation, achieving superior performance on CUDA and Triton benchmarks.

The article announces PyTorch Conference North America 2026, where Andrey Talman from Meta will discuss modernizing the PyTorch release process, including coordination with Triton and vllm, and the use of AI agentic workflows.

The paper presents tritonBLAS, an analytical model for optimizing GPU GEMM kernel parameters without runtime autotuning, achieving near-optimal performance and significantly reducing compilation time.

The article introduces AMDKernelVault, an open dataset and training framework for AMD GPU kernel optimization, featuring large-scale HIP and Triton kernels and agent-driven pipelines for generating and validating kernels.

The article introduces ParaRNN, an open-source PyTorch/Triton library that speeds up training of non-linear RNNs using parallel Newton-Raphson scans.

A YouTube creator returns after a hiatus to teach building a distributed training framework from first principles, focusing on advanced AI topics like DeepSeek, MoE, and MLA, with an emphasis on developing problem-solving skills and self-confidence.

Meta's FBTriton is a downstream fork of OpenAI's Triton compiler that enables rapid development of GPU optimizations like TLX and autoWS while staying synced upstream. This blog details its continuous upstream ingestion strategy, hierarchical L1/L2/L3 validation framework, and the practical challenges of balancing innovation with production stability.

Presents MKEvolve, a modular multi-agent framework that iteratively co-evolves modular decomposition and LLM-generated kernels for hardware accelerators, achieving improved correctness and speedup over direct synthesis while reducing token usage.

SonicSampler presents a unified suite of tile-aware Triton kernels that vertically fuse the entire LLM sampling pipeline, supporting dynamic per-request behaviors and speculative verification, achieving up to 16x speedup over state-of-the-art baselines.

Technical post detailing how to run DeepSeek V4 Flash on two Nvidia 4090d GPUs using custom Triton kernels and vLLM, achieving ~105 tokens/second with 262k context.

The PyTorch-Triton 3.7 release introduces the Triton Plugin Extensions system, enabling dynamic loading of custom compiler passes and DSL extensions into upstream Triton without forking, with Meta's TLX now supported out of the box.

Elliot Arledge explains why he prefers using a Kimi Linear megakernel over Qwen 3.6 for kernel performance, comparing parameter counts, layer synchronization, hidden dimensions, and architecture-specific optimizations. The discussion highlights that Kimi Linear architecture is more suitable for megakernel implementation, especially for batch-1 decode on RTX PRO 6000 Blackwell.

A CMU PhD who developed the kernels now used by NVIDIA in TensorRT-LLM explains fast attention, covering fused CUDA kernels, FlashInfer, Triton, and paged-KV attention, enabling more tokens per second on the same GPU.

This post explains the Decompose-K technique for accelerating skinny large-K matrix multiplications by splitting the K dimension into chunks, running batched matmuls, and summing partials. It provides a PyTorch implementation and benchmarks showing significant speedups over standard torch.compile for bad-shaped matmuls.

This blog post continues the profiling in PyTorch series, exploring nn.Linear, MLP blocks, and fusion techniques using Triton kernels to optimize performance.