@zhaoran_wang: for me, the coolest finding is that you can connect/interpolate all softmax/linear variants and give a promising direct…

X AI KOLs Timeline Papers
attention softmax linear-attention muon-optimizer parallax research

Summary

Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.

for me, the coolest finding is that you can connect/interpolate all softmax/linear variants and give a promising direction - affine-linear : )
Original Article
View Cached Full Text

Cached at: 05/30/26, 10:33 AM

for me, the coolest finding is that you can connect/interpolate all softmax/linear variants and give a promising direction - affine-linear : )

Yifei Zuo (@YifeiZuoX): For me, the coolest finding is that Muon optimizer is crucial for Parallax to move beyond Softmax Attention.

Lesson — don’t evaluate new architectures solely under AdamW, you’ll miss the good ones.

paper: https://t.co/fMY17lRQtn code: https://t.co/LVBjxCiVVW

For the origin of

Similar Articles

@Phoenixyin13: I think this is a top-notch work in ICML 2026. The attention mechanism of traditional Transformers is essentially point-to-point matching: it cuts input into a bunch of tokens (discrete points), computes similarity between Query and Key, and then weights the Value. In NLP...

X AI KOLs Timeline

Introduces the ICML 2026 paper Functional Attention, which treats functions as first-class citizens and replaces softmax point-to-point similarity with structured linear operators. It addresses issues of discretization, resolution sensitivity, and high computational complexity in traditional Transformers when handling continuous functions. Achieves or surpasses SOTA in tasks like PDE solving and 3D segmentation, and exhibits strong OOD generalization.

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

arXiv cs.LG

This paper compares softmax attention with four linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) and introduces cross-layer routing mechanisms. Experiments at 350M parameters show Kimi Delta Attention with Muon achieves lowest validation loss, while pure Gated DeltaNet with AdamW has highest throughput.