softmax

This paper proposes Jacobian-guided noise injection, a training strategy that improves quantization robustness in large language models by injecting noise into pre-attention logits with variance derived from the Jacobian norm, leading to significant performance gains in low-bit quantization settings.

A blog post discussing optimization techniques for constrained categorical probability distributions, using softmax reparameterization and log barrier methods, applied to protein binder design.

Explains that super weights in large language models arise from the SoftMax-Attention interaction creating a 'Nothing Dump' token that serves as a stable reference point; removing these weights cripples performance.

An educational overview of knowledge distillation, covering its history, core concepts like softmax and temperature, types, scaling laws, and practical examples including DeepSeek-R1.

A tweet sharing a naive CUDA softmax implementation using shared memory reduction, noting that reduction is straightforward.

This paper analyzes precision loss in FP8 attention due to the attention sink phenomenon when casting the softmax output to FP8 (E4M3). It shows that forward KV iteration causes underflow of non-sink attention values, and proposes reverse iteration and a static scaling factor S=256 to eliminate underflow, achieving 3-10x MSE improvement.

The paper identifies a misalignment between the softmax-based InfoNCE loss and the normalized embedding setting in modern contrastive learning. It proposes WEINCE, a simple modification that blends softmax logits with an endpoint shortfall correction using extreme value theory, yielding consistent improvements across vision benchmarks.

Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.