Tag
A detailed mathematical explainer of the softmax function, its probabilistic interpretation in multiclass classification, and a full derivation of its Jacobian/derivative using the quotient rule, including the gradient of a softmax-based loss.
The paper investigates quantizing softmax in transformer attention during pretraining, comparing calibration and rounding strategies to reduce validation loss gaps. Experiments at 124M parameters show minimal loss degradation with specific techniques.
This paper analyzes the backward-pass geometry of Z-loss in AI model training, providing a framework for understanding and optimizing gradients in dense output heads and sparse mixture-of-experts routers, with evaluations on GPT-2 and Pythia models.
A tweet explains the optimization principle behind Flash attention, focusing on minimizing data movement and maximizing on-chip memory reuse, in the context of a language modeling course lecture.
This paper proposes Jacobian-guided noise injection, a training strategy that improves quantization robustness in large language models by injecting noise into pre-attention logits with variance derived from the Jacobian norm, leading to significant performance gains in low-bit quantization settings.
A blog post discussing optimization techniques for constrained categorical probability distributions, using softmax reparameterization and log barrier methods, applied to protein binder design.
Explains that super weights in large language models arise from the SoftMax-Attention interaction creating a 'Nothing Dump' token that serves as a stable reference point; removing these weights cripples performance.
An educational overview of knowledge distillation, covering its history, core concepts like softmax and temperature, types, scaling laws, and practical examples including DeepSeek-R1.
A tweet sharing a naive CUDA softmax implementation using shared memory reduction, noting that reduction is straightforward.
This paper analyzes precision loss in FP8 attention due to the attention sink phenomenon when casting the softmax output to FP8 (E4M3). It shows that forward KV iteration causes underflow of non-sink attention values, and proposes reverse iteration and a static scaling factor S=256 to eliminate underflow, achieving 3-10x MSE improvement.
The paper identifies a misalignment between the softmax-based InfoNCE loss and the normalized embedding setting in modern contrastive learning. It proposes WEINCE, a simple modification that blends softmax logits with an endpoint shortfall correction using extreme value theory, yielding consistent improvements across vision benchmarks.
Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.