layer-normalization

This paper constructs a transformer with linear self-attention that performs in-context learning of the closed-form least squares solution for simple linear regression, using layer normalization to approximate the analytical solution rather than gradient descent.

This paper identifies that in LayerNorm-based GNNs, positive per-node scalars like node degree are erased when placed before LayerNorm but survive after LayerNorm. The authors propose PostDeg, a parameter-free post-LayerNorm inverse-degree scale, achieving significant gains on influence maximization, network dismantling, and maximum independent set tasks.

This paper introduces the Curse of Depth in LLMs, where deep layers become ineffective due to Pre-Layer Normalization causing output variance explosion. The authors propose LayerNorm Scaling to mitigate this, showing consistent improvements in pre-training and fine-tuning across model sizes up to 7B.