normalization

提出 LayerRoPE，将 Transformer 中隐式的深度位置编码显式化：用单一共享 γ 向量与深度条件标量替代逐层 γ，在参数减少、FLOPs 几乎不变的情况下，在最高 512 层的深度上实现更稳定的训练与显著更高的计算效率（达到 Pre-Norm 1.3B 损失所需算力减少 3.4 倍）。

This paper introduces DepthBench, a controlled benchmark for studying computational depth in Transformers by varying width-depth ratios. It identifies that residual connection designs like HC and FullAttnRes are key to translating architectural depth into effective computation.

CryptoL is a unified framework for cryptocurrency multivariate time series forecasting that mitigates scale dominance and enforces physical constraints to enhance prediction accuracy and financial validity.

OntologyAligner presents a three-stage framework combining ontology-aligned retrieval and LLM reranking for biomedical ontology normalization, achieving state-of-the-art performance on HPO tasks and introducing the PhenoNormBench benchmark.

This paper proposes the Triangular Fuzzy Rescaling Distance (d_TR), a metric that integrates normalization directly into distance calculations for comparing triangular fuzzy numbers, suitable for heterogeneous data applications.

This paper shows that sequence-pooled normalization in convolutional networks provides global context beyond the receptive field, supplying most of the context needed for sequence labeling and affecting attribution in network ablation studies.

This paper introduces V-Simba, a visual RL architecture that adds normalization layers and pointwise convolutions to improve sample efficiency and stability. It matches or outperforms state-of-the-art methods across DMC, Adroit, and Meta-World benchmarks while being more computationally efficient than DrQ-v2.

This paper unifies Euclidean residual connections and geodesic normalization on the hypersphere, introducing a one-parameter family of angular retractions called p-SpheretNorm. The proposed norm-preserving algebraic methods outperform existing deep connection schemes on nanoGPT, showing the exponential map is just one end of a spectrum.

Mojibake is a self-contained Unicode 17 library for C11/C++17, providing normalization, case conversion, and character database functions with zero dependencies.

This course note summarizes the architectural evolution from the original Transformer to modern LLMs, focusing on convergent developments such as pre-normalization, RMS normalization, and RoPE, and provides hyperparameter selection recommendations.

A blog post defending the necessity of keys and normalization in the relational model, arguing that they reflect ontological requirements for coherent discourse about reality, countering criticisms about practical difficulties in defining keys and domains.

Proposes LieBN, a framework for batch normalization over Lie groups, applicable to SPD, rotation, and correlation manifolds, with theoretical guarantees and extensive experiments.

This blog post from PyTorch presents novel kernel fusion techniques for normalization ops like LayerNorm and RMSNorm, achieving significant speedups by reducing memory-IO overhead. Techniques include Lazy Pre-Norm and Multi-CTA Norm Fusion, hiding up to 90% of normalization latency when fused with GEMMs, and the FlashNormAttention algorithm achieving up to 35% kernel speedup.

Meta introduces techniques like Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention to fuse normalization operations with GEMM and Attention kernels, hiding up to 90% of normalization latency on NVIDIA B200 hardware and achieving up to 35% latency reduction in attention blocks.

Wiola is a novel Small Language Model architecture introducing five independently designed components—SRPE, GCLA, ATM, DSFF, and WiolaRMSNorm—aimed at improving efficiency and coherence, released in sizes from 120M to 1.5B parameters and integrated with HuggingFace Transformers.

This document discusses the challenges of Unicode filename composition (NFC vs NFD) across different operating systems in the context of the Subversion version control system, and proposes solutions for handling these differences.

This paper proposes TOPSIS-RAD, a modified version of the TOPSIS method that incorporates decision-maker-defined reference levels (VPL and DPL) to address issues like misalignment with preferences, outlier sensitivity, and rank reversal.

The article compares two methods for normalizing RGB values (dividing by 255 vs 256) and explains the consequences for floating-point conversion and rounding, including uneven bin widths at the extremes.

This paper systematically studies scale vectors in LLM normalization layers, showing they optimize training through a self-amplifying preconditioning effect, and proposes three lightweight improvements that enhance performance and scaling behavior with negligible overhead.

This paper introduces a new energy-based model for linear inverse problems that learns normalized posterior densities, overcoming limitations of diffusion models. It enables unbiased sampling, adaptive sampling, and blind degradation estimation, with competitive performance on ImageNet, CelebA, and AFHQ.