Tag
提出ShellElastic，一种将隐藏维度组织为稠密核心与递增外壳的弹性Transformer架构，使单次全宽前向即可同时包含所有子网络的精确隐藏状态；并引入μE扩展最大化更新参数化（μP），保证子网络在超网络加宽或增加外壳时仍被最大化更新且超参数可迁移，性能优于MatFormer等弹性基线并逼近分别训练的模型。
The paper proposes a compute-efficient two-step hyperparameter transfer framework to predict optimal learning rates for large Mixture-of-Experts models, enabling efficient pretraining without costly hyperparameter sweeps.
This paper analyzes residual scaling in looped (weight-tied) transformers, showing that weight sharing requires stronger scaling (1/N) than standard residual networks, and derives a factored parameterization that enables hyperparameter transfer across loop counts without retuning.
This paper derives scaling rules for Gated Delta Networks using Maximal Update Parametrization (μP), enabling zero-shot hyperparameter transfer across model widths for efficient sub-quadratic LLM architectures. Experiments confirm stable learning-rate transfer under both AdamW and SGD, whereas standard parametrization fails.
This paper introduces a framework to quantify hyperparameter transfer in LLMs and finds that the benefit of μP over SP in AdamW training largely comes from increasing the embedding layer learning rate. It also explores the impact of weight decay and other factors.
This paper extends the maximal update parameterization (μP) framework to grouped-query attention (GQA), deriving scaling laws for hyperparameter transfer across model architectures. It introduces spectral norm conditions for feature learning and addresses issues with low-rank weight matrices in GQA.