Tag
This paper examines the coupling between the preconditioning exponent and learning rate in adaptive optimizers, showing that the exponent maximizing cross-environment accuracy decreases with log learning rate, and identifies a conflict between source-domain validation and robustness to environmental change.
The paper demonstrates that using a shared learning rate as a control in selective on-policy distillation experiments is not neutral, leading to varying performance and conclusions, and advocates for reporting full learning-rate matrix comparisons for fair evaluation.
This paper introduces a trust-region framework for understanding adaptive moment estimation methods like Adam, deriving a family of learning-rate mechanisms based on p-th moment constraints (2≤p≤4), including a kurtosis-based variant (Gmake). Experiments on GPT2-124M show that second-moment versions become competitive when trust-region controls are stronger.
The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.
This paper investigates whether GRPO post-training improves a small (4B-8B) language and vision-language model web agent. It finds a controlled null result: no configuration yields credible gains on mastered tasks, and moderate-to-high learning rates cause degradation or collapse, revealing a double dissociation between degrade and collapse regimes.
This paper introduces a framework to quantify hyperparameter transfer in LLMs and finds that the benefit of μP over SP in AdamW training largely comes from increasing the embedding layer learning rate. It also explores the impact of weight decay and other factors.
A tweet suggests that scaling the embedding learning rate by model width can replace the need for µP (micro-parameterization), referencing Muon optimizer for hidden layers and Adam for the rest.
A user found that reducing the learning rate from 2e-4 to 1e-4 significantly improved QLoRA fine-tuning of Llama 3.1 8B on a small dataset (8k samples), preventing overfitting and leading to better evaluation results.
This paper derives a closed-form upper bound for admissible learning-rate steps in belief-space dynamics using KL divergence and Bregman geometry, focusing on cross-entropy classification.
This paper presents a closed-form upper bound for admissible learning-rate steps in belief-space dynamics, providing a theoretical result for optimization in robotics or control.