@maximelabonne: To clarify, this paper basically says: under AdamW, µP's embedding LR rule (constant) is essentially right and explains…
Summary
This paper clarifies that under AdamW, µP's embedding learning rate rule (constant) is essentially correct and explains most of µP's benefit, contrary to a previous finding by Hayou et al. about realistic LLM vocab sizes.
View Cached Full Text
Cached at: 05/23/26, 02:09 PM
Turns out you never really needed µP, you just needed to scale the embedding learning rate by model width
I’m no nanoGPT speedrunner, but isn’t it something people stumbled into by using Muon for hidden layers + Adam for the rest?
To clarify, this paper basically says: under AdamW, µP’s embedding LR rule (constant) is essentially right and explains most of µP’s benefit.
Last year, Hayou et al. found that µP’s embedding LR rule is wrong for realistic LLM vocab sizes. They found that the optimal embedding LR decreases as 1/√width
These predictions look contradictory. But this paper successfully tested its thesis in a regime (vocab=50k, width=128-2048) that shouldn’t work according to Hayou.
Not sure why this is the case tbh, but interesting future work to explore!
https://x.com/Ham_TheFog/status/2057617101360451886…
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate (first screenshot, Kalra and Barkeshli): https://arxiv.org/abs/2605.21486
Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size (Hayou and Liu): https://arxiv.org/abs/2506.15025
Similar Articles
@maximelabonne: Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate (first screenshot, Kalra and Ba…
This paper introduces a framework to quantify hyperparameter transfer in LLMs and finds that the benefit of μP over SP in AdamW training largely comes from increasing the embedding layer learning rate. It also explores the impact of weight decay and other factors.
@dair_ai: New paper on giving LLM agents experience that improves the weights and stays readable at the same time. Agent-experien…
JERP introduces a method for LLM agents to jointly learn interpretable natural-language rules and update policy parameters from the same interaction trajectories, improving performance on AlfWorld and WebShop while maintaining inspectability.
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
The paper compares large language models and embedding models across 37 tasks, finding that while aggregate performance is similar, embedding models are far cheaper and faster, supporting a division of labor for cost-efficiency.
On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
This paper investigates how LLMs' internal priors affect zero-shot annotation performance, finding that nearly two-thirds of errors resist prompt-based correction and introducing Definition-Specific Familiarity as a better predictor than memorization metrics.
@agarwl_: Good blog, makes you think about the empirical observation that cureent RL methods that work for LLMs are *low bias* - …
A blog post explores the paradox of reinforcement learning for LLMs achieving rapid sample efficiency despite being information-theoretically inefficient, and highlights the importance of low-bias value functions.