@maximelabonne: Turns out you never really needed µP, you just needed to scale the embedding learning rate by model width I'm no nanoGP…
Summary
A tweet suggests that scaling the embedding learning rate by model width can replace the need for µP (micro-parameterization), referencing Muon optimizer for hidden layers and Adam for the rest.
View Cached Full Text
Cached at: 05/22/26, 03:49 AM
Turns out you never really needed µP, you just needed to scale the embedding learning rate by model width
I’m no nanoGPT speedrunner, but isn’t it something people stumbled into by using Muon for hidden layers + Adam for the rest? https://t.co/Ybs5C2rhlH
Similar Articles
Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
The paper proves almost sure convergence of stochastic gradient descent with clipping and additive noise, including momentum variants, under smoothness and bounded gradient noise assumptions, providing theoretical foundations for stable training in convex and nonconvex settings.
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
This paper introduces a constant-size state cache for block diffusion models, demonstrating significant reductions in memory and latency compared to attention-based approaches, enabling efficient long-context generation without quality loss.
@YRSM_Simon: Amazing!
shi3z shares optimization experience for running DeepSeek v4.1 Flash on an A100 GPU without FP4 support, boosting inference speed from 33 tok/s to 673 tok/s, surpassing the official API speed.
Pacing the Frontier – Tahuna: AI Training Infrastructure, Now Open Source [P]
Tahuna is an open-source AI training infrastructure designed for small teams to train models, run inference, orchestrate GPUs, and experiment with autonomous research, featuring reproducible runs and tools like Hillclimb.
Hoping for Optimized Smarter Upcoming Models .... Like DeepSeek-V4.1-Flash( KVCache + Engram) in Small/Medium/Big sizes
The article expresses hope for future AI models with optimizations like KVCache and Engram to reduce memory usage, enabling larger models to run on consumer GPUs with limited VRAM.