Scaling Laws for Looped Mixture of Experts
Summary
Introduces Loop Scaling Laws, the first scaling law jointly modeling recurrence (looped transformers) and sparsity (MoE) alongside model size and data, showing looped MoEs match models ~2x larger on reasoning benchmarks at matched compute.
View Cached Full Text
Cached at: 10/01/26, 08:24 AM
Paper page - Scaling Laws for Looped Mixture of Experts
Source: https://huggingface.co/papers/2609.40316
Abstract
LoopedtransformersandMixture-of-Experts(MoE)offercomplementaryroutestoefficientscaling:recurrenceincreasescomputationaldepthatfixedparameters,whileMoEsparsityexpandstotalcapacityatfixedactivecompute.Yetexistingscalinglawsmodelrecurrenceorsparsityinisolation.Inthiswork,weintroduceLoopScalingLaws,thefirstscalinglawtojointlymodelrecurrenceandsparsityalongsidemodelsizeanddata.Atitscoreisabounded,sparsity-conditionalrecurrencemappingthatcharacterizestheeffective-parametergainfromloopingandhowsparsityraisesthisgain.Thelawspredicttheheld-outlossofloopedmodelsmoreaccuratelythanprioralternatives,andrecoverthestandarddenseandMoEscalinglawsasspecialcases.Beyondprediction,thefittedlawsprovideaprincipledfoundationfordesigningloopedMoEmodelsundercomputeandmemoryconstraints.Downstreamevaluationsfurtherdemonstratethecomplementarybenefitsofthetwoaxes:sparsitydelivers~3xactive-parameterefficiency,recurrenceyields~2xtotal-parameterefficiencyonreasoning,andjointscalingfurtheradvancestheperformancefrontier.Asapracticalextension,weshowthesegainsholdattrillion-tokenscale:atmatchedtrainingcompute,aloopedMoEwithlaw-derivedrecurrencematchesa~2xlargernon-loopedMoEonthereasoningbenchmarks,whileenablingtest-timescalingthroughrecurrence.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.40316
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.40316 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.40316 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.40316 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
The paper proposes LOOM, a scalable recipe for Looped Mixture-of-Experts models that stabilizes recurrence via residual scaling and input re-injection while diversifying expert usage with per-loop routers, achieving stable scaling to 9-12 loops and improved perplexity and zero-shot accuracy on 100M-1.7B models.
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
论文提出 LatentLoop,通过非对称计算分配(压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度)来扩展循环 MoE 模型,在 1.3B–7B 参数范围内一致降低验证 NLL,相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
This paper develops a principled scaling theory for Mixture-of-Experts (MoE) architectures, introducing the Maximally Scale-Stable Parameterization (MSSP) that ensures stable training and hyperparameter transfer across width, depth, expert width, and number of experts, validated by experiments.
Model Merging Scaling Laws in Large Language Models
This paper establishes empirical scaling laws for language model merging, identifying power-law relationships between model size, expert count, and performance to enable predictive planning for optimal model composition.