Scaling Laws for Looped Mixture of Experts

Hugging Face Daily Papers Papers

Summary

Introduces Loop Scaling Laws, the first scaling law jointly modeling recurrence (looped transformers) and sparsity (MoE) alongside model size and data, showing looped MoEs match models ~2x larger on reasoning benchmarks at matched compute.

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.
Original Article
View Cached Full Text

Cached at: 10/01/26, 08:24 AM

Paper page - Scaling Laws for Looped Mixture of Experts

Source: https://huggingface.co/papers/2609.40316

Abstract

LoopedtransformersandMixture-of-Experts(MoE)offercomplementaryroutestoefficientscaling:recurrenceincreasescomputationaldepthatfixedparameters,whileMoEsparsityexpandstotalcapacityatfixedactivecompute.Yetexistingscalinglawsmodelrecurrenceorsparsityinisolation.Inthiswork,weintroduceLoopScalingLaws,thefirstscalinglawtojointlymodelrecurrenceandsparsityalongsidemodelsizeanddata.Atitscoreisabounded,sparsity-conditionalrecurrencemappingthatcharacterizestheeffective-parametergainfromloopingandhowsparsityraisesthisgain.Thelawspredicttheheld-outlossofloopedmodelsmoreaccuratelythanprioralternatives,andrecoverthestandarddenseandMoEscalinglawsasspecialcases.Beyondprediction,thefittedlawsprovideaprincipledfoundationfordesigningloopedMoEmodelsundercomputeandmemoryconstraints.Downstreamevaluationsfurtherdemonstratethecomplementarybenefitsofthetwoaxes:sparsitydelivers~3xactive-parameterefficiency,recurrenceyields~2xtotal-parameterefficiencyonreasoning,andjointscalingfurtheradvancestheperformancefrontier.Asapracticalextension,weshowthesegainsholdattrillion-tokenscale:atmatchedtrainingcompute,aloopedMoEwithlaw-derivedrecurrencematchesa~2xlargernon-loopedMoEonthereasoningbenchmarks,whileenablingtest-timescalingthroughrecurrence.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.40316

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.40316 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.40316 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.40316 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Hugging Face Daily Papers

The paper proposes LOOM, a scalable recipe for Looped Mixture-of-Experts models that stabilizes recurrence via residual scaling and input re-injection while diversifying expert usage with per-loop routers, achieving stable scaling to 9-12 loops and improved perplexity and zero-shot accuracy on 100M-1.7B models.

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Hugging Face Daily Papers

SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Hugging Face Daily Papers

论文提出 LatentLoop,通过非对称计算分配(压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度)来扩展循环 MoE 模型,在 1.3B–7B 参数范围内一致降低验证 NLL,相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。

Model Merging Scaling Laws in Large Language Models

Hugging Face Daily Papers

This paper establishes empirical scaling laws for language model merging, identifying power-law relationships between model size, expert count, and performance to enable predictive planning for optimal model composition.