Model Merging Scaling Laws in Large Language Models
Summary
This paper establishes empirical scaling laws for language model merging, identifying power-law relationships between model size, expert count, and performance to enable predictive planning for optimal model composition.
View Cached Full Text
Cached at: 05/12/26, 07:32 AM
Paper page - Model Merging Scaling Laws in Large Language Models
Source: https://huggingface.co/papers/2509.24244
Abstract
Empirical scaling laws for language model merging reveal power-law relationships between model size, expert count, and cross-entropy performance, enabling predictive planning for optimal model composition.
We study empiricalscaling lawsforlanguage model mergingmeasured bycross-entropy. Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we add experts or scale themodel size. We identify a compactpower lawthat linksmodel sizeandexpert number: the size-dependent floor decreases withmodel capacity, while the merging tail exhibits cleardiminishing returnsin the number of experts. The law holds in-domain and cross-domain, tightly fits measured curves across diverse architectures and methods (Average, TA, TIES, DARE), and explains two robust regularities: most gains arrive early, and variability shrinks as more experts are included. Building on this, we present a simple theory that explains why gains fall roughly as 1/k and links the floor and tail to properties of the base model and the diversity across domains. This law enablespredictive planning: estimate how many experts are needed to reach a target loss, decide when to stop adding experts, and trade off scaling the base model versus adding experts under a fixed budget--turning merging from heuristic practice into a computationally efficient, planable alternative tomultitask training. This suggests a scaling principle fordistributed generative AI: predictable gains can be achieved by composing specialists, offering a complementary path towardAGI-level systems.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2509\.24244
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2509.24244 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2509.24244 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2509.24244 in a Space README.md to link it from this page.
Collections including this paper2
Similar Articles
Scaling laws for neural language models
Foundational empirical study demonstrating power-law scaling relationships between language model performance and model size, dataset size, and compute budget, with implications for optimal training allocation and sample efficiency.
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
InfoLaw is a data-aware scaling framework that predicts model loss based on token consumption, model size, data mixture weights, and repetition, enabling efficient data-recipe selection under varying compute budgets.
On the Smallness of the Large Language Models Scaling Exponents
The paper discusses the small scaling exponents of large language models, arguing that they indicate an unsustainable regime in terms of energy resources. It also examines the 'pedestal effect' and draws analogies with fluid turbulence to comment on data smoothness.
Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
The paper investigates scaling laws for hypernetwork-based knowledge injection into LLMs, finding predictive power law scaling and reliable out-of-distribution generalization, establishing hypernetworks as a scalable alternative to LoRA and full fine-tuning.
@omarsar0: Impressive new paper from Meta. (bookmark it) Scaling laws assume model size and training data act on loss independentl…
Meta researchers introduce the Skaling law, a generalized scaling law that couples model capacity and data through a single interaction exponent, reducing prediction error by 1.5–3x and enabling roughly 10x less compute for full-grid extrapolation.