Impressive new paper from Meta.

(bookmark it)

Scaling laws assume model size and training data act on loss independently.

This work introduces Skaling law, which couples capacity and data through a single interaction exponent. The extra term cuts mean absolute percentage error by 1.5x to 3x across both interpolation and extrapolation.

The largest corrections land in the data-scarce and heavy-overtraining regimes where the standard Chinchilla and Kaplan forms drift.

Paired with a sparse grid restricted to low-compute runs, it extrapolates the full grid using roughly 10x less compute than a uniform sweep.

Why does it matter?

Deployment now happens well past compute optimal. A law that stays accurate there, and that can be fit from small runs, changes how a pretraining budget gets planned.

Paper: https://arxiv.org/abs/2608.07222

Track more trending AI papers in our academy: https://academy.dair.ai

Skaling: Chinchilla’s Exponents Meet Kaplan’s Coupling

Source: https://arxiv.org/html/2608.07222 1]FAIR at Meta

(August 7, 2026)

Abstract

Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law (\tipaencoding/“skeIlIN/), a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by1.51.5–3×3\timesacross both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately10×10\timesless compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

\correspondence

Mathurin Videau at

Figure 1:The additive Chinchilla law carries a systematic, boundary-concentrated prediction bias that the Skaling law removes.Each marker is a trained(N,D)(N,D)configuration (model sizeNNhorizontal, training tokensDDvertical).*Left, centre:*signed percentage error (positive = overestimation (red), negative = underestimation (blue)) of the fitted Chinchilla and Skaling laws (shared colorbar); Chinchilla shows a saddle-shaped residual that grows toward the corners, whereas Skaling stays near zero throughout.*Right:*the per-run ratio of the two laws’ errors (capped at16×16\times); red runs where Skaling is x times more accurate. Skaling wins at76%76\%of configurations (median2.2×2.2\times, and≥4×\geq\!4\timesat a third of them), with the largest gains at the cheaper edges.## 1Introduction

The success of modern Large Language Models (LLMs) rests on their predictable and consistent improvement as training data and model size scale. This property is formalized by neural scaling laws(Kaplan et al.,2020; Hoffmann et al.,2022; Bi et al.,2024), which predict the performance of high-compute runs using only a handful of low-compute models. Guided by these empirical laws, researchers can confidently optimize critical decisions regarding pretraining budgets, architectural dimensions, and resource allocation.

Despite their widespread adoption, standard scaling formulations possess a structural flaw(Li et al.,2025a; Ardalani et al.,2026). While the original Kaplan form(Kaplan et al.,2020)coupled model size and data, the widely used Chinchilla law(Hoffmann et al.,2022)models the reducible loss as a simple sum of independent terms for model size and training tokens. This mathematical structure implies a strict independence between the two variables, artificially enforcing a cross-derivative of exactly zero. However, early analysisFigure˜1shows Chinchilla is accurate in the interior of the grid but develops large, oppositely-signed errors toward the corners, reaching several percent whereNNandDDare most imbalanced. This is the saddle-shaped residual expected when theNN–DDinteraction is omitted. Furthermore, our empirical analysis of the loss gradients reveals an interaction between these dimensions (Section˜2). Because they ignore this coupling, additive scaling models systematically under- and overestimate loss at the boundaries of the training grid.

To resolve this, we propose the Skaling law, a minimal generalization of the Chinchilla form that introduces a single coupling exponent between model size and data volume, restoring the interaction of the Kaplan form that the additive Chinchilla law discards. Despite adding only one parameter, the Skaling law reduces extrapolation MAPE by1.51.5–3×3\timesrelative to Chinchilla across multiple cross-validation strategies. We further show that this improved functional form enables an efficient “L-shape” profiling strategy. By restricting training to the cheaper edges of the compute grid, sweeping data volume for small models (varying training tokensDD, referred to as the D-band) and sweeping model size on a small, fixed amount of data (varying model sizeNN, referred to as the N-band), the Skaling law achieves predictive accuracy matching that of Chinchilla fitted on the full grid, while requiring up to10×10\timesless compute. Together, these contributions provide a practical, well-grounded framework for scaling decisions under constrained experimental budgets.

Contributions

1. C1.Skaling: a coupled scaling form. We introduce a single interaction exponent between model size and data volume that corrects the boundary biases of additive laws (Section˜3.1).
2. C2.Sparse profiling grids. We show that Skaling’s predictive accuracy is preserved under a more effective sampling scheme, enabling “L-shape” grids that reduce profiling compute by∼10×\sim\!10\timeswhile maintaining accuracy (Section˜3.2).

Figure 2:First-order derivative structure on Farseer (Equation˜1; MLS estimates, log–log axes; the colorbar shows the cross-variable).Top:same-variable projections,ln⁡|∂L/∂N|\ln|\partial L/\partial N|vsln⁡N\ln N(a) andln⁡|∂L/∂D|\ln|\partial L/\partial D|vsln⁡D\ln D(b), whose linear trends indicate power-law decay (αN,αD\alpha_{N},\alpha_{D}).Bottom:cross-variable projections,ln⁡|∂L/∂N|\ln|\partial L/\partial N|vsln⁡D\ln D(c) andln⁡|∂L/∂D|\ln|\partial L/\partial D|vsln⁡N\ln N(d); the dominant structure is horizontal bands induced by the same-variable dependence, while the cross-slopesγN,γD\gamma_{N},\gamma_{D}remain small.

2The loss surface couples model size and data

Before committing to a functional form, we ask the data directly whether model size and training data interact. We probe the loss surface through its derivatives, estimated with the Moving least squares (MLS)Lancaster and Salkauskas (1981)estimator (described inSection˜7.1); because the grid is logarithmically spaced, MLS returns log-slopes that we convert to real-space derivatives,

∂L∂N=LN​∂ln⁡L∂ln⁡N,∂L∂D=LD​∂ln⁡L∂ln⁡D,\frac{\partial L}{\partial N}=\frac{L}{N}\frac{\partial\ln L}{\partial\ln N},\qquad\frac{\partial L}{\partial D}=\frac{L}{D}\frac{\partial\ln L}{\partial\ln D},which removes the irreducible errorEEand isolates the structure of the reducible loss.

We first summarize the first-order structure with the log-linear diagnostic

ln⁡|∂L∂N|=αN​ln⁡N+γN​ln⁡D+cN,ln⁡|∂L∂D|=γD​ln⁡N+αD​ln⁡D+cD,\ln\!\left|\frac{\partial L}{\partial N}\right|=\alpha_{N}\,\ln N+\gamma_{N}\,\ln D+c_{N},\qquad\ln\!\left|\frac{\partial L}{\partial D}\right|=\gamma_{D}\,\ln N+\alpha_{D}\,\ln D+c_{D},(1)whereαN,αD\alpha_{N},\alpha_{D}capture the dominant same-variable decay andγN,γD\gamma_{N},\gamma_{D}the residual dependence on the other axis; we use it only as a diagnostic, not as a scaling law.Figure˜2(a,b) shows the same-variable projections are close to linear, withαN≈αD≈−1.3\alpha_{N}\approx\alpha_{D}\approx-1.3, so the marginal derivatives decay approximately as power laws. The cross-slopes are small,γN≈0.13\gamma_{N}\approx 0.13andγD≈0.07\gamma_{D}\approx 0.07, so at first order the surface looks nearly separable; first-order projections alone, however, cannot rule out a weaker interaction.

The mixed derivative is the decisive test. Any additive lawL=f​(N)+g​(D)+EL=f(N)+g(D)+Esatisfies∂2L/∂N​∂D=0\partial^{2}L/\partial N\partial D=0identically, whatever power laws are chosen forffandgg.Figure˜3shows the estimated mixed derivative is instead non-zero across the entire grid, with its own power-law decay,

ln⁡|∂2L∂N​∂D|=a​ln⁡N+b​ln⁡D+c,\ln\!\left|\frac{\partial^{2}L}{\partial N\,\partial D}\right|=a\,\ln N+b\,\ln D+c,(2)witha≈b≈−1.1a\approx b\approx-1.1and a predominantlynegativesign: scalingNNandDDtogether lowers the loss more than scaling either alone, a synergy an additive law cannot represent. This motivates replacing Chinchilla’s additive sum with a single coupling exponent, the Skaling form introduced next;Section˜7.2shows that this form reproduces both the negative mixed derivative and the small, asymmetric first-order slopes, and why a multiplicative coupling is preferable to an additive interaction term.

Figure 3:Cross-derivative|∂2L/∂N​∂D||\partial^{2}L/\partial N\partial D|(from the second-order term of the local quadratic fit, log scale), testingEquation˜2.(a)Dependence onln⁡N\ln N, colored byDD.(b)Dependence onln⁡D\ln D, colored byNN. An additive law predicts∂2L/∂N​∂D=0\partial^{2}L/\partial N\partial D=0; the estimates are non-zero throughout the grid.

3Methods

3.1The Skaling form

Two functional forms anchor the scaling-law literature, differing primarily in how model size and data combine. The widely used Chinchilla law(Hoffmann et al.,2022)decouples them completely: it sums two terms with independent inner exponentsα\alphaandβ\beta, alongside a free irreducible constantEE. While mathematically convenient, this purely additive structure forces the cross-derivative∂2L/∂N​∂D\partial^{2}L/\partial N\partial Dto vanish identically, implicitly assuming thatNNandDDdo not interact. The earlier Kaplan form(Kaplan et al.,2020)takes the opposite approach:

L​(N,D)=[(Nc/N)αN/αD+Dc/D]αDL(N,D)=\bigl[(N_{c}/N)^{\alpha_{N}/\alpha_{D}}+D_{c}/D\bigr]^{\alpha_{D}}Here the outer exponentαD\alpha_{D}plays the role of Skaling’skk, but Kaplan additionally ties the inner terms through the ratioαN/αD\alpha_{N}/\alpha_{D}, so their per-axis decay rates are no longer independent.

Skaling bridges these two paradigms. It retains Chinchilla’s interpretable base terms and independent inner exponents but, following Kaplan, raises their sum to a single free outer exponentkk:

L​(N,D)=(ANα+BDβ)k+EL(N,D)=\left(\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}\right)^{k}+E(3) This single parameter interpolates between the two prior forms. Atk=1k=1, the Skaling law recovers the purely additive Chinchilla law. For anyk≠1k\neq 1, it reinstates a Kaplan-style coupling and a non-zero cross-derivative. Crucially, unlike Kaplan, the Skaling law achieves this coupling through the outer exponentkkwhile preserving Chinchilla’s independent inner exponents. Despite this added flexibility, the Skaling law retains two desirable properties of the additive law. First, becausek>0k>0, the function remainsstrictly decreasingin bothNNandDD: adding model capacity or training data never increases the predicted loss. Second, it preserves Chinchilla’s structure. The exponentkkdictates how the two source termsaggregateinto the final loss, so each retains its individual interpretability even though their joint contribution no longer separates additively.

A practical payoff of this structure concerns compute allocation. Minimizing the loss under a fixed budgetCCreduces to the same stationarity condition as the additive Chinchilla law, so the Skaling law inherits Chinchilla’s closed-form compute-optimal allocation: the optimal token-to-parameter ratioD∗/N∗D^{*}/N^{*}keeps its closed form and stays constant across scales whenα≈β\alpha\approx\beta. Importantly, while the algebraic formula is identical, the resulting optimal ratio differs in practice because the underlying fitted parametersAA,BB,α\alpha, andβ\betado not transfer between the two functional forms. We give the full derivation inSection˜8.Section˜8for the full derivation).

Accurately identifying this optimal ratio is critical because it dictates baseline architectural decisions. Frontier models such as DeepSeekBi et al. (2024)typically lock in a fixed token-to-parameter ratio; therefore, the true optimum must be known to accurately quantify the performance penalty of suboptimal training. As detailed inSection˜7.3(Figure˜6), the optimal ratio predicted by Chinchilla drifts significantly from that of Skaling, accumulating to a 100-fold discrepancy at frontier compute scales. When validated against empirical measurements, Skaling’s predicted ratio tracks the true underlying trend more accurately.

3.2Sampling Strategies

To obtain robust parameter estimates without prohibitive computational costs, the design of the experimental training grid plays an important role. Standard full-grid setups typically sampleNNandDDon a square spaced logarithmically to capture behavior across multiple orders of magnitude. Because of this, the total compute is overwhelmingly dominated by the top-right corner of the grid, which contains the largest models trained for the longest horizons. This geometric concentration of compute makes dense sampling particularly wasteful. Rather than expending the vast majority of an experimental budget on a few massive runs, compute could be better allocated to a more discriminative set of points. The mathematical structure of the scaling laws naturally motivates our primary approach: the L-shape sampling strategy (Figure˜4(a)).

(a)Sampling strategies. (b)Evaluation regimes.

Figure 4:Partitions of the(N,D)(N,D)grid.(a)Sampling strategies:Randomspreads held-out points across the grid, whereas theL-shapegrid restricts training to the low-compute edges.(b)Evaluation regimes used for cross-validation: interpolation, extrapolation inNNand inDD, and far extrapolation beyond both.Consider the asymptotic behavior of the loss function. As data volume increases, the estimation error vanishes and the loss approaches the size-dependent approximation error:

limD→∞L​(N,D)=(ANα)k+E\lim_{D\to\infty}L(N,D)=\left(\frac{A}{N^{\alpha}}\right)^{k}+EConsequently, increasingDDcleanly isolates theNN-dependent coefficients. Symmetrically, as model size increases, the approximation error diminishes, cleanly isolating the data-dependent parameters:

limN→∞L​(N,D)=(BDβ)k+E\lim_{N\to\infty}L(N,D)=\left(\frac{B}{D^{\beta}}\right)^{k}+E In practice, we do not need to reach these theoretical limits. Simply varying one axis while holding the other fixed provides sufficient signal to trace the corresponding decay rate. The L-shape strategy applies this principle at the lowest compute scales. Rather than filling the entire parameter space, we sweep data volumeDDexclusively for the smallest models to fit the data parameters (B,βB,\beta), and sweep model sizeNNexclusively on the shortest training horizons to fit the size parameters (A,αA,\alpha). By anchoring the independent decay rates along the grid boundaries, this sparse geometry maps the interaction betweenNNandDDmore efficiently than a full grid sweep under same total compute budget constrain.

3.3Evaluation Protocol

To rigorously test the predictive capabilities and algorithmic stability of the fitted methods, we employ a comprehensive cross-validation framework. Rather than relying on a single static train-test split, we repeatedly resample the training data and construct corresponding hold-out sets. This approach allows us to explicitly quantify the uncertainty of our predictions, assess the variance of the fitted parameters, and test the models’ ability to extrapolate reliably to unseen scales. For each cross-validation fold, the evaluation sets are partitioned as follows (Figure˜4(b)):

• •Validation (Interpolation):Randomly held-out points lying within the established boundaries of the training grid.
• •Extrapolation N:Points at larger model sizes beyond the active training set, simulating prediction for larger architectures.
• •Extrapolation D:Points at larger data volumes, testing predictions for extended training horizons.
• •Far Extrapolation:The most challenging regime, consisting of the largest models trained on the largest data volumes, lying completely outside both theNNandDDboundaries of the training grid.

On all cross-validation folds, we report the MAPE on every evaluation set and the coefficient of determination (R2R^{2}) on the interpolation set only. For an evaluation set𝒮\mathcal{S}in which runiihas measured lossLiL_{i}and the fitted law predictsL^i\hat{L}_{i}, the MAPE is the mean absolute relative deviation,

MAPE(𝒮)=100|𝒮|∑i∈𝒮|L^i−Li|Li[%].\mathrm{MAPE}(\mathcal{S})\;=\;\frac{100}{\lvert\mathcal{S}\rvert}\sum_{i\in\mathcal{S}}\frac{\bigl\lvert\hat{L}_{i}-L_{i}\bigr\rvert}{L_{i}}\quad[\%].(4)We restrictR2R^{2}to interpolation because the extrapolation sets contain few points covering a narrow, somewhat arbitrary slice of the grid; sinceR2R^{2}normalizes error by the variance of the held-out targets, it becomes unstable and uninformative there (often small or strongly negative), whereas MAPE stays directly comparable across all regimes. Aggregating over folds, we report the mean of each metric together with its variance.

3.4Compute extrapolation along iso-ratio slices

Figure 5:Iso-ratio compute extrapolation.Frontier labs rely almost exclusively on one-dimensional compute power laws to project the performance of their massive pretraining runs. Because these models are typically scaled along a fixed token-to-parameter ratio, as demonstrated by DeepSeek(Bi et al.,2024), extrapolating compute along these specific operational rays represents an interesting point of comparison for any scaling formulation. While our earlier cross-validation comprehensively maps theNN–DDgrid, here we address a strictly operational challenge: can a single, globally fitted law reliably predict the most computationally expensive runs using only cheap, low-compute data?

To mirror this scenario, we group the runs into iso-ratio slices of constant token-to-parameter ratioD/ND/N(Figure˜5). Within each slice, we hold out theKKhighest-compute points (K=8K{=}8). Every scaling law is refittedonceon the pooled remainder, the union of the low-compute points across all slices—and then evaluated on the held-out high-compute runs. This ensures each functional form sees the exact same training set for a rigorous global comparison. As a strong baseline, we additionally fit an independent single power lawL=A​Ca+EL=A\,C^{a}+Estrictly within each slice (per-ratio), replicating the DeepSeek methodology(Bi et al.,2024). While this per-ratio baseline is heavily tailored to a single recipe and cannot inform jointNN–DDallocation, it serves as an empirical upper bound on how well a dedicated, one-dimensional compute law can extrapolate along a fixed ray.

4Experiments

4.1Experimental setup

Data.

We fit and evaluate all scaling laws on two grids of pretraining runs.

Farseer(Li et al.,2025a)records the final validation loss of404404(N,D)(N,D)configurations across2525model sizes (100100M to6.46.4B parameters) and5555data budgets (11B to512512B tokens), with compute ranging from1.6×10181.6\times 10^{18}to4.1×10214.1\times 10^{21}FLOPs. All models use a sequence length of20482048. From this grid we hold out three evaluation sets:ExtrapolationNN(the33largest model sizes,4.54.5B–6.46.4B,3636points),ExtrapolationDD(the top-33data budgets per remaining model size,6666points), andFar extrapolation(77additional runs at larger scales,2.32.3B–2525B parameters trained on126126B–453453B tokens, well beyond both axes of the training grid). The remaining302302configurations used for fitting total∼5.0×1022\sim\!5.0\times 10^{22}FLOPs.

SK-Grid, our own grid of training runs, is a complementary set of134134configurations across1515model sizes (134134M to4.94.9B) and1616data budgets (316316M to316316B tokens), with compute from9.0×10169.0\times 10^{16}to9.9×10209.9\times 10^{20}FLOPs. The same hold-out scheme yields77points for ExtrapolationNN(2.82.8B–4.94.9B),3333points for ExtrapolationDD, and33far-extrapolation runs at∼1022\sim\!10^{22}FLOPs (5.85.8B–10.810.8B parameters). The fitting grid totals∼3.1×1021\sim\!3.1\times 10^{21}FLOPs.

Optimizer.

All scaling laws are fitted by minimizing a Huber loss in log space using L-BFGS-B with basin-hopping and autograd. Full optimizer settings and per-law bounds are given inSection˜9.

Baselines.

We benchmark the Skaling law against the two most widely used reducible-loss forms: the additive Chinchilla law(Hoffmann et al.,2022)and the more heavily parameterized Farseer law(Li et al.,2025a). All laws are fit with the same optimizer and log-space objective (Section˜9), so that differences in predictive accuracy reflect the functional form rather than the fitting procedure. We fit each functional form to the remaining data and measure its predictive error under the cross-validation protocol ofSection˜3.3.

Training.

The models in the grid are trained following the StepLaw hyperparameter prescriptions(Li et al.,2025b), which set the learning rate and batch size as functions of the model size and token budget. Using these near-optimal settings at every(N,D)(N,D)ensures that the measured loss reflects the scaling behaviour of the architecture rather than hyperparameter mistuning, providing a clean target for the scaling-law fits. Further details on model configurations and training hyperparameters are given inSection˜11.

4.2Results

Boundary errors.

The clearest gains appear at the boundaries of the grid. Interior interpolation is already accurate for the additive Chinchilla law, but the error grows on the single-axis and far-extrapolation sets, where the saddle-shaped residual ofFigure˜1is most pronounced. InTable˜1, the Skaling law reduces this boundary error consistently: on the full grids, the single-axis MAPE falls from1.481.48to0.470.47and from1.981.98to0.880.88on Farseer, and from0.830.83to0.390.39and from1.441.44to0.580.58on SK-Grid. The largest gains occur in the most imbalanced corners. On SK-Grid, for example, the far-extrapolation error drops from5.175.17to0.700.70on the full grid and from14.6314.63to1.151.15on the L-shape grid; the largest-NNL-shape error drops from6.096.09to0.770.77.

Sparse profiling.

Sparse profiling preserves much more of Skaling’s accuracy. When trained only on the L-shape grid, which uses roughly10×10\timesless fitting compute, the Skaling law remains close to or better than the full-grid Chinchilla baseline on interpolation and single-axis extrapolation. Chinchilla, by contrast, degrades substantially under the same restriction: on Farseer, interpolation MAPE increases from0.770.77to2.512.51, and on SK-Grid, far-extrapolation MAPE increases from5.175.17to14.6314.63. The coupled form therefore preserves predictive accuracy when the training grid is concentrated on the low-compute edges, which is the setting needed for the sparse profiling strategy inFigure˜4(a).

Table 1:Fit quality and predictive error on the Farseer and SK-Grid datasets. For each dataset we report the interpolation coefficient of determinationR2R^{2}(↑\uparrow) and the MAPE (%,↓\downarrow; mean±\pmstd over 5 CV folds) on interpolation and three extrapolation regimes (larger model sizeNN, larger dataDD, far extrapolation beyond both), for the full grid and the sparse L-shape grid (total training FLOPs per grid shown in the panel headers).R2R^{2}is reported on the interpolation set only (seeSection˜3.3). Best per column within each panel inbold.

Interpolation is not sufficient.

High interpolation fit quality is not enough to validate a scaling law. Chinchilla attains strong interpolationR2R^{2}on the full grids (0.9950.995on Farseer and0.9920.992on SK-Grid), yet its extrapolation errors are several times larger than Skaling’s at the grid boundaries. The relevant failure mode is therefore not a poor fit to the interior, but a systematic misprediction of how the loss surface bends away from the observed region.

Stable coupling.

The fitted parameters explain why the boundary predictions improve. Across all grids inTable˜2, the Skaling law recovers a sub-unit coupling exponent,k≈0.31k\approx 0.31–0.450.45, rather than collapsing back to the additive casek=1k=1. The fitted irreducible loss is systematically lower than Chinchilla’s, and on Farseer data it nearly vanishes: from0.450.45to0.030.03on the full grid and from0.590.59to0.050.05on the L-shape, while on SK-Grid it stays substantial (1.751.75to1.141.14). We do not readE≈0E\approx 0as a vanishing loss floor. The two quantities that set how the loss flattens, the couplingkkand the floorEE, trade off against each other: withk<1k<1the concave outer map makes the coupled reducible term decay more slowly at large scale, so it can absorb curvature that the additive law can only represent through a largerEE. Because none of our runs reach the scale where the loss saturates, the data fix the total loss but not this split between a decaying term and a constant floor, and the Skaling law resolves the ambiguity by pushingEEdown, on Farseer data almost to zero. BeyondEE, every parameter is determined precisely within each fit, with small fold-to-fold standard deviations: a few percent for the exponents and coupling, and at most∼40%\sim\!40\%for the amplitudes.

Table 2:Fitted coefficients (mean±\pmstd over55CV folds) behindTable˜1. Chinchilla is thek=1k{=}1special case of Skaling; the nine-parameter Farseer law is omitted (see(Li et al.,2025a)).

Parameter count.

The more heavily parameterized Farseer baseline does not remove the boundary failure by itself. Despite its additional degrees of freedom, it is less accurate than the Skaling law in most regimes ofTable˜1, and its largest errors concentrate on data extrapolation (MAPE4.134.13and4.454.45on the Farseer and SK-Grid full grids). This suggests that the gain is not simply a consequence of adding parameters, but of using a functional form whose inductive bias matches the observedNN–DDinteraction. Part of the difference may also be that the richer form is harder to fit; we tried several optimizers, including Farseer’s own pipeline, without obtaining a substantially better fit (Section˜9), so we report the best Farseer results we could obtain.

Coupling strength.

The benefit weakens when the observed coupling is closer to additive. On Farseer-code and on the original Chinchilla measurements inSection˜10, the fitted coupling is much closer to additive (k≈0.77k\approx 0.77–0.900.90), and the Skaling law correspondingly performs at roughly Chinchilla-level accuracy. This behaviour is expected from the nested form: because Chinchilla is recovered atk=1k=1, the Skaling law departs from the additive law when the data support a coupled surface and otherwise remains close to the additive fit.

Farseer data allocation frontier.

On Farseer data, the same interaction has a concrete allocation consequence. Using numerical loss gradients recover token-to-parameter ratio that decreases with compute, with fitted exponents−0.14-0.14and−0.15-0.15. This agrees with Skaling’s analytic exponent (−0.11-0.11) and has the opposite sign from Chinchilla’s near-flat prediction (+0.03+0.03). One order of magnitude beyond the data, the two prescriptions differ by roughly10×10\timesin the recommended token-to-parameter ratio (Figure˜6).

Allocation direction.

The sign of this allocation trend is dataset-specific rather than a universal consequence of the coupled form. For Farseer, the fitted Skaling exponents satisfyα<β\alpha<\beta, giving a decreasing analyticD⋆/N⋆D^{\star}/N^{\star}as compute grows. On SK-Grid, however, the fitted exponents satisfyα>β\alpha>\betaon both the full and L-shape grids, so the same closed-form optimum would increase the token-to-parameter ratio with compute. The robust conclusion is therefore that coupling changes large-scale allocation, while the direction of that change depends on the fitted data and architecture.

4.3Compute extrapolation

The cross-validation above held out gridcorners; we now test the operational case ofSection˜3.4: predicting the most computationally expensive runs of each recipe from computationally inexpensive ones.Table˜3reports the error on the112112held-out high-compute Farseer runs, split by training regime (the iso-ratio slices grouped into undertrained / optimal / overtrained thirds byD/ND/N), with every law refit only on the computationally inexpensive remainder. Skaling is the bestgloballaw in every regime and overall (pooled MAPE0.60±0.27%0.60{\pm}0.27\%, a3.9×3.9\timesreduction over the additive Chinchilla law and below the far more heavily parameterized Farseer law), and the most stable—its error never exceeds0.9%0.9\%in any regime. Chinchilla, by contrast, is strongly regime-dependent and weakest of all laws in the optimal band (3.47%3.47\%), so its pooled number hides where it fails. The only reference that edges the Skaling law is theper-ratiopower law, and only near the optimum (0.770.77vs0.88%0.88\%), where a one-dimensional law along a fixed ratio is naturally well behaved; it is fit separately per recipe and so cannot inform jointNN–DDallocation. Chinchilla exceed2.3%2.3\%, confirming the gain comes from the functional form rather than the protocol.

Table 3:Compute extrapolation on Farseer (Section˜3.4) by training regime: iso-ratio slices grouped into undertrained (D/N=1.8D/N{=}1.8–77), optimal (1010–4040) and overtrained (5656–158158) thirds, plus all1414setups. Predictive error on the112112highest-compute runs (held out88per slice), every law refit only on the computationally inexpensive runs.R2R^{2}is the per-group mean; MAPE (%) is mean±\pmstd over the slices in each group.Per-ratiois a single power law fit independently within each slice. Best per group inbold.

5Related Work

The modern practice of scaling laws rests on a small set of functional forms. Early work established that model loss falls predictably as a power law in scale(Hestness et al.,2017; Rosenfeld et al.,2019), andKaplan et al. (2020)modeled it jointly in model sizeNNand dataDD, treating the two as coupled.Hoffmann et al. (2022)recast the reducible loss as a purely additive sum of independent power laws,A/Nα+B/DβA/N^{\alpha}+B/D^{\beta}above an irreducible termEE, and this additive form became the field’s default. It underlies the familiar compute-optimal rule of roughly2020tokens per parameter, reaffirmed in a careful replication(Besiroglu et al.,2024), as well as the broader effort to predict the loss of large training runs from much smaller ones(Bi et al.,2024), a compute-extrapolation problem we study inSection˜3.4. The additive form’s convenience, however, conceals a strong assumption: thatNNandDDact on the loss independently, an identically zero mixed derivative∂2L/∂N​∂D\partial^{2}L/\partial N\,\partial D, so the marginal value of parameters does not depend on how much data they see.

Efforts to reconcile competing scaling results have targeted the fitting, not the form. When Kaplan- and Chinchilla-style studies disagreed on compute allocation, the gap was traced to parameter counting(Pearce and Song,2024)and to last-layer FLOP accounting, warmup, and optimizer tuning(Porian et al.,2024), all artifacts of the fitting procedure, resolved while leaving the additive form intact. Our evidence points elsewhere: with the fit held fixed, the form itself bends the wrong way at the edges of the grid, whereNNandDDare most imbalanced (Figures˜1and3), and a single coupling exponent is enough to straighten it.

Most closely related,Busbridge et al. (2025)use the same untied outer-exponent form,E+(A/Nα+B/Dβ)γE+(A/N^{\alpha}+B/D^{\beta})^{\gamma}, as the supervised scaling law that supplies teacher losses and the corresponding supervised student losses inside their distillation scaling law. Their main contribution is the distillation law: it predicts student loss from teacher loss, student size, and distillation tokens, captures the teacher–student capacity gap, and supports compute-optimal allocation between teacher and student. They state in a footnote that leaving the exponents untied improves fit and extrapolation relative to the tied Kaplan and Chinchilla cases, but do not report a controlled comparison of those supervised forms or study the resultingNN–DDinteraction. Our work makes that interaction the central question and provides direct evidence through boundary residuals, mixed derivatives, controlled predictive comparisons, and sampling and extrapolation experiments.

Richer functional forms can also capture the interaction, at a higher cost. The Farseer law(Li et al.,2025a)makes the data exponent and amplitude depend onNN, introducing anNN–DDinteraction through nine fitted parameters; the added flexibility is harder to fit, and in our experiments it is less accurate than Skaling at the grid boundaries. Scaling laws have also been extended along other axes: to repeated data, mapped to effective token and parameter counts within the additive form(Muennighoff et al.,2023), and to data mixtures, which add domain weights as inputs to the loss(Ye et al.,2024; Shukor et al.,2026). Skaling instead isolates the effect of a single parameter that couplesNNandDD, and we expect similar couplings to matter along other scaling axes.

6Conclusion

Additive scaling laws inherently assume that model size and data volume act independently. As we have shown, this assumption breaks down exactly where accurate predictions matter most: the imbalanced extremes of the training grid. The Skaling law resolves this with a single coupling exponent, a natural and straightforward extension that captures theNN–DDinteraction already present in the empirical loss surface. By simply allowing these variables to interact, this single parameter eliminates boundary bias and predicts performance more accurately than both the additive baseline and richer nine-parameter formulations. Furthermore, it yields a strictly sub-additive coupling with a closed-form optimal allocation, a result independently verified by our model-free gradient estimates.

Crucially, because this coupled form is anchored by the edges of the parameter space rather than its interior, we do not need to measure the entire surface to reconstruct it. A sparse, L-shaped grid of inexpensive boundary runs is entirely sufficient to recover the full-grid law at a fraction of the computational cost. Reliable scaling prediction therefore no longer requires massive, dense empirical sweeps, dramatically lowering the barrier to principled compute allocation. We expect this fundamental coupling dynamic to apply along other axes of model scaling, presenting a clear direction for future work.

References

• Ardalani et al. (2026)Newsha Ardalani, Feiyang Kang, Michael Kuchnik, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, and Carole-Jean Wu.How text quality interventions reshape neural scaling laws for LLMs: Empirical study.InThe Fourteenth International Conference on Learning Representations, 2026.https://openreview.net/forum?id=ZC5QBfdOw7.
• Besiroglu et al. (2024)Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You.Chinchilla scaling: A replication attempt.arXiv preprint arXiv:2404.10102, 2024.
• Bi et al. (2024)Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al.Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024.
• Busbridge et al. (2025)Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb.Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025.
• Hansen (2016)Nikolaus Hansen.The cma evolution strategy: A tutorial.arXiv preprint arXiv:1604.00772, 2016.
• Hestness et al. (2017)Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409, 2017.
• Hoffmann et al. (2022)Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022.
• Kaplan et al. (2020)Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.
• Lancaster and Salkauskas (1981)Peter Lancaster and Kes Salkauskas.Surfaces generated by moving least squares methods.Mathematics of computation, 37(155):141–158, 1981.
• Li et al. (2025a)Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, et al.Predictable scale: Part ii, farseer: A refined scaling law in large language models.arXiv preprint arXiv:2506.10972, 2025a.
• Li et al. (2025b)Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, et al.Predictable scale: Part i, step law–optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025b.
• Li et al. (2024)Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al.Datacomp-lm: In search of the next generation of training sets for language models.Advances in Neural Information Processing Systems, 37:14200–14282, 2024.
• Muennighoff et al. (2023)Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.Scaling data-constrained language models.Advances in Neural Information Processing Systems, 36, 2023.
• Pearce and Song (2024)Tim Pearce and Jinyeop Song.Reconciling kaplan and chinchilla scaling laws.arXiv preprint arXiv:2406.12907, 2024.
• Porian et al. (2024)Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon.Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146, 2024.
• Rosenfeld et al. (2019)Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit.A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673, 2019.
• Shukor et al. (2026)Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin.Scaling laws for optimal data mixtures.Advances in Neural Information Processing Systems, 38:129554–129579, 2026.
• Videau et al. (2024)Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez-Paz.Meta Lingua: A minimal PyTorch LLM training library, 2024.https://github.com/facebookresearch/lingua.
• Ye et al. (2024)Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu.Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024.

\beginappendix

7Differential analyses

7.1Numerical gradients

We estimate derivatives of the loss surface directly from the measured runs. At each grid point, we estimate∂z/∂xj\partial z/\partial x_{j}using two mesh-free procedures: a local moving least-squares estimator (MLS) and a global Gaussian-process estimator (GP).

Moving least squares (MLS).

The local estimator fits a truncated Taylor expansion of the surface in a neighbourhood of each query pointx⋆x_{\star}(Lancaster and Salkauskas,1981). To estimate the derivatives at a specific target pointx⋆x_{\star}, we locally approximate the loss surface using a multivariate polynomial of degreepp. This approach comes directly from Taylor expansion: if we fit a local polynomial, its coefficients correspond mathematically to the function’s derivatives of various orders.

Letxxrepresent the spatial coordinates, and letΔ​xi=xi−x⋆\Delta x_{i}=x_{i}-x_{\star}denote the centered distance to a neighboring point. By Taylor’s theorem, the local loss valuez​(xi)z(x_{i})can be expanded as:

z​(xi)≈c0+g⊤​Δ​xi+12​Δ​xi⊤​H​Δ​xi+…z(x_{i})\approx c_{0}+g^{\top}\Delta x_{i}+\frac{1}{2}\Delta x_{i}^{\top}H\Delta x_{i}+\dotswhere the polynomial expansion continues with higher-order basis terms up to degreepp. In this formulation,c0c_{0}is the local constant, the vectorggis the gradient∇z​(x⋆)\nabla z(x_{\star}), andHHis the Hessian matrix containing our target cross-derivative.

To solve for these unknowns simultaneously, we flatten all the coefficients of this expansion into a single vectorcc. We then determine the optimal coefficient estimatec^\hat{c}by fitting this polynomial to thekknearest neighbors using distance-weighted ridge regression:

c^=(Φ⊤​W​Φ+λ​I)−1​Φ⊤​W​z\hat{c}=\big(\Phi^{\top}W\Phi+\lambda I\big)^{-1}\Phi^{\top}Wz The components of this generalized regression map directly to the Taylor expansion:

• •Φ\Phiis the design matrix. Each row corresponds to a neighbor and contains its evaluated polynomial basis terms up to degreepp(for example:11,Δ​xi,1\Delta x_{i,1},Δ​xi,2\Delta x_{i,2},Δ​xi,12\Delta x_{i,1}^{2}, the interaction termΔ​xi,1​Δ​xi,2\Delta x_{i,1}\Delta x_{i,2}, and higher-order combinations).
• •WWis a diagonal weight matrix wherewi=exp⁡(−‖Δ​xi‖2/σ2)w_{i}=\exp\!\big(\!-\|\Delta x_{i}\|^{2}/\sigma^{2}\big). This acts as a Gaussian filter, ensuring that points closer tox⋆x_{\star}exert more influence on the local fit. The bandwidthσ\sigmascales dynamically with the neighborhood radius.
• •λ​I\lambda Iadds a small ridge regularization penalty to keep the matrix inversion numerically stable, which is crucial if the scattered local data points are poorly distributed.

Once we solve forc^\hat{c}, extracting the necessary derivatives is straightforward. We simply read them directly from their designated slots in the vector: the first-order block provides the gradient estimate, and the coefficient corresponding to theΔ​xi,1​Δ​xi,2\Delta x_{i,1}\Delta x_{i,2}basis term provides the exact mixed cross-derivative required for our analysis. Because this Moving Least Squares (MLS) approach relies entirely on local geometry, its accuracy is naturally sensitive to local noise, the chosen neighborhood sizekk, and the polynomial degreepp.

Gaussian process (GP).

The global estimator fits a single Gaussian process to all points and differentiates its posterior mean in closed form. We model the empirical loss targetsz∈ℝn×1z\in\mathbb{R}^{n\times 1}as a smooth function of the input coordinates, corrupted by some observation noiseσn2\sigma_{n}^{2}. In standard GP regression, the predicted log-lossz^​(x⋆)\hat{z}(x_{\star})at a target pointx⋆x_{\star}is given directly by the posterior mean:

z^​(x⋆)=μ​(x⋆)=k​(x⋆,X)​α\hat{z}(x_{\star})=\mu(x_{\star})=k(x_{\star},X)\alphaHere,k​(x⋆,X)∈ℝ1×nk(x_{\star},X)\in\mathbb{R}^{1\times n}is a row vector of kernel evaluations between the target point and allnntraining data points. The learned weight vector is defined asα=(K+σn2​I)−1​z∈ℝn×1\alpha=(K+\sigma_{n}^{2}I)^{-1}z\in\mathbb{R}^{n\times 1}, whereK∈ℝn×nK\in\mathbb{R}^{n\times n}is the dense covariance matrix of the training inputs andIIis then×nn\times nidentity matrix.

The key advantage of using a GP is that differentiation is a linear operator, meaning the derivative of a GP is simply another GP. We can compute the estimated scalar gradient of the loss surface in closed form by directly differentiating the predicted surfacez^​(x⋆)\hat{z}(x_{\star})with respect to the input dimensions. Because the weightsα\alphaare independent ofx⋆x_{\star}, the derivative simply applies to the1×n1\times nrow vector:

∂z∂xj^​(x⋆)=∂z^​(x⋆)∂xj=∂k​(x⋆,X)∂xj​α\widehat{\frac{\partial z}{\partial x_{j}}}(x_{\star})=\frac{\partial\hat{z}(x_{\star})}{\partial x_{j}}=\frac{\partial k(x_{\star},X)}{\partial x_{j}}\alpha Because standard kernel functions are analytically differentiable, this computation is exact. In this work, we are using a Radial Basis Function (RBF) kernel with length-scaleℓj\ell_{j}, the derivative is∂k​(x,x′)/∂xj=k​(x,x′)​(xj′−xj)/ℓj2\partial k(x,x^{\prime})/\partial x_{j}=k(x,x^{\prime})(x^{\prime}_{j}-x_{j})/\ell_{j}^{2}. Composite kernels can be differentiated using standard sum and product rules, provided that any additive white-noise terms are excluded from the predictive kernel.

Additionally, the GP provides a closed-form predictive variance for this gradient, quantifying our estimation uncertainty. It is calculated by taking the scalar prior variance of the gradient and subtracting the information gained from the observed data:

Var​[∂z∂xj​(x⋆)]=∂2k​(x,x′)∂xj​∂xj′|x=x′=x⋆−∂k​(x⋆,X)∂xj​(K+σn2​I)−1​∂k​(X,x⋆)∂xj\mathrm{Var}\!\left[\frac{\partial z}{\partial x_{j}}(x_{\star})\right]=\left.\frac{\partial^{2}k(x,x^{\prime})}{\partial x_{j}\,\partial x^{\prime}_{j}}\right|_{x=x^{\prime}=x_{\star}}-\frac{\partial k(x_{\star},X)}{\partial x_{j}}\,(K+\sigma_{n}^{2}I)^{-1}\,\frac{\partial k(X,x_{\star})}{\partial x_{j}}Notice how the matrix dimensions elegantly resolve to a scalar: a1×n1\times nvector multiplied by ann×nn\times nmatrix, multiplied by ann×1n\times 1column vector∂k​(X,x⋆)∂xj\frac{\partial k(X,x_{\star})}{\partial x_{j}}.

Unlike MLS, which requires manual tuning of neighborhood sizes and polynomial degrees, the GP explicitly models observation noise and automatically tunes its length-scales and noise levels by maximizing the marginal likelihood.

7.2Form of the coupling

The non-zero mixed derivative ofSection˜2constrains the functional form: it should preserve the marginal power-law behaviour visible in the first derivatives without making the two axes additively separable. The Skaling form (Equation˜3) satisfies this. Writingu=A​N−α+B​D−βu=AN^{-\alpha}+BD^{-\beta}, its mixed derivative is

∂2L∂N​∂D=k​(k−1)​uk−2​α​A​N−α−1​β​B​D−β−1.\frac{\partial^{2}L}{\partial N\,\partial D}=k(k-1)\,u^{k-2}\,\alpha A\,N^{-\alpha-1}\,\beta B\,D^{-\beta-1}.Thusk=1k=1recovers the additive Chinchilla law and forces the mixed derivative to vanish, while anyk≠1k\neq 1introduces an interaction. In the empirically relevant case0<k<10<k<1, the mixed derivative is negative, matching the sign observed inFigure˜3. At the same time, the first derivatives,

∂L∂N=−k​α​A​N−α−1​uk−1,∂L∂D=−k​β​B​D−β−1​uk−1,\frac{\partial L}{\partial N}=-k\alpha A\,N^{-\alpha-1}u^{k-1},\qquad\frac{\partial L}{\partial D}=-k\beta B\,D^{-\beta-1}u^{k-1},remain dominated by the same-variable power-law factors, with cross-variable dependence entering only through the shared factoruk−1u^{k-1}. Consequently, the Skaling law can appear nearly separable at first order while still allowing the non-zero mixed derivative indicated by the data.

Why a multiplicative coupling, not an additive interaction term.

A natural alternative to the Skaling law would be to keep the Chinchilla law additive and append a separate product term,

L=A​N−α+B​D−β+G​N−μ​D−ν+E.L=AN^{-\alpha}+BD^{-\beta}+G\,N^{-\mu}D^{-\nu}+E.This term can create a non-zero mixed derivative, but its sign creates an immediate constraint. For this model, the mixed derivative and the contribution of the interaction term to the size derivative are

∂2L∂N​∂D=μ​ν​G​N−μ−1​D−ν−1,∂L∂N​(G​N−μ​D−ν)=−μ​G​N−μ−1​D−ν.\frac{\partial^{2}L}{\partial N\,\partial D}=\mu\nu\,G\,N^{-\mu-1}D^{-\nu-1},\qquad\frac{\partial L}{\partial N}\!\left(GN^{-\mu}D^{-\nu}\right)=-\,\mu\,G\,N^{-\mu-1}D^{-\nu}.Thus the sign ofGGcontrols two quantities in opposite ways. Matching the observed negative mixed derivative requiresG<0G<0. But withG<0G<0, the interaction term contributes positively to∂L/∂N\partial L/\partial N, opposing the desired monotonic decrease with model size. This is not just a local inconvenience: the fitted mixed-derivative decay impliesμ≈0.1\mu\approx 0.1, smaller than the main size exponentα\alpha, so this positive contribution decays more slowly inNNthan the leading negative term. At largeNNand smallDD, it can therefore dominate and make∂L/∂N>0\partial L/\partial N>0, predicting that loss increases when the model becomes larger. ChoosingG>0G>0avoids this monotonicity failure, but then the mixed derivative becomes positive and the desired synergy disappears. Thus a single additive product term cannot simultaneously preserve monotonicity and match the observed negative interaction.

Skaling avoids this sign conflict because the interaction is introduced by a positive multiplicative factor rather than by a separate signed term. Its size gradient is

∂L∂N=−k​α​A​N−α−1​uk−1,\frac{\partial L}{\partial N}=-k\,\alpha A\,N^{-\alpha-1}u^{k-1},which is negative for everyk>0k>0. The dependence on data enters only through the positive factoruk−1u^{k-1}. When0<k<10<k<1, increasingDDdecreasesuu, which increasesuk−1u^{k-1}and therefore increases the magnitude of the already-negative size gradient. This yields∂2L/∂N​∂D<0\partial^{2}L/\partial N\partial D<0without ever changing the sign of∂L/∂N\partial L/\partial N. In this parameterization, monotonicity and synergy are compatible by construction.

The same factor also explains the asymmetric first-order slopes inFigure˜2. LetwD=B​D−β/uw_{D}=BD^{-\beta}/uandwN=A​N−α/uw_{N}=AN^{-\alpha}/udenote the data and size shares of the inner sum. Differentiatingln⁡|∂L/∂N|\ln|\partial L/\partial N|andln⁡|∂L/∂D|\ln|\partial L/\partial D|givesγN=(1−k)​β​wD\gamma_{N}=(1-k)\,\beta\,w_{D}andγD=(1−k)​α​wN\gamma_{D}=(1-k)\,\alpha\,w_{N}, soγN/γD=(β/α)​(wD/wN)\gamma_{N}/\gamma_{D}=(\beta/\alpha)\,(w_{D}/w_{N}). This ratio exceeds one wheneverβ>α\beta>\alphaand the inner sum is data-leaning, reproducing the measuredγN≈0.13>γD≈0.07\gamma_{N}\approx 0.13>\gamma_{D}\approx 0.07through the single coupling exponent rather than through a skewed interaction term.

7.3Empirical optimal token to parameter ratio

The numerical gradients also provide an empirical estimate of the compute-optimal allocation, without fitting a parametric scaling law. For a fixed training budgetC=6​N​DC=6ND, feasible configurations lie on a single iso-compute curve: increasing model size requires decreasing the number of training tokens, and conversely. The empirical optimum for that budget is the lowest-loss point on this curve,

minN,DL(N,D)s.t.6ND=C.\min_{N,D}\;L(N,D)\quad\mathrm{s.t.}\quad 6ND=C. For an interior optimum on a differentiable loss surface, the Lagrangian

𝒥​(N,D,λ)=L​(N,D)+λ​(6​N​D−C)\mathcal{J}(N,D,\lambda)=L(N,D)+\lambda(6ND-C)has stationarity conditions

∂L∂N=−6​λ​D,∂L∂D=−6​λ​N.\frac{\partial L}{\partial N}=-6\lambda D,\qquad\frac{\partial L}{\partial D}=-6\lambda N.Dividing the two equations cancels the multiplier and leaves a condition on the loss gradients alone:

N​∂L∂N=D​∂L∂D⟺∂ln⁡L∂ln⁡N=∂ln⁡L∂ln⁡D.N\,\frac{\partial L}{\partial N}=D\,\frac{\partial L}{\partial D}\quad\Longleftrightarrow\quad\frac{\partial\ln L}{\partial\ln N}=\frac{\partial\ln L}{\partial\ln D}.(5)Thus, at a compute-optimal point, a1%1\%increase in model size and a1%1\%increase in data have the same signed effect on the loss. Equivalently, the directional derivative along the tangent to the iso-compute curve must vanish. This is a constrained stationarity condition: the full gradient is not required to vanish, since increasing the total compute would still reduce loss.

To solve this condition non-parametrically, we write the loss in log coordinates,

x=ln⁡N,y=ln⁡D,z​(x,y)=ln⁡L​(N,D).x=\ln N,\qquad y=\ln D,\qquad z(x,y)=\ln L(N,D).The optimality condition becomes∂z/∂x=∂z/∂y\partial z/\partial x=\partial z/\partial y. We locate this equality using the numerical log-slopes fromSection˜7.1; the GP and MLS estimators give two fully independent estimates of the frontier through the same construction.

GP surrogate.

The optimal compute allocation occurs exactly where the partial derivatives balance. Because our empirical data consists of discrete points, we cannot locate this exact equilibrium directly from the grid. Instead, we use the GP posterior mean to provide a continuous, differentiable surface. We can then simply track the points on this surface where the difference between the two gradients is exactly zero. This continuous zero-crossing boundary directly yields the optimal token-to-parameter ratio for any compute budget. If multiple roots exist for a given budget, we selects the minimum with the lowest predicted loss.

MLS estimates.

The local estimator is itself a mesh-free surrogate, so we follow exactly the same procedure: around any query point it fits a weighted polynomial to thekknearest runs, from which we read the gradient difference on a gride of point track its zero-crossing as above.

In both cases, trackingD⋆/N⋆D^{\star}/N^{\star}asCCvaries gives an empirical token-to-parameter ratio, which we compare with the closed-formRo​p​tR_{opt}ofSection˜8on the Farseer data (Figure˜6). The two independent estimates agree qualitatively: both recover a ratio that decreases with compute, closely following Skaling’sRo​p​tR_{opt}, whereas Chinchilla predicts a nearly flat ratio. Fitting each empirical frontier as a power lawD⋆/N⋆∝CmD^{\star}/N^{\star}\propto C^{m}givesm≈−0.14m\approx-0.14for the GP andm≈−0.15m\approx-0.15for MLS, close to Skaling’s exponent of−0.11-0.11and opposite in sign to Chinchilla’s+0.03+0.03. When extrapolated one order of magnitude beyond the data, to2×10252\times 10^{25}FLOPs, the allocations differ by more than10×10\times: Chinchilla approaches∼380\sim\!380tokens per parameter, while the empirical fits and the Skaling law fall to2020–4040. The coupling therefore has a material effect on large-scale allocation.

Figure 6:Empirical compute-optimal token-to-parameter ratio on Farseer, recovered without a parametric fit.The optimumD⋆/N⋆D^{\star}/N^{\star}is located where the log-slopes balance (∂ln⁡L/∂ln⁡N=∂ln⁡L/∂ln⁡D\partial\ln L/\partial\ln N=\partial\ln L/\partial\ln D), using a global GP surrogate and a local MLS surrogate (both evaluated as continuous mesh-free fields), and compared with the closed-formRo​p​tR_{opt}of Skaling and Chinchilla fitted on the same(N,D)(N,D)data.*Left:*both empirical optima, over the run cloud, follow Skaling more closely than the nearly flat Chinchilla prediction.*Right:*power-law fits of each empirical frontier (dotted) against the analytic laws, extrapolated to2×10252\times 10^{25}FLOPs (shaded==data range); the empirical exponents (−0.14-0.14,−0.15-0.15) are close to Skaling (−0.11-0.11) and have the opposite sign from Chinchilla (+0.03+0.03).

8Compute-optimal training

The primary utility of a scaling law is to determine the optimal allocation of a given compute budgetCCbetween model sizeNNand data volumeDD. We formally define the training compute budget asC=6​N​DC=6ND. To identify the compute-optimal configuration, we minimize the Skaling loss ofEquation˜3subject to this budget constraint.

By substitutingD=C6​ND=\frac{C}{6N}, we can isolate the inner additive term as a function ofNN, definingZ​(N)=ANα+B​(C6​N)−βZ(N)=\frac{A}{N^{\alpha}}+B\left(\frac{C}{6N}\right)^{-\beta}. Our loss objective then simplifies to:

L​(N)=Z​(N)k+EL(N)=Z(N)^{k}+E(6) Taking the derivative with respect toNNyields:

d​Ld​N=k⋅Z​(N)k−1⋅Z′​(N)\frac{dL}{dN}=k\cdot Z(N)^{k-1}\cdot Z^{\prime}(N)(7) SinceZ​(N)Z(N)is a sum of strictly positive terms and our empirical fits consistently yieldk>0k>0, the scaling factork⋅Z​(N)k−1k\cdot Z(N)^{k-1}is non-zero. This means minimizing the loss strictly requires thatZ′​(N)=0Z^{\prime}(N)=0, precisely the stationarity condition of the additive Chinchilla law(Hoffmann et al.,2022); the Skaling law therefore inherits Chinchilla’s compute-optimal allocation unchanged. SolvingZ′​(N)=−α​A​N−α−1+β​B​(6/C)β​Nβ−1=0Z^{\prime}(N)=-\alpha A\,N^{-\alpha-1}+\beta B\,(6/C)^{\beta}N^{\beta-1}=0givesNα+β=α​Aβ​B​(C/6)βN^{\alpha+\beta}=\frac{\alpha A}{\beta B}(C/6)^{\beta}; substitutingD∗=C/(6​N∗)D^{*}=C/(6N^{*})then yields the optimal token-to-parameter ratio (i.e.,D∗/N∗D^{*}/N^{*}):

Ro​p​t=6β−αα+β​(β​Bα​A)2α+β​Cα−βα+βR_{opt}=6^{\frac{\beta-\alpha}{\alpha+\beta}}\left(\frac{\beta B}{\alpha A}\right)^{\frac{2}{\alpha+\beta}}C^{\frac{\alpha-\beta}{\alpha+\beta}}(8) This reveals that ifα≈β\alpha\approx\beta(as observed in Chinchilla), the optimal ratioD/ND/Nremains constant across scales. More broadly, the Skaling law cleanly separates the shape of the loss landscape (controlled bykk) from the location of the optimal allocation (Z′​(N)=0Z^{\prime}(N)=0): the monotone outer mapx↦xk+Ex\mapsto x^{k}+Erescales the loss but leaves its minimizer unchanged, so the Skaling law gains expressivity without sacrificing tractability.

9Fitting Details

9.1Challenges

Despite the small number of parameters, fitting scaling laws to empirical loss data is a surprisingly fragile non-convex optimization problem. Several structural factors contribute to this instability. First, the parameters operate on vastly different numerical scales. For instance, the large magnitudes of coefficientsAAandBBcontrast sharply with the fractional exponentsα\alphaandβ\beta, ill-conditioning the problem for standard gradient-based optimizers like L-BFGS. Second, the optimization landscape is notoriously flat. Because the loss decreases logarithmically, optimizers are prone to halting prematurely, a vulnerability that is worsened when relying on finite-difference gradient approximations. Finally, the parameters exhibit strong compensatory behaviors; the irreducible lossEE, in particular, is difficult to accurately estimate because other coefficients can easily shift to offset its value. This creates wide valleys of distinct local minima that yield nearly identical overall surface fits.

We fit every law by minimizing the objective in log space, which (as in Chinchilla) absorbs much of this scale disparity. Within this setup, a carefully configured L-BFGS, paired with basin-hopping and automatic differentiation(Besiroglu et al.,2024; Shukor et al.,2026), and the scale-independent, gradient-free CMA-ES reach equally good fits. The difference we observe is practical rather than in final quality: L-BFGS must be carefully tuned (initialization, restarts, and the log-space objective) to reach them reliably, whereas CMA-ES attains the same solutions out of the box, without any such tweaks.

9.2Hyperparameters

We implement two global optimization strategies. The first, L-BFGS-B with basin-hopping, pairs a gradient-based local optimizer (L-BFGS-B with analytical gradients via autograd) with multi-start basin-hopping for global search; starting points are drawn from a Sobol quasi-random sequence over the bounded parameter space. The second, BIPOP-CMA-ES(Hansen,2016), is a gradient-free evolutionary strategy; we use doubled population size, active CMA, and 9 BIPOP restarts. As discussed inSection˜9, both reach equally good fits; all results reported in this paper use L-BFGS-B with basin-hopping, as it is more common in the literature(Besiroglu et al.,2024; Shukor et al.,2026). Every law minimizes a Huber loss (δ=0.05\delta=0.05) in log space; coefficientsAAandBBare optimized in log scale to absorb their large dynamic range. The per-law configurations are summarized inTable˜4.

Table 4:Per-law parameter bounds and fitting configuration. All laws use20002000basin-hopping restarts, a log-space Huber objective, and analytic (autograd) gradients;Paramsis the number of free parameters andLog paramsthose optimized in log scale. Farseer uses the ExpExpExp parameterization of(Li et al.,2025a).#### Fitting Farseer.

We fit Farseer, like every other law, with this common direct optimizer, so the comparison reflects the functional form and not the fitting recipe. This is also a practical necessity: Farseer’s original pipeline estimates its components from consecutive-DDdifferences at fixedNN, which requires several evenly spacedDDvalues per model size. The sparse L-shape grids and the scattered, non-gridded datasets do not provide this, and even on a fully gridded dataset the cross-validation hold-outs remove points and break the difference structure the pipeline relies on; direct optimization is the only procedure that applies uniformly across our settings. Run under the same cross-validation, the original pipeline did not give significantly better predictions: its errors lay within the fold-to-fold standard deviation of the direct fit in every regime except larger-DDextrapolation, where it only matched Skaling. We were unable to fit this form substantially better with any of these procedures, so we report the common direct fit as the best Farseer result we obtained.

10Additional datasets

We repeat the cross-validation protocol ofSection˜3.3on two further datasets (Table˜5).Farseer-codeis the code-domain counterpart of the Farseer grid(Li et al.,2025a):117117runs over99model sizes (201201M–3.183.18B) and2020token budgets (22B–128128B), spanning computeC=6​N​DC{=}6NDfrom2.4×10182.4\times 10^{18}to2.4×10212.4\times 10^{21}FLOPs. Being grid-structured, it admits both the full-grid and L-shape splits; its larger-NNand larger-DDhold-outs are the largest model sizes and longest token horizons at the top edge of the grid. TheChinchillaloss measurements(Besiroglu et al.,2024)are245245scattered(N,D)(N,D)points covering5757M–16.216.2B parameters and245245M–318318B tokens (1.4×10181.4\times 10^{18}–1.3×10221.3\times 10^{22}FLOPs); they do not lie on a regular grid, so only the full-grid (random) split applies (no L-shape), with the largest-NN(16.216.2B) and largest-DD(318318B-token) points forming the two extrapolation sets.

Table 5:Fit quality and predictive error on two additional datasets, following the protocol ofTable˜1: interpolationR2R^{2}(↑\uparrow) and MAPE (%,↓\downarrow; mean±\pmstd over 5 CV folds) on interpolation and two extrapolation regimes (largerNN, largerDD). Farseer-code is grid-structured (full and L-shape splits); the Chinchilla measurements are not gridded, so only a full-grid split applies. Best per column within each block inbold.Table˜6lists the corresponding coefficients for Farseer-code and the Chinchilla data. The same pattern holds, with the Skaling law fittingk<1k<1and a smallerEEthan Chinchilla, though the coupling is weaker here (k≈0.77k\approx 0.77–0.900.90), in line with the more mixed accuracy gains inTable˜5.

Table 6:Fitted coefficients (mean±\pmstd over55CV folds) behindTable˜5. Chinchilla is thek=1k{=}1special case of Skaling.

11Pre-training Details

11.1Model

The configurations in this section describe the SK-Grid, an internal set of pretraining runs; for the Farseer grid we use the released runs and refer the reader to(Li et al.,2025a)for details. SK-Grid comprises125125runs across1414model sizes (Table˜8).

Architecture.

The models span1414sizes from134134M to4.94.9B parameters, scaled by growing width and depth together: the model dimensiondmodeld_{\text{model}}increases from672672to32643264and the depth from77to3434layers. All models use the Llama 3 tokenizer, with a vocabulary of128,256128{,}256.

Token budgets.

Each size is trained on a geometric ladder of token budgets spanning316316M to316316B tokens, five budgets per decade. Per-run compute is capped, so larger models are trained on fewer budgets: the smallest models cover all1616horizons while the largest is trained on a single budget, producing the staircase grid whose row counts appear in the last column ofTable˜8.

11.2Hyperparameters

The two run-specific hyperparameters, the global batch sizeBB(tokens) and the peak learning rateη\eta, follow the StepLaw prescription(Li et al.,2025b)as power laws in the per-token computeFF(FLOPs per token, a monotone proxy for model sizeNN) and the token budgetDD, with coefficients refit to our own setup:

B=896.07​F0.231,η=0.0709​F−0.4303​D0.2785.B=896.07\,F^{0.231},\qquad\eta=0.0709\,F^{-0.4303}\,D^{0.2785}.(9)Because the shape of the loss surface is highly sensitive to these tuning choices, different parameterizations of hyperparameters in the training recipe can alter the apparentNN–DDinteraction and shift the inferred compute-optimal ratio. For instance, a broadly mistuned grid might artificially dampen the measured interaction or skew the optimal ratio. This sensitivity also implies that differences in hyperparameter policies, such as the original Chinchilla recipe compared to our StepLaw-tuned SK-Grid, partly explain why different datasets exhibit varying degrees ofNN–DDcoupling. Consequently, cross-dataset comparisons inherently reflect the specific training recipes aand data used to generate each grid.

All remaining settings are fixed across the grid (Table˜8), based on the defaults of the Meta Lingua framework(Videau et al.,2024). Every run uses the same data mixture:60%60\%DCLM-Edu web text(Li et al.,2024),30%30\%code, and10%10\%math. The loss we fit is the validation loss on a held-out split of this mixture. For the other datasets, refer to their respective papers: Farseer(Li et al.,2025a)and Chinchilla(Besiroglu et al.,2024; Hoffmann et al.,2022).

Table 7:SK-Grid model configurations. Table 8:Fixed configuration shared by all SK-Grid runs; per-run LR and batch size followEquation˜9.

12Dominated-pair fitting

A recurring difficulty above is that the irreducible lossEEis only weakly identified: interior points constrain it poorly and the other coefficients shift to absorb it. Thedominated-pairfit removesEEfrom the objective entirely. For any ordered pair in which configurationiidominatesjj(Fi≥FjF_{i}\geq F_{j},Di≥DjD_{i}\geq D_{j}andLi<LjL_{i}<L_{j}), the additive floor cancels in the loss difference:

Lj−Li=(A​Fj−α+B​Dj−β)k−(A​Fi−α+B​Di−β)k,L_{j}-L_{i}=\big(A\,F_{j}^{-\alpha}+B\,D_{j}^{-\beta}\big)^{k}-\big(A\,F_{i}^{-\alpha}+B\,D_{i}^{-\beta}\big)^{k},(10)which no longer involvesEE(Chinchilla is thek=1k{=}1case). We fit the shape parameters(A,B,α,β,k)(A,B,\alpha,\beta,k)on all such pairwise differences, then recover the floor asE=mediank⁡(Lk−(A​Fk−α+B​Dk−β)k)E=\operatorname{median}_{k}\big(L_{k}-(A\,F_{k}^{-\alpha}+B\,D_{k}^{-\beta})^{k}\big), decoupling the reducible shape from the constant offset.

Table˜9shows that dominated-pair fitting serves primarily as a correction for the additive Chinchilla law. Across both datasets, it reduces most of Chinchilla’s extrapolation errors, yielding the clearest gains in boundary regimes where accurately fitting the irreducible loss floor matters most. For example, on the full grids, Chinchilla’s far-extrapolation error drops from2.46%2.46\%to0.79%0.79\%on Farseer and from5.17%5.17\%to3.67%3.67\%on SK-Grid. The L-shape grids show similar improvements across all reported metrics.

This consistent pattern suggests that much of Chinchilla’s extrapolation error stems from weak identification of the floorEE; onceEEis removed from the objective, the constant offset no longer distorts the fit of the reducible component. Furthermore, because Chinchilla’s functional form does not perfectly capture the global shape of the loss surface, the dominated-pair objective implicitly reweights the data, forcing the optimization to anchor more heavily on the extreme boundary points of the grid. In contrast, this correction does not consistently improve Skaling.

Table 9:Dominated-pair (“+dom”) fitting versus the default joint L-BFGS fit, for Chinchilla and Skaling on every dataset and grid. Each “+dom” row is the dominated-pair fit (Equation˜10) of the law above it. Columns: interpolationR2R^{2}(↑\uparrow) and MAPE (%,↓\downarrow; mean±\pmstd over 5 CV folds) on interpolation and the larger-NN, larger-DD, and far regimes. The Chinchilla measurements are not gridded, so they have no L-shape and no far set; the Farseer-code far set is a single held-out run and is omitted ( – ). Both variants come from one fitting run, so the baselines can differ marginally fromTable˜1. Best per column within each panel inbold(far excluded where not comparable).