@kimiahmdh: Can we tell whether data domains cooperate or compete during pretraining? Adding code to the mix makes models better at…

Can we tell whether data domains cooperate or compete during pretraining?

Adding code to the mix makes models better at math, while some other combinations hurt each other. We call this data synergy. Turns out you can incorporate data synergy into scaling laws and estimate it

Here’s a motivating example: Fit scaling laws on validation loss of a coding benchmark, across models trained on different data mixtures.

a) With the same data scaling exponent for all mixtures: bad fit! b) Let the data exponent depend on the mixture: near perfect fit!

We propose domain-aware scaling laws that account for two types of synergy:

First-order synergy: every (domain, benchmark) pair gets a coefficient that modifies the data scaling exponent.

Second-order: additional bonus tokens that appear when two domains co-occur

No training sweep required. We estimate the synergies observationally, from open-data models that already exist, and fit the scaling laws.

What do we find? Math data is a massive accelerant for coding benchmarks, meanwhile books and encyclopedias interfere with code benchmarks.

The strongest pairwise synergies, shared across benchmarks are:

Code × Science, Code × Math Math × Books: negative (they interfere with each other)

Takeaway: if your target benchmark benefits one domain, don’t spend tokens on its interfering partners.

Our domain-aware laws predict the performance of held-out models more accurately across all benchmarks.

Once you can predict performance from a mixture, you can optimize it: we use the laws to design data mixtures that perform well on a given task.

Joint work with my amazing collaborators @LesterMackey and @elmelis

Paper: https://arxiv.org/abs/2607.11052 Website: http://data-syn.github.io

To appear at #COLM2026!