@WilliamBarrHeld: There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together ? For Mari…

There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?

For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.

Here’s the work between downloading those and training a model 🧵

The collection includes books, math problems, genetic sequences, and SVG illustrations.

We build on datasets from @allen_ai, @NVIDIAAI, @huggingface, @hplt_eu, @AiEleuther, and many smaller contributors. An enormous amount of community work!

Different datasets often contain versions of the same webpage (including synthetic rewrites that retain much of the original text).

We want repetition to be something we control in our experiments, so we deduplicate globally.

This removed 2.13T tokens from the collection!

When ingesting this much data, we want to make sure we don’t accidentally train on test (we’ve unfortunately done it before…)!

We filter the entire corpus for any 13-gram overlap with our evals (including our development set evals, since they guide our research decisions!).

Now we have “normalized” datasets. How much should we sample from each?

Suppose we want more math. Some comes from dedicated math datasets, but plenty is buried in web crawls alongside recipes and advertisements!

We need to group documents by content, regardless of source.

We embed each document with Harrier, an OSS embedding model from @MSFTResearch. We keep the embeddings so we can try different categorization schemes without running the (expensive) embedding model over everything again.

For topic clusters, we cluster documents into 40 topics.

We assign five quality bands within each topic, giving us 200 sampling groups.

We don’t filter any data yet, since the best mix depends on compute! Short runs may only use the best data. In longer runs, more variety—even at lower quality—can help avoid excessive repetition.

To choose the sampling weights, we train small models on different mixtures and measure their performance.

We scale down each group’s available data too (otherwise, the small models would see fresh documents where the large model would already be repeating them!)

We fit a regression predicting evaluation performance from those mixture weights, then use it to optimize the mixture.

We also compare against sampling in proportion to token counts at larger model sizes, checking whether the small-model findings transfer.

Great research to read that informed this pipeline

Organize the Web https://arxiv.org/abs/2502.10341 FineWeb-Edu https://arxiv.org/abs/2406.17557 Data Curation cannot be Compute Agnostic https://arxiv.org/abs/2404.07177 RegMix https://arxiv.org/abs/2407.01492 OLMix https://arxiv.org/abs/2602.12237

This covers research and infrastructure from a whole bunch of folks at Marin: @ravwojdyla, @Bartleby_Kamo, Russell Power, @wbmoss, and @pinlinxu!

Huge thanks to everyone who built the datasets we use (this collection exists because of an enormous amount of community work!).

As always all of this is made possible by a variety of compute and financial support! Huge thanks to the folks at the Huang Foundation, TPU Research Cloud, Siegel Family Endowment and others.

We’ll be writing more here and gains can definitely be small v.s. vibe mixing (I have received many a :rick-rubin: emoji in slack).

A couple things that might help in your setup while I’m on the train! Mini-🧵

Yes! From the embeddings! More detail in the (poorly linked in this thread) blog:

Yup! Both are here:

From your slides, it looks like you are optimizing the hard accuracy metrics for the tasks which can be hard.

If either the signal to noise ratio is low (https://arxiv.org/abs/2508.13144) or scaling isn’t predictable (https://openathena.ai/blog/delphi/), mixing tends to be weak!

For models that have a token horizon that requires epochs, one of the biggest things mixes learn is: for what N “epoch N of great data” is worse than “epoch 1 of okay data”?

This can be tricky to vibe a priori, but proxies don’t model this by default because they are small!

https://arxiv.org/abs/2608.14071 from Tsinghua/Bytedance has a lot of great detail here, but @pinlinxu will have a ton more detail here soon based on his work in Marin!

Even for vibe based approaches modeling the discount factor is key for long token horizons: https://arxiv.org/abs/2501.11747

(off train now, but these are the two biggest things! Underlying papers are of course clearer than I am in brief here! Even for vibes, I’m a strong advocate for modeling epoching across domains experimentally using nice smooth metrics!)