i can’t stop checking in on this. the marin team is training the largest fully open model ever. 535B params (23B active), 18T tokens. nobody has been this transparent in a hero run before. beyond the weights, the data, logs, and decisions are all open. you can, and should, follow the run live, it’s ~15% through so far! https://mtracker.oa.dev/hero-run-535b

Marin 535B-A23B MoE — Marin Tracker

Source: https://mtracker.oa.dev/hero-run-535b Training remains stable for nearly twenty hours, since mcwittreturned the run to pooled wave, its previous expert-routing backend, at 9:20 p.m. (ET) on September 2, after a trial of a replacement backend hung twice. The loss stays on the line the run followed before the swap. No one has posted to the run’s threads since 1:19 a.m. (ET), nearly sixteen hours ago.

The gate and router weight decay in#8833is still unmerged. It would pull the attention-gate and router weights steadily toward zero, to stop the attention gate’s weight norm from growing further in the run’s first few layers (#8818); the gate scales each attention head’s output, so a growing norm leaves more of that scaling resting on the gate, whose value shifts from token to token. The test is a second run forked from a permanent checkpoint — one kept for good rather than overwritten — and compared against the run as it stands.#8854asked the recipe to pin such a checkpoint at step 55,000, but neither it nor#8833reached the running job beforethe run passed that markat about 2:50 a.m. (ET): the rollback relaunched the older commit on purpose, and a restart takes the recipe as the repository holds it at the commit it launches from. The next permanent checkpoint the recipe already pins is step 60,000, about fourteen hours away at the pace the run has held since the rollback, and ClassicLarryholds #8833 unmergeduntil a planned restart, so that no unplanned resume switches the decay on early.

The hang that ended the swap stays open as#8870. The backend the swap tried, ragged all-to-all, sends each token to only the few experts that handle it, and was chosen because it discards far fewer tokens for the same compute. It hung twice within the first hundred steps, each time inside a single one of the run’s eleven racks, with no layer of the stack logging an error. There is no fix yet, because neither a single-rack test nor a smaller version of the run reproduces the hang, so a candidate fix has nothing to test against. At 1:19 a.m. (ET) on September 3 a group training its own large mixture-of-experts modelsreported a related early-layer problem: in their runs the first few expert layers stopped learning after a few thousand steps while the loss and the evaluations stayed normal (write-up). They reached it from the opposite direction, with weight norms falling rather than growing, and they caution that the two setups share almost no components, so the link is a possibility rather than a diagnosis.

Percy Liang (@percyliang): Marin 535B-A23B is 13% through training. This hero run would not be possible without the generous support of the Jen-Hsun and Lori Huang Foundation, which provided the funding for the compute (Coreweave). Thanks @JensenHuang for supporting open models!