@alesfav: AI needs vastly more data than we do. One idea might close the gap: don't predict raw signals (tokens), predict your ow…

AI needs vastly more data than we do. One idea might close the gap: don’t predict raw signals (tokens), predict your own abstract latent representation (JEPA, data2vec).

With @DanKorchinski @MatthieuWyart, on a toy model, we prove how much that helps: the gap is exponential.

We study recovering the hidden latent tree of a hierarchical grammar.

Token-level SSL pays a depth tax: the data it needs grows exponentially with the tree’s depth. We prove that iteratively supervising on latents escapes it, recovering the tree with constant-in-depth data!

Surprisingly, we found data2vec already does this with a single module. Through its teacher, it implicitly supervises on latents at every level, reaching the same constant-in-depth scaling.

The hierarchy unfolds during training rather than being stacked into the architecture.

This result also suggests that explicit stacking, like H-JEPA, may be redundant.

Many open questions!

Our paper:

Thanks for sharing! This indeed sounds very interesting. We’ll definitely read your work soon!

Thanks. We did try to keep the main text light on maths and lead with intuition, and even got feedback from neuroscience friends. The core idea is simple: token prediction has to infer high-level causes through many lower-level choices, so the data cost grows with depth.

Thanks! Yes, the idea has predecessors: predictive coding, BYOL, DINO, etc. which we cite in the intro. Here we mention data2vec/JEPA for space reasons and because they are modern formulations of predict your own latent, which is the setting our analysis maps to most directly.

Latent prediction avoids that bottleneck by learning one level, then using that learned level as the target/context for the next.

We may write a more accessible blog post version at some point!