@reza_byt: World Modeling with JEPA has recently gained traction thanks to a novel anti-collapse mechanism called "SIGReg" (by @yl…

World Modeling with JEPA has recently gained traction thanks to a novel anti-collapse mechanism called “SIGReg” (by @ylecun and @randall_balestr). The math is clean, but it is rarely explained from first principles, so I broke it down in a detailed blog post

Here is the summary ():

The problem: in JEPA, both sides of the prediction loss go through the same encoder. Map every input to a single constant point, the predictor matches it trivially, and the loss hits exactly zero. Perfect score, zero information. Gradient descent finds this solution by default.

Prior work patched around it with stop-gradients, EMA teacher-student models, frozen pretrained encoders, and VICReg-style losses with 6+ hand-tuned coefficients. Each adds instability, hyperparameters, or a dependency on someone else’s pretraining.

SIGReg replaces all of that with one regularizer built on a single claim: force the batch of embeddings to look like samples from an isotropic Gaussian, N(0, I). LeJEPA proves this isn’t arbitrary. It’s the distribution that minimizes worst-case downstream risk across linear and nonlinear probes.

But “make the embeddings Gaussian” is easy to say and hard to compute. You can’t estimate a density in 200+ dimensions from a small batch. The fix is a chain of four classical results: compare Fourier transforms instead of densities, turn that comparison into a single scalar, approximate the integral with ~16 points, and lift the whole thing from 1D to any dimension with a theorem from 1936. Each step is simple. The stack is what makes it work. Read the subsequent posts in this thread to understand each step.

But why it provably works: the only distribution for which SIGReg = 0 is N(0, I), which is full rank by construction, with every eigenvalue equal to 1. A collapsed low-rank encoder cannot be a minimum. It’s not just unlikely to converge there—it’s mathematically excluded.

And the training loop stays boring:

total loss = prediction + λ·SIGReg

The encoder gets gradients from both terms; the predictor gets gradients from the prediction term only. No alternating updates, no stop-gradients, no two-timescale tricks.

Now check out the breakdown below.

2/ Eight sections, each with runnable code and figures you can regenerate. Below is the tour.

3/ · The problem

When both sides of the prediction loss use the same encoder, collapse is not a bug you can train your way out of. It’s a global minimum of the objective. The post shows the degenerate solution explicitly, then makes the case for an isotropic Gaussian as the target you actually want.

4/ · Characteristic functions

Every distribution has a Fourier transform, and it’s unique. Two distributions are identical iff their CFs are. The estimator is one line: average e^(it·z) over the batch. No density estimation, fully differentiable.

5/ · The 1D Gaussianity test

Epps–Pulley: integrate the squared gap between your empirical CF and the target across frequencies.

6/ · Quadrature

That integral has to run every step. It doesn’t need to be expensive: the integrand is smooth and effectively zero outside a narrow band, so ~8-16 trapezoidal knots land within 0.04% of the true value.

7/ · Cramér–Wold

The best part. A 1936 theorem says a distribution in any dimension is fully determined by all its 1D projections. So you never test in 200D. Sample random directions, project, run the cheap 1D test on each, average. That average is SIGReg.

8/ · Why it provably prevents collapse

Not a heuristic. SIGReg = 0 only at N(0, I), full rank by construction. Ran a toy version: start from a deliberately rank-1 initialization, train under SIGReg alone, watch the covariance get pulled to isotropic in a few hundred steps.

9/ · The training loop

Everything trains jointly. One loss, one λ, no scheduling tricks. The prediction path is untouched; the regularizer path just collects embeddings into a batch and adds its term.

If you’re about to read LeJEPA or LeWorldModel, this post gives you the complete rundown: