Tag
The paper develops an early-training stability theory for Joint-Embedding Predictive Architectures that unifies empirical collapse-avoidance heuristics via a drive/decay stability ratio, predicts a phase boundary confirmed across 800+ Tabular-JEPA configurations, and introduces a ResidualPred transformer predictor that improves effective rank and accuracy on tabular and I-JEPA image benchmarks.
LeVJEPA introduces a more efficient self-supervised video pretraining approach by eliminating the need for target networks and predictors, using a single shared encoder with SIGReg regularizer, and achieving competitive performance with lower compute costs.
Proposes Topologically Regularized Side-Path (TRSP) to mitigate representation collapse in LLMs by balancing spectral trade-offs between mixing efficiency and information capacity, achieving significant gains on long-context benchmarks.
This paper demonstrates that attention sinks, representation collapse, and norm stratification are not unique to attention mechanisms but are general consequences of content-based routing under a norm-blind similarity metric, as shown across multiple architectures including transformers, graph attention, state-space models, and recurrent mixers.
This paper studies representation collapse in sequential post-training of large language models, showing that repeated adaptation stages compress internal representations, reducing plasticity and out-of-domain generalization. The authors propose lightweight interventions to preserve future learnability without sacrificing behavioral gains.