lejepa

This paper proposes an attention-only white-box Transformer trained with LeJEPA-based self-supervised pretraining, achieving competitive accuracy on CIFAR-10/100 while cutting parameters by ~31% compared to CRATE, and further shows MLP redundancy in standard ViTs.