Tag
This paper proposes an attention-only white-box Transformer trained with LeJEPA-based self-supervised pretraining, achieving competitive accuracy on CIFAR-10/100 while cutting parameters by ~31% compared to CRATE, and further shows MLP redundancy in standard ViTs.
This paper presents a controlled study comparing attention-only transformers (Simple Attention Networks, SANs) against standard transformers matched for parameters, compute, and depth. It finds that removing feed-forward layers largely closes the performance gap when the freed capacity is reallocated to attention depth, with the remaining deficit attributed to parametric recall.