SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Hugging Face Daily Papers Papers
self-supervised-learning representation-learning computer-vision siamese-networks jepa efficient-training

Summary

This paper proposes SiamJEPA, which uses masked Siamese student encoders with an EMA teacher network in JEPA models, showing improved representation separability and training efficiency compared to single-encoder variants and MAE.

Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.
Original Article
View Cached Full Text

Cached at: 07/09/26, 07:53 AM

Paper page - SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Source: https://huggingface.co/papers/2607.04044

Abstract

Siamese student encoders in JEPA models improve representation separability and training efficiency through effective regularization, outperforming single-encoder variants and MAE under limited training budgets.

Recently,Joint Embedding Predictive Architectures(JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervisedrepresentation learning. Unlikemasked autoencodersthat reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, usingSiamese encodersfor student network is more naturally aligned with brain-inspiredrepresentation learningframeworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-basedrepresentation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with anexponential moving average(EMA)teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspiredrepresentation learningmodel PhiNet. Through extensive experiments on ImageNetlinear probing, we demonstrate thatSiamese encodersact as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higherlinear probingaccuracy thanMasked Autoencoders(MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an importantinductive biasfor predictiverepresentation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervisedrepresentation learning.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.04044

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.04044 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.04044 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.04044 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

The Annotated JEPA

Hacker News Top

A step-by-step annotated implementation and explanation of Joint Embedding Predictive Architectures (JEPA) for self-supervised learning, covering I-JEPA, V-JEPA, and LeJEPA.

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

arXiv cs.LG

SJEPA introduces a reconstruction-free JEPA framework that learns hybrid symbolic-neural latent dynamics, aiming for the simplest adequate predictive representation. Experiments show it discovers simpler symbolic dynamics with lower rollout error than post-hoc fitting, while controlling symbolic-neural allocation under grammar misspecification.

Representation Without Reward: A JEPA Audit for LLM Fine-Tuning

arXiv cs.LG

This paper audits Joint-embedding predictive architectures (JEPA) for LLM fine-tuning on a natural-language-to-regex task, testing twenty-two auxiliary objectives. The results show that hidden-state representation improvements are only weakly coupled to decoded-task accuracy, with no auxiliary surviving family-wise correction.