We introduce LeVLJEPA: the first fully non-contrastive end-to-end vision-language pretraining method competitive with CLIP & SigLIP

No negatives. No temperature. No momentum encoder. No teacher-student.

TL;DR: LeVLJEPA learns image to text structure by prediction: each modality predicts the other’s embedding, while SIGReg keeps each embedding isotropic Gaussian.

https://arxiv.org/abs/2607.00784

1 Introduction

Source: https://arxiv.org/html/2607.00784 LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

Lukas Kuhn1,2Giuseppe Serra1Randall Balestriero*,4Florian Buettner*,1,2,3

1German Cancer Research Center2German Cancer Consortium3Goethe University Frankfurt4Brown University

*Joint last author. Correspondence:[email protected]

Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders has shifted: they are increasingly deployed not as zero-shot classifiers but as the frozen visual backbone of vision-language models and dense prediction systems, which consume the full grid of patch tokens rather than a single pooled embedding. We introduceLeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. LeVLJEPA learns through cross-modal prediction with stop-gradient targets and per-modality distributional regularization, without negatives, temperature, momentum encoder, or teacher-student schedule, and trains stably at large scale. We find that the resulting encoder provides markedly stronger dense semantic features for downstream use: as a frozen vision-language-model backbone, LeVLJEPA is the strongest of the evaluated encoders across GQA, VQAv2, and POPE under two distinct language models, and outperforms contrastive baselines on semantic segmentation, while remaining on par on global readouts such as linear probing. These results establish non-contrastive pretraining as an effective means of producing dense semantic vision features.

Self-supervised pretraining for vision has increasingly shifted toward non-contrastive objectives. Methods such as DINO[6], I-JEPA[1], and LeJEPA[3]learn strong visual representations without negative pairs, relying instead on predictive or distributional objectives. Vision-language pretraining has not followed the same trajectory: CLIP[23], SigLIP[28], and their extensions still use contrastive image-text alignment as the default objective for learning from paired image-caption data. This raises a basic question: can end-to-end vision-language pretraining work without contrastive negatives?

This question has practical implications. Vision-language encoders are increasingly reused as visual backbones in systems where the quality of the image representation matters directly, including visual instruction-tuned models such as LLaVA[20], dense prediction methods such as DenseCLIP[25], and language-conditioned robotics systems such as CLIPort[26]and Voltron[18]. In many such settings the pretrained vision encoder serves as a frozen feature extractor, and the downstream system does not consume a single pooled image vector but attends over the full grid of patch tokens. What matters is therefore the quality of these dense, per-token visual features rather than zero-shot classification alone. Yet contrastive vision-language objectives optimize primarily for cross-modal alignment of a single pooled embedding which is well suited to zero-shot retrieval and classification, but is supervising the per-token features only as a byproduct.

We introduceLeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. LeVLJEPA learns image-text alignment through cross-modal prediction: image embeddings predict stop-gradient text embeddings, and text embeddings predict stop-gradient image embeddings through modality-specific predictors. To prevent collapse, each modality is regularized independently with SIGReg[3], encouraging isotropic-Gaussian marginal embedding distributions. This gives a simple training recipe with no negatives, temperature parameter, momentum encoder, or teacher-student schedule.

Figure 1:LeVLJEPA overview.ImageXVX_{V}and textXTX_{T}are encoded into embeddingsZV,ZTZ_{V},Z_{T}, which are then passed through modality-specific predictors to produce cross-modal predictionsZ^V,Z^T\hat{Z}_{V},\hat{Z}_{T}. The training objective combines (i) a cross-modal MSE loss between each predictor’s output and the stop-gradient target from the other modality, and (ii) SIGReg[3]applied independently toZVZ_{V}andZTZ_{T}to keep each modality’s marginal embedding distribution close to isotropic Gaussian. SIGReg (right) enforces this distribution by projecting embeddings onto random univariate directions and applying a characteristic-function-based normality test to each projection, avoiding the curse of dimensionality of direct density matching.We validate our method at two scales, training on CC12M (12M samples)[7]and on Datacomp-L (92M samples)[13]. At the smaller CC12M scale, LeVLJEPA is competitive with CLIP and SigLIP on both zero-shot transfer and linear probing, establishing that the non-contrastive objective matches contrastive pretraining when the alignment signal is the primary axis of evaluation. At the larger Datacomp-L scale, contrastive objectives retain an advantage on zero-shot transfer, while LeVLJEPA remains on par with CLIP and SigLIP under linear probing of the global class representation and outperforms both on background robustness. The objectives diverge most strongly under evaluations that operate on the patch-token sequence rather than a pooled embedding. LeVLJEPA outperforms both contrastive baselines on semantic segmentation, and when employed as the frozen visual backbone of a vision-language model—with only a lightweight bridge trained between the frozen encoder and a frozen language model—it attains the highest downstream accuracy on GQA, VQAv2, and POPE, consistently across two distinct language-model families. These results indicate that the benefit of non-contrastive vision-language pretraining is concentrated in the quality of the dense, per-token features on which downstream systems operate, rather than in the global image-text alignment measured by zero-shot transfer.

The contributions of this work are as follows:

• •We introduceLeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. The objective combines cross-modal prediction with stop-gradient targets and per-modality distributional regularization, and requires no negative pairs, temperature, momentum encoder, or teacher-student schedule, yielding a simple and stable training procedure.
• •We establish that our method is competitive with contrastive pretraining across two training scales. At smaller scale training it matches CLIP and SigLIP on both zero-shot transfer and linear probing, while at the scale of Datacomp-L it remains on par under linear probing and trails the contrastive baselines only on zero-shot transfer, the objective these methods optimize directly.
• •We demonstrate that LeVLJEPA learns stronger dense semantic features for downstream use. Evaluated under protocols that operate on the patch-token sequence rather than a pooled embedding, it outperforms both contrastive baselines on semantic segmentation and, as the frozen visual backbone of a vision-language model, attains the highest accuracy on GQA, VQAv2, and POPE across two distinct language-model families.

2 Background

2.1 Notation and Setup

We consider a paired dataset𝒟={(xi,yi)}i=1N\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}of imagesxi∈𝒳x_{i}\in\mathcal{X}and captionsyi∈𝒴y_{i}\in\mathcal{Y}. An image encoderfθ:𝒳→ℝdf_{\theta}:\mathcal{X}\to\mathbb{R}^{d}maps images to embeddingszv=fθ​(x)z^{v}=f_{\theta}(x), and a text encodergϕ:𝒴→ℝdg_{\phi}:\mathcal{Y}\to\mathbb{R}^{d}maps captions to embeddingszt=gϕ​(y)z^{t}=g_{\phi}(y). A minibatch of sizeBByields embedding matricesZv,Zt∈ℝB×dZ^{v},Z^{t}\in\mathbb{R}^{B\times d}with rowsziv,zitz^{v}_{i},z^{t}_{i}. For contrastive baselines and evaluation, embeddings areℓ2\ell_{2}-normalized before computing cosine similarities. For LeVLJEPA, the embeddings regularized by SIGReg are notℓ2\ell_{2}-normalized during training, since SIGReg targets an isotropic Gaussian marginal distribution inℝd\mathbb{R}^{d}. When cosine similarities are needed for zero-shot evaluation or retrieval, LeVLJEPA embeddings are normalized only at evaluation time.

2.2 Contrastive Vision-Language Pretraining

CLIP[23]trainsfθf_{\theta}andgϕg_{\phi}jointly by pulling matched image-caption pairs together and pushing unmatched pairs apart, using a symmetric InfoNCE objective over the batch,

ℒInfoNCE=−12​B∑i=1B[log⁡exp⁡(ziv⋅zit/τ)∑j=1Bexp⁡(ziv⋅zjt/τ)+logexp⁡(ziv⋅zit/τ)∑j=1Bexp⁡(zjv⋅zit/τ)],\begin{split}\mathcal{L}_{\text{InfoNCE}}=-\frac{1}{2B}\sum_{i=1}^{B}\Bigg[&\log\frac{\exp(z^{v}_{i}\cdot z^{t}_{i}/\tau)}{\sum_{j=1}^{B}\exp(z^{v}_{i}\cdot z^{t}_{j}/\tau)}\\ &+\log\frac{\exp(z^{v}_{i}\cdot z^{t}_{i}/\tau)}{\sum_{j=1}^{B}\exp(z^{v}_{j}\cdot z^{t}_{i}/\tau)}\Bigg],\end{split}(1)whereτ\tauis a learned temperature. The softmax over the batch turns every other sample into a negative, so the quality of the signal scales withBB: larger batches provide harder and more diverse negatives, and CLIP-style methods are typically trained with batch sizes in the tens of thousands.

SigLIP[28]replaces the softmax with an independent sigmoid classification over every pair(i,j)(i,j)in the batch,

ℒSigLIP=−1B​∑i=1B∑j=1Blog⁡σ​(ti​j​(ziv⋅zjt+b)),\mathcal{L}_{\text{SigLIP}}=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B}\log\sigma\!\left(t_{ij}\left(z^{v}_{i}\cdot z^{t}_{j}+b\right)\right),(2)whereti​j=+1t_{ij}=+1ifi=ji=jand−1-1otherwise, andt,bt,bare learnable scalars. Decoupling the pairs removes the global normalization and improves training stability at small and large batch sizes alike, though performance still benefits meaningfully from scalingBB.

VL-JEPA[8]recasts vision-language learning in the language of joint-embedding predictive architectures: rather than aligning two independently pooled embeddings, a predictor maps the image to the continuous embedding of the target caption, which is learned in latent space instead of by autoregressive token generation. Despite this predictive, JEPA-style framing, the training signal is contrastive. VL-JEPA optimizes a bidirectional InfoNCE objective of the same form as CLIP between predicted and target embeddings, which simultaneously aligns each predicted embedding with its matched target and, through the in-batch denominator, pushes it away from the other captions in the batch. Collapse is therefore prevented by negative pairs drawn from the batch rather than by an explicit distributional regularizer, and VL-JEPA inherits the batch-level negative sampling and batch-size dependence of contrastive pretraining; we accordingly group it with the contrastive methods. Notably,Chenet al.[8]observe that the InfoNCE term could in principle be replaced by a sample-independent anti-collapse regularizer but leave this to future work.

2.3 Non-Contrastive Self-Supervised Learning

In vision-only pretraining, non-contrastive methods have become the dominant paradigm. SimCLR[9]remains a canonical contrastive baseline, DINO[6]learns without negatives via self-distillation between student and teacher networks, and I-JEPA[1]predicts the representations of masked image regions from visible ones in latent space, avoiding both negatives and pixel-level reconstruction. BYOL[15]prevents collapse without negatives through architectural asymmetry: an online network equipped with a prediction head regresses the representation produced by a momentum-averaged target network, with a stop-gradient applied to the target. This asymmetric predictor-plus-stop-gradient design, which later was simplified by SimSiam[10]—which shows the momentum encoder is not required—is the mechanism we adapt to the cross-modal setting (Section3), and it is complementary to the distributional regularizer of LeJEPA.

LeJEPA[3]provides a simple and theoretically grounded alternative within the JEPA family by replacing such mechanisms with an explicit distributional regularizer. The central result is that isotropic Gaussian embeddings minimize downstream risk, and SIGReg (Sketched Isotropic Gaussian Regularization) is the objective that enforces this distribution in a scalable way. Rather than matching densities directly indddimensions, SIGReg projects the embeddings onto a set of random 1D directions𝔸\mathbb{A}and applies a characteristic-function-based statistical test (Epps–Pulley) to each projection, yielding an objective with linear time and memory complexity in both batch size and embedding dimension.

Given a batch ofBBsamples withVVaugmented views per sample (VgV_{g}global views andVlV_{l}local views), letzn,v∈ℝdz_{n,v}\in\mathbb{R}^{d}denote the embedding of viewvvof samplenn. LeJEPA combines SIGReg, applied per-view and averaged across theVVviews, with a view-prediction loss averaged across theBBsamples in the batch:

ℒLeJEPA=λV​∑v=1VSIGReg​({zn,v}n=1B)+1−λB​∑n=1Bℒpred(Vg)​({zn,v}v=1V),\begin{split}\mathcal{L}_{\text{LeJEPA}}={}&\frac{\lambda}{V}\sum_{v=1}^{V}\text{SIGReg}\!\left(\{z_{n,v}\}_{n=1}^{B}\right)\\ &+\frac{1-\lambda}{B}\sum_{n=1}^{B}\mathcal{L}_{\text{pred}}^{(V_{g})}\!\left(\{z_{n,v}\}_{v=1}^{V}\right),\end{split}(3)with a single trade-off hyperparameterλ∈[0,1]\lambda\in[0,1]. The prediction term draws views closer in latent space while SIGReg keeps the embedding distribution isotropic Gaussian, and together they are sufficient to prevent collapse without stop-gradients, momentum encoders, or teacher-student architectures. We build directly on LeJEPA and reuse its view structure and prediction-plus-SIGReg formulation as the non-contrastive component of our method.

3 Learning Image-Text Alignment Without Negatives

We investigate whether non-contrastive objectives, of the kind that have replaced contrastive learning in vision-only self-supervised pretraining, can be applied to the vision-language setting. We adopt the LeJEPA framework[3]as a representative instance: it provides a simple, theoretically grounded non-contrastive objective and serves as our starting point for cross-modal extension. Concretely, given a paired batch{(xi,yi)}i=1B\{(x_{i},y_{i})\}_{i=1}^{B}with image embeddingsziv=fθ​(xi)z^{v}_{i}=f_{\theta}(x_{i})and text embeddingszit=gϕ​(yi)z^{t}_{i}=g_{\phi}(y_{i}), we want an objective that (i) aligns the two modalities, (ii) regularizes each embedding distribution to be isotropic Gaussian via SIGReg, and (iii) uses no negatives. An overview of the resulting architecture is shown in Figure1.

3.1 Direct Alignment Collapses

A direct adaptation of LeJEPA’s view-prediction objective to the cross-modal setting replaces CLIP’s contrastive term with a latent-space regression between matched image-text pairs,

ℒdirect=1B​∑i=1B‖ziv−zit‖22+λv​SIGReg​(Zv)+λt​SIGReg​(Zt),\begin{split}\mathcal{L}_{\text{direct}}={}&\frac{1}{B}\sum_{i=1}^{B}\|z^{v}_{i}-z^{t}_{i}\|_{2}^{2}\\ &+\lambda_{v}\,\text{SIGReg}(Z^{v})+\lambda_{t}\,\text{SIGReg}(Z^{t}),\end{split}(4)trained jointly over both encoders. This symmetric objective is insufficient in practice. Without SIGReg, direct MSE collapses almost completely: the effective rank ofZvZ^{v}andZtZ^{t}drops rapidly during training and the two distributions converge onto a shared low-dimensional subspace. Adding per-modality SIGReg improves the effective rank of both embeddings, but still yields poor zero-shot and linear-probing performance (AppendixD). Thus, marginal regularization alone does not resolve the degeneracy introduced by symmetric cross-modal regression.

The cause is the symmetry of the regression term. Gradients flow into both encoders from the same target‖ziv−zit‖22\|z^{v}_{i}-z^{t}_{i}\|_{2}^{2}, so the objective encourages both embeddings to meet in a shared space determined by the information common to image and text. Since captions describe images at a coarse level of detail, this shared signal under-specifies fine visual structure. Although SIGReg can pull each marginal distribution toward an isotropic Gaussian and improve rank, the symmetric cross-modal term still couples the two encoders too strongly and does not recover useful cross-modal geometry.

3.2 Cross-Modal Prediction with Stop-Gradient

This suggests that stable non-contrastive cross-modal alignment requires an asymmetric objective. Instead of regressing one embedding against the other directly, we introduce across-modal predictoron each side: a small MLPhv:ℝd→ℝdh_{v}:\mathbb{R}^{d}\to\mathbb{R}^{d}that predicts the text embedding from the image embedding, and a mirror predictorhth_{t}that predicts the image embedding from the text embedding. Crucially, the prediction targets are detached: gradients flow only through the predictor and its own encoder, not back into the encoder producing the target. The cross-modal loss is

ℒcross=1B∑i=1B[‖hv​(ziv)−sg​(zit)‖22+∥ht(zit)−sg(ziv)∥22],\begin{split}\mathcal{L}_{\text{cross}}=\frac{1}{B}\sum_{i=1}^{B}\Big[&\|h_{v}(z^{v}_{i})-\text{sg}(z^{t}_{i})\|_{2}^{2}\\ &+\|h_{t}(z^{t}_{i})-\text{sg}(z^{v}_{i})\|_{2}^{2}\Big],\end{split}(5)wheresg​(⋅)\text{sg}(\cdot)denotes stop-gradient. This asymmetry is the central design choice: with stopped targets, each encoder’s gradients come only from its own branch, so SIGReg can shape each modality’s embedding distribution without competition from a symmetric cross-modal term. The predictorshv,hth_{v},h_{t}absorb the residual asymmetry between image and caption content rather than forcing it into the encoders themselves, similar to the predictor design in SimSiam[10]and BYOL[15]for vision-only SSL.

Architecture.Both encoders follow the standard CLIP setup: a ViT[12]image encoder producing a CLS token embedding, and a GPT-2[24]text encoder using the final-token hidden state. Each backbone output is passed through a one-layer MLP projection (hidden dimension 2048, GELU, BatchNorm, Dropout) that maps to the shared embedding space of dimensiondd, where we calculate the SIGReg loss. This projection is necessary because both the final ViT layer and the final GPT-2 layer apply a Layer Normalization[2]which prevents the SIGReg anti-collapse objective from being optimized effectively[21]. The cross-modal predictorshv,hth_{v},h_{t}are separate MLPs operating on top of these embeddings, so the target embeddings (used with stop-gradient) are exactly the same representations that SIGReg regularizes (see AppendixAfor details).

Objective.The full LeVLJEPA objective combines cross-modal prediction with SIGReg on each modality,

ℒLeVLJEPA=(1−λv−λt)​ℒcross+λv​SIGReg​(Zv)+λt​SIGReg​(Zt),\begin{split}\mathcal{L}_{\text{LeVLJEPA}}={}&(1-\lambda_{v}-\lambda_{t})\,\mathcal{L}_{\text{cross}}\\ &+\lambda_{v}\,\text{SIGReg}(Z^{v})+\lambda_{t}\,\text{SIGReg}(Z^{t}),\end{split}(6)with a single degree of freedom per modality (λv,λt∈[0,1]\lambda_{v},\lambda_{t}\in[0,1]). Unlike contrastive or teacher-student objectives, LeVLJEPA does not introduce negative-pair sampling, temperature scaling, momentum-averaged encoders, or auxiliary teacher networks. The method is trained with a standard forward-backward pass and is compatible with distributed data parallelism.

3.3 Batch Size Invariance

Contrastive objectives draw their negative signal from the batch, so their performance depends on the batch sizeBB; CLIP-style methods accordingly improve asBBgrows[23,28]. LeVLJEPA has no batch-level term: cross-modal prediction is computed per sample, and SIGReg acts on the empirical marginal of the embedding distribution, converging to the same population objective regardless ofBB. We confirm this on CC12M acrossB∈{1024,2048,4096}B\in\{1024,2048,4096\}at matched compute, where LeVLJEPA’s ImageNet zero-shot and linear-probing accuracy stay within noise while CLIP improves withBB. Non-contrastive cross-modal pretraining is therefore largely insensitive to batch size, removing a constraint that ties contrastive performance to training infrastructure.

4 Experimental Setup

Two-stage protocol.

We separate method development from final evaluation to prevent overfitting our design choices to the regime on which we report results. Hyperparameter selection and all objective ablations are conducted on CC12M, whose moderate scale permits sweeping the predictor depth and width, the SIGReg trade-off weightsλv\lambda_{v}andλt\lambda_{t}, and the learning rate across a large number of configurations at tractable cost; these studies are reported in AppendixF. The configuration selected on CC12M is then held fixed and evaluated at the scale of Datacomp-L, where all results in the main paper are obtained. No additional method-specific tuning is performed at the Datacomp-L scale, ensuring that the large-scale results reflect a single transferred configuration rather than a configuration optimized on the evaluation regime itself.

Baselines.

For the contrastive baselines we use the InfoNCE and SigLIP objectives with the well-tested OpenCLIP[17]training hyperparameters, which are widely validated for CLIP-style training at this scale, rather than tuning them ourselves. This ensures the contrastive baselines are configured as a practitioner would deploy them and that any gap is not an artifact of under-tuned baselines. All methods share the same ViT-B/16 vision encoder, the same data, the same number of samples seen, and the same evaluation protocols; LeVLJEPA and the contrastive baselines differ only in the pretraining objective.

Scale.

All Datacomp-L runs are trained for a total of819819M samples seen. We adopt this samples-seen budget, rather than a fixed number of epochs, as the axis along which all methods are matched, so that total compute is held constant across objectives independently of the effective dataset size each method traverses. Datacomp-L is distributed as a list of image URLs, and a fraction of these are no longer retrievable due to link rot: of the approximately140140M image-text pairs in the original release, roughly9292M remained downloadable at the time of our experiments, and all methods are therefore trained on this reduced∼92\sim\!92M-sample pool. This bears on cross-paper comparison. Earlier results on Datacomp-L were obtained from substantially larger downloadable fractions, so our absolute numbers are not directly comparable to those reported elsewhere and are expected to be lower. Because link rot affects all methods identically, the comparisons drawn in this paper remain controlled: every encoder is trained on the same surviving pool under the same samples-seen budget.

Training dynamics and ablations.

Under this protocol LeVLJEPA trains stably from initialization: the effective rank of bothZvZ^{v}andZtZ^{t}rises steadily throughout training until saturation, the SIGReg terms converge to small values consistent with isotropic Gaussian marginals, and the cross-modal loss decreases monotonically. Training is also robust to architectural and optimization choices, with final performance largely insensitive to predictor depth and width and to the trade-off weightsλv,λt\lambda_{v},\lambda_{t}across a wide range (AppendixF,A). Objective ablations on CC12M (AppendixD) confirm that each component is necessary: direct symmetric MSE collapses, SIGReg alone does not prevent that collapse, predictors without SIGReg are unstable or underperform, and only the full objective is both stable and performant.

5 Global Representation Quality

We first evaluate the encoders under the two standard protocols for vision-language representations: zero-shot transfer and linear probing. Each protocol reduces an image to a single pooled vector prior to scoring—zero-shot transfer compares the pooled image embedding to a text embedding, whereas linear probing classifies the pooled image embedding directly—and both therefore characterize the global representation rather than the per-token features. The two protocols yield distinct conclusions. On zero-shot transfer, the contrastive baselines hold an advantage at the scale of Datacomp-L, consistent with this protocol measuring precisely the image-text alignment they optimize directly. Under linear probing, by contrast, the objectives are on par at both the CC12M and Datacomp-L scales: removing the dependence on cross-modal alignment eliminates the zero-shot gap and leaves the global visual features of all three methods equally discriminative. These protocols thus establish parity in global representation quality while attributing the zero-shot gap to alignment rather than to feature quality. In the following subsections we report detailed results for both protocols across multiple classification benchmarks.

5.1 Zero-Shot Alignment Without Negatives

Although LeVLJEPA receives no negative-pair signal, cross-modal prediction still induces nontrivial zero-shot alignment. Table1reports zero-shot accuracy on Datacomp-L across four classification benchmarks. LeVLJEPA produces meaningful predictions across all four, confirming that matched-pair prediction with marginal regularization is sufficient to learn an image-text correspondence usable for classification.

Table 1:Zero-shot classification accuracy (%) on Datacomp-L with an identical ViT-B/16 backbone. LeVLJEPA learns nontrivial alignment without negatives but trails the contrastive baselines, which optimize the matched-vs-unmatched objective that zero-shot directly measures. LeVLJEPA uses the text-to-image prediction direction.It does, however, trail CLIP and SigLIP. This is expected as zero-shot classification scores matched against unmatched image-text pairs, which corresponds directly to the objective contrastive methods optimize. In contrast, LeVLJEPA is trained only through matched-pair prediction and never sees the discriminative signal this protocol rewards. We therefore do not present zero-shot as a setting where the non-contrastive objective is competitive, but as evidence that alignment emerges without negatives.

The magnitude of this gap is scale-dependent. On CC12M, trained at a batch size of20482048, all three objectives are nearly on par on zero-shot transfer, with LeVLJEPA exceeding CLIP on the majority of benchmarks (AppendixE). The contrastive advantage that is evident at the scale of Datacomp-L is therefore substantially reduced in this regime, consistent with the dependence of contrastive training on large batches: at the moderate batch size used here, the negative signal that drives zero-shot alignment is weaker, and the gap between the objectives narrows accordingly.

5.2 Global-Feature Linear Probing

A linear probe on the frozen CLS token isolates the quality of the global image feature from cross-modal alignment: it discards the text encoder and asks only whether the pooled visual embedding is linearly separable by class. Table2reports linear-probing accuracy on Datacomp-L. Here the three objectives track closely. LeVLJEPA stays within about1.51.5points of the strongest baseline on every benchmark and exceeds InfoNCE on Aircraft, with no method dominating across datasets. Removing the dependence on image-text alignment is thus enough to erase the zero-shot gap of Table1: the global visual feature LeVLJEPA learns is as discriminative as those of CLIP and SigLIP.

Table 2:Linear probing accuracy (%) on frozen CLS features, Datacomp-L with an identical ViT-B/16 backbone. On this global-feature readout the three objectives are on par, in contrast to the zero-shot gap in Table1.This parity is consistent across training scale. On CC12M the three objectives are likewise equivalent under ImageNet linear probing, with LeVLJEPA reaching31.84%31.84\%Top-1, against31.91%31.91\%for CLIP and31.83%31.83\%for SigLIP.

5.3 Background Robustness

IID linear probing measures whether global features are class-discriminative, but not whether they rely on the object itself or on correlated background context. We therefore evaluate background robustness on the ImageNet-9 backgrounds challenge[27]. A linear classifier is trained on frozen CLS features from theOriginaltraining split and evaluated without retraining on two controlled variants:Mixed-Same, where the foreground is composited onto a background from the same class, andMixed-Rand, where the background is drawn from a randomly selected class. A large accuracy drop under these shifts indicates sensitivity to background cues.

Table 3:ImageNet-9 background robustness. A linear classifier is trained on frozen CLS features fromOriginalimages and evaluated without retraining on background-shifted variants. All encoders use the same ViT-B/16 backbone and are pretrained on Datacomp-L.Table3shows that LeVLJEPA attains the highest accuracy in all three settings and exhibits the smallest observed degradation under both background shifts. From Original to Mixed-Same, its accuracy drops by5.955.95points, compared with6.576.57for InfoNCE and7.037.03for SigLIP. Under the stronger Mixed-Rand shift, LeVLJEPA drops by17.2117.21points, versus18.6718.67and18.0918.09points, respectively.

Thus, although the methods are closely matched on standard IID linear probing, LeVLJEPA’s global representation is less sensitive to changes in background context. This is consistent with a more object-focused representation.

Both readouts considered so far reduce the image to a single vector—zero-shot compares it to a caption embedding, linear probing classifies it directly. Modern uses of vision-language encoders, however, do not pool the image at all. In visual instruction tuning, dense prediction, and language-conditioned control, the downstream model attends over the full grid of patch tokens, and its performance is bounded by the semantic quality of those per-token features. In the next section we evaluate encoders on exactly this property—the dense token features their downstream use depends on.

6 Non-Contrastive Pretraining Yields Stronger Dense Features

A frozen ViT-B/16 emits a sequence of196196patch tokens per image, and it is this sequence, rather than the pooled CLS embedding, that downstream systems consume—a segmentation head classifying each spatial location, or a language model cross-attending to the visual tokens during instruction tuning. The semantic and spatial structure of these tokens is therefore the representational property on which the downstream use of vision-language encoders depends, and it is not constrained by the contrastive image-text objective, which supervises only the pooled embedding.

We assess this property in two stages, in both of which the encoder is held frozen and only a lightweight task head or bridge is trained, so that every difference is attributable to the pretrained representation rather than to downstream adaptation. We first evaluate dense prediction through semantic segmentation, which reads the patch grid with a minimal task head and measures the spatial and semantic structure of the token features directly (Section6.1). We then evaluate the setting these encoders are increasingly deployed in: serving as the frozen visual backbone of a vision-language model, where the patch tokens are consumed by a language model in place of a task head (Section6.2).

6.1 Semantic Segmentation

A frozen backbone evaluated through a minimal task head is a direct readout of representation quality: with the encoder held fixed and only a linear head trained on top, accuracy on a dense task is bounded by the spatial and semantic structure already present in the patch tokens. We apply this protocol to semantic segmentation, training a single linear layer on the frozen patch tokens to predict a class label at each location of the14×1414\times 14grid, with the same ViT-B/16 backbone across all methods. We report mean intersection-over-union on ADE20K[30]and COCO-Stuff[5].

Table 4:Linear semantic segmentation (mIoU, %) on frozen patch tokens, Datacomp-L with an identical ViT-B/16 backbone. A single linear head is trained per method.Table4shows a consistent advantage for LeVLJEPA, which exceeds the stronger contrastive baseline by2.252.25mIoU on ADE20K and by2.082.08mIoU on COCO-Stuff. This margin is substantially larger than the differences observed on the global readouts of Section5, and its direction is preserved across two segmentation benchmarks of differing label granularity. Because the task head is a single linear layer, the difference cannot be attributed to head capacity and instead reflects the linear predictability of semantic class from the patch tokens at each spatial location. The contrastive objective supervises only the pooled image-text similarity and shapes the per-token features as a byproduct, which is consistent with the weaker per-location structure observed here.

6.2 Visual Question Answering with a Frozen Backbone

A principal contemporary use of a vision-language encoder is as the frozen visual front-end of a vision-language model, in which the patch tokens are projected into a language model’s embedding space and consumed by the language model during multimodal instruction following. We evaluate each pretrained encoder in this role under a strict frozen-transfer protocol: both the ViT-B/16 vision encoder and the language model are held frozen, and only a lightweight MLP bridge mapping the patch tokens into the language-model input space is trained on the corresponding train split of each dataset. As neither the encoder nor the language model is updated, downstream accuracy is a function of the pretrained visual features and the bridge alone, isolating the contribution of the encoder in the same manner as the frozen heads of Section6.1. To distinguish properties of the visual features from those of a particular language model, we repeat the full evaluation with two distinct frozen language models, Llama-1B and Qwen-1.5B, and report accuracy on GQA[16], VQAv2[14], and POPE[19]. A randomly initialized encoder is included as a floor quantifying the accuracy attainable from the bridge alone. Results are reported in Table6; LeVLJEPA attains the highest accuracy in every benchmark–language-model combination, and all pretrained encoders exceed the random floor by a wide margin, confirming that the bridge reads pretrained structure rather than learning the task itself.

Compositional question answering (GQA).

GQA evaluates compositional reasoning over object attributes, spatial relations, and their conjunctions, and therefore depends on spatially localized visual information rather than a global image descriptor. LeVLJEPA is the strongest backbone under both language models, exceeding the best contrastive baseline by1.91.9points with Llama-1B and1.51.5points with Qwen-1.5B. That the advantage on a relational task is realized through a frozen bridge is consistent with the per-location structure observed under semantic segmentation: the spatial information GQA requires is present in the patch tokens prior to any downstream training.

General visual question answering (VQAv2).

VQAv2 measures open-ended visual question answering across a broad range of object, attribute, and scene queries. The margin in favor of LeVLJEPA is largest on this benchmark, at2.62.6points with Llama-1B and4.84.8points with Qwen-1.5B over the strongest contrastive backbone. The contrastive baselines also diverge from each other here—InfoNCE exceeds SigLIP under both language models despite their proximity on the global readouts of Section5—indicating that the global-feature parity established earlier does not extend to the patch-level features this task consumes.

Object hallucination (POPE).

POPE evaluates object hallucination through balanced yes/no questions on object presence, on which a backbone that induces a default “yes” response inflates recall while hallucinating absent objects; we therefore report accuracy together with F1 and the fraction of “yes” responses in Table5. LeVLJEPA attains the highest accuracy under both language models and the highest F1 under Qwen-1.5B, and produces the most balanced answer distribution, with a yes-rate nearest the calibrated50%50\%, whereas the contrastive backbones and the random floor are skewed toward “yes.” Under Llama-1B its F1 falls below the contrastive baselines despite its higher accuracy; this reflects a lower yes-rate, which trades recall on present objects for fewer false positives, rather than weaker hallucination behavior. Across both language models, the LeVLJEPA backbone yields the most accurate and least answer-biased model on this benchmark.

Table 5:POPE object-hallucination results with frozen encoders and frozen LLMs. Accuracy and F1 are higher-is-better. Bias is the absolute deviation of the yes-rate from the balanced50%50\%,|yes-rate−50||\text{yes-rate}-50|, for which lower is better. Subscripts are standard deviations over three seeds.

Consistency across language models.

The ordering of the three pretrained encoders is identical under both language models—LeVLJEPA, then InfoNCE, then SigLIP. As Llama-1B and Qwen-1.5B differ in tokenizer, pretraining corpus, and architecture, an ordering preserved across the substitution reflects a property of the visual features rather than an interaction with a specific language model. This comparison further dissociates backbone quality from zero-shot alignment: SigLIP is the strongest zero-shot model in Table1yet the weakest of the three backbones, whereas LeVLJEPA is the weakest zero-shot model yet the strongest backbone. On these encoders, zero-shot accuracy is thus not merely uninformative of backbone quality but inversely related to it, providing direct evidence for the central claim of this work: the representational property that yields a strong vision-language classifier is distinct from the one that yields a strong visual feature encoder.

Table 6:Frozen VLM-backbone accuracy (%) on GQA, VQAv2, and POPE, with two frozen language models. Both the ViT-B/16 vision encoder and the LLM are frozen; only an MLP bridge is trained. All encoders are pretrained on Datacomp-L. Subscripts are standard deviations over three bridge-training seeds. LeVLJEPA is the strongest backbone in every column.These results extend the dense-prediction findings to the deployment setting. The advantage observed as stronger per-location semantic structure under semantic segmentation manifests, when the encoder serves as a frozen visual front-end, as a more accurate and better-calibrated vision-language model, across two language models and three benchmarks and without updating either the encoder or the language model. Non-contrastive vision-language pretraining thus yields the dense semantic features on which downstream language-model use depends.

7 Discussion

LeVLJEPA demonstrates that end-to-end vision-language pretraining can be performed without contrastive negatives. Using only matched-pair cross-modal prediction with stop-gradient targets and per-modality distributional regularization and no negatives, temperature, momentum encoder, or teacher-student schedule, it trains stably at the scale of Datacomp-L and learns representations that match contrastive pretraining on global-feature readouts. Negative-pair discrimination can therefore bee seen as only one route to useful vision-language representations rather than a precondition for them.

The central finding of this work is that non-contrastive, JEPA-style vision-language pretraining yields a vision encoder with markedly stronger dense semantic features than contrastive pretraining. As the frozen backbone of a vision-language model, LeVLJEPA is the strongest of the evaluated encoders across GQA, VQAv2, and POPE under two distinct language models, and it outperforms both contrastive baselines on semantic segmentation. This advantage is specific to evaluations that consume the full patch-token sequence: under zero-shot transfer and linear probing, both of which reduce an image to a single pooled vector, the three objectives are difficult to separate, with the contrastive baselines retaining an advantage only on the zero-shot alignment they optimize directly. The standard evaluation protocols therefore do not measure the property on which these encoders are now deployed, and the divergence between objectives is visible only at the token level.

These results motivate a vision-centric formulation of vision-language pretraining. Rather than optimizing the global image-text alignment rewarded by classification metrics, the objective is to extract as much usable structure as possible from both modalities and to produce a vision encoder whose dense per-token features serve the systems that now consume them—visual instruction tuning, dense prediction, and language-conditioned control, none of which read the pooled embedding. The non-contrastive predictive objective is suited to this formulation: by supervising each modality through prediction and regularizing each marginal toward isotropy, rather than collapsing the image to a single vector matched against a caption, it preserves stronger semantic structure in the patch tokens that downstream models attend over. The background-robustness results are consistent with this account: even under a global readout, LeVLJEPA’s representation is less sensitive to background substitution than the contrastive baselines, indicating a more object-centric representation of the kind that dense, language-grounded use depends on.

Several questions remain open. The contrastive objectives retain a stronger mechanism for zero-shot image-text alignment, and we do not close this gap; whether the dense-feature advantage of non-contrastive pretraining can be combined with competitive alignment within a single objective is a natural direction for future work. Our experiments are conducted with a ViT-B/16 backbone, and while LeVLJEPA trains stably and remains competitive at the scale of Datacomp-L, establishing that these advantages persist at larger model and data scales remains important to verify. More broadly, these results indicate that as vision-language encoders are increasingly deployed as the visual front-end of language models, the objectives and evaluation protocols used to train and compare them should be reconsidered around the dense token features on which these systems rely.

Acknowledgments

We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003).

This work was co-funded by the European Union (ERC, TAIPO, 101088594 to F.B.) grant. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or ERC. Neither the European Union nor the granting authority can be held responsible for them.

References

• [1](2023)Self-supervised learning from images with a joint-embedding predictive architecture.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp. 15619–15629.Cited by:§1,§2.3.
• [2]J. L. Ba, J. R. Kiros, and G. E. Hinton(2016)Layer normalization.External Links:1607.06450,LinkCited by:§3.2.
• [3]R. Balestriero and Y. LeCun(2025)Lejepa: provable and scalable self-supervised learning without the heuristics.arXiv preprint arXiv:2511.08544.Cited by:Figure 1,§1,§1,§2.3,§3.
• [4]R. Balestriero, H. Van Assel, S. BuGhanem, and L. Maes(2025)Stable-pretraining-v1: foundation model research made simple.arXiv preprint arXiv:2511.19484.Cited by:Appendix A.
• [5]H. Caesar, J. Uijlings, and V. Ferrari(2018)Coco-stuff: thing and stuff classes in context.InProceedings of the IEEE conference on computer vision and pattern recognition,pp. 1209–1218.Cited by:Appendix B,§6.1.
• [6]M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin(2021)Emerging properties in self-supervised vision transformers.InProceedings of the IEEE/CVF international conference on computer vision,pp. 9650–9660.Cited by:§1,§2.3.
• [7]S. Changpinyo, P. Sharma, N. Ding, and R. Soricut(2021)Conceptual 12m: pushing web-scale image-text pre-training to recognize long-tail visual concepts.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp. 3558–3568.Cited by:Appendix B,§1.
• [8]D. Chen, M. Shukor, T. Moutakanni, W. Chung, J. Yu, T. Kasarla, Y. Bang, A. Bolourchi, Y. LeCun, and P. Fung(2025)Vl-jepa: joint embedding predictive architecture for vision-language.arXiv preprint arXiv:2512.10942.Cited by:§2.2.
• [9]T. Chen, S. Kornblith, M. Norouzi, and G. Hinton(2020)A simple framework for contrastive learning of visual representations.External Links:2002.05709,LinkCited by:§2.3.
• [10]X. Chen and K. He(2021)Exploring simple siamese representation learning.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp. 15750–15758.Cited by:§2.3,§3.2.
• [11]J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei(2009)Imagenet: a large-scale hierarchical image database.In2009 IEEE conference on computer vision and pattern recognition,pp. 248–255.Cited by:Appendix B.
• [12]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly,et al.(2020)An image is worth 16x16 words: transformers for image recognition at scale.arXiv preprint arXiv:2010.11929.Cited by:§3.2.
• [13]S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang,et al.(2023)Datacomp: in search of the next generation of multimodal datasets.Advances in Neural Information Processing Systems36,pp. 27092–27112.Cited by:Appendix B,§1.
• [14]Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh(2017)Making the v in vqa matter: elevating the role of image understanding in visual question answering.InProceedings of the IEEE conference on computer vision and pattern recognition,pp. 6904–6913.Cited by:Appendix B,§6.2.
• [15]J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar,et al.(2020)Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems33,pp. 21271–21284.Cited by:§2.3,§3.2.
• [16]D. A. Hudson and C. D. Manning(2019)Gqa: a new dataset for real-world visual reasoning and compositional question answering.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp. 6700–6709.Cited by:Appendix B,§6.2.
• [17]OpenCLIPNote:If you use this software, please cite it as below.External Links:Document,LinkCited by:Appendix A,§4.
• [18]S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang(2023)Language-driven representation learning for robotics.arXiv preprint arXiv:2302.12766.Cited by:§1.
• [19]Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen(2023)Evaluating object hallucination in large vision-language models.InProceedings of the 2023 conference on empirical methods in natural language processing,pp. 292–305.Cited by:Appendix B,§6.2.
• [20]H. Liu, C. Li, Q. Wu, and Y. J. Lee(2023)Visual instruction tuning.Advances in neural information processing systems36,pp. 34892–34916.Cited by:§1.
• [21]L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero(2026)Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels.arXiv preprint arXiv:2603.19312.Cited by:§3.2.
• [22]S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi(2013)Fine-grained visual classification of aircraft.Technical reportExternal Links:1306.5151Cited by:Appendix B.
• [23]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al.(2021)Learning transferable visual models from natural language supervision.InInternational conference on machine learning,pp. 8748–8763.Cited by:§1,§2.2,§3.3.
• [24]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever,et al.(2019)Language models are unsupervised multitask learners.OpenAI blog1(8),pp. 9.Cited by:§3.2.
• [25]Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu(2022)Denseclip: language-guided dense prediction with context-aware prompting.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp. 18082–18091.Cited by:§1.
• [26]M. Shridhar, L. Manuelli, and D. Fox(2022)Cliport: what and where pathways for robotic manipulation.InConference on robot learning,pp. 894–906.Cited by:§1.
• [27]K. Xiao, L. Engstrom, A. Ilyas, and A. Madry(2020)Noise or signal: the role of image backgrounds in object recognition.arXiv preprint arXiv:2006.09994.Cited by:Appendix B,§5.3.
• [28]X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer(2023)Sigmoid loss for language image pre-training.InProceedings of the IEEE/CVF international conference on computer vision,pp. 11975–11986.Cited by:§1,§2.2,§3.3.
• [29]B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba(2017)Places: a 10 million image database for scene recognition.IEEE transactions on pattern analysis and machine intelligence40(6),pp. 1452–1464.Cited by:Appendix B.
• [30]B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba(2017)Scene parsing through ade20k dataset.InProceedings of the IEEE conference on computer vision and pattern recognition,pp. 633–641.Cited by:Appendix B,§6.1.

Appendix ATraining Protocol and Hyperparameter Tuning

All methods are trained under a matched compute budget and a shared optimization schedule, so that reported differences reflect the pretraining objective rather than differences in training configuration. Each model uses the same ViT-B/16 backbone, the same global batch size, the same warmup, and the same cosine learning-rate decay schedule, and is evaluated under an identical protocol. Hyperparameter selection is performed on CC12M and the resulting configuration is transferred unchanged to the Datacomp-L runs on which the main results are reported. We are using stable-pretraining[4]throughout all experiments.

Optimization and hardware.

Each model is trained on1616GPUs with a per-rank batch size of256256, for a global batch size of40964096. Training uses a1,0001{,}000-step warmup followed by cosine decay. Each run requires approximately256256H100 GPU-hours.

Learning-rate tuning.

The learning rate is tuned independently for each method over a small grid on CC12M, and the best-performing configuration under the shared schedule is retained. This allows each objective to operate at an appropriate step size while holding the data, backbone, number of updates, warmup, and evaluation protocol fixed. At the Datacomp-L scale, the contrastive baselines instead use the InfoNCE and SigLIP objectives with the established OpenCLIP[17]hyperparameters, which are well validated for CLIP-style training at this scale, rather than a learning rate tuned by us; this ensures the contrastive baselines are configured as a practitioner would deploy them and that any observed gap is not an artifact of under-tuned baselines.

LeVLJEPA configuration.

LeVLJEPA uses a learning rate of4×10−24\times 10^{-2}, which we found stable across the batch sizes considered. The vision and text SIGReg weights are set toλv=λt=0.01\lambda_{v}=\lambda_{t}=0.01. The cross-modal predictor is a depth-44MLP of width20482048with BatchNorm and GELU, and applies 10% dropout to limit overfitting at scale. SIGReg is computedwith cross-device gathering similar to the contrastive baselines.

This protocol compares objectives under matched compute rather than maximizing absolute performance through method-specific scaling recipes. All reported differences should accordingly be interpreted as differences under the same data, backbone, number of updates, warmup, and evaluation setup, with the learning rate tuned per method.

Appendix BDatasets

Pretraining.

We pretrain on two image-text datasets. CC12M[7]contains approximately1212M image-caption pairs and is used for hyperparameter selection, ablations, and small-scale validation. Datacomp-L[13]is used for the main results; of its∼140\sim\!140M originally released pairs,∼92\sim\!92M remained downloadable at the time of our experiments due to link rot (AppendixA), and all models are trained on this surviving pool.

Classification (zero-shot and linear probing).

We evaluate on ImageNet[11](10001000classes), Places365[29](scene recognition,365365classes), FGVC-Aircraft[22](fine-grained recognition,100100classes), and Oxford-IIIT Pets[parkhi2012cats](3737classes).

Semantic segmentation.

ADE20K[30]contains20,21020{,}210training and2,0002{,}000validation images across150150semantic classes. COCO-Stuff[5]provides dense annotations over171171classes. Both are evaluated with a linear head on frozen patch tokens at the14×1414\times 14grid resolution.

Background robustness.

ImageNet-9[27]is a99-superclass subset of ImageNet with controlled foreground/background recompositions. We use theOriginal,Mixed-Same, andMixed-Randvariants, training a linear classifier onOriginaland evaluating on all three.

Vision-language model evaluation.

We evaluate the frozen-backbone VLMs on GQA[16](compositional question answering), VQAv2[14](general visual question answering), and POPE[19](object hallucination, balanced yes/no presence questions).

Appendix CEvaluation Protocol

For linear probing we evaluate frozen ViT-B/16 vision features without the projection head. For each image we extract the CLS token from the frozen vision encoder, yielding a768768-dimensional feature vector, and applyℓ2\ell_{2}normalization. Images are resized so that the shortest side is256256, center-cropped to224×224224\times 224, and normalized with the ImageNet mean and standard deviation.

Linear probing.

The linear probe is a single linear layer from the frozen feature dimension to the number of classes, without bias. Prior to fitting, features are standardized per dimension using the training-set mean and standard deviation, with standard deviations clamped at10−810^{-8}, and the same statistics are applied to the test set. The probe is trained for5050epochs with AdamW at learning rate0.10.1, weight decay10−410^{-4}, and batch size40964096, with the training set reshuffled each epoch.

Zero-shot evaluation.

For zero-shot classification we use the frozen image and text encoders without fitting any classifier. Unless stated otherwise, class names are embedded with the single prompt template

a photo of a {class}.\texttt{a photo of a \{class\}}.Image embeddings are compared against text embeddings by cosine similarity, and the nearest text class is taken as the prediction. We apply this single-template protocol to all methods to keep the evaluation controlled and to avoid confounding objective-level differences with prompt engineering, prompt ensembling, or dataset-specific calibration. These choices lower absolute zero-shot accuracy relative to fully tuned CLIP-style evaluations but yield a cleaner comparison between objectives. We therefore interpret the zero-shot results of Table1as a controlled comparison of training objectives rather than as an attempt to maximize absolute accuracy.

Appendix DObjective Ablations

We ablate the components of LeVLJEPA to test which parts are necessary for stable non-contrastive vision-language pretraining. We compare four variants: direct latent MSE between matched image and text embeddings, direct MSE with per-modality SIGReg, cross-modal predictors with stop-gradient targets but without SIGReg, and the full LeVLJEPA objective. All ablations are run for20,00020{,}000training steps under the same reduced experimental setting; absolute numbers are therefore not directly comparable to the main results.

Table 7:Objective ablation for LeVLJEPA after20,00020{,}000training steps. We report whether SIGReg is used, effective rank of the text and vision embeddings, ImageNet zero-shot Top-1, and ImageNet linear probing Top-1. Direct symmetric alignment collapses. Adding SIGReg improves rank but remains insufficient without predictor/stop-gradient asymmetry. Predictors with stop-gradient improve zero-shot alignment but underperform without marginal regularization. The full LeVLJEPA objective is required for stable and performant non-contrastive vision-language pretraining.The results in Table7show that symmetric cross-modal regression is not sufficient. Direct MSE collapses almost completely, especially on the vision side, yielding effective rank33and near-random ImageNet performance. Adding SIGReg increases the effective rank, but does not recover useful image-text alignment or transferable vision features, indicating that marginal regularization alone cannot overcome the degeneracy induced by symmetric regression. Introducing predictors and stop-gradient targets improves zero-shot alignment substantially, but remains unstable and produces weak linear probing without SIGReg. The full LeVLJEPA objective combines both ingredients: the predictor/stop-gradient asymmetry stabilizes cross-modal alignment, while SIGReg preserves high-rank modality-specific representations. This combination yields the highest effective ranks and the strongest zero-shot and linear probing performance among the ablated variants.

Appendix ECC12M Global Representation Results

This appendix reports the full CC12M results summarized in Section5. All models use an identical ViT-B/16 backbone and are trained on CC12M at a batch size of20482048under the protocol of AppendixA.

Zero-shot transfer.

Table8reports zero-shot accuracy across five classification benchmarks. At this scale the three objectives are nearly on par: LeVLJEPA exceeds CLIP on STL-10, ImageNet Top-1, CIFAR-100 Top-1, Places365, and SUN397, and matches SigLIP on SUN397 Top-1, while SigLIP retains an advantage on ImageNet and CIFAR-100. LeVLJEPA’s largest margins occur on scene classification, where it improves over CLIP by4.64.6Top-1 and6.36.3Top-5 on Places365 and by2.82.8Top-1 and4.64.6Top-5 on SUN397. The contrastive advantage observed on zero-shot transfer at the scale of Datacomp-L (Table1) is thus substantially reduced in this regime, consistent with the batch-size dependence of contrastive training.

Table 8:Zero-shot classification accuracy (%) on CC12M at batch size20482048with an identical ViT-B/16 backbone. At this scale the three objectives are nearly on par, with LeVLJEPA strongest on STL-10 and scene classification.

Linear probing.

Under linear probing of the frozen CLS feature on ImageNet, the three objectives are indistinguishable: LeVLJEPA reaches31.84%31.84\%Top-1, against31.91%31.91\%for CLIP and31.83%31.83\%for SigLIP. This matches the parity observed under linear probing at the scale of Datacomp-L (Table2), indicating that the equivalence of the global visual features across objectives is not specific to scale.

Appendix FHyperparameter Sensitivity

We study LeVLJEPA’s sensitivity to two architectural and optimization choices: the depth of the cross-modal predictor and the SIGReg trade-off weightλv\lambda_{v}andλt\lambda_{t}. All experiments use the configuration described in the main paper except for the ablated dimension; numbers are ImageNet zero-shot and linear probing Top-1 at batch size20482048on CC12M.

Predictor depth.

Figure 2:Sensitivity to predictor depth.Zero-shot (left) and linear probing (right) Top-1 accuracy as a function of the number of hidden layers in the cross-modal predictor.Figure2sweeps the cross-modal predictor’s depth from11to44hidden layers. A single hidden layer is insufficient. Zero-shot drops to7.4%7.4\%and linear probing to26.1%26.1\%, both well below the configurations with more capacity. Beyond two layers, performance is stable and improves monotonically with depth, gaining0.70.7points on zero-shot and1.91.9points on linear probing between depths22and44. The default configuration of44hidden layers used throughout the paper sits within the stable region.

SIGReg weight.

Figure 3:Sensitivity to the SIGReg trade-off weightλ\lambda.Zero-shot (left) and linear probing (right) Top-1 accuracy as a function ofλ\lambdaon a log scale.Figure3sweeps the SIGReg weightsλv\lambda_{v}andλt\lambda_{t}(tested symmetrically and denoted asλ\lambda) across two orders of magnitude. Atλ=0.001\lambda=0.001, SIGReg is too weak to enforce isotropy and training degenerates: zero-shot drops below1%1\%and linear probing to3%3\%. Forλ∈[0.005,0.04]\lambda\in[0.005,0.04], performance is stable, with probing accuracy varying within a44-point range and zero-shot within a22-point range. LeVLJEPA does not require precise tuning ofλ\lambdaas long as SIGReg is given enough weight to actively shape the embedding distribution.

Together, these ablations indicate that LeVLJEPA is robust to architectural and optimization choices once minimum thresholds (two predictor layers and a sufficiently strong SIGReg term) are satisfied.

Appendix GScaling

G.1 Scaling Model Size

Figure 4:Model size scaling on CC12M.LeVLJEPA vs. CLIP on ImageNet zero-shot (left) and linear probing (right) across three ViT backbone sizes: Tiny, Small, and Base. Both methods improve with model size, and LeVLJEPA remains competitive with CLIP across the full range.Figure4reports ImageNet performance across three ViT backbone sizes (Tiny, Small, Base) under the same update budget. Both methods benefit from increased model capacity. On zero-shot classification, LeVLJEPA outperforms CLIP at Tiny (4.90%4.90\%vs.2.26%2.26\%), is within0.340.34points at Small (9.95%9.95\%vs.10.29%10.29\%), and slightly exceeds CLIP at Base (13.49%13.49\%vs.12.39%12.39\%). Linear probing shows the same competitive pattern: the two methods are essentially tied at Tiny (22.13%22.13\%vs.22.10%22.10\%), CLIP is ahead at Small (29.37%29.37\%vs.27.43%27.43\%), and LeVLJEPA slightly exceeds CLIP at Base (32.20%32.20\%vs.31.89%31.89\%). These results suggest that the non-contrastive formulation scales with model capacity and is not restricted to a single backbone size.

G.2 Data Efficiency

Figure 5:Data efficiency on CC12M.LeVLJEPA vs. CLIP (InfoNCE) on ImageNet zero-shot (left) and linear probing (right) when trained on25%25\%,50%50\%,75%75\%, and100%100\%of CC12M under the same update budget. Both methods improve steadily with more data and converge to nearly identical linear probing accuracy at the full dataset.Figure5reports ImageNet zero-shot and linear probing accuracy when both methods are trained on subsets of CC12M under the same update budget. Both methods scale smoothly with dataset size. At smaller data fractions, CLIP holds a modest advantage: at25%25\%of CC12M, CLIP reaches5.15%5.15\%zero-shot and18.21%18.21\%linear probing Top-1, compared to4.78%4.78\%and17.33%17.33\%for LeVLJEPA; at50%50\%, CLIP reaches9.86%9.86\%zero-shot and25.62%25.62\%linear probing, compared to8.41%8.41\%and23.57%23.57\%for LeVLJEPA. The gap narrows as data increases. At75%75\%, LeVLJEPA slightly exceeds CLIP on zero-shot accuracy (10.91%10.91\%vs.10.46%10.46\%), while CLIP remains ahead on linear probing (29.54%29.54\%vs.28.53%28.53\%). At the full dataset, the methods are nearly tied on linear probing (32.20%32.20\%for CLIP vs.32.09%32.09\%for LeVLJEPA), while LeVLJEPA slightly exceeds CLIP in zero-shot accuracy (12.84%12.84\%vs.12.37%12.37\%). These results indicate that LeVLJEPA scales competitively with data under the matched training regime.