AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning

# Extending LeJEPA to Audio-Visual Self-Supervised Learning
Source: [https://arxiv.org/html/2607.15295](https://arxiv.org/html/2607.15295)
###### Abstract

We present AV\-JEPA, an elegant multimodal extension of LeJEPA to audio\-visual self\-supervised learning\. Using an early\-fusion Vision Transformer and*modality dropout as masking*, the model is trained to align the embeddings of global and per\-modality local views, while the SIGReg objective encourages a theoretically optimal distribution\. This achieves cross\-modal alignment in the latent space, resulting in a remarkably clean architecture with no decoder, EMA teacher, complex multi\-term losses, or contrastive negatives\. The proposed AV\-JEPA backbone delivers competitive classification performance on VGGSound \(57\.1% top\-1\) and AudioSet \(32\.7 mAP\) and supports zero\-shot audio\-video retrieval out of the box\.

Self\-supervised learning, audio, video, JEPA, multimodal

## 1Introduction

Self\-supervised learning \(SSL\) has become the dominant paradigm for learning high\-quality representations from unlabelled data\. In the audio\-visual domain, the prevailing approach is*masked autoencoding*: models such as AV\-MAE\(Georgescu et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib8)\), CAV\-MAE\(Gong et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib11)\), MAViL\(Huang et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib14)\), and CAV\-MAE Sync\(Araujo et al\.,[2025](https://arxiv.org/html/2607.15295#bib.bib1)\)reconstruct masked audio spectrograms and video patches through dedicated decoder networks\. While effective, these methods lack formal guarantees on embedding quality and require architectural overhead such as separate decoders, carefully tuned masking ratios, or contrastive negative\-pair losses\.

An alternative paradigm, the*Joint\-Embedding Predictive Architecture*\(JEPA, LeCun,[2022](https://arxiv.org/html/2607.15295#bib.bib15)\), avoids reconstruction entirely by operating in the latent space: the model learns to predict the embedding of one view from another\. Recently, LeJEPA\(Balestriero & LeCun,[2025](https://arxiv.org/html/2607.15295#bib.bib3)\)established a rigorous theoretical foundation for JEPAs by proving that the isotropic Gaussian is the uniquely optimal embedding distribution for minimizing downstream prediction risk, and enforces it via*Sketched Isotropic Gaussian Regularization*\(SIGReg\)\. However, LeJEPA has only been validated on single\-modality vision tasks\.

Concretely, we ask:*‘Can a JEPA achieve cross\-modal audio\-visual alignment, and in particular yield strong audio representations, without a decoder, contrastive negatives, EMA teachers, stop\-gradients, or per\-modality pretraining?’*We extend LeJEPA to the audio\-visual setting withAV\-JEPA\. Our key contributions are:\(i\)an extension of JEPA\-based self\-supervised learning to audio\-visual representation learning;\(ii\)*cross\-modal view generation*, where local views alternate between audio\-only and video\-only inputs \(the other modality zeroed\), creating an implicit cross\-modal prediction task entirely in latent space;\(iii\)an early\-fusion ViT architecture that processes both modalities jointly through a single shared transformer; and\(iv\)empirical results from our main experiment, AudioSet\-2M pretraining followed by fine\-tuning, showing that the resulting backbone reaches57\.1%top\-1 on VGGSound and32\.7 mAPon AudioSet, with a single\-modality breakdown confirming a strongly audio\-driven representation, and supports cross\-modal retrieval out of the box\. Qualitatively, meaningful cross\-modal attention to the sound source emerges purely from the JEPA objective, without any localization supervision\.

Video ClipAudio Spectrogram3×16×224×2243\{\\times\}16\{\\times\}224\{\\times\}2241×128×8011\{\\times\}128\{\\times\}801Global View 1VideoAudioGlobal View 2VideoAudioLocal View 1∅\\varnothingAudio\(audio only\)Local View 2Video∅\\varnothing\(video only\)EncoderShared ViT\-Base12 layers, 768 dimProjector768→2048→2048→128768\{\\to\}2048\{\\to\}2048\{\\to\}128Linear ProbeLN\+\+Linear on \[CLS\]Attentive ProbeQuery\+\+CrossAttndetachedLeJEPA Lossℒ=\(1−λ\)​∑i‖𝒛¯−𝒛i‖2\+λ​SIGReg​\(\{𝒛i\}\)\\mathcal\{L\}=\(1\{\-\}\\lambda\)\\\!\\sum\_\{i\}\\\|\\bar\{\{\\bm\{z\}\}\}\-\{\\bm\{z\}\}\_\{i\}\\\|^\{2\}\+\\lambda\\,\\mathrm\{SIGReg\}\(\\\{\{\\bm\{z\}\}\_\{i\}\\\}\)Cross\-Entropy Loss309 classes

Figure 1:AV\-JEPA training pipeline\.Each clip is split intoG=2G\{=\}2global views \(both modalities\) andK=2K\{=\}2local views \(alternating audio\-only / video\-only, the absent modality zeroed\)\. All views go through a shared ViT\-Base early\-fusion encoder over video tubelets and audio mel\-spectrogram patches\. The LeJEPA loss pulls every view embedding toward the joint\-modality center𝒛¯\\bar\{\{\\bm\{z\}\}\}while SIGReg enforces an isotropic Gaussian embedding distribution\. During VGGSound pretraining we additionally attach detached linear and attentive classification probes\.
## 2Methods

AV\-JEPA adapts LeJEPA to cross\-modal audio\-visual learning through\(i\)an early\-fusion architecture that embeds both modalities into a single token sequence, and\(ii\)a view\-generation strategy that uses modality dropout as the partial\-view mechanism\. The full pipeline is shown in[Fig\.1](https://arxiv.org/html/2607.15295#S1.F1)\.

#### LeJEPA loss

LeJEPA\(Balestriero & LeCun,[2025](https://arxiv.org/html/2607.15295#bib.bib3)\)shows that the isotropic Gaussian𝒩​\(𝟎,𝑰\)\\mathcal\{N\}\(\{\\bm\{0\}\},\{\\bm\{I\}\}\)is the optimal embedding distribution for both linear and nonlinear downstream probes, and enforces it through SIGReg: a sliced characteristic\-function test that projects embeddings ontoMMrandom unit\-norm directions and matches each univariate projection to the Gaussian target via Epps–Pulley\. GivenGGglobal andKKlocal views with embeddings𝒛i=Proj​\(fθ​\(𝒙i\)\)\{\\bm\{z\}\}\_\{i\}=\\mathrm\{Proj\}\(f\_\{\\theta\}\(\{\\bm\{x\}\}\_\{i\}\)\)and the joint\-view center𝒛¯=1G​∑g=1G𝒛g\\bar\{\{\\bm\{z\}\}\}=\\tfrac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}\{\\bm\{z\}\}\_\{g\}, the LeJEPA loss is

ℒ=\(1−λ\)​1G\+K​∑i=1G\+K‖𝒛¯−𝒛i‖2⏟invariance\+λ​SIGReg​\(\{𝒛i\}\)⏟regularization,\\mathcal\{L\}=\(1\-\\lambda\)\\,\\underbrace\{\\frac\{1\}\{G\+K\}\\sum\_\{i=1\}^\{G\+K\}\\\|\\bar\{\{\\bm\{z\}\}\}\-\{\\bm\{z\}\}\_\{i\}\\\|^\{2\}\}\_\{\\text\{invariance\}\}\\;\+\\;\\lambda\\,\\underbrace\{\\vphantom\{\\sum\_\{i=1\}^\{G\+K\}\}\\mathrm\{SIGReg\}\(\\\{\{\\bm\{z\}\}\_\{i\}\\\}\)\}\_\{\\text\{regularization\}\},\(1\)with a single trade\-off scalarλ\\lambda\.

#### Audio\-video early\-fusion encoder

Raw audio is resampled to 16 kHz and converted to a1×128×8011\{\\times\}128\{\\times\}801mel spectrogram \(128 mel bins, 801 time frames from an 8 s clip\); a16×1616\{\\times\}16Conv2D patch embedding yields8×50=4008\{\\times\}50\{=\}400audio tokens with factorized \(frequency, time\) positional embeddings\. Video frames of shape3×T×224×2243\{\\times\}T\{\\times\}224\{\\times\}224\(T=16T\{=\}16\) are tokenized by a2×16×162\{\\times\}16\{\\times\}16Conv3D tubelet, giving15681568video tokens with factorized \(spatial, temporal\) positional embeddings\. A learnable\[CLS\]token is prepended:

\[CLS;𝒗1,…,𝒗1568;𝒂1,…,𝒂400\],\[\\texttt\{CLS\};\\,\{\\bm\{v\}\}\_\{1\},\\ldots,\{\\bm\{v\}\}\_\{1568\};\\,\{\\bm\{a\}\}\_\{1\},\\ldots,\{\\bm\{a\}\}\_\{400\}\],\(2\)with learned modality\-type embeddings \(ID 0 = video, ID 1 = audio\) added to distinguish modalities\. The full 1969\-token sequence is processed by a ViT\-Base\(Dosovitskiy et al\.,[2021](https://arxiv.org/html/2607.15295#bib.bib6)\)encoder \(12L,d=768d\{=\}768, 12 heads\)\. The\[CLS\]output is projected by a 3\-layer MLP \(768→2048→2048→128768\{\\to\}2048\{\\to\}2048\{\\to\}128, BatchNorm, GELU\) before the LeJEPA loss is applied\. See[Fig\.5](https://arxiv.org/html/2607.15295#A12.F5)for the full encoder schematic\.

#### Modality dropout as partial\-view mechanism

Standard LeJEPA generates view diversity through spatial augmentations\. We add an audio\-visual axis: each 10 s clip is split into two temporally offset 8 s crops, used to form two global views \(both modalities present, light augmentation\) and two local views, oneaudio\-only\(video tokens zeroed\) and onevideo\-only\(audio tokens zeroed, video with standard augmentations\)\. The invariance term in[Equation1](https://arxiv.org/html/2607.15295#S2.E1)then pushes each single\-modality embedding toward the joint\-modality center𝒛¯\\bar\{\{\\bm\{z\}\}\}, so the model must learn, from audio alone, an embedding predictive of the joint audio\-video representation \(and vice versa\)\. Cross\-modal alignment happens entirely in latent space, with no decoder or reconstruction target\. SIGReg simultaneously prevents collapse of these dropout\-induced embeddings\.

#### Online probing

On the labelled VGGSound dataset, we additionally train two classification heads on*detached*backbone features: a linear probe \(LayerNorm \+ linear on\[CLS\]\) and an attentive probe \(one learnable query, 12\-head cross\-attention over patch tokens\), both with cross\-entropy and label smoothing 0\.1 at learning rate10−310^\{\-3\}\.

## 3Experiments

Ourmain experimentpretrains AV\-JEPA on AudioSet\-2M\(Gemmeke et al\.,[2017](https://arxiv.org/html/2607.15295#bib.bib7)\)and fine\-tunes the resulting backbone on VGGSound\(Chen et al\.,[2020](https://arxiv.org/html/2607.15295#bib.bib5)\)for audio\-visual classification \([Sec\.3\.1](https://arxiv.org/html/2607.15295#S3.SS1)\)\. We additionally fine\-tune the same AudioSet\-pretrained backbone on AudioSet itself \([Sec\.3\.2](https://arxiv.org/html/2607.15295#S3.SS2)\) as a sanity check that pretraining transfers to its source distribution, and probe cross\-modal retrieval \([Sec\.3\.3](https://arxiv.org/html/2607.15295#S3.SS3)\) as a complementary check that the learned embedding space is genuinely shared across modalities\. As a controlled secondary study, we also pretrain \(and fine\-tune\) on VGGSound alone, isolating the contribution of the JEPA objective from data scale\. All runs share the same ViT\-Base early\-fusion encoder and LeJEPA recipe, differing only in pretraining dataset, batch size, and training budget\.

#### Datasets

AudioSet\(Gemmeke et al\.,[2017](https://arxiv.org/html/2607.15295#bib.bib7)\):∼\{\\sim\}2M YouTube clips, multi\-label across 527 sound classes \(unbalanced split for pretraining and downstream fine\-tuning; 20k balanced subset for AS\-20k\)\.VGGSound\(Chen et al\.,[2020](https://arxiv.org/html/2607.15295#bib.bib5)\):∼\{\\sim\}184k train / 15k test 10 s clips spanning 309 audio\-visual event classes\.

#### Pretraining

We train AV\-JEPA on 8×\\timesNVIDIA H200 GPUs \(DDP, bf16\) with AdamW\(Loshchilov & Hutter,[2019](https://arxiv.org/html/2607.15295#bib.bib16)\)at learning rate5×10−45\{\\times\}10^\{\-4\}, weight decay 0\.05, linear warmup over 15% of training followed by cosine decay to10−610^\{\-6\}, gradient clipping 5\.0,G=2G\{=\}2global andK=2K\{=\}2cross\-modal local views, andλ=0\.05\\lambda\{=\}0\.05\. The AudioSet\-2M pretraining runs for 57 epochs at batch size 40/GPU \(effective 320\); a smaller VGGSound\-only run uses 50 epochs at batch size 50/GPU \(effective 400\)\. The pretraining loss decreases monotonically in both regimes \([Fig\.2](https://arxiv.org/html/2607.15295#S3.F2)\), and the per\-dimension embedding standard deviation rises from∼0\.8\{\\sim\}0\.8to∼1\.01\{\\sim\}1\.01on AudioSet \(∼0\.8\{\\sim\}0\.8to∼1\.01\{\\sim\}1\.01on VGGSound\), confirming that SIGReg converges to the isotropic Gaussian target at both scales\.

![Refer to caption](https://arxiv.org/html/2607.15295v1/x1.png)\(a\)AudioSet\-2M \(57 ep\.\)
![Refer to caption](https://arxiv.org/html/2607.15295v1/x2.png)\(b\)VGGSound \(50 ep\.\)

Figure 2:LeJEPA pretraining losson\(a\)AudioSet\-2M and\(b\)VGGSound, decomposed into weighted invariance and SIGReg terms\. Both decrease steadily without signs of collapse\.
#### Fine\-tuning

We attach a LayerNorm \+ linear classifier \(with an auxiliary attentive head\) on top of\[CLS\]and unfreeze the full backbone\. AdamW with head LR2×10−42\{\\times\}10^\{\-4\}and a low\-LR backbone \(0\.05×\\timeshead for the AS\-2M→\\toVGGSound headline, 0\.1×\\timesfor the controlled and AudioSet runs\), weight decay 0\.05, label smoothing 0\.1, gradient clipping 1\.0, warmup 5% then cosine to10−710^\{\-7\}, bf16\. We fine\-tune VGGSound on 4×\\timesNVIDIA H200 for 13 epochs \(6 epochs for the controlled VGGS\-only study\) and AudioSet on 8×\\times\(AS\-2M,∼29\{\\sim\}29epochs\) and 2×\\times\(AS\-20k,∼46\{\\sim\}46epochs\) H200, reporting best top\-1 \(resp\. mAP\) along the trajectory with multi\-clip aggregation\.

### 3\.1Audio\-Visual Classification on VGGSound

[Table1](https://arxiv.org/html/2607.15295#S3.T1)compares AV\-JEPA against state\-of\-the\-art audio\-visual SSL methods on VGGSound\. After 57 epochs of AudioSet pretraining and 13 epochs of fine\-tuning, AV\-JEPA reaches57\.1%top\-1 with the attentive head and 56\.6% with the linear head\. To our knowledge, this is the first JEPA\-based result at this level of classification accuracy\.

Table 1:Audio\-visual classification on VGGSound\. AV\-JEPA is the only JEPA\-based method\. The headline \(top\) fine\-tunes the AS\-2M\-pretrained backbone on VGGSound; the middle reports a VGGSound\-only controlled study; the bottom lists published MAE\-based baselines\.MethodTypePre\-trainEpochsEvalTop\-1AS\-2M→\\toVGGS fine\-tune \(ours, headline\)AV\-JEPA \(ours\)JEPAAS\-2M57\+13FT \(Att\.\)57\.1AV\-JEPA \(ours\)JEPAAS\-2M57\+13FT \(Lin\.\)56\.6Controlled VGGS\-only \(ours\)AV\-JEPA \(ours\)JEPAVGGS50\+6FT49\.8AV\-JEPA \(ours\)JEPAVGGS50Att\. \(frozen\)48\.6AV\-JEPA \(ours\)JEPAVGGS50Lin\. \(frozen\)46\.0LiteratureMAViLMAEAS\-2M\+IN80\+60FT67\.1CAV\-MAEMAEAS\-2M25\+10FT65\.4AV\-MAEMAEVGGS800\+50FT63\.5CAV\-MAE SyncMAEAS\-2M25Lin\. \(frozen\)52\.7The end\-to\-end fine\-tuned results of AV\-MAE\(Georgescu et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib8)\), CAV\-MAE\(Gong et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib11)\), and MAViL\(Huang et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib14)\)\(63–67%\) sit higher, but those methods rely on reconstruction decoders and contrastive objectives, AV\-MAE in particular pretrains for up to800800epochs, and MAViL adds an ImageNet\-pretrained visual encoder\. The remaining gap is consistent with these architectural advantages, the absence of a video\-specific pretraining stage, and this being the first JEPA recipe in the audio\-visual setting\.

### 3\.2Audio Classification on AudioSet

Table 2:Audio\-visual mAP on the AudioSet eval split\. AV\-JEPA fine\-tunes the 57\-epoch AS\-2M\-pretrained backbone end\-to\-end on the full AS\-2M set \(∼29\{\\sim\}29epochs\) and on the balanced AS\-20k subset \(∼46\{\\sim\}46epochs\), and reports per\-modality \(audio\-only, video\-only\) eval at inference time\. The Epochs column lists AudioSet\-2M pretraining epochs\. Baselines report joint A\+V mAP\.†linear probe\.MethodTypePre\-trainEpochsEvalAS\-2MAS\-20kEnd\-to\-end fine\-tuning \(ours\)AV\-JEPAJEPAAS\-2M57A\+V32\.729\.6AV\-JEPAJEPAAS\-2M57A\-only26\.023\.7AV\-JEPAJEPAAS\-2M57V\-only12\.810\.3Baselines \(end\-to\-end fine\-tuning\)AV\-MAEMAEAS\-2M100A\+V47\.3–CAV\-MAEMAEAS\-2M25A\+V51\.242\.0MAViLMAEAS\-2M\+IN80A\+V53\.344\.9CAV\-MAE SyncMAEAS\-2M–A\+V–30\.5†[Table2](https://arxiv.org/html/2607.15295#S3.T2)reports AudioSet mAP after end\-to\-end fine\-tuning of the AS\-2M\-pretrained AV\-JEPA backbone on AS\-2M\. The model reaches32\.7 mAPon the AudioSet eval split\. The per\-modality breakdown \(26\.0 audio\-only vs\. 12\.8 video\-only\) shows a clear audio dominance: the JEPA objective produces a backbone whose decisions are mostly carried by audio, consistent with the dominant role of audio in many AudioSet classes\. The MAE\-based baselines reach4242–5353mAP through dedicated reconstruction decoders and contrastive objectives \(and, for MAViL, an ImageNet\-pretrained encoder\); closing this gap is left to future work\. The same audio\-driven behaviour is visible during VGGSound training \([Fig\.3](https://arxiv.org/html/2607.15295#S3.F3)\): audio\-only and audio\+video accuracy track each other closely while video\-only lags substantially\.

![Refer to caption](https://arxiv.org/html/2607.15295v1/figures/fig_modality_probe_combined.png)\(a\)VGGSound pretraining
![Refer to caption](https://arxiv.org/html/2607.15295v1/x3.png)\(b\)VGGSound fine\-tuning

Figure 3:Top\-1 by input modalityduring \(a\) VGGSound\-only pretraining \(frozen\-feature linear probe\) and \(b\) end\-to\-end fine\-tuning of the AS\-2M backbone on VGGSound\.
### 3\.3Cross\-Modal Retrieval and Attention

#### Audio↔\\leftrightarrowvideo retrieval

We probe whether the embedding space is genuinely shared via audio↔\\leftrightarrowvideo retrieval on the projected\[CLS\]features\. For each dataset we build a balanced 5\-per\-class evaluation subset of the official test split\. Each clip is encoded twice through the backbone, once with the video zeroed \(audio\-only embedding\) and once with the audio zeroed \(video\-only embedding\), and we rank candidates by cosine similarity in both A→\\toV \(audio query, video gallery\) and V→\\toA directions\.[Table3](https://arxiv.org/html/2607.15295#S3.T3)reports Recall@kkand median rank\. On both datasets retrieval is well above the1/N1/Nchance level \(0\.06%0\.06\\%and0\.05%0\.05\\%R@1 respectively\), with R@10 reaching∼36%\{\\sim\}36\\%on both VGGSound and AudioSet, and the two directions broadly symmetric\. These rankings are obtained from the same backbone with no contrastive training and no paired\-retrieval supervision\.

Table 3:Cross\-modal retrievalon the VGGSound and AudioSet eval splits \(balanced 5\-per\-class subsets\)\. Recall@kk\(%\) and median rank in both directions, on projected\[CLS\]embeddings\.
#### Cross\-modal attention

[Fig\.4](https://arxiv.org/html/2607.15295#S3.F4)qualitatively visualizes audio↔\\leftrightarrowvideo attention from the last transformer layer on three VGGSound test clips\. We extract audio\-to\-video and video\-to\-audio attention weights, average across heads, and overlay them on the original RGB frame \(audio→\\tovideo\) and on the mel spectrogram \(video→\\toaudio\)\. Across all clips the attention concentrates on the sound\-producing region in the video \(the guitar and player’s hands, the flute and player’s mouth, the body of the flying bird\)*and*on the harmonic / temporal structure in the audio \(fundamentals and overtones for the guitar and flute, the wing\-beat envelope for the bird\)\. This emerges from the JEPA objective alone, with no localization supervision and no contrastive negatives\.

![Refer to caption](https://arxiv.org/html/2607.15295v1/attention_diagrams/audio_video_combined/guitar.png)V![Refer to caption](https://arxiv.org/html/2607.15295v1/x4.png)A

![Refer to caption](https://arxiv.org/html/2607.15295v1/attention_diagrams/audio_video_combined/flute.png)V![Refer to caption](https://arxiv.org/html/2607.15295v1/x5.png)A

![Refer to caption](https://arxiv.org/html/2607.15295v1/attention_diagrams/audio_video_combined/bird.png)V![Refer to caption](https://arxiv.org/html/2607.15295v1/attention_diagrams/audio_video_combined/bird-spectrogram.png)A

Figure 4:Cross\-modal attentionon three VGGSound test clips \(guitar, flute, bird\)\. Last\-layer audio→\\tovideo attention overlaid on RGB frames \(V\); video→\\toaudio attention overlaid on mel spectrograms \(A\)\. The model attends to the visually salient sound source and to the harmonic / temporal structure of the sound, with no localization supervision\.

## 4Discussion and Conclusion

We presented AV\-JEPA, the first extension of LeJEPA to cross\-modal audio\-visual self\-supervised learning\. Replacing spatial masking with*modality dropout*, AV\-JEPA turns alignment between audio\-only and video\-only views into an implicit cross\-modal prediction task in latent space, without decoders, reconstruction targets, stop\-gradient, or EMA teachers\. The recipe is clean, a single shared ViT\-Base encoder, the LeJEPA loss, and one trade\-off scalarλ\\lambda, yet reaches57\.1%top\-1 on VGGSound and32\.7 mAPon AudioSet after fine\-tuning, and supports zero\-shot audio↔\\leftrightarrowvideo retrieval\. This positions theoretically grounded JEPAs as a viable alternative to masked\-autoencoding pipelines for multimodal representation learning\.

#### Limitations and future work

Modality\-probe and per\-modality mAP results show that AV\-JEPA’s predictions are largely carried by audio on both VGGSound and AudioSet, making the backbone most useful as an*audio*representation with a visual side\-channel from pretraining\. The gap with end\-to\-end fine\-tuned MAE\-based methods \(63–67% on VGGSound, 42–53 mAP on AudioSet\) likely reflects their reconstruction and contrastive objectives, the absence of a video\-specific pretraining stage, and \(for MAViL\) ImageNet initialisation\. Next steps include longer AudioSet pretraining, larger ViTs, initialising the visual stream from a video\-only stage, evaluating against single\-modality SSL backbones\(Gong et al\.,[2021](https://arxiv.org/html/2607.15295#bib.bib10); Huang et al\.,[2022](https://arxiv.org/html/2607.15295#bib.bib13)\), and extending the modality\-dropout recipe to text or optical flow\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- Araujo et al\. \(2025\)Araujo, E\., Rouditchenko, A\., Gong, Y\., Bhati, S\., Thomas, S\., Kingsbury, B\., Karlinsky, L\., Feris, R\., Glass, J\. R\., and Kuehne, H\.CAV\-MAE Sync: Improving contrastive audio\-visual mask autoencoders via fine\-grained alignment\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2025\.
- Assran et al\. \(2023\)Assran, M\., Duval, Q\., Misra, I\., Bojanowski, P\., Vincent, P\., Rabbat, M\., LeCun, Y\., and Ballas, N\.Self\-supervised learning from images with a joint\-embedding predictive architecture\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023\.
- Balestriero & LeCun \(2025\)Balestriero, R\. and LeCun, Y\.LeJEPA: Provable and scalable self\-supervised learning without the heuristics\.*arXiv preprint arXiv:2511\.08544*, 2025\.
- Bardes et al\. \(2024\)Bardes, A\., Garrido, Q\., Ponce, J\., Chen, X\., Rabbat, M\., LeCun, Y\., Assran, M\., and Ballas, N\.Revisiting feature prediction for learning visual representations from video\.*arXiv preprint arXiv:2404\.08471*, 2024\.
- Chen et al\. \(2020\)Chen, H\., Xie, W\., Vedaldi, A\., and Zisserman, A\.VGGSound: A large\-scale audio\-visual dataset\.In*International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2020\.
- Dosovitskiy et al\. \(2021\)Dosovitskiy, A\., Beyer, L\., Kolesnikov, A\., Weissenborn, D\., Zhai, X\., Unterthiner, T\., Dehghani, M\., Minderer, M\., Heigold, G\., Gelly, S\., Uszkoreit, J\., and Houlsby, N\.An image is worth 16x16 words: Transformers for image recognition at scale\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Gemmeke et al\. \(2017\)Gemmeke, J\. F\., Ellis, D\. P\. W\., Freedman, D\., Jansen, A\., Lawrence, W\., Moore, R\. C\., Plakal, M\., and Ritter, M\.Audio set: An ontology and human\-labeled dataset for audio events\.In*International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2017\.
- Georgescu et al\. \(2023\)Georgescu, M\.\-I\., Fonseca, E\., Ionescu, R\. T\., Lucic, M\., Schmid, C\., and Arnab, A\.Audiovisual masked autoencoders\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023\.
- Girdhar et al\. \(2023\)Girdhar, R\., El\-Nouby, A\., Liu, Z\., Singh, M\., Alwala, K\. V\., Joulin, A\., and Misra, I\.ImageBind: One embedding space to bind them all\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023\.
- Gong et al\. \(2021\)Gong, Y\., Chung, Y\.\-A\., and Glass, J\.AST: Audio spectrogram transformer\.In*Interspeech*, 2021\.
- Gong et al\. \(2023\)Gong, Y\., Rouditchenko, A\., Liu, A\. H\., Harwath, D\., Karlinsky, L\., Kuehne, H\., and Glass, J\.Contrastive audio\-visual masked autoencoder\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Guzhov et al\. \(2022\)Guzhov, A\., Raue, F\., Hees, J\., and Dengel, A\.AudioCLIP: Extending CLIP to image, text and audio\.In*International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2022\.
- Huang et al\. \(2022\)Huang, P\.\-Y\., Xu, H\., Li, J\., Baevski, A\., Auli, M\., Galuba, W\., Metze, F\., and Feichtenhofer, C\.Masked autoencoders that listen\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Huang et al\. \(2023\)Huang, P\.\-Y\., Sharma, V\., Xu, H\., Ryali, C\., Fan, H\., Li, Y\., Li, S\.\-W\., Ghosh, G\., Malik, J\., and Feichtenhofer, C\.MAViL: Masked audio\-video learners\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- LeCun \(2022\)LeCun, Y\.A path towards autonomous machine intelligence\.Technical report, OpenReview, 2022\.
- Loshchilov & Hutter \(2019\)Loshchilov, I\. and Hutter, F\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.
- Nagrani et al\. \(2021\)Nagrani, A\., Yang, S\., Arnab, A\., Jansen, A\., Schmid, C\., and Sun, C\.Attention bottlenecks for multimodal fusion\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2021\.

## Appendices

## Appendix ALLM/Agent Usage

The authors made use of LLMs and agents throughout this work\. In writing, they assisted with drafting, editing, and producing plots\. They were also used during model development\. LLMs/agents were not used for ideation\.

## Appendix BRelated Work

#### Audio\-visual SSL

The dominant approach uses masked autoencoding: AV\-MAE\(Georgescu et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib8)\)jointly reconstructs masked audio and video; CAV\-MAE\(Gong et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib11)\)combines contrastive learning with masked reconstruction; MAViL\(Huang et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib14)\)adds self\-training; and CAV\-MAE Sync\(Araujo et al\.,[2025](https://arxiv.org/html/2607.15295#bib.bib1)\)introduces fine\-grained temporal alignment\. All require decoder networks for pixel/spectrogram reconstruction or contrastive negatives\. AV\-JEPA keeps neither\.

#### Audio\-only SSL backbones

On the audio side, AST\(Gong et al\.,[2021](https://arxiv.org/html/2607.15295#bib.bib10)\)introduces a transformer backbone for spectrograms and AudioMAE\(Huang et al\.,[2022](https://arxiv.org/html/2607.15295#bib.bib13)\)adapts MAE\-style masked spectrogram modelling to audio\. AV\-JEPA bypasses per\-modality pretraining and trains a single shared backbone from scratch with a cross\-modal JEPA objective\.

#### JEPAs

I\-JEPA\(Assran et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib2)\)applies JEPA to images using spatial masking, and V\-JEPA\(Bardes et al\.,[2024](https://arxiv.org/html/2607.15295#bib.bib4)\)extends it to video\. LeJEPA\(Balestriero & LeCun,[2025](https://arxiv.org/html/2607.15295#bib.bib3)\)provides the theoretical grounding via SIGReg\. All prior JEPA work operates on a single modality; AV\-JEPA is, to our knowledge, the first to extend JEPA to cross\-modal audio\-visual learning, with modality dropout as the partial\-view mechanism rather than spatial masking\.

#### Cross\-modal alignment

AudioCLIP\(Guzhov et al\.,[2022](https://arxiv.org/html/2607.15295#bib.bib12)\)and ImageBind\(Girdhar et al\.,[2023](https://arxiv.org/html/2607.15295#bib.bib9)\)learn shared embedding spaces across modalities via large\-scale contrastive pretraining\. AV\-JEPA achieves cross\-modal alignment as a by\-product of the LeJEPA invariance loss on modality\-dropout views, with no contrastive negatives\.

#### Multimodal fusion

Early fusion concatenates modality tokens before processing\(Nagrani et al\.,[2021](https://arxiv.org/html/2607.15295#bib.bib17)\); late fusion uses separate encoders\. AV\-JEPA uses early fusion with modality\-type embeddings, enabling the shared transformer to learn cross\-modal interactions from the first layer\.

## Appendix CEncoder Architecture

[Fig\.5](https://arxiv.org/html/2607.15295#A12.F5)shows the full AV\-JEPA encoder used for every view \(global and local\) during pretraining\. Video frames of shape3×16×224×2243\{\\times\}16\{\\times\}224\{\\times\}224are tokenized by a2×16×162\{\\times\}16\{\\times\}16Conv3D tubelet into15681568video tokens; the mel spectrogram of shape1×128×8011\{\\times\}128\{\\times\}801is tokenized by a16×1616\{\\times\}16Conv2D into400400audio tokens\. Both streams receive factorized positional embeddings \(spatial \+ temporal for video; frequency \+ time for audio\) and a learnable modality\-type embedding \(ID 0 for video, ID 1 for audio\)\. A learnable\[CLS\]token is prepended and the resulting19691969\-token sequence is passed through a1212\-layer ViT\-Base \(d=768d\{=\}768,1212heads, FlashAttention\-2\)\. On local views, the tokens of the dropped modality are zeroed before patch embedding, so the same encoder handles joint, audio\-only, and video\-only inputs without architectural changes\. The\[CLS\]output is taken as the clip embedding and fed to the projection MLP for the LeJEPA loss \([Fig\.1](https://arxiv.org/html/2607.15295#S1.F1)\)\.

## Appendix DTraining Configurations

We summarise all training configurations used in the paper in two consolidated tables\.[Table4](https://arxiv.org/html/2607.15295#A12.T4)lists the fine\-tuning recipe applied on top of either backbone for VGGSound classification\.[Table5](https://arxiv.org/html/2607.15295#A12.T5)lists the pretraining recipes for the AudioSet\-2M run \(the backbone used in our main experiment\) and the VGGSound\-only run \(the controlled secondary study\)\.

## Appendix EAdditional Embedding\-Quality Curves

[Fig\.6](https://arxiv.org/html/2607.15295#A12.F6)tracks the per\-dimension standard deviation of the projected embeddings over training for both pretraining runs\. In each case the value rises rapidly toward∼1\.01\{\\sim\}1\.01\(from∼0\.8\{\\sim\}0\.8on VGGSound and∼0\.8\{\\sim\}0\.8on AudioSet\), confirming that SIGReg drives the embedding distribution toward the target isotropic Gaussian𝒩​\(𝟎,𝑰\)\\mathcal\{N\}\(\{\\bm\{0\}\},\{\\bm\{I\}\}\)at both VGGSound and AudioSet scales\.

## Appendix FAdditional Pretraining Curves

#### Pretraining loss components

[Fig\.7](https://arxiv.org/html/2607.15295#A12.F7)decomposes the LeJEPA objective into its two unweighted terms\. The invariance term decreases steadily, indicating that view embeddings converge toward the global center𝒛¯\\bar\{\{\\bm\{z\}\}\}and that cross\-modal alignment is successful\. The SIGReg term drops sharply during the first epoch and then plateaus near zero, confirming that the embedding distribution matches the target isotropic Gaussian early in training and remains there for the rest of the run\.

#### Online probe curves

[Fig\.8](https://arxiv.org/html/2607.15295#A12.F8)reports the online linear and attentive probes on frozen backbone features during pretraining: attentive probe top\-1 accuracy, top\-5 accuracy for both probes, and the combined cross\-entropy loss\. The attentive probe consistently outperforms the linear probe by∼2\{\\sim\}2–3 points across both top\-1 and top\-5\.

## Appendix GAudioSet Pretraining Curves

We complement the AudioSet pretraining loss in the main text with the per\-component invariance and SIGReg curves \([Fig\.9](https://arxiv.org/html/2607.15295#A12.F9)\)\. The qualitative behaviour mirrors the VGGSound\-only run: the invariance term decreases steadily and the SIGReg term drops sharply during the first epoch and then plateaus near zero, confirming that SIGReg behaves as expected at AudioSet scale\.

## Appendix HFine\-Tuning Training Curves

[Fig\.10](https://arxiv.org/html/2607.15295#A12.F10)shows the end\-to\-end fine\-tuning curves \(top\-1/top\-5 accuracy and cross\-entropy loss\) when the pretrained AV\-JEPA backbone is fine\-tuned with a linear classifier on the\[CLS\]token\.

## Appendix IJoint Embedding t\-SNE by Modality

[Fig\.11](https://arxiv.org/html/2607.15295#A12.F11)shows two\-dimensional t\-SNE projections of the projected\[CLS\]embeddings on the same balanced 5\-per\-class evaluation subsets used for retrieval \(AudioSetN=2015N\{=\}2015, VGGSoundN=1545N\{=\}1545clips\)\. Each clip contributes two points: one obtained by encoding the clip with the video tokens zeroed \(audio\-only, orange\) and one with the audio tokens zeroed \(video\-only, blue\), so an unshared embedding space would manifest as two well\-separated colour clusters\. Instead, on both datasets the two modalities are interleaved across the same regions of the projected space, with no visible modality\-conditioned partition, indicating that AV\-JEPA has learned a genuinely shared cross\-modal embedding space rather than two parallel single\-modality manifolds\. This is consistent with the cross\-modal retrieval results in[Table3](https://arxiv.org/html/2607.15295#S3.T3): a shared embedding space is exactly what makes cosine ranking of audio against video, and vice versa, meaningful in the first place\.

## Appendix JSemantic Structure of the Embedding Space

Where[App\.I](https://arxiv.org/html/2607.15295#A9)asks whether the two modalities share a space,[Fig\.12](https://arxiv.org/html/2607.15295#A12.F12)asks whether that space is organised by semantic content\. We encodeN=11,143N\{=\}11\{,\}143VGGSound training clips spanning6060classes, grouped into six coarse semantic families \(instruments, animals, vehicles, water/weather, human voice, and sports\), take the\[CLS\]embedding of each clip from the fine\-tuned AV\-JEPA ViT\-B backbone \(the same backbone behind[Table1](https://arxiv.org/html/2607.15295#S3.T1)\), and project the L2\-normalised embeddings to two dimensions with t\-SNE; points are coloured by family\. Clips organise into compact, well\-separated clusters that respect the family grouping: each family occupies coherent regions of the projected space, and the finer per\-class structure within a family \(for example the individual instruments\) is visible as distinct same\-colour sub\-clusters\. The residual overlap is concentrated between families that are acoustically and visually related, most notably animal vocalisations and human voice, which is consistent with the cross\-modal attention behaviour of[Fig\.4](https://arxiv.org/html/2607.15295#S3.F4)and indicates that the learned space is structured by semantics rather than by low\-level modality cues\.

## Appendix KFeature PCA of Video Patch Tokens

[Fig\.20](https://arxiv.org/html/2607.15295#A12.F20)visualises the video patch\-token features of the fine\-tuned AV\-JEPA ViT\-B encoder \(the same backbone behind the VGGSound classification results in[Table1](https://arxiv.org/html/2607.15295#S3.T1)\) in the style of DINO/DINOv2 feature PCA\. For each instrument class we fit a*single*PCA jointly over the pooled last\-layer video patch tokens of four VGGSound clips, map the top three principal components to RGB, and overlay the result on the video frames \(rows are clips, columns are frames; audio tokens are left uncoloured\); because the basis is shared across the four clips, a given colour denotes the same direction in feature space\. The sounding object \(the piano keyboard, the body and neck of the bass guitar, the violin\) then takes a consistent colour across clips and frames, clearly separated from the player and the background, indicating that the encoder represents the visually salient sound source as a coherent, instance\-corresponding region of feature space, even though it is trained only with clip\-level objectives and never receives pixel\- or region\-level supervision, echoing in the video stream the localization behaviour seen in the cross\-modal attention maps of[Fig\.4](https://arxiv.org/html/2607.15295#S3.F4)\.

## Appendix LAblations and LeJEPA Hyperparameter Tuning

We complement the main results with one architectural ablation and two hyperparameter sensitivity studies, all run as VGGSound\-only pretraining with the same ViT\-B early\-fusion recipe as the controlled run reported in the main text\. The dual\-encoder ablation is run on 8×\\timesNVIDIA H200 to match the controlled main\-text run; the SIGReg\-weightλ\\lambdaand local\-viewKKsweeps are run on 1×\\timesNVIDIA H200 \(batch size 32 per GPU\) under a tighter compute budget\. All curves are clipped to the shortest variant’s final epoch, ensuring a common training budget\.

#### Shared vs\. dual encoder \(ablation\)\.

[Fig\.13](https://arxiv.org/html/2607.15295#A12.F13)compares the shared early\-fusion encoder used in AV\-JEPA against a dual\-encoder variant that processes audio and video through separate ViT\-B encoders and averages the two\[CLS\]tokens\. The shared encoder matches the dual\-encoder variant on both linear\- and attentive\-probe accuracy throughout training; we adopt early fusion for its simpler, single\-encoder design\.

#### SIGReg weightλ\\lambda\(LeJEPA hyperparameter\)\.

[Fig\.14](https://arxiv.org/html/2607.15295#A12.F14)sweeps the SIGReg weightλ\\lambdain the LeJEPA lossℒ=\(1−λ\)​ℒinv\+λ​ℒSIGReg\\mathcal\{L\}=\(1\-\\lambda\)\\mathcal\{L\}\_\{\\mathrm\{inv\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}across\{0\.03,0\.05,0\.10\}\\\{0\.03,0\.05,0\.10\\\}\. The probe\-accuracy curves are tightly clustered, indicating that AV\-JEPA is robust to moderate variation inλ\\lambdaon VGGSound\. The SIGReg loss itself naturally rises withλ\\lambda\(since the optimiser tolerates more residual SIGReg in exchange for a stronger invariance gradient\), but the resulting embedding standard deviation remains close across settings\.

#### Number of local viewsKK\(LeJEPA hyperparameter\)\.

[Fig\.15](https://arxiv.org/html/2607.15295#A12.F15)sweeps the number of local \(modality\-dropout\) viewsKKacross\{2,4,6,8\}\\\{2,4,6,8\\\}while keeping the two global views fixed\. The linear and attentive probes are largely insensitive toKK, suggesting that the default of two local views already provides sufficient cross\-modal regularisation; we did not see a clear benefit from increasingKK\.

### L\.1Loss\-component and pipeline ablations

We additionally run four hyperparameter ablations of the VGGSound recipe, each with one change from the 1\-GPU ablation baseline \(ViT\-B, batch size 32,λ=0\.05\\lambda\{=\}0\.05, modality dropout and video tubelet/frequency\-time masking, attentive probe\)\. All runs share the same wall\-clock budget and reach∼\\sim6–7 epochs of VGGSound pretraining; figures are clipped to the common range\. We deliberately plot only the LeJEPA loss components and the embedding standard deviation, not probe top\-1: withλ=0\\lambda\{=\}0the embedding collapses to a point and withλ=1\\lambda\{=\}1views never align, so probe accuracy is not a meaningful basis for comparison across these runs\. The losses and the embedding distribution remain directly comparable\.

#### Removing the SIGReg term \(λ=0\\lambda\{=\}0\)\.

[Fig\.16](https://arxiv.org/html/2607.15295#A12.F16)setsλ=0\\lambda\{=\}0, training only on the invariance loss without the SIGReg regulariser\. The invariance loss drops to∼10−7\{\\sim\}10^\{\-7\}almost immediately and the embedding standard deviation collapses from the baseline trajectory towards∼10−4\{\\sim\}10^\{\-4\}, both consistent with the trivial constant\-output solution\. The SIGReg loss itself, although not optimised, rises far above the baseline range, confirming that the resulting embedding distribution is arbitrarily far from the target isotropic Gaussian\. This isolates SIGReg as the component that prevents representation collapse in the audio\-visual setting, mirroring the role it plays in LeJEPA on images\.

#### Removing the invariance term \(λ=1\\lambda\{=\}1\)\.

[Fig\.17](https://arxiv.org/html/2607.15295#A12.F17)setsλ=1\\lambda\{=\}1, training only on SIGReg\. The embedding standard deviation reaches the target∼1\{\\sim\}1, confirming that SIGReg alone can shape the distribution, but the invariance loss stays roughly an order of magnitude above the baseline throughout training: there is no signal pulling different views of the same clip together\. This isolates the invariance loss as the component responsible for cross\-modal alignment\.

#### Removing modality dropout and masking\.

[Fig\.18](https://arxiv.org/html/2607.15295#A12.F18)disables modality dropout, tube masking, and frequency/time masking \(pdrop=0p\_\{\\mathrm\{drop\}\}\{=\}0,rvid=0r\_\{\\mathrm\{vid\}\}\{=\}0,raud=0r\_\{\\mathrm\{aud\}\}\{=\}0\)\. Local views then differ from the global views only through random resized crops and horizontal flips, while always retaining both modalities\. The invariance loss drops one to two orders of magnitude below the baseline trajectory because the alignment task is trivialised: there is no partial\-view perturbation forcing the model to predict the missing modality from the surviving one\. SIGReg behaves similarly to the baseline, so the embedding distribution is still well shaped, but the LeJEPA loss as a whole loses its information bottleneck\. This confirms that modality dropout \(rather than just spatial cropping\) is what makes the JEPA target non\-trivial in the audio\-visual setting\.

#### Backbone capacity: ViT\-Small vs\. ViT\-Base\.

[Fig\.19](https://arxiv.org/html/2607.15295#A12.F19)replaces the ViT\-B backbone \(dz=768d\_\{z\}\{=\}768,∼86\{\\sim\}86M parameters\) with ViT\-S \(dz=384d\_\{z\}\{=\}384,∼22\{\\sim\}22M parameters\), keeping every other hyperparameter fixed\. ViT\-S follows the same overall loss trajectory as the baseline: the LeJEPA, invariance, and SIGReg losses track the ViT\-B curves within a small constant offset, and the embedding standard deviation converges toward the same regime\. The smaller backbone reaches a slightly lower invariance loss within the 6–7\-epoch budget, plausibly because there is less representational capacity to spread across, but the qualitative dynamics are unchanged\. This indicates that the LeJEPA recipe is not specific to ViT\-B and transfers to a smaller backbone without retuningλ\\lambdaor the view schedule\.

![Refer to caption](https://arxiv.org/html/2607.15295v1/x6.png)Figure 5:AV\-JEPA early\-fusion encoder\.Video tubelets and mel\-spectrogram patches are embedded, summed with factorized positional and modality\-type embeddings, concatenated with a\[CLS\]token, and processed by a1212\-layer ViT\-Base\. The same encoder is used for every view, including the modality\-dropout local views, where the absent modality’s input tensor is zeroed before patch embedding\.Table 4:Fine\-tuning configuration\(VGGSound classification, main experiment fine\-tuning the AudioSet\-pretrained backbone for the headline 57\.1% top\-1 result\)\. The controlled VGGSound\-only secondary study uses the same recipe but with 6 epochs\.Table 5:Pretraining configurations\.Both runs share the same backbone \(ViT\-Base early\-fusion\), optimizer, and LeJEPA recipe, and differ only in dataset, batch size, and number of epochs\. The online linear and attentive classification probes are attached only to the VGGSound\-only run\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x7.png)\(a\)AudioSet\-2M \(57 ep\.\)
![Refer to caption](https://arxiv.org/html/2607.15295v1/x8.png)\(b\)VGGSound \(50 ep\.\)

Figure 6:Embedding standard deviationover training\. SIGReg drives the per\-dimension std toward 1\.0 \(the isotropic Gaussian target\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x9.png)\(a\)Invariance loss
![Refer to caption](https://arxiv.org/html/2607.15295v1/x10.png)\(b\)SIGReg loss

Figure 7:Pretraining loss components\(VGGSound\-only run, 50 epochs\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x11.png)\(a\)Attentive probe top\-1
![Refer to caption](https://arxiv.org/html/2607.15295v1/x12.png)\(b\)Probe top\-5 \(linear vs\. attentive\)
![Refer to caption](https://arxiv.org/html/2607.15295v1/x13.png)\(c\)Probe cross\-entropy loss

Figure 8:Online probing curveson frozen backbone features during pretraining\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x14.png)\(a\)Invariance loss
![Refer to caption](https://arxiv.org/html/2607.15295v1/x15.png)\(b\)SIGReg loss

Figure 9:AudioSet pretraining loss componentsover 57 epochs of AudioSet\-2M\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x16.png)\(a\)Top\-1 / Top\-5 accuracy
![Refer to caption](https://arxiv.org/html/2607.15295v1/x17.png)\(b\)Cross\-entropy loss

Figure 10:Fine\-tuning curvesfor end\-to\-end fine\-tuning of the pretrained AV\-JEPA backbone\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x18.png)\(a\)AudioSet
![Refer to caption](https://arxiv.org/html/2607.15295v1/x19.png)\(b\)VGGSound

Figure 11:Joint embedding t\-SNE by modality\.Projected\[CLS\]embeddings on the 5\-per\-class retrieval subsets, with each clip contributing an audio\-only \(orange\) and a video\-only \(blue\) point\. The two modalities are mixed rather than separated, indicating a shared cross\-modal embedding space\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x20.png)Figure 12:Semantic structure of the embedding space\.t\-SNE of the\[CLS\]embeddings of the fine\-tuned AV\-JEPA ViT\-B backbone forN=11,143N\{=\}11\{,\}143VGGSound training clips drawn from6060classes grouped into six semantic families \(colours\)\. Clips cluster by family with finer per\-class sub\-structure of the same colour; the residual overlap falls mainly between acoustically related families \(animal calls vs\. human voice\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x21.png)Figure 13:Architectural ablation: shared vs\. dual encoderon VGGSound pretraining\. Linear\-/attentive\-probe top\-1 accuracy, embedding standard deviation, invariance loss, SIGReg loss, and total LeJEPA loss are plotted against pretraining epoch\. Curves are clipped to the shorter of the two runs \(∼\\sim53 epochs\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x22.png)Figure 14:LeJEPA hyperparameter sensitivity: SIGReg weightλ\\lambdaon VGGSound pretraining, sweepingλ∈\{0\.03,0\.05,0\.10\}\\lambda\\in\\\{0\.03,0\.05,0\.10\\\}withλ=0\.05\\lambda\{=\}0\.05as the baseline used elsewhere in the paper\. Curves are clipped to the shortest run \(λ=0\.10\\lambda\{=\}0\.10,∼\\sim21 epochs\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x23.png)Figure 15:LeJEPA hyperparameter sensitivity: number of local viewsKKon VGGSound pretraining, sweepingK∈\{2,4,6,8\}K\\in\\\{2,4,6,8\\\}withK=2K\{=\}2as the paper’s default\. Curves are clipped to the shortest run \(K=8K\{=\}8,∼\\sim7 epochs\)\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x24.png)Figure 16:Removing SIGReg \(λ=0\\lambda\{=\}0\)vs\. the baseline \(λ=0\.05\\lambda\{=\}0\.05\) on VGGSound pretraining\. Without the SIGReg term, the invariance loss collapses to zero and the embedding standard deviation drops several orders of magnitude below the baseline, indicating representation collapse\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x25.png)Figure 17:Removing the invariance loss \(λ=1\\lambda\{=\}1\)vs\. the baseline \(λ=0\.05\\lambda\{=\}0\.05\) on VGGSound pretraining\. SIGReg alone drives the embedding standard deviation to its target, but the invariance loss never decreases, indicating that the model never learns to align global and local views without the invariance term\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x26.png)Figure 18:Removing modality dropout and tube/freq\-time maskingvs\. the baseline on VGGSound pretraining\. Without partial\-view perturbations the invariance loss collapses to near zero, indicating that alignment between global and local views becomes trivial when both modalities are always present and unmasked\.![Refer to caption](https://arxiv.org/html/2607.15295v1/x27.png)Figure 19:ViT\-Small backbonevs\. the ViT\-Base baseline on VGGSound pretraining\. All three loss components and the embedding standard deviation track the baseline curves within a small offset, indicating that the LeJEPA recipe transfers to a smaller backbone without retuning\.![Refer to caption](https://arxiv.org/html/2607.15295v1/figures/fig_pca_piano.png)\(a\)Playing piano
![Refer to caption](https://arxiv.org/html/2607.15295v1/figures/fig_pca_bass.png)\(b\)Playing bass guitar
![Refer to caption](https://arxiv.org/html/2607.15295v1/figures/fig_pca_violin.png)\(c\)Playing violin

Figure 20:Feature PCA of video patch tokensfor three VGGSound instrument classes\. A single PCA is fit jointly over the last\-layer video patch tokens of four clips per class \(rows\); its top three components are mapped to RGB and overlaid on four frames per clip \(columns\)\. The instrument takes a consistent colour across clips and frames, distinct from the player and the background\.