self-supervised-learning

Introduces CardioState-JEPA, a cardiac foundation model that learns a shared representation across ECG, PPG, and PCG signals using a delay-aware joint-embedding predictive architecture, improving downstream cardiac classification tasks.

This paper presents a controlled study on ECG self-supervised representation learning, examining how temporal context length (16s to 10min) and encoding strategy (continuous patch embeddings vs discretized tokens) affect downstream rhythm detection and patient-level retrieval. Results show longer context and continuous encoders improve performance, motivating extended-context ECG foundation models.

BioM-JEPA introduces a joint-embedding predictive architecture that learns single-cell representations by predicting graph-connected gene blocks instead of individual genes, showing improved efficiency and downstream performance in perturbation-response tasks.

This paper introduces HiLP, a hierarchical representation training method that adds multi-scale self-predictive learning to transformer pretraining, aiming to reduce compounding error and improve long-horizon reasoning and speculative decoding efficiency.

This paper introduces Spectral Aliasing Pretext (SAP), a self-supervised learning method for fault diagnosis in rotating machinery. By deliberately undersampling vibration signals and training a Transformer to reconstruct the original spectrum, SAP learns discriminative frequency-domain representations that achieve strong classification performance with limited labeled data.

This paper introduces NodeJEPA, a joint-embedding predictive architecture for node-level graph self-supervised learning that predicts latent representations of masked structure-aware ego-subgraphs, avoiding reconstruction and hand-crafted augmentations. The method is evaluated on node classification benchmarks and shows competitive performance.

This paper proposes an attention-only white-box Transformer trained with LeJEPA-based self-supervised pretraining, achieving competitive accuracy on CIFAR-10/100 while cutting parameters by ~31% compared to CRATE, and further shows MLP redundancy in standard ViTs.

This preprint introduces hierarchical self-supervised world models for music co-creation agents, with fast CPU-friendly models and a live demo for MIDI inpainting and generation.

WorldCycle proposes a self-verifiable reinforcement learning method for long-horizon video world models, using reversible action cycles as free supervision to reduce state-returning drift by up to 44% and boost composite-action accuracy nearly 4x. It also introduces CycleBench to evaluate world models as simulators.

This paper introduces CoCoEvolve, a self-supervised method that improves cross-representation understanding across charts, tables, and code by enforcing one-to-one consistency between representations, with training-time and test-time co-evolution objectives.

A ten-seed study of self-supervised speech encoders shows that the learning objective (reconstruction vs. prediction) governs cross-lingual perceptual narrowing, with reconstruction degrading non-native phoneme discrimination and prediction improving it.

This paper proposes DualIFM, an interpretable-by-design foundation model for retinal fundus images, achieving performance comparable to RETFound with far fewer parameters while providing interpretable predictions.

MoLGE assigns dedicated expert modules to clusters of similar languages in a mixture-of-experts framework for large-scale multilingual ASR, achieving improvements across 495 languages with minimal parameter increase.

This paper analyzes why deterministic JEPA-style latent prediction works for images but not for text, attributing the failure to high conditional variance in language where masked contexts admit multiple valid completions whose representations lack a coherent center.

This paper introduces AMRD, an adaptive multi-teacher relational distillation method for compressing large self-supervised speech emotion recognition models into lightweight student models for edge devices. It addresses teacher reliability variation and relational structure loss, showing improvements on IEMOCAP and CREMA-D datasets.

Proposes Unbiased Open World Regularization (UOWReg), an encoder-only framework that enforces conditional distribution matching to achieve statistical independence between learned representations and sensitive attributes, reducing bias while maintaining accuracy.

An analysis of Yann LeCun's bet that intelligence starts with world models via JEPA, not language, supported by AMI Labs' $1.03 billion funding. The article explains why next-pixel prediction fails and how JEPA predicts in latent space to avoid blurry futures.

The author revisits influential open-source works in representation learning, listing key papers from MoCo v1 to LeJEPA that advanced vision foundation models and self-supervised learning.

SenCos-GEM introduces a physics-guided molecular representation learning framework that uses Squeeze-and-Excitation modules and a law-of-cosines constraint to improve 3D geometric understanding, achieving state-of-the-art results on MoleculeNet regression benchmarks.

Explains SIGReg, a novel regularizer for JEPA that prevents representation collapse by forcing embeddings to follow an isotropic Gaussian distribution, with theoretical guarantees and a clean training loop.