A Theory of Contrastive Learning with Natural Images

Hugging Face Daily Papers Papers
contrastive-learning theory natural-images representation-learning augmentations cnn analytical-computation

Summary

This paper analytically computes the optimal representations under a contrastive loss for basic augmentations and natural images with stationary statistics, showing that the optimal CNN first-layer filters are sinusoids and that weights can be computed via a waterfilling algorithm.

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening
Original Article
View Cached Full Text

Cached at: 07/14/26, 08:17 PM

Paper page - A Theory of Contrastive Learning with Natural Images

Source: https://huggingface.co/papers/2607.07470 Published on Jul 8

·

Submitted byhttps://huggingface.co/ShadenA

Shadenon Jul 14

Abstract

Whydoescontrastivelearningwithsimpleimagesandaugmentationsyieldusefulrepresentationsfordownstreamtasks?Weaddressthisquestionbyanalyticallycomputingtheoptimalrepresentationintermsofacontrastivelossforarangeofbasicaugmentationsandanyimagedatasetwithstationarystatistics.WeshowthatforcertainaugmentationstheoptimumcanbeattainedbyaCNNwhosefirstlayerfiltersaresinusoids,followedbyapointwisenonlinearity,globalaveragepooling,andafinallinearlayerthatperformspartialwhitening.WealsoshowthattheoptimalweightsinsuchCNNsformorecomplicatedaugmentationsarestillsinusoids.Thefrequenciesofthesinusoidsandtheirweightscanbecomputedusingasimplewaterfillingalgorithmgiventhedataset’sexpectedpowerspectrum.ExperimentswithdifferentimagedatasetsandaugmentationsshowthatsuchCNNstrainedwithSGDempiricallylearnsinusoidsintheirfirstlayerandtoperformpartialwhitening

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.07470 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.07470 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.07470 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

A Unified Geometric Framework for Weighted Contrastive Learning

arXiv cs.LG

This paper introduces a unified geometric framework showing that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, providing exact characterizations of optimal embeddings for supervised and weakly supervised contrastive learning methods and revealing when such embeddings are geometrically realizable, degenerate, or inconsistent.

Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks

arXiv cs.AI

This paper evaluates the biological plausibility and representational alignment of feedback alignment algorithms in convolutional networks, comparing them to standard backpropagation on CIFAR-10. The authors find that modified feedback alignment methods converge on internal representations similar to those produced by backpropagation, suggesting functional success through mimicking representational geometry.

Data-driven Video Codec with Implicit Neural Representations

arXiv cs.AI

This paper presents a video codec that stores video and audio as weights of a sinusoidal representation network, using knowledge distillation and quantization for compression. Experiments show a 2.61x compression ratio compared to the original network, but quality lags behind standard codecs like H.264 and HEVC.