A Theory of Contrastive Learning with Natural Images
Summary
This paper analytically computes the optimal representations under a contrastive loss for basic augmentations and natural images with stationary statistics, showing that the optimal CNN first-layer filters are sinusoids and that weights can be computed via a waterfilling algorithm.
View Cached Full Text
Cached at: 07/14/26, 08:17 PM
Paper page - A Theory of Contrastive Learning with Natural Images
Source: https://huggingface.co/papers/2607.07470 Published on Jul 8
·
Submitted byhttps://huggingface.co/ShadenA
Shadenon Jul 14
Abstract
Whydoescontrastivelearningwithsimpleimagesandaugmentationsyieldusefulrepresentationsfordownstreamtasks?Weaddressthisquestionbyanalyticallycomputingtheoptimalrepresentationintermsofacontrastivelossforarangeofbasicaugmentationsandanyimagedatasetwithstationarystatistics.WeshowthatforcertainaugmentationstheoptimumcanbeattainedbyaCNNwhosefirstlayerfiltersaresinusoids,followedbyapointwisenonlinearity,globalaveragepooling,andafinallinearlayerthatperformspartialwhitening.WealsoshowthattheoptimalweightsinsuchCNNsformorecomplicatedaugmentationsarestillsinusoids.Thefrequenciesofthesinusoidsandtheirweightscanbecomputedusingasimplewaterfillingalgorithmgiventhedataset’sexpectedpowerspectrum.ExperimentswithdifferentimagedatasetsandaugmentationsshowthatsuchCNNstrainedwithSGDempiricallylearnsinusoidsintheirfirstlayerandtoperformpartialwhitening
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.07470 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.07470 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.07470 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
This paper develops a measure-theoretic framework analyzing when contrastive learning recovers meaningful latent geometry, introducing a 'diversity condition' on positive-pair sampling and a support-corrected InfoNCE variant, with experiments validating that sampling diversity and architectural inductive bias interact critically in contrastive representation learning.
A Unified Geometric Framework for Weighted Contrastive Learning
This paper introduces a unified geometric framework showing that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, providing exact characterizations of optimal embeddings for supervised and weakly supervised contrastive learning methods and revealing when such embeddings are geometrically realizable, degenerate, or inconsistent.
Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling
This paper derives a scaling law for sketched linear contrastive learning under a Gaussian latent-variable model, analyzing how risk decomposes into approximation, optimization, and statistical terms, and provides theoretical guidance for balancing model size, data, and compute in contrastive learning.
A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss
SimLoss uses embedding-space contrastive supervision to enable single-pass fine-grained image captioning that matches multi-stage quality at much lower latency, with variants like SimLoss FFT achieving high precision.
Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks
This paper evaluates the biological plausibility and representational alignment of feedback alignment algorithms in convolutional networks, comparing them to standard backpropagation on CIFAR-10. The authors find that modified feedback alignment methods converge on internal representations similar to those produced by backpropagation, suggesting functional success through mimicking representational geometry.