Invisible Shortcuts: Why Vision Encoders Know Your Camera

Hugging Face Daily Papers Papers
vision-encoders metadata-traces shortcut-learning pretraining distribution-shift generalization

Summary

This paper identifies invisible metadata traces at the pixel level as shortcuts that vision encoders exploit, leading to performance degradation under metadata distribution shifts. Mitigation strategies during and after pretraining reduce sensitivity to both targeted and unseen metadata without sacrificing downstream performance.

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:55 AM

Paper page - Invisible Shortcuts: Why Vision Encoders Know Your Camera

Source: https://huggingface.co/papers/2608.05424

Abstract

Deepvisionmodelsexploitshortcuts,relyingoncuesthatcorrelatewithsupervisionsignals.Priorworkhasfocusedonvisiblebiases,suchasobject-backgroundortexturecorrelations.Weidentifyadifferentsourceofshortcutlearning:invisiblemetadatatracesembeddedatthepixellevel,formetadatasuchasimageprocessingandphotoacquisition.Wehypothesizethatlarge-scalesemanticsupervision,whetherthroughcategoricallabels(ImageNet)orbillion-scalecaptions(LAION),naturallyinducesmetadata-semanticscorrelationsduringpretraining,leadingmodelstoconvertlow-levelsignalsintopredictivefeatures.Byintroducingcontrolledmetadata-semanticscorrelations,weshowthatstrongeronesproducesystematicallyhighersensitivitytometadatatracesandlargerperformancedegradationundermetadatadistributionshifts.Wefurtherexploremitigationstrategiesappliedduringandafterpretrainingthatreducesensitivitynotonlytotargetedmetadatabutalsotounseenones,withoutsacrificingperformanceondownstreamtasks.Metadatasensitivityalsohasapositiveside:itpartlyexplainsthestronggenerated-imagedetectionabilityofsomeencoders,whileitsmitigationcanimproveout-of-distributiongeneralization.Code:https://github.com/ryan-caesar-ramos/visual-encoder-traces

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.05424 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.05424 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.05424 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles