You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences

Hugging Face Daily Papers Papers
self-supervised-learning video-representation temporal-difference causal-assumption inductive-bias visual-learning

Summary

The paper introduces Temporal Difference in Vision (TDV), a self-supervised learning method for video that relies only on a causal assumption that past causes future, avoiding strong inductive biases while matching state-of-the-art on dense spatial tasks.

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.
Original Article
View Cached Full Text

Cached at: 06/16/26, 07:33 PM

Paper page - You Don’t Need Strong Assumptions: Visual Representation Learning via Temporal Differences

Source: https://huggingface.co/papers/2606.15956

Abstract

Temporal Difference in Vision (TDV) presents a novel self-supervised learning approach for video data that eliminates traditional inductive biases by leveraging causal relationships between past and future frames.

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weakerinductive biasesgenerally outperform those with stronger assumptions. This is particularly characteristic of the field ofVisual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success ofSelf-Supervised Learningwithout human labels. Yet, even modernSelf-Supervised Learningapproaches still depend on stronginductive biasessuch as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength ofinductive biasesdecreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduceTemporal Differencein Vision (TDV), a new paradigm forself-supervised learningfrom video that avoids existinginductive biases, relying instead on acausal assumptionthat the past causes the future. TDV functions by jointly training animage encoderand amotion encoderso that the current frame’s representation plus the encoded motion equals the next frame’s representation. Despite not leveraging any stronginductive biases, TDV matches state-of-the-art recipes ondense spatial tasks, laying the foundation for representation learning without strong assumptions.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2606\.15956

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.15956 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.15956 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.15956 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Temporal Difference Learning for Diffusion Models

arXiv cs.LG

This paper introduces a temporal difference (TD) learning objective for diffusion models that enforces cross-time consistency along the denoising trajectory. It reformulates denoising as a reinforcement learning policy evaluation problem, showing significant improvements in sample quality (FID), especially for few-step samplers.