A Stationary (and Therefore Compatible) Representation is All You Need

Hugging Face Daily Papers Papers
stationary-representations compatible-representations d-simplex-fixed-classifiers model-updates retrieval-services fine-tuning contrastive-loss

Summary

Introduces stationary representations learned via d-Simplex fixed classifiers to ensure model compatibility during sequential fine-tuning, enabling continuous retrieval services without reprocessing. Combines cross-entropy and contrastive losses to capture higher-order dependencies.

Learning compatible representations aims to learn feature representations that can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate that stationary representations learned by d-Simplex fixed classifiers imply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility using d-Simplex fixed classifiers when the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with the cross-entropy loss aligns feature distributions at the first-order statistics. Consequently, it may not fully capture higher-order dependencies in the representation between model updates. To address this issue, we demonstrate that training the model using a d-Simplex fixed classifier through a convex combination of the cross-entropy loss and a contrastive loss not only captures higher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show that stationary representations enable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.
Original Article
View Cached Full Text

Cached at: 06/12/26, 02:53 PM

Paper page - A Stationary (and Therefore Compatible) Representation is All You Need

Source: https://huggingface.co/papers/2606.12488

Abstract

Stationary representations learned through d-Simplex fixed classifiers ensure model compatibility during sequential fine-tuning and updates, enabling continuous retrieval services without reprocessing.

Learning compatible representations aims to learnfeature representationsthat can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate thatstationary representationslearned byd-Simplex fixed classifiersimply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility usingd-Simplex fixed classifierswhen the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with thecross-entropy lossaligns feature distributions at the first-order statistics. Consequently, it may not fully capturehigher-order dependenciesin the representation between model updates. To address this issue, we demonstrate that training the model using a d-Simplex fixed classifier through aconvex combinationof thecross-entropy lossand acontrastive lossnot only captureshigher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show thatstationary representationsenable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.12488

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.12488 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.12488 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.12488 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Representation Collapse in Sequential Post-Training of Large Language Models

arXiv cs.LG

This paper studies representation collapse in sequential post-training of large language models, showing that repeated adaptation stages compress internal representations, reducing plasticity and out-of-domain generalization. The authors propose lightweight interventions to preserve future learnability without sacrificing behavioral gains.

A Theory of Contrastive Learning with Natural Images

Hugging Face Daily Papers

This paper analytically computes the optimal representations under a contrastive loss for basic augmentations and natural images with stationary statistics, showing that the optimal CNN first-layer filters are sinusoids and that weights can be computed via a waterfilling algorithm.

Bayes-Sufficient Representations in Supervised Learning

arXiv cs.LG

This paper formalizes the concept of Bayes-sufficient representations in supervised learning, defining when a representation retains exactly the information needed for Bayes-optimal prediction under a given loss function. It introduces the Bayes quotient as a canonical loss-dependent object and connects the framework to property elicitation, illustrating distinctions between sufficiency, minimality, and excess retained information through experiments.

Which Tasks Survive Self-Supervised Learning?

arXiv cs.LG

This Texas A&M paper introduces "semantic recoverability" to explain which downstream tasks survive same-instance self-supervised learning, linking recoverability to class-distance-normalized variance, few-shot nearest-centroid transfer, and the spectral subspace spanned by cross-view-stable modes of the SSL objective.