IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

Hugging Face Daily Papers Papers
discrete-representation autoencoder alignment image-reconstruction image-generation vision-foundation-models

Summary

IDEAL proposes an in-depth alignment framework for discrete representation autoencoding, jointly aligning quantized tokens with shallow and deep VFM features to achieve superior reconstruction and generation performance.

Built on pretrained vision foundation models (VFMs), representation autoencoders (RAEs) have recently emerged as a promising approach for constructing semantically rich latent spaces for image generation. However, their reconstruction quality often remains suboptimal, largely because deep VFM representations do not preserve sufficient fine-grained visual detail. This limitation becomes even more severe after discretization, where missing low-level information is difficult to recover. In fact, we observe that shallow VFM features retain considerably richer local appearance and structural detail, which complements the high-level semantics carried by deep features used in existing RAEs. Motivated by this complementary property, we propose Ideal, an In-depth Alignment framework for discrete representation autoencoding. By jointly aligning quantized tokens with both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve both visual fidelity and rich semantics. Extensive experiments demonstrate that Ideal yields superior reconstruction performance, achieving 0.61 rFID on ImageNet and outperforming the previous best method by 0.28. When used for autoregressive image generation, Ideal further produces a gFID of 1.89, establishing a new state of the art for autoregressive image generation.
Original Article
View Cached Full Text

Cached at: 06/12/26, 06:50 AM

Paper page - IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

Source: https://huggingface.co/papers/2606.11096

Abstract

Representation autoencoders using deep learning frameworks can improve image reconstruction quality by combining shallow and deep visual feature representations for better semantic richness and visual fidelity.

Built on pretrainedvision foundation models(VFMs),representation autoencoders(RAEs) have recently emerged as a promising approach for constructing semantically rich latent spaces for image generation. However, their reconstruction quality often remains suboptimal, largely because deep VFM representations do not preserve sufficient fine-grained visual detail. This limitation becomes even more severe after discretization, where missing low-level information is difficult to recover. In fact, we observe that shallow VFM features retain considerably richer local appearance and structural detail, which complements the high-level semantics carried by deep features used in existing RAEs. Motivated by this complementary property, we propose Ideal, an In-depth Alignment framework fordiscrete representation autoencoding. By jointly aligningquantized tokenswith both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve bothvisual fidelityand rich semantics. Extensive experiments demonstrate that Ideal yields superior reconstruction performance, achieving 0.61rFIDon ImageNet and outperforming the previous best method by 0.28. When used forautoregressive image generation, Ideal further produces agFIDof 1.89, establishing a new state of the art forautoregressive image generation.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2606\.11096

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.11096 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.11096 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.11096 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse

arXiv cs.LG

This paper addresses the issue of dimensional collapse in VQ-VAEs, showing that representations often occupy a low-dimensional subspace. It proposes an 'AE Warm-Up' strategy that trains the model as an unquantized autoencoder first, which improves reconstruction quality and increases effective latent dimensionality.

FRAPPE: Full Input, Residual Output Autoencoding with Projection Pursuit Encoder

Hugging Face Daily Papers

FRAPPE is a novel autoencoding framework that uses a projection pursuit encoder to predict residuals from full input, enabling efficient variable-rate image compression with fast CPU-based encoding. At high compression ratios, FRAPPE-Image achieves higher perceptual quality than AVIF with 47x faster encoding, making real-time 1080p 30fps CPU-only encoding possible.