HiRAE: Hierarchical Representation Autoencoding with Residual Budgets

Hugging Face Daily Papers Papers
autoencoding hierarchical-representation image-reconstruction generative-models residual-budgets

Summary

HiRAE is a hierarchical representation autoencoder that improves image reconstruction fidelity and generative modeling compatibility, reducing FID and enhancing alignment in text-to-image generation.

Pretrained visual representations support image generation, but may not fully preserve the fine-grained details needed for faithful reconstruction. Meanwhile, intermediate encoder layers contain complementary visual details, but learning to fuse them for reconstruction can produce a latent distribution that is difficult to model. Existing fusion methods require empirical tuning of layer selection or staged optimization of fusion and decoding, increasing configuration effort or training complexity. We introduce HiRAE (Hierarchical Representation Autoencoder), which learns a hierarchical fusion framework over the full encoder hierarchy to improve reconstruction fidelity while maintaining compatibility with generative modeling. HiRAE groups encoder layers by depth and learns residual corrections to the deepest representation. Group-wise norm caps bound these corrections relative to the deep anchor, with tighter budgets for shallower groups. Our HiRAE-24 preserves the latent token count and channel dimension. On ImageNet-256, HiRAE-24 reduces reconstruction FID from 0.299 to 0.209 relative to RAEv2 while maintaining competitive guided generation quality. For text-to-image generation, HiRAE-24 improves alignment over RAEv2 on GenEval, DPG-Bench, and GenAI-Bench both before and after supervised fine-tuning. Under the same generator-training and evaluation protocol, post-fine-tuning GenEval increases from 84.86 to 87.70.
Original Article
View Cached Full Text

Cached at: 09/30/26, 04:21 AM

Paper page - HiRAE: Hierarchical Representation Autoencoding with Residual Budgets

Source: https://huggingface.co/papers/2609.37775

Abstract

Pretrainedvisualrepresentationssupportimagegeneration,butmaynotfullypreservethefine-graineddetailsneededforfaithfulreconstruction.Meanwhile,intermediateencoderlayerscontaincomplementaryvisualdetails,butlearningtofusethemforreconstructioncanproducealatentdistributionthatisdifficulttomodel.Existingfusionmethodsrequireempiricaltuningoflayerselectionorstagedoptimizationoffusionanddecoding,increasingconfigurationeffortortrainingcomplexity.WeintroduceHiRAE(HierarchicalRepresentationAutoencoder),whichlearnsahierarchicalfusionframeworkoverthefullencoderhierarchytoimprovereconstructionfidelitywhilemaintainingcompatibilitywithgenerativemodeling.HiRAEgroupsencoderlayersbydepthandlearnsresidualcorrectionstothedeepestrepresentation.Group-wisenormcapsboundthesecorrectionsrelativetothedeepanchor,withtighterbudgetsforshallowergroups.OurHiRAE-24preservesthelatenttokencountandchanneldimension.OnImageNet-256,HiRAE-24reducesreconstructionFIDfrom0.299to0.209relativetoRAEv2whilemaintainingcompetitiveguidedgenerationquality.Fortext-to-imagegeneration,HiRAE-24improvesalignmentoverRAEv2onGenEval,DPG-Bench,andGenAI-Benchbothbeforeandaftersupervisedfine-tuning.Underthesamegenerator-trainingandevaluationprotocol,post-fine-tuningGenEvalincreasesfrom84.86to87.70.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.37775

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.37775 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.37775 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.37775 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

V-RAE: Rethinking Video Latent Spaces for Generation

Hugging Face Daily Papers

V-RAE proposes a video representation autoencoder that builds semantically organized latents from frozen vision representations to enhance video generation quality, convergence speed, and predictive modeling.

Rational Sparse Autoencoder

arXiv cs.LG

Introduces Rational Sparse Autoencoder (RSAE), which replaces fixed encoder activations with trainable rational functions, improving reconstruction and sparsity trade-offs on residual-stream activations of open-weight language models across multiple baseline families.