Paper page - UniSpace: Unified Visual Representation and Scalable Multimodal Modeling

Source: https://huggingface.co/papers/2608.08676 Published on Aug 9

·

Submitted byhttps://huggingface.co/yjb6

yjbon Aug 24

Abstract

A reparameterized pretrained vision transformer unifies semantic understanding, high-fidelity reconstruction, and image generation within a single visual space without requiring a separate VAE.

Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrainedsemantic ViT. We show that the frozen Transformer blocks of asemantic ViTare not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introducePatch Reparameterization, which preserves the original semantic pathway while adding areconstruction-aware patch embeddingthat provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation intoUniSpace, an 8BMixture-of-Transformer-Expertsmodel that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as aunified visual interfacefor scalable multimodal modeling.

View arXiv pageView PDFProject pageGitHub5Add to collection

Get this paper in your agent:

hf papers read 2608\.08676

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### yjb6/UniSpace

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.08676 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.08676 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.