HiUni: Harmonizing Understanding and Generation in Shared Visual Space
Summary
HiUni explores extending a pretrained vision-language model into a unified multimodal model via extensive parameter sharing, reusing a single dense backbone for both understanding and generation with a PixelDiffusionHead and HiGate pathway to address optimization and spatial-detail challenges. Experiments show it matches the pretrained VLM's understanding score while achieving competitive image generation and editing performance.
View Cached Full Text
Cached at: 11/20/26, 11:35 AM
Paper page - HiUni: Harmonizing Understanding and Generation in Shared Visual Space
Source: https://huggingface.co/papers/2611.26107 Authors:
,
,
,
,
,
,
,
,
,
,
,
Abstract
Unifiedmultimodalmodelsaimtosupportvisualunderstandingandgenerationwithinasingleframework.However,prevailingapproachesoftenseparatetheirvisualrepresentationsorbackbonecomputationtoaccommodatethedifferentrequirementsofthesetwotasks,whichlimitsthedegreeofunification.Inthispaper,wepresentHiUni,whichrevisitsthisdesignchoiceandexploreswhetherapretrainedvisionlanguagemodelcanbedirectlyextendedintoaunifiedmodelwithextensiveparametersharing.Specifically,HiUnireusesthesamelanguage-alignedViTanddenseTransformerbackboneforbothcapabilities,whileconfiningtask-specificcomputationtotheoutputside.Yetsuchadesignintroducestwokeychallenges.Generationinpixelspaceisdifficulttooptimize,andsemanticabstractioninpretrainedViTdiscardsfinespatialdetailsrequiredforsynthesis.Toaddressthesechallenges,HiUniintroducesaPixelDiffusionHeadtailoredtodirectpixel-spaceoptimizationandalightweightHiGatepathwaythatprovidesfine-grainedspatialdetailsneartheoutputstage.ExtensiveexperimentsshowthatHiUniachievesasimilaraverageunderstandingscoretothepretrainedVLMwhileattainingcompetitiveperformanceonimagegenerationandediting.Controlledcomparisonsandanalysesfurtherdemonstratethatasingledensebackbonecaneffectivelysupportbothcapabilitieswithouttask-specificbackbonecomputation,highlightingasimplerpathtowardunifiedmultimodalmodeling.
View arXiv pageView PDFProject pageGitHub7Add to collection
Get this paper in your agent:
hf papers read 2611\.26107
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### ernie-research/HiUni Image-Text-to-Text• 5B• Updatedabout 2 hours ago • 1
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2611.26107 in a dataset README.md to link it from this page.
Spaces citing this paper1
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
UniSpace: Unified Visual Representation and Scalable Multimodal Modeling
UniSpace introduces a unified visual representation that unifies semantic understanding, high-fidelity reconstruction, and image generation in a single space using a reparameterized ViT, eliminating the need for a separate VAE.
UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation
UniSpark-9B is a unified multimodal model combining full attention with Gated DeltaNet linear attention in a Mixture-of-Transformer-Experts backbone, jointly supporting visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, achieving 2470.3 on MME-S and 0.83 on GenEval.
Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
UniAR presents a unified autoregressive framework that uses a single discrete visual tokenizer to bridge visual understanding and generation, achieving state-of-the-art results in image generation and editing.
Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
This paper introduces UNO, an Understanding-Oriented Post-Training framework that uses comprehension tasks as supervisory signals to enhance image generation and editing in unified multimodal models.
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
UniDDT proposes a decoupled diffusion transformer framework that unifies multimodal understanding and generation by leveraging a Noisy ViT encoder and LLM for semantic encoding, achieving strong performance on both tasks.