HiUni: Harmonizing Understanding and Generation in Shared Visual Space

Hugging Face Daily Papers Papers

Summary

HiUni explores extending a pretrained vision-language model into a unified multimodal model via extensive parameter sharing, reusing a single dense backbone for both understanding and generation with a PixelDiffusionHead and HiGate pathway to address optimization and spatial-detail challenges. Experiments show it matches the pretrained VLM's understanding score while achieving competitive image generation and editing performance.

Unified multimodal models aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of these two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained vision language model can be directly extended into a unified model with extensive parameter sharing. Specifically, HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side. Yet such a design introduces two key challenges. Generation in pixel space is difficult to optimize, and semantic abstraction in pretrained ViT discards fine spatial details required for synthesis. To address these challenges, HiUni introduces a Pixel Diffusion Head tailored to direct pixel-space optimization and a lightweight HiGate pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons and analyses further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.
Original Article
View Cached Full Text

Cached at: 11/20/26, 11:35 AM

Paper page - HiUni: Harmonizing Understanding and Generation in Shared Visual Space

Source: https://huggingface.co/papers/2611.26107 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Unifiedmultimodalmodelsaimtosupportvisualunderstandingandgenerationwithinasingleframework.However,prevailingapproachesoftenseparatetheirvisualrepresentationsorbackbonecomputationtoaccommodatethedifferentrequirementsofthesetwotasks,whichlimitsthedegreeofunification.Inthispaper,wepresentHiUni,whichrevisitsthisdesignchoiceandexploreswhetherapretrainedvisionlanguagemodelcanbedirectlyextendedintoaunifiedmodelwithextensiveparametersharing.Specifically,HiUnireusesthesamelanguage-alignedViTanddenseTransformerbackboneforbothcapabilities,whileconfiningtask-specificcomputationtotheoutputside.Yetsuchadesignintroducestwokeychallenges.Generationinpixelspaceisdifficulttooptimize,andsemanticabstractioninpretrainedViTdiscardsfinespatialdetailsrequiredforsynthesis.Toaddressthesechallenges,HiUniintroducesaPixelDiffusionHeadtailoredtodirectpixel-spaceoptimizationandalightweightHiGatepathwaythatprovidesfine-grainedspatialdetailsneartheoutputstage.ExtensiveexperimentsshowthatHiUniachievesasimilaraverageunderstandingscoretothepretrainedVLMwhileattainingcompetitiveperformanceonimagegenerationandediting.Controlledcomparisonsandanalysesfurtherdemonstratethatasingledensebackbonecaneffectivelysupportbothcapabilitieswithouttask-specificbackbonecomputation,highlightingasimplerpathtowardunifiedmultimodalmodeling.

View arXiv pageView PDFProject pageGitHub7Add to collection

Get this paper in your agent:

hf papers read 2611\.26107

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### ernie-research/HiUni Image-Text-to-Text• 5B• Updatedabout 2 hours ago • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2611.26107 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Hugging Face Daily Papers

UniSpark-9B is a unified multimodal model combining full attention with Gated DeltaNet linear attention in a Mixture-of-Transformer-Experts backbone, jointly supporting visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, achieving 2470.3 on MME-S and 0.83 on GenEval.