UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Hugging Face Daily Papers Papers

Summary

UniSpark-9B is a unified multimodal model combining full attention with Gated DeltaNet linear attention in a Mixture-of-Transformer-Experts backbone, jointly supporting visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, achieving 2470.3 on MME-S and 0.83 on GenEval.

We present UniSpark-9B, a unified multimodal model that investigates hybrid attention for visual understanding and generation. Built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch, UniSpark combines full attention for explicit token interactions with GatedDeltaNet linear attention for compact context propagation. Autoregressive text prediction and visual flow matching jointly support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction. In interleaved generation, noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction through attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers. The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum jointly trains multimodal understanding and visual generation, refining answer accuracy and instruction following while extending image synthesis to 2K. UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. Qualitative examples further illustrate its capabilities in interleaved image generation, video synthesis, and video editing. Together, these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation.
Original Article
View Cached Full Text

Cached at: 11/16/26, 08:56 AM

Paper page - UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Source: https://huggingface.co/papers/2611.16517

Abstract

WepresentUniSpark-9B,aunifiedmultimodalmodelthatinvestigateshybridattentionforvisualunderstandingandgeneration.BuiltonaMixture-of-Transformer-Expertsbackbonewithapproximately9Bparametersineachbranch,UniSparkcombinesfullattentionforexplicittokeninteractionswithGatedDeltaNetlinearattentionforcompactcontextpropagation.Autoregressivetextpredictionandvisualflowmatchingjointlysupportvisualquestionanswering,text-to-imagegeneration,instruction-guidedediting,andinterleavedtext-imageprediction.Ininterleavedgeneration,noisyvisualtargetsareisolatedfromsubsequentpredictions,whilecleanvisualrepresentationsprovidecontextforlateroutputs.Ablockwiseprotocolenforcesthisdistinctionthroughattentionmasksinfull-attentionlayersandrecurrent-stateretentionrulesinlinear-attentionlayers.Theselectedtrainingsourcescontain159.7Mgenerationsamples,213.4Munderstandingsamples,and624.8Kinterleavedsequences.Afour-stagecurriculumjointlytrainsmultimodalunderstandingandvisualgeneration,refiningansweraccuracyandinstructionfollowingwhileextendingimagesynthesisto2K.UniSparkscores2470.3onMME-Sand0.83onGenEval.Qualitativeexamplesfurtherillustrateitscapabilitiesininterleavedimagegeneration,videosynthesis,andvideoediting.Together,theseresultsdemonstratethathybridattentionprovidesaneffectivebackboneforunifiedmultimodalunderstandingandgeneration.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2611\.16517

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2611.16517 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2611.16517 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2611.16517 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

HiUni: Harmonizing Understanding and Generation in Shared Visual Space

Hugging Face Daily Papers

HiUni explores extending a pretrained vision-language model into a unified multimodal model via extensive parameter sharing, reusing a single dense backbone for both understanding and generation with a PixelDiffusionHead and HiGate pathway to address optimization and spatial-detail challenges. Experiments show it matches the pretrained VLM's understanding score while achieving competitive image generation and editing performance.