Decoupled Vision-Language System for Multimodal Understanding and Generation

# Decoupled Vision-Language System for Multimodal Understanding and Generation
Source: [https://arxiv.org/html/2608.20382](https://arxiv.org/html/2608.20382)
Yifan Xu, Baochen Xiong, Xiaoshan Yang, Donglin Di, Yaowei Wang, Changsheng XuCorresponding author: Changsheng Xu\. Yifan Xu, Baochen Xiong, Xiaoshan Yang, and Changsheng Xu are with MAIS, Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing 100190, China, and also with the Peng Cheng Laboratory, Shenzhen 518066, China\. Email: yifan\.xu9674@gmail\.com, xiongbaochen2022@ia\.ac\.cn, \{xiaoshan\.yang, csxu\}@nlpr\.ia\.ac\.cn\. Donglin Di is with Li Auto, Beijing 101399, China\. Email: donglin\.ddl@gmail\.com\. Yaowei Wang is with Peng Cheng Laboratory, Shenzhen 518066, China\. Email: wangyw@pcl\.ac\.cn

###### Abstract

We introduce a new architecture design for multimodal large language models \(MLLMs\), Libra, capable of both multimodal understanding and generation\. Libra architecture contains one vision system and one language system, connected by cross\-modal bridges\. This design decouples self\-modal modeling and cross\-modal interaction, enabling each modality to learn its unique representations while maintaining effective cross\-modal comprehension\. The decoupling is mainly achieved in a switch attention module and a switch FFN module, which dynamically routes the computation flow for self\-modal modeling and cross\-modal interaction scenarios\. We evaluate the effectiveness in two important settings:Libra\-1for the understanding\-only image\-to\-text setting, andLibra\-2for unified image\-to\-text understanding and text\-to\-image generation\. In addition to the architecture design, we discuss various improvements on tokenization, positional encoding, and supervision\. Experiments demonstrate that the dedicated Libra design enables mutual improvements on multimodal understanding and generation, achieving strong performance on both understanding and generation benchmarks\. Code will be available at[https://github\.com/YifanXu74/Libra](https://github.com/YifanXu74/Libra)\.

## IIntroduction

Generative modeling, particularly the auto\-regressive paradigm in large language models \(LLMs\)\[gpt3,llama3,gemini\], has emerged as a promising pathway toward general\-purpose language systems over the past few years\. Its effectiveness stems from an inherent capacity to reconstruct data distributions, thereby implicitly capturing the underlying world knowledge embedded in the data\. Taking this a step further, in the realm of multimodal large language models \(MLLMs\)\[libra,llava,flamingo\], an intuitive idea emerges: if a model can effectively generate text and images, it should have acquired a deeper understanding of multimodal world knowledge, while achieving stronger alignment between the two modalities\. This, in turn, would lead to more robust multimodal understanding capabilities\.

There are two representative paradigms in vision\-language MLLMs: \(i\) understanding\-only models\[llava,flamingo,blip2\], which take images as conditions and generate text, namely image\-to\-text understanding, and \(ii\) unified models\[unified\-io,show\-o,janus\]for both image\-to\-text understanding and text\-to\-image generation\. Image\-to\-text understanding is a fundamental capacity of MLLMs as it enables the model to “see”, while text\-to\-image generation enables the model to learn visual world from the bottom up\. Consequently, understanding\-only models focus on the most fundamental image\-to\-text settings, often enabling more efficient and stable optimization; Unified models are, in principle, more general and scalable, but typically introducing greater training complexity and instability\.

Both paradigms faces a fundamental challenge in current research: the modality imbalance between vision and language\. Language data, with its highly structured symbolic representation, can compactly encode abstract concepts and relationships, while visual data organizes information in fundamentally different ways: a single image carries rich fine\-grained visual details, but it has much lower semantic density than text\. As a result, existing visual datasets lag behind text corpora in the breadth of knowledge coverage, while textual datasets fall short in describing the fine\-grained appearances compared to visual data\. This imbalance can easily lead to biased training, where the model favors one modality over the other, weakening performance on the less favored side\[flamingo\]\. Therefore, a reasonable model design is essential to address the modality imbalance\. As shown in Fig\.[1](https://arxiv.org/html/2608.20382#S1.F1), several types of model designs are proposed in current research:

![Refer to caption](https://arxiv.org/html/2608.20382v1/x1.png)Figure 1:Architecture comparison among generative MLLMs\. \(a\) A unified architecture models vision and language within a trainable shared backbone\. \(b\) A cascade architecture typically uses a frozen LLM as the language backbone and adds a trainable visual backend to provide visual generation capabilities\. \(c\) The Libra architecture combines the modularity of cascade architectures with the layer\-wise cross\-modal interaction of unified architectures\.\(1\) Unified architecture\[llava,unified\-io,chameleon,show\-o,janus\], shown in Fig\.[1](https://arxiv.org/html/2608.20382#S1.F1)\(a\), jointly models vision and language with a shared backbone\. Most understanding\-only MLLMs\[llava\]are also within this architecture but are trained solely with language supervision\. This design is conceptually optimal but not well suited for the current stage of data development\. A unified architecture, introducing minimal human prior, has the flexibility to learn arbitrary patterns for organizing vision and language, given large\-scale and appropriate data\. However, previous researchhas struggled with the limitations of existing vision\-language datasets, which typically contain superficial knowledge and noisily aligned pairs\. To overcome this, works such as Unified\-IO\[unified\-io\]and Show\-O\[show\-o\]rely heavily on large\-scale pure text datasets to preserve basic language comprehension capacities\. This requires vision data at a comparable scale to avoid biased training, which imposes strict constraints on data balancing\.

\(2\) Cascade architecture\[emu,dreamllm,lavit\], shown in Fig\.[1](https://arxiv.org/html/2608.20382#S1.F1)\(b\), decouples language and vision generation into a language backbone and a largepretrainedvisual backend\[ldm\]\. The use of the pretrained visual backend enables the pretraining of MLLMs to focus on language modeling, serving as a workaround for modality imbalance\. However, a powerful visual generation backend reduces the language backbone’s visual modeling to a naive role, merely serving as a prompt generator for the backend, resulting in poorly scalable multimodal understanding\.

We propose a new architecture, Libra, named for its balance\-scale\-like design as shown in Fig\.[1](https://arxiv.org/html/2608.20382#S1.F1)\(c\)\. The core idea of Libra is to explicitly decouple both vision and language computing flows into two streams: self\-modal modeling and cross\-modal interaction\. At the macro level, vision and language are separated into two dual branches for self\-modal modeling, and connected by cross\-modal bridges for layer\-wise cross\-modal interaction\. This design enables learning from the bottom up within each modality, as in unified architectures, while mitigating modality imbalance by modularly decoupling self\-modal modeling and cross\-modal interaction, as in cascade architectures\.

More concretely, the decoupling is primarily achieved through a proposed switch attention mechanism and a switch FFN mechanism\. The switch attention induces distinct computation patterns for the query\-key dot product in self\-modal modeling versus cross\-modal interaction\. In the self\-modal case, it reduces to standard attention\. In the cross\-modal case, it maps the attention queries, keys, and values into a new feature space via a cross\-modal bridge\. This strategy avoids directly aligning one modality to the feature space of another, thereby preserving modality\-specific representations while enabling effective cross\-modal interaction\. The switch FFN also decouples multimodal understanding and generation by inducing distinct computation patterns for different scenarios\.

We instantiate the Libra framework in two configurations:Libra\-1, tailored for understanding\-only tasks, andLibra\-2, designed for unified multimodal understanding and generation\. Beyond architectural design, we introduce several new strategies for tokenization, representation, and supervision\. The main contributions of this paper can be summarized as:

- •We introduce Libra, a new MLLM architecture whose core innovation includes a switch attention mechanism and a switch FFN mechanism, which decouple self\-modal modeling from cross\-modal interaction in both vision and language\. This design enables the model to learn multimodal world knowledge from the bottom up while mitigating the modality imbalance that previous MLLMs suffered from\. We instantiate the Libra framework in the following two configurations\.
- •Libra\-1is designed for understanding\-only tasks with two key innovations: \(1\)Supervision: unified discrete autoregression over both vision and language ensures stable training; \(2\)Hybrid tokenization: we integrate continuous signals with discrete IDs as vision inputs, enabling both fine\-grained perception and training stability\.
- •Libra\-2extends the framework to unified multimodal understanding and generation with three key innovations: \(1\)Supervision: we integrate continuous\-space masked visual generation into a unified MLLM; \(2\)Unified RoPEjointly represents 2D images and 1D text while remaining natively compatible with standard RoPE; \(3\)Continuous\-space visual tokenization:Libra\-2processes only continuous visual features for both image\-to\-text understanding and text\-to\-image generation\.
- •The Libra series demonstrate strong performance on over 10 benchmarks including traditional VQA, captioning, and comprehensive MLLM benchmarks for image\-to\-text understanding, as well as FID metrics and general text\-to\-image generation benchmarks for text\-to\-image generation\. Further analysis shows that Libra architecture exhibits lower learning redundancy than commonly used LLaVA\[llava1\.5\]architecture\.

This work extends our ICML 2024 paper\[libra\], which proposedLibra\-1that implements the framework in understanding\-only settings\. In this paper, we further developLibra\-2that substantially advancing the Libra framework toward unified multimodal understanding and generation\.

## IIRelated Work

In recent years, researchers have attempted to extend the LLMs to the field of multimodality\. Existing model frameworks can be roughly categorized into the following two paradigms\.

Unified Architecture Models\. This category of methods performs joint modeling of vision\-language modalities by sharing parameter spaces, enabling end\-to\-end multimodal learning\. The advantage of this type of method lies in granting the model greater flexibility to adaptively balance visual and language modalities within a unified framework\. However, this comes at the cost of complex data mixing strategies and high training expenses\. Most understanding\-only and unified MLLMs are within this framework\. According to the understanding\-only setting, the models are usually trained solely with language supervision, and integrate pretrained vision encoders with pretrained LLMs\. The integrating strategies include simple projections\[llava,cogvlm,qwen\-vl,otter,dreamllm,emu\], cross\-attention\[flamingo\], and Q\-Former prejection\[blip2\]\. According to the unified setting, a representative work is Unified\-IO\[unified\-io,unified\-io2\], which discretizes images into sequences of visual tokens and unifies multimodal generation tasks as sequence prediction\. However, this approach requires carefully designed mixing ratios for over 20 types of multimodal task data, and it still struggles to ensure robust multimodal understanding capabilities\. Subsequent studies, such as Show\-O\[show\-o\]and Janus\[janus\], inject visual generation capabilities into pretrained large language models, achieving significant improvements\. Nevertheless, they rely heavily on large\-scale pure\-text datasets to maintain language modeling performance, which not only significantly increases training costs but also necessitates fine\-grained data balancing strategies\.

Cascaded Architecture Models\. Another category of methods adopts a staged strategy that decouples text generation and visual generation: the backbone model focuses on language modeling, while visual modeling is handled by a separate backend module\. For example, Emu\[emu,emu2\]proposes a unified framework based on autoregressive continuous visual features and discrete language tokens\. In this framework, visual generation is realized by autoregressively generating compressed one\-dimensional visual features, which are then used as conditional input for a backend diffusion model to generate images\. Similarly, DreamLLM\[dreamllm\]adopts a comparable strategy\. LaViT\[lavit\]designs a dynamic visual encoder\-decoder, which first filters and discretizes the input visual features, and then uses a diffusion model decoder to reconstruct the image\. Based on this encoder\-decoder, LaViT achieves joint visual and textual generation via discrete autoregression\. NExT\-GPT\[nextgpt\]further extends to multimodal generation \(image, audio, video, text\), employing different input encoders and designing diffusion model\-based decoders for visual, audio, and video modalities to enhance generation quality\. These methods appropriately decouple vision and language modeling at the architectural level, thus avoiding training imbalance issues\. However, they usually rely on heavy backend generation modules to ensure high\-quality visual outputs\. This leads to a disconnect between visual modeling and language understanding: the backbone language model treats the input visual features merely as conditional signals for the backend visual generation module, and still primarily encodes visual information through language modeling approaches, making it difficult to truly internalize cross\-modal knowledge\.

## IIIPreliminaries

### III\-ABasic Process of a Multimodal Large Language Models

MLLMs extend standard LLMs by adding visual perception to the text\-only pipeline\. In an LLM, raw text is tokenized into subword tokens, mapped to numeric IDs, known as tokenization\. Then the IDs are embedded via a learned lookup table\. A Transformer processes these embeddings in context to produce hidden representations, which a final linear classifier converts into logits over the vocabulary; applying a softmax yields next\-token probabilities\. MLLMs follow the same flow but augment it with an image pathway: an image encoder \(*e\.g\.*, CLIP\[clip\]\) converts pixels into visual embeddings\. The model then take a sequence that interleaves text and visual embeddings as the inputs\.

Many understanding\-only MLLMs\[llava,qwen\-vl\]supervise solely on language tokens, treating image features as conditioning context\. When generative modeling of images is required in a unified framework, a common strategy is to discretize input visual features with a vector\-quantization process\[vqgan\], producing visual token IDs so that both images and text can be modeled with the same next\-token prediction objective\.

### III\-BLookup\-free Quantization

A quantization process is used to discretize input visual features into token IDs so that both images and text can be modeled with the same next\-token prediction objective, commonly performing matching with a set of learnable vectors defined in a vector\-quantization \(VQ\) codebook\. Specifically, a batch of imagesIIare first encoded with a vision encoderΦe​n​c\\Phi\_\{enc\}to produce the output featuresf∈ℝB×H×W×Cf\\in\\mathbb\{R\}^\{B\\times H\\times W\\times C\}, whereBBis the batch size,H×WH\\times Ware the spatial dimensions, andCCis the channel dimension\. LetZ∈ℝB×N×CZ\\in\\mathbb\{R\}^\{B\\times N\\times C\}withN=H​WN=HWdenote the spatially flattened features, and let𝐄=\{ek\}k=1K∈ℝK×C\\mathbf\{E\}=\\\{e\_\{k\}\\\}\_\{k=1\}^\{K\}\\in\\mathbb\{R\}^\{K\\times C\}be a learnable VQ codebook\. Then the output features are used to match with the vectors in the VQ codebook with the largest similarity:

qb,n\\displaystyle q\_\{b,n\}=arg⁡maxq∈𝐄⁡⟨zb,n,q⟩‖zb,n‖2​‖q‖2,\\displaystyle=\\arg\\max\_\{q\\in\\mathbf\{E\}\}\\frac\{\\langle z\_\{b,n\},q\\rangle\}\{\\\|z\_\{b,n\}\\\|\_\{2\}\\,\\\|q\\\|\_\{2\}\},\(1\)wherezb,n∈ℝCz\_\{b,n\}\\in\\mathbb\{R\}^\{C\}is the feature at locationnnof samplebb, andqb,nq\_\{b,n\}is its quantized counterpart\. The matched vectors are used to reconstruct the images by training with a reconstruction loss,*e\.g\.*,ℒrec=‖I−Φd​e​c​\(q\)‖22\\mathcal\{L\}\_\{\\mathrm\{rec\}\}=\\\|I\-\\Phi\_\{dec\}\(q\)\\\|\_\{2\}^\{2\}, whereΦd​e​c\\Phi\_\{dec\}is a decoder to decode the quantized counterpartqqinto the pixel space\.

Compared with previous quantization methods\[vqgan,residualvq,fsq\], Lookup\-free Quantization \(LFQ\)\[lfq\]reduces the embedding dimension of the VQ codebook\[vqvae\]to zero\. Specifically, the codebook𝐄∈ℝK×C\\mathbf\{E\}\\in\\mathbb\{R\}^\{K\\times C\}, similar to the one in VQGAN\[vqgan\], is replaced with an integer set𝔼\\mathbb\{E\}where\|𝔼\|=K\|\\mathbb\{E\}\|=K, whereKKis the vision vocabulary size\. This approach eliminates the need for embedding lookup entirely\. Unlike previous quantization methods\[vqgan,vqvae\]that require codebook embeddings to mimic input features for image reconstruction, LFQ does not require such emulation as it has no codebook embeddings\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x2.png)Figure 2:An overview of the Libra architecture\. The core design is the switch attention and the switch FFN that decoupling self\-modal modeling and cross\-modal interaction\. We build two variants,Libra\-1andLibra\-2, to discuss several improvements in tokenization, positional encoding, and supervision\.
### III\-CRotary Position Embedding

Rotary Position Embedding \(RoPE\)\[rope\]encodes token positions by rotating query and key vectors within each Transformer head\. Letddbe the \(even\) channel dimension of a head\. For the token at positionii, define the block\-diagonal rotation matrixRΘ,i\(d\)∈ℝd×dR\_\{\\Theta,i\}^\{\(d\)\}\\in\\mathbb\{R\}^\{d\\times d\}as

RΘ,id=\[cos⁡\(i​θ0\)−sin⁡\(i​θ0\)sin⁡\(i​θ0\)cos⁡\(i​θ0\)cos⁡\(i​θ1\)−sin⁡\(i​θ1\)sin⁡\(i​θ1\)cos⁡\(i​θ1\)⋱cos⁡\(i​θd/2\)−sin⁡\(i​θd/2\)sin⁡\(i​θd/2\)cos⁡\(i​θd/2\)\]R\_\{\\Theta,i\}^\{d\}=\\begin\{bmatrix\}\\cos\(i\\theta\_\{0\}\)&\-\\sin\(i\\theta\_\{0\}\)&&&&&\\\\ \\sin\(i\\theta\_\{0\}\)&\\cos\(i\\theta\_\{0\}\)&&&&&\\\\ &&\\cos\(i\\theta\_\{1\}\)&\-\\sin\(i\\theta\_\{1\}\)&&&\\\\ &&\\sin\(i\\theta\_\{1\}\)&\\cos\(i\\theta\_\{1\}\)&&&\\\\ &&&&\\ddots&&\\\\ &&&&&\\cos\(i\\theta\_\{d/2\}\)&\-\\sin\(i\\theta\_\{d/2\}\)\\\\ &&&&&\\sin\(i\\theta\_\{d/2\}\)&\\cos\(i\\theta\_\{d/2\}\)\\\\ \\end\{bmatrix\}where the per\-pair angular frequencies are collected in

Θ=\{θm=b−2​m/d:m=0,1,…,d2−1\},b=10000\.\\Theta=\\bigl\\\{\\theta\_\{m\}=b^\{\-2m/d\}\\;:\\;m=0,1,\\ldots,\\tfrac\{d\}\{2\}\-1\\bigr\\\},\\quad b=10000\.Each2×22\\times 2block rotates the coordinates of a feature pair\(2​m,2​m\+1\)\(2m,2m\+1\)by anglei​θmi\\theta\_\{m\}\. These rotations satisfy the relative\-shift property:

RΘ,i\(d\)​\(RΘ,j\(d\)\)⊤=RΘ,i−j\(d\)\.R\_\{\\Theta,i\}^\{\(d\)\}\\bigl\(R\_\{\\Theta,j\}^\{\(d\)\}\\bigr\)^\{\\\!\\top\}=R\_\{\\Theta,i\-j\}^\{\(d\)\}\.\(2\)
RoPE is applied to queries and keys before their dot\-product interaction\. GivenQi,Kj∈ℝ1×dQ\_\{i\},K\_\{j\}\\in\\mathbb\{R\}^\{1\\times d\}for the query/key row vectors at positionsiiandjj,

\(Qi​RΘ,i\(d\)\)​\(Kj​RΘ,j\(d\)\)⊤=Qi​RΘ,i−j\(d\)​Kj⊤,\\bigl\(Q\_\{i\}R\_\{\\Theta,i\}^\{\(d\)\}\\bigr\)\\bigl\(K\_\{j\}R\_\{\\Theta,j\}^\{\(d\)\}\\bigr\)^\{\\\!\\top\}\\;=\\;Q\_\{i\}\\,R\_\{\\Theta,i\-j\}^\{\(d\)\}\\,K\_\{j\}^\{\\top\},\(3\)so the attention score depends only on the relative positioni−ji\-jrather than absolute positions\. In conclusion, RoPE provide a simple and parameter\-free way to endow attention with relative positional inductive bias\.

### III\-DMasked Image Generation in Continuous Space

Masked generative modeling\[maskgit,mage\]learns an image prior by predicting randomly masked tokens from their visible context\. LetX=\{xi\}i=1NX=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}denote the sequence of spatial tokens for an image, where each tokenxi∈ℝDx\_\{i\}\\in\\mathbb\{R\}^\{D\}may be continuous \(e\.g\., from a latent encoder\) or discrete \(e\.g\., a codebook index\)\. A learnable special tokenx∗x\_\{\\ast\}indicates missing content\. For a mask setM⊆\[N\]M\\subseteq\[N\], define the masked input

Xi\(M\)=\{x∗,i∈Mxi,i∉M\.X^\{\(M\)\}\_\{i\}\\;=\\;\\begin\{cases\}x\_\{\\ast\},&i\\in M\\\\ x\_\{i\},&i\\notin M\\;\.\\end\{cases\}Masked modeling maximizes a pseudo\-likelihood objective that predicts each masked token from its unmasked context:

ℒMIM​\(θ\)=𝔼X∼pdata​𝔼M∼π​∑i∈Mlog⁡pθ​\(xi∣X\(M\)\),\\mathcal\{L\}\_\{\\text\{MIM\}\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{X\\sim p\_\{\\text\{data\}\}\}\\;\\mathbb\{E\}\_\{M\\sim\\pi\}\\;\\sum\_\{i\\in M\}\\log p\_\{\\theta\}\\\!\\bigl\(x\_\{i\}\\mid X^\{\(M\)\}\\bigr\),\(4\)whereπ\\piis a masking distribution\. This objective reduces to cross\-entropy for discretexix\_\{i\}and to reconstruction\-style losses for continuousxix\_\{i\}\.

MAR\[mar\]introduces masked image generation in continuous space by attaching a lightweight diffusion head to each position\. Training proceeds as follows:

1. 1\.Encode an image with a VAE\[vae\]to obtain tokensX=\{xi\}i=1NX=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}withxi∈ℝDx\_\{i\}\\in\\mathbb\{R\}^\{D\}\.
2. 2\.Sample a random mask setM⊆\[N\]M\\subseteq\[N\]and constructX\(M\)X^\{\(M\)\}by replacing\{xi:i∈M\}\\\{x\_\{i\}:i\\in M\\\}withx∗x\_\{\\ast\}\.
3. 3\.FeedX\(M\)X^\{\(M\)\}to the Transformer backboneΦ\\Phito produce per\-position featuresF=Φ​\(X\(M\)\)F=\\Phi\\\!\\bigl\(X^\{\(M\)\}\\bigr\)\. Letf\(i\)∈Ff^\{\(i\)\}\\in Fdenote the feature at positionii; for masked positionsi∈Mi\\in M, writef∗\(i\)f\_\{\\ast\}^\{\(i\)\}\.
4. 4\.For eachi∈Mi\\in M, apply the per\-token diffusion headΦdec\\Phi\_\{\\text\{dec\}\}tof∗\(i\)f\_\{\\ast\}^\{\(i\)\}and minimize the diffusion lossℒdiff​\(Φdec​\(f∗\(i\)\),xi\)\\mathcal\{L\}\_\{\\text\{diff\}\}\\\!\\bigl\(\\Phi\_\{\\text\{dec\}\}\(f\_\{\\ast\}^\{\(i\)\}\),\\,x\_\{i\}\\bigr\), which can be viewed as a reconstruction\-style objective\.

At inference time, MAR starts from a fully masked sequence and generates iteratively\. In each iteration, it predicts all tokens, then randomly accepts a fixed proportion of the newly predicted tokens, re\-masks the remainder, and repeats until all positions are accepted\.

## IVDecoupled Vision\-Language System

Libra framework is built on three basic principles:

1. 1\.Modularity: The vision and language systems should be partially decoupled, thereby mitigating biased learning that results from modality entanglement\.
2. 2\.Interactivity: A well\-designed interaction mechanism is essential to connect the two systems and achieve effective cross\-modal interaction\.
3. 3\.Integrity: Each modality should be modeled and learned end\-to\-end from the bottom up to ensure deep and complete understanding\.

Fig\.[2](https://arxiv.org/html/2608.20382#S3.F2)shows an overview of the Libra framework\. This framework comprises a language system and a vision system \(Modularity\), connected via cross\-modal bridges for interaction \(Interactivity\)\. Each system serves as a modality\-specific generator, learning in\-depth world knowledge in a bottom\-up manner \(Integrity\)\.

The methodology part is organized as follows\. Sec\.[IV](https://arxiv.org/html/2608.20382#S4)\(this section\) presents the overall pipeline and the core architectural design of the Libra framework\. Since understanding\-only and unified models require tailored tokenization and supervision strategies, we instantiate distinct configurations to buildLibra\-1for understanding\-only MLLMs in Sec\.[V](https://arxiv.org/html/2608.20382#S5)andLibra\-2for unified MLLMs in Sec\.[VI](https://arxiv.org/html/2608.20382#S6)\.

### IV\-AOverall Pipeline

Libra’s pipeline comprises several components: tokenization, forward pass, positional encoding, and supervision\. Here we present a meta\-framework to illustrate the overall functionality\. Concrete instantiations are provided in Sec\.[V](https://arxiv.org/html/2608.20382#S5)for understanding\-only settings and Sec\.[VI](https://arxiv.org/html/2608.20382#S6)for unified settings\.

Tokenization and Forward Pass\. The tokenization and forward pass of Libra follow a similar process to that of a basic MLLM introduced in Sec\.[III\-A](https://arxiv.org/html/2608.20382#S3.SS1)\. For coherence, we reiterate the process here\. Libra extends standard LLMs by adding visual perception to the text\-only pipeline\. In an LLM, raw text is tokenized into subword tokens, mapped to numeric IDs, known as tokenization\. Then the IDs are embedded via a learned lookup table\. A Transformer\[transformer\]processes these embeddings in context to produce hidden representations, which a final linear classifier converts into logits over the vocabulary; applying a softmax yields next\-token probabilities\. Libra follows the same flow but augment it with an image pathway: an image tokenization process converts pixels into visual embeddings\. The model then take a sequence that interleaves text and visual embeddings as the inputs\.

Positional Embedding\. Libra adopts rotary positional embeddings \(RoPE\)\[rope\], introduced in Sec\.[III\-C](https://arxiv.org/html/2608.20382#S3.SS3)\. To unify positional representations for 2D images and 1D text within a single model, we propose a unified RoPE \(UniRoPE\) method, described in Sec\.[VI](https://arxiv.org/html/2608.20382#S6)\.

Supervision\. Libra conducts supervision on both the image and language sides for both the understanding\-only setting and the unified\-generation setting\. On the language side, Libra uses vanilla next\-token prediction\. On the vision side, we instantiate different modeling strategies: discrete auto\-regression in Sec\.[V](https://arxiv.org/html/2608.20382#S5)for the understanding\-only setting, and continuous\-space masked generation in Sec\.[VI](https://arxiv.org/html/2608.20382#S6)for unified generation\.

### IV\-BCore Architecture

Switch Attention\. The interaction between the two systems in Libra primarily occurs within the attention mechanism, where each token dynamically switches its computational pathway between self\-modal modeling and cross\-modal interaction\. We refer to this as switch attention\. This mechanism helps prevent self\-modal generation from interfering with cross\-modal comprehension during training\.

Given a multimodal input consisting of image and text features with channel dimensionDD, denoted asX=\[XI,XT\]X=\[X\_\{I\},X\_\{T\}\], the switch attention is computed as:

Attn⁡\(X\)\\displaystyle\\operatorname\{Attn\}\(X\)=softmax⁡\(Fb​\(Q,K\)D\)​Fb′​\(V\),\\displaystyle=\\operatorname\{softmax\}\\left\(\\frac\{F\_\{b\}\(Q,K\)\}\{\\sqrt\{D\}\}\\right\)F\_\{b\}^\{\\prime\}\(V\),\(5\)Q\\displaystyle Q=\[QI,QT\],\\displaystyle=\\left\[Q\_\{I\},Q\_\{T\}\\right\],K\\displaystyle K=\[KI,KT\],\\displaystyle=\\left\[K\_\{I\},K\_\{T\}\\right\],V\\displaystyle V=\[VI,VT\],\\displaystyle=\\left\[V\_\{I\},V\_\{T\}\\right\],whereQQ,KK,VVare projections ofXX\. The cross\-modal bridgesFbF\_\{b\}andFb′F\_\{b\}^\{\\prime\}are the core design, as:

Fb​\(Q,K\)\\displaystyle F\_\{b\}\(Q,K\)=\[QI​KI⊤QI′​KT′⁣⊤QT′​KI′⁣⊤QT​KT⊤\],\\displaystyle=\\left\[\\begin\{array\}\[\]\{ll\}Q\_\{I\}K\_\{I\}^\{\\top\}&Q\_\{I\}^\{\\prime\}K\_\{T\}^\{\\prime\\top\}\\\\ Q\_\{T\}^\{\\prime\}K\_\{I\}^\{\\prime\\top\}&Q\_\{T\}K\_\{T\}^\{\\top\}\\end\{array\}\\right\],\(6\)Fb′​\(V\)\\displaystyle F\_\{b\}^\{\\prime\}\(V\)=\[VIVI′VTVT′\],\\displaystyle=\\left\[\\begin\{array\}\[\]\{ll\}V\_\{I\}&V\_\{I\}^\{\\prime\}\\\\ V\_\{T\}&V\_\{T\}^\{\\prime\}\\end\{array\}\\right\],where

Q∗′\\displaystyle Q\_\{\*\}^\{\\prime\}=Q∗\+Q∗​W∗Q′,\\displaystyle=Q\_\{\*\}\+Q\_\{\*\}W\_\{\*\}^\{Q^\{\\prime\}\},\(7\)K∗′\\displaystyle K\_\{\*\}^\{\\prime\}=K∗​W∗K′,\\displaystyle=K\_\{\*\}W\_\{\*\}^\{K^\{\\prime\}\},V∗′\\displaystyle V\_\{\*\}^\{\\prime\}=V∗​W∗V′,\\displaystyle=V\_\{\*\}W\_\{\*\}^\{V^\{\\prime\}\},∗\\displaystyle\*∈\{I,T\}\.\\displaystyle\\in\\\{I,T\\\}\.Here, the projection matricesW∗Q′W\_\{\*\}^\{Q^\{\\prime\}\},W∗K′W\_\{\*\}^\{K^\{\\prime\}\},W∗V′W\_\{\*\}^\{V^\{\\prime\}\}are the cross\-modal bridges\.W∗K′W\_\{\*\}^\{K^\{\\prime\}\}andW∗V′W\_\{\*\}^\{V^\{\\prime\}\}are standard matrix in a linear layer, whileW∗Q′W\_\{\*\}^\{Q^\{\\prime\}\}is a low\-rank matrix of rank88, serving as a low\-rank adaptation\[lora\]applied toQQduring cross\-modal interactions\. This enables lightweight finetuning of the original modality\-specific query patterns, rather than learning entirely new ones, preserving modality\-specific feature structures while enhancing cross\-modal alignment\.

An intuitive explanation of the switch attention mechanism is as follows: For self\-modal modeling \(*e\.g\.*,QI​KI⊤Q\_\{I\}K\_\{I\}^\{\\top\}\), Libra retains the standard self\-attention operation\. In contrast, for cross\-modal interaction \(*e\.g\.*,QI′​KT′⁣⊤Q\_\{I\}^\{\\prime\}K\_\{T\}^\{\\prime\\top\}\), it redirects the computation pathways ofQQ,KK, andVV, projecting them into a new latent space via cross\-modal bridges before performing attention alignment\. This strategy avoids directly aligning one modality with the feature space of another, thereby preserving the integrity of modality\-specific representations while facilitating effective cross\-modal interaction\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x3.png)Figure 3:Assessing the cross‑modal bridge via image inpainting and text‑to‑image generation\. We finetuneLibra\-1and its variant without cross\-modal bridges with additional 10M image\-text pairs from the pretraining data by disabling the contiguous visual signal in Sec\.[V\-C](https://arxiv.org/html/2608.20382#S5.SS3)\. We also present the results ofLibra\-2for reference\.Cross\-modal Bridge\. The cross\-modal bridge in Eqn\. \([6](https://arxiv.org/html/2608.20382#S4.E6)\) is the core component of the switch attention mechanism\. Here we further elaborate on its role in mitigating modality imbalance\. Rather than forcing one modality to align directly to the feature space of another, the bridge serves as an intermediate adapter that preserves modality\-specific characteristics while enabling controlled information exchange\. As shown in Fig\.[3](https://arxiv.org/html/2608.20382#S4.F3), removing the bridge yields an entangled and underperforming vision system that \(1\) learns repetitive patterns in image completion and \(2\) poorly follows language instructions in text\-to\-image generation\. In contrast, the bridge implements a principled cross\-modal interaction strategy that substantially strengthens visual learning within the MLLM\.

Switch FFN\. Multimodal understanding and generation are distinct processes\. While understanding concerns the comprehension of overall meaning, generation emphasizes the production of detailed content\. These two processes should be treated in different manners\. Inspired by switch attention, we further propose switch FFN that dynamically routes computation in the visual branch to decouple multimodal understanding from generation\. The realization of switch FFN is simple\. Given image featuresXI=\{XIund,XIgen\}X\_\{I\}=\\\{X\_\{I\}^\{\\text\{und\}\},X\_\{I\}^\{\\text\{gen\}\}\\\}, whereXIundX\_\{I\}^\{\\text\{und\}\}andXIgenX\_\{I\}^\{\\text\{gen\}\}are the feature subsets dedicated to understanding and generation, switch FFN applies separate FFNs to these subsets and concatenates the results:

S​w​i​c​h​F​F​N​\(XI\)=c​o​n​c​a​t​\(F​F​Nu​\(XIu​n​d\),F​F​Ng​\(XIg​e​n\)\)\.SwichFFN\(X\_\{I\}\)=concat\(FFN\_\{u\}\(X\_\{I\}^\{und\}\),FFN\_\{g\}\(X\_\{I\}^\{gen\}\)\)\.\(8\)In words, if the image features are associated with understanding, it is routed to an understanding expertF​F​NuFFN\_\{u\}, and vice versa\. Together with switch attention, the simple yet effective switch FFN further strengthens the decoupling, leading to improved performance in both understanding and generation\. We use the switch FFN only in the vision system\. This prevents the pretrained LLM’s weights in the language system from being significantly disturbed, as no additional parameters are added to its forward pass\. The final FFN for image and text featuresX=\{XI,XT\}X=\\\{X\_\{I\},X\_\{T\}\\\}is computed as:

F​F​N​\(X\)=c​o​n​c​a​t​\(S​w​i​c​h​F​F​N​\(XI\),F​F​N​\(XT\)\)\.FFN\(X\)=concat\(SwichFFN\(X\_\{I\}\),FFN\(X\_\{T\}\)\)\.\(9\)

## VLibra\-1: Towards Complete Multimodal Understanding

### V\-AModel Overview

Multimodal understanding, typically in image\-to\-text settings, is the fundamental ability of MLLMs\. Here we builtLibra\-1to validate the effectiveness of the Libra framework under understanding\-only image\-to\-text settings\.

Libra\-1follows the core architecture described in Sec\.[IV\-B](https://arxiv.org/html/2608.20382#S4.SS2)\. Although the model targets understanding only, we supervise both the image and text sides to meet Libra’s integrity principle that each modality should be modeled and learned end\-to\-end from the bottom up to ensure deep and complete understanding\. SinceLibra\-1does not generate images, we replace the switch FFN in Eqn\. \([9](https://arxiv.org/html/2608.20382#S4.E9)\) with a standard FFN, as:

FFN⁡\(X\)=concat⁡\(FFNI⁡\(XI\),FFNT⁡\(XT\)\)\.\\operatorname\{FFN\}\(X\)=\\operatorname\{concat\}\\\!\\left\(\\operatorname\{FFN\}\_\{I\}\\\!\\left\(X\_\{I\}\\right\),\\,\\operatorname\{FFN\}\_\{T\}\\\!\\left\(X\_\{T\}\\right\)\\right\)\.\(10\)
The following subsections detail the tokenization and supervision ofLibra\-1\.

### V\-BHybrid Tokenization

Libra\-1unifies both vision and language modeling into adiscretenext\-token\-prediction paradigm\. Given an input sequence with both image and corresponding language parts, we separately tokenize the image and the language parts into discrete tokens through a VQGAN\[vqgan\]and a SentencePiece\[sentencepiece\]tokenizer\. We respectively prefix and suffix the image sequence with one⟨BOI⟩\\langle\\textrm\{BOI\}\\rangle\(beginning of image\) token and one⟨EOI⟩\\langle\\textrm\{EOI\}\\rangle\(end of image\) token\. We use a newline token “`\\n`” to separate images and texts\. All token embeddings except the separation newline token “`\\n`” are updated through a cross\-entropy classification loss\.

This unified discrete next\-token\-prediction paradigm enables stable training convergence but raising two obstacles for effective vision\-language comprehension: 1\) The discretization process of VQGAN can cause severe visual information loss, leading to low perception on visual details\. 2\) Naive discrete sequential modeling hardly benefits from the pretrained knowledge of the vision encoder, since the model receives newly\-constructed embeddings based on the input IDs instead of the features of the vision encoder\. To this end, we propose a hybrid image tokenization process from two aspects: contiguous visual signals and pretrained visual knowledge\.

Contiguous Visual Signal vs\. Discrete Modeling\.We leverage a hybrid tokenization strategy with a combination of contiguous visual signals from the vision encoder and discrete modeling using tokenized IDs\. Formally:

fc\\displaystyle f\_\{c\}=Φ​\(I\),\\displaystyle=\\Phi\(I\),\(11\)ID=quantize⁡\(fc\),\\displaystyle=\\operatorname\{quantize\}\(f\_\{c\}\),fd\\displaystyle f\_\{d\}=Ed​\(ID\),\\displaystyle=E\_\{d\}\(\\text\{ID\}\),XI\\displaystyle X\_\{I\}=concat⁡\(fc,fd\)\.\\displaystyle=\\operatorname\{concat\}\(f\_\{c\},f\_\{d\}\)\.Eqn\. \([11](https://arxiv.org/html/2608.20382#S5.E11)\) proceeds as follows\. Given an input imageII, we first pass it through the vision encoderΦ\\Phito obtain continuous visual signalsfcf\_\{c\}\. We then discretize these features using a codebook\-based quantization step, producing token IDs\. The token IDs are mapped to learnable “word” embeddings via the embedding tableEdE\_\{d\}in the main MLLM, yielding discrete signalsfdf\_\{d\}\. Finally, we concatenatefcf\_\{c\}andfdf\_\{d\}along the channel dimension to form the final visual inputXIX\_\{I\}to theLibra\-1model\. Meanwhile, a discrete auto\-regressive image modeling is performed on the output features ofLibra\-1,*i\.e\.*, each vision input is used to predict the token ID of the next position\. This simple design enables both continuous\-space visual comprehension and stable discrete sequential modeling\. However, since it requires the continuous signal of the entire image as input, which is not available before the image is generated,Libra\-1is unable to perform image generation\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x4.png)Figure 4:Image reconstruction results of the tokenizers\. Directly replacing the image encoder of VQGAN with CLIP distorts the visual information\.Libra\-1largely alleviates this problem via lookup\-free quantization\.Pretrained Visual Knowledge\.To leverage the pretrained knowledge in existing well\-established vision encoders like CLIP\[clip\], we replace the vision encoder of VQGAN with afrozenCLIP\-ViT\-L\. However, training a CLIP\-based VQGAN is non\-trivial\. The features in CLIP are highly semantic with less low\-level visual information\. Directly emulating such features in the quantization process of the original VQGAN results in poor reconstruction performance, as demonstrated in Fig\.[3](https://arxiv.org/html/2608.20382#S4.F3)\(CLIP\-VQGAN\)\. Instead, we find that the lookup\-free quantization \(LFQ\)\[lfq\], described in Sec\.[III\-B](https://arxiv.org/html/2608.20382#S3.SS2), does not need to emulate the input features, largely addressing this problem\. To this end, we use a CLIP\-based VQGAN with LFQ asLibra\-1’s image tokenizer\. This is the first time that a highly reconstructive image tokenizer can be constructed based on a frozen vision encoder like CLIP, which has not even been investigated in the work of LFQ\.

### V\-CUnified Discrete Supervision

Libra\-1is pretrained under unified sequential modeling, where a next\-token\-prediction objective is performed on all input tokens\., as:

p​\(X\)=∏ℓ=1Lp​\(Xℓ∣X<ℓ\),p\(X\)=\\prod\_\{\\ell=1\}^\{L\}p\\left\(X\_\{\\ell\}\\mid X\_\{<\\ell\}\\right\),\(12\)whereX=\[XI,XT\]X=\[X\_\{I\},X\_\{T\}\]is the input multimodal sequence andLLis the sequence length\. In practice, the objective is computed through a discrete cross\-entropy classification loss\. However, since we introduce additional continuous visual signals to form hybrid inputs in Sec\.[V\-B](https://arxiv.org/html/2608.20382#S5.SS2),Libra\-1is unable to generate images, as the full continuous signal is not available before the image is generated\. Therefore, the discrete autoregressive supervision on vision outputs functions solely as a regularization objective, enabling the model to learn visual world knowledge from the bottom up\.

The training commonly consists of pretraining states and fine\-tuning stages\. The pretraining stages follows the training objective in Eqn\. \([12](https://arxiv.org/html/2608.20382#S5.E12)\)\. The fine\-tuning stages helps MLLMs to align with user intentions\[rlhf,self\-instruct\]and generalize to unseen tasks\[wei2021finetuned,chung2022scaling\]\. All instruction\-tuning data are arranged based on this template:

<System Message\>\[USER\]: <Image\> <Instruction\>\[ASSISTANT\]: <Answer\>\\begin\{split\}&\{\\color\[rgb\]\{0,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\texttt\{\\text\{<System Message\>\}\}\}\\\\ &\{\\color\[rgb\]\{0,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\texttt\{\\text\{\[USER\]: <Image\> <Instruction\>\}\}\}\\\\ &\{\\color\[rgb\]\{0,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\texttt\{\\text\{\[ASSISTANT\]: <Answer\>\}\}\}\\\\ \\end\{split\}\(13\)where only<Answer\>is accounted for computing loss, as:

p​\(Xa∣Xv,Xi​n​s​t​r​u​c​t\)=∏ℓ=1Lp​\(xℓ∣Xv,Xi​n​s​t​r​u​c​t,x<ℓ\),p\(X\_\{a\}\\mid X\_\{v\},X\_\{instruct\}\)=\\prod\_\{\\ell=1\}^\{L\}p\\left\(x\_\{\\ell\}\\mid X\_\{v\},X\_\{instruct\},x\_\{<\\ell\}\\right\),\(14\)whereXa=\{xℓ\}ℓ=1LX\_\{a\}=\\\{x\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\},XvX\_\{v\},Xi​n​s​t​r​u​c​tX\_\{instruct\}are the answers, images, and instructions\.

## VILibra\-2: Towards Unified Multimodal Understanding and Generation

### VI\-AModel Overview

We further extend Libra framework to unify multimodal understanding and generation and deriveLibra\-2\. The improvements are made on the following aspects:

1. 1\.Architecture\.Libra\-2also follows the core architecture design described in Sec\.[IV\-B](https://arxiv.org/html/2608.20382#S4.SS2)\. We use switch FFN to decouple multimodal understanding and generation\.
2. 2\.Representation\. Vision and language differ fundamentally in structure: images are inherently two\-dimensional, whereas text is one\-dimensional\. To unify multimodal understanding and generation, we need a unified position encoding method for strong generation on both modalities in one model\. Therefore we propose unified rotary position encoding \(UniRoPE\) in Sec\.[VI\-B](https://arxiv.org/html/2608.20382#S6.SS2)\.
3. 3\.Supervision\. Prior works typically employs discrete autoregression for both vision and language\. To mitigate the information loss introduced by discretizing vision,Libra\-1takes both continuous visual features and discrete IDs as hybrid inputs\. This, however, disables image generation, renderingLibra\-1an understanding\-only model\. In contrast,Libra\-2implements continuous\-space visual tokenization in Sec\.[VI\-C](https://arxiv.org/html/2608.20382#S6.SS3)and continuous\-space masked image generation in Sec\.[VI\-D](https://arxiv.org/html/2608.20382#S6.SS4)to enable lossless multimodal understanding and generation\.

### VI\-BUnified Rotary Position Embedding

Images are inherently two\-dimensional, whereas text is one\-dimensional\. This fundamental difference necessitates an effective position encoding strategy for unified multimodal modeling\. Rotary Position Embedding \(RoPE\)\[rope\]is one of the most commonly used techniques in current MLLMs, as introduced in Sec\.[III\-C](https://arxiv.org/html/2608.20382#S3.SS3)\. However, its existing variants in multimodal settings exhibit notable limitations:

1. 1\.Linear flattening \(*e\.g\.*, LLaVA\[llava\]\): Visual features are flattened into a 1D sequence to align with the input format of language models\. While this enables seamless integration, it sacrifices the intrinsic spatial structure of visual data, potentially hindering visual understanding\.
2. 2\.Grouped encoding \(*e\.g\.*, Unified\-IO\[unified\-io\]\): Attention heads are divided into two groups to encode horizontal and vertical positions separately\. Although this strategy preserves spatial structure, it manually constrains the model’s expressiveness: each attention head encodes only one spatial direction\.

Therefore, we propose a Unified RoPE \(UniRoPE\) strategy for better multimodal integration\. UniRoPE splits the frequencies of RoPE into two groups for horizontal and vertical positions\. Namely, for a token with position\(i,j\)\(i,j\), its RoPE matrixRΘ,i,jdR\_\{\\Theta,i,j\}^\{d\}is formulated as:

\[cos⁡\(i​θ0\)−sin⁡\(i​θ0\)sin⁡\(i​θ0\)cos⁡\(i​θ0\)cos⁡\(j​θ1\)−sin⁡\(j​θ1\)sin⁡\(j​θ1\)cos⁡\(j​θ1\)⋱cos⁡\(i​θd/2−1\)−sin⁡\(i​θd/2−1\)sin⁡\(i​θd/2−1\)cos⁡\(i​θd/2−1\)cos⁡\(j​θd/2\)−sin⁡\(j​θd/2\)sin⁡\(j​θd/2\)cos⁡\(j​θd/2\)\]\\begin\{bmatrix\}\\cos\(i\\theta\_\{0\}\)&\-\\sin\(i\\theta\_\{0\}\)&&&&&&&\\\\ \\sin\(i\\theta\_\{0\}\)&\\cos\(i\\theta\_\{0\}\)&&&&&&&\\\\ &&\\cos\(j\\theta\_\{1\}\)&\-\\sin\(j\\theta\_\{1\}\)&&&&&\\\\ &&\\sin\(j\\theta\_\{1\}\)&\\cos\(j\\theta\_\{1\}\)&&&&&\\\\ &&&&\\ddots&&&&\\\\ &&&&&\\cos\(i\\theta\_\{d/2\-1\}\)&\-\\sin\(i\\theta\_\{d/2\-1\}\)&&\\\\ &&&&&\\sin\(i\\theta\_\{d/2\-1\}\)&\\cos\(i\\theta\_\{d/2\-1\}\)&&\\\\ &&&&&&&\\cos\(j\\theta\_\{d/2\}\)&\-\\sin\(j\\theta\_\{d/2\}\)\\\\ &&&&&&&\\sin\(j\\theta\_\{d/2\}\)&\\cos\(j\\theta\_\{d/2\}\)\\\\ \\end\{bmatrix\}whereΘ=\{θi=10000−2​\(i−1\)/d,i∈\[1,2,…,d/2\]\}\\Theta=\\left\\\{\\theta\_\{i\}=10000^\{\-2\(i\-1\)/d\},i\\in\[1,2,\\ldots,d/2\]\\right\\\}is the set of predefined frequencies, andddis the channel dimension\. It is easy to prove that UniRoPE satisfies the relative\-shift property, as:

RΘ,i1,j1⋅RΘ,i2,j2⊤=RΘ,i1−i2,j1−j2\.R\_\{\\Theta,i\_\{1\},j\_\{1\}\}\\cdot R\_\{\\Theta,i\_\{2\},j\_\{2\}\}^\{\\top\}=R\_\{\\Theta,i\_\{1\}\-i\_\{2\},j\_\{1\}\-j\_\{2\}\}\.\(15\)In addition, in the case of text input,*i\.e\.*, wheni=ji=j, UniRoPE degenerates to the standard RoPE, ensuring compatibility with existing LLMs\. In fact, this is equivalent to treating each text token as an1×11\\times 1image\.

UniRoPE enables seamless integration with existing LLM architectures while offering greater representational capacity than previous encoding strategies\. For more detailed background on RoPE, please refer to Sec\.[III\-C](https://arxiv.org/html/2608.20382#S3.SS3)and the original paper\[rope\]\.

### VI\-CContinuous\-Space Visual Tokenization

Libra\-2operates directly on continuous\-space visual features for both multimodal understanding and generation\. Given an input imageII, we first encode it with a VAE\[vae\]\. We then semantically enrich these VAE features using a CLIP\-style encoder via cross\-attention:

XIvae\\displaystyle X\_\{I\}^\{\\text\{vae\}\}=ΦVAE​\(I\),\\displaystyle=\\Phi\_\{\\text\{VAE\}\}\(I\),\(16\)XIclip\\displaystyle X\_\{I\}^\{\\text\{clip\}\}=ΦCLIP​\(I\),\\displaystyle=\\Phi\_\{\\text\{CLIP\}\}\(I\),XI\\displaystyle X\_\{I\}=CrossAttn⁡\(XIvae,XIclip\),\\displaystyle=\\operatorname\{CrossAttn\}\(X\_\{I\}^\{\\text\{vae\}\},\\,X\_\{I\}^\{\\text\{clip\}\}\),whereΦVAE\\Phi\_\{\\text\{VAE\}\}is a VAE with downsampling factor 16 and latent channel dimension 16, andΦCLIP\\Phi\_\{\\text\{CLIP\}\}is a CLIP\-style\[clip\]vision encoder \(we use SigLIP\-Large/384\[siglip\]\)\. The cross\-attention module takesXIvaeX\_\{I\}^\{\\text\{vae\}\}as queries andXIclipX\_\{I\}^\{\\text\{clip\}\}as keys and values, using the unified RoPE introduced in Sec\.[VI\-B](https://arxiv.org/html/2608.20382#S6.SS2)\.

Training interleaves multimodal understanding and image\-conditioned generation examples\. For generation samples, we only takeXIvaeX\_\{I\}^\{\\text\{vae\}\}as input features\. For understanding samples, we use the fused featuresXIX\_\{I\}\. During inference, the generated visual latent features are decoded to the pixel space by a VAE decoder\.

### VI\-DAdaptive Supervision

Libra\-2uses different modeling strategies for the language and visual modalities\. The language branch is supervised by auto\-regression in a discrete space, while the visual branch is supervised by masked prediction in a continuous space\[mar\]introduced in Sec\.[III\-D](https://arxiv.org/html/2608.20382#S3.SS4)\. We claim that continuous\-space visual modeling is crucial for MLLMs: it enables lossless visual perception and supports learning visual world knowledge from the bottom up in a generative manner\. However, a key open challenge is how to equip MLLMs with strong continuous\-space image generation\. Although some cascaded architectures\[emu\]train auto\-regressively on continuous\-space visual features, the resulting models only generate an indirect prompt for a backend generator and lack explicit spatial structure\. To this end, we seek to achieve continuous, strong, and direct visual generation within a unified MLLM, enabling bottom\-up learning of visual knowledge\.

We extend the continuous\-space masked image generation described in Sec\.[III\-D](https://arxiv.org/html/2608.20382#S3.SS4)to the field of MLLMs\. It is worth noting that this technique\[mar\]was originally designed for class\-to\-image generation, and was later extended to text\-to\-image generation\[fluid\]\. However, the mechanism of continuous\-space masked image generation in unified MLLMs remains unexplored\.Libra\-2is among the first to take this step\.

Given a multimodal input comprising visual tokens𝐯=\{v1,v2,…,vN\}\\mathbf\{v\}=\\\{v\_\{1\},v\_\{2\},\\ldots,v\_\{N\}\\\}and text tokens𝐭=\{t1,t2,…,tM\}\\mathbf\{t\}=\\\{t\_\{1\},t\_\{2\},\\ldots,t\_\{M\}\\\}, the training objective of language modeling is to maximize its conditional probability:

P​\(𝐭∣𝐯\)=∏i=1Mpθ​\(ti∣𝐯<p​o​s​\(ti\),t<i\),P\(\\mathbf\{t\}\\mid\\mathbf\{v\}\)=\\prod\_\{i=1\}^\{M\}p\_\{\\theta\}\\left\(t\_\{i\}\\mid\\mathbf\{v\}\_\{<pos\(t\_\{i\}\)\},\\ t\_\{<i\}\\right\),\(17\)wherep​o​s​\(x\)pos\(x\)denotes the position of tokenxxin the overall multimodal sequence\. Eqn\. \([17](https://arxiv.org/html/2608.20382#S6.E17)\) reflects a causal attention mechanism, whereby the model predicts each text tokentit\_\{i\}based only on the preceding visual information𝐯<p​o​s​\(ti\)\\mathbf\{v\}\_\{<pos\(t\_\{i\}\)\}and prior text tokenst<it\_\{<i\}\.

According to the visual modeling, we adopt a masked prediction strategy in the continuous space\. Given multimodal tokens𝐯\\mathbf\{v\}and𝐭\\mathbf\{t\}, the modeling process proceeds as: 1\) Visual tokens are randomly replaced with a learnable \[MASK\] token, denoted asv∗v\_\{\*\}, based on a predefined probability\. 2\) The model predicts the masked visual tokens based on the multimodal context\. Formally,

P​\(𝐯∣𝐭\)=∏j=1Npθ​\(vj∣v∗,v2,⋯,v∗,⋯,vN,𝐭<loc​\(vj\)\)\.P\(\\mathbf\{v\}\\mid\\mathbf\{t\}\)=\\prod\_\{j=1\}^\{N\}p\_\{\\theta\}\\left\(v\_\{j\}\\mid v\_\{\*\},v\_\{2\},\\cdots,v\_\{\*\},\\cdots,v\_\{N\},\\mathbf\{t\}\_\{<\\text\{loc\}\(v\_\{j\}\)\}\\right\)\.\(18\)Eqn\. \([18](https://arxiv.org/html/2608.20382#S6.E18)\) indicates that the prediction for each masked token depends on all visual tokens in the image, and the preceding textual tokens\. By comparing Eqns\. \([17](https://arxiv.org/html/2608.20382#S6.E17)\) and \([18](https://arxiv.org/html/2608.20382#S6.E18)\), it can be observed that the textual tokens still remain a causal unidirectional attention mechanism, while the visual tokens utilize a bidirectional attention mechanism inLibra2\. This design preserves the 2D spatial structure of the visual input within the model\.

To this end, the final loss function is defined as:

ℒ=−α​log⁡P​\(𝐭∣𝐯\)−log⁡P​\(𝐯∣𝐭\),\\mathcal\{L\}=\-\\alpha\\log P\(\\mathbf\{t\}\\mid\\mathbf\{v\}\)\-\\log P\(\\mathbf\{v\}\\mid\\mathbf\{t\}\),\(19\)
During practical training, the modeling objective of language in Eqn\. \([17](https://arxiv.org/html/2608.20382#S6.E17)\) is realized by a cross\-entropy \(CE\) loss that predict the discrete IDs of the next tokens, while the modeling objective of vision in Eqn\. \([18](https://arxiv.org/html/2608.20382#S6.E18)\) is implemented by a mean squared error \(MSE\) loss that predict the continuous masked visual tokens\. We observe that the \(MSE\) loss on continuous visual tokens is significantly smaller than the CE loss on discrete text tokens\. Therefore, we reweight the MSE loss by a factor ofα=0\.25\\alpha=0\.25\.

## VIIExperiments

### VII\-AImplementation Details

The cross\-modal bridges in Libra architecture allow the vision and language branches to use different hidden dimensions and depths\. We adopt a simple implementation: cross\-modal interactions occur at every layer, and the vision branch has the same hidden size as the language branch\. ForLibra\-1, we initialize the language branch with LLaMA2\-7B\-Chat\[llama2\]; forLibra\-2, we use LLaMA3\.2\-1B\-Instruct\[llama3\]\. The vision branch and all cross\-modal bridges are initialized from scratch\. This configuration yields an 11BLibra\-1model and a 3BLibra\-2model\. We adopt the SDXL refiner\[sdxl\]as a post\-processing step to reduce noise inLibra\-2’s generated images\. This design does not interfere with the model’s bottom\-up learning of visual knowledge: the core model is trained to reconstruct the low\-level VAE latent representations, while the refiner participates neither in training nor in tokenization\. serving only as post processing\.

We trainLibra\-1’s image tokenizer using 10M images collected by\[sam\]\. The vision vocabulary size is enlarged to2182^\{18\}thanks to LFQ\. For computational efficiency, we predict in two concatenated codebooks, each of size292^\{9\}\. For theLibra\-2image tokenizer, we employ the officially pretrained VAE\[vae\]and SigLiP\[siglip\]encoders\. No further training is required\.

### VII\-BTraining Pipeline and Data

Training MLLMs typically comprises a multi\-stage pretraining phase followed by instruction fine\-tuning\. We train two variants:Libra\-1, aimed at multimodal understanding, andLibra\-2, which unifies multimodal understanding and generation\. Their training pipelines are summarized below\.

Libra\-1is trained in two stages\. The training hyperparameters ofLibra\-1are shown in Tab\.[I](https://arxiv.org/html/2608.20382#S7.T1)\. Below, we summarize te key pipeline:

1. 1\.Pretraining\. We sample 50M image\-text pairs from COYO\-700M\[coyo700m\]and CC12M\[cc12m\]\. To normalize caption style, we additionally include 500K image\-text pairs from the COCO training split\[cococap\]\. The data are arranged as “<image\><text\>” for multimodal understanding\. During pretraining, the language branch is frozen; only the vision branch and the cross\-modal bridges are updated\.
2. 2\.High\-quality instruction fine\-tuning\. We fine\-tune the full model on 665K high\-quality supervised samples from LLaVA\-Instruct\[llava\]\.

Libra\-2is trained in five stages\. In Stages 1\-4, the language branch is frozen, and only the vision branch and cross\-modal bridges are updated\. In Stage 5, we fine\-tune the entire model\. For all pretraining image\-text pairs, we recaption them by a pretrained BLIP\-2 model\[blip2\]\. The training hyperparameters ofLibra\-2are shown in Tab\.[II](https://arxiv.org/html/2608.20382#S7.T2)\. Below, we summarize the key pipeline\.

1. 1\.Alignment\. We initialize the vision branch on ImageNet\-1K\[imagenet\]to learn basic pixel dependencies, arranging the data in a class\-to\-image generation format, namely, “<name\><image\>”, where<name\>represents the text names of each class\.
2. 2\.Pretraining \(PT\)\. This is the main training stage\. We train on 200M image\-text pairs from LAION\-COCO\[laion\-coco\], interleaving two sequence formats: “<image\><text\>” for multimodal understanding and “<text\><image\>” for multimodal generation\. As understanding is easier to learn than generation, we upweight the latter by sampling the two formats with probabilities of 20% \(understanding\) and 80% \(generation\)\.
3. 3\.*Optional: Resolution scaling\.*Higher input resolutions generally improve multimodal performance\. We therefore optionally increase the training resolution on LAION–COCO from256×256256\\times 256to384×384384\\times 384pixels\. If the resolution is scaled at this stage, the same scaled resolution should be used in all subsequent steps\.
4. 4\.Aesthetic modulation \(CT\)\. As images from LAION\-COCO contain massive low\-quality visual renderings, we further modulateLibra\-2’s image generation toward higher aesthetic quality\. We use 14M aesthetic\-oriented samoles from JourneyDB\[journeydb\]and LAION\-Aesthetic\[laion\-aesthetic\], using the same data arrangement as in Stage 2\.
5. 5\.High\-quality instruction fine\-tuning \(SFT\)\. We fine\-tune the full model on the 1\.2M high\-quality supervised samples from InternVL\[internvl\]for multimodal understanding and on 1M samples drawn from JourneyDB and LAION\-Aesthetic to maintain the multimodal generation ability\.

TABLE I:Training hyperparameters ofLibra\-1\.TABLE II:Training hyperparameters ofLibra\-2\.TABLE III:Evaluation of multimodal understanding\. Models are categorized based on their architecture types, vision input formats, and input resolution\. The reported metrics include accuracy \(%\) for Visual Question Answering \(VQA\), CIDEr scores for image captioning, and scores for general MLLM benchmarks\.VQACaptioningGeneral BenchmarkModel\#Params\.Arch\.VisionRes\.VQAv2GQAFlickrCOCOMMEPOPEMMBSEEDSpecialistBEiT\-3\[beit3\]1\.9B\-\-22484\.0\-\-147\.6\-\-\-\-PaLI\-X\[palix\]55B\-\-25686\.1\-\-149\.2\-\-\-\-OFA\[ofa\]1B\-\-48082\.0\-\-154\.9\-\-\-\-Understanding GeneralistBLIP\-2\[blip2\]12BQ\-FormerContinuous22465\.044\.774\.9144\.51293\.885\.3\-46\.4Flamingo\[flamingo\]80BCrossAttnContinuous32056\.3\-67\.284\.3\-\-\-\-Qwen\-VL\[qwen\-vl\]9\.6BVanillaContinuous22478\.257\.581\.0\-1487\.5\-60\.658\.2LLaVA1\.5\[llava1\.5\]13BVanillaContinuous38480\.063\.3\-129\.81510\.785\.964\.358\.6Libra\-111BLibraHybrid38477\.363\.886\.6135\.21494\.788\.265\.262\.7Understanding & Generation GeneralistEmu\[emu\]14BCascadeContinuous22452\.9\-\-117\.7\-\-\-\-DreamLLM\[dreamllm\]7BCascadeContinuous33656\.6\-\-115\.4\-49\.9\-LaVIT\[lavit\]7BCascadeDiscrete22466\.046\.883\.0\-\-\-\-\-NExT\-GPT\[nextgpt\]13BCascadeDiscrete\-66\.7\-84\.5124\.9\-\-\-\-Gemini\-Nano\-1\[gemini\]1\.8BUnified\-\-62\.7\-\-\-\-\-\-\-Chameleon\[chameleon\]34BUnifiedDiscrete\-66\.0\-74\.7\-\-\-\-\-Janus\[janus\]1\.3BUnifiedDiscrete38477\.359\.1\-\-1338\.087\.069\.463\.7LWM\[lwm\]7BUnifiedDiscrete25655\.844\.8\-\-\-\-\-\-Show\-O\[show\-o\]1\.3BUnifiedDiscrete25659\.348\.736\.2\-948\.473\.8\-\-3BLibraContinuous25677\.558\.878\.5115\.01475\.088\.560\.066\.2Libra\-23BLibraContinuous38478\.459\.377\.0107\.81648\.389\.860\.771\.8TABLE IV:Evaluation of text\-to\-image generation on GenEval benchmark\. We present the results of both generation\-only methods and methods for unified understanding and generation\.
### VII\-CMultimodal Understanding

Traditional understanding benchmarks\. We evaluate the multimodal understanding capabilities of the Libra series across a broad range of benchmarks, including traditional VQA and image captioning\. Tab\.[III](https://arxiv.org/html/2608.20382#S7.T3)summarizes the results\. We first compare the understanding\-only model,Libra\-1, with prior work and find that it surpasses contemporary MLLMs on multimodal understanding\. We also observed thatLibra\-2underperforms on image captioning\. This probably stems from a distribution mismatch: our training data predominantly contain long captions, whereas COCO emphasizes concise, single\-sentence captions\. This mismatch discourages the model from compressing key content into short summaries\.

General\-purpose MLLM benchmarks\.We then conduct a systematic comparison in the field of unified multimodal understanding and generation\. The main observations are:

1. 1\.The Libra architecture consistently outperforms unified and cascaded architectures\. Compared with Show\-O, whose data scale and composition most closely match ours,Libra\-2achieves substantially higher scores across all benchmarks\.
2. 2\.Higher input resolution is important for accurate multimodal understanding\. IncreasingLibra\-2’s resolution from2562256^\{2\}to3842384^\{2\}yields significant performance gains\.
3. 3\.Generation helps understanding\. Despite having far fewer parameters thanLibra\-1,Libra\-2attains comparable results on several benchmarks \(e\.g\., VQAv2, MME, POPE, SEED\), suggesting that multimodal generation can benefit multimodal understanding\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x5.png)Figure 5:Comparison on text\-to\-image generation\. We highlight the key parts of the text prompts, whereLibra\-2demonstrates strong prompt\-following capacity\.
### VII\-DText\-to\-Image Generation

TABLE V:Evaluation of text\-to\-image generation via FID\. We present the results of both generation\-only methods and methods for unified understanding and generation\.†We build an image\-generation\-capable variant ofLibra\-1by disabling the contiguous visual signal\.General ability of text\-to\-image generation\. A key metric for text\-to\-image generation is a model’s ability to follow textual instructions\. We evaluateLibra\-2’s general generative capability using the widely adopted GenEval\[geneval\]benchmark\. As shown in Tab[IV](https://arxiv.org/html/2608.20382#S7.T4),Libra\-2ranks among the leading models and even surpasses systems designed specifically for generation tasks\. Notably,Libra\-2excels at “counting,” a long\-standing weakness of prior models, outperforming the previous state of the art by 12%\. This may because counting is closely tied to language comprehension: the model must interpret numerals in text and map them to their visual realizations\. Prior approaches may not have captured this alignment effectively\. By enabling bottom\-up alignment between visual and textual modalities, the Libra architecture affords a more faithful grounding of numeric concepts\.

Visual quality of image generation\. We evaluateLibra\-2’s image generation quality using the FID metric\[fid\], as shown in Tab\.[V](https://arxiv.org/html/2608.20382#S7.T5)\. Two benchmarks are used, where COCO\[coco\]focuses on realism and MJHQ\[mjhq\]emphasizes aesthetics\. Compared to other unified models,Libra\-2consistently produces high\-quality images, achieving the best FID score\. To quantify the improvement ofLibra\-2overLibra\-1, we build an image\-generation\-capable variant ofLibra\-1by disabling the contiguous visual signal \(see Sec\.[V\-C](https://arxiv.org/html/2608.20382#S5.SS3)\) and fine\-tuning it on an additional 10M image\-text pairs from the pretraining data\.Libra\-2substantially mitigatesLibra\-1’s shortcomings in image generation, achieving a relative FID reduction of over 80% on both COCO and MJHQ\.

It is worth noting that FID is a vision\-centric metric, focusing solely on how closely the generated images resemble the target images, thus primarily reflecting the effectiveness of vision modeling, rather than cross\-modal comprehension\. Based on this, we attributeLibra\-2’s superior performance on FID to its separated vision branch, which models visual information independently\. In contrast, previous works typically couple vision and language modeling within a single backbone, potentially reduce the focus on vision\. By separating the vision branch,Libra\-2allows for more specialized and effective vision modeling, reducing the impact of language modeling on vision\.

Case study\. Fig\.[5](https://arxiv.org/html/2608.20382#S7.F5)showcases representative text\-to\-image results fromLibra\-2\. We compareLibra\-2with two baselines—Stable Diffusion XL \(SDXL\)\[sdxl\], a widely used state\-of\-the\-art text\-to\-image model, and Show\-O\[show\-o\], a unified MLLM trained on data most similar to ours\. As shown,Libra\-2demonstrates a deeper understanding of complex prompts \(*e\.g\.*, “Sydney Opera House next to the Eiffel Tower”\) and more precise control over object counts, accurately rendering the specified quantities \(*e\.g\.*, “two rosehips” and “four cut stickers”\)\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x6.png)Figure 6:Several failure cases ofLibra\-2’s generated images\.We present several failure cases in Fig\.[6](https://arxiv.org/html/2608.20382#S7.F6), which illustrate multiple shortcomings ofLibra\-2’s image generation\. First, the visual detail is insufficient: for instance, the clock in the first image lacks clear tick marks\. This may be due to the limited resolution of the generated images and constraints imposed by the VAE encoder\. Second, the model struggles to render accurate, legible text; in the second image, the word “Libra” is not correctly produced\. A plausible explanation is that textual content is highly structured, which is friendly to perception \(*e\.g\.*, OCR\[ocr\]\) but correspondingly raising difficulties in generation: even small errors can cause large readability degradations\. Achieving robust text generation typically requires a lot of specialized data to strengthen the model’s text\-generation capability\. However, the Libra series is trained on noisy internet image\-text pairs with only sparse textual signals, which hinders reliable alignment between words and visual content\. Third, the model often fails to resolve spatial relations\. In the third image, when it does not properly understand the relation “to the right of,” it tends to merge the two objects into one\. This may stem from training data that mostly feature single objects and lack explicit positional supervision\.

### VII\-EAnalysis on Understanding\-only Modeling

In this section, we present ablation studies of the core architectural design of the Libra series, using theLibra\-1model as a representative case\. The evaluation mainly focuses on three general multimodal understanding benchmarks: MME\[mme\], POPE\[pope\], and SEED\[seed\]\.

Libra architecture v\.s Unified architecture\.In Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(a\), we remove the switch attention design in Libra, where Libra degenerates to a unified architecture like Show\-O\[show\-o\]\. The results show that the coupled vision and language systems exhibit obvious performance degradation on zero\-shot tasks\.

We further investigate the impact of the cross\-modal bridge within the switch\-attention module, as shown in Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(b\)\. The results indicate that adding only an additional vision branch without cross\-modal bridge yields limited gains and consistently performs worse than the originalLibra\-1model\. Moreover, compared to the unified\-architecture variant \(Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(a\)\), the variant without the cross\-modal bridge \(Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(b\)\) shows limited improvement, despite its larger parameter size\. his indicates the importance of a reasonable cross\-modal interaction strategy\. Fig,[3](https://arxiv.org/html/2608.20382#S4.F3)provides additional evidence of the effectiveness of the cross\-modal bridge\.

TABLE VI:Ablation of the basic architecture design onLibra\-1\.Impact of hybrid vision inputs\.InLibra\-1, we take the contiguous visual signal along with the discrete tokenized embeddings as hybrid inputs for both accurate visual perception and robust generative visual modeling\. We claim that the contiguous visual signal plays a crucial role in accurate visual perception\. To validate this, we remove the contiguous visual signals inLibra\-1’s hybrid inputs and only retain the discrete embeddings\. As shown in Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(c\), a clear performance degradation rises when solely using discrete inputs \(*e\.g,*, \-14\.2% on SEED\)\.

Impact of unified supervision\.Libra\-1employs unified discrete autoregressive modeling across both the language and vision modalities\. Although it cannot generate images due to the hybrid visual inputs, the vision\-side modeling enables the model to acquire visual knowledge in a bottom\-up, generative manner\. Tab\.[VI](https://arxiv.org/html/2608.20382#S7.T6)\(d\) shows a variant trained with supervision only on the language side, which yields a consistent drop in performance\. This indicates the importance of vision modeling for in\-depth visual understanding\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x7.png)\(a\)Dll​a​y​e​rD\_\{l\}^\{layer\}
![Refer to caption](https://arxiv.org/html/2608.20382v1/x8.png)\(b\)Dlh​e​a​dD\_\{l\}^\{head\}

Figure 7:Mean absolute error \(MAE\) of the attention activations\. \(a\) Layer\-wise attention diversity\. \(b\) Head\-wise attention diversity\.Attention diversity\. We analyzeLibra\-1’s attention patterns on a VQA task in Fig\.[7](https://arxiv.org/html/2608.20382#S7.F7), averaging results over 100 VQA samples\. We report two metrics based on the mean absolute error:

1. 1\.Layer\-wise attention diversity\. In Fig\.[7](https://arxiv.org/html/2608.20382#S7.F7)\(a\), we measure how each layer’s answer\-to\-image attention differs from the cross\-layer mean\. For a sampless, letAl,h\(s\)∈ℝNA\_\{l,h\}^\{\(s\)\}\\in\\mathbb\{R\}^\{N\}denote the attention activation overNNimage tokens induced by the answer token at layerlland headhh\. Define the layer\-averaged activationAl\(s\)=1H​∑h=1HAl,h\(s\)A\_\{l\}^\{\(s\)\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}A\_\{l,h\}^\{\(s\)\}and the cross\-layer meanA¯\(s\)=1L​∑l=1LAl\(s\)\.\\bar\{A\}^\{\(s\)\}=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}A\_\{l\}^\{\(s\)\}\.The layer\-wise attention difference is Dllayer=1S​∑s=1S1N​‖Al\(s\)−A¯\(s\)‖1,D^\{\\text\{layer\}\}\_\{l\}=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}\\frac\{1\}\{N\}\\left\\lVert A\_\{l\}^\{\(s\)\}\-\\bar\{A\}^\{\(s\)\}\\right\\rVert\_\{1\},whereS=100S=100\. Larger values indicate greater across\-layer diversity\.
2. 2\.Head\-wise attention diversity\. In Fig\.[7](https://arxiv.org/html/2608.20382#S7.F7)\(b\), we quantify the diversity across heads within each layer\. For samplessand layerll, letA¯l\(s\)=1H​∑h=1HAl,h\(s\)\.\\bar\{A\}\_\{l\}^\{\(s\)\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}A\_\{l,h\}^\{\(s\)\}\.The head\-wise deviation for layerllis Dlhead=1S​∑s=1S1H​∑h=1H1N​‖Al,h\(s\)−A¯l\(s\)‖1\.D^\{\\text\{head\}\}\_\{l\}=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\frac\{1\}\{N\}\\left\\lVert A\_\{l,h\}^\{\(s\)\}\-\\bar\{A\}\_\{l\}^\{\(s\)\}\\right\\rVert\_\{1\}\.Larger values indicate greater across\-head diversity within that layer\.

Fig\.[7](https://arxiv.org/html/2608.20382#S7.F7)reveals several interesting findings: 1\) LLaVA1\.5\[llava1\.5\], a well\-known MLLM baseline, demonstrates similar attention patterns across layers, with lower attention diversity, whileLibra\-1shows diverse attention patterns across all layers\. These results suggest that the decoupled vision\-language system has lower learning redundancy in both layer\-wise and head\-wise aspects, as observed through diverse attention patterns\.

TABLE VII:Ablation of the improvements for unified multimodal understanding and generation onLibra\-2\.
### VII\-FAnalysis on Unified Modeling

InLibra\-2, we extend the Libra architecture to unified multimodal understanding and generation\. We introduce important improvements including Switch\-FFN, UniRoPE, and continuous visual modeling, focusing on architecture, representation, and supervision, respectively\. Below, we analyze the impact of these improvements\. We evaluate on one understanding benchmark, MME\[mme\], and one generation benchmark, GenEval\[geneval\]\.

Impact of Switch FFN\. In Sec\.[IV](https://arxiv.org/html/2608.20382#S4), we incorporate a Switch FFN intoLibra\-2to decouple understanding from generation\. Tab\.[VII](https://arxiv.org/html/2608.20382#S7.T7)\(a\) presents a variant ofLibra\-2in which the Switch FFN is replaced with a vanilla FFN\. Removing the Switch FFN decreases MME by 69 points and GenEval by 2%\. We attribute this degradation to a granularity mismatch between understanding and generation: a single vanilla FFN forces both into a shared feature space, resulting in inferior performance\.

Impact of unified RoPE\. As discussed in Sec\.[VI\-B](https://arxiv.org/html/2608.20382#S6.SS2), vision and language exhibit a key structural difference: visual data are organized in two\-dimensional spatial layouts, whereas language is arranged as a one\-dimensional causal sequence\. We therefore propose UniRoPE, a unified positional encoding that reconciles these two modalities\. This design is crucial for high\-quality cross\-modal understanding and generation\. As shown in Tab\.[VII](https://arxiv.org/html/2608.20382#S7.T7)\(b\), replacing UniRoPE with vanilla RoPE inLibra\-2substantially reduces the MME score from 1475 to 1252 and the GenEval score from 0\.63 to 0\.42, which largely support the effectiveness of UniRoPE\.

Impact of continuous image modeling\. InLibra\-1, we adopt a hybrid tokenization strategy that allows the model to take both continuous and discrete signals as inputs while being supervised with discrete autoregression\. This hybrid design lets the model access lossless continuous visual features while performing discrete modeling that promotes deeper data understanding; however, it cannot be used for generation, because the whole continuous signals of an image are unavailable before the image itself is generated\. To address this,Libra\-2employs masked image generation in continuous space, feeding continuous image features directly to the model and training it with a masked generation manner\. The continuous vision space is crucial for MLLMs, where multimodal understanding relies on fine\-grained visual information\.

Tab\.[VII](https://arxiv.org/html/2608.20382#S7.T7)\(c\) compares these two tokenization strategies by replacing theLibra\-2’s inputs with hybrid tokens and training the model, as inLibra\-1, under a unified discrete autoregressive manner for both the vision and language modalities\. Under this setting, the model cannot generate images and exhibits an obvious degradation in multimodal understanding \(\-59 on the MME score\)\. This is because the hybrid tokenization strategy inLibra\-1introduces discrete noise into the continuous features during the hybridization process\.Libra\-2’s continuous\-space masked image generation not only enables high\-quality image generation but also strengthens understanding\.

TABLE VIII:Analysis on the mutual impact of understanding and generation withinLibra2\.Mutual impact of generation and understanding\.Tab\.[VIII](https://arxiv.org/html/2608.20382#S7.T8)compares the unifiedLibra\-2\(Und\. & Gen\.\) model with its single\-task variants: an understanding\-only model and a generation\-only model\. The single\-task variants are obtained by adjusting the sampling probabilities of understanding and generation data during training \(see “T2I:I2T ratio” in Tab\.[II](https://arxiv.org/html/2608.20382#S7.T2)\)\. We construct understanding and generation samples from the same image\-text pairs by swapping the order of modalities in the sequence,*i\.e\.*, “<image\><text\>” for understanding and “<text\><image\>” for generation\. During instruction tuning, we directly control the ratio of understanding to generation data; the understanding\-only and generation\-only variants simply use one type of data exclusively\.

The results in Tab\.[VIII](https://arxiv.org/html/2608.20382#S7.T8)indicate that adding generation training has a modest yet generally positive effect on understanding\. Compared with the understanding\-only variant, the unifiedLibra\-2improves VQAv2 from 76\.0% to 77\.5% and MMB from 58\.2% to 60\.0%\. In the opposite direction, understanding training also benefits generation: compared with the generation\-only variant, the unifiedLibra\-2consistently achieves better performance across all three generation benchmarks\. Overall, understanding and generation are mutually reinforcing\.

![Refer to caption](https://arxiv.org/html/2608.20382v1/x9.png)Figure 8:GenEval performance versus sampling steps for multiple CFG scales: \(a\) masking sampling step, \(b\) diffusion sampling step\.Impact of sampling steps\.We consider two types of sampling steps inLibra\-2: \(1\) masking sampling steps, where masked image generation is an autoregressive\-like process that progressively generates image tokens over multiple steps; the number of masking steps controls the granularity of whole\-image generation; and \(2\) diffusion steps in the lightweight diffusion head used to predict each image token; the number of diffusion steps controls the token\-level granularity\.

We evaluate GenEval across different classifier\-free guidance \(CFG\) scales\[cfg\]\. As shown in Fig\.[8](https://arxiv.org/html/2608.20382#S7.F8)\(a\), increasing the number of masking sampling steps consistently improves generation quality\. In contrast, Fig\.[8](https://arxiv.org/html/2608.20382#S7.F8)\(b\) shows that the number of diffusion steps has only a modest effect on overall performance\. This is because diffusion is applied only to individual tokens rather than to the entire image, so a relatively small number of steps is sufficient for this easier subtask\. Meanwhile, higher CFG scales generally yield better generations\. However, at high CFG \(*e\.g\.*, 8\.0 and 10\.0\) combined with a low number diffusion steps, performance degrades substantially: when the number of diffusion steps is below 400, stronger guidance makes sampling more brittle, and too few sampling steps cause accumulated noise, resulting in meaningless images\.

## VIIIConclusion

Through developing the Libra series, includingLibra\-1andLibra\-2, we show that a decoupled vision\-language system can improve both multimodal understanding and generation\. The Libra architecture achieves this via switch attention and switch FFN modules\. We observe that this design yields diverse attention patterns across layers, suggesting reduced learning redundancy\. In addition, we integrate continuous\-space masked image generation into a unified MLLM, which endowsLibra\-2with both accurate multimodal understanding and high\-quality image generation\. We would like to claim that vision and language should be integrated in a more reasonable manner beyond simple modality alignment\. We hope our work could provoke more consideration in MLLM designs\.

## References