Representation Forcing for Bottleneck-Free Unified Multimodal Models
Summary
Introduces Representation Forcing (RF), a technique that enables unified multimodal models to perform both perception and generation end-to-end without external VAE latent spaces, matching state-of-the-art VAE-based models in image generation while improving understanding.
View Cached Full Text
Cached at: 06/01/26, 03:18 AM
Paper page - Representation Forcing for Bottleneck-Free Unified Multimodal Models
Source: https://huggingface.co/papers/2605.31604 Published on May 29
#3 Paper of the day Authors:
,
,
,
,
,
,
,
,
,
,
,
Abstract
Representation Forcing enables unified multimodal models to perform both perception and generation tasks end-to-end without relying on external latent spaces, matching state-of-the-art performance in image generation while improving understanding capabilities.
Unified multimodal models(UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we proposeRepresentation Forcing(RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predictvisual representationsas intermediate tokens before pixels; these tokens then stay in context to guidepixel diffusionwithin the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2605\.31604
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.31604 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.31604 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.31604 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
How to setup a local coding agent on macOS
A detailed tutorial on setting up a local coding agent on macOS using Gemma 4 with MTP draft model and llama.cpp, achieving ~24% speed improvement through speculative decoding.
@GitHub_Daily: To dive deep into model research, you can't just stay at the application layer—you need to understand how the underlying system is trained and optimized. I stumbled upon LLMSys-PaperList, a carefully curated collection of papers related to large model systems. It is continuously updated from 2022 to the latest top conference papers in 2026, and organized by categories such as training, inference, multimodality...
A carefully curated collection of papers related to large model systems, covering training, inference, multimodality, and more. It is continuously updated and includes technical reports, frameworks, and courses, making it a valuable reference for researchers and developers.
APCyc: Property-Informed Design of Cyclic Peptides via Automated Cyclization
APCyc is a target-aware generative framework that designs cyclic peptides with controlled physicochemical properties by explicitly modeling cyclization patterns and using Bayesian posterior guidance.
PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization
PRISMR proposes a framework using hypernetworks and LoRA to internalize list structure, overcoming parse collapse in multimodal listwise ranking. It introduces a large-scale benchmark and shows reduced parse collapse and improved ranking performance across domains and backbones.
MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
MLUBench is a large-scale benchmark for lifelong unlearning in multimodal large language models (MLLMs), featuring 127 entities across 9 classes. The paper identifies that existing unlearning methods suffer from cumulative degradation and proposes LUMoE to mitigate this, showing significant improvements.