Rethinking Projector Training in Multimodal LLMs

Hugging Face Daily Papers Papers
multimodal-llm projector-training optimal-transport vision-language training-free feature-alignment

Summary

该论文提出 PORTAL，一种无需配对图像-文本数据、基于最优传输的多模态投影器初始化方法，可以完全跳过投影器训练阶段，在六个 LLM 主干和 16 个基准上匹敌使用 558k 图文对的标准训练流程。

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only 10 to 20% of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in 3 of 7 (vision encoder, LLM) settings and reaches an average 98% of its score across the other four.
Original Article
View Cached Full Text

Cached at: 10/15/26, 02:58 PM

Paper page - Rethinking Projector Training in Multimodal LLMs

Source: https://huggingface.co/papers/2610.18598

Abstract

PORTAL initializes multimodal projectors without paired data by aligning vision and language feature distributions via optimal transport, matching standard training performance while eliminating projector training.

Multimodal LLMsare typically trained by first aligning avision encoderwith a language model viaprojector trainingon large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of theprojector trainingstage remains poorly understood. We find that (i) using only 10 to 20% of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest thatprojector trainingmainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we proposePORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics.PORTALcomputes an optimal-transport-based map betweenGaussian approximationsof the vision and language feature distributions on a sharedprincipal subspace, so that theprojector trainingstage can be skipped entirely. Without any paired data,PORTALmatches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, twovision encoders, and 16 benchmarks. It outperforms that pipeline in 3 of 7 (vision encoder, LLM) settings and reaches an average 98% of its score across the other four.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.18598

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.18598 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.18598 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.18598 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

arXiv cs.CL

This paper proposes a query-based cross-modal projector that compresses visual tokens via cross-attention to improve Mamba-based multimodal LLMs, boosting both performance and throughput on vision-language benchmarks while eliminating the need for manual 2D scan order design.

Projector Is All You Train

arXiv cs.CL

This paper demonstrates that training only the projector in multimodal large language models achieves strong performance on 3D tasks, avoids language model drift, and improves training efficiency compared to joint training methods.

Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

Hugging Face Daily Papers

ART (Art-based Reinforcement Training) enables parameter-efficient fine-tuning of frozen multimodal LLMs by optimizing raw visual input via gradient backpropagation, achieving performance comparable to LoRA while supporting pre-compiled computational graphs for high-throughput engines like vLLM.