MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

# MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Source: [https://arxiv.org/html/2610.20965](https://arxiv.org/html/2610.20965)
Robert MoseleyMd Kaykobad RezaAffiliation:University of California,Affiliation:RiversideEmail:[mreza025@ucr\.edu](mailto:)Ameya PatilAffiliation:AmazonEmail:[ameyapat@amazon\.com](mailto:)Edward AyrapetianAffiliation:AmazonEmail:[edayrap@amazon\.com](mailto:)Salman AsifAffiliation:University of California,Affiliation:RiversideEmail:[sasif@ucr\.edu](mailto:)

###### Abstract

Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data\. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress\. We introduce*Modality Self\-Play*\(MSP\), a training framework for multimodal foundation models based on proxy\-mediated modality composition\. Instead of relying on paired cross\-modal data, MSP uses anchor\-supporting proxy textual descriptions of unseen modalities as semantic bridges during training\. Given an observed modality \(e\.g\., image or audio\), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned with the observed modality through learned bridges, enabling cross\-modal interaction without real paired non\-text supervision\. Our central hypothesis is that proxy text, combined with contrastively trained pretrained encoders, provides sufficient semantic structure for multimodal composition to emerge\. We show that MSP enables zero\-shot modality composition: models trained only with a real anchor modality and accompanying proxy descriptions can integrate and reason over multiple modalities at inference time despite never observing real paired non\-text data during training\. Empirically, MSP achieves state\-of\-the\-art or competitive performance on multiple multimodal benchmarks\. Ablation studies further demonstrate that learned bridges improve alignment, while anchor\-supporting proxy descriptions enable effective cross\-modal composition\. MSP removes the need for real paired non\-text \(e\.g\., audio\-vision\) supervision, while continuing to rely on modality\-text pairs, pretrained contrastive encoders, and modality\-distribution bridge calibration\.

## 1Introduction

Large multimodal models \(LMMs\)\([Huang et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib2);[Wu et al\., 2023a](https://arxiv.org/html/2610.20965#bib.bib1)\)have advanced the frontier of intelligent systems by enabling unified reasoning over language, vision, audio, and other heterogeneous modalities\. Much of this progress has been driven by scaling: larger models, broader modality coverage, and increasingly large paired multimodal datasets have led to strong gains across perception and reasoning tasks\([Qwen Team, 2025a](https://arxiv.org/html/2610.20965#bib.bib15);[Li et al\., 2025b](https://arxiv.org/html/2610.20965#bib.bib16)\)\. However, most existing approaches rely on a common assumption: effective cross\-modal reasoning requires explicit supervision from paired multimodal data\. This dependence on paired multimodal data presents a key scalability challenge\. High\-quality aligned multimodal datasets are expensive to collect, difficult to scale across modality combinations, and often domain\-specific\. As models expand to incorporate more modalities, the cost of acquiring comprehensive paired data grows combinatorially\. As a result, progress in multimodal learning is increasingly constrained by the availability of paired multimodal datasets rather than model design\.

In contrast, large amounts of bimodal data \(e\.g\., vision\-language or audio\-language\) are readily available on open source repositories\. These datasets contain rich semantic information, raising an important question:*can models learn to reason over multiple modalities without ever observing real paired non\-text data during training?*Prior work has explored related directions through shared encoders\([Han et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib4)\), modular model design\([Su et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib8);[Han et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib3)\), and weak supervision\([Panagopoulou et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib5)\)\. However, most approaches still rely on some form of paired multimodal supervision during training\([Wu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib6)\)or exhibit limited performance on standard multimodal benchmarks\([Zhao et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib20);[Han et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib4);[Panagopoulou et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib5)\)\.

In this work, we explore an alternative approach based on*proxy\-mediated modality alignment*\. We introduceModality Self\-Play \(MSP\), a training paradigm that replaces real paired non\-text supervision with proxy textual descriptions\. Given an observed modality \(e\.g\., an image\), MSP generates proxy text describing complementary unseen modalities \(e\.g\., audio\)\. These proxy descriptions are encoded using contrastively pretrained text encoders \(e\.g\., CLIP/CLAP\-style encoders\([Radford et al\., 2021](https://arxiv.org/html/2610.20965#bib.bib9);[Wu et al\., 2023b](https://arxiv.org/html/2610.20965#bib.bib10)\)\) and projected to the shared language embedding space via a bridge and MLP projector\. During training, the model never observes real paired non\-text data; instead, it learns from single modalities paired with proxy descriptions that act as semantic bridges\. This design leverages the fact that contrastively pretrained encoders provide transferable cross\-modal structure\([Girdhar et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib27)\)\. Proxy text does not replicate unseen modalities, but provides a shared semantic space through which representations can be aligned\. We therefore do not assume the absence of cross\-modal priors, but rather eliminate the need for real paired non\-text data during training\.

Our central hypothesis is that proxy\-mediated alignment is sufficient for multimodal composition to emerge, enabling LMMs to reason jointly over multiple modalities without real paired non\-text data during training\. We evaluate this hypothesis empirically across multiple multimodal reasoning benchmarks\. We show that MSP enables*zero\-shot modality composition*: models trained using single modalities and proxy descriptions can reason over multiple modalities at inference time despite never observing them jointly during training\. MSP achieves*competitive or state\-of\-the\-art performance*on standard multimodal benchmarks\([Yang et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib44);[Li et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib30);[Li et al\., 2024b](https://arxiv.org/html/2610.20965#bib.bib48)\)without using real paired non\-text training data \(Section[4\.2](https://arxiv.org/html/2610.20965#S4.SS2)\)\. We further analyze the role of proxy descriptions and alignment mechanisms through ablation studies, showing that learned bridges improve alignment between proxy embeddings and the corresponding modality representations, resulting in improved multimodal reasoning performance \(Section[4\.3](https://arxiv.org/html/2610.20965#S4.SS3)\)\.Our contributions are summarized as follows:

- •We introduceModality Self\-Play \(MSP\), a training framework that replaces real paired non\-text supervision with proxy\-mediated alignment\.
- •We show that proxy textual descriptions can serve as semantic bridges, enabling zero\-shot multimodal composition without real paired non\-text training data\. MSP achieves competitive or state\-of\-the\-art performance on multiple multimodal benchmarks \(Section[4\.2](https://arxiv.org/html/2610.20965#S4.SS2)\)\.
- •Ablation studies show that bridges improve alignment between proxy and modality representations, leading to stronger multimodal reasoning performance without real paired non\-text data \(Section[4\.3](https://arxiv.org/html/2610.20965#S4.SS3)\)\.

MSP removes the need for real paired non\-text \(e\.g\., audio\-vision\) supervision, while continuing to rely on modality\-text pairs, pretrained contrastive encoders, and modality\-distribution bridge calibration\.

## 2Related Work

Our work relates to three lines of research in multimodal foundation models: \(i\) multimodal LLMs \(MLLMs\) trained with paired multimodal data, \(ii\) approaches that align independently observed modalities to a shared language embedding space, and \(iii\) text\-mediated proxy methods that use language as an intermediate representation\.

Training MLLMs with paired multimodal data\.The dominant paradigm for MLLMs relies on training with explicitly paired multimodal data, where language or instruction prompts are aligned with one or more observed modalities\. Early vision\-language models such as BLIP\-2\([Li et al\., 2023a](https://arxiv.org/html/2610.20965#bib.bib11)\)and LLaVA\([Liu et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib13)\)demonstrate strong performance through large\-scale image\-text pretraining and instruction fine\-tuning\. This paradigm has been extended to broader modality coverage: NExT\-GPT\([Wu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib6)\), AnyGPT\([Zhan et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib7)\), Unified\-IO 2\([Lu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib34)\), and MIO\([Wang et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib14)\)integrate multiple modalities within unified architectures, while recent omni\-modal systems such as Qwen2\.5\-Omni\([Qwen Team, 2025a](https://arxiv.org/html/2610.20965#bib.bib15)\), Baichuan\-Omni\-1\.5\([Li et al\., 2025b](https://arxiv.org/html/2610.20965#bib.bib16)\), and MiniCPM\([Yao et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib32)\)scale this approach with large multimodal training pipelines\. While effective, these methods depend on the availability of aligned paired multimodal datasets, which can be costly to curate and difficult to scale across modality combinations\. In contrast, our work explores whether multimodal composition can be learned without requiring explicit paired multimodal supervision\.

Multimodal learning via modality\-text alignment\.A growing line of work reduces reliance on direct multimodal pairing by aligning each modality to language or to a shared embedding space\. ImageBind\-LLM\([Han et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib3)\)leverages pretrained modality\-specific encoders to align multiple modalities to a shared image embedding space before connecting them to a large language model\. PandaGPT\([Su et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib8)\), X\-InstructBLIP\([Panagopoulou et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib5)\), and OneLLM\([Han et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib4)\)extend this approach by aligning multiple modalities to a shared language embedding space through projectors and modality\-specific adapters\. ChatBridge\([Zhao et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib20)\)utilizes language as an intermediate representation connecting modalities, reducing the need for direct modality\-to\-modality pairing\. These approaches demonstrate that language can serve as a unifying interface for multimodal learning\. However, they often struggle with cross\-modal composition at inference time, since the models are not explicitly trained to reason jointly over multiple modalities\. In contrast, MSP introduces proxy\-mediated alignment that incorporates compositional signals during training without requiring real paired non\-text datasets\.

Text\-mediated proxies and representation bridging\.Several methods use text representations as proxies for other modalities\. DeCap\([Li et al\., 2023b](https://arxiv.org/html/2610.20965#bib.bib21)\)and CapDec\([Nukrai et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib22)\)train decoders using only text embeddings and map visual representations into the text embedding space during inference, enabling zero\-shot captioning\. Similar ideas have been explored for image, video, and audio captioning\([Wang et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib23);[Deshmukh et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib24);[Li et al\., 2025a](https://arxiv.org/html/2610.20965#bib.bib25)\)\. These approaches demonstrate that text can act as an intermediate representation when multimodal data is limited\.

However, these methods primarily focus on single\-modality generation tasks and do not investigate whether text proxies can be used to learn multimodal composition without paired multimodal data\. MSP differs from existing approaches in two important ways\. First, proxy text conditioned on an observed modality is generated and used during training to guide cross\-modal alignment, rather than serving as a substitute for unseen modalities at inference time\. Second, these proxies are used to train the model for cross\-modal reasoning by using them as semantic surrogates for unseen modalities within the LLM\. This positions MSP as a training\-time framework for inducing multimodal composition through proxy\-mediated alignment\.

## 3Method

We introduce*Modality Self\-Play*\(MSP\), a training framework that enables MLLMs to reason jointly over audio and vision at inference time without observing any real audio\-vision pairs during training\. We define*paired cross\-modal data*as samples containing two or more non\-text modalities originating from the same underlying source \(e\.g\., audio and vision from the same recording\)\. MSP uses standard modality\-text supervision \(e\.g\., image\-language or audio\-language\) to train modality\-specific projectors and calibrate the alignment bridges described below, but never uses paired non\-text modality data\. We first describe the model architecture in Section[3\.1](https://arxiv.org/html/2610.20965#S3.SS1), followed by the two training stages: per\-modality projector training \(Section[3\.2](https://arxiv.org/html/2610.20965#S3.SS2)\) and self\-play composition training \(Section[3\.3](https://arxiv.org/html/2610.20965#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2610.20965v1/msp_pipeline.png)Figure 1:MSP training pipeline\.\(1\)Phase 1 trains one projector per modality using modality–text pairs only\.\(2\)The frozen Phase\-1 model then generates a text proxy for the modality it does not observe\.\(3\)Phase 2 trains on one real modality together with the proxy for the missing one\. No training sample ever contains two real non\-text modalities\. A component\-level view of the same architecture, with the bridge formulation and the full token sequence, is given in Figure[2](https://arxiv.org/html/2610.20965#A1.F2)\.### 3\.1Model Architecture

Figure[1](https://arxiv.org/html/2610.20965#S3.F1)summarizes the full training pipeline, and Figure[2](https://arxiv.org/html/2610.20965#A1.F2)illustrates the four main components of MSP: modality\-specific encoders, modality\-specific projectors, text\-to\-modality bridges, and the language model\.

Modality\-specific encoders\.We use frozen contrastive encoders to map each non\-text modality into a fixed\-dimensional feature representation\. Specifically, we use LAION\-CLAP\([Wu et al\., 2023b](https://arxiv.org/html/2610.20965#bib.bib10)\)for audio and MetaCLIP ViT\-B/16\([Xu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib26)\)for vision\. Each encoder produces a single512512\-dimensional feature embedding per input\. The encoders remain frozen throughout training to ensure that both real and proxy representations reside in a consistent embedding space, enabling the bridge modules to learn meaningful transformations between them\.

Modality\-specific projectors\.Each modality uses a lightweight trainable MLP projectorPm:ℝ512→ℝDP\_\{m\}:\\mathbb\{R\}^\{512\}\\rightarrow\\mathbb\{R\}^\{D\}to map modality\-specific features into the language embedding space, whereD=4096D\{=\}4096is the hidden dimension of Qwen3\-8B\. Given an inputxmx\_\{m\}from modalitymm, the encoder produces a feature representationzmz\_\{m\}, which is projected into the language embedding space as

hm=Pm​\(Em​\(xm\)\)\.h\_\{m\}=P\_\{m\}\(E\_\{m\}\(x\_\{m\}\)\)\.\(1\)Each modality has its own projector\. We represent every modality using a single token, ensuring that both real and proxy features share the same representation format\.

LLM backbone\.We use Qwen3\-8B\([Qwen Team, 2025b](https://arxiv.org/html/2610.20965#bib.bib29)\)as the language backbone\. The input prompt contains modality\-specific special tokens,<audio\>and<vision\>, corresponding to audio and vision inputs, respectively\. During the forward pass, these tokens are replaced with the projected modality embeddingshmh\_\{m\}, following the standard design used in multimodal LLMs\([Liu et al\., 2023](https://arxiv.org/html/2610.20965#bib.bib13);[Li et al\., 2023a](https://arxiv.org/html/2610.20965#bib.bib11)\)\. We apply LoRA adapters to the attention and MLP layers of Qwen3\-8B while keeping the remaining LLM parameters frozen\.

### 3\.2Phase 1: Per\-Modality Projector Training

In Phase 1, we train the audio and vision projectors independently using standard modality\-caption datasets\. For each batch from modalitym∈\{a,v\}m\\in\\\{a,v\\\}, the input prompt contains a single special token \(e\.g\.,<audio\>or<vision\>\) along with the text instruction\. The corresponding modality\-specific feature is computed and projected into the LLM embedding space using \([1](https://arxiv.org/html/2610.20965#S3.E1)\), which replaces the special token in the prompt\. The encoderEmE\_\{m\}is frozen and used only for feature extraction, while the projectorPmP\_\{m\}and the LoRA adapters in the LLM backbone are trained\. Training is performed using standard next\-token prediction over the response tokens:

ℒP1=−∑j∈ℛlogpθ\(yj∣y<j,q,hm\),\\mathcal\{L\}\_\{\\mathrm\{P1\}\}=\-\\sum\_\{j\\in\\mathcal\{R\}\}\\log p\_\{\\theta\}\(y\_\{j\}\\mid y\_\{<j\},q,h\_\{m\}\),\(2\)whereqqdenotes the text prompt,ℛ\\mathcal\{R\}denotes response tokens, and both prompt and placeholder tokens are excluded from the loss computation\. The resulting Phase 1 model serves two roles in Phase 2: it initializes the modality\-specific projectors and LoRA adapters, and it is used as the generator for self\-play proxy text describing unseen modalities\.

### 3\.3Phase 2: Self\-Play Composition Training

Phase 2 enables the model to learn cross\-modal composition between audio and vision without using any real paired non\-text data\. Since no audio\-vision pairs are available, each training example uses a single real modality as an*anchor*\(e\.g\., vision\) and a*proxy text description*for the unseen modality \(e\.g\., audio\)\. The proxy text is first encoded using a contrastive text encoder and then transformed through a learned text\-to\-modality bridge before being mapped into the LLM embedding space via the modality\-specific projector\. This allows proxy representations to be injected into the model through the same interface as real modality features\. When preparing our dataset, any data that could be considered “paired” by definition \(i\.e\., containing both vision and audio\) was exclusively utilized for one modality for a single training sample\. If, for example, there was some sample containing both vision and audio, we designated the sample for either being used with its audio and not with its vision, or vice versa\. This means that any given sample would only ever be used with one of its modalities—not one pass with vision, and a later pass with its audio\. We refer to this as the one\-real\-anchor rule\.

In the following, we describe the proxy generation process, the text\-to\-modality bridge that aligns proxy embeddings with real modality representations, and the Phase 2 training objective\.

Self\-Play Proxy Generation\.For each Phase\-2 example, we select one modality as the anchor \(e\.g\., audio\) and a different modality as the target \(e\.g\., vision\)\. The Phase\-1 model is conditioned on the anchor via its corresponding pathway and prompted to generate a short text descriptiontm′′t^\{\\prime\}\_\{m^\{\\prime\}\}of the unseen modalitym′m^\{\\prime\}\. We use prompt templates aligned with the original training style of the contrastive encoders \(AudioCaps\-style for CLAP, COCO\-style for MetaCLIP\); full prompts are provided in Appendix[C](https://arxiv.org/html/2610.20965#A3)\. Since the proxy is generated from the anchor, it is anchor\-supporting by construction\. All proxies are generated once before Phase 2 and cached, so no online generation is required during training\.

Text\-to\-Modality Bridge\.The text\-encoder and modality\-encoder outputs of the contrastive encoders lie in related but shifted regions of a shared embedding space\([Liang et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib28)\)\. Directly feeding text\-encoder features into the projectorPmP\_\{m\}would lead to a distribution mismatch, sincePmP\_\{m\}is trained on modality\-encoder features during Phase 1\. To address this, we learn a linear bridgeBmB\_\{m\}that maps the embeddings from the text\-encoder into the modality\-encoder embedding space\. The bridge is trained via ridge regression on 1,000 paired \(caption, modality\) samples \(AudioCaps\([Kim et al\., 2019](https://arxiv.org/html/2610.20965#bib.bib45)\)for audio and LLaVA\-665K\([Liu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib12)\)for vision\), split into 500 training and 500 held\-out samples:

Wm=\(T⊤​T\+λ​I\)−1​T⊤​R,Bm​\(t\)=Wm⊤​t,λ=1\.0,W\_\{m\}=\(T^\{\\top\}T\+\\lambda I\)^\{\-1\}T^\{\\top\}R,\\quad B\_\{m\}\(t\)=W\_\{m\}^\{\\top\}t,\\quad\\lambda=1\.0,\(3\)whereTTandRRdenote stacked text\-tower and modality\-tower embeddings, respectively\. The bridge is trained once before Phase 2 and kept fixed thereafter\. The resulting bridged proxy featurez^m′=Bm′​\(Em′text​\(tm′′\)\)\\hat\{z\}\_\{m^\{\\prime\}\}=B\_\{m^\{\\prime\}\}\(E^\{\\mathrm\{text\}\}\_\{m^\{\\prime\}\}\(t^\{\\prime\}\_\{m^\{\\prime\}\}\)\)is then passed through the projectorPm′P\_\{m^\{\\prime\}\}and inserted at the corresponding modality special token using the same mechanism as real modality inputs\. Bridge calibration is evaluated in Appendix[H](https://arxiv.org/html/2610.20965#A8), and remains frozen during Phase 2 training\.

Phase 2 Forward Pass\.Letmmbe the real anchor modality andm′m^\{\\prime\}be the missing target modality\. Phase 2 constructs one real feature,hm=Pm​\(Em​\(xm\)\)h\_\{m\}=P\_\{m\}\(E\_\{m\}\(x\_\{m\}\)\), and one bridged proxy feature,hm′′=Pm′​\(Bm′​\(Em′text​\(tm′′\)\)\)h^\{\\prime\}\_\{m^\{\\prime\}\}=P\_\{m^\{\\prime\}\}\(B\_\{m^\{\\prime\}\}\(E^\{\\mathrm\{text\}\}\_\{m^\{\\prime\}\}\(t^\{\\prime\}\_\{m^\{\\prime\}\}\)\)\)\. Both vectors have the same LLM hidden widthDDand are inserted into the prompt at their reserved modality placeholders\. An audio\-anchored example therefore contains real audio at<audio\>and a bridged vision proxy at<vision\>; a vision\-anchored example reverses the roles\. This construction gives real and proxy inputs identical token shape and insertion semantics\.

Phase 2 Training and Data\.Audio anchors are drawn from Clotho\-AQA\([Lipping et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib47)\), AudioCaps\([Kim et al\., 2019](https://arxiv.org/html/2610.20965#bib.bib45)\), WavCaps\([Mei et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib46)\), and from the audio tracks of MSRVTT\-QA and MSVD\-QA training videos\([Xu et al\., 2017](https://arxiv.org/html/2610.20965#bib.bib49)\)\. Vision anchors are drawn from VQAv2\([Goyal et al\., 2017](https://arxiv.org/html/2610.20965#bib.bib50)\), LLaVA\-665K\([Liu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib12)\), and additional frames from MSRVTT\-QA and MSVD\-QA\. The trainable parameters are the modality\-specific projectors and shared LoRA adapters, while the modality\-specific encoders and bridges remain frozen\. The objective is the same as \([2](https://arxiv.org/html/2610.20965#S3.E2)\), now conditioned on both the real anchor and the proxy feature\. This enables cross\-modal composition during training without requiring any real paired audio–vision data\.

Inference\.At inference time, real modalities are encoded and projected through the same encoder\-projector pathway used during training\. Tables[5](https://arxiv.org/html/2610.20965#S4.T5)and[6](https://arxiv.org/html/2610.20965#S4.T6)demonstrate that despite the LLM’s own ability to easily distinguish between proxies and modalities in residual space, the model converges to the same tokens in the output layer indicating that the model is able to use proxies as substitutes for real modalities during training\. We do not generate proxies at inference\.

Figure[3](https://arxiv.org/html/2610.20965#A7.F3)\(Appendix[G](https://arxiv.org/html/2610.20965#A7)\) shows representative proxy examples generated for AVQA and MUSIC\-AVQA samples\. These examples illustrate the intended role of proxy text: it should preserve the semantic content of the anchor while occupying the representation space of the missing modality\.

## 4Experiments and Results

### 4\.1Dataset and Implementation Details

Phase 1 data\.In Phase 1, we train the modality\-specific projectors together with the LLM decoder using standard modality\-text pairs to align each modality into a shared language embedding space that can be processed by the LLM\. We represent each modality using a single token\. We use the audio\-text pairs from AudioCaps\([Kim et al\., 2019](https://arxiv.org/html/2610.20965#bib.bib45)\)and WavCaps\([Mei et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib46)\)datasets, while image\-text pairs are obtained from the LLaVA\-665K dataset\([Liu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib12)\)\.

Proxy generation and Phase 2 data\.Self\-play proxies are generated after Phase 1 and cached for the remainder of training\. For each training sample, the Phase 1 model is conditioned on the observed anchor modality and prompted to generate textual descriptions of the unseen target modality in its native captioning style\. These proxy descriptions are then used as semantic surrogates for the missing modalities during subsequent MSP training\. To generate vision proxies, we use audio samples from the Clotho\-AQA\([Lipping et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib47)\), AudioCaps\([Kim et al\., 2019](https://arxiv.org/html/2610.20965#bib.bib45)\), and WavCaps\([Mei et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib46)\)datasets, together with the audio tracks of MSRVTT\-QA and MSVD\-QA\([Xu et al\., 2017](https://arxiv.org/html/2610.20965#bib.bib49)\)\. Conversely, to generate audio proxies, we use images from VQAv2\([Goyal et al\., 2017](https://arxiv.org/html/2610.20965#bib.bib50)\)and LLaVA\-665K\([Liu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib12)\), together with frames from MSRVTT\-QA and MSVD\-QA\. For video samples, we choose either audio or uniformly sample 4 frames, encode each with MetaCLIP ViT\-B/16, and mean\-pool the resulting features into a single 512\-dimensional vision token\.

Importantly, MSP never observes paired audio\-vision\-text modalities during composition training\. During Phase 2 training, we alternate between*audio \+ proxy vision description*and*vision \+ proxy audio description*samples to encourage proxy\-mediated multimodal composition\.

Bridge calibration\.Each bridge is a linear map that re\-projects text\-tower vectors into the modality\-tower space, correcting the offset between a contrastive encoder’s two towers\([Liang et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib28)\)\. We fit one per modality using the ridge regression and calibration split introduced in Section[3\.3](https://arxiv.org/html/2610.20965#S3.SS3)\(λ=1\.0\\lambda\{=\}1\.0\)\. Both bridges are fit once before Phase 2 begins and then frozen\.

Evaluation benchmarks\.We evaluate our method on three cross\-modal question answering \(QA\) benchmarks that require joint reasoning over audio and vision to answer each question:AVQA\([Yang et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib44)\),MUSIC\-AVQA\([Li et al\., 2022](https://arxiv.org/html/2610.20965#bib.bib30)\), andOmniBench\([Li et al\., 2024b](https://arxiv.org/html/2610.20965#bib.bib48)\)\. AVQA and MUSIC\-AVQA focus on audio\-visual question answering, while OmniBench extends this setting to audio\-visual\-text reasoning, enabling evaluation in a broader multimodal context\. Here, “zero\-shot” means we do not finetune specifically for performance on these benchmarks \(meaning we do not use the training data for these benchmarks to enhance performance\)\.

Implementation details\.We use Qwen3\-8B\([Qwen Team, 2025b](https://arxiv.org/html/2610.20965#bib.bib29)\)with LoRA adapters applied to attention and MLP modules with ranksr∈\{128,256,512\}r\{\\in\}\\\{128,256,512\\\}\. The modality\-specific projectors are MLPs and are fully trainable\. Frozen contrastive encoders: MetaCLIP B/16\([Xu et al\., 2024](https://arxiv.org/html/2610.20965#bib.bib26)\)for vision and LAION\-CLAP\([Wu et al\., 2023b](https://arxiv.org/html/2610.20965#bib.bib10)\)for audio; per\-modality features are pooled to a single512512\-dim token before projection\. Full optimization details, including the projector\-warmup schedule and per\-rank effective batch sizes, are listed in Appendix[D](https://arxiv.org/html/2610.20965#A4)\.

### 4\.2Performance on Multimodal Benchmarks

We evaluate MSP on three multimodal benchmark datasets and compare it against state\-of\-the\-art multimodal models\. We report results for models trained with LoRA ranks\{128,256,512\}\\\{128,256,512\\\}\. Despite never observing real paired non\-text data during training, MSP achieves competitive performance and, in several cases, outperforms baseline MLLMs on multimodal question answering tasks that require joint reasoning over paired modalities\. We report audio\-vision performance for all the datasets\.

Table 1:Performance comparison across different multimodal models on the MUSIC\-AVQA test split\. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks\.Table 2:Performance comparison across different multimodal models on the AVQA test split\. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks\.Table 3:Performance comparison across different multimodal models on OmniBench test split\. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks\.Table 4:Proxy and bridge help improve model performance\. Every cell is atr=512r\{=\}512on the test splits\.Performance on MUSIC\-AVQA benchmark\.Table[1](https://arxiv.org/html/2610.20965#S4.T1)reports zero\-shot accuracy on the MUSIC\-AVQA benchmark\. MSP with LoRA rankr=512r\{=\}512achieves65\.6565\.65, outperforming OneLLM by\+18\.05\+18\.05pp and GPT\-4o by\+15\.57\+15\.57pp, despite GPT\-4o having a substantially larger parameter count\. Ther=256r\{=\}256andr=128r\{=\}128configurations achieve64\.1764\.17and60\.2460\.24, respectively, both surpassing the baseline methods\. Unlike OneLLM, which is trained only on modality\-text pairs, MSP additionally uses proxy textual descriptions of unseen modalities during training\. This enables effective reasoning over paired audio\-visual inputs at inference time and demonstrates the effectiveness of proxy supervision for learning cross\-modal composition\.

Performance on AVQA\.Table[2](https://arxiv.org/html/2610.20965#S4.T2)reports performance on the AVQA benchmark, where MSP achieves85\.7385\.73in zero\-shot setting with LoRA rankr=256r\{=\}256, representing the highest reported zero\-shot AVQA performance among methods not finetuned on the AVQA dataset\. Performance remains consistent across ranksr∈\{128,256,512\}r\{\\in\}\\\{128,256,512\\\}, suggesting performance saturation rather than insufficient model capacity\. MSP outperforms zero\-shot baselines such as ImageBind\-LLM and X\-InstructBLIP by large margins \(30\.7330\.73–41\.4441\.44pp\)\. Notably, MSP also surpasses methods including PSAC, HGA, and HCRN, despite those models being trained on the paired AVQA train set\. These results highlight the effectiveness of MSP in learning cross\-modal composition without real paired non\-text training data\.

Performance on OmniBench\.Table[3](https://arxiv.org/html/2610.20965#S4.T3)reports performance on the OmniBench benchmark, which requires joint reasoning over vision, audio, and text modalities\. MSP with LoRA rankr=256r\{=\}256achieves43\.2643\.26, outperforming most baseline methods and achieving performance competitive with Baichuan\-Omni\-1\.5 and Gemini\-1\.5\-Pro despite never being trained on real paired non\-text data\. Notably, the closed\-source Gemini\-1\.5\-Pro achieves42\.9142\.91, slightly below MSP on this benchmark\. MSP is the only method in this comparison trained without real paired cross\-modal data, and it remains competitive with models that use it, further demonstrating the effectiveness of proxy\-based cross\-modal composition\.

### 4\.3Learning Modality Composition via Proxy

We analyze the model to understand how proxy supervision and the text\-to\-modality bridge contribute to learning cross\-modal composition\. Table[4](https://arxiv.org/html/2610.20965#S4.T4)compares no proxies and no bridge, proxies without a bridge, and full MSP with both proxies and bridge\. At the audio\-video setting, proxies are worth\+12\.70\+12\.70on AVQA and\+12\.85\+12\.85on MUSIC\-AVQA, and the bridge a further\+2\.67\+2\.67and\+7\.05\+7\.05\. This shows that proxies provide strong learning signals for modality composition, while the bridge further improves alignment\.

We also examine representation behavior across LLM layers in Table[5](https://arxiv.org/html/2610.20965#S4.T5)using both residual\-stream similarity and the*logit lens*\. While residual similarity decreases across layers, vocabulary\-space similarity recovers at the final layer and ends well above it \(Δ=\+0\.166\\Delta=\+0\.166audio,\+0\.124\+0\.124vision\)\. This indicates that proxies do not need to match real modality features in hidden space; instead, the model learns to map them to similar output predictions, enabling effective cross\-modal composition\.

Table 5:Paired residual and output\-layer cosine similarity\. Each proxy is matched to its nearest real counterpart by last\-layer residual cosine, and similarities are computed over those pairs\. Here,Δ=vocab cosLlast−residual cosLlast\\Delta=\\text\{vocab cos\}\_\{L\_\{\\text\{last\}\}\}\-\\text\{residual cos\}\_\{L\_\{\\text\{last\}\}\}\. These are not comparable to the trajectory cosines in Table[10](https://arxiv.org/html/2610.20965#A5.T10), which measure a different quantity\.We fit a logistic\-regression probe at the input, middle, and final layers of the LLM backbone to predict real vs\. bridged\-proxy from the hidden state at that layer \(n=400n\{=\}400per cell\)\. Real and bridged\-proxy features remain linearly separable at every layer for both audio and vision \(Table[6](https://arxiv.org/html/2610.20965#S4.T6)\); MSP does not erase the distinction in the residual stream\. This supports the interpretation that the LM\-head readout, rather than residual indistinguishability, explains the proxy–real alignment\.

Table 6:Real and proxy hidden states remain distinguishable\. A logistic\-regression probe predicts whether a hidden state came from a real feature or a bridged\-proxy feature at three layers;n=400n\{=\}400per cell\. Values are probe accuracies, so higher values mean real and proxy states are easier to separate\. The high accuracies show that MSP does not make the hidden states identical; Table[5](https://arxiv.org/html/2610.20965#S4.T5)tests whether the LM\-head readout is nevertheless aligned\.
Additional alignment analyses are provided in Appendix[E](https://arxiv.org/html/2610.20965#A5)\.

Phase 2 trains on QA\-formatted data, so part of the gain over Phase 1 comes from that supervision alone rather than from composition\. Separating the two \(Table[13](https://arxiv.org/html/2610.20965#A9.T13)\), we still find that MSP composition training with proxies enables significantly better performance than the QA supervision by itself\.

Table 7:Real modalities vs\. proxy modalities at inference and token\-matched controls\. Qwen3\-8B LoRAr=512r\{=\}512, test splits: AVQA \(n=16,910n\{=\}16\{,\}910\), MUSIC\-AVQA \(n=9,185n\{=\}9\{,\}185\)\.
Real modalities vs\. proxy modalities at inference\.We found across the board that real modalities outperform their proxy counterparts \(Table[7](https://arxiv.org/html/2610.20965#S4.T7)a\)\. Token count, position, attention mask, and decoding are unchanged\. Proxies for this ablation are generated via the same mechanism used during the training of the model\.

Token\-matched controls\.Table[7](https://arxiv.org/html/2610.20965#S4.T7)b varies the audio slot with real vision and the vision slot with real audio\. The key takeaway is that the model utilizes real modality content through these slots, and that only correct content recovers full performance\. Additional details are provided in Appendix[I](https://arxiv.org/html/2610.20965#A9)\. Question\-type breakdowns atr=512r\{=\}512are provided in Appendix[B](https://arxiv.org/html/2610.20965#A2)\.

Sensitivity to proxy quality\.As might be expected based on the token\-matched controls, the quality and content of the modality given to the model has a large impact\. While we do not have results for a model trained on degenerate proxies, the token\-matched controls indicate that the model is reading real modality content from the token slot, thus generating realistic proxies is also important for the downstream model to learn the desired cross\-modal relationships \(Appendix[K](https://arxiv.org/html/2610.20965#A11)\)\.

MUSIC\-AVQA\-R results are provided in Appendix[J](https://arxiv.org/html/2610.20965#A10)\.

## 5Limitations and Future Directions

While MSP demonstrates that proxy\-mediated alignment can enable multimodal composition without real paired non\-text supervision, it relies on the quality and semantic consistency of proxy textual descriptions\. Noisy or weakly aligned proxies can degrade model performance \(Appendix[K](https://arxiv.org/html/2610.20965#A11)\)\. Additionally, MSP builds on contrastively pretrained encoders \(e\.g\., CLIP/CLAP\-style models\) to provide cross\-modal structure, and therefore inherits their coverage limitations and potential biases\. Future work can address these limitations along several directions\. Improving proxy generation—through stronger generative models or retrieval\-based methods—could yield more accurate and semantically grounded descriptions\. Reducing the training–inference gap, either by better integrating proxy information at inference or by learning proxy\-free alignment mechanisms, is another promising avenue\. Extending MSP to a wider range of modalities and more complex reasoning tasks, as well as studying its behavior at larger scales, would further evaluate its generality\. Finally, developing a deeper theoretical understanding may help clarify when and why multimodal composition can emerge without real paired non\-text supervision\. Future work should focus on using MSP in paradigms with significantly more tokens and also determine how closely MSP compares with paired data ceilings \(or if they can be exceeded by using additional text proxies; Appendix[L](https://arxiv.org/html/2610.20965#A12)\)\.

Broader impacts\.This work improves multimodal reasoning without requiring real paired non\-text data, which can benefit data\-limited or privacy\-sensitive applications\. However, improved cross\-modal synthesis may also facilitate misuse such as multimodal deepfakes or fabricated content\. To mitigate these dual\-use risks, we advocate for the use of ethically sourced training data, model access control, and the adoption of provenance\-tracking techniques for any synthesized outputs\.

## 6Conclusion

In this work, we revisited a central assumption in multimodal learning that large\-scale paired multimodal data is necessary for learning cross\-modal reasoning\. We introducedModality Self\-Play \(MSP\), a training paradigm in which proxy textual descriptions serve as semantic bridges between observed and unseen modalities\. MSP removes the need for real paired non\-text \(e\.g\., audio\-vision\) supervision, while continuing to rely on modality\-text pairs, pretrained contrastive encoders, and modality\-distribution bridge calibration\. Our results show that proxy\-based supervision is both viable and effective\. Models trained with MSP demonstrate strong performance on standard multimodal benchmarks, while reducing reliance on costly data curation pipelines\. More broadly, our findings suggest that multimodal composition can emerge from structured interactions over independently learned representations mediated through proxy text, rather than from exhaustive aligned multimodal data\. We hope this work encourages the development of more scalable and data\-efficient approaches to multimodal learning based on principled alignment mechanisms\.

## Acknowledgments

This work is partially supported by a gift award from Amazon and an NSF award 2406199\.

## References

- \[1\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\)Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.7.1.1)\.
- \[2\]S\. Deshmukh, B\. Elizalde, D\. Emmanouilidou, B\. Raj, R\. Singh, and H\. Wang\(2024\)Training audio captioning models without audio\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p4.1)\.
- \[3\]C\. Fan, X\. Zhang, S\. Zhang, W\. Wang, C\. Zhang, and H\. Huang\(2019\)Heterogeneous memory enhanced multimodal attention model for video question answering\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 1999–2007\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.4.1.1)\.
- \[4\]C\. Fu, H\. Lin, Z\. Long, Y\. Shen, Y\. Dai, M\. Zhao, Y\. Zhang, S\. Dong, Y\. Li, X\. Wang,et al\.\(2024\)Vita: towards open\-source interactive omni multimodal llm\.arXiv preprint arXiv:2408\.05211\.Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.5.1.1)\.
- \[5\]C\. Fu, H\. Lin, X\. Wang, Y\. Zhang, Y\. Shen, X\. Liu,et al\.\(2025\)VITA\-1\.5: towards GPT\-4o level real\-time vision and speech interaction\.arXiv preprint arXiv:2501\.01957\.Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.6.1.1)\.
- \[6\]R\. Girdhar, A\. El\-Nouby, Z\. Liu, M\. Singh, K\. V\. Alwala, A\. Joulin, and I\. Misra\(2023\)ImageBind: one embedding space to bind them all\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p3.1)\.
- \[7\]Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. Parikh\(2017\)Making the V in VQA matter: elevating the role of image understanding in visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6904–6913\.Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[8\]J\. Han, K\. Gong, Y\. Zhang, J\. Wang, K\. Zhang, D\. Lin, Y\. Qiao, P\. Gao, and X\. Yue\(2024\)Onellm: one framework to align all modalities with language\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26584–26595\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p3.1),[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.6.1.1)\.
- \[9\]J\. Han, R\. Zhang, W\. Shao, P\. Gao, P\. Xu, H\. Xiao, K\. Zhang, C\. Liu, S\. Wen, Z\. Guo,et al\.\(2023\)Imagebind\-llm: multi\-modality instruction tuning\.arXiv preprint arXiv:2309\.03905\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p3.1),[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.2.1.1),[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.9.1.1)\.
- \[10\]D\. Huang, C\. Yan, Q\. Li, and X\. Peng\(2024\)From large language models to large multimodal models: a literature review\.Applied Sciences14\(12\),pp\. 5068\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p1.1)\.
- \[11\]A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.8.1.1)\.
- \[12\]P\. Jiang and Y\. Han\(2020\)Reasoning with heterogeneous graph alignment for video question answering\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 11109–11116\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.7.1.1)\.
- \[13\]C\. D\. Kim, B\. Kim, H\. Lee, and G\. Kim\(2019\)AudioCaps: generating captions for audios in the wild\.InNAACL\-HLT,Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p4.1),[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[14\]T\. M\. Le, V\. Le, S\. Venkatesh, and T\. Tran\(2020\)Hierarchical conditional relation networks for video question answering\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9972–9981\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.8.1.1)\.
- \[15\]G\. Li, Y\. Wei, Y\. Tian, C\. Xu, J\. Wen, and D\. Hu\(2022\)Learning to answer questions in dynamic audio\-visual scenarios\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p5.1)\.
- \[16\]J\. Li, D\. Li, S\. Savarese, and S\. Hoi\(2023\)BLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[§3\.1](https://arxiv.org/html/2610.20965#S3.SS1.p4.1)\.
- \[17\]W\. Li, L\. Zhu, L\. Wen, and Y\. Yang\(2023\)DeCap: decoding CLIP latents for zero\-shot captioning via text\-only training\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p4.1)\.
- \[18\]X\. Li, L\. Gao, X\. Wang, W\. Liu, X\. Xu, H\. T\. Shen, and J\. Song\(2019\)Learnable aggregating net with diversity learning for video question answering\.InProceedings of the 27th ACM international conference on multimedia,pp\. 1166–1174\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.5.1.1)\.
- \[19\]X\. Li, J\. Song, L\. Gao, X\. Liu, W\. Huang, X\. He, and C\. Gan\(2019\)Beyond rnns: positional self\-attention with co\-attention for video question answering\.InProceedings of the AAAI conference on artificial intelligence,Vol\.33,pp\. 8658–8665\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.2.1.1)\.
- \[20\]X\. Li, W\. Chen, Z\. Ma, X\. Xu, Y\. Liang, Z\. Zheng, Q\. Kong, and X\. Chen\(2025\)DRCap: decoding CLAP latents with retrieval\-augmented generation for zero\-shot audio captioning\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p4.1)\.
- \[21\]Y\. Li, J\. Liu, T\. Zhang, T\. Zhang, S\. Chen, T\. Li, Z\. Li, L\. Liu, L\. Ming, G\. Dong,et al\.\(2025\)Baichuan\-Omni\-1\.5 technical report\.arXiv preprint arXiv:2501\.15368\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p1.1),[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.12.1.1)\.
- \[22\]Y\. Li, H\. Sun, M\. Lin, T\. Li, G\. Dong, T\. Zhang, B\. Ding, W\. Song, Z\. Cheng, Y\. Huo,et al\.\(2024\)Baichuan\-Omni technical report\.arXiv preprint arXiv:2410\.08565\.Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.4.1.1)\.
- \[23\]Y\. Li, Y\. Ma, G\. Zhang, R\. Yuan, K\. Zhu, H\. Guo, Y\. Liang, J\. Liu, Z\. Wang, J\. Yang,et al\.\(2024\)Omnibench: towards the future of universal omni\-language models\.arXiv preprint arXiv:2409\.15272\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p5.1)\.
- \[24\]V\. W\. Liang, Y\. Zhang, Y\. Kwon, S\. Yeung, and J\. Y\. Zou\(2022\)Mind the gap: understanding the modality gap in multi\-modal contrastive representation learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p4.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p4.1)\.
- \[25\]S\. Lipping, P\. Sudarsanam, K\. Drossos, and T\. Virtanen\(2022\)Clotho\-AQA: a crowdsourced dataset for audio question answering\.In2022 30th European Signal Processing Conference \(EUSIPCO\),pp\. 1140–1144\.Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[26\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2024\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 26296–26306\.Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p4.1),[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[27\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[§3\.1](https://arxiv.org/html/2610.20965#S3.SS1.p4.1)\.
- \[28\]J\. Lu, C\. Clark, S\. Lee, Z\. Zhang, S\. Khosla, R\. Marten, D\. Hoiem, and A\. Kembhavi\(2024\)Unified\-io 2: scaling autoregressive multimodal models with vision language audio and action\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26439–26455\.Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.10.1.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.3.1.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.7.1.1)\.
- \[29\]R\. Luo, T\. Lin, H\. Zhang, Y\. Wu, X\. Liu, M\. Yang, and Y\. Li\(2025\)OpenOmni: advancing open\-source omnimodal large language models with progressive multimodal alignment and real\-time self\-aware emotional speech synthesis\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.9.1.1)\.
- \[30\]J\. Ma, M\. Hu, P\. Wang, W\. Sun, L\. Song, H\. Pei, J\. Liu, and Y\. Du\(2024\)Look, listen, and answer: overcoming biases for audio\-visual question answering\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/123a18dfd821c8b440f42a00a27648d6-Abstract-Conference.html)Cited by:[Appendix J](https://arxiv.org/html/2610.20965#A10.p1.1)\.
- \[31\]X\. Mei, C\. Meng, H\. Liu, Q\. Kong, T\. Ko, C\. Zhao, M\. D\. Plumbley, Y\. Zou, and W\. Wang\(2024\)Wavcaps: a chatgpt\-assisted weakly\-labelled audio captioning dataset for audio\-language multimodal research\.IEEE/ACM Transactions on Audio, Speech, and Language Processing32,pp\. 3339–3354\.Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[32\]D\. Nukrai, R\. Mokady, and A\. Globerson\(2022\)Text\-only training for image captioning using noise\-injected CLIP\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p4.1)\.
- \[33\]A\. Panagopoulou, L\. Xue, N\. Yu, J\. Li, D\. Li, S\. Joty, R\. Xu, S\. Savarese, C\. Xiong, and J\. C\. Niebles\(2023\)X\-instructblip: a framework for aligning x\-modal instruction\-aware representations to llms and emergent cross\-modal reasoning\.arXiv preprint arXiv:2311\.18799\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p3.1),[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.3.1.1),[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.4.1.1),[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.10.1.1)\.
- \[34\]Qwen Team\(2025\)Qwen2\.5\-Omni technical report\.arXiv preprint arXiv:2503\.20215\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p1.1),[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.14.1.1)\.
- \[35\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1](https://arxiv.org/html/2610.20965#S3.SS1.p4.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p6.1)\.
- \[36\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p3.1)\.
- \[37\]M\. K\. Reza, A\. Patil, E\. Ayrapetian, and M\. S\. Asif\(2026\)SSAM: singular subspace alignment for merging multimodal large language models\.arXiv preprint arXiv:2603\.21584\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.6.1.1)\.
- \[38\]Y\. Su, T\. Lan, H\. Li, J\. Xu, Y\. Wang, and D\. Cai\(2023\)Pandagpt: one model to instruction\-follow them all\.InProceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants\!,pp\. 11–23\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p3.1)\.
- \[39\]G\. Sun, W\. Yu, C\. Tang, X\. Chen, T\. Tan, W\. Li, L\. Lu, Z\. Ma, Y\. Wang, and C\. Zhang\(2024\)Video\-SALMONN: speech\-enhanced audio\-visual large language models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.8.1.1)\.
- \[40\]G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang,et al\.\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.13.1.1)\.
- \[41\]J\. Wang, M\. Yan, Y\. Zhang, and J\. Sang\(2023\)From association to generation: text\-only captioning by unsupervised cross\-modal mapping\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 4326–4334\.Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p4.1)\.
- \[42\]Z\. Wang, K\. Zhu, C\. Xu, W\. Zhou, J\. Liu, Y\. Zhang, J\. Wang, N\. Shi, S\. Li, Y\. Li,et al\.\(2024\)MIO: a foundation model on multimodal tokens\.arXiv preprint arXiv:2409\.17692\.Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1)\.
- \[43\]J\. Wu, W\. Gan, Z\. Chen, S\. Wan, and P\. S\. Yu\(2023\)Multimodal large language models: a survey\.In2023 IEEE International Conference on Big Data \(BigData\),pp\. 2247–2256\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p1.1)\.
- \[44\]S\. Wu, H\. Fei, L\. Qu, W\. Ji, and T\. Chua\(2024\)Next\-gpt: any\-to\-any multimodal llm\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p2.1)\.
- \[45\]Y\. Wu, K\. Chen, T\. Zhang, Y\. Hui, T\. Berg\-Kirkpatrick, and S\. Dubnov\(2023\)Large\-scale contrastive language\-audio pretraining with feature fusion and keyword\-to\-caption augmentation\.InICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.20965#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p6.1)\.
- \[46\]D\. Xu, Z\. Zhao, J\. Xiao, F\. Wu, H\. Zhang, X\. He, and Y\. Zhuang\(2017\)Video question answering via gradually refined attention over appearance and motion\.InProceedings of the 25th ACM International Conference on Multimedia \(MM\),pp\. 1645–1653\.Cited by:[§3\.3](https://arxiv.org/html/2610.20965#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p2.1)\.
- \[47\]H\. Xu, S\. Xie, X\. E\. Tan, P\. Huang, R\. Howes, V\. Sharma, S\. Li, G\. Ghosh, L\. Zettlemoyer, and C\. Feichtenhofer\(2024\)Demystifying CLIP data\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.1](https://arxiv.org/html/2610.20965#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p6.1)\.
- \[48\]P\. Yang, X\. Wang, X\. Duan, H\. Chen, R\. Hou, C\. Jin, and W\. Zhu\(2022\)AVQA: a dataset for audio\-visual question answering on videos\.InProceedings of the 30th ACM International Conference on Multimedia \(MM\),Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.20965#S4.SS1.p5.1)\.
- \[49\]Y\. Yao, T\. Yu, A\. Zhang, C\. Wang, J\. Cui, H\. Zhu, T\. Cai, H\. Li, W\. Zhao, Z\. He,et al\.\(2024\)MiniCPM\-v: a gpt\-4v level mllm on your phone\.arXiv preprint arXiv:2408\.01800\.Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.11.1.1)\.
- \[50\]J\. Zhan, J\. Dai, J\. Ye, Y\. Zhou, D\. Zhang, Z\. Liu, X\. Zhang, R\. Yuan, G\. Zhang, L\. Li,et al\.\(2024\)Anygpt: unified multimodal llm with discrete sequence modeling\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9637–9662\.Cited by:[§2](https://arxiv.org/html/2610.20965#S2.p2.1),[Table 3](https://arxiv.org/html/2610.20965#S4.T3.4.2.1.1)\.
- \[51\]J\. Zhang, J\. Shao, R\. Cao, L\. Gao, X\. Xu, and H\. T\. Shen\(2020\)Action\-centric relation transformer network for video question answering\.IEEE Transactions on Circuits and Systems for Video Technology32\(1\),pp\. 63–74\.Cited by:[Table 2](https://arxiv.org/html/2610.20965#S4.T2.4.3.1.1)\.
- \[52\]Z\. Zhao, L\. Guo, T\. Yue, S\. Chen, S\. Shao, X\. Zhu, Z\. Yuan, and J\. Liu\(2023\)ChatBridge: bridging modalities with large language model as a language catalyst\.arXiv preprint arXiv:2305\.16103\.Cited by:[§1](https://arxiv.org/html/2610.20965#S1.p2.1),[§2](https://arxiv.org/html/2610.20965#S2.p3.1),[Table 1](https://arxiv.org/html/2610.20965#S4.T1.4.5.1.1)\.

## Appendix ADetailed architecture diagram

Figure[2](https://arxiv.org/html/2610.20965#A1.F2)gives the component\-level view of MSP referred to in Section[3\.1](https://arxiv.org/html/2610.20965#S3.SS1)\. It shows the frozen contrastive encoders and their text towers, the trainable per\-modality projectors, the frozen text\-to\-modality bridge used on the proxy path in Phase 2, the reserved token slots in the prompt, and the Qwen3\-8B backbone with its LoRA adapters\.

![Refer to caption](https://arxiv.org/html/2610.20965v1/paper_sections/figures/fixed_textfuse.png)Figure 2:MSP architecture, component\-level view\. MSP uses frozen audio and vision encoders with their associated text encoders, trainable per\-modality projectors, a frozen text\-to\-modality bridge for Phase 2, and a Qwen3\-8B language model with LoRA adapters\.
## Appendix BPer\-question\-type breakdowns

The single\-token model does not collapse: AVQA*Come From*\(sound localization\)90\.990\.9and*Where*\(spatial\)80\.380\.3; MUSIC\-AVQA*Counting*72\.672\.6and*Location*57\.257\.2\(recomputed using ther=512r\{=\}512checkpoint; Table[8](https://arxiv.org/html/2610.20965#A2.T8)\)\.

Table 8:Reported question\-type and overall modality results atr=512r\{=\}512\. The question\-type block reports the four categories recomputed for this checkpoint\.

## Appendix CSelf\-play prompt templates

Phase 2 proxies are generated by the Phase\-1 checkpoint conditioned on the real anchor modality\. The prompt template controls which target encoder distribution the generated text is meant to match; we use encoder\-native terse prompts that mirror the caption style each contrastive encoder was trained on \(Appendix[H](https://arxiv.org/html/2610.20965#A8)\)\. The verbatim templates used throughout this paper are reproduced below\.

#### Audio proxy from a vision anchor \(CLAP\-targeting\)\.

> <vision\> Describe the sounds in this scene as a single AudioCaps\-style caption: ONE short sentence, 8–15 words, present tense, naming 1–3 specific sound sources directly\. Do not use multiple sentences and do not describe pitch, timbre, or volume\. Do NOT use: image, picture, photo, visible, scene, look, see, shown, view, visually\. Examples: “A man is speaking while water is flowing\.” “A train engine is running and blowing its horn\.” “Music is playing as a vehicle passes by\.” “An animal is chewing loudly in the background\.” “Wind is blowing and leaves are rustling\.” Write ONE such sentence\. No other text\.

#### Vision proxy from an audio anchor \(MetaCLIP\-targeting\)\.

> <audio\> Write a single COCO\-style image caption describing what the source of the sound would look like as a photograph\. Use one short sentence, 8–18 words, present tense\. Do NOT use any of these words or concepts: sound, audio, noise, hear, hearing, listen, volume, pitch, timbre, loud, quiet, rhythmic, echo\. Name the main object, person, or animal and the setting\. Mention visible attributes only\. Examples: “A dog stands on grass with its mouth open\.” “A man sits indoors beside a running kitchen sink\.” “A train passes along tracks near a platform\.” Write ONE such sentence\. No other text\.

Both templates share a common structure: one paragraph naming the target encoder’s caption\-style family, one paragraph of negative\-word constraints to prevent leakage from the source modality’s vocabulary, and a short positive instruction listing the content axes the target caption distribution covers\.

## Appendix DOptimization

Both Phase 1 and Phase 2 use AdamW with bfloat16 mixed precision, peak learning rate5×10−55\{\\times\}10^\{\-5\}, cosine decay, one epoch per phase, and seed4242\. Effective batch size is1616\(micro\-batch22×\\timesgradient\-accumulation88\) for LoRAr=128r\{=\}128and1616\(micro\-batch11×\\timesgradient\-accumulation1616\) forr∈\{256,512\}r\\in\\\{256,512\\\}\. Each reported LoRA run uses DeepSpeed ZeRO\-2 on one NVIDIA H100 \(80 GB\)\.

#### Phase 1 schedule\.

For the first2,0002\{,\}000optimizer steps the LLM is frozen and only the modality projectors are updated; afterward the LoRA parameters are unfrozen\. Each Phase 1 batch carries real features from exactly one modality, and audio and vision batches alternate so the update parameters see both pathways within each rotation\. The Phase\-1 checkpoint is then used both as the initialization for Phase 2 and as the generator that produces the cached self\-play proxies\.

#### Phase 2 schedule\.

Phase 2 continues from the Phase\-1 checkpoint with frozen bridges and the cached self\-play proxies; only the modality projectors and the chosen LLM update parameters are trained\. Dataset sources for both phases are listed in Section[4\.1](https://arxiv.org/html/2610.20965#S4.SS1)\.

One additional note is that for all models, modality dropout is applied during training so that absent modalities are not treated as malformed input\.

## Appendix EDetailed mechanism probes

To analyze the effect of bridge on modality alignment, we calculate the cosine similarity between proxy and real modality features\. As shown in Table[9](https://arxiv.org/html/2610.20965#A5.T9), alignment improves substantially after applying the bridge, increasing similarity from0\.2180\.218–0\.4220\.422in the raw encoder space to0\.5480\.548–0\.8230\.823after bridging \(a1\.8×1\.8\\times–2\.6×2\.6\\timesincrease\), with additional smaller gains from the projector\.

Table 9:Bridge and projector progressively align proxy with real\-modality features\.Table[10](https://arxiv.org/html/2610.20965#A5.T10)reports the companion surrogate\-input control used in Section[4\.3](https://arxiv.org/html/2610.20965#S4.SS3)\. Zero and Gaussian features reduce placeholder engagement, while valid modality features remain closer to the real\-feature trajectory\. The cosines in this table are trajectory similarities against the unmodified real\-input run on the same example, whereas Table[5](https://arxiv.org/html/2610.20965#S4.T5)reports similarity between nearest\-matched proxy–real pairs\. The two quantities are measured differently and their magnitudes are not comparable\.

Table 10:Valid features engage the placeholder\. We compare real features with surrogate inputs using the last\-layer residual cosine of the whole hidden\-state trajectory against the unmodified real\-input run on the same example \(n=200n\{=\}200pairs\)\. This is a different quantity from the nearest\-pair cosine in Table[5](https://arxiv.org/html/2610.20965#S4.T5)and the two should not be compared directly\. Higher is closer to the real trajectory\. Zero and Gaussian inputs reduce engagement; valid features remain closer\.

## Appendix FEncoder selection

Our initial hypothesis when selecting encoders for our final model was that whichever encoders had the highest cosine similarity between text proxies passing through their text encoder and their modality counterparts would be the most useful for MSP\. We calculated the mean cosine similarity between all COCO images and their text captions \(Table[11](https://arxiv.org/html/2610.20965#A6.T11)\)\.

Table 11:Mean cosine similarity between all COCO images and their text captions, used to inform encoder selection\.Cosine similarity is not the only determinant of success with a given contrastive encoder, but we believe that it does matter for effectiveness of MSP training\. One note is that sigmoid\-loss models produce lower raw cosine values by design \(SigLIP\)\. For our final model that we report in the paper we utilized MetaCLIP due to its strong text\-vision alignment and highest starting cosine similarity\. This process was also repeated with a selection of audio encoders to inform our final model architecture\.

## Appendix GAnchor\-supporting proxy examples

Figure 3:Examples of anchor\-supporting proxy generation\. Each panel shows a real anchor in one modality and the proxy text generated for the other modality\. The held\-out ground truth is shown only for context; it is not used to generate the proxy\.The qualitative figure \(Figure[3](https://arxiv.org/html/2610.20965#A7.F3)\) shows faithful proxy generations, where the Phase\-1 model correctly describes the held\-out modality\. The two examples below \(Figure[4](https://arxiv.org/html/2610.20965#A7.F4)\) illustrate a more nuanced case: the proxy is not literally faithful to the held\-out scene, but remains consistent with the real anchor\. MSP can still benefit from these examples because the training signal is anchor consistency rather than literal reconstruction of the held\-out modality, matching the shuffled\-input result in Section[4\.3](https://arxiv.org/html/2610.20965#S4.SS3)\.

Figure 4:Anchor\-supporting proxy examples\. Generated vision proxies can differ from the held\-out scene while remaining compatible with the real audio anchor\.
## Appendix HBridge variant comparison

Two calibration\-time choices in Section[3](https://arxiv.org/html/2610.20965#S3)are validated by ablation\. The linear ridge bridge family beats identity, mean\-shift, and residual\-MLP variants on held\-out residual on both audio and vision\. Earlier co\-trained bridge variants collapsed toward the identity under the QA loss, so the linear ridge is fit offline and frozen\. Encoder\-native terse prompt templates beat literary 4\-sentence templates by\+0\.39\+0\.39post\-bridge cosine on audio \(CLAP\) and\+0\.28\+0\.28on vision \(MetaCLIP\), so we adopt the terse templates throughout Phase 2\.

The deployed bridge of Section[3\.3](https://arxiv.org/html/2610.20965#S3.SS3)is a ridge\-regularized linear map \(Eq\.[3](https://arxiv.org/html/2610.20965#S3.E3)\)\. For completeness, Table[12](https://arxiv.org/html/2610.20965#A8.T12)compares four bridge families on held\-out residualεm′=𝔼​‖Bm′​Em′text​\(c\)−Em′modality​\(o\)‖2\\varepsilon\_\{m^\{\\prime\}\}=\\mathbb\{E\}\\\|B\_\{m^\{\\prime\}\}E\_\{m^\{\\prime\}\}^\{\\text\{text\}\}\(c\)\-E\_\{m^\{\\prime\}\}^\{\\text\{modality\}\}\(o\)\\\|\_\{2\}\. The linear ridge family achieves the smallest residual on both audio and vision\.

Table 12:Bridge calibration residuals\.Held\-out bridge error from text\-tower to modality\-encoder space; lower is better\. The linear bridge gives the lowest residual for audio and vision\.

## Appendix IAdditional composition controls

All numbers below are from the reported Qwen3\-8B LoRA model atr=512r\{=\}512, on the test splits of the given benchmarks\.

#### Training baselines\.

In Table[4](https://arxiv.org/html/2610.20965#S4.T4), the “without proxy and bridge” row omits the non\-anchor modality entirely, meaning that no token is given, and it is not zeroed or replaced by a learned null\. The “without bridge” row keeps the same proxies, but injects them without the learned text\-modality mapping the bridge contributes\.

Table 13:QA supervision and MSP composition\. All numbers are atr=512r\{=\}512on the test splits\.StageAVQAMUSIC\-AVQAPhase 1 only \(per\-modality projectors\)70\.2145\.75\+ QA supervision only, no proxies, no bridge77\.4655\.18\+ full MSP composition85\.5865\.65QA\-supervision contribution\+7\.25\+9\.43composition contribution\+8\.12\+10\.47
#### Real modalities vs\. proxy modalities at inference\.

We observe that for both benchmarks, highest accuracy is obtained when both modality inputs are real modality samples\. Additionally, we see consistent decreases in performance when a single modality is swapped for a proxy, and an even larger decrease when both modality inputs are swapped for generated proxies\. All six drops are significant \(allp<10−6p<10^\{\-6\}, paired McNemar test over per\-sample predictions from a fixed model\)\. These results are consistent with our view that proxies function as a useful training\-time scaffold for learning cross\-modal structure, but do not carry the same fidelity of information as their real counterparts\.

#### Token\-matched controls\.

The Gaussian is matched to the real features’ per\-dimension mean and standard deviation, and shuffled is a genuine encoder output \(just one taken from a different test example\)\. Notably, real examples with the wrong features cost the most on both controls, which suggests that a real sample from the same distribution can actively mislead a model when its content does not help answer the given query\.

## Appendix JMUSIC\-AVQA\-R

As an additional control, we tested our model against MUSIC\-AVQA\-R\[[30](https://arxiv.org/html/2610.20965#bib.bib51)\], which is the debiased re\-split of MUSIC\-AVQA that contains a tail reweighted toward rarer answers, with the specific goal of defeating answer\-prior exploitation\. On the full211,572211\{,\}572\-question test set we achieved the following \(Table[14](https://arxiv.org/html/2610.20965#A10.T14)\):

Table 14:MUSIC\-AVQA\-R results on the full211,572211\{,\}572\-question test set\.We believe that this result shows additional evidence that MSP learns to use cross\-modal structure effectively and can perform well even when compared against models that use direct supervision\.

## Appendix KSensitivity to proxy quality at test time

We compare the model at its best \(all real inputs\) with proxies at two levels of quality: \(1\) scene\-grounded proxies are proxies generated from the Phase 1 model by having it observe an anchor modality and imagine what the counterpart modality might be, and \(2\) a question\-derived proxy with no accompanying modality input\. A question\-derived proxy is generated by asking the model for a possible audio proxy based on the given question’s prompt \(Table[15](https://arxiv.org/html/2610.20965#A11.T15)\)\.

Table 15:Audio proxy quality at test time with real vision\.For example:

> Question:“What’s driving in the video?” Audio:<proxy\>A car engine roars as tires screech and glass shatters</proxy\> Vision:<real\>A blue car drives past a red truck on a sunny highway</real\>

From the token\-matched controls, the table above, and this example \(generated by the 8Br=512r\{=\}512model\) it is clear why weaker proxies might not function as well as stronger counterparts\. In this example there is clearly conflicting information—the audio paints the picture of a car racing and creating some kind of accident, while the vision input specifies a more peaceful highway scene\.

This also implies that a multimodal model may not always be necessary for generating high\-quality proxies if the generator has a decent enough signal \(e\.g\., detailed captions as input\) and sufficient multimodal priors learned from text\. This is a future direction worth exploring\.

## Appendix LPaired data at a smaller scale

For this control we took a text\-only Qwen3 4B model and finetuned it on AVQA across 5 training paradigms: \(1\) paired data only, \(2\) 1 anchor \+ 1 proxy \(general MSP recipe, alternating the anchor modality\), \(3\) interleaved unimodal \(1 modality per sample, other slot is left empty\), \(4\) vision only data, and \(5\) audio only data \(Table[16](https://arxiv.org/html/2610.20965#A12.T16)\)\.

Table 16:Paired data at a smaller scale: a text\-only Qwen3 4B model finetuned on AVQA\.While this does not determine the upper bound of our headliner=512r\{=\}512model, it does indicate MSP is a strong alternative to fully paired data\. The important takeaway is not just the gap between fully paired and MSP \(which is relatively small\), but the gap between MSP and interleaved unimodal training\. This shows that when fully paired data is unavailable, there is an option beyond leaving absent modality slots empty\.