Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

# Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
Source: [https://arxiv.org/html/2607.18666](https://arxiv.org/html/2607.18666)
Abdul Basit Tonmoy1,2,3Kazi Fardinul Hoque2Md\. Shahrier Islam Arham1,2Arman Luthra1,2 1Eximius Labs2Wabash College3Skop Intelligence Co\. atonmoy27@wabash\.edukfhfar@amazon\.com marham27@wabash\.edualuthra26@wabash\.edu

###### Abstract

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose\. Embedding models built on vision–language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio–text retrieval is led by specialist systems that serve no other modality\. We present the Fusion Embedding family, which adds audio to a frozen vision–language embedding base whose parameters are never updated: generation 1 \(fusion\-embedding\-1\) trains only a 16\.4M\-parameter connector between a frozen audio tower and the frozen base, and generation 2 \(fusion\-embedding\-2\) adds modality\-gated deep adapters \(44\.2M parameters\) whose branch never executes on text, image, or video inputs: their outputs are bit\-for\-bit those of the released base, verified after every training run\. Because the base already binds text, images, and video, aligning audio to text alone makes audio–image retrieval*emerge*, with zero paired audio–visual training data\. Alongside the recipe we map its design space with controlled negative results \(rewriting training captions with an LLM, substituting a leaderboard\-stronger audio tower, and widening the connector each*reduce*retrieval\) and with training\-protocol findings that we expect to transfer to any frozen decoder\-LM embedding backbone\. Both generations train in hours on a single GPU\. Weights, code, and the evaluation harness are openly released\.

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

## 1Introduction

Retrieval systems increasingly need a*single*vector space covering every modality a user might store or query: text, images, video, and audio\. Contrastive dual encoders trained per modality pair, such as CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2607.18666#bib.bib25)\)for vision–language and the CLAP family for audio–language\(Wu et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib34); Mei et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib21); Zhu and Duan,[2024](https://arxiv.org/html/2607.18666#bib.bib39); Niizumi et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib23)\), achieve strong in\-pair retrieval but each model spans only two modalities\. Unified\-space models such as ImageBind\(Girdhar et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib11)\)bind many modalities to a pivot modality, demonstrating that alignment between two modalities can emerge through a shared third\. Meanwhile, embedding models built on large vision–language backbones now lead text/image/video retrieval benchmarks such as MMEB\-V2\(Jiang et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib15); Meng et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib22)\), but lack audio entirely\.

Two routes to adding the missing modality are emerging, and they trade different things\.*Audio\-native specialist systems*adapt a multimodal LLM from the inside: OEA\(Yoo et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib37)\)trains LoRA adapters in the backbone attention of an audio\-native LLM, and AuroLA\(Xu et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib35)\)trains 418M LoRA parameters inside both the audio tower and the LLM of Qwen2\.5\-Omni\-7B, adding a second 7B model as a re\-ranker\. These systems set the pace on audio–text benchmarks, and they serve exactly one modality pair, with backbones that are no longer the models their ecosystems shipped\.*Frozen\-base fusion*, the route taken here, holds the deployed model fixed and attaches the new modality from the outside\. The trade is capability for invariance: every embedding a deployment has already computed and indexed with the base model remains exactly valid, and the base’s published text/image/video benchmark results carry over unchanged; not approximately, but by construction\.

This report describes the two generations of the Fusion Embedding family,[fusion\-embedding\-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)and[fusion\-embedding\-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview), both built on a byte\-frozen Qwen3\-VL\-Embedding\-2B\(Li et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib19)\)base and the frozen audio tower of Qwen2\.5\-Omni\(Xu et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib36)\)\. Generation 1 establishes the space: a 16\.4M\-parameter perceiver\-resampler connector, trained contrastively against cached frozen\-text targets, is enough to make audio a first\-class modality: AudioCaps audio\-to\-text R@10 0\.741 with both towers frozen, Clotho transfer strictly zero\-shot, and audio\-to\-image retrieval at 29×\\timeschance with no audio–visual supervision\. Generation 2 asks where the remaining headroom lives\. Three controlled negative results \(§[8](https://arxiv.org/html/2607.18666#S8)\) locate it in the frozen LM’s ability to*process*audio tokens, not in the data or the connector;fusion\-embedding\-2therefore adds bottleneck adapters at every decoder layer, hard\-gated to audio inputs so that non\-audio computation is bitwise untouched\. The adapters improve every text\-to\-audio benchmark \(AudioCaps R@10 0\.775\) while the invariance guarantee holds exactly\.

Our contributions:

- •*System\.*A two\-generation recipe for extending a frozen vision–language embedding base with audio: a 16\.4M connector \(generation 1\), then modality\-gated deep adapters \(\+44\.2M, generation 2\) whose gate guarantees that non\-audio inputs execute the base’s computation unmodified \(§[3\.3](https://arxiv.org/html/2607.18666#S3.SS3)\)\. Connector\-only systems preserve the base structurally; the gate extends the same exactness to*in\-layer*capacity, where prior methods only mitigate drift: unmerged LoRA preserves base*parameters*but executes on the text path, and zero\-initialized expansion trains away from identity\.
- •*Emergent Cross\-Modal Retrieval\.*With zero audio–image training pairs, audio\-to\-image retrieval on VGGSound\-696 reaches R@10 0\.418 \(29×\\timeschance; 0\.443 for the strongest generation\-2 checkpoint, under a different readout template; see Table[6](https://arxiv.org/html/2607.18666#S6.T6)\)\. Head\-to\-head under one protocol, each of ImageBind andfusion\-embedding\-1wins its supervised pair andfusion\-embedding\-1wins the two remaining pairs;fusion\-embedding\-1beats LanguageBind on all three pairs including LanguageBind’s supervised one, and we trace that failure to measurable divergence of its branch text towers \(§[7](https://arxiv.org/html/2607.18666#S7)\)\.
- •*Negative results\.*Controlled experiments in which LLM\-rewritten training captions*reduce*retrieval despite being cleaner; a sound\-event audio tower that dominates shallow\-projection comparisons loses by 16 R@10 points inside a frozen\-LLM splice; and wider connectors lose by memorizing\. Together with a \+14\.5\-point training\-protocol effect and a loss\-floor audit, these map the design space of frozen\-backbone audio fusion \(§[8](https://arxiv.org/html/2607.18666#S8)\)\.
- •*Release\.*Open weights for both generations with pinned revision tags, Apache\-2\.0 training and evaluation code, and result records for every reported number \(§[10](https://arxiv.org/html/2607.18666#S10)\)\.

## 2Related Work

#### Contrastive audio–text pretraining \(CLAP family\)\.

LAION\-CLAP\(Wu et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib34)\), WavCaps\(Mei et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib21)\), Cacophony\(Zhu and Duan,[2024](https://arxiv.org/html/2607.18666#bib.bib39)\), and M2D\-CLAP\(Niizumi et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib23)\)train audio and text encoders end\-to\-end on hundreds of thousands of audio–caption pairs and set the AudioCaps retrieval state of the art among dual encoders \(audio\-to\-text R@10 0\.906–0\.928\)\. These models fine\-tune both towers and include in\-domain AudioCaps training; M2D\-CLAP additionally includes Clotho in its training data, so its Clotho rows are not zero\-shot\. WavCaps also demonstrated the value of large LLM\-assisted caption corpora, and AudioSetCaps\(Bai et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib2)\)scaled this recipe to 1\.9M pairs; notably, its authors report that in\-domain AudioCaps fine\-tuning after pretraining is worth \+13–15 R@1 points, and that a 478K high\-quality\-caption subset outperforms a similarly sized mix of WavCaps\+AudioCaps\+Clotho; caption quality wins at equal scale\. GLAP\(Dinkel et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib6)\)compares five frozen audio encoders under one shared contrastive recipe and selects Dasheng\(Dinkel et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib7)\)as the most versatile, with Whisper\-family encoders trailing sound\-event encoders by∼\\sim9–12 mAP@10 on sound retrieval; our tower\-swap experiment shows this ranking does*not*transfer to frozen\-LLM splicing \(§[8](https://arxiv.org/html/2607.18666#S8)\)\. On the data side, AudioCaps 2\.0\(Kim et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib17)\)\(91,256 train / 4,875 test pairs, audio distribution gated behind a request process\) doubles the in\-domain pool; we verified its train split is disjoint from the v1 test protocol we evaluate on \(0 of our 883 evaluation clips appear in its train CSV\), making it a natural extension of our fine\-tuning stage\. Our models differ from this entire family in that*neither*tower is trained\.

#### Audio\-native specialist systems\.

Recent work builds audio–text retrieval on multimodal LLM backbones\. OEA\(Yoo et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib37)\)is the closest system to generation 1 in trainable\-parameter budget: it keeps an audio\-native multimodal LLM backbone frozen and trains LoRA adapters on the attention layers plus projection heads \(11–16M trainable parameters against our 16\.4M connector\), reaching AudioCaps text\-to\-audio R@1 0\.389\. The architectural difference is decisive for our setting: OEA’s LoRA sits*inside*the backbone attention, so the text pathway is adapted too and the shared backbone is no longer byte\-identical to its release, which is exactly the property our design exists to preserve\. Two further observations from OEA inform our analysis: its 7B backbone performs on par with its 3B on the audio benchmarks, and its Clotho\-trained variants are not zero\-shot on Clotho\. AuroLA\(Xu et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib35)\)trains LoRA in both towers of Qwen2\.5\-Omni\-7B \(418M trainable parameters\) on a curated 1\.4M\-pair corpus and adds a second 7B model for pairwise re\-ranking, reaching AudioCaps R@1 0\.656 \(A→\\toT\) / 0\.510 \(T→\\toA\) as a two\-pass system; its scaling ablation indicates that large gains lie exactly in the corpus range above ours\. Its Hybrid\-NCE loss \(soft positive sets with weighted negatives\) is convergent evidence for the soft\-label and false\-negative\-masking terms we adopt at large corpus scale \(§[4\.1](https://arxiv.org/html/2607.18666#S4.SS1)\)\. Both lines validate LLM backbones for audio retrieval while adapting far more than the audio path, and neither serves, nor evaluates, any modality beyond audio and text\.

#### Unified embedding spaces\.

ImageBind\(Girdhar et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib11)\)binds six modalities to images and established that cross\-modal alignment emerges between modalities never paired in training\. We use the same emergence mechanism with text as the pivot, but in the opposite regime: instead of training modality towers to bind to a pivot encoder, we freeze an existing multi\-modality base and train only the new modality’s path, which preserves the base’s benchmark performance exactly\. LanguageBind\(Zhu et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib38)\)binds modality branches to language by fine\-tuning per\-branch copies of a text tower, a design whose consequences we measure in §[7](https://arxiv.org/html/2607.18666#S7)\. ONE\-PEACE\(Wang et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib30)\)trains a 4B three\-modality \(vision/audio/language\) model from scratch and remains the strongest unified space on audio–text retrieval, with every parameter trained and its retrieval rows fine\-tuned on the merged AudioCaps\+Clotho\+MACS train splits \(Table[4](https://arxiv.org/html/2607.18666#S6.T4)\)\. A separate*space\-stitching*line \(OmniBind,Wang et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib32); FreeBind,Wang et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib31)\) binds many pre\-trained specialist spaces with small learned projectors and routers; these systems report AudioCaps/Clotho R@1/R@5 only \(retrieval direction unstated\), and the bound experts include checkpoints trained on AudioCaps/Clotho train data, so their “zero\-shot” applies to the stitching stage rather than the underlying spaces\. Matryoshka representation learning\(Kusupati et al\.,[2022](https://arxiv.org/html/2607.18666#bib.bib18)\)provides the nested\-truncation property we preserve end\-to\-end; our contrastive objective is applied at every rung of the base’s ladder, and our whitening is deliberately diagonal so truncation and whitening commute\.

#### Positioning\.

Concurrent with jina\-embeddings\-v5\-omni\(Hönicke et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib14)\), which composes a frozen*text*embedding backbone with frozen vision and audio encoders and trains only connecting components,fusion\-embedding\-1is, to our knowledge, the first to extend an already*vision–language*embedding model \(Qwen3\-VL\-Embedding\-2B\) to audio with both the base and the audio tower frozen and only a connector trained: a fourth modality added to an existing retriever rather than a multimodal space built from a text base\.fusion\-embedding\-2adds audio capacity*inside*the frozen backbone’s layers, gated by modality\. Unlike gated cross\-attention\(Alayrac et al\.,[2022](https://arxiv.org/html/2607.18666#bib.bib1)\)or block expansion\(Wu et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib33)\), which preserve the original behavior only approximately or only at initialization, the gate makes the text, image, and video outputs*bitwise\-identical*to the frozen base—a guarantee we verify on the released checkpoint, and one we are not aware of any prior multimodal embedding model providing across several preserved modalities at once\. On AudioCaps audio\-to\-text R@10,fusion\-embedding\-1andfusion\-embedding\-2\(∼0\.74\{\\sim\}0\.74\) are the frontier of the both\-towers\-frozen class; the next such system is jina\-embeddings\-v5\-omni at 0\.672, and every model above them trains at least one tower\. ONE\-PEACE reaches 0\.92, but in a different class: all 4B parameters trained, with in\-domain fine\-tuning\. The audio\-native specialist systems above sit well higher on in\-domain audio–text R@1 \(§[6\.2](https://arxiv.org/html/2607.18666#S6.SS2)\); they are single\-pair systems, and the comparison across system classes is drawn explicitly in Table[4](https://arxiv.org/html/2607.18666#S6.T4)\.

## 3The Model Family

![Refer to caption](https://arxiv.org/html/2607.18666v1/x1.png)Figure 1:The Fusion Embedding family\. Both generations share the frozen stack: a byte\-frozen Qwen3\-VL\-Embedding\-2B base \(last\-token pooling, Matryoshka output ladder\) and the frozen Qwen2\.5\-Omni audio tower\. Generation 1 \(fusion\-embedding\-1\) trains only the FusionResampler \(16\.4M parameters, width 384, 64 latent queries\), which maps audio\-tower frames into the base’s input embedding space at placeholder token positions\. Generation 2 \(fusion\-embedding\-2\) keeps that connector and adds a bottleneck adapter to the residual stream of each of the 28 frozen decoder layers \(44\.2M parameters\), gated so the adapter branch executes only while audio is being encoded: text, image, and video inputs take the unmodified frozen path and their outputs are bit\-for\-bit identical to the released base\. Training updates only the orange components; every run assertsbase\_drift=0=0afterwards\.Figure[1](https://arxiv.org/html/2607.18666#S3.F1)shows the family: two frozen towers, a trained connector \(generation 1\), gated per\-layer adapters \(generation 2\), and the base’s own read\-out\. Table[1](https://arxiv.org/html/2607.18666#S3.T1)summarizes the released models\.

Table 1:The released family at a glance \(Gen\. 1==fusion\-embedding\-1, Gen\. 2==fusion\-embedding\-2\)\. “Bitwise base invariance”: inputs containing no audio execute the unmodified base computation, so their outputs are bit\-for\-bit those of the released base model under matched execution \(§[3\.3](https://arxiv.org/html/2607.18666#S3.SS3)\)\. Both models keep the base’s Matryoshka ladder and instruction interface\.### 3\.1Generation 1: a connector and nothing else

#### Frozen components\.

The base is Qwen3\-VL\-Embedding\-2B\(Li et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib19)\): a decoder\-LM embedding model with hidden widthdllm=2048d\_\{\\mathrm\{llm\}\}=2048, last\-token \(EOS\) pooling, instruction conditioning via a chat template, and a Matryoshka ladder\{2048,1536,1024,512,256,128,64\}\\\{2048,1536,1024,512,256,128,64\\\}\. The audio tower is the audio encoder of Qwen2\.5\-Omni\-7B\(Xu et al\.,[2025](https://arxiv.org/html/2607.18666#bib.bib36)\), a Whisper\-large\-v3\-derived\(Radford et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib26)\)encoder consuming 128\-bin log\-mel spectrograms of 30 s windows at 16 kHz\. We tap the tower’s*post\-projection*output: packed frames of dimension 3584 at∼\\sim25 frames per second; an early A/B confirmed this tap outperforms the 1280\-d pre\-projection encoder states\. Both components are frozen: they are set to evaluation mode, gradients are disabled, and a regression guard \(§[3\.3](https://arxiv.org/html/2607.18666#S3.SS3)\) verifies at the end of every run that no base parameter changed\.

#### FusionResampler \(the only trained module\)\.

The connector is a Flamingo\-style perceiver resampler\(Alayrac et al\.,[2022](https://arxiv.org/html/2607.18666#bib.bib1)\)operating at an internal bottleneck widthdr=384d\_\{r\}=384: an input projection3584→3843584\\to 384with sinusoidal temporal positions,N=64N=64learnable latent queries processed byL=6L=6pre\-norm blocks \(latent self\-attention→\\tocross\-attention over the audio frames with key\-padding masks→\\to4×\\timesFFN\), and an output projection384→2048384\\to 2048followed by LayerNorm\. This totals 16\.4M trained parameters, under 1% of the 2B base\. A learnable temperature \(initialized tolog⁡\(1/0\.07\)\\log\(1/0\.07\), clamped atlog⁡100\\log 100\) is the only other trained scalar\.

#### Injection at placeholder positions\.

The connector’sNNoutput tokens overwrite the input embeddings ofNNplaceholder tokens in the base’s input stream, an inert special token of the base vocabulary reserved for this purpose, exactly mirroring the base’s own image\-token mechanism\. The frozen LLM then runs its ordinary forward pass over the spliced sequence, and the embedding is read out by the base’s own last\-token pooling, truncated to a Matryoshka rung, and L2\-normalized\. Audio thus conforms to the base’s read\-out conventions in every respect; nothing about the base’s text, image, or video paths is altered\.

### 3\.2Generation 2: modality\-gated deep adapters

Generation 1 routes all audio understanding through a 16\.4M input\-side connector into layers trained only on text and images\. The negative results of §[8](https://arxiv.org/html/2607.18666#S8)locate the remaining bottleneck in exactly those layers; OEA reaches near\-specialist retrieval at a comparable trainable budget precisely because its LoRA adapts the layers themselves \(a route closed to us, since LoRA touches every token and forfeits backbone byte\-identity\)\.

fusion\-embedding\-2adds the in\-layer capacity without giving up the guarantee\. At each of the base’s 28 decoder layers, a bottleneck adapter \(LayerNorm→2048×384→SiLU→384×2048\\mathrm\{LayerNorm\}\\to 2048\{\\times\}384\\to\\mathrm\{SiLU\}\\to 384\{\\times\}2048; 44\.2M parameters total\) is attached to the residual stream,*hard\-gated to audio encoding*: for any text, image, or video input the hook returns the frozen layer’s output before any adapter arithmetic executes, so non\-audio forwards are bitwise identical to the unmodified base\. The adapter up\-projections are zero\-initialized, making a fresh adapter stack the exact identity, so training starts at the generation\-1 architecture\. One implementation constraint matters in practice: with gradient checkpointing, layer forwards re\-run during the backward pass, so the gate must be held open across forward*and*backward of each audio step \(non\-reentrant checkpointing detects a mismatch loudly\); we additionally hard\-fail any text encode issued under an open gate, so the frozen text targets cannot be silently corrupted\.

### 3\.3The invariance guarantee

The family’s defining property is that the base model is provably unmodified, at two levels\.*Parameters*: a regression guard snapshots every base parameter before training and asserts bitwise equality after; a run with nonzero drift fails, and the released checkpoints contain only the connector, adapters, a temperature, and normalization statistics\.*Computation*: in generation 2, the adapter branch does not execute outside the audio gate, so text, image, and video forwards are the frozen base’s forwards: the same operations on the same weights\. Stated precisely, the property is*computation\-graph identity for inputs containing no audio tokens*: a mixed sequence that does contain audio is adapted as a whole, including its text tokens, while any input without audio never touches an adapter\. Bitwise output equality is the verified corollary, checked by unit test and release smoke test under matched kernels, precision, and execution \(§[6\.4](https://arxiv.org/html/2607.18666#S6.SS4)\); we do not claim bit equality across arbitrary hardware, where floating\-point kernels themselves differ\. This converts “the base’s MMEB\-V2 text/image/video results are preserved” from an empirical claim requiring re\-benchmarking into a mechanical consequence of the architecture, checked on every run\. It also has a deployment reading: adding audio to an index built on the base model re\-embeds*nothing*; every stored vector remains exactly valid\.

## 4Training Recipe

### 4\.1Objective and negatives

Letai,ti∈ℝ2048a\_\{i\},t\_\{i\}\\in\\mathbb\{R\}^\{2048\}be the pooled audio and text vectors of a batch ofBBpairs\. The loss is symmetric InfoNCE\(van den Oord et al\.,[2018](https://arxiv.org/html/2607.18666#bib.bib29)\)computed at*every*Matryoshka rungDD\(prefix\-truncate toDD, renormalize\), weighted equally and summed, plus a light CORAL covariance penalty\(Sun and Saenko,[2016](https://arxiv.org/html/2607.18666#bib.bib28)\)\(λ=0\.05\\lambda=0\.05\) that discourages audio from forming its own cluster:

ℒ=∑D∈MRLwD12\[InfoNCED​\(a→t\)\+InfoNCED\(t→a\)\]\+λ∥Cov​\(a\)−Cov​\(t\)∥F2/d2\.\\begin\{split\}\\mathcal\{L\}=\\sum\_\{D\\in\\mathrm\{MRL\}\}w\_\{D\}\\,\\tfrac\{1\}\{2\}\\bigl\[&\\mathrm\{InfoNCE\}\_\{D\}\(a\\to t\)\\\\\[\-4\.0pt\] &\+\\mathrm\{InfoNCE\}\_\{D\}\(t\\to a\)\\bigr\]\\\\ \+\\lambda\\,\\bigl\\\|\\mathrm\{Cov\}\(a\)&\-\\mathrm\{Cov\}\(t\)\\bigr\\\|\_\{F\}^\{2\}/d^\{2\}\.\\end\{split\}\(1\)
#### Full\-corpus frozen\-text negative bank\.

The audio\-to\-text direction augments the in\-batch negatives with the*entire corpus*of cached caption embeddings as shared negatives: 484,372 rows in the generation\-1 headline run\. The classic memory\-bank failure mode is staleness: banked representations drift as their encoder trains\. Here the text tower is frozen, so cached text embeddings are exact forever; the bank has zero staleness*by construction*\. Each anchor’s own caption and its exact\-string duplicates are masked out of the denominator\. Text anchors use in\-batch audio negatives only \(audio embeddings do change during training and are never banked\)\. Because the frozen text side is precomputed once into a cache, the bank costs one matrix multiply per step; the cache itself gave \+27% steps/min and−\-45% peak VRAM \(43\.8 GB→\\to23\.9 GB at batch 128\), which is what makes effective batch 1024 practical on a single accelerator\.

#### Soft labels and false\-negative masking at large scale\.

Past∼\\sim500K corpus rows the bank itself becomes the constraint \(§[5\.4](https://arxiv.org/html/2607.18666#S5.SS4)\): thousands of moderately similar captions crowd the denominator\. Two terms counteract this and are enabled for all runs on 500K\+ corpora: a soft\-label relaxation of the InfoNCE targets \(β=0\.3\\beta=0\.3\) and masking of bank entries whose caption similarity to the anchor exceedsτ=0\.98\\tau=0\.98\(the threshold at which captions stop being paraphrases; §[5\.3](https://arxiv.org/html/2607.18666#S5.SS3)\)\. In a matched A/B at the 592K scale the two terms recover \+2\.6 R@10 over the unmodified objective \(0\.501→\\to0\.527 on the in\-run probe protocol111The*in\-run probe protocol*is the automated AudioCaps rescore that every training job runs at its own precision\. Probe numbers are comparable between matched arms within this paper; release numbers come from the protocol of §[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)at matched bf16 lineage\.\)\. AuroLA’s Hybrid\-NCE \(soft positive sets with weighted negatives, of which InfoNCE is a special case\) arrives at the same structure independently\(Xu et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib35)\), which we read as convergent evidence that plain InfoNCE is the wrong objective at this corpus scale\.

### 4\.2The native\-protocol effect: \+14\.5 points

Decoder\-LM embedding models are trained and released with a specific input protocol\. For Qwen3\-VL\-Embedding this is the chat template with the task instruction in the system turn, the content in the user turn, and generation priming, with the embedding pooled at the final position\. Our early runs fed*bare*sequences \(“\{instruction\} \{caption\}<\|im\_end\|\>”\), off the base’s training manifold\. Rebuilding the caption\-embedding cache in the native format and retraining the connector at otherwise identical configuration improved AudioCaps audio\-to\-text R@10 from 0\.481 to0\.626\(\+14\.5 points; R@1 \+73%\) at half the schedule of the previous best\. This is the single largest improvement in this work, exceeding the gains from whitening \(\+11\.5 in the bare format\) and from a 19K→\\to131K data scale\-up \(\+9\.3\)\.

![Refer to caption](https://arxiv.org/html/2607.18666v1/x2.png)Figure 2:Protocol sensitivity \(AudioCaps A→\\toT R@10\)\. The*same*generation\-1 checkpoint reads 25\.6 points apart under the bare vs\. native input format \(left\), while the bare\-trained control \(right\) shows the native format inflates nothing\. The deployment lesson: evaluate a frozen backbone in its training protocol, or silently forfeit double\-digit recall\. Both formats are printed verbatim in Appendix[G](https://arxiv.org/html/2607.18666#A7)\.The gain is not an evaluation artifact, and the sensitivity cuts both ways \(Figure[2](https://arxiv.org/html/2607.18666#S4.F2)\)\. The v0\.1 checkpoint scored under the bare input format reads AudioCaps R@100\.370; the same checkpoint under the native chat\-template format reads0\.626, a 25\.6\-point swing from formatting alone, because a connector trained against native\-format targets is evaluated off its manifold otherwise\. The control rules out score inflation: a checkpoint trained on*bare*\-format targets scores 0\.488 under the native format vs\. 0\.504 under its own: native formatting does not flatter models not trained for it\. The general lesson:*a frozen embedding backbone must be trained against, and evaluated in, its native input protocol*; off\-protocol targets silently forfeit double\-digit recall, and published comparisons between LLM\-backbone embedding models should be read with the input protocol attached\.

### 4\.3Diagonal, Matryoshka\-safe text whitening

Decoder\-LM embedding spaces are anisotropic; on our training captions the mean pairwise cosine of raw frozen\-text embeddings was 0\.946 under the bare input format\. We standardize the frozen text side per dimension \(mean\-center, divide by per\-dimension standard deviation\), fitted once from cached caption embeddings and stored as buffers\. The transform is deliberately*diagonal*: truncating to the firstDDdimensions and whitening with the first\-DDstatistics equals whitening then truncating, so the Matryoshka nesting survives; full\-covariance whitening would mix dimensions and break it\. Only the frozen text side is whitened; the trainable audio side learns to match the whitened targets\. Under the bare input format whitening was worth \+11\.5 R@10 \(0\.475→\\to0\.590 in a 4K\-clip A/B\)\. Under the native protocol the measured anisotropy drops sharply \(mean pairwise cosine 0\.61–0\.64; §[7](https://arxiv.org/html/2607.18666#S7.SS0.SSS0.Px3)\), and whitening’s contribution shrinks to \+3\.3 R@10 \(0\.593→\\to0\.626\), still real, so it stays in the recipe\.

### 4\.4Stage two: in\-domain fine\-tuning

End\-to\-end CLAP systems owe a documented \+13–15 R@1 to in\-domain AudioCaps fine\-tuning after pretraining\(Bai et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib2)\)\. We add the same stage under our constraint that only the trained components may change\. The fine\-tune warm\-starts the resampler \(and in generation 2 the adapters\), the whitening buffers, and the temperature from the finished pretraining checkpoint, keeps the*pretraining*whitening transform \(the connector was trained against that exact target geometry; refitting on the small in\-domain corpus would move the targets under it\), initializes a fresh optimizer, and trains 400 steps on the AudioCaps train split alone, in minutes of GPU time\.

Two gates, pre\-registered in the execution plan before the run, decide acceptance: AudioCaps audio\-to\-text R@10 must gain at least\+2\+2points, and Clotho zero\-shot audio\-to\-text R@10 must lose no more than22points \(a forgetting guard\)\. Generation 1’s accepted checkpoint \(v0\.3\) passes both: AudioCaps audio\-to\-text moves from 0\.717/0\.279 to0\.741/0\.332\(R@10/R@1;\+2\.4\+2\.4\) and text\-to\-audio R@10 to 0\.746; Clotho lands at 0\.433 audio\-to\-text \(−1\.5\-1\.5, inside the gate\) with text\-to\-audio R@10*up*1\.1 points at 0\.460\. The cost is measurable but small elsewhere: emergent audio→\\toimage on VGGSound\-696 gives back about one point \(R@10 0\.418→\\to0\.407\); in\-domain sharpening trades a little cross\-modal generality\. The schedule length is at an R@1\-preserving optimum, with shorter and longer arms both worse \(§[8\.3](https://arxiv.org/html/2607.18666#S8.SS3.SSS0.Px1)\)\.

### 4\.5Configuration and cost

The generation\-1 headline run: 3,200 steps at effective batch 1024 \(micro\-batch 128×\\times8 accumulation\)≈\\approx6\.8 epochs over 484K pairs, AdamW with cosine decay and 5% linear warmup, bf16 base precision, full\-corpus negative bank, native\-format targets, whitening on: 5\.1 hours on a single H100 \(peak 49\.5 GB\), training loss 9\.48→\\to4\.59,base\_drift=0\.0=0\.0\. The generation\-2 pretrain is the same recipe on the 518K cleaned corpus \(§[5\.4](https://arxiv.org/html/2607.18666#S5.SS4)\) with adapters enabled and 3,900 steps\. Audio\-tower frames and caption embeddings are precomputed once \(both towers frozen\) and streamed from shards, so every expensive forward pass is paid exactly once\.

#### Precision lineage as protocol\.

Train, fine\-tune, and score run at one precision, end to end\. We measured that rescoring a bf16\-trained checkpoint through a 4\-bit base silently costs 5\.6 R@10 points, and the trap is bidirectional \(a 4\-bit\-trained generation\-2 lineage lost∼\\sim2 points scored at bf16\)\. Checkpoints therefore record their base precision and the scorer warns on mismatch; all release numbers come from matched bf16 lineages\.

## 5Data and Scaling

### 5\.1Corpus

The generation\-1 headline \(v0\.2\) corpus is 484,372 audio–caption pairs: the full AudioCaps train split \(∼\\sim45K\)\(Kim et al\.,[2019](https://arxiv.org/html/2607.18666#bib.bib16)\), FSD50K\(Fonseca et al\.,[2022](https://arxiv.org/html/2607.18666#bib.bib10)\), the WavCaps AudioSet\-SL subset\(Mei et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib21)\), and a 318K\-clip subset of LAION\-FreeSound\. All evaluation sets \(AudioCaps test, Clotho, ESC\-50, VGGSound\) are excluded from training by clip id at ingestion\. Long clips are randomly cropped to 10 s training windows \(250 frames, the CLAP\-standard window\), which bounds the per\-clip cache volume and acts as light temporal augmentation\. The generation\-2 corpus grows this to 592K pairs \(a FreeSound tail and BBC sound effects\), then drops 73,716 clips whose metadata describes no sound content, leaving 518,183 training pairs with their original captions\.

![Refer to caption](https://arxiv.org/html/2607.18666v1/x3.png)Figure 3:AudioCaps A→\\toT R@10 \(883 clips, five\-reference min\-rank\) versus training\-corpus size across the milestones of Table[2](https://arxiv.org/html/2607.18666#S5.T2)\. Grey circles: runs with bare\-format text targets; orange squares: native\-protocol targets\. The dotted riser marks the \+14\.5\-point native\-protocol effect at*fixed*data \(§[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)\); the 131K→\\to484K step adds a further \+9\.1 points, and the 400\-step in\-domain stage reaches 0\.741\. Points differ in schedule as labeled, so this is a milestone progression, not a controlled scaling law\.StageCorpusConfigChangeR@1R@10\[S0\]19Kbare, whiten, batch 128, 4000 stepsbaseline0\.1000\.400\[S1\]131K\+ eff\. batch 1024, 131K bank,dr=256d\_\{r\}\{=\}256, 1800 stepsdata \+ negatives0\.1340\.493\[S2\]131Kdr=384d\_\{r\}\{=\}384, 1800 stepscapacity0\.1430\.504\[S3\]131Kdr=384d\_\{r\}\{=\}384, 800 steps,native targets\(v0\.1\)protocol0\.2160\.626131Knative,*without*whitening \(control\)−\-whitening0\.1960\.593\[S4\]484Kdr=384d\_\{r\}\{=\}384, 3200 steps, native \(v0\.2\)data scale0\.2790\.717\[S5\]484K\+ 400\-step in\-domain FT \(v0\.3\)fine\-tune stage0\.3320\.741*Generation 2*\[S6\]518K\+ gated adaptersr=384r\{=\}384, soft\-label/FN\-mask, 3900 steps‡in\-layer capacity0\.2680\.708592Kfull corpus, no adapters, no loss terms \(same\-scale baseline\)‡—0\.2320\.674\[S7\]518K\+ 400\-step in\-domain FT \(fusion\-embedding\-2release\)fine\-tune stage0\.3020\.743Table 2:The family recipe ladder on AudioCaps \(A→\\toT\)\. Each row changes one axis relative to the row above unless noted; the \[S2\]→\\to\[S3\] step changes schedule and target format together \(its protocol component is isolated in §[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)\)\. “Bare”/“native” refers to the text\-target input format\.‡\[S6\] rows are the in\-run probe protocol at the jobs’ 4\-bit training precision \(a matched pair, comparable to each other but not to the release\-protocol rows\); \[S7\] is the release protocol \(§[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\)\.
### 5\.2The ladder from 0\.400 to 0\.741

Table[2](https://arxiv.org/html/2607.18666#S5.T2)and Figure[3](https://arxiv.org/html/2607.18666#S5.F3)decompose the generation\-1 path\. The individually largest levers were, in order: the native\-protocol targets \(\+14\.5 R@10 at fixed data, half schedule\), corpus scale \(19K→\\to131K: \+9\.3; 131K→\\to484K with multi\-epoch training: \+9\.1 over v0\.1\), connector width at matched schedule \(\+4\.7; §[8\.3](https://arxiv.org/html/2607.18666#S8.SS3)\), whitening \(\+11\.5 bare / \+3\.3 native\), the in\-domain fine\-tuning stage \(\+2\.4 R@10, \+5\.3 R@1; §[4\.4](https://arxiv.org/html/2607.18666#S4.SS4)\), and schedule length at matched width \(\+2\.3\)\. Notably, at 19K scale a negatives\-at\-1024 arm did*not*beat the plain 4000\-step baseline \(R@10 0\.392 vs\. 0\.400\): large negative pools pay off only once data scale stops being the binding constraint, consistent with the floor audit below\.

### 5\.3Loss floor: crowding, not caption noise

Training loss at 131K saturated at 4\.0–5\.0 from around step 400\. A natural hypothesis is a noise floor from semantically near\-duplicate captions \(different clips, near\-identical captions\) acting as false negatives\. We tested it directly: re\-run the exact training loss \(same batch size 128, same 131K\-row bank, same exact\-duplicate masking, the checkpoint’s own temperature\) with the audio embedding*replaced by the whitened text embedding*, i\.e\. a hypothetically perfect connector\.

Table 3:Loss\-floor audit at the exact 131K training configuration\. A perfect connector would reach 2\.27; training saturates at 4\.0–5\.0, so∼\\sim2 nats of headroom are attributable to connector capacity / optimization, not caption noise\. False\-negative masking of the bank moves the floor by at most 0\.12 nats even at an aggressively low threshold\.Table[3](https://arxiv.org/html/2607.18666#S5.T3): the perfect\-connector floor is2\.27, roughly 2 nats below the observed saturation, so the saturation is*not*a caption\-noise floor; capacity and optimization bind\. Masking near\-duplicate bank entries buys only 0\.01–0\.12 nats: near\-duplicates at cosine≥\\geq0\.95 average just 5\.4 per anchor\. The bank’s∼\\sim2\-nat floor cost \(2\.27 vs\. 0\.25 without it\) comes from*crowding*, not twins: the median nearest\-neighbor cosine among 131K whitened caption embeddings is 0\.914 \(p90 0\.966\), i\.e\. the caption distribution has low effective dimensionality \(∼\\sim30 by max\-cosine statistics\), so hundreds of moderate\-similarity negatives each leak a fraction of a positive’s mass\. The audit also fixes a practical threshold: genuinely distinct captions \(“a woman gives a speech” vs\. “a man gives a speech”\) sit at cosine 0\.976, so any relevance\-based masking must useτ≳0\.98\\tau\\gtrsim 0\.98\. The audit’s verdict \(widen the connector, then scale data\) was subsequently confirmed by intervention \(§[8\.3](https://arxiv.org/html/2607.18666#S8.SS3)and the 484K run\)\.

### 5\.4Scaling past 500K: bank crowding is real

Growing the corpus from 484K to 592K pairs at the fixed recipe*regressed*both benchmarks, the effect the floor audit predicts: with a full\-corpus bank, every added caption is also an added negative, and the caption distribution’s low effective dimensionality means added mass lands close to existing anchors\. The scale\-up also changes corpus composition \(a FreeSound tail and BBC captions\), so the regression is*consistent with*bank crowding rather than proof of it; the intervention supports the diagnosis: in a matched A/B at 592K, enabling the soft\-label and false\-negative\-masking terms \(§[4\.1](https://arxiv.org/html/2607.18666#S4.SS1)\) recovers \+2\.6 R@10 \(0\.501→\\to0\.527, in\-run probe protocol\), and these flags are adopted for all 500K\+ runs\. Corpus scale beyond∼\\sim500K pairs is therefore not free under a full\-corpus negative bank\. The mechanism is the false\-negative problem studied by debiased and hard\-negative contrastive learning\(Chuang et al\.,[2020](https://arxiv.org/html/2607.18666#bib.bib5); Robinson et al\.,[2021](https://arxiv.org/html/2607.18666#bib.bib27)\), surfacing here as a bank\-scaling caveat; AuroLA’s soft\-positive loss design independently corroborates it\(Xu et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib35)\)\.

## 6Evaluation

### 6\.1Protocols

#### AudioCaps\.

We evaluate on 883 test clips with 4,411 reference captions \(∼\\sim5 per clip\), scoring audio\-to\-text as min\-rank over the five references \(any reference in top\-kkcounts\) and text\-to\-audio with each caption as a query; this is the protocol used by published CLAP\-family results\. Our 883\-clip pool is the currently live\-audio subset of the∼\\sim957\-clip test convention; a larger candidate pool is, if anything, harder\. Because no single canonical split exists \(papers variously use 957\-, 816\-, and private splits\), we also built a machine\-readable allowlist of the 957\-clip WavCaps/ACT convention \(our shard covers 877/957\) and support scoring against it; on the 877\-clip canonical subset an earlier checkpoint scored R@10 0\.503 vs\. 0\.504 on the full 883, indicating the pool difference is immaterial\. All retrieval is at the 1024\-d Matryoshka rung\.

#### Clotho \(zero\-shot\)\.

We use the canonical Clotho v2\.1 evaluation set\(Drossos et al\.,[2020](https://arxiv.org/html/2607.18666#bib.bib8)\)built directly from the Zenodo release: 1,045 clips×\\timesexactly 5 captions, min\-rank scoring\. Clotho never appears in our training data, so all Clotho numbers are strictly zero\-shot; we compare only against baselines that are also zero\-shot on Clotho’s captions \(most published Clotho rows are not\)\.

#### Cross\-modal\.

VGGSound\-696\(Chen et al\.,[2020](https://arxiv.org/html/2607.18666#bib.bib4)\)\(696 audio clips paired with video frames; chance R@10=0\.014=0\.014\)\. VGGSound is blacklisted from our training data\. Images pass through the base’s native vision path with its official embedding prompt; audio through the trained audio path; text through the frozen text path\. Rankings use per\-modality mean\-centering of the gallery \(§[7](https://arxiv.org/html/2607.18666#S7)\)\.

#### Paired\-baseline replay\.

Evaluation protocol can dominate the measured number \(§[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)\), and the sensitivity applies*longitudinally*within a project: a fresh evaluation may only be compared against an archived result after replaying the archived run’s exact evaluation flags\. During generation\-2 development, a new checkpoint’s cross\-modal score appeared to collapse relative to an archived table; re\-running the archived model as a paired baseline under the fresh flags reproduced the same collapse, isolating the difference to a template setting rather than the model\. Our working practice, used for every table in this section: score a paired baseline alongside every new evaluation configuration before interpreting any cross\-checkpoint difference\.

#### Uncertainty\.

All results are single\-seed\. At the pool sizes here a recall proportion carries a binomial standard error of roughly 1\.5 points \(883 queries\) to 1\.4 points \(1,045\), so we read single deltas below∼\\sim2 points as parity: the generation\-2 AudioCaps audio\-to\-text R@10 edge over v0\.3 \(0\.741→\\to0\.743\) is within noise, while its text\-to\-audio gains \(\+2\.9 AudioCaps, \+2\.2 Clotho R@10\) and its VGGSound audio–text gains \(\+4\.0/\+3\.6\) exceed it\. Bold in the tables marks the best point estimate per column and class, not a significance claim\. No headline number uses test\-time correction; a querybank\-normalization study is reported in Appendix[B](https://arxiv.org/html/2607.18666#A2)\.

### 6\.2Audio–text retrieval

Table 4:AudioCaps test retrieval, grouped by system class; bold marks the best per column*within each class*\. Our rows: 883 clips, five\-reference min\-rank protocol, bf16, native templates\. All other rows are the numbers reported by their authors \(retrieved 2026\-07\-12\) on their respective AudioCaps test protocols, which vary in split and pool across papers \(§[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\)\.∗Our reproduction: the publiclaion/clap\-htsat\-fusedcheckpoint \(pinned revision\) scored on our exact 883\-clip five\-reference protocol; the cited row above it is the paper’s own best AudioCaps configuration on its own protocol, so the two rows differ in both checkpoint and protocol\. M2D\-CLAP’s direction labels are single\-sourced \(the reproduction inYoo et al\. \([2026](https://arxiv.org/html/2607.18666#bib.bib37)\)reports T→\\toA R@1 0\.414\)\. AuroLA is a two\-pass system \(7B retriever, then a second 7B pairwise re\-ranker\)\. ONE\-PEACE trains all 4B parameters and fine\-tunes on the merged AudioCaps\+Clotho\+MACS train splits \(its evaluation split is not stated\)\. Specialist and CLAP systems embed audio and text only; our models additionally embed image and video in the same space, with the base’s performance there preserved bitwise \(§[6\.4](https://arxiv.org/html/2607.18666#S6.SS4)\)\. Dashes: not reported / not recorded\.Table 5:Clotho v2\.1 evaluation \(1,045 clips×\\times5 references\), zero\-shot rows only: every listed model excludes Clotho’s*caption*data from training; bold: best per column within each class\. A caveat applies to the baselines’ audio: WavCaps’ FreeSound portion overlaps 61% of the Clotho evaluation audio \(as measured byYoo et al\. \([2026](https://arxiv.org/html/2607.18666#bib.bib37)\)\), so zero\-shot rows for WavCaps\-trained models \(including GLAP, which trains on WavCaps\) are optimistic; AuroLA’s training\-audio overlap with Clotho is not reported\. Our corpus excludes all 5,929 Clotho FreeSound clip ids by blacklist at ingestion\. Baseline numbers as reported by their authors \(retrieved 2026\-07\-12\); GLAP numbers are single\-sourced\. Dashes: not reported / not recorded\.Table[4](https://arxiv.org/html/2607.18666#S6.T4)reports AudioCaps retrieval across the system classes; Table[5](https://arxiv.org/html/2607.18666#S6.T5)the strictly zero\-shot Clotho comparison\. Consistent with §[4\.2](https://arxiv.org/html/2607.18666#S4.SS2), the cited rows ride their authors’ input protocols and splits, so we read these tables at the class level rather than cell against cell\. Within the frozen\-base class, generation 1’s pretraining checkpoint \(v0\.2\) reaches audio\-to\-text R@10 0\.717 \(R@1 0\.279, R@5 0\.590, mAP@10 0\.208\); the in\-domain stage \(v0\.3\) lifts this to 0\.741/0\.332, and generation 2 posts the family’s best text\-to\-audio results on both benchmarks \(AudioCaps 0\.775, Clotho 0\.482\) while matching v0\.3 on audio\-to\-text R@10 \(0\.743, within noise; §[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\); full R@1/R@5/R@10 grids are in Appendix[D](https://arxiv.org/html/2607.18666#A4)\. The remaining gap to the fully fine\-tuned CLAP family is 16–19 R@10 points, and the audio\-native specialist systems sit higher still; these are comparisons whose structural terms \(both towers trained, in\-domain data, LoRA inside the backbone, a second re\-ranking pass\) are drawn in the table and taken up in §[9](https://arxiv.org/html/2607.18666#S9)\. As a protocol anchor, we also scored the released public LAION\-CLAP checkpoint on our exact protocol \(the starred row\): audio\-to\-text R@10 0\.684, below both v0\.2 and v0\.3\. The gap to the cited row is therefore a cross\-protocol, cross\-checkpoint reading, the effect §[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)documents, and the tables support class\-level conclusions rather than cell\-level ones\. On Clotho, our models sit within 10 R@10 points text\-to\-audio \(17 audio\-to\-text\) of the strongest nominally zero\-shot dual encoder despite frozen towers, with the caveat in the caption that those baselines’ training audio overlaps the Clotho evaluation set while ours is excluded by construction; ONE\-PEACE’s Clotho rows \(audio\-to\-text 0\.271/0\.654, text\-to\-audio 0\.224/0\.627 R@1/R@10\) are omitted from Table[5](https://arxiv.org/html/2607.18666#S6.T5)because its training includes Clotho’s train split\. The R@10/R@1 ratio, a measure of top\-of\-ranking sharpness \(lower is sharper\), improved from 2\.9×\\times\(v0\.1\) to 2\.57×\\times\(v0\.2\) to 2\.23×\\times\(v0\.3\); end\-to\-end systems sit at 1\.7–2\.2×\\times\.

#### The family delta\.

Table[6](https://arxiv.org/html/2607.18666#S6.T6)compares the two generations cell by cell under the release protocol\. The generation\-2 gains concentrate in the text\-to\-audio direction and the cross\-modal audio–text pair \(\+4\.0\+4\.0/\+3\.6\+3\.6R@10,\+5\.3\+5\.3T→\\toA R@1 on VGGSound\); generation 1 retains audio\-to\-text R@1 on both caption benchmarks\. The pattern is consistent with the mechanism: in\-layer capacity lets the audio representation organize for discrimination against text queries \(the direction that improves everywhere\), at a small cost in audio\-to\-text top\-1 on the caption benchmarks\. In the controlled A/B behind this comparison \(matched 45K\-scale arms, 800 steps, identical recipe; Appendix[C](https://arxiv.org/html/2607.18666#A3)\), adapters at rank 384 improve every retrieval direction \(audio\-to\-text R@10 0\.631→\\to0\.665 \(\+3\.4\+3\.4\), R@1 0\.202→\\to0\.229, text\-to\-audio R@10 0\.684→\\to0\.708\), with rank 128 recovering\+2\.5\+2\.5, so the gains scale with adapter capacity and had not saturated at the largest rank trained\. The experiment replicates across three seeds, with paired audio\-to\-text R@10 deltas of\+3\.4\+3\.4/\+4\.6\+4\.6/\+5\.4\+5\.4and every retrieval direction positive at every seed; the published pairing is the most conservative of the three\. At full 518K scale the pretraining gain reproduces \(R@10 0\.708 vs\. 0\.674, matched protocol\)\.

Table 6:The family delta under the release protocol \(bf16, native templates; Clotho zero\-shot\)\. Bold marks the better generation per row \(deltas under∼\\sim2 points are within single\-seed noise; §[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\)\. The pre\-fine\-tune generation\-2 checkpoint reaches the family’s best emergent audio\-to\-image alignment \(R@10 0\.443, measured under the earlier readout template\), indicating the adapters strengthen rather than erode the text bridge; the in\-domain fine\-tune then gives back part of that alignment for its AudioCaps gains \(0\.392 under the release readout; the two numbers straddle a template change, so the exact size of the trade is not protocol\-clean\), as the generation\-1 fine\-tune did \(−1\.1\-1\.1\)\.

### 6\.3Breadth: MAEB, user\-intent queries, and coverage

#### MAEB\.

We evaluated the released generation\-1 v0\.2\-preview checkpoint withmteb==2\.18\.0on the sound\-event tier of the MAEB benchmark\(El Assadi et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib9)\): nine task results submitted to the public leaderboard \(BeijingOpera, ClothoT2ARetrieval, GTZANAudioReranking, GTZANGenre, MACST2ARetrieval, RavdessZeroshot, SpeechCommandsZeroshotv0\.02, UrbanSound8KT2ARetrieval, VehicleSoundClustering\); against the public leaderboard as of 2026\-07\-09 these scores place roughly \#3–\#6 per task on the tasks closest to our training domain \(UrbanSound8K \#3, Ravdess \#4\) despite the speech\- and music\-light corpus; per\-task scores are in Appendix[H](https://arxiv.org/html/2607.18666#A8)\. A tenth task, FSD2019Kaggle, was evaluated but withheld: matching its test clips by FreeSound id against the Zenodo post\-competition metadata shows 608 of its 4,481 test clips \(13\.6%\) appear in the FSD50K development split \(part of our training corpus\), and the test set is in fact entirely contained in FSD50K\. Any FSD50K\-trained model is affected on that task; we flagged this to the benchmark maintainers\.

#### User\-intent queries\.

Yoo et al\. \([2026](https://arxiv.org/html/2607.18666#bib.bib37)\)argue that caption\-style queries understate real search behavior and introduce user\-intent queries: questions, commands, keyword tags, paraphrases, and exclusion queries\. On our own evaluation in that style over the identical 1,045\-clip Clotho pool, v0\.3 averages R@10 0\.518 across the four positive query formulations: retrieval degrades gracefully rather than collapsing off the caption manifold, which we attribute to the LLM backbone’s instruction conditioning\.

#### Coverage\.

The evaluation surface itself separates the system classes, and Table[7](https://arxiv.org/html/2607.18666#S6.T7)makes the claim explicit\. Every specialist row in Tables[4](https://arxiv.org/html/2607.18666#S6.T4)and[5](https://arxiv.org/html/2607.18666#S6.T5)embeds audio and text only: none of those systems can populate a single cell of the cross\-modal matrix in §[7](https://arxiv.org/html/2607.18666#S7), and none inherits a text/image/video benchmark result at all\. Our models populate every cell of every table in this paper with one checkpoint and one space, and the text/image/video cells of that space are the base model’s published MMEB\-V2 results\(Li et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib19)\), carried over bitwise \(§[6\.4](https://arxiv.org/html/2607.18666#S6.SS4)\)\. Figure[4](https://arxiv.org/html/2607.18666#S6.F4)places the family among the systems that*can*enter the unified comparison\.

Table 7:Coverage across the unified evaluation surface\. Cells are each system’s published results on the named benchmark, or our reproductions \(∗, VGGSound\-696 average R@10 over both directions on the protocol of §[7](https://arxiv.org/html/2607.18666#S7)\); base\-benchmark cells are official 0–100 scores \(Text==MMTEB Mean\(Task\); Image/Video/VisDoc==MMEB\-V2 category overalls\), audio cells are recall fractions\.∘marks cells inherited from the released Qwen3\-VL\-Embedding\-2B card\(Li et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib19)\)bitwise \(§[6\.4](https://arxiv.org/html/2607.18666#S6.SS4)\): not re\-measured, mechanically preserved\. The table’s claim is*population*, not cell\-vs\-cell comparison: one Fusion Embedding checkpoint covers the entire row; every other system leaves most of the surface empty\. ImageBind’s audio–image cell is its supervised training pair\.![Refer to caption](https://arxiv.org/html/2607.18666v1/x4.png)Figure 4:Positioning among unified embedding models on VGGSound\-696 cross\-modal retrieval \(R@10 averaged over both directions; log axis of*trained*parameters\)\. Left: audio↔\\leftrightarrowtext\. Right: emergent audio↔\\leftrightarrowimage, where ImageBind trains on its supervised audio–image pair while our models have seen zero audio–image pairs\. Trained\-parameter positions:fusion\-embedding\-116\.4M,fusion\-embedding\-260\.6M, ImageBind≈\\approx0\.21B trained of 1\.2B total \(frozen CLIP towers\), LanguageBind≈\\approx0\.30B \(the fully fine\-tuned audio tower we benchmark\)\. Baselines are our reproductions on the identical protocol \(§[7](https://arxiv.org/html/2607.18666#S7)\); the dashed line is Gemini Embedding 2, an API of undisclosed size, evaluated 2026\-07\-09\.

### 6\.4Invariance verification

The guarantee of §[3\.3](https://arxiv.org/html/2607.18666#S3.SS3)is enforced, not assumed, and every check reads exactly zero\.*Parameters*: the regression guard’sbase\_driftassertion, the maximum absolute change over every base parameter, returned0\.00\.0on every training run reported in this paper, including all generation\-2 adapter runs\.*Outputs*: unit tests encode identical text through the base model and throughfusion\-embedding\-2with adapters attached and assert bitwise\-equal activations \(maximum absolute difference0, not merely small\); the release smoke test repeats the check on the packaged checkpoint through the public loading path\.*Controls that must differ, do*: the same test suite verifies that audio\-gated forwards*change*when adapters are enabled, that gradients reach only adapter and connector parameters, and that a gate held open across an interrupted backward is detected loudly rather than silently degrading the guarantee\. Text–image retrieval cells forfusion\-embedding\-1andfusion\-embedding\-2in Table[9](https://arxiv.org/html/2607.18666#S7.T9)are identical to the third decimal because the underlying vectors are identical to the last bit\. Table[8](https://arxiv.org/html/2607.18666#S6.T8)summarizes the checks; the full verification protocol, mapped to the open\-source test suite, is in Appendix[F](https://arxiv.org/html/2607.18666#A6)\.

CheckGen\. 1Gen\. 2Base param\. drift, max\|Δ\|\|\\Delta\|, every run00Text forwardΔ\\Deltavs\. base \(unit test\)00Text forwardΔ\\Delta\(released ckpt\., smoke\)00Text↔\\leftrightarrowimage vectors across familyidenticalTable 8:Invariance verification: every entry is a maximum absolute difference, exactly zero \(not approximately\) under matched execution \(§[3\.3](https://arxiv.org/html/2607.18666#S3.SS3)\)\. Image and video inputs take the same gated path as text \(the gate keys on audio presence, not token type\), so the text checks cover the mechanism; the identity of every text↔\\leftrightarrowimage cell across all four released checkpoints \(Table[9](https://arxiv.org/html/2607.18666#S7.T9)\) confirms it end\-to\-end\.

## 7Emergent Cross\-Modal Retrieval

The connector never sees an audio–image pair; VGGSound itself is blacklisted from training\.Because we train no audio–visual supervision of any kind, we call retrieval between audio and the base’s visual modalities*emergent cross\-modal retrieval*, the retrieval analogue of the emergent zero\-shot classification that ImageBind established for bind\-to\-pivot training\(Girdhar et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib11)\), here arising from a frozen base whose text–image geometry the new modality simply inherits\. The setting is strictly harder than supervised cross\-modal retrieval, and published baselines for it are scarce: ImageBind is the natural reference point, with its supervised audio–image pair printed for calibration rather than competition\.

Generation 1 v0\.2 retrieves the matching video frame from sound alone at R@10 0\.418 \(R@1 0\.088, R@5 0\.315, mAP@10 0\.179\), 29×\\timeschance, and the reverse direction reaches 0\.440 \(31×\\times\)\. An earlier checkpoint, before the native\-protocol change, had already established the existence proof at R@10 0\.310 \(21×\\timeschance\); audio–text quality improvements have since lifted the emergent direction in lockstep: benchmark gains on the supervised pair compound directly into the unsupervised ones\. The strongest emergent checkpoint in the family is the generation\-2 pretrain at R@10 0\.443\.

Modelaudio↔\\leftrightarrowimageaudio↔\\leftrightarrowtexttext↔\\leftrightarrowimageChance0\.0140\.0140\.014ImageBind\-Huge\(Girdhar et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib11)\)0\.718 / 0\.7200\.404 / 0\.3480\.243 / 0\.282LanguageBind\(Zhu et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib38)\)0\.365 / 0\.4150\.547 / 0\.3310\.221 / 0\.283Gemini Embedding 2\(Google DeepMind,[2026](https://arxiv.org/html/2607.18666#bib.bib13)\)0\.312 / 0\.3160\.379 / 0\.3740\.273 /0\.366fusion\-embedding\-1v0\.10\.368 / 0\.3880\.555 / 0\.5920\.331/ 0\.319fusion\-embedding\-1v0\.20\.418 / 0\.4400\.588 / 0\.6310\.331/ 0\.319fusion\-embedding\-1v0\.30\.407 / 0\.4280\.625 / 0\.6450\.331/ 0\.319fusion\-embedding\-20\.392 / 0\.4300\.665 / 0\.6810\.331/ 0\.319Table 9:VGGSound\-696 cross\-modal retrieval, R@10 in both directions \(chance=0\.014=0\.014\)\. Identical clips, frames, and scoring for all models; ImageBind numbers computed with the releasedimagebind\_hugecheckpoint, LanguageBind with revision\-pinnedAudio\_FTandImagecheckpoints \(its audio–text shown with its strongest readout, the audio branch’s own text tower\); our readout uses per\-modality mean\-centering\. ImageBind trains directly on audio–image pairs \(its supervised pair\); LanguageBind trains audio against language; Gemini Embedding 2 is a commercial natively\-multimodal API \(training undisclosed\), evaluated 2026\-07\-09 at its documented default invocation; our models train on audio–text only; each model’s other pairs are emergent\. Our text↔\\leftrightarrowimage rows are identical across the family because text and image never touch the trained components \(§[6\.4](https://arxiv.org/html/2607.18666#S6.SS4)\)\. Evaluation captions are the model\-generated descriptions distributed with the MAEB VGGSound retrieval set\(El Assadi et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib9)\); the generating model is not documented by the benchmark, and model\-generated caption style could favor text towers trained on similar registers \(a sensitivity §[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)shows can be large\), but all models received identical inputs\. Bold: best per pair \(per direction where split\)\.![Refer to caption](https://arxiv.org/html/2607.18666v1/figures/fig_gallery.png)Figure 5:Qualitative audio→\\toimage retrieval on VGGSound\-696 \(v0\.2 checkpoint; aggregate R@10 0\.418, chance 0\.014, zero audio–image training pairs\)\. Each row: the query clip’s own frame \(left, blue\), then the top\-5 retrieved frames; green marks the clip’s exact frame, with its rank noted above each row\. Retrievals are organized by sound category even when the exact frame ranks lower\. Example frames from the VGGSound dataset\(Chen et al\.,[2020](https://arxiv.org/html/2607.18666#bib.bib4)\)\(CC\-BY\-4\.0\), shown for evaluation illustration\.#### Head\-to\-heads\.

Table[9](https://arxiv.org/html/2607.18666#S7.T9)reports the comparison on our exact protocol\. Against ImageBind\-Huge, each model wins its*supervised*pair: ImageBind trains on audio–image \(0\.718/0\.720 vs\. our emergent 0\.418/0\.440\); we train on audio–text \(0\.588/0\.631 for v0\.2, 0\.665/0\.681 forfusion\-embedding\-2, vs\. its emergent 0\.404/0\.348\)\. Comparing emergent\-vs\-emergent, ImageBind’s audio–text \(0\.404/0\.348\) and our audio–image \(0\.418/0\.440\) are of the same order\. Our models additionally win text–image \(0\.331/0\.319 vs\. 0\.243/0\.282\), where both models are exercising pretrained vision–language alignment; two of three pairs on this protocol go to the frozen\-base models\.

Against LanguageBind the comparison is uniform: v0\.2 exceeds it on all three pairs, including LanguageBind’s own supervised direction \(audio–text: 0\.588/0\.631 vs\. 0\.547/0\.331 under its strongest readout\), andfusion\-embedding\-2extends the margin\. A measurement on the released checkpoints suggests why\. LanguageBind’s modality branches each fine\-tune their own copy of the text tower, and the copies diverge: the mean cosine between the audio branch’s and the image branch’s embeddings of identical captions is 0\.553, so the language pivot intended to bind the branches is no longer a shared space, and its emergent audio–image \(0\.365/0\.415\) falls below ours despite a far larger trained parameter count\. This failure mode is structurally unavailable to a frozen\-base design: every modality of the Fusion Embedding family conforms to one immutable space, asserted parameter\-identical after every run\.

Gemini Embedding 2\(Google DeepMind,[2026](https://arxiv.org/html/2607.18666#bib.bib13)\), a commercial natively\-multimodal embedding API released in 2026, provides the closest product\-level analogue to our design goal \(one space for text, images, video, audio, and documents\)\. On this protocol our models lead it on audio–image in both directions \(our*emergent*pair against a natively\-trained system\) and on audio–text in both directions, while text–image splits by direction\. We evaluate the API at its documented default invocation and report the evaluation date; as a served model it may change over time, and its training data and protocol are undisclosed, so this row is a product snapshot rather than a method comparison\.

#### Why alignment emerges\.

The mechanism is the one ImageBind established, inverted in an economically useful direction\. The base model already places text, images, and video in one space\. Our audio path is optimized solely to place audio near its captions*in that space*, and captions of a barking dog live near images of barking dogs by the base’s own training\. Alignment to text is therefore alignment to everything text is aligned to\. Quantitatively, the emergent audio↔\\leftrightarrowimage direction reaches roughly 70% of the strength of the trained audio↔\\leftrightarrowtext direction on the same clips \(0\.418/0\.440 vs\. 0\.588/0\.631\), and improvements to the trained direction have transferred to the emergent one at every checkpoint measured \(0\.368/0\.388→\\to0\.418/0\.440 from v0\.1 to v0\.2, with the text↔\\leftrightarrowimage control unchanged at 0\.331/0\.319\), confirming the gain is attributable to audio moving, not to any change in the base’s geometry, which is impossible here by construction\. Figure[5](https://arxiv.org/html/2607.18666#S7.F5)shows the emergent direction qualitatively: retrievals organize by sound category, with the query clip’s exact frame frequently in the top ranks\.

#### Geometry: anisotropy is substantially a protocol artifact\.

Under the bare input format, the frozen text embeddings of random training captions have mean pairwise cosine 0\.946 \(measured at whitening\-fit time; 0\.93–0\.95 across fits\), the “severe anisotropy” of decoder\-LM embedding spaces\. Under the native chat template the same statistic drops to 0\.640 \(131K corpus\) and 0\.612 \(484K corpus\)\. Much of the reported anisotropy of decoder\-LM embedding spaces, in this model at least, is the cost of querying the model off its training distribution rather than an intrinsic property of the space\. This is consistent with the whitening ablation \(§[4\.3](https://arxiv.org/html/2607.18666#S4.SS3)\): the correction is worth \+11\.5 R@10 where the anisotropy is severe \(bare\) and \+3\.3 where it is mild \(native\)\. On the cross\-modal benchmark, readout geometry is a second\-order effect: per\-modality mean\-centering of the gallery is the best readout \(audio→\\toimage 0\.397 raw, 0\.402 whitened, 0\.418 centered for v0\.2\), and the mismatch between the whitened text space and the raw image space costs only∼\\sim1 point\.

## 8What Did Not Work

Three controlled experiments refuted the three most natural improvement hypotheses for this architecture\. We report them at the same standard as the positive results, each with its experiment, its number, and the mechanism we believe explains it, because together they redraw the design map: the binding constraint of frozen\-backbone audio fusion is not the captions, not the audio encoder, and not the connector, but the frozen LM’s ability to process audio tokens\. Generation 2 \(§[3\.2](https://arxiv.org/html/2607.18666#S3.SS2)\) is the direct response, and its gains land exactly where this section predicts\.

### 8\.1Do cleaner captions help?

No: style dominates cleanliness\. Roughly three quarters of our corpus carries metadata\-derived captions \(titles, tags, recording notes\) rather than descriptive sentences, an obvious target for model\-based rewriting, which the caption\-quality literature \(§[2](https://arxiv.org/html/2607.18666#S2)\) would predict to help\. We rewrote all 426K FreeSound and BBC captions with an instruction\-following LLM into short descriptive caption style \(median 7 words\), dropped the 74K clips whose metadata carried no sound content at all, and retrained the full recipe on the remaining 518K pairs with the large\-scale loss terms enabled\. The rewritten corpus is measurably cleaner \(exact\-duplicate caption rows fall from 50\.7% to 45\.0%\), yet in\-domain retrieval*drops*: AudioCaps R@10 0\.649 against 0\.674 for the same\-scale baseline trained on the raw captions \(592K before the junk\-clip removal, without the loss terms; all arms in Appendix Table[13](https://arxiv.org/html/2607.18666#A5.T13)\)\. The−2\.5\-2\.5is a*lower bound*on the rewrite’s cost there: the loss terms alone are worth\+2\.6\+2\.6in a matched A/B \(§[5\.4](https://arxiv.org/html/2607.18666#S5.SS4)\) and the junk\-clip removal is caption\-preserving, so both differences between the arms favor the rewritten one\. \(Measured against the 484K v0\.2 checkpoint the drop reads−6\.8\-6\.8, but most of that is the scale regression of §[5\.4](https://arxiv.org/html/2607.18666#S5.SS4), not the captions\. Clotho reads 0\.438, above the same\-scale baseline’s 0\.400 and below v0\.2’s 0\.448, consistent with the style analysis below rather than with caption quality\.\)

The mechanism is a style\-region shift, not a failure to learn: the run’s internal held\-out evaluation*on rewritten captions*reaches R@10 0\.84, while evaluation on original\-style captions collapses, and the run’s whitening\-fit statistics shifted sharply away from the raw\-caption fit, recording the text\-distribution move directly\. The rewrite also targeted the wrong register for the benchmark it was meant to help: the outputs are short AudioCaps\-like sentences, where Clotho’s references are long, elaborate descriptions\. The conclusion:*caption style specificity dominates caption cleanliness*for retrieval transfer; rewriting a corpus into a generic clean style moves the model’s operating point rather than improving it\. The fix implied is style\-targeted: match the reference register of the benchmark, and augment rather than replace original captions\. \(The 74K\-clip junk removal, by contrast, was kept: it is caption\-preserving and defines the generation\-2 corpus\.\)

### 8\.2Is a leaderboard\-stronger audio encoder a better tower?

No: co\-training compatibility dominates encoder ranking\. GLAP’s controlled ablation\(Dinkel et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib6)\)trains five frozen audio encoders under one contrastive recipe with MLP projections and finds Whisper\-family encoders∼\\sim9–12 mAP@10 behind sound\-event encoders on sound retrieval \(AudioCaps T2A mAP@10: CED\-Base 58\.6, BEATs 55\.1, Dasheng 55\.8, Whisper\-Base 46\.5\)\. Our tower is Whisper\-family, so we ran the implied experiment: a matched A/B at 45K AudioCaps\-only scale, 800 steps, identical recipe and hyperparameters, each arm evaluated on frames from its own tower\. The ranking*inverts*: the Qwen2\.5\-Omni tower \(Whisper\-family,∼\\sim640M, co\-trained to feed a Qwen LM\) reaches audio\-to\-text R@10 0\.631 against 0\.469 for Dasheng\-base\(Dinkel et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib7)\), a 16\-point margin, consistent in both directions \(text\-to\-audio 0\.684 vs\. 0\.538\)\. We note the size confound: the towers are not parameter\-matched \(86M vs\.∼\\sim640M\), so the experiment does not isolate co\-training from scale\. It does establish the practical point: encoder\-quality rankings measured under shallow projections do not transfer to architectures that splice tokens into a frozen LLM, where compatibility with the LLM’s expected input distribution dominates\. Selecting a tower for this architecture class by CLAP\-style leaderboards would have selected the wrong tower\.

Two adjacent probes complete the picture\. Whisper\-family encoders are reported to hide sound\-event information in intermediate layers, with last\-layer features specializing toward speech\(Gong et al\.,[2023](https://arxiv.org/html/2607.18666#bib.bib12)\); we linear\-probed the actual frozen tower on ESC\-50\(Piczak,[2015](https://arxiv.org/html/2607.18666#bib.bib24)\)\(5\-fold\) and found the layer curve rises*monotonically*to 0\.924 at the last layer, with a learned weighted average over all layers*worse*\(0\.893\); the Omni tower’s continued pretraining evidently repaired the vanilla\-Whisper pathology, so multi\-layer taps offer nothing for this tower\. And the only controlled same\-connector/same\-LLM ablation we are aware of\(Liu et al\.,[2024](https://arxiv.org/html/2607.18666#bib.bib20)\)finds adapting a Whisper\-family encoder buys nothing for audio captioning \(SPIDEr\-FL: LoRA 44\.8 vs\. frozen 44\.8, with full fine\-tuning*worse*at 44\.0, versus 49\.6 for a frozen event encoder\); adaptation capacity is not what frozen Whisper\-derived features lack\. Together these results rule out audio\-tower adaptation as a productive direction for this architecture\.

### 8\.3Is the connector too small?

Table 10:Connector\-width study on AudioCaps\-883 \(A→\\toT\)\. In both groupsdr=512d\_\{r\}\{=\}512shows better*training*metrics but equal or worse held\-out retrieval\.No: wider connectors memorize\. At 131K with matched 800\-step schedules \(Table[10](https://arxiv.org/html/2607.18666#S8.T10)\),dr=384d\_\{r\}\{=\}384beatsdr=256d\_\{r\}\{=\}256by \+4\.7 R@10 points \(\+30% R@1\) and reaches parity with the 256\-width 1800\-step run \(0\.481 vs\. 0\.493, within single\-seed noise; §[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\) at 44% of the steps\. Width 512 ties 384 on held\-out retrieval*despite*far better training metrics \(tail in\-batch accuracy∼\\sim0\.73 vs\.∼\\sim0\.53 and a much stronger in\-domain probe\): the extra width fits the training distribution without generalizing\. The re\-test at 484K confirms the knee is not an artifact of the smaller corpus: an exact v0\.2 mirror atdr=512d\_\{r\}\{=\}512loses on every metric \(R@10 0\.675 vs\. 0\.717\) with the same memorization signature\. We fixdr=384d\_\{r\}\{=\}384\. Input\-side width, like the input\-side tower, is not where the headroom lives: capacity helps only when placed*inside*the layers, behind the modality gate \(§[6\.2](https://arxiv.org/html/2607.18666#S6.SS2)\)\.

#### Fine\-tune length\.

The fine\-tuning schedule shows the same shape in miniature: a 600\-step generation\-1 arm improves AudioCaps R@10 to 0\.750 and text\-to\-audio R@10 to 0\.762, with Clotho at 0\.442/0\.477, but AudioCaps R@1 falls from 0\.332 to 0\.316; a 200\-step generation\-2 arm is worse than 400 on both metrics \(0\.273/0\.695 vs\. 0\.302/0\.743\)\. The two arms sit on different generations, so they are consistent with—not proof of—a shared optimum at 400 steps; we report 400\-step checkpoints throughout\.

## 9Discussion and Limitations

On in\-domain audio–text retrieval, the specialist systems remain ahead: the CLAP family reaches AudioCaps audio\-to\-text R@10 0\.906–0\.928 by training both towers end\-to\-end with in\-domain data, and AuroLA unsurprisingly achieves the best results of any system, but it trains 418M parameters inside a 7B backbone, adds a second 7B model as a re\-ranking pass, and, like every audio–text system in Table[4](https://arxiv.org/html/2607.18666#S6.T4), serves exactly one modality pair with no preservation claim of any kind\. The trade this family makes is explicit: some in\-domain audio–text accuracy, in exchange for one space over four modalities, a bitwise\-preserved base, single\-GPU\-hours training cost, and an index that never needs re\-embedding\.

Our own scale\-up evidence tempers the obvious next step: growing the corpus from 484K to 592K pairs at the fixed recipe*regressed*both benchmarks \(bank crowding, §[5\.4](https://arxiv.org/html/2607.18666#S5.SS4); partially recovered by the soft\-label and false\-negative\-masking terms\), so corpus scale beyond∼\\sim500K pairs is not free under a full\-corpus negative bank; AuroLA’s scaling curve, measured without one, suggests headroom in the 0\.5M–1\.4M range\(Xu et al\.,[2026](https://arxiv.org/html/2607.18666#bib.bib35)\)\. At fixed 2B base size the measured levers are otherwise spent: connector capacity has a knee at 384 \(§[8\.3](https://arxiv.org/html/2607.18666#S8.SS3)\), caption rewriting and tower substitution reduce accuracy \(§[8](https://arxiv.org/html/2607.18666#S8)\), and the fine\-tune schedule is at its optimum from both sides \(§[8\.3](https://arxiv.org/html/2607.18666#S8.SS3.SSS0.Px1)\)\. The headroom we can identify: additional in\-domain data, and the adapter rank axis, which had not saturated at rank 384\. The first lever has since been confirmed: after this report’s evaluations were frozen, a 1,600\-step fine\-tune on the AudioCaps 2\.0\-expanded pool \(86,394 pairs; train split verified disjoint from our evaluation protocol\) lifted AudioCaps R@10 to 0\.759 audio\-to\-text and 0\.785 text\-to\-audio with Clotho and VGGSound within noise, and is released as v0\.2 of generation 2; all numbers reported here remain the pinned v0\.1 checkpoints\. Systems with native\-audio backbones and in\-backbone adaptation sit∼\\sim15 R@1 higher on AudioCaps, which bounds what this family can reach if the byte\-identity constraint were relaxed, though OEA’s observation that its 7B backbone matches its 3B cautions against expecting backbone scale alone to close it\.

Three further limitations\.*Speech and music are not yet trained to parity*: the instruction taxonomy includes speech content, language, paralinguistics, and music tasks, but the corpus is sound\-event\-centric, and MAEB’s speech tasks are the models’ weakest\.*English\-only, with input constraints*: captions are English; audio is 16 kHz mono, processed in 30 s windows \(longer audio chunked, up to 8 windows\)\.*Evaluation breadth*: cross\-modal results are on one benchmark \(VGGSound\-696\) with a single readout convention; MMEB\-V2 preservation is mechanical \(parameter identity\) rather than re\-measured; and some baseline numbers \(M2D\-CLAP direction labels, GLAP\) are single\-sourced, as flagged in the tables\. Finally, the training mix includes YouTube\-sourced and research\-licensed corpora, so the preview checkpoints are released under CC\-BY\-NC\-4\.0; a commercially clean corpus is a separate, ongoing track\.

## 10Conclusion

A frozen vision–language embedding base can be extended to audio without touching a single base weight: first with a 16\.4M connector trained in five GPU\-hours \(fusion\-embedding\-1: AudioCaps R@10 0\.741 at the frontier of the frozen\-both\-towers class, strictly zero\-shot Clotho transfer, emergent audio–image retrieval at 29×\\timeschance\), then with modality\-gated deep adapters that add in\-layer capacity while keeping text, image, and video outputs bit\-for\-bit identical to the released base \(fusion\-embedding\-2: the family’s best text\-to\-audio results on every benchmark\)\. Along the way we isolated a training\-protocol effect \(\+14\.5 R@10\) that we believe generalizes to any frozen decoder\-LM embedding backbone; a scaling caveat for full\-corpus negative banks past 500K rows; and controlled negative results \(caption rewriting, tower substitution, and connector widening all*reduce*accuracy\) that map where the headroom of frozen\-backbone audio fusion actually lives, and that generation 2 converts into gains\. Future work follows directly: speech and music data to make the instruction taxonomy real, the unsaturated adapter\-rank axis, and the same recipe on the 8B base tier, primarily for the quality of the shared text/image/video space that audio binds into\.

## Reproducibility statement

The training and evaluation pipeline is open source \([https://github\.com/Eximius\-Labs/fusion\-embedding](https://github.com/Eximius-Labs/fusion-embedding)\), and preview checkpoints \(∼\\sim60 MB connector\-only; the frozen towers download from their original repositories\) are released with pinned version tags corresponding to the checkpoints evaluated here\. The generation\-1 model is integrated in themteblibrary, so the MAEB evaluations of §[6\.3](https://arxiv.org/html/2607.18666#S6.SS3)reproduce end\-to\-end from the public package\. Architecture and hyperparameters are specified in §[3](https://arxiv.org/html/2607.18666#S3)–§[5](https://arxiv.org/html/2607.18666#S5)and Appendix[A](https://arxiv.org/html/2607.18666#A1); evaluation protocols and their data sources \(the AudioCaps test crawl, the Clotho v2\.1 Zenodo release, and the public VGGSound retrieval pairing\) in §[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\. Every training run emits a result record containing its configuration, corpus size, loss trace endpoints, thebase\_driftassertion, and an automatically scored AudioCaps protocol result; checkpoints record their base precision so evaluations cannot silently mix precisions\. The full pipeline also runs end\-to\-end on CPU\-only stand\-ins with over 125 unit and integration tests, including the bitwise\-invariance and gate tests of §[6\.4](https://arxiv.org/html/2607.18666#S6.SS4), so the training logic can be verified without GPUs\.

## Acknowledgments

This work builds on outstanding open releases: Qwen3\-VL\-Embedding and Qwen2\.5\-Omni \(the frozen towers\), ImageBind’s emergent\-alignment result, Matryoshka representation learning, and the open audio–caption data ecosystem \(AudioCaps, WavCaps, Clotho, FSD50K, VGGSound, ESC\-50, and LAION\-FreeSound\)\.

## References

- Alayrac et al\. \(2022\)Jean\-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al\. 2022\.Flamingo: a visual language model for few\-shot learning\.*Advances in Neural Information Processing Systems \(NeurIPS\)*\.ArXiv:2204\.14198\.
- Bai et al\. \(2024\)Jisheng Bai et al\. 2024\.Audiosetcaps: An enriched audio\-caption dataset using automated generation pipeline with large audio and language models\.In*NeurIPS 2024 Workshop on Audio Imagination*\.ArXiv:2411\.18953\.
- Bogolin et al\. \(2022\)Simion\-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie\. 2022\.Cross modal retrieval with querybank normalisation\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.ArXiv:2112\.12777\.
- Chen et al\. \(2020\)Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman\. 2020\.Vggsound: A large\-scale audio\-visual dataset\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\.ArXiv:2004\.14368\.
- Chuang et al\. \(2020\)Ching\-Yao Chuang, Joshua Robinson, Yen\-Chen Lin, Antonio Torralba, and Stefanie Jegelka\. 2020\.Debiased contrastive learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.ArXiv:2007\.00224\.
- Dinkel et al\. \(2026\)Heinrich Dinkel, Zhiyong Yan, Tianzi Wang, Yongqing Wang, Xingwei Sun, Yadong Niu, Jizhong Liu, Gang Li, Junbo Zhang, and Jian Luan\. 2026\.Glap: General contrastive audio\-text pretraining across domains and languages\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\.ArXiv:2506\.11350\.
- Dinkel et al\. \(2024\)Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang\. 2024\.Scaling up masked audio encoder learning for general audio classification\.In*Proceedings of Interspeech*\.ArXiv:2406\.06992\.
- Drossos et al\. \(2020\)Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen\. 2020\.Clotho: an audio captioning dataset\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\.ArXiv:1910\.09387\. Evaluation split v2\.1 from Zenodo record 4783391\.
- El Assadi et al\. \(2026\)Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muennighoff, and Kenneth Enevoldsen\. 2026\.MAEB: Massive audio embedding benchmark\.*arXiv preprint arXiv:2602\.16008*\.
- Fonseca et al\. \(2022\)Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra\. 2022\.Fsd50k: An open dataset of human\-labeled sound events\.*IEEE/ACM Transactions on Audio, Speech, and Language Processing*\.ArXiv:2010\.00475\.
- Girdhar et al\. \(2023\)Rohit Girdhar, Alaaeldin El\-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra\. 2023\.Imagebind: One embedding space to bind them all\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.ArXiv:2305\.05665\.
- Gong et al\. \(2023\)Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass\. 2023\.Whisper\-at: Noise\-robust automatic speech recognizers are also strong general audio event taggers\.In*Proceedings of Interspeech*\.ArXiv:2307\.03183\.
- Google DeepMind \(2026\)Google DeepMind\. 2026\.Gemini embedding 2: Our first natively multimodal embedding model\.[https://deepmind\.google/models/gemini/embedding/](https://deepmind.google/models/gemini/embedding/)\.Commercial API, model idgemini\-embedding\-2; evaluated 2026\-07\-09\.
- Hönicke et al\. \(2026\)Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Scott Martens, Saba Sturua, and Han Xiao\. 2026\.jina\-embeddings\-v5\-omni: Geometry\-preserving embeddings via locked aligned towers\.*arXiv preprint arXiv:2605\.08384*\.
- Jiang et al\. \(2024\)Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen\. 2024\.Vlm2vec: Training vision\-language models for massive multimodal embedding tasks\.*arXiv preprint arXiv:2410\.05160*\.The MMEB benchmark\.
- Kim et al\. \(2019\)Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim\. 2019\.Audiocaps: Generating captions for audios in the wild\.In*Proceedings of NAACL\-HLT*\.
- Kim et al\. \(2025\)Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim\. 2025\.Audiocaps 2\.0\.[https://github\.com/cdjkim/audiocaps/blob/master/dataset2\.0/README\.md](https://github.com/cdjkim/audiocaps/blob/master/dataset2.0/README.md)\.Dataset release \(February 2025\) in the AudioCaps repository: 91,256 train / 2,475 val / 4,875 test pairs\.
- Kusupati et al\. \(2022\)Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard\-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi\. 2022\.Matryoshka representation learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.ArXiv:2205\.13147\.
- Li et al\. \(2026\)Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin\. 2026\.Qwen3\-vl\-embedding and qwen3\-vl\-reranker: A unified framework for state\-of\-the\-art multimodal retrieval and ranking\.*arXiv preprint arXiv:2601\.04720*\.
- Liu et al\. \(2024\)Jizhong Liu, Gang Li, Junbo Zhang, Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Yujun Wang, and Bin Wang\. 2024\.Enhancing automated audio captioning via large language models with optimized audio encoding\.In*Proceedings of Interspeech*\.ArXiv:2406\.13275\.
- Mei et al\. \(2024\)Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D\. Plumbley, Yuexian Zou, and Wenwu Wang\. 2024\.Wavcaps: A chatgpt\-assisted weakly\-labelled audio captioning dataset for audio\-language multimodal research\.*IEEE/ACM Transactions on Audio, Speech, and Language Processing*\.ArXiv:2303\.17395\.
- Meng et al\. \(2025\)Rui Meng et al\. 2025\.VLM2Vec\-V2: Advancing multimodal embedding for videos, images, and visual documents\.*arXiv preprint arXiv:2507\.04590*\.The MMEB\-V2 benchmark\.
- Niizumi et al\. \(2025\)Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino\. 2025\.M2d\-clap: Exploring general\-purpose audio\-language representations beyond clap\.*arXiv preprint arXiv:2503\.22104*\.
- Piczak \(2015\)Karol J\. Piczak\. 2015\.Esc: Dataset for environmental sound classification\.In*Proceedings of the 23rd ACM International Conference on Multimedia*\.
- Radford et al\. \(2021\)Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever\. 2021\.Learning transferable visual models from natural language supervision\.In*Proceedings of the 38th International Conference on Machine Learning \(ICML\)*\.ArXiv:2103\.00020\.
- Radford et al\. \(2023\)Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever\. 2023\.Robust speech recognition via large\-scale weak supervision\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*\.ArXiv:2212\.04356\.
- Robinson et al\. \(2021\)Joshua Robinson, Ching\-Yao Chuang, Suvrit Sra, and Stefanie Jegelka\. 2021\.Contrastive learning with hard negative samples\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2010\.04592\.
- Sun and Saenko \(2016\)Baochen Sun and Kate Saenko\. 2016\.Deep coral: Correlation alignment for deep domain adaptation\.In*European Conference on Computer Vision \(ECCV\) Workshops*\.ArXiv:1607\.01719\.
- van den Oord et al\. \(2018\)Aäron van den Oord, Yazhe Li, and Oriol Vinyals\. 2018\.Representation learning with contrastive predictive coding\.*arXiv preprint arXiv:1807\.03748*\.
- Wang et al\. \(2023\)Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou\. 2023\.ONE\-PEACE: Exploring one general representation model toward unlimited modalities\.*arXiv preprint arXiv:2305\.11172*\.
- Wang et al\. \(2024\)Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, and Zhou Zhao\. 2024\.Freebind: Free lunch in unified multimodal space via knowledge fusion\.In*International Conference on Machine Learning \(ICML\)*\.ArXiv:2405\.04883\.
- Wang et al\. \(2025\)Zehan Wang, Ziang Zhang, Minjie Hong, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Shengpeng Ji, Tao Jin, Hengshuang Zhao, and Zhou Zhao\. 2025\.Omnibind: Large\-scale omni multimodal representation via binding spaces\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2407\.11895\.
- Wu et al\. \(2024\)Chengyue Wu et al\. 2024\.LLaMA pro: Progressive LLaMA with block expansion\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.ArXiv:2401\.02415\.
- Wu et al\. \(2023\)Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg\-Kirkpatrick, and Shlomo Dubnov\. 2023\.Large\-scale contrastive language\-audio pretraining with feature fusion and keyword\-to\-caption augmentation\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\.ArXiv:2211\.06687\.
- Xu et al\. \(2026\)Jilan Xu, Carl Thomé, Danijela Horak, Weidi Xie, and Andrew Zisserman\. 2026\.Scaling audio\-text retrieval with multimodal large language models\.*arXiv preprint arXiv:2602\.18010*\.
- Xu et al\. \(2025\)Jin Xu et al\. 2025\.Qwen2\.5\-omni technical report\.*arXiv preprint arXiv:2503\.20215*\.
- Yoo et al\. \(2026\)HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung\-Wan Koo, and Du\-Seong Chang\. 2026\.Omni\-embed\-audio: Leveraging multimodal llms for robust audio\-text retrieval\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.ArXiv:2604\.18360\.
- Zhu et al\. \(2024\)Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan\. 2024\.Languagebind: Extending video\-language pretraining to n\-modality by language\-based semantic alignment\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2310\.01852\.
- Zhu and Duan \(2024\)Ge Zhu and Zhiyao Duan\. 2024\.Cacophony: An improved contrastive audio\-text model\.*IEEE/ACM Transactions on Audio, Speech, and Language Processing*\.ArXiv:2402\.06986\.

## Appendix AHyperparameters

Table 11:Training configuration for the released checkpoints \(§[4\.5](https://arxiv.org/html/2607.18666#S4.SS5)\)\.
## Appendix BTest\-time hubness correction is benchmark\-specific

Querybank normalization \(QB\-Norm\)\(Bogolin et al\.,[2022](https://arxiv.org/html/2607.18666#bib.bib3)\)corrects retrieval hubness at inference time by renormalizing gallery similarities against a bank of query embeddings\. We built the querybank from training captions and used the dynamic inverted softmax variant, selecting the temperature \(β=20\\beta=20\) on the AudioCaps benchmark, where it adds \+1\.75 text\-to\-audio R@10 \(0\.7506→\\to0\.7681\)\. All numbers in this study are the generation\-1 v0\.3 checkpoint under the study’s own evaluation run, whose baselines differ slightly from the release\-protocol cells of Table[4](https://arxiv.org/html/2607.18666#S6.T4)\. Applied*once*, with no further tuning, to Clotho it adds \+0\.9 text\-to\-audio R@1 and \+1\.0 text\-to\-audio R@10 \(0\.4624→\\to0\.4720\); applied to a third benchmark \(MECAT\) it*hurts*:−1\.2\-1\.2R@1 and−1\.7\-1\.7R@10\. We therefore ship it as an optional inference\-time setting, off by default, and no headline number in this paper includes it\. The sign flip is a useful caution: test\-time corrections tuned on one benchmark do not transfer as reliably as their single\-benchmark gains suggest\.

## Appendix CAdapter gate probe

The controlled experiment behind generation 2 \(matched 45K AudioCaps\-only arms, 800 steps, identical recipe, in\-run protocol\): control \(no adapters\) R@1 0\.202 / R@5 0\.484 / R@10 0\.631, text\-to\-audio R@10 0\.684; adapters rank 128 \(14\.8M\) 0\.217 / 0\.512 / 0\.656; adapters rank 384 \(44\.2M\)0\.229/0\.531/0\.665, text\-to\-audio R@100\.708, with a lower final training loss\. The acceptance gate was≥\\geq\+3 R@10; it passed at \+3\.4, every direction improved, and the gain had not saturated at the largest rank trained\. At full scale the pretraining\-stage gain reproduced at the same magnitude \(R@10 0\.708 with adapters at 518K with the large\-scale loss terms vs\. 0\.674 for the 592K baseline without either, matched protocol; the probe above is the controlled isolation of the adapters themselves\)\.

## Appendix DFull AudioCaps retrieval grids

Table 12:Released\-checkpoint AudioCaps grids under the release protocol \(§[6\.1](https://arxiv.org/html/2607.18666#S6.SS1)\)\. Generation\-1 cells that were not tabulated at release time were recovered from the archived release\-time evaluation artifacts \(each artifact reproduces every previously published cell exactly\); no checkpoint was re\-scored\. Remaining dashes: not recorded\.
## Appendix ENegative\-result tables

Table 13:Recaptioning arms \(§[8\.1](https://arxiv.org/html/2607.18666#S8.SS1)\), audio\-to\-text\. The 592K row is the same\-scale baseline; the loss terms favor the rewritten arm by \+2\.6 \(§[5\.4](https://arxiv.org/html/2607.18666#S5.SS4)\), so−2\.5\-2\.5under\-states the rewrite’s cost\. Clotho baseline cells are from the run logs\.Table 14:Tower\-swap A/B \(§[8\.2](https://arxiv.org/html/2607.18666#S8.SS2)\): 45K AudioCaps\-only corpus, 800 steps, identical recipe, each arm on frames from its own tower\. Text\-to\-audio cells are from the run logs\. The towers are not parameter\-matched; the experiment establishes non\-transfer of shallow\-projection rankings, not encoder superiority per parameter\.
## Appendix FInvariance verification protocol

The checks behind §[6\.4](https://arxiv.org/html/2607.18666#S6.SS4), by artifact:fusion\_embedding/adapters\.pyimplements the gate \(a depth\-counted context manager\) and the layer hooks, which return the frozen layer’s output*before*any adapter arithmetic when the gate is closed\.tests/test\_adapters\.pyasserts: \(i\) bitwise text invariance: identical activations through the base and through the adapter\-attached model with the gate closed, and exact identity of a zero\-initialized adapter stack under an open gate; \(ii\) audio\-gated forwards*change*while a text forward in the same process remains bit\-identical; \(iii\) gradient isolation: only adapter and connector parameters receive gradients, and the frozen towers reject them; \(iv\) checkpoint resume round\-trips adapters and refuses adapter\-presence mismatches; \(v\) warm\-start semantics from a connector\-only checkpoint; \(vi\) a gate that fails to span the re\-run forwards of gradient checkpointing raises loudly \(CheckpointError\) rather than silently degrading the guarantee\. The training loop assertsbase\_drift=0=0\(maximum absolute parameter change\) at the end of every run, and the release smoke test loads the packaged checkpoint from the public repository, repeats the text output\-equality check, and exercises the audio, text, and image paths end\-to\-end through the public loading code\.

## Appendix GInput formats, verbatim

The native chat\-template format \(the base’s official protocol; §[4\.2](https://arxiv.org/html/2607.18666#S4.SS2)\), with the task instruction in the system turn and pooling at the final position:

> ``` <|im_start|>system {instruction}<|im_end|> <|im_start|>user {caption}<|im_end|> <|im_start|>assistant ```

The bare format used by our early runs \(off\-manifold for the frozen base;−14\.5\-14\.5R@10 at matched configuration\):

> ``` {instruction} {caption}<|im_end|> ```

An example instruction \(the sound\-retrieval task\): “Retrieve audio by sound description\.” Sequences are truncated at 254 tokens; the embedding is pooled at the last position in both formats\.

## Appendix HMAEB per\-task scores

Table 15:MAEB sound\-event tier,fusion\-embedding\-1v0\.2\-preview withmteb==2\.18\.0: each task’smain\_scoreas defined by the benchmark \(metrics differ per task type\), with public\-leaderboard ranks as of 2026\-07\-09\. Absolute scores on UrbanSound8KT2ARetrieval are low across the leaderboard \(best published 0\.010\); the rank, not the magnitude, is informative there\. FSD2019Kaggle withheld for the containment issue described in §[6\.3](https://arxiv.org/html/2607.18666#S6.SS3)\.