Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Hugging Face Daily Papers Papers
omni-modal embeddings contrastive-learning multi-modal universal-embeddings state-of-the-art

Summary

The paper introduces Ovis-Embedding, a state-of-the-art omni-modal embedding model that uses a shared backbone to encode text, image, video, and audio in a common representation space, achieving top performance on benchmarks like MMEB-v3 and MVEB.

In this report, we introduce Ovis-Embedding, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make three key advances: (1) native omni-modal initialization: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) data-centric omni-modal training: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) embedding-specific training and inference optimization: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the Ovis-Embedding family achieves state-of-the-art performance on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.
Original Article
View Cached Full Text

Cached at: 09/23/26, 03:31 AM

Paper page - Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Source: https://huggingface.co/papers/2609.25165

Abstract

Inthisreport,weintroduceOvis-Embedding,astate-of-the-artomni-modalembeddingfamilybuiltonnativeintegrationoftext,image,video,andaudio.Insteadofassemblingseparatemodalitytowers,Ovis-Embeddingusesasharedmultimodalbackbonetoencodedifferentmodalitiesinacommonrepresentationspace.Specifically,wemakethreekeyadvances:(1)nativeomni-modalinitialization:weadoptapretrainedQwen-omnimodelastheembeddingbackboneandadaptitthroughcontrastivetrainingwithlow-rankinitialization;(2)data-centricomni-modaltraining:weconstructabroad,high-qualitycorpusspanningtext,images,video,audio,andinterleavedmultimodaldata.Toimprovedataefficiency,weintroducehomogeneous-sourcesamplingtoformtask-consistentbatcheswithinformativein-batchnegatives;and(3)embedding-specifictrainingandinferenceoptimization:weusefocallosstoemphasizehardexamplesandsimilarity-basedEmbeddingDistillationtotransferfine-grainedsimilaritystructurefromcomplementaryexperts.Atinferencetime,low-rankfeaturedecompositionenablescompactembeddingswithflexibledimensionalityandminimalperformanceloss.EmpiricalevaluationsshowthattheOvis-Embeddingfamilyachievesstate-of-the-artperformanceonMMEB-v3,MMEB-v2,MVEB,MAEB,andRTEB,demonstratingitseffectivenessacrosstext,image,video,andaudiomodalities.Theseresultshighlightthepotentialofunifiedomni-modaltrainingtoovercomemodalityfragmentationandadvanceuniversalembeddingmodelsforany-to-anyretrieval.

View arXiv pageView PDFGitHub19Add to collection

Get this paper in your agent:

hf papers read 2609\.25165

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.25165 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.25165 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.25165 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MVEB: Massive Video Embedding Benchmark

Hugging Face Daily Papers

This paper introduces MVEB, a large-scale benchmark for evaluating video embeddings across 23 tasks, finding that no single model dominates and that audio's contribution depends on dataset annotation provenance. It integrates into the MTEB ecosystem for unified multimodal evaluation.

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Hugging Face Daily Papers

UEmbed is a decoder-only multimodal embedding model that produces both sparse and dense representations in a single forward pass, released at 2B, 4B, and 9B scales. It outperforms existing public-data-trained multimodal embedding models on MMEB-v2 and remains competitive on BEIR.