Douyin Multimodal Embedding Model Technical Report

Hugging Face Daily Papers Papers
multimodal-embedding technical-report douyin contrastive-learning search-recommendation

Summary

This technical report introduces Douyin Multimodal Embedding (DME), a two-stage trained model that combines contrastive pre-training with evidence-grounded latent reasoning and cross-conditional reconstruction, achieving state-of-the-art results on MMEB-v2 and deployment in Douyin search.

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.
Original Article
View Cached Full Text

Cached at: 08/10/26, 06:14 AM

Paper page - Douyin Multimodal Embedding Model Technical Report

Source: https://huggingface.co/papers/2608.02148

Abstract

MultimodalrepresentationlearningisacornerstoneofmodernAI.Byencodingmultimodalqueriesandtargetsintovectors,itpowersindustrialsearchandrecommendationandunderpinsmodernagents.Real-worldplatformswithcomplexmodalitiesandmassive-scalecontent,suchasDouyin,Xiaohongshu,andYouTube,demandbothefficiencyunderbillion-scaleindexingandfine-graineddiscriminationforhardmatching.ExistingMLLMembeddingmodelsrarelysatisfyboth.Contrastivemodelsareefficientbutrelyonpair-levelsupervisiontoocoarseforfine-graineddistinctions,whileCoT-basedmodelsimprovediscriminationthroughexplicitgenerationimpracticaltoserveonline.WepresentDouyinMultimodalEmbedding(DME),amodeltrainedintwostagestocombinebothstrengths.Stage1performslarge-scalecontrastivepre-trainingthatestablishesaunifiedmultimodalembeddingspacewithbroadmodalityandtaskcoverage.Stage2supplementssemanticsufficiency,thepropertythatanembeddingisgroundedinretrieval-relevantevidenceandpreservesfine-grainedcounterpart-sidesemantics,viatwomechanisms.Evidence-GroundedTypedLatentReasoningorganizesretrievalevidencethroughhidden-spacelatentreasoning,andCross-ConditionalReconstructionenforcescounterpart-sidesemanticsthroughcross-directionalautoregressivereconstruction.Bothactonlyduringtrainingandaddonlymarginalquery-sideoverhead,soDMEservesasefficientlyasastandardcontrastiveencoder.OnMMEB-v2,DMEreachesstate-of-the-artresultsatcomparablescalesforits2Band9Bvariants(74.8and78.4),withespeciallystrongvideoandvisual-documenttasks.Inproduction,DMEdeliversa2.92%relativegainonDouyin’sin-houseofflineevaluationset,isdeployedacrossDouyinscenariossuchasgenerative,image,andAIsearch,andyieldsa0.1%Lifetime(LT)gaininonlineA/BtestingonDouyinsearch.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.02148 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.02148 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.02148 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Hugging Face Daily Papers

UEmbed is a decoder-only multimodal embedding model that produces both sparse and dense representations in a single forward pass, released at 2B, 4B, and 9B scales. It outperforms existing public-data-trained multimodal embedding models on MMEB-v2 and remains competitive on BEIR.

Kwai Keye-VL-2.0 Technical Report

Hugging Face Daily Papers

This technical report presents Kwai Keye-VL-2.0, an open-source Mixture-of-Experts multimodal foundation model designed for long-video understanding and agentic intelligence, leveraging DeepSeek Sparse Attention and cross-modal distillation to achieve state-of-the-art performance among similar-scale models.

MVEB: Massive Video Embedding Benchmark

Hugging Face Daily Papers

This paper introduces MVEB, a large-scale benchmark for evaluating video embeddings across 23 tasks, finding that no single model dominates and that audio's contribution depends on dataset annotation provenance. It integrates into the MTEB ecosystem for unified multimodal evaluation.

LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

arXiv cs.LG

LongMoE proposes a unified framework that jointly addresses modality missingness and longitudinal dynamics in multimodal clinical learning, using context-aware imputation, attentional tokenization, trajectory-aware encoding, and sparse mixture-of-experts routing. Experiments on ADNI, OASIS-3, and MIMIC-IV demonstrate improved robustness under missing modalities while remaining competitive in full-modality settings.