CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation
Summary
CORE introduces a distillation method that transfers compositional ranking judgments from a reranker to an embedding model using a Rank-KL objective, enhancing compositional retrieval performance across benchmarks without compromising standard tasks.
View Cached Full Text
Cached at: 09/04/26, 03:56 AM
Paper page - CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation
Source: https://huggingface.co/papers/2609.04083
Abstract
CORE distills compositional ranking judgments from a cross-attentive reranker into an embedding model via synthesized multi-level candidates and a Rank-KL objective, improving compositional retrieval without degrading standard performance.
MLLM-based embedding modelsremain limited incompositional retrieval, often failing to distinguish scenes containing the same concepts but different attribute-object bindings. Yet the same backbone can resolve such distinctions when used as across-attentive reranker, motivating us to distill its compositional judgments into the embedding model. We propose CORE, which synthesizes candidate lists spanning five compositional matching levels and introduces aRank-KL objectivethat trains the embedding model to reproduce the reranker’s fine-grained ranking. We further introduce a graded evaluation protocol and comparecontrastive learning, pairwiseCoSENT, and listwise Rank-KL under the same data and tuning budget. Our comparison shows that bothCoSENTand Rank-KL use the multi-level supervision more effectively thancontrastive learning, with Rank-KL achieving the strongest overall performance. Across threecompositional reasoning benchmarks(COLA, SUGARCREPE++, NEGBENCH), CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points, while CORE-EMBED-8B achieves the best total average (0.666) among all evaluated embedding models. The improvements transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.04083
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper4
#### Alibaba-NLP/core-reranker-2b Feature Extraction• 2B• Updatedabout 2 hours ago • 1
#### Alibaba-NLP/core-emb-2b Feature Extraction• Updatedabout 2 hours ago • 1
#### Alibaba-NLP/core-reranker-8b Feature Extraction• 9B• Updatedabout 2 hours ago
#### Alibaba-NLP/core-emb-8b Feature Extraction• Updatedabout 2 hours ago
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.04083 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04083 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
This paper presents a method to fine-tune LLaMA 3 8B as an efficient reranker for Retrieval-Augmented Generation using knowledge distillation and 4-bit quantization, achieving 14-21% gains in retrieval metrics over cross-encoder baselines with reduced inference cost.
MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval
MemReranker is a reasoning-aware reranking model family (0.6B/4B) designed for agent memory retrieval, addressing limitations in semantic similarity by incorporating LLM knowledge distillation for better temporal and causal reasoning.
DecisionRank: On-Policy Distillation for Decision-Model Rerankers
Meta's DecisionRank paper introduces item-space on-policy distillation to transfer a generative LLM teacher's ranking knowledge into a single-token decision-model reranker, achieving state-of-the-art results on TREC DL, BEIR, and BRIGHT while running up to 24.5x faster than its generative counterpart and surpassing its 27B teacher at just 1.7B parameters.
RenderRank: Learning to Rerank Text with Compressed Visual Tokens
RenderRank introduces a reranking method that uses compressed visual document representations to reduce token usage and improve efficiency while maintaining accuracy across multiple datasets.
KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking
KaLM-Reranker-V1 is a fast reranker that decouples query and passage computation using an encoder-decoder architecture with Matryoshka embedding pooling and cross-attention, achieving state-of-the-art reranking performance on BEIR and competitive results on multilingual benchmarks.