CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation
Summary
CORE introduces a distillation method that transfers compositional ranking judgments from a reranker to an embedding model using a Rank-KL objective, enhancing compositional retrieval performance across benchmarks without compromising standard tasks.
View Cached Full Text
Cached at: 09/04/26, 03:56 AM
Paper page - CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation
Source: https://huggingface.co/papers/2609.04083
Abstract
CORE distills compositional ranking judgments from a cross-attentive reranker into an embedding model via synthesized multi-level candidates and a Rank-KL objective, improving compositional retrieval without degrading standard performance.
MLLM-based embedding modelsremain limited incompositional retrieval, often failing to distinguish scenes containing the same concepts but different attribute-object bindings. Yet the same backbone can resolve such distinctions when used as across-attentive reranker, motivating us to distill its compositional judgments into the embedding model. We propose CORE, which synthesizes candidate lists spanning five compositional matching levels and introduces aRank-KL objectivethat trains the embedding model to reproduce the reranker’s fine-grained ranking. We further introduce a graded evaluation protocol and comparecontrastive learning, pairwiseCoSENT, and listwise Rank-KL under the same data and tuning budget. Our comparison shows that bothCoSENTand Rank-KL use the multi-level supervision more effectively thancontrastive learning, with Rank-KL achieving the strongest overall performance. Across threecompositional reasoning benchmarks(COLA, SUGARCREPE++, NEGBENCH), CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points, while CORE-EMBED-8B achieves the best total average (0.666) among all evaluated embedding models. The improvements transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.04083
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper4
#### Alibaba-NLP/core-reranker-2b Feature Extraction• 2B• Updatedabout 2 hours ago • 1
#### Alibaba-NLP/core-emb-2b Feature Extraction• Updatedabout 2 hours ago • 1
#### Alibaba-NLP/core-reranker-8b Feature Extraction• 9B• Updatedabout 2 hours ago
#### Alibaba-NLP/core-emb-8b Feature Extraction• Updatedabout 2 hours ago
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.04083 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04083 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
This paper presents a method to fine-tune LLaMA 3 8B as an efficient reranker for Retrieval-Augmented Generation using knowledge distillation and 4-bit quantization, achieving 14-21% gains in retrieval metrics over cross-encoder baselines with reduced inference cost.
MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval
MemReranker is a reasoning-aware reranking model family (0.6B/4B) designed for agent memory retrieval, addressing limitations in semantic similarity by incorporating LLM knowledge distillation for better temporal and causal reasoning.
KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking
KaLM-Reranker-V1 is a fast reranker that decouples query and passage computation using an encoder-decoder architecture with Matryoshka embedding pooling and cross-attention, achieving state-of-the-art reranking performance on BEIR and competitive results on multilingual benchmarks.
Enhancing Rubric-based RL via Self-Distillation
Introduces CriPO, a method that enhances rubric-based reinforcement learning for LLMs via on-policy self-distillation, addressing unexplored and suppressed criteria. Experiments on medicine and science benchmarks show stronger final performance with roughly 2x fewer optimization steps.
Learning to Refine Hidden States for Reliable LLM Reasoning
Proposes ReLAR, a reinforcement-guided latent refinement framework that iteratively updates hidden representations in LLMs before decoding, improving reasoning reliability and efficiency compared to chain-of-thought methods.