It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Hugging Face Daily Papers Papers
generative-retriever reinforcement-learning llm information-retrieval keyword-matching co-optimization

Summary

CoGR is a retrieval framework that uses co-evolving reinforcement learning to train LLMs for generating keywords in both query and item sides, achieving significant performance improvements in retrieval tasks.

Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leverages LLMs to improve retrieval through query expansion, data synthesis, and retrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduce CoGR, a retrieval framework that instead trains LLMs to directly construct retrieval representations on both query and item sides. Each generator produces a compact set of keywords, which are matched directly through an inverted index, preserving compatibility with existing keyword-based retrieval infrastructure. CoGR uses a two-stage training pipeline. Supervised fine-tuning first establishes an aligned keyword space, after which co-evolving reinforcement learning alternately optimizes the query- and item-side generators with GRPO against the opposite side's frozen index. Both sides optimize the same query-to-item retrieval F_1 objective: the query side receives retrieval F_1 directly, while the item side receives a counterfactual marginal reward measuring the change in query-side F_1 caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines, CoGR achieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving F_1 over the strongest baseline by 10.9% and 36.1%, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.
Original Article
View Cached Full Text

Cached at: 09/03/26, 03:52 AM

Paper page - It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Source: https://huggingface.co/papers/2609.00638

Abstract

CoGR trains LLMs to generate compact keywords for both queries and items, enabling direct inverted-index retrieval optimized via co-evolving reinforcement learning.

Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leveragesLLMsto improve retrieval throughquery expansion, data synthesis, andretrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduceCoGR, a retrieval framework that instead trainsLLMsto directly constructretrieval representationson both query and item sides. Each generator produces a compact set of keywords, which are matched directly through aninverted index, preserving compatibility with existing keyword-based retrieval infrastructure.CoGRuses a two-stage training pipeline.Supervised fine-tuningfirst establishes an aligned keyword space, after whichco-evolving reinforcement learningalternately optimizes the query- and item-side generators withGRPOagainst the opposite side’s frozen index. Both sides optimize the same query-to-item retrieval F_1 objective: the query side receives retrieval F_1 directly, while the item side receives acounterfactual marginal rewardmeasuring the change in query-side F_1 caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines,CoGRachieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving F_1 over the strongest baseline by 10.9% and 36.1%, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.00638

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.00638 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.00638 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.00638 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Reciprocal Co-Training (RCT): Coupling Gradient-Based and Non-Differentiable Models via Reinforcement Learning

arXiv cs.CL

Researchers from Fordham University introduce Reciprocal Co-Training (RCT), a framework that couples LLMs and Random Forest classifiers via reinforcement learning, creating an iterative feedback loop where each model improves using signals from the other. Experiments on three medical datasets show consistent performance gains for both models, demonstrating a general mechanism for integrating incompatible model families.

Self-Evolving Deep Research via Joint Generation and Evaluation

arXiv cs.CL

Researchers from HKUST, ByteDance, and UCL propose SCORE, a co-evolutionary training framework that jointly trains an LLM as both a deep research report generator and an evaluator, using a meta-harness to dynamically adjust evaluation difficulty and prevent reward saturation. Experiments show consistent improvement in open-ended research report quality.