DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning
Summary
DeepRefine is a research paper introducing an LLM-based reasoning model that refines agent-compiled knowledge bases using reinforcement learning and multi-turn interactions to improve downstream task performance.
View Cached Full Text
Cached at: 05/13/26, 12:20 AM
Paper page - DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning
Source: https://huggingface.co/papers/2605.10488
Abstract
DeepRefine is an LLM-based reasoning model that refines agent-compiled knowledge bases through multi-turn interactions and targeted updates to improve downstream task performance.
Agent-compiled knowledge basesprovide persistent external knowledge for large language model (LLM) agents in open-ended, knowledge-intensive downstream tasks. Yet their quality is systematically limited by incompleteness, incorrectness, and redundancy, manifested as missing evidence or cross-document links, low-confidence or imprecise claims, and ambiguous or coreference resolution issues. Such defects compound under iterative use, degrading retrieval fidelity and downstream task performance. We present DeepRefine, a general LLM-based reasoning model for agent-compiledknowledge refinementthat improves the quality of any pre-constructed knowledge bases with user queries to make it more suitable for the downstream tasks. DeepRefine performsmulti-turn interactionswith the knowledge base and conductsabductive diagnosisover interaction history, localizes likely defects, and executes targeted refinement actions for incremental knowledge base updates. To optimize refinement policies of DeepRefine without gold references, we introduce a Gain-Beyond-Draft (GBD) reward and train the reasoning process end-to-end viareinforcement learning. Extensive experiments demonstrate consistent downstream gains over strong baselines.
View arXiv pageView PDFGitHub2Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.10488 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.10488 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.10488 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Learning to Refine Hidden States for Reliable LLM Reasoning
Proposes ReLAR, a reinforcement-guided latent refinement framework that iteratively updates hidden representations in LLMs before decoding, improving reasoning reliability and efficiency compared to chain-of-thought methods.
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
This paper investigates whether reinforcement learning can improve the direct recall of parametric knowledge in LLMs beyond reasoning tasks. It demonstrates that RL with binary rewards yields significant gains in factual QA benchmarks by redistributing probability mass to unlock latent knowledge rather than acquiring new facts.
Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
A new verifier-free breadth-depth refinement framework improves LLM reasoning at test time by sampling multiple rollouts, iteratively refining each via self-critique, and aggregating with majority voting. It consistently outperforms greedy decoding, majority voting, and verifier-based selection across several math benchmarks and open-weight models.
Retrospective Progress-Aware Self-Refinement for LLM Agent Training
This paper introduces RePro, a framework that trains LLM agents to self-generate progress signals through a forward-then-reflect rollout paradigm, achieving up to 12% absolute success rate gains on WebShop, ALFWorld, and Sokoban benchmarks.
Enhancing Rubric-based RL via Self-Distillation
Introduces CriPO, a method that enhances rubric-based reinforcement learning for LLMs via on-policy self-distillation, addressing unexplored and suppressed criteria. Experiments on medicine and science benchmarks show stronger final performance with roughly 2x fewer optimization steps.