OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models

# OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models
Source: [https://arxiv.org/html/2604.19766](https://arxiv.org/html/2604.19766)
Haijian Liang1,Zenghao Niu1,Junjie Wu2, Changwang Zhang2,Wangchunshu Zhou2,Jun Wang2, 1Shenzhen University, 2OPPO Research Institute changwangzhang@foxmail\.com

###### Abstract

Retrieval\-Augmented Generation \(RAG\) expands the knowledge of Large Language Models \(LLMs\), yet current static retrieval methods struggle with complex, multi\-hop problems\. While recent dynamic retrieval strategies offer improvements, they face two key challenges: 1\) irrelevant retrieved noise can misdirect the reasoning process, and 2\) processing full documents incurs prohibitive computational and latency costs\. To address these issues, we propose OThink\-SRR1, a framework that enhances large models with an iterative Search\-Refine\-Reason process trained via reinforcement learning\. Its core Refine stage distills retrieved documents into concise, relevant facts before reasoning\. We introduce GRPO\-IR, an end\-to\-end reinforcement learning algorithm that rewards accurate evidence identification while penalizing excessive retrievals, thus training the model to be both focused and efficient\. Experiments on four multi\-hop QA benchmarks show our approach achieves superior accuracy over strong baselines while using fewer retrieval steps and tokens\. This positions OThink\-SRR1 as a potent foundational model for information\-seeking agents\.

OThink\-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models

Haijian Liang1, Zenghao Niu1, Junjie Wu2,Changwang Zhang2,Wangchunshu Zhou2,Jun Wang2,1Shenzhen University,2OPPO Research Institutechangwangzhang@foxmail\.com

## 1Introduction

Large language models \(LLMs\) have achieved remarkable results in natural language understanding and text generation through large\-scale pre\-trainingMatarazzo and Torlone \([2025](https://arxiv.org/html/2604.19766#bib.bib5)\)\. However, relying solely on their internal knowledge often leads to hallucinations, outdated facts, and knowledge gaps\. Retrieval\-Augmented Generation \(RAG\) addresses this by integrating external search engines or knowledge bases into the generation process, grounding outputs in real\-world evidenceLewiset al\.\([2020](https://arxiv.org/html/2604.19766#bib.bib28)\)\. Most RAG systems follow either a single\-step “retrieve–reason” paradigm or a multi\-step “plan–retrieve–reason” paradigm with a fixed retrieval planLeeet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib17)\)\. Although these methods improve factuality, they cannot adapt dynamically to intermediate findings, which can result in redundant searches and suboptimal use of retrieved documents\.

Recent advances in reasoning\-oriented LLMs, such as DeepSeek\-R1Guoet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib6)\), OpenAI\-o1OpenAI \([2024](https://arxiv.org/html/2604.19766#bib.bib8)\), and Qwen\-QwQQwen Team \([2025](https://arxiv.org/html/2604.19766#bib.bib12)\), have greatly enhanced chain\-of\-thought planning and tool invocation\. Stronger reasoning capabilities enable precise decomposition of complex queries, dynamic adjustment of subsequent search steps based on partial evidence, and more effective synthesis and conflict resolution across multiple documents\. This synergy has led to dynamic reasoning–retrieval model & frameworks such as R1\-SearcherSonget al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib16)\), Search\-R1Jinet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib15)\), and ReSearchChenet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib14)\), which interleave reasoning and search actions for more flexible information gathering\.

However, existing dynamic retrieval–reasoning methods face two main challenges\.First, unfiltered retrieval results often contain substantial irrelevant content, which can mislead the reasoning process—over 60% of retrievals introduce errorsWuet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib1)\), and the utility of added context declines as prompt length growsTianet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib2)\), especially for smaller modelsFanget al\.\([2024b](https://arxiv.org/html/2604.19766#bib.bib4)\)\.Second, appending full\-text documents to the prompt dramatically increases token usage and computation cost, degrading inference efficiency and latencyZhanget al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib3)\)\. Addressing both noise suppression and inference efficiency is essential for truly effective dynamic retrieval reasoning\.

To address these challenges, we propose OThink\-SRR1, a novel reasoning\-retrieval model & framework, enhancing the iterativeSearch,Refinement andReasoning \(OThink\-SRR1\) capability of large reasoning models through reinforcement learning\. Specifically, once the retrieval results are obtained, they are immediately refined to extract the key information most relevant to the current context\. The original documents are then removed so that subsequent reasoning focuses only on this refined content to stablish a robust foundation for subsequent reasoning\. We also design a reward function that provides rewards when the ground truth is found in the last retrieved document, its refined summary, and the final answer, and penalizes excessive retrievals\. This encourages the model to generate more precise search queries, improve retrieval efficiency, and refine information to filter out noise, supporting more focused reasoning and leading to higher answer accuracy\. We train OThink\-SRR1 on Qwen2\.5\-7B\-Instruct & Qwen2\.5\-3B\-Instruct and evaluate it on four standard multi\-hop QA benchmarks\. Experimental results show that OThink\-SRR1 achieves higher answer accuracy and better retrieval efficiency than baseline methods\. In multi\-round settings, OThink\-SRR1 also completes tasks with lower generation costs\. In summary, we make the following contributions:

- •We propose the OThink\-SRR1, which enhances the iterative Search\-Refine\-Reason capability of large reasoning models through reinforcement learning, establishing a foundational model for agents with information retrieval capabilities\. To our best knowledge, OThink\-SRR1 is the first large search reasoning model that internalizes and optimizes the ability to refine retrieval content through end\-to\-end training\.
- •We propose the RL algorithm GRPO\-IR, which rewards accurate hits of the ground truth in the most recently retrieved documents, its refined summary, and the final response, while also penalizing excessive retrievals\. GRPO\-IR trains the large reasoning model to formulate more precise queries, enhance efficiency, and maintain focus on refined information\.
- •We systematically evaluate the OThink\-SRR1 on multihop question answering benchmarks\. Our model demonstrates superior performance compared to the latest baseline models in terms of EM/F1, while also decreasing the average retrieval count and total token usage\.

## 2Related Work

### 2\.1Retrieval Augmented Generation

Since being introduced, Retrieval\-Augmented Generation \(RAG\) technology has seen substantial advancements\. Early works focused on establishing an end\-to\-end framework for zero\-shot QADu and Ji \([2022](https://arxiv.org/html/2604.19766#bib.bib29)\)\. With the growth in multimodal demands, the MuRAG system first achieved cross\-modal knowledge fusionChenet al\.\([2022](https://arxiv.org/html/2604.19766#bib.bib30)\)\. Recent studies have made breakthroughs in dynamic retrieval strategiesSuet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib31)\), long\-tail knowledge augmentationLiet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib32)\), and noise\-resistant trainingFanget al\.\([2024a](https://arxiv.org/html/2604.19766#bib.bib33)\)\. Among these, the LongRAGE system improved long\-text question\-answering accuracy by 17% through dual\-perspective attention distillationZhaoet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib34)\), and the Provenance framework innovatively combined natural language inference models for factual verification of generated resultsSankararamanet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib35)\)\.

As RAG frameworks continue to evolve to handle increasingly complex information needs,multi\-hop retrievalhas emerged as a crucial extension for reasoning across multiple pieces of evidence\. Early research focused on constraint query processing in knowledge graphs, such as the KG embedding method proposed inMitraet al\.\([2022](https://arxiv.org/html/2604.19766#bib.bib36)\)\. The generative multi\-hop retrieval model GMRLeeet al\.\([2022](https://arxiv.org/html/2604.19766#bib.bib37)\)optimized resource usage through text sequence generation\. Dense\-ATOMICShenet al\.\([2023](https://arxiv.org/html/2604.19766#bib.bib38)\)enhanced multi\-hop path coverage by constructing dense knowledge graphs\. Although RAG significantly expands the knowledge boundaries of LLMs, but current approaches primarily rely on single static retrievals, which limits LLMs’ ability to handle complex tasks such as multi\-hop reasoning\.

### 2\.2Retrieval for Reasoning in Large Language Models

Recent research has explored various strategies to enhance the reasoning abilities of large language models \(LLMs\) by combining them with retrieval\-augmented generation\. Several works focus on modeling the retrieval process as a sequence of adaptive decisions, such as RATWanget al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib20)\), which iteratively revises reasoning steps using relevant retrieved information to reduce hallucination\. Other frameworks, including ReARTeRSunet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib21)\)and AutoRAGYuet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib22)\), introduce methods like factuality scoring, process explanation, or autonomous multi\-turn retrieval planning to further improve the integration of external information and the overall reliability of reasoning\.

More recent approaches have adopted reinforcement learning \(RL\) to help LLMs actively learn when and how to interact with search engines during multi\-step reasoning\. For example, Search\-R1Jinet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib15)\), R1\-SearcherSonget al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib16)\), and ReSearchChenet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib14)\)present RL\-based frameworks that allow LLMs to autonomously generate and refine search queries, select relevant information, and make step\-wise decisions throughout the reasoning chain\. These methods demonstrate that outcome\-based rewards and the integration of retrieval actions within the reasoning process can significantly boost performance on complex question answering and knowledge\-intensive tasks\. However, current retrieval\-reasoning models still suffer from issues such as excessive retrieval noise and high computational costs during inference\.

## 3Method

### 3\.1Motivation

Current RAG systems suffer from noisy retrievals and growing inefficiency\. Irrelevant or incorrect documents can disrupt generation: over 60% of retrievals introduce errorsWuet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib1)\), and relevance utility declines as context growsTianet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib2)\)\. Excessive content also distracts LLMs and wastes computation, degrading performanceZhanget al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib3)\)\. To address this, we introduce step‐wise reward for retrieval\-reasoning modelsChenet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib14)\); Jinet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib15)\); Songet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib16)\): after each retrieval, the LLM filters and retains only the most relevant documents, shortening context, boosting accuracy, and enhancing robustness\.

![Refer to caption](https://arxiv.org/html/2604.19766v1/x1.png)Figure 1:Overview of our OThink\-SRR1 framework\. At each step the model \(1\) formulates a reasoning prompt \("Think"\), \(2\) issues a retrieval query \("Query"\), \(3\) condenses retrieved content into a concise answer \("Refine"\) to extract the key information most relevant to the current scenario\. This process is iteratively executed until sufficient information is obtained to derive the final answer\. Training is guided by rewards from a reference model and a stepwise reward function\.
### 3\.2Search, Refine and Reasoning with Reinforcement Learning

In this work, we introduce OThink\-SRR1, a framework for multi\-step Search\-Refine\-Reasoning integration in multi\-hop retrieval tasks\. Figure[1](https://arxiv.org/html/2604.19766#S3.F1)provides an overview of our framework\. Given a complex questionqq, our approach enables the model to autonomously develop search strategies, execute multiple rounds of querying with subsequent information refinement, and progressively accumulate essential knowledge until the final answer can be derived\. Unlike other retrieval\-reasoning methodsSonget al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib16)\); Chenet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib14)\); Jinet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib15)\), OThink\-SRR1 distinguishes itself through two key features: \(1\) dynamic generation of high\-quality retrieval queries during reasoning, and \(2\) a dedicated refinement step that systematically filters retrieved information to maximize relevance, which significantly improves the signal\-to\-noise ratio\. OThink\-SRR1 requires models to demonstrate four core competencies: problem decomposition, retrieval strategy planning, information filtering, and evidence synthesis\.

To achieve the aforementioned goals, we formulate the OThink\-SRR1 task as a sequential decision\-making problem, optimized via a reinforcement learning framework based on Group Relative Policy Optimization \(GRPO\)\. Unlike the original GRPO, we have modified the reward mechanism in consideration of the task’s characteristics: instead of relying on a reward model, we use an explicit reward function that is easier to control and measure\. This approach not only reduces dependence on step\-by\-step human annotations or external reward models but also facilitates flexible adjustment of the reward strategy based on task requirements, thus efficiently driving the model to discover optimal solution strategies\.

#### 3\.2\.1Training System Prompt

System Prompt For Instruction\-Tuned ModelYou are a helpful assistant who employs a step\-by\-step approach using the Wikipedia search tool\. Each step must be based ONLY on currently known information and lead to either a new search query or a final answer\. Continue this sequential process until you have sufficient information to answer\.You must STRICTLY follow the output format below:1\. Place all reasoning, thinking, explanations, and process language INSIDE <think\>…</think\> tags\.2\. Place query INSIDE <query\>…</query\> tags for wikipedia search\.3\. After receiving search results \(in <documents\>…</documents\>\), based on currently known information, extract information relevant to the user question and current query INSIDE <documents\_refine\>…</documents\_refine\> tags\.4\. Place the final, direct, and concise answer ONLY INSIDE <answer\>…</answer\> tags, and wrap the direct, concise result WITHIN \\boxed\{\}\.5\. Do not output any text outside the <think\>, <query\>, <documents\_refine\>, and <answer\> tags\.For example:<think\>This is your thinking process\.</think\> <query\>This is a search query for retrieval\.</query\> <documents\>This is the search result\.</documents\> <documents\_refine\>This is the refined information extracted from the documents that are most relevant to the query and enough to promote the next step\.</documents\_refine\> <think\>Further reasoning, explanation or search\.</think\> <answer\>The final answer is \\boxed\{exact answer here\}</answer\>Figure 2:System Prompt for instruction\-tuned model\.To ensure that the model consistently generates outputs in a well\-defined structured format, we design prompts that explicitly specify the required output tags, as shown in Figure[2](https://arxiv.org/html/2604.19766#S3.F2)\. In complex, multi\-step reasoning tasks that require information retrieval, the model’s internal thought process can be opaque, making it difficult to diagnose the root cause of errors\. Conventional free\-form text outputs exacerbate this challenge, as they are notoriously difficult to parse and evaluate automatically\. Our structured format addresses this directly by requiring the model to externalize its reasoning into distinct, machine\-readable components\. Specifically, the model is instructed to use <query\>…</query\> tags to initiate retrieval operations and <documents\_refine\>…</documents\_refine\> tags to present refined information extracted from the retrieved documents\. Furthermore, to enable precise Exact Match \(EM\) evaluation, the final answer is enclosed within a \\box\{\} environment\. This design makes the final answer trivial to extract, eliminating ambiguity and facilitating robust, automated performance measurement\.

This structured output format serves two main purposes\. First, it allows for the automated evaluation of the model’s intermediate reasoning and retrieval steps, creating a transparent audit trail of its problem\-solving process\. By parsing the structured output, we can programmatically assess the quality of each step—for example, by evaluating the relevance of a <query\> or the faithfulness of the information in <documents\_refine\> to the source documents\. This capability makes detailed error analysis and performance tracking possible, allowing us to move beyond simply knowing if a final answer is correct to understanding why the model succeeded or failed\. This detailed insight is invaluable for targeted model improvement and debugging\.

Second, this format provides clear and fine\-grained feedback at different stages of the reasoning process, which is crucial for reinforcement learning \(RL\)\. In many complex tasks, a reward signal based solely on the final answer is often too sparse and delayed to be effective\. Our approach allows for a more sophisticated, multi\-faceted reward function that considers: format correctness, retrieval accuracy, refinement quality, final answer accuracy, and retrieval efficiency\. This design enables the model to receive explicit and interpretable feedback for each discrete action it takes, such as formulating a query or synthesizing information\. By providing immediate rewards for successful intermediate steps, we can guide the model toward more effective reasoning strategies and systematically improve its performance on these critical sub\-tasks\.

#### 3\.2\.2GRPO‐IR

Multi‐hop retrieval tasks require models to chain together information from several documents, but intermediate retrieval and reasoning steps usually lack direct supervision\. To address this, we adapt Group Relative Policy Optimization \(GRPO\)Shaoet al\.\([2024](https://arxiv.org/html/2604.19766#bib.bib7)\)by inserting a non‐differentiable information retrieval \(IR\) operation into the autoregressive loop\.

Concretely, letq∼𝒬q\\sim\\mathcal\{Q\}be an input question drawn from distribution𝒬\\mathcal\{Q\}\. At each dedicated “<query\>” token, the policyπθ\\pi\_\{\\theta\}issues a search to an external retriever and appends the top\-KKdocuments to the generation context\. These retrieved tokens are treated as non\-differentiable during training: we block gradients through them and exclude them from importance\-weight ratios and KL computations\.

We then updateπθ\\pi\_\{\\theta\}by maximizing the following clipped, KL‐regularized objective over groups ofGGsampled trajectories\{ai\}i=1G\\\{a\_\{i\}\\\}\_\{i=1\}^\{G\}:

𝒥\\displaystyle\\mathcal\{J\}\(θ\)=𝔼q∼𝒬,\{ai\}i=1G∼πθold\(⋅∣q\)\\displaystyle\(\\theta\)=\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\},\\left\\\{a\_\{i\}\\right\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old \}\}\}\(\\cdot\\mid q\)\}\(1\)1G∑i=1G\[min\(πθ​\(ai∣q;I​R\)πθold​\(ai∣q;I​R\)Ai,\\displaystyle\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\Biggl\[\\min\\Biggl\(\\frac\{\\pi\_\{\\theta\}\\left\(a\_\{i\}\\mid q;IR\\right\)\}\{\\pi\_\{\\theta\_\{\\text\{old \}\}\}\\left\(a\_\{i\}\\mid q;\{IR\}\\right\)\}A\_\{i\},clip\(πθ​\(ai∣q;I​R\)πθold​\(ai∣q;I​R\),1−ϵ,1\+ϵ\)Ai\)\\displaystyle\\operatorname\{clip\}\\left\(\\frac\{\\pi\_\{\\theta\}\\left\(a\_\{i\}\\mid q;\{IR\}\\right\)\}\{\\pi\_\{\\theta\_\{\\text\{old \}\}\}\\left\(a\_\{i\}\\mid q;\{IR\}\\right\)\},1\-\\epsilon,1\+\\epsilon\\right\)A\_\{i\}\\Biggr\)−β𝔻KL\(πθ\|\|πθref\)\]\\displaystyle\-\\beta\\mathbb\{D\}\_\{\\text\{KL \}\}\\left\(\\pi\_\{\\theta\}\|\|\\pi\_\{\\theta\_\{\\text\{ref \}\}\}\\right\)\\Biggr\]
Eachaia\_\{i\}is a full generated sequence, including reasoning tokens, query tokens, retrieved documents, and refinements and𝐫=\{ri\}i=1G\\mathbf\{r\}=\\\{r\_\{i\}\\\}\_\{i=1\}^\{G\}are the rewards computed by our hierarchical reward function\. Here,θ\\thetais the current policy,θold\\theta\_\{\\rm old\}the policy at the start of the update, andθref\\theta\_\{\\rm ref\}a fixed “reference” policy used for KL regularization,ϵ\\epsilonis the clipping parameter \(e\.g\.,0\.20\.2\),β\\betacontrols the weight of the KL penalty,β\\betacontrols the weight of the KL penalty,KKis the number of top documents retrieved at each query step,GGis the group size \(number of trajectories\) used to compute relative advantages, andAi=r~i=ri−mean⁡\(𝐫\)std⁡\(𝐫\)A\_\{i\}=\\widetilde\{r\}\_\{i\}=\\frac\{r\_\{i\}\-\\operatorname\{mean\}\(\\mathbf\{r\}\)\}\{\\operatorname\{std\}\(\\mathbf\{r\}\)\}\.

In practice, for eachqqwe \(1\) sampleGGfull sequences underπθold\\pi\_\{\\theta\_\{\\rm old\}\}, \(2\) compute per‐sequence rewardsrir\_\{i\}with our reward function \(see Section[3\.2\.3](https://arxiv.org/html/2604.19766#S3.SS2.SSS3)\), \(3\) standardize them to obtain advantagesAiA\_\{i\}, and \(4\) take a gradient step on𝒥​\(θ\)\\mathcal\{J\}\(\\theta\)\.

#### 3\.2\.3Reward Function

We design a hierarchical and interpretable reward function that decomposes the complex reasoning task into distinct sub\-goals: format correctness, retrieval accuracy, and answer relevance\. The total reward is structured as a sum of bonuses awarded for successfully completing each step\.

First, outputs must strictly adhere to the required format \(e\.g\., <think\>…</think\>\)\. Failure to do so results in a reward of zero, enforcing structural integrity\. For correctly formatted outputs, the reward calculation proceeds as follows: The model receives a base reward \(βb​a​s​e\\beta\_\{base\}\) for any valid attempt\. A significant retrieval bonus \(βr​e​t​r​i​e​v​a​l\\beta\_\{retrieval\}\) is added if the last retrieved document contains the ground truth \(GT\)\. Crucially, to encourage the model to not only find but also correctly utilize the information, an additional refinement bonus \(βr​e​f​i​n​e\\beta\_\{refine\}\) is granted if the last refined content also contains the GT\. This layered bonus system explicitly incentivizes effective information synthesis beyond simple retrieval\.

Alongside correctness, the reward function also incorporates retrieval efficiency, scored byrc​o​u​n​tr\_\{count\}\. This component encourages the model to minimize the number of retrievals\. We define a maximum efficiency reward,ηm​a​x\\eta\_\{max\}, awarded when the number of retrieval stepsCCis at or below an ideal thresholdCs​t​a​r​tC\_\{start\}\. Beyond this threshold, the reward linearly decays to a minimum efficiency reward,ηm​i​n\\eta\_\{min\}, asCCapproaches the upper limitCm​a​xC\_\{max\}\.

The overall rewardrris then determined by combining these components\. We define three indicator variables:

- •F: equals 1 if the output format is correct, 0 otherwise\.
- •D: equals 1 if the last retrieved document contains the ground truth \(GT\), 0 otherwise\.
- •R: equals 1 if the last refined content contains GT, 0 otherwise\.

r=\{βb​a​s​e\+βr​e​t​r​i​e​v​a​l\+βr​e​f​i​n​e\+f1\+rcount,if​F=D=R=1βb​a​s​e\+βr​e​t​r​i​e​v​a​l\+rcount,if​F=D=1,R=0βb​a​s​e,if​F=1,D=00,if​F=0\\displaystyle r=\\begin\{cases\}\\beta\_\{base\}\+\\beta\_\{retrieval\}\+\\beta\_\{refine\}\+\\text\{f1\}\+r\_\{\\text\{count\}\},&\\text\{if \}F=D=R=1\\\\ \\beta\_\{base\}\+\\beta\_\{retrieval\}\+r\_\{\\text\{count\}\},&\\text\{if \}F=D=1,R=0\\\\ \\beta\_\{base\},&\\text\{if \}F=1,D=0\\\\ 0,&\\text\{if \}F=0\\\\ \\end\{cases\}

\(2\)
Here, f1 denotes the F1 score between the model’s final answer and the GT, computed as:

f1=2×mp\+q\\text\{f1\}=\\frac\{2\\times m\}\{p\+q\}\(3\)whereppandqqare the word counts of the predicted and reference answers, respectively, andmmis the number of overlapping words between the two\. This metric captures the similarity between the two answers by considering both precision and recall\. The efficiency scorerc​o​u​n​tr\_\{count\}is defined as:

rc​o​u​n​t=\{ηm​a​x,if​C≤Cs​t​a​r​tηm​i​n\+\(ηm​a​x−ηm​i​n\)​Cm​a​x−CCm​a​x−Cs​t​a​r​t,if​Cs​t​a​r​t<C<Cm​a​xηm​i​n,if​C≥Cm​a​x\\displaystyle r\_\{count\}=\\begin\{cases\}\\eta\_\{max\},&\\text\{if \}C\\leq C\_\{start\}\\\\ \\eta\_\{min\}\+\(\\eta\_\{max\}\-\\eta\_\{min\}\)\\frac\{C\_\{max\}\-C\}\{C\_\{max\}\-C\_\{start\}\},&\\text\{if \}C\_\{start\}<C<C\_\{max\}\\\\ \\eta\_\{min\},&\\text\{if \}C\\geq C\_\{max\}\\\\ \\end\{cases\}

\(4\)
In our experiments, the reward hyperparameters are set toβb​a​s​e=0\.1\\beta\_\{base\}=0\.1,βr​e​t​r​i​e​v​a​l=0\.2\\beta\_\{retrieval\}=0\.2,βr​e​f​i​n​e=0\.3\\beta\_\{refine\}=0\.3,ηm​a​x=0\.2\\eta\_\{max\}=0\.2, andηm​i​n=0\\eta\_\{min\}=0\. This configuration creates a clear reward hierarchy that prioritizes correct refinement over simple retrieval, and concise reasoning over excessive steps\. The modular design also allows for future work to systematically tune the relative importance of each reasoning component\.

The reward function thus considers the accuracy of the final answer, the efficiency in retrieval steps, and the format correctness, promoting both effectiveness and efficiency in the model’s generation process\.

## 4Experiments

### 4\.1Experimental Setting

To evaluate the effectiveness of OThink\-SSR1, we conducted extensive experiments on multi\-hop QA benchmarks that require multiple information retrievals\. Our OThink\-SSR1 models were trained using Qwen2\.5\-7B\-Instruct and Qwen2\.5\-3B\-Instruct\. During the training process, we exclusively utilized data from the MuSiQue training set\.

BenchmarksWe evaluated on four multi\-hop QA benchmarks: HotpotQAYanget al\.\([2018](https://arxiv.org/html/2604.19766#bib.bib23)\), 2WikiMultiHopQAHoet al\.\([2020](https://arxiv.org/html/2604.19766#bib.bib24)\), MuSiQueTrivediet al\.\([2022](https://arxiv.org/html/2604.19766#bib.bib25)\), and BambooglePresset al\.\([2023](https://arxiv.org/html/2604.19766#bib.bib26)\)\. The first three were built from Wikipedia/WikidataVrandečić and Krötzsch \([2014](https://arxiv.org/html/2604.19766#bib.bib27)\)using different multi\-hop strategies, while Bamboogle contains challenging 2\-hop questions\. We used development sets \(HotpotQA: 7405; 2WikiMultiHopQA: 12576; MuSiQue: 2417\) and Bamboogle’s test set \(125 samples\), removing original contexts and relying solely on Wikipedia retrieval for background knowledge\.

BaselinesWe compared OThink\-SRR1 with several baselines: \(1\) No RAG \(direct generation\); \(2\) Basic RAG: simple retrieval\-augmented generation; \(3\) Iter\-RetGenShaoet al\.\([2023](https://arxiv.org/html/2604.19766#bib.bib18)\): iterative retrieval\-generation; and \(4\) IRCoTTrivediet al\.\([2023](https://arxiv.org/html/2604.19766#bib.bib19)\): interleaved retrieval and Chain\-of\-Thought reasoning\. To ensure a fair comparison, all methods were based on the same family of instruction\-tuned models\. Specifically, the Search\-R1Jinet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib15)\)and the 7B\-Instruct version of the ReSearchChenet al\.\([2025](https://arxiv.org/html/2604.19766#bib.bib14)\)baseline leveraged publicly available open\-source models\. For ReSearch’s 3B\-Instruct variant, we trained the model using its official implementation with the same hyperparameters as OThink\-SRR1 to create a controlled comparison\.

Evaluation MetricsTo assess the correctness of final answers, we evaluated answer quality using: \(1\) Exact Match \(EM\) for strict correctness, and \(2\) f1 score for partial matches at token level\. Both metrics are reported to comprehensively assess performance\.

Implementation DetailsExperiments ran on 4 × A100\-80GB GPUs, training used batch size 256 for 2 epochs with 5e\-6 learning rate\. Inference applied sampling \(temp=0\.7, top\_\\\_p=0\.9\) with 8192\-token limit\. All methods shared the same ElasticSearch\-based Wikipedia retrieval \(6M articles, 2023 version\), fetching top\-5 passages, the maximum retrieval limitCm​a​x=5C\_\{max\}=5\.

### 4\.2Results Analysis

In this section, we report and analyze the effectiveness of the proposed method \(OThink\-SRR1\) across various settings and benchmarks, focusing on its performance and generalization capabilities as evidenced by Table[1](https://arxiv.org/html/2604.19766#S4.T1)\.

##### OThink\-SRR1 demonstrates superior performance on multi\-hop QA benchmarks\.

As detailed in Table[1](https://arxiv.org/html/2604.19766#S4.T1), OThink\-SRR1 consistently outperforms established baselines across most evaluated datasets when using Qwen2\.5\-7B\-Instruct\. It achieves an average Exact Match \(EM\) of 36\.26% and an F1 score of 45\.85%, surpassing the strong ReSearch baseline by 0\.91 EM and 0\.27 F1 points\. Notably, on the complex MuSiQue dataset, OThink\-SRR1 improves EM to 20\.73% from ReSearch’s 19\.07%\. These results underscore the efficacy of our orchestrated search\-retrieve\-reasoning framework\. This performance superiority is also maintained with the smaller Qwen2\.5\-3B\-Instruct model, where OThink\-SRR1 achieves an average EM of 28\.61%, significantly ahead of other methods\.

##### OThink\-SRR1 exhibits strong generalization to out\-of\-domain datasets\.

Despite being trained exclusively on the MuSiQue dataset, OThink\-SRR1 shows robust generalization to unseen benchmarks such as 2WikiMultiHopQA and Bamboogle\. On these out\-of\-domain datasets, our method still achieves leading performance, with the 7B model attaining 41\.52% EM on 2WikiMultiHopQA and 44\.00% EM on Bamboogle, exceeding ReSearch\. This effective transfer suggests that OThink\-SRR1 learns fundamental and adaptable reasoning and retrieval strategies rather than overfitting to the characteristics of the training data\. The generalization capability is also evident with the 3B model, further highlighting the robustness of our approach across different model scales and data distributions\.

Table 1:The results of different methods on four multi\-hop QA datasets, bolded result represent better outcomes for correspond model parameters and metrics\.ModelMuSiQue2WikiBamboogleHotpotQAAvg\.EMF1EMF1EMF1EMF1EM↑\\uparrowF1↑\\uparrowBase model: Qwen2\.5\-7B\-InstructNo RAG3\.6011\.5925\.5230\.3210\.4017\.5719\.4127\.5014\.7321\.75Basic RAG6\.4513\.0925\.5932\.0120\.0029\.3432\.1342\.1621\.0429\.15Iter\-RetGen8\.1915\.7927\.8534\.1720\.0029\.3234\.6445\.4022\.6731\.17IRCOT6\.8313\.5021\.4929\.2822\.4034\.5430\.6142\.2020\.3329\.88Search\-R116\.8824\.5833\.0839\.0339\.2049\.7138\.2448\.5031\.8540\.45ReSearch19\.0728\.6640\.648\.6642\.4053\.6539\.3251\.3535\.3545\.58OThink\-SRR120\.7329\.8541\.5249\.3344\.0053\.6938\.8050\.5536\.2645\.85Base model: Qwen2\.5\-3B\-InstructNo RAG2\.367\.9224\.7928\.732\.408\.9416\.0222\.4311\.3917\.00Basic RAG5\.6310\.9424\.5430\.189\.6019\.0628\.1037\.4516\.9724\.41Iter\-RetGen7\.1212\.6125\.9731\.4711\.2019\.0930\.0939\.5618\.5925\.68IRCOT6\.7912\.5121\.4530\.0020\.8031\.4727\.2037\.5019\.0627\.87Search\-R110\.2617\.0532\.1137\.8728\.0038\.1230\.2039\.6225\.1433\.16ReSearch12\.4519\.3226\.8232\.2226\.4036\.4530\.1139\.9123\.9531\.98OThink\-SRR114\.9823\.9435\.2743\.5431\.2043\.5832\.9944\.2128\.6138\.82

### 4\.3Further Analysis

![Refer to caption](https://arxiv.org/html/2604.19766v1/x2.png)Figure 3:The reward scores, response lengths, and retrieval counts during the training process of Qwen2\.5\-3B\-Instruct and Qwen2\.5\-7B\-Instruct models\.Training Dynamics Analysis\.Figure[3](https://arxiv.org/html/2604.19766#S4.F3)plots reward score, response length, and retrieval count for both Qwen\-7B \(solid\) and Qwen\-3B \(dashed\) under OThink\-SRR1\. Both models exhibit an early exploration phase—retrieval count rise to about 2\.5–2\.8 per question, followed by consolidation\-retrieval count fall to 2\.2\-2\.4 and responses shorten by roughly 25%\. Simultaneously, rewards climb from 0\.4 to 0\.6 for 7B and from 0\.3 to 0\.5 for 3B before plateauing, demonstrating that OThink\-SRR1 learns to filter noise and focus on key information\. This transition from broad search to efficient reasoning highlights OThink\-SRR1’s strength in reducing redundant retrievals and context size—a critical advantage for smaller models prone to distraction by noisy inputs\.

![Refer to caption](https://arxiv.org/html/2604.19766v1/x3.png)Figure 4:Comparison of Average Retrieval Count and Total Token Usage Between OThink\-SRR1 and the Baseline ReSearch on Multi\-hop QA Datasets, based on Qwen2\.5\-7B\-Instruct model\.Question:What was the population of the city that is found in the same county as Eric Marcus Municipal Airport in 1900?Ground Truth:7,531ReSearch \(7B\) Thinking Process:tokens fed to inference: 10347<think\> To find \.\.\. planing \.\.\. </think\>,
<search\> What county is Eric Marcus Municipal Airport in? </search\>,<result\> \.\.\. Eric Marcus Municipal Airport is a county\-owned, public\-use airport in Pima County, Arizona, \.\.\. </result\>,
<think\> \.\.\. I will now find the city in that county \.\.\.</think\>,
<search\> What is the largest city in Pima County, Arizona? </search\>,
<result\> \.\.\. Pima County includes the Tucson, \.\.\. </result\>,
<think\> \.\.\. I will find the population of Tucson in the year 1900 \.\.\.</think\>,
<search\> What was the population of Tucson, Arizona in 1900? </search\>,
<result\> \.\.\. By 1900, 7,531 people lived in the city \.\.\. </result\>,
<think\> After finding the population of Tucson in 1900, I can now provide the answer to the question\.</think\>,
<search\> What was the population of Tucson, Arizona in 1900? </search\>,
<result\> \.\.\. By 1900, 7,531 people lived in the city \.\.\. </result\>,
<think\> Having found the population of Tucson in 1900, I can now provide the final answer to the question\.</think\>,
<answer\> \\boxed\{Tucson\} </answer\>OThink\-SRR1\-7B Thinking Process:tokens fed to inference: 3638
<think\> First, \.\.\. identify the \.\.\. Airport is located\. </think\>,
<query\> What county is Eric Marcus Municipal Airport in?</query\>
<documents\> …airport in Pima County, Arizona,… </documents\>
<documents\_refine\> Pima County, Arizona\. </documents\_refine\>,
<think\> \.\.\.find the city in that county and its 1900 population\. </think\>
<query\> What city in Pima County, Arizona had a population in 1900?</query\>
<documents\> …By 1900, 7,531 people lived in the city… </documents\>
<documents\_refine\> Tucson, Arizona had a population of 7,531 in 1900\. </documents\_refine\>
<think\> The city that was in the same county as Eric Marcus Municipal Airport in 1900 was Tucson, and its population in 1900 is found to be 7,531\. </answer\>
<answer\> \\boxed\{7531\} </answer\>Figure 5:Case Study Comparison Between OThink\-SRR1 and ReSearch Baseline\.Redqueries show search operations,strikethroughshows discarded documents\.Figure[4](https://arxiv.org/html/2604.19766#S4.F4)provides a detailed comparison of average retrieval counts and cumulative token consumption between our proposedOThink\-SRR1framework and the baseline ReSearch across four representative multi\-hop QA datasets\.

As shown in the left panel, our method consistently achieves a lower average retrieval count \(2\.35\) compared to the baseline \(3\.42\) on all datasets, indicating thatOThink\-SRR1is able to reduce unnecessary or redundant retrieval steps during the reasoning process\. Correspondingly, the right panel demonstrates a significant reduction in the total number of input tokens processed by the model \(4270\.81 vs\. 8045\.26\), reflecting improved computational efficiency and lower resource consumption\. This reduction in both retrieval count and token usage not only alleviates the burden on the model but also contributes to more focused and effective reasoning\. Importantly, these efficiency gains do not come at the cost of performance; instead, our method achieves stronger results, validating that the Search\-Refine\-Reasoning paradigm enables the model to filter out noise and concentrate on relevant information, leading to both better accuracy and improved efficiency\.

### 4\.4Case Study

The case study \(Figure[5](https://arxiv.org/html/2604.19766#S4.F5)\) vividly illustrates OThink\-SRR1’s practical superiority\. OThink\-SRR1 efficiently solves the multi\-hop query in just two retrieval\-refinement cycles, processing only 3,638 tokens\. Each refinement cycle acts as a powerful distillation step, transforming verbose documents into compact, factual assertions\. By iteratively retaining only these essential facts \(e\.g\., "Pima County, Arizona," then "Tucson…population of 7,531"\), it prevents the reasoning context from being cluttered\. This strategic pruning is not merely about efficiency; it critically reduces the model’s cognitive load, allowing it to maintain focus and correctly extract the population "7,531"\. In stark contrast, the ReSearch baseline, lacking this refinement, processes significantly more tokens \(10,347\) across more steps\. It becomes overwhelmed by the noisy, unpruned context, leading to a classic failure of contextual distraction where a salient but incorrect entity \("Tucson"\) hijacks the model’s attention\. This comparison demonstrates that for complex tasks, the ability to intelligently filter and forget is as vital as the ability to retrieve and reason\. OThink\-SRR1’s success proves that effective reasoning is not just about finding information, but about building a clear path to the answer by judiciously discarding noise\.

## 5Conclusion

We introduced OThink\-SRR1, a novel framework that synergistically integrates search, refinement, and R1\-style reasoning through reinforcement learning, specifically targeting retrieval noise and inefficiency in multi\-hop question answering\. Our GRPO\-IR reward mechanism enables dynamic information management without supervised intermediate steps\. Extensive experiments demonstrate OThink\-SRR1’s significant outperformance against strong baselines across four benchmarks in terms of accuracy, retrieval efficiency, and token economy\. The framework also exhibits strong generalization capabilities from single\-dataset training, offering a robust and efficient approach to advance LLM performance in complex, knowledge\-intensive reasoning tasks\.

## 6Limitation

Despite its effectiveness, OThink\-SRR1’s refinement process occasionally discards critical information needed for accurate reasoning\. This loss of essential details can lead to incorrect answers, particularly when relevant context is subtle or dispersed across documents\. Future work will enhance context\-awareness through adaptive filtering to preserve key information while reducing noise\. Incorporating dynamic reward functions based on task\-specific feedback could improve flexibility\. These refinements aim to broaden OThink\-SRR1’s applicability to complex reasoning tasks\.

## References

- M\. Chen, T\. Li, H\. Sun, Y\. Zhou, C\. Zhu, F\. Yang, Z\. Zhou, W\. Chen, H\. Wang, J\. Z\. Pan,et al\.\(2025\)Learning to reason with search for llms via reinforcement learning\.arXiv preprint arXiv:2503\.19470\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1),[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2604.19766#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p3.1)\.
- MuRAG: multimodal retrieval\-augmented generator for open question answering over images and text\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 5558–5570\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.375/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.375)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- X\. Du and H\. Ji \(2022\)Retrieval\-augmented generative question answering for event argument extraction\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 4649–4666\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.307/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.307)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- F\. Fang, Y\. Bai, S\. Ni, M\. Yang, X\. Chen, and R\. Xu \(2024a\)Enhancing noise robustness of retrieval\-augmented language models with adaptive adversarial training\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10028–10039\.External Links:[Link](https://aclanthology.org/2024.acl-long.540/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.540)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- F\. Fang, Y\. Bai, S\. Ni, M\. Yang, X\. Chen, and R\. Xu \(2024b\)Enhancing noise robustness of retrieval\-augmented language models with adaptive adversarial training\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10028–10039\.External Links:[Link](https://aclanthology.org/2024.acl-long.540/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.540)Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p3.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1)\.
- X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,D\. Scott, N\. Bel, and C\. Zong \(Eds\.\),Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://aclanthology.org/2020.coling-main.580/),[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p2.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1),[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2604.19766#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p3.1)\.
- H\. Lee, S\. Yang, H\. Oh, and M\. Seo \(2022\)Generative multi\-hop retrieval\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 1417–1436\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.92/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.92)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p2.1)\.
- M\. Lee, S\. An, and M\. Kim \(2024\)PlanRAG: a plan\-then\-retrieval augmented generation for generative large language models as decision makers\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6537–6555\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p1.1)\.
- D\. Li, J\. Yan, T\. Zhang, C\. Wang, X\. He, L\. Huang, H\. Xue’, and J\. Huang \(2024\)On the role of long\-tail knowledge in retrieval augmented large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 120–126\.External Links:[Link](https://aclanthology.org/2024.acl-short.12/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-short.12)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- A\. Matarazzo and R\. Torlone \(2025\)A survey on large language models with some insights on their capabilities and limitations\.arXiv preprint arXiv:2501\.04040\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p1.1)\.
- S\. Mitra, R\. Ramnani, and S\. Sengupta \(2022\)Constraint\-based multi\-hop question answering with knowledge graph\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track,A\. Loukina, R\. Gangadharaiah, and B\. Min \(Eds\.\),Hybrid: Seattle, Washington \+ Online,pp\. 280–288\.External Links:[Link](https://aclanthology.org/2022.naacl-industry.31/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-industry.31)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p2.1)\.
- OpenAI \(2024\)Learning to reason with llms\.Note:[https://openai\.com/index/learning\-to\-reason\-with\-llms/](https://openai.com/index/learning-to-reason-with-llms/)Accessed: 2024\-06\-13Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1)\.
- O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. Smith, and M\. Lewis \(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5687–5711\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.378/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p2.1)\.
- Qwen Team \(2025\)QwQ\-32b: embracing the power of reinforcement learning\.Note:[https://qwenlm\.github\.io/blog/qwq\-32b/](https://qwenlm.github.io/blog/qwq-32b/)Accessed: 2025\-03\-6Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1)\.
- H\. Sankararaman, M\. N\. Yasin, T\. Sorensen, A\. D\. Bari, and A\. Stolcke \(2024\)Provenance: a light\-weight fact\-checker for retrieval augmented LLM generation output\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 1305–1313\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.97/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.97)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- Z\. Shao, Y\. Gong, Y\. Shen, M\. Huang, N\. Duan, and W\. Chen \(2023\)Enhancing retrieval\-augmented large language models with iterative retrieval\-generation synergy\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9248–9274\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.620/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.620)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p3.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.2\.2](https://arxiv.org/html/2604.19766#S3.SS2.SSS2.p1.1)\.
- X\. Shen, S\. Wu, and R\. Xia \(2023\)Dense\-ATOMIC: towards densely\-connected ATOMIC with high knowledge coverage and massive multi\-hop paths\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13292–13305\.External Links:[Link](https://aclanthology.org/2023.acl-long.742/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.742)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p2.1)\.
- H\. Song, J\. Jiang, Y\. Min, J\. Chen, Z\. Chen, W\. X\. Zhao, L\. Fang, and J\. Wen \(2025\)R1\-searcher: incentivizing the search capability in llms via reinforcement learning\.arXiv preprint arXiv:2503\.05592\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p2.1),[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2604.19766#S3.SS2.p1.1)\.
- W\. Su, Y\. Tang, Q\. Ai, Z\. Wu, and Y\. Liu \(2024\)DRAGIN: dynamic retrieval augmented generation based on the real\-time information needs of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12991–13013\.External Links:[Link](https://aclanthology.org/2024.acl-long.702/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.702)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.
- Z\. Sun, Q\. Wang, W\. Yu, X\. Zang, K\. Zheng, J\. Xu, X\. Zhang, S\. Yang, and H\. Li \(2025\)ReARTeR: retrieval\-augmented reasoning with trustworthy process rewarding\.arXiv preprint arXiv:2501\.07861\.Cited by:[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p1.1)\.
- F\. Tian, D\. Ganguly, and C\. Macdonald \(2025\)Is relevance propagated from retriever to generator in rag?\.InEuropean Conference on Information Retrieval,pp\. 32–48\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p3.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p2.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2023\)Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 10014–10037\.External Links:[Link](https://aclanthology.org/2023.acl-long.557/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.557)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p3.1)\.
- D\. Vrandečić and M\. Krötzsch \(2014\)Wikidata: a free collaborative knowledgebase\.Commun\. ACM57\(10\),pp\. 78–85\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/2629489),[Document](https://dx.doi.org/10.1145/2629489)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p2.1)\.
- Z\. Wang, A\. Liu, H\. Lin, J\. Li, X\. Ma, and Y\. Liang \(2024\)Rat: retrieval augmented thoughts elicit context\-aware reasoning in long\-horizon generation\.arXiv preprint arXiv:2403\.05313\.Cited by:[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p1.1)\.
- K\. Wu, E\. Wu, and J\. Y\. Zou \(2024\)Clasheval: quantifying the tug\-of\-war between an llm’s internal prior and external evidence\.Advances in Neural Information Processing Systems37,pp\. 33402–33422\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p3.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2369–2380\.External Links:[Link](https://aclanthology.org/D18-1259/),[Document](https://dx.doi.org/10.18653/v1/D18-1259)Cited by:[§4\.1](https://arxiv.org/html/2604.19766#S4.SS1.p2.1)\.
- T\. Yu, S\. Zhang, and Y\. Feng \(2024\)Auto\-rag: autonomous retrieval\-augmented generation for large language models\.arXiv preprint arXiv:2411\.19443\.Cited by:[§2\.2](https://arxiv.org/html/2604.19766#S2.SS2.p1.1)\.
- G\. Zhang, Z\. Xu, Q\. Jin, F\. Chen, Y\. Fang, Y\. Liu, J\. F\. Rousseau, Z\. Xu, Z\. Lu, C\. Weng,et al\.\(2024\)A mapreduce approach to effectively utilize long context information in retrieval augmented language models\.arXiv preprint arXiv:2412\.15271\.Cited by:[§1](https://arxiv.org/html/2604.19766#S1.p3.1),[§3\.1](https://arxiv.org/html/2604.19766#S3.SS1.p1.1)\.
- Q\. Zhao, R\. Wang, Y\. Cen, D\. Zha, S\. Tan, Y\. Dong, and J\. Tang \(2024\)LongRAG: a dual\-perspective retrieval\-augmented generation paradigm for long\-context question answering\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 22600–22632\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1259/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1259)Cited by:[§2\.1](https://arxiv.org/html/2604.19766#S2.SS1.p1.1)\.