RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation

# Rims: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
Source: [https://arxiv.org/html/2607.16431](https://arxiv.org/html/2607.16431)
Pei Tian1, Zihan Dong2, Tianci Liu3, Linjun Zhang2, Haoyu Wang4 1Department of Computer Science, Columbia University 2Department of Statistics, Rutgers University 3School of Electrical and Computer Engineering, Purdue University 4Departiment of Computer Science, SUNY Albany pt2632@columbia\.edu;hwang28@albany\.edu

###### Abstract

Small\-scale language models \(SLMs\) are attractive for retrieval\-augmented generation \(RAG\) in resource\-constrained settings, but their limited capacity makes them highly sensitive to noisy or spurious retrieved evidence\. Existing preference\-based methods such as RoseRAG select only the hardest single preference pair via hard argmin/argmax, discarding the remaining signal; others treat multiple pairs as independent binary comparisons, resulting in low data utilization\. We proposeRims, a three\-stage preference optimization framework comprising \(1\) synthetic chain\-of\-thought preference data generation via rejection sampling using the target SLM itself without relying on proprietary models, \(2\) a differentiable soft aggregation mechanism that replaces hard selection with a smooth operator, preserving gradient signal from all preference pairs while retaining the discriminative structure of margin\-aware selection, and \(3\) preference optimization with the smoothed objective applied to multiple alignment algorithms\. We theoretically show that the smoothed approximation admits a controllable error bound and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection\. Experiments on four multi\-hop question answering benchmarks show that our approach outperforms state\-of\-the\-art baselines across multiple SLM backbones, achieving consistent gains in Exact Match and F1 under noisy retrieval conditions\. Our implementation is available at[https://github\.com/tptrix29/RIMS](https://github.com/tptrix29/RIMS)\.

## 1Introduction

Large language models \(LLMs\)\(Achiamet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib36); Comaniciet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib37); Yanget al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib38); Grattafioriet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib39)\)have achieved strong performance across many natural language tasks by learning vast amounts of linguistic and world knowledge during pretraining\. However, their high computational and memory requirements limit practical deployment in resource\-constrained environments such as on\-device systems or low\-infrastructure settings\(Zhouet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib40); Xuet al\.,[2024b](https://arxiv.org/html/2607.16431#bib.bib41)\)\. To address this, small\-scale language models \(SLMs\) with dramatically fewer parameters have become a promising alternative due to their lower inference cost, portability, and ease of deployment\(Belcaket al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib42); Luet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib43); Vernikoset al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib44)\)\.

Despite their computational efficiency, SLMs inherently cannot internalize as much knowledge as larger models, and their reduced capacity often leads to weaker semantic understanding and generalization on knowledge\-intensive tasks\(Ovadiaet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib45)\)\. Retrieval\-augmented generation \(RAG\)\(Lewiset al\.,[2021](https://arxiv.org/html/2607.16431#bib.bib46); Gaoet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib63); Jianget al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib62)\)offers a natural remedy by coupling the model with an external knowledge source, allowing it to answer queries that require up\-to\-date or domain\-specific information beyond its pretrained parameters\. Applying RAG to SLMs can substantially enhance performance by offloading factual grounding to an external retrieval component\(Wanget al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib49); Asaiet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib50); Xuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib51)\)\. Yet even with RAG, robust integration of retrieved content remains challenging for small models: their limited capacity makes it difficult to distinguish relevant evidence from distractors, to reason over multiple retrieved passages, and to suppress noisy or contradictory context during generation\(Yoranet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib47); Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)\. This underscores the need for mechanisms that go beyond retrieval quality and directly improve how SLMs learn to utilize imperfect evidence\.

Existing methods for improving SLM robustness under RAG fall into three broad categories\.Prompt\-based approaches\(Wanget al\.,[2025a](https://arxiv.org/html/2607.16431#bib.bib33);[2024](https://arxiv.org/html/2607.16431#bib.bib21)\)instruct the model to disregard irrelevant retrieved content, but their effectiveness hinges on the model’s own ability to identify noise, an assumption that often fails for SLMs\.Supervised fine\-tuning \(SFT\)\(Yoranet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib47); Weiet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib22)\)aligns models with curated instruction–response pairs, yet is highly sensitive to noise in retrieved contexts and prone to overfitting on imperfect evidence\(Chuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib48)\), particularly for capacity\-limited SLMs\. A third line of work,preference optimization\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib9); Xuet al\.,[2024a](https://arxiv.org/html/2607.16431#bib.bib10); Honget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib11); Ethayarajhet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib60); Menget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib61)\), offers a more flexible alternative by learning from relative quality signals rather than fixed targets\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12); Chenet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib35)\)\. However, effectively constructing and utilizing preference data under noisy RAG conditions remains an open challenge\.

In RAG settings, preference optimization is particularly appealing because it learns from relative quality comparisons rather than treating any single response as ground truth, making it naturally more robust to noisy retrieved contexts than supervised fine\-tuning\. Moreover, by sampling multiple responses per query through techniques such as rejection sampling with chain\-of\-thought prompting, one can readily obtain multiple preferred and non\-preferred candidates, yielding rich multi\-pair preference data for each training instance\. Despite the abundance of such supervision, existing methods utilize only a small fraction of it\. Concretely, when multiple preference pairs are available for a single query, RoseRAG\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)selects only the single hardest pair based on the initial model’s likelihoods, discarding the remaining signal entirely and fixing the selection throughout training even as the policy evolves\. NCA\(Chenet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib35)\)aggregates multiple negative responses but does not jointly model positive candidates, resulting in an incomplete use of the available preference structure\. In both cases, the rich multi\-pair preference information is substantially underutilized, motivating the need for a principled mechanism that can fully leverage all available preference pairs during optimization\.

To address these limitations, we propose a preference optimization framework via smoothedmulti\-pair aggregation forSLM retrieval\-augmented generation \(Rims\)\. Given multiple preference pairs generated per query through chain\-of\-thought \(CoT\)\(Weiet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib18)\)rejection sampling,Rimsintroduces a differentiable soft aggregation mechanism that replaces hard pair selection with a smooth, temperature\-controlled operator\. This operator consolidates all positive and negative candidates into a unified pseudo\-pair, preserving gradient signal from every candidate while retaining the discriminative structure of margin\-aware selection\. The smoothed objective is general and can be applied to multiple preference optimization algorithms\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib9); Xuet al\.,[2024a](https://arxiv.org/html/2607.16431#bib.bib10); Honget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib11)\)\. By adaptively aggregating gradient signals across multiple samples of the same question,Rimsimproves the model’s ability to identify and integrate relevant evidence from noisy retrieval contexts\. We further provide theoretical guarantees that the smoothed objective admits a controllable approximation error and that its gradient updates are provably closer to the oracle than those of hard selection\.

Our contributions can be summarized as follows:

- •We proposeRims, a preference optimization framework for SLM\-based RAG that generates multi\-pair CoT preference data via rejection sampling using the target SLM itself, without relying on proprietary models\.
- •We introduce a differentiable soft aggregation mechanism that consolidates multiple preference pairs into a unified, algorithm\-agnostic objective, replacing hard pair selection while preserving the discriminative structure of margin\-aware optimization\.
- •We provide theoretical analysis showing that the LSE\-based smoothing admits a controllable approximation error bound, and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection under noisy reward estimation\.
- •Experiments on four multi\-hop question answering benchmarks with two SLM backbones show thatRimsconsistently outperforms state\-of\-the\-art baselines\.

## 2Related Work

Retrieval\-augmented generation \(RAG\) has emerged as an effective paradigm for alleviating the knowledge limitations of language models by grounding generation in external corpora\. While early RAG frameworks\(Izacardet al\.,[2022](https://arxiv.org/html/2607.16431#bib.bib53)\)were primarily designed for large language models \(LLMs\), recent studies\(Wanget al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib49); Liet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib54); Yuet al\.,[2024b](https://arxiv.org/html/2607.16431#bib.bib55)\)have explored adapting RAG to SLMs to improve efficiency and deployability in resource\-constrained settings\. However, due to limited representational capacity, SLMs are particularly vulnerable to noisy or irrelevant retrieved contexts, often leading to degraded reasoning and hallucinated outputs\(Yoranet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib47)\)\. Prior work like Blendfilter\(Wanget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib21)\)and RobustRAG\(Xianget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib2)\)has attempted to address this issue through prompt\-based filtering, but these approaches rely heavily on the model’s intrinsic reasoning ability to identify irrelevant context, which is often insufficient for SLMs\. In the meantime, some methods\(Weiet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib22); Yuet al\.,[2024a](https://arxiv.org/html/2607.16431#bib.bib57)\)investigate training LLMs to handle noisy contexts via supervised finetuning on data synthesized with RAG\-retrieved content, but they significantly suffer from low robustness under noisy retrieval conditions\.

Preference\-based optimization methods\(Zhanget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib58); Donget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib59)\)have gained traction as alternatives to supervised fine\-tuning by directly learning from relative preferences, offering improved robustness under noisy RAG settings\. However, most approaches rely on single\-pair comparisons, underutilizing the richer supervision available from multiple candidate responses, motivating recent work on multi\-pair preference optimization\. RoseRAG\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)introduces margin\-aware preference optimization by selecting the hardest positive and negative pair based on model likelihoods, improving robustness to noisy retrieval but discarding the remaining preference signals\. Noise Contrastive Alignment \(NCA\)\(Chenet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib35)\)instead aggregates multiple negative responses within a contrastive framework, enhancing robustness but neglecting structured relationships among positive candidates\. Other recent efforts\(Xieet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib3); Guptaet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib4); Wanget al\.,[2025b](https://arxiv.org/html/2607.16431#bib.bib5); Leet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib6)\)explore various strategies for leveraging multiple responses per query, including selective subset sampling, multi\-perspective preference signals, and multiple reference distributions, yet they generally rely on hard selection or heuristic aggregation that limits full exploitation of structured multi\-pair preference data\.

In contrast, our framework jointly incorporates multiple preference pairs into a single optimization objective through a differentiable smoothing mechanism, preserving informative signals while maintaining computational efficiency\.

## 3Preliminaries

We briefly review the standard retrieval\-augmented generation framework, preference optimization, and margin\-aware preference selection, which form the basis of our method\.

### 3\.1Retrieval\-Augmented Generation

Given a language model parameterized byθ\\theta, a knowledge base𝒦=\{𝒦i\}i=1k\\mathcal\{K\}=\\\{\\mathcal\{K\}\_\{i\}\\\}\_\{i=1\}^\{k\}\(wherekkdenotes the number of documents\), a retrieverℛ​\(⋅\)\\mathcal\{R\}\(\\cdot\), and a queryqq, the vanilla retrieval\-augmented generation \(RAG\) framework follows a retrieve\-then\-generate paradigm\. Specifically, it first retrieves the top\-KKrelevant documents from the knowledge base and then generates an answer conditioned on the retrieved information\. This process can be formulated as

𝒦q=ℛ​\(q,𝒦;K\),y∼Pθ​\(y∣PromptCoT​\(q,𝒦q\)\)\\begin\{gathered\}\\mathcal\{K\}\_\{q\}=\\mathcal\{R\}\(q,\\mathcal\{K\};K\),\\\\ y\\sim P\_\{\\theta\}\\bigl\(y\\mid\\text\{Prompt\}\_\{\\text\{CoT\}\}\(q,\\mathcal\{K\}\_\{q\}\)\\bigr\)\\end\{gathered\}\(1\)whereyydenotes the generated response andPromptCoT​\(⋅\)\\text\{Prompt\}\_\{\\text\{CoT\}\}\(\\cdot\)represents the CoT prompt\.

### 3\.2Preference Optimization

Preference alignment for language models has traditionally relied on multi\-stage procedures, such as reinforcement learning from human feedback \(RLHF\)\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.16431#bib.bib64); Baiet al\.,[2022](https://arxiv.org/html/2607.16431#bib.bib65)\), that require an additional reference model to guide and stabilize training\. Recent advances have led to a class of methods that learn directly from preference data, avoiding or simplifying explicit reward modeling\. These methods generally optimize a policyπθ\\pi\_\{\\theta\}over preference pairs\(yw,yl\)\(y\_\{w\},y\_\{l\}\)given a promptxx, whereywy\_\{w\}denotes a preferred response andyly\_\{l\}a dispreferred one\. A general form of such objectives can be expressed as

ℒ​\(θ\)=ℒbase​\(πθ\)−λ​𝔼\(x,yw,yl\)​\[log⁡σ​\(β⋅hθ​\(yw,yl,x\)\)\]\\mathcal\{L\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{base\}\}\(\\pi\_\{\\theta\}\)\-\\lambda\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\left\[\\log\\sigma\\\!\\left\(\\beta\\cdot h\_\{\\theta\}\(y\_\{w\},y\_\{l\},x\)\\right\)\\right\]\(2\)
whereℒbase\\mathcal\{L\}\_\{\\text\{base\}\}is an optional base objective \(e\.g\., supervised fine\-tuning loss\),σ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function,β\\betacontrols the strength of the preference signal,λ\\lambdaweights the preference term, andhθ​\(yw,yl,x\)h\_\{\\theta\}\(y\_\{w\},y\_\{l\},x\)is an algorithm\-specific scoring function that quantifies the relative quality ofywy\_\{w\}overyly\_\{l\}\. Different instantiations ofhθh\_\{\\theta\}andℒbase\\mathcal\{L\}\_\{\\text\{base\}\}give rise to distinct algorithms, including Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib9)\), Contrastive Preference Optimization \(CPO\)\(Xuet al\.,[2024a](https://arxiv.org/html/2607.16431#bib.bib10)\), and Odds\-Ratio Preference Optimization \(ORPO\)\(Honget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib11)\), which differ in how reference normalization, base supervision, and contrast signals are incorporated\.

### 3\.3Margin\-Aware Preference Optimization

In RAG settings, preference optimization is particularly appealing because it learns from relative quality comparisons rather than treating any single response as ground truth, making it more robust to noisy retrieved context than supervised fine\-tuning\. When multiple preference pairs are available for a single query, selecting the most informative pair becomes critical for effective optimization\. RoseRAG\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)addresses this with a margin\-aware selection strategy that identifies the hardest positive and negative examples based on current model likelihoods:

yw∗=arg⁡mini⁡log⁡Pθ​\(ywi∣x\),yl∗=arg⁡maxj⁡log⁡Pθ​\(ylj∣x\),\\begin\{gathered\}y\_\{w\}^\{\*\}=\\arg\\min\_\{i\}\\log P\_\{\\theta\}\(y\_\{w\}^\{i\}\\mid x\),\\\\ y\_\{l\}^\{\*\}=\\arg\\max\_\{j\}\\log P\_\{\\theta\}\(y\_\{l\}^\{j\}\\mid x\),\\end\{gathered\}\(3\)That is, the preferred response with the lowest model confidence and the non\-preferred response with the highest model confidence are selected, forming the most challenging contrast for the current policy\. However, this hard argmin/argmax selection retains only a single pair per query, discarding the gradient signal from all remaining candidates\.

## 4Method

We presentRims, shown in Fig\.[1](https://arxiv.org/html/2607.16431#S4.F1), a framework that improves preference optimization for SLM\-based RAG by fully utilizing multi\-pair preference data\.Rimsconsists of three components: \(1\)Multi\-Pair Preference Data Generation, which produces diverse chain\-of\-thought rationales via rejection sampling; \(2\)Multi\-Pair Preference Data Aggregation, which consolidates multiple preference pairs into a unified pseudo\-pair through a differentiable smoothing operator; and \(3\)Preference Optimization, which applies the smoothed objective to standard alignment algorithms\.

### 4\.1Multi\-Pair Preference Data Generation

To construct preference data for training, we use the target SLM itself to generate chain\-of\-thought \(CoT\) rationales, and apply rejection sampling to filter for correctness\. This avoids dependence on proprietary models and produces preference pairs that are well\-calibrated to the target model’s own capability\.

We instantiate this procedure by constructing a multi\-turn prompt consisting of a system message, a user message, and an initial assistant message\. The system message specifies the task and instructs the model to reason step\-by\-step, acknowledging that the provided knowledge may contain irrelevant information\. The user message provides the retrieved knowledge documents𝒦q\\mathcal\{K\}\_\{q\}, the questionqq, and the ground\-truth answera∗a^\{\*\}\. The assistant message initializes the reasoning process\. The complete prompt template is provided in Appendix[F](https://arxiv.org/html/2607.16431#A6)\. Given these inputs, the model generates a rationalerrthat captures its analytical reasoning\.

To ensure quality, we apply rejection sampling\(Liuet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib7); Guoet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib8)\)\. Given the generated rationalerr, we extract an answeraaand compare it against the ground\-truth labela∗a^\{\*\}\. Ifaamatchesa∗a^\{\*\}, the full response\(r,a\)\(r,a\)is retained as a preferred responseywy\_\{w\}\. The non\-preferred responseyly\_\{l\}is obtained separately as the vanilla RAG output \(Eqn\. \([1](https://arxiv.org/html/2607.16431#S3.E1)\)\), which is generated without ground\-truth guidance\. By repeating this process with different sampling runs, we obtain multiple preferred–non\-preferred pairs per query, yielding multi\-pair preference data\.

![Refer to caption](https://arxiv.org/html/2607.16431v1/x1.png)Figure 1:Framework of Smoothing Preference Optimization with Multi\-Pair Preference Data
### 4\.2Multi\-Pair Preference Data Aggregation

The margin\-aware strategy \(Eqn\. \([3](https://arxiv.org/html/2607.16431#S3.E3)\)\) selects the single hardest preference pair via hard argmin/argmax, discarding all remaining candidates\. Directly relaxing this selection poses competing challenges: the argmin/argmax operators are non\-differentiable, preventing gradient signal from propagating through the selection step; yet naively replacing them with a simple average dilutes the focus on informative hard cases, weakening the discriminative structure that makes margin\-aware optimization effective\. Moreover, hard selection is performed once based on the initial model’s likelihoods and remains fixed throughout training; as the policy evolves, the initially selected pair may no longer represent the most informative contrast\. What is needed is a smooth, differentiable operator that preserves the emphasis on hard examples while incorporating signal from all candidates and adapting to the evolving policy\.

We address this with the LogSumExp \(LSE\) function\(Nielsen and Sun,[2016](https://arxiv.org/html/2607.16431#bib.bib13)\), which provides a temperature\-controlled smooth approximation to the maximum operator:

LSEτ​\(x1,⋯,xn\)=1τ​log⁡\(∑i=1nexp⁡\(τ​xi\)\)\\text\{LSE\}\_\{\\tau\}\(x\_\{1\},\\cdots,x\_\{n\}\)=\\frac\{1\}\{\\tau\}\\log\\Big\(\\sum\_\{i=1\}^\{n\}\\exp\(\{\\tau x\_\{i\}\}\)\\Big\)\(4\)whereτ\>0\\tau\>0controls the sharpness of the approximation: asτ→∞\\tau\\to\\infty, the LSE operator converges to the exact maximum; asτ→0\\tau\\to 0, it reduces to a uniform average\. This provides a principled interpolation between hard selection and soft aggregation\. Because the LSE operator is evaluated using the current policyπθ\\pi\_\{\\theta\}at each training step, the aggregation weights adapt dynamically as the model evolves, in contrast to the static one\-time selection of RoseRAG\. A formal analysis of the approximation bound is provided in Section[5](https://arxiv.org/html/2607.16431#S5)and Appendix[A\.1](https://arxiv.org/html/2607.16431#A1.SS1)\.

Applying this operator to the policy likelihoods, we aggregate all preferred and non\-preferred candidates into a unified pseudo\-pair\(x,y~w,y~l\)\(x,\\tilde\{y\}\_\{w\},\\tilde\{y\}\_\{l\}\)\. For the preferred set, we use the negated LSE to approximate the minimum\-likelihood \(hardest\) positive; for the non\-preferred set, we use LSE directly to approximate the maximum\-likelihood \(hardest\) negative:

log⁡πθ​\(y~w\|x\)=−LSEτ​\(−log⁡πθ​\(yw1∣x\),⋯,−log⁡πθ​\(ywn∣x\)\),log⁡πθ​\(y~l\|x\)=LSEτ​\(log⁡πθ​\(yl1∣x\),⋯,log⁡πθ​\(yln∣x\)\),\\begin\{gathered\}\\log\\pi\_\{\\theta\}\(\\tilde\{y\}\_\{w\}\|x\)=\-\\text\{LSE\}\_\{\\tau\}\(\-\\log\\pi\_\{\\theta\}\(y\_\{w\}^\{1\}\\mid x\),\\cdots,\-\\log\\pi\_\{\\theta\}\(y\_\{w\}^\{n\}\\mid x\)\),\\\\ \\log\\pi\_\{\\theta\}\(\\tilde\{y\}\_\{l\}\|x\)=\\text\{LSE\}\_\{\\tau\}\(\\log\\pi\_\{\\theta\}\(y\_\{l\}^\{1\}\\mid x\),\\cdots,\\log\\pi\_\{\\theta\}\(y\_\{l\}^\{n\}\\mid x\)\),\\end\{gathered\}\(5\)

### 4\.3Preference Optimization

For a multi\-pair preference data tuple\(x,\{ywi\}i=1n,\{yli\}i=1n\)\(x,\\\{y\_\{w\}^\{i\}\\\}\_\{i=1\}^\{n\},\\\{y\_\{l\}^\{i\}\\\}\_\{i=1\}^\{n\}\), wherexxis the input prompt,\{ywi\}\\\{y\_\{w\}^\{i\}\\\}is the set of preferred responses, and\{yli\}\\\{y\_\{l\}^\{i\}\\\}is the set of non\-preferred responses, the smoothed preference optimization objective is:

ℒ~​\(θ\)=ℒbase​\(πθ\)−λ​𝔼\(x,yw,yl\)​\[log⁡σ​\(β⋅hθ​\(y~w,y~l,x\)\)\]\\tilde\{\\mathcal\{L\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{base\}\}\(\\pi\_\{\\theta\}\)\-\\lambda\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\left\[\\log\\sigma\\left\(\\beta\\cdot h\_\{\\theta\}\(\\tilde\{y\}\_\{w\},\\tilde\{y\}\_\{l\},x\)\\right\)\\right\]\(6\)
wherey~w\\tilde\{y\}\_\{w\}andy~l\\tilde\{y\}\_\{l\}are the smoothed pseudo\-responses computed via Eqn\. \([5](https://arxiv.org/html/2607.16431#S4.E5)\)\. This formulation is algorithm\-agnostic: for any preference algorithm instantiated through Eqn\. \([2](https://arxiv.org/html/2607.16431#S3.E2)\), the smoothed variant is obtained by replacing the original policy likelihoodsπθ​\(yw\|x\)\\pi\_\{\\theta\}\(y\_\{w\}\|x\)andπθ​\(yl\|x\)\\pi\_\{\\theta\}\(y\_\{l\}\|x\)withinhθh\_\{\\theta\}with their aggregated counterpartsπθ​\(y~w\|x\)\\pi\_\{\\theta\}\(\\tilde\{y\}\_\{w\}\|x\)andπθ​\(y~l\|x\)\\pi\_\{\\theta\}\(\\tilde\{y\}\_\{l\}\|x\), directly yielding smoothed versions of DPO\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib9)\), CPO\(Xuet al\.,[2024a](https://arxiv.org/html/2607.16431#bib.bib10)\), and ORPO\(Honget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib11)\)\.

## 5Theoretical Analysis

In this section, we provide theoretical justifications for the proposed smoothed aggregation mechanism\. We first establish a formal error bound showing that the LSE approximation can be controlled via an interpretable hyperparameter \(Section[5\.1](https://arxiv.org/html/2607.16431#S5.SS1)\), and then analyze gradient stability to show that smooth aggregation yields provably lower variance than hard selection under candidate\-pool sampling \(Section[5\.2](https://arxiv.org/html/2607.16431#S5.SS2)\)\.

### 5\.1Approximation Error Analysis

The smoothed aggregation mechanism introduced in Section 4 relies on the LSE operator as a differentiable proxy for hard max/min selection\. A natural question is how tightly this approximation tracks the exact maximum, and whether the approximation error can be controlled in a principled way\. We establish a formal error bound and show how it can be used as a practical hyperparameter\. Based on the LogSumExp smoothing in Eqn\. \([4](https://arxiv.org/html/2607.16431#S4.E4)\), we establish the following theorem to control the upper bound of the approximation error:

###### Theorem 1\(LSE Error Bound Control\)\.

LetM\>0M\>0,n\>1n\>1, andx1,…,xn∈ℝx\_\{1\},\\dots,x\_\{n\}\\in\\mathbb\{R\}\. Letx\(1\)≤⋯≤x\(n\)x\_\{\(1\)\}\\leq\\cdots\\leq x\_\{\(n\)\}and assume the top\-gapδ:=x\(n\)−x\(n−1\)\>0\\delta:=x\_\{\(n\)\}\-x\_\{\(n\-1\)\}\>0,ϵ:=LSEτ​\(x1,…,xn\)−x\(n\)\\epsilon:=\\text\{LSE\}\_\{\\tau\}\(x\_\{1\},\\dots,x\_\{n\}\)\-x\_\{\(n\)\}~forτ\>0\\tau\>0\. Ifτ≥Fδ,n​\(M\):=1δ​W​\(δ​\(n−1\)M\)\\tau\\;\\geq\\;F\_\{\\delta,n\}\(M\)\\;:=\\;\\frac\{1\}\{\\delta\}\\,W\\\!\\Big\(\\frac\{\\delta\(n\-1\)\}\{M\}\\Big\), thenϵ≤M\\epsilon\\leq M\. Moreover, for anyM\>0M\>0,δ\>0\\delta\>0, andn\>1n\>1,Fδ,n​\(M\)∈ℝF\_\{\\delta,n\}\(M\)\\in\\mathbb\{R\}\.

Given Theorem[1](https://arxiv.org/html/2607.16431#Thmtheorem1), we can treat the error boundMMas a hyperparameter in our smoothing strategy and adaptively control the temperatureτ\\tauduring training viaFδ,n​\(M\)F\_\{\\delta,n\}\(M\), since the top\-gapδ\\deltaand smoothing sizennare known quantities\. The proof is provided in Appendix[A\.1](https://arxiv.org/html/2607.16431#A1.SS1)\.

### 5\.2Gradient Stability under Candidate\-Pool Sampling

We now analyze stability by treating the sampled candidate pool itself as the source of randomness\. For a fixed promptxx, one training step draws

Y=\(Yw1,…,Ywn,Yl1,…,Yln\)∼Px,Y=\(Y\_\{w\}^\{1\},\\dots,Y\_\{w\}^\{n\},Y\_\{l\}^\{1\},\\dots,Y\_\{l\}^\{n\}\)\\sim P\_\{x\},and the CPO objective maps this pool to either a hard or a smoothed gradient signal\. Hard selection keeps exactly one preferred/non\-preferred pair, while smoothing aggregates the full pool through LSE\. The key instability of hard selection is that the chosen pair itself is random: when different sampled pools select different hard pairs and these pairs induce different updates, this discrete selector randomness produces an intrinsic variance floor\. By contrast, smoothing replaces this one\-hot switch by a continuous weighted mixture whose fluctuation can be controlled through the softmax weighting structure\.

Define

Vsm​\(x,θ\):=𝔼Y​\[‖∇θℒsm​\(θ;x,Y\)−𝔼Y​∇θℒsm​\(θ;x,Y\)‖2\],V\_\{\\mathrm\{sm\}\}\(x,\\theta\):=\\mathbb\{E\}\_\{Y\}\\\!\\Big\[\\big\\\|\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{sm\}\}\(\\theta;x,Y\)\-\\mathbb\{E\}\_\{Y\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{sm\}\}\(\\theta;x,Y\)\\big\\\|^\{2\}\\Big\],Vhard​\(x,θ\):=𝔼Y​\[‖∇θℒhard​\(θ;x,Y\)−𝔼Y​∇θℒhard​\(θ;x,Y\)‖2\]\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\):=\\mathbb\{E\}\_\{Y\}\\\!\\Big\[\\big\\\|\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{hard\}\}\(\\theta;x,Y\)\-\\mathbb\{E\}\_\{Y\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{hard\}\}\(\\theta;x,Y\)\\big\\\|^\{2\}\\Big\]\.
###### Theorem 2\(Smooth Aggregation Improves Gradient Stability\)\.

Under Appendix Assumptions[1](https://arxiv.org/html/2607.16431#Thmassumption1),[2](https://arxiv.org/html/2607.16431#Thmassumption2), and[3](https://arxiv.org/html/2607.16431#Thmassumption3),

Vhard​\(x,θ\)\>Vsm​\(x,θ\)\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\>V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\.

Theorem[2](https://arxiv.org/html/2607.16431#Thmtheorem2)reduces the stability comparison to an interpretable tradeoff between two quantities\. The termp2​m2p^\{2\}m^\{2\}is the selector\-mixture variance floor induced by hard selection: it becomes large when the hard selector places non\-trivial mass on multiple pair indices whose average updates are well separated\. The right\-hand side captures the fluctuation of smoothing:Var⁡\(g¯​\(Y\)\)\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)is the branchwise averaging baseline, while the second term is the additional variance induced by soft weighting\. Crucially, this extra term is multiplied by the shrinkage factorδn​\(τ,R\)\\delta\_\{n\}\(\\tau,R\), which vanishes as the branch score range collapses\. Hence, whenever hard selector competition is strong but the softmax weights remain close to a stable average, smooth aggregation yields strictly smaller gradient variance than hard selection\. The full notation is given in Appendix[A\.2](https://arxiv.org/html/2607.16431#A1.SS2), and the supporting lemmas and proof appear in Appendix[A\.3](https://arxiv.org/html/2607.16431#A1.SS3)\.

## 6Experiments

In this section, we evaluate the proposedRimsby addressing the following research questions:

1. RQ1How doesRimsperform compared to state\-of\-the\-art baselines?
2. RQ2How does the proposed smoothed aggregation compare with alternative strategies?
3. RQ3How doesRimsperform across different preference optimization methods?
4. RQ4How doesRimsperform with varying numbers of retrieved documents?
5. RQ5How sensitive isRimsto the hyperparameters error boundMMand smoothing sizenn?

### 6\.1Datasets and Experiment Settings

Datasets and Evaluation Metrics\.We conduct experiments on four public multi\-hop question answering benchmarks: HotPotQA \(HPQA,Yanget al\.\([2018](https://arxiv.org/html/2607.16431#bib.bib14)\)\), 2WikiMultiHopQA \(2WMHQA,Hoet al\.\([2020](https://arxiv.org/html/2607.16431#bib.bib15)\)\), MultiHopRAGQA \(MHRAGQA,Tang and Yang \([2024](https://arxiv.org/html/2607.16431#bib.bib1)\)\), and MuSiQueQA \(MSQQA,Trivediet al\.\([2022](https://arxiv.org/html/2607.16431#bib.bib16)\)\)\. Following prior work\(Shaoet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib28); Wanget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib21); Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\), we evaluate the first 500 questions from the development splits of HPQA and 2WMHQA\. For MSQQA, we select the first 500 questions from its development set\. As MHRAGQA lacks an official development split, we randomly set aside 20% of the training data for evaluation and use the remaining 80% for training, with the final results also computed over 500 questions\. We employ exact match \(EM\) and F1 score as evaluation metrics, followingLiuet al\.\([2025](https://arxiv.org/html/2607.16431#bib.bib12)\)\.

Baselines\.We compare our method against the following state\-of\-the\-art baselines: 1\) CoT Prompting\(Weiet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib18)\), 2\) ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib19)\), 3\) SelfAsk\(Presset al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib20)\), 4\) BlendFilter\(Wanget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib21)\), 5\) InstructRAG\(Weiet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib22)\), 6\) RoseRAG\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)\. Further background and details are deferred to Appendix[C](https://arxiv.org/html/2607.16431#A3)\.

Implementation Details\.Our experiments are conducted using two representative SLMs: Qwen2\.5\-1\.5B\-Instruct\(Team,[2024](https://arxiv.org/html/2607.16431#bib.bib23)\)and Gemma\-2\-2b\-it\(Teamet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib24)\)\. For the retriever, we utilize the ColBERTv2 retriever\(Santhanamet al\.,[2022](https://arxiv.org/html/2607.16431#bib.bib25)\), a state\-of\-the\-art efficient retrieval method implemented byKhattabet al\.\([2023a](https://arxiv.org/html/2607.16431#bib.bib26)\)\. The knowledge base we employ is the 2017 Wikipedia abstracts dump\(Khattabet al\.,[2023b](https://arxiv.org/html/2607.16431#bib.bib27)\)\. More implementation details can be found in Appendix[D](https://arxiv.org/html/2607.16431#A4)\.

### 6\.2Overall Performance Comparison

We evaluate the effectiveness ofRimsacross four benchmarks against six strong baseline methods to answer RQ1\. Results for Qwen2\.5\-1\.5B\-Instruct and Gemma\-2\-2B\-it are reported in Table[1](https://arxiv.org/html/2607.16431#S6.T1)\.

Rimsconsistently outperforms strong baselines across all benchmarks by a substantial margin\. The gains are especially pronounced on the more challenging multi\-hop datasets \(2WMHQA and MHRAGQA\), where retrieved passages tend to be lengthy and noisy\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\), indicating thatRimsis effective at aggregating diverse evidence while maintaining reasoning consistency\. Prompt\-based and decomposition\-based methods \(ReAct, SelfAsk, BlendFilter\) are limited by SLMs’ intrinsic reasoning capacity, while fine\-tuning\-based approaches that explicitly enhance robustness to noisy contexts show stronger performance\. Overall, the superior and stable performance ofRimshighlights the importance of designing optimization strategies specifically for SLMs rather than directly transferring techniques developed for large\-scale models\.

Table 1:Performance comparison of different methods with Qwen2\.5\-1\.5B\-Instruct and gemma\-2\-2b\-it as the backbone models\. The best and second\-best results are highlighted inboldandunderlined, respectively\. \(HPQA: HotPotQA, 2WMHQA: 2WikiMultiHopQA, MHRAGQA: MultiHopRAGQA, MSQQA: MuSiQueQA\)
### 6\.3Effectiveness of Data Aggregation Methods

Table 2:Performance \(EM/F1\) comparison ofRimswith different aggregation strategies\. Experiments are conducted on the MHRAGQA using Qwen2\.5\-1\.5B\-Instruct as the backbone model\. \(Weighted: weight training data based on untrained model’s likelihood;Average: assign equal weights\. Detailed explanations of these strategies are provided in Appendix Table[3](https://arxiv.org/html/2607.16431#A2.T3)\.Aggregation StrategySmoothing size = 3Smoothing size = 4PositiveNegativeEMF1EMF1AverageAverage48\.048\.550\.651\.2AverageWeighted47\.047\.443\.444\.3WeightedAverage45\.246\.048\.849\.1WeightedWeighted48\.248\.747\.447\.8Rims\-DPO50\.8051\.4352\.4052\.98To answer RQ2, we compare our proposed smoothed aggregation with two alternative strategies for combining multi\-pair preference data\(x,yw1,⋯,ywn,yl1,⋯,yln\)\(x,y\_\{w\}^\{1\},\\cdots,y\_\{w\}^\{n\},y\_\{l\}^\{1\},\\cdots,y\_\{l\}^\{n\}\)into a pseudo\-pair:*Weighted*\(fixed weights based on base model likelihoods\),*Average*\(equal weights\)\. More details can be found in Appendix[B](https://arxiv.org/html/2607.16431#A2)\.

Results are reported in Table[2](https://arxiv.org/html/2607.16431#S6.T2), using Qwen2\.5\-1\.5B\-Instruct as the backbone\. Across two smoothing sizes, all heuristic\-based methods exhibit significantly worse performance than our proposedRims\. Average and Weighted aggregation dilute informative signals by mixing responses of heterogeneous quality\. In contrast, our proposedRimsconsistently achieves the best performance and remains stable across different smoothing sizes, confirming that LSE smoothing effectively balances the focus on hard examples with the utilization of all candidates\.

### 6\.4Effectiveness Across Different Preference Optimization Methods

![Refer to caption](https://arxiv.org/html/2607.16431v1/x2.png)Figure 2:Performance \(EM/F1\) ofRimswhen using different preference optimization methods\. The backbone model is Qwen2\.5\-1\.5B\-Instruct\.Figure[2](https://arxiv.org/html/2607.16431#S6.F2)reports the performance of three preference optimization methods \(DPO, CPO, ORPO\) with a fixed smoothing component to answer RQ3\.Rimsconsistently achieves strong and stable performance across all three methods, demonstrating the robustness of the proposed smoothing mechanism\. Among them, DPO yields the highest EM and F1 scores, with the gap being especially pronounced on more challenging multi\-hop datasets such as MHRAGQA, suggesting that DPO’s reference\-policy regularization is particularly beneficial for capturing complex reasoning chains under noisy retrieval\.

### 6\.5Effectiveness with a Varying Number of Retrieved Documents

Figure[3\(a\)](https://arxiv.org/html/2607.16431#S6.F3.sf1)comparesRimsand vanilla RAG on HPQA, with the number of retrieved documentsKKvarying from 0 to 10 to answer RQ4\. WhenK=0K=0, the backbone relies solely on its internal knowledge;Rimsdemonstrates comparable performance to vanilla RAG, indicating that the optimization did not degrade the model’s intrinsic reasoning abilities and pretrained knowledge\.

AsKKincreases from 1 to 5, both methods benefit from improved retrieval recall, butRimsexhibits much steadier improvement while vanilla RAG shows only marginal gains beyondK=1K=1\. WhenK\>5K\>5, vanilla RAG performance begins to degrade due to increased noise, whereasRimsmaintains its superior performance and shows greater resilience\. These results demonstrate thatRimsprovides more robust evidence utilization than vanilla RAG, enabling consistent performance gains even as the retrieved context grows noisier\.

![Refer to caption](https://arxiv.org/html/2607.16431v1/x3.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.16431v1/x4.png)\(b\)
![Refer to caption](https://arxiv.org/html/2607.16431v1/x5.png)\(c\)

Figure 3:Sensitivity analysis ofRimsacross different hyperparameter settings\. Experiments are conducted on the HPQA dataset using Qwen2\.5\-1\.5B\-Instruct as the backbone model\. \(a\) EM under varying retrieved document counts\. \(b\) EM/F1 under varying LSE approximation error boundMMwith smoothing sizen=4n=4\. \(c\) EM/F1 under varying smoothing sizenn; inner box shows the interquartile range across 4 replicates and outer violin shows the density\.
### 6\.6Sensitivity Analysis of Hyperparameters

LSE Approximation Error Analysis\.The error boundMMin Theorem[1](https://arxiv.org/html/2607.16431#Thmtheorem1)controls how closely the LSE operator approximates the exact max/min, with the temperatureτ\\tauadaptively adjusted viaFδ,n​\(M\)F\_\{\\delta,n\}\(M\)during training\. Figure[3\(b\)](https://arxiv.org/html/2607.16431#S6.F3.sf2)shows the impact ofMMon model performance\. WhenMMis small,Rimsachieves performance comparable to RoseRAG, as the LSE operator degenerates to hard selection, validating our theoretical analysis\. Performance peaks atM=1M=1, suggesting that a moderately relaxed bound enables the model to leverage a richer set of preference pairs while preserving the discriminative structure of margin\-aware selection\.

Smoothing Size\.The smoothing sizenncontrols the number of candidate responses aggregated in the LSE operator\. Figure[3\(c\)](https://arxiv.org/html/2607.16431#S6.F3.sf3)shows the performance under different smoothing sizes with a fixed error boundMM, whereτ\\tauis adaptively adjusted following Theorem[1](https://arxiv.org/html/2607.16431#Thmtheorem1)\. Both EM and F1 scores exhibit a clear degradation asnnincreases, even when the worst\-case approximation error is controlled\. This indicates that larger smoothing sizes dilute the influence of top\-ranked candidates by aggregating over a broader spectrum of scores; while controllingMMguarantees closeness to the true max in magnitude, increasingnnalters the relative contributions of subdominant candidates\. Consequently, moderate smoothing sizes achieve a better balance between optimization smoothness and ranking fidelity\.

## 7Conclusions

We presentedRims, a preference optimization framework that addresses the underutilization of multi\-pair preference data in SLM\-based RAG\. The core idea is to replace hard pair selection with a differentiable LSE\-based smoothing mechanism that aggregates all preference candidates into a unified objective, preserving informative gradient signals while retaining the discriminative structure of margin\-aware optimization\. The resulting formulation is algorithm\-agnostic, directly applicable to DPO, CPO, and ORPO without modification\. We further provided theoretical guarantees showing that the LSE approximation error is controllable via an interpretable hyperparameter, and that smooth aggregation yields provably tighter gradient alignment than hard selection\. Extensive experiments on four multi\-hop question answering benchmarks with two representative SLM backbones demonstrate thatRimsconsistently outperforms state\-of\-the\-art baselines in both Exact Match and F1, highlighting the importance of structured multi\-pair aggregation for robust evidence utilization under noisy retrieval conditions\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hSyW5go0v8)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§3\.2](https://arxiv.org/html/2607.16431#S3.SS2.p1.5)\.
- P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. Molchanov \(2025\)Small language models are the future of agentic ai\.External Links:2506\.02153,[Link](https://arxiv.org/abs/2506.02153),[Document](https://dx.doi.org/10.48550/arXiv.2506.02153)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- H\. Chen, G\. He, L\. Yuan, G\. Cui, H\. Su, and J\. Zhu \(2024\)Noise contrastive alignment of language models with explicit rewards\.External Links:2402\.05369,[Link](https://arxiv.org/abs/2402.05369)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§1](https://arxiv.org/html/2607.16431#S1.p4.1),[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. Ma \(2025\)SFT memorizes, rl generalizes: a comparative study of foundation model post\-training\.External Links:2501\.17161,[Link](https://arxiv.org/abs/2501.17161)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- G\. Dong, Y\. Zhu, C\. Zhang, Z\. Wang, Z\. Dou, and J\. Wen \(2024\)Understand what llm needs: dual preference alignment for retrieval\-augmented generation\.External Links:2406\.18676,[Link](https://arxiv.org/abs/2406.18676)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)Kto: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1)\.
- Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, H\. Wang, H\. Wang,et al\.\(2023\)Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.109972\(1\),pp\. 32\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§4\.1](https://arxiv.org/html/2607.16431#S4.SS1.p3.8)\.
- T\. Gupta, R\. Madhavan, X\. Zhang, C\. Bansal, and S\. Rajmohan \(2025\)AMPO: active multi\-preference optimization for self\-play preference selection\.External Links:2502\.18293,[Link](https://arxiv.org/abs/2502.18293)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.External Links:2011\.01060,[Link](https://arxiv.org/abs/2011.01060)Cited by:[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)ORPO: monolithic preference optimization without reference model\.External Links:2403\.07691,[Link](https://arxiv.org/abs/2403.07691)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§1](https://arxiv.org/html/2607.16431#S1.p5.1),[§3\.2](https://arxiv.org/html/2607.16431#S3.SS2.p3.9),[§4\.3](https://arxiv.org/html/2607.16431#S4.SS3.p3.7)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p2.1)\.
- G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave \(2022\)Atlas: few\-shot learning with retrieval augmented language models\.External Links:2208\.03299,[Link](https://arxiv.org/abs/2208.03299)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- Z\. Jiang, F\. F\. Xu, L\. Gao, Z\. Sun, Q\. Liu, J\. Dwivedi\-Yu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Active retrieval augmented generation\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 7969–7992\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- O\. Khattab, K\. Santhanam, X\. L\. Li, D\. Hall, P\. Liang, C\. Potts, and M\. Zaharia \(2023a\)Demonstrate\-search\-predict: composing retrieval and language models for knowledge\-intensive nlp\.External Links:2212\.14024,[Link](https://arxiv.org/abs/2212.14024)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p3.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2023b\)DSPy: compiling declarative language model calls into self\-improving pipelines\.External Links:2310\.03714,[Link](https://arxiv.org/abs/2310.03714)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p3.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1)\.
- H\. Le, Q\. Tran, D\. Nguyen, K\. Do, S\. Mittal, K\. Ogueji, and S\. Venkatesh \(2024\)Multi\-reference preference optimization for large language models\.External Links:2405\.16388,[Link](https://arxiv.org/abs/2405.16388)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2021\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.External Links:2005\.11401,[Link](https://arxiv.org/abs/2005.11401)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- X\. Li, C\. Zhu, L\. Li, Z\. Yin, T\. Sun, and X\. Qiu \(2024\)LLatrieval: llm\-verified retrieval for verifiable generation\.External Links:2311\.07838,[Link](https://arxiv.org/abs/2311.07838)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- T\. Liu, H\. Jiang, T\. Wang, R\. Xu, Y\. Yu, L\. Zhang, T\. Zhao, and H\. Wang \(2025\)RoseRAG: robust retrieval\-augmented generation with small\-scale LLMs via margin\-aware preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 13036–13054\.Cited by:[6th item](https://arxiv.org/html/2607.16431#A3.I1.i6.p1.1),[§1](https://arxiv.org/html/2607.16431#S1.p2.1),[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§1](https://arxiv.org/html/2607.16431#S1.p4.1),[§2](https://arxiv.org/html/2607.16431#S2.p2.1),[§3\.3](https://arxiv.org/html/2607.16431#S3.SS3.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.16431#S6.SS2.p2.1)\.
- T\. Liu, Y\. Zhao, R\. Joshi, M\. Khalman, M\. Saleh, P\. J\. Liu, and J\. Liu \(2024\)Statistical rejection sampling improves preference optimization\.External Links:2309\.06657,[Link](https://arxiv.org/abs/2309.06657)Cited by:[§4\.1](https://arxiv.org/html/2607.16431#S4.SS1.p3.8)\.
- Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, X\. Zhang, N\. D\. Lane, and M\. Xu \(2025\)Small language models: survey, measurements, and insights\.External Links:2409\.15790,[Link](https://arxiv.org/abs/2409.15790)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)Simpo: simple preference optimization with a reference\-free reward\.Advances in Neural Information Processing Systems37,pp\. 124198–124235\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1)\.
- F\. Nielsen and K\. Sun \(2016\)Guaranteed bounds on information\-theoretic measures of univariate mixtures using piecewise log\-sum\-exp inequalities\.Entropy18\(12\),pp\. 442\.External Links:ISSN 1099\-4300,[Link](http://dx.doi.org/10.3390/e18120442),[Document](https://dx.doi.org/10.3390/e18120442)Cited by:[§4\.2](https://arxiv.org/html/2607.16431#S4.SS2.p2.5)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§3\.2](https://arxiv.org/html/2607.16431#S3.SS2.p1.5)\.
- O\. Ovadia, M\. Brief, M\. Mishaeli, and O\. Elisha \(2024\)Fine\-tuning or retrieval? comparing knowledge injection in LLMs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 237–250\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.15/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.15)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. Smith, and M\. Lewis \(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5687–5711\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.378/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378)Cited by:[3rd item](https://arxiv.org/html/2607.16431#A3.I1.i3.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§1](https://arxiv.org/html/2607.16431#S1.p5.1),[§3\.2](https://arxiv.org/html/2607.16431#S3.SS2.p3.9),[§4\.3](https://arxiv.org/html/2607.16431#S4.SS3.p3.7)\.
- K\. Santhanam, O\. Khattab, J\. Saad\-Falcon, C\. Potts, and M\. Zaharia \(2022\)ColBERTv2: effective and efficient retrieval via lightweight late interaction\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),pp\. 3715–3734\.Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p3.1)\.
- Z\. Shao, Y\. Gong, Y\. Shen, M\. Huang, N\. Duan, and W\. Chen \(2023\)Enhancing retrieval\-augmented large language models with iterative retrieval\-generation synergy\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 9248–9274\.Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p2.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1)\.
- Y\. Tang and Y\. Yang \(2024\)Multihop\-rag: benchmarking retrieval\-augmented generation for multi\-hop queries\.arXiv preprint arXiv:2401\.15391\.Cited by:[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p3.1)\.
- Q\. Team \(2024\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p3.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.External Links:2108\.00573,[Link](https://arxiv.org/abs/2108.00573)Cited by:[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1)\.
- G\. Vernikos, A\. Bražinskas, J\. Adamek, J\. Mallinson, A\. Severyn, and E\. Malmi \(2024\)Small language models improve giants by rewriting their outputs\.External Links:2305\.13514,[Link](https://arxiv.org/abs/2305.13514)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. Gallouédec \(2020\)TRL: transformer reinforcement learning\.GitHub\.Note:[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1)\.
- F\. Wang, X\. Wan, R\. Sun, J\. Chen, and S\. Ö\. Arık \(2025a\)Astute rag: overcoming imperfect retrieval augmentation and knowledge conflicts for large language models\.External Links:2410\.07176,[Link](https://arxiv.org/abs/2410.07176)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1)\.
- H\. Wang, L\. Xu, H\. Liu, Y\. Liu, X\. Zhao, B\. Zeng, L\. Shao, L\. Wang, W\. Luo, and K\. Zhang \(2025b\)Beyond single\-reward: multi\-pair, multi\-perspective preference optimization for machine translation\.External Links:2510\.13434,[Link](https://arxiv.org/abs/2510.13434)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- H\. Wang, R\. Li, H\. Jiang, J\. Tian, Z\. Wang, C\. Luo, X\. Tang, M\. X\. Cheng, T\. Zhao, and J\. Gao \(2024\)BlendFilter: advancing retrieval\-augmented large language models via query generation blending and knowledge filtering\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 1009–1025\.Cited by:[4th item](https://arxiv.org/html/2607.16431#A3.I1.i4.p1.1),[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§2](https://arxiv.org/html/2607.16431#S2.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1)\.
- Y\. Wang, P\. Li, M\. Sun, and Y\. Liu \(2023\)Self\-knowledge guided retrieval augmentation for large language models\.External Links:2310\.05002,[Link](https://arxiv.org/abs/2310.05002)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1),[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2023\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[1st item](https://arxiv.org/html/2607.16431#A3.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.16431#S1.p5.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1)\.
- Z\. Wei, W\. Chen, and Y\. Meng \(2025\)InstructRAG: instructing retrieval\-augmented generation via self\-synthesized rationales\.External Links:2406\.13629,[Link](https://arxiv.org/abs/2406.13629)Cited by:[5th item](https://arxiv.org/html/2607.16431#A3.I1.i5.p1.1),[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§2](https://arxiv.org/html/2607.16431#S2.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations,pp\. 38–45\.Cited by:[Appendix D](https://arxiv.org/html/2607.16431#A4.p1.1)\.
- C\. Xiang, T\. Wu, Z\. Zhong, D\. Wagner, D\. Chen, and P\. Mittal \(2024\)Certifiably robust rag against retrieval corruption\.External Links:2405\.15556,[Link](https://arxiv.org/abs/2405.15556)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- S\. Xie, F\. Zhu, J\. Wang, L\. Wen, W\. Dai, X\. Chen, J\. Zhu, K\. Zhou, and B\. Zheng \(2024\)MPPO: multi pair\-wise preference optimization for llms with arbitrary negative samples\.External Links:2412\.15244,[Link](https://arxiv.org/abs/2412.15244)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- H\. Xu, A\. Sharaf, Y\. Chen, W\. Tan, L\. Shen, B\. V\. Durme, K\. Murray, and Y\. J\. Kim \(2024a\)Contrastive preference optimization: pushing the boundaries of llm performance in machine translation\.External Links:2401\.08417,[Link](https://arxiv.org/abs/2401.08417)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§1](https://arxiv.org/html/2607.16431#S1.p5.1),[§3\.2](https://arxiv.org/html/2607.16431#S3.SS2.p3.9),[§4\.3](https://arxiv.org/html/2607.16431#S4.SS3.p3.7)\.
- J\. Xu, Z\. Li, W\. Chen, Q\. Wang, X\. Gao, Q\. Cai, and Z\. Ling \(2024b\)On\-device language models: a comprehensive review\.External Links:2409\.00088,[Link](https://arxiv.org/abs/2409.00088)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- R\. Xu, H\. Liu, S\. Nag, Z\. Dai, Y\. Xie, X\. Tang, C\. Luo, Y\. Li, J\. C\. Ho, C\. Yang, and Q\. He \(2025\)SimRAG: self\-improving retrieval\-augmented generation for adapting large language models to specialized domains\.External Links:2410\.17952,[Link](https://arxiv.org/abs/2410.17952)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2369–2380\.External Links:[Link](https://aclanthology.org/D18-1259/),[Document](https://dx.doi.org/10.18653/v1/D18-1259)Cited by:[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[2nd item](https://arxiv.org/html/2607.16431#A3.I1.i2.p1.1),[§6\.1](https://arxiv.org/html/2607.16431#S6.SS1.p2.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2024\)Making retrieval\-augmented language models robust to irrelevant context\.External Links:2310\.01558,[Link](https://arxiv.org/abs/2310.01558)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p2.1),[§1](https://arxiv.org/html/2607.16431#S1.p3.1),[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- W\. Yu, H\. Zhang, X\. Pan, P\. Cao, K\. Ma, J\. Li, H\. Wang, and D\. Yu \(2024a\)Chain\-of\-note: enhancing robustness in retrieval\-augmented language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 14672–14685\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.813/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.813)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- Y\. Yu, W\. Ping, Z\. Liu, B\. Wang, J\. You, C\. Zhang, M\. Shoeybi, and B\. Catanzaro \(2024b\)RankRAG: unifying context ranking with retrieval\-augmented generation in LLMs\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=S1fc92uemC)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p1.1)\.
- R\. Zhang, Y\. Xu, Y\. Xiao, R\. Zhu, X\. Jiang, X\. Chu, J\. Zhao, and Y\. Wang \(2024\)KnowPO: knowledge\-aware preference optimization for controllable knowledge selection in retrieval\-augmented language models\.External Links:2408\.03297,[Link](https://arxiv.org/abs/2408.03297)Cited by:[§2](https://arxiv.org/html/2607.16431#S2.p2.1)\.
- Z\. Zhou, X\. Ning, K\. Hong, T\. Fu, J\. Xu, S\. Li, Y\. Lou, L\. Wang, Z\. Yuan, X\. Li, S\. Yan, G\. Dai, X\. Zhang, Y\. Dong, and Y\. Wang \(2024\)A survey on efficient inference for large language models\.External Links:2404\.14294,[Link](https://arxiv.org/abs/2404.14294)Cited by:[§1](https://arxiv.org/html/2607.16431#S1.p1.1)\.

## Appendix AOmitted Proofs

This section collects all proofs omitted from the main text\. Appendix[A\.1](https://arxiv.org/html/2607.16431#A1.SS1)proves the approximation\-error result used in the main text\. Appendix[A\.2](https://arxiv.org/html/2607.16431#A1.SS2)then switches to the CPO setting and analyzes gradient stability under candidate\-pool sampling\. The two analyses are independent: the first controls LSE approximation error, while the second explains why smoothing produces a more stable training signal than hard pair selection\.

### A\.1LogSumExp Error Bound Analysis

###### Lemma 1\(LSE Error Bound\)\.

Letn\>1n\>1,τ\>0\\tau\>0, andx1,…,xn∈ℝx\_\{1\},\\dots,x\_\{n\}\\in\\mathbb\{R\}\. Letx\(1\)≤⋯≤x\(n\)x\_\{\(1\)\}\\leq\\cdots\\leq x\_\{\(n\)\}be the order statistics and define

LSEτ⁡\(x1,…,xn\):=1τ​log⁡\(∑i=1neτ​xi\)\.\\operatorname\{LSE\}\_\{\\tau\}\(x\_\{1\},\\dots,x\_\{n\}\)\\;:=\\;\\frac\{1\}\{\\tau\}\\log\\Big\(\\sum\_\{i=1\}^\{n\}e^\{\\tau x\_\{i\}\}\\Big\)\.Ifδ:=x\(n\)−x\(n−1\)≥0\\delta:=x\_\{\(n\)\}\-x\_\{\(n\-1\)\}\\geq 0, then the errorϵ:=LSEτ⁡\(x1,…,xn\)−x\(n\)\\epsilon\\;:=\\;\\operatorname\{LSE\}\_\{\\tau\}\(x\_\{1\},\\dots,x\_\{n\}\)\-x\_\{\(n\)\}satisfies

ϵ≤1τ​log⁡n​and​ϵ≤n−1τ​e−τ​δ\\epsilon\\leq\\frac\{1\}\{\\tau\}\\log n\\text\{ and \}\\epsilon\\leq\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}

###### Proof\.

Factor outeτ​x\(n\)e^\{\\tau x\_\{\(n\)\}\}from the exponential sum:

∑i=1neτ​xi=eτ​x\(n\)​∑i=1neτ​\(x\(i\)−x\(n\)\)=eτ​x\(n\)​\(1\+∑i=1n−1eτ​\(x\(i\)−x\(n\)\)\)\\begin\{split\}\\sum\_\{i=1\}^\{n\}e^\{\\tau x\_\{i\}\}&=e^\{\\tau x\_\{\(n\)\}\}\\sum\_\{i=1\}^\{n\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\\\ &=e^\{\\tau x\_\{\(n\)\}\}\\Big\(1\+\\sum\_\{i=1\}^\{n\-1\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\Big\)\\end\{split\}Therefore

ϵ=1τ​log⁡\(1\+∑i=1n−1eτ​\(x\(i\)−x\(n\)\)\)\\epsilon=\\frac\{1\}\{\\tau\}\\log\\Big\(1\+\\sum\_\{i=1\}^\{n\-1\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\Big\)
Uniform bound\.Sincex\(i\)≤x\(n\)x\_\{\(i\)\}\\leq x\_\{\(n\)\}for allii, we haveeτ​\(x\(i\)−x\(n\)\)≤1e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\leq 1, and thus

∑i=1neτ​\(x\(i\)−x\(n\)\)≤n\.\\sum\_\{i=1\}^\{n\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\leq n\.Consequently,

ϵ=1τ​log⁡\(∑i=1neτ​\(x\(i\)−x\(n\)\)\)≤1τ​log⁡n\.\\epsilon=\\frac\{1\}\{\\tau\}\\log\\\!\\Big\(\\sum\_\{i=1\}^\{n\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\Big\)\\leq\\frac\{1\}\{\\tau\}\\log n\.
δ\\delta\-Bound\.For eachi≤n−1i\\leq n\-1, we havex\(i\)≤x\(n−1\)x\_\{\(i\)\}\\leq x\_\{\(n\-1\)\}, hencex\(i\)−x\(n\)≤x\(n−1\)−x\(n\)=−δ,x\_\{\(i\)\}\-x\_\{\(n\)\}\\leq x\_\{\(n\-1\)\}\-x\_\{\(n\)\}=\-\\delta,, soeτ​\(x\(i\)−x\(n\)\)≤e−τ​δe^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\leq e^\{\-\\tau\\delta\}\. Summing these bounds gives

∑i=1n−1eτ​\(x\(i\)−x\(n\)\)≤\(n−1\)​e−τ​δ,\\sum\_\{i=1\}^\{n\-1\}e^\{\\tau\(x\_\{\(i\)\}\-x\_\{\(n\)\}\)\}\\leq\(n\-1\)e^\{\-\\tau\\delta\},and thus

ϵ≤1τ​log⁡\(1\+\(n−1\)​e−τ​δ\)\.\\epsilon\\leq\\frac\{1\}\{\\tau\}\\log\\Big\(1\+\(n\-1\)e^\{\-\\tau\\delta\}\\Big\)\.Finally, sincelog⁡\(1\+u\)≤u\\log\(1\+u\)\\leq ufor allu≥0u\\geq 0, we obtain

ϵ≤1τ​log⁡\(1\+\(n−1\)​e−τ​δ\)≤n−1τ​e−τ​δ\.\\epsilon\\leq\\frac\{1\}\{\\tau\}\\log\\Big\(1\+\(n\-1\)e^\{\-\\tau\\delta\}\\Big\)\\leq\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}\.∎

###### Lemma 2\(LambertWWinversion equivalence\)\.

Leta\>0a\>0\. For anyy\>0y\>0,

y​ey≥a⟺y≥W​\(a\),ye^\{y\}\\geq a\\;\\Longleftrightarrow\\;y\\geq W\(a\),whereWWis the \(principal\) LambertWWfunction andW​\(a\)∈ℝW\(a\)\\in\\mathbb\{R\}\.

###### Proof\.

Fixa\>0a\>0and consider the function

f:\(0,∞\)→\(0,∞\),f​\(y\)=y​ey\.f:\(0,\\infty\)\\to\(0,\\infty\),\\quad f\(y\)=ye^\{y\}\.We first show thatffis strictly increasing on\(0,∞\)\(0,\\infty\)\. Indeed,

f′​\(y\)=ey​\(1\+y\)\>0for all​y\>0,f^\{\\prime\}\(y\)=e^\{y\}\(1\+y\)\>0\\qquad\\text\{for all \}y\>0,soffis strictly increasing and hence injective\.

By definition of the \(principal\) LambertWWfunction,W​\(a\)W\(a\)is the unique real numberwwsuch that

Existence and uniqueness follow here becauseffis continuous, strictly increasing, and satisfieslimy→0f​\(y\)=0\\lim\_\{y\\to 0\}f\(y\)=0andlimy→∞f​\(y\)=∞\\lim\_\{y\\to\\infty\}f\(y\)=\\infty; thus for eacha\>0a\>0there is a uniquew\>0w\>0withf​\(w\)=af\(w\)=a\.

Sinceffis strictly increasing andf​\(W​\(a\)\)=af\(W\(a\)\)=a, for anyy\>0y\>0we have

y​ey≥a=f​\(W​\(a\)\)⟺f​\(y\)≥f​\(W​\(a\)\)⟺y≥W​\(a\)\.∎ye^\{y\}\\geq a=f\(W\(a\)\)\\;\\Longleftrightarrow\\;f\(y\)\\geq f\(W\(a\)\)\\;\\Longleftrightarrow\\;y\\geq W\(a\)\.\\qed

###### Theorem 3\(LSE Error Bound Control\)\.

LetM\>0M\>0,n\>1n\>1, andx1,…,xn∈ℝx\_\{1\},\\dots,x\_\{n\}\\in\\mathbb\{R\}\. Letx\(1\)≤⋯≤x\(n\)x\_\{\(1\)\}\\leq\\cdots\\leq x\_\{\(n\)\}and assume the top\-gapδ:=x\(n\)−x\(n−1\)\>0\\delta:=x\_\{\(n\)\}\-x\_\{\(n\-1\)\}\>0,ϵ:=LSEτ⁡\(x1,…,xn\)−x\(n\)\\epsilon:=\\operatorname\{LSE\}\_\{\\tau\}\(x\_\{1\},\\dots,x\_\{n\}\)\-x\_\{\(n\)\}~forτ\>0\\tau\>0\. If

τ≥Fδ,n​\(M\):=1δ​W​\(δ​\(n−1\)M\),\\tau\\;\\geq\\;F\_\{\\delta,n\}\(M\)\\;:=\\;\\frac\{1\}\{\\delta\}\\,W\\\!\\Big\(\\frac\{\\delta\(n\-1\)\}\{M\}\\Big\),thenϵ≤M\\epsilon\\leq M\. Moreover, for anyM\>0M\>0,δ\>0\\delta\>0, andn\>1n\>1,Fδ,n​\(M\)∈ℝF\_\{\\delta,n\}\(M\)\\in\\mathbb\{R\}\.

###### Proof\.

By Lemma[1](https://arxiv.org/html/2607.16431#Thmlemma1),

ϵ≤n−1τ​e−τ​δ\.\\epsilon\\leq\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}\.Thus it suffices to show thatτ≥Fδ,n​\(M\)\\tau\\geq F\_\{\\delta,n\}\(M\)impliesn−1τ​e−τ​δ≤M\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}\\leq M\. Lety:=τ​δ\>0y:=\\tau\\delta\>0anda:=δ​\(n−1\)M\>0a:=\\frac\{\\delta\(n\-1\)\}\{M\}\>0\. Sinceτ≥Fδ,n​\(M\)\\tau\\geq F\_\{\\delta,n\}\(M\)meansy≥W​\(a\)y\\geq W\(a\), Lemma[2](https://arxiv.org/html/2607.16431#Thmlemma2)\(applied right\-to\-left\) givesy​ey≥aye^\{y\}\\geq a, i\.e\.,

δ​\(n−1\)M≤y​ey⟺δ​\(n−1\)M​e−y≤y⟺n−1τ​e−τ​δ≤M\.\\frac\{\\delta\(n\-1\)\}\{M\}\\leq ye^\{y\}\\;\\Longleftrightarrow\\;\\frac\{\\delta\(n\-1\)\}\{M\}\\,e^\{\-y\}\\leq y\\;\\Longleftrightarrow\\;\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}\\leq M\.Henceϵ≤n−1τ​e−τ​δ≤M\\epsilon\\leq\\frac\{n\-1\}\{\\tau\}e^\{\-\\tau\\delta\}\\leq M\.

For the final claim, sinceM\>0M\>0,δ\>0\\delta\>0, andn\>1n\>1, the argumentδ​\(n−1\)M\\displaystyle\\frac\{\\delta\(n\-1\)\}\{M\}is strictly positive\. The principal branchW​\(z\)W\(z\)is real\-valued for allz≥0z\\geq 0, henceW​\(δ​\(n−1\)M\)∈ℝ\\displaystyle W\\\!\\big\(\\frac\{\\delta\(n\-1\)\}\{M\}\\big\)\\in\\mathbb\{R\}and thereforeFδ,n​\(M\)∈ℝF\_\{\\delta,n\}\(M\)\\in\\mathbb\{R\}\. ∎

### A\.2Notation for Gradient Stability Analysis

Throughout this appendix,∥⋅∥\\\|\\cdot\\\|denotes the Euclidean norm\. For probability vectorsp,q∈ℝnp,q\\in\\mathbb\{R\}^\{n\}, we use the total variation distance

TV⁡\(p,q\):=12​∑i=1n\|pi−qi\|=1−∑i=1nmin⁡\{pi,qi\}\.\\operatorname\{TV\}\(p,q\):=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\|p\_\{i\}\-q\_\{i\}\|=1\-\\sum\_\{i=1\}^\{n\}\\min\\\{p\_\{i\},q\_\{i\}\\\}\.
#### CPO\-Specific Quantities\.

Let

ϕ​\(u\):=−log⁡σ​\(u\)\.\\phi\(u\):=\-\\log\\sigma\(u\)\.For a fixed promptxxand sampled candidate pool

Y=\(Yw1,…,Ywn,Yl1,…,Yln\)∼Px,Y=\(Y\_\{w\}^\{1\},\\dots,Y\_\{w\}^\{n\},Y\_\{l\}^\{1\},\\dots,Y\_\{l\}^\{n\}\)\\sim P\_\{x\},define branchwise scores and score gradients by

siw​\(Y\):=log⁡πθ​\(Ywi∣x\),sil​\(Y\):=log⁡πθ​\(Yli∣x\),s\_\{i\}^\{w\}\(Y\):=\\log\\pi\_\{\\theta\}\(Y\_\{w\}^\{i\}\\mid x\),\\qquad s\_\{i\}^\{l\}\(Y\):=\\log\\pi\_\{\\theta\}\(Y\_\{l\}^\{i\}\\mid x\),ui​\(Y\):=∇θlog⁡πθ​\(Ywi∣x\),vi​\(Y\):=∇θlog⁡πθ​\(Yli∣x\)\.u\_\{i\}\(Y\):=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\_\{w\}^\{i\}\\mid x\),\\qquad v\_\{i\}\(Y\):=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\_\{l\}^\{i\}\\mid x\)\.The hard\-selected indices and smoothed branch weights are

iw⋆​\(Y\):=arg⁡mini⁡siw​\(Y\),il⋆​\(Y\):=arg⁡maxi⁡sil​\(Y\),I​\(Y\):=\(iw⋆​\(Y\),il⋆​\(Y\)\),i\_\{w\}^\{\\star\}\(Y\):=\\arg\\min\_\{i\}s\_\{i\}^\{w\}\(Y\),\\qquad i\_\{l\}^\{\\star\}\(Y\):=\\arg\\max\_\{i\}s\_\{i\}^\{l\}\(Y\),\\qquad I\(Y\):=\(i\_\{w\}^\{\\star\}\(Y\),i\_\{l\}^\{\\star\}\(Y\)\),qw,i​\(Y\):=e−τ​siw​\(Y\)∑j=1ne−τ​sjw​\(Y\),ql,i​\(Y\):=eτ​sil​\(Y\)∑j=1neτ​sjl​\(Y\)\.q\_\{w,i\}\(Y\):=\\frac\{e^\{\-\\tau s\_\{i\}^\{w\}\(Y\)\}\}\{\\sum\_\{j=1\}^\{n\}e^\{\-\\tau s\_\{j\}^\{w\}\(Y\)\}\},\\qquad q\_\{l,i\}\(Y\):=\\frac\{e^\{\\tau s\_\{i\}^\{l\}\(Y\)\}\}\{\\sum\_\{j=1\}^\{n\}e^\{\\tau s\_\{j\}^\{l\}\(Y\)\}\}\.For hard selection, we interpret the update on each sampled poolYYas the active\-branch gradient, i\.e\., the gradient evaluated with the selected indicesiw⋆​\(Y\)i\_\{w\}^\{\\star\}\(Y\)andil⋆​\(Y\)i\_\{l\}^\{\\star\}\(Y\)held fixed\. Equivalently, away from selector\-switching boundaries, the hard objective is locally smooth and the formula below is its ordinary gradient\. Define the hard and smoothed margins by

Δhard​\(Y\):=β​\(siw⋆​\(Y\)w​\(Y\)−sil⋆​\(Y\)l​\(Y\)\),\\Delta\_\{\\mathrm\{hard\}\}\(Y\):=\\beta\\bigl\(s\_\{i\_\{w\}^\{\\star\}\(Y\)\}^\{w\}\(Y\)\-s\_\{i\_\{l\}^\{\\star\}\(Y\)\}^\{l\}\(Y\)\\bigr\),Δsm​\(Y\):=β​\(−LSEτ⁡\(−s1w​\(Y\),…,−snw​\(Y\)\)−LSEτ⁡\(s1l​\(Y\),…,snl​\(Y\)\)\)\.\\Delta\_\{\\mathrm\{sm\}\}\(Y\):=\\beta\\Big\(\-\\operatorname\{LSE\}\_\{\\tau\}\(\-s\_\{1\}^\{w\}\(Y\),\\dots,\-s\_\{n\}^\{w\}\(Y\)\)\-\\operatorname\{LSE\}\_\{\\tau\}\(s\_\{1\}^\{l\}\(Y\),\\dots,s\_\{n\}^\{l\}\(Y\)\)\\Big\)\.The corresponding gradients are

ghard​\(Y\)\\displaystyle g\_\{\\mathrm\{hard\}\}\(Y\):=β​ϕ′​\(Δhard​\(Y\)\)​\(uiw⋆​\(Y\)​\(Y\)−vil⋆​\(Y\)​\(Y\)\),\\displaystyle:=\\beta\\,\\phi^\{\\prime\}\(\\Delta\_\{\\mathrm\{hard\}\}\(Y\)\)\\bigl\(u\_\{i\_\{w\}^\{\\star\}\(Y\)\}\(Y\)\-v\_\{i\_\{l\}^\{\\star\}\(Y\)\}\(Y\)\\bigr\),\(7\)gsm​\(Y\)\\displaystyle g\_\{\\mathrm\{sm\}\}\(Y\):=β​ϕ′​\(Δsm​\(Y\)\)​\(∑i=1nqw,i​\(Y\)​ui​\(Y\)−∑i=1nql,i​\(Y\)​vi​\(Y\)\)\.\\displaystyle:=\\beta\\,\\phi^\{\\prime\}\(\\Delta\_\{\\mathrm\{sm\}\}\(Y\)\)\\left\(\\sum\_\{i=1\}^\{n\}q\_\{w,i\}\(Y\)u\_\{i\}\(Y\)\-\\sum\_\{i=1\}^\{n\}q\_\{l,i\}\(Y\)v\_\{i\}\(Y\)\\right\)\.\(8\)
We also use

u¯​\(Y\):=1n​∑i=1nui​\(Y\),v¯​\(Y\):=1n​∑i=1nvi​\(Y\),\\bar\{u\}\(Y\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}u\_\{i\}\(Y\),\\qquad\\bar\{v\}\(Y\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}v\_\{i\}\(Y\),and the corresponding averaged smoothed gradient

g¯​\(Y\):=β​ϕ′​\(Δsm​\(Y\)\)​\(u¯​\(Y\)−v¯​\(Y\)\)\.\\bar\{g\}\(Y\):=\\beta\\,\\phi^\{\\prime\}\(\\Delta\_\{\\mathrm\{sm\}\}\(Y\)\)\\bigl\(\\bar\{u\}\(Y\)\-\\bar\{v\}\(Y\)\\bigr\)\.
For any square\-integrable vector random variableXX, we use

Var⁡\(X\):=𝔼​\[‖X−𝔼​X‖2\]\.\\operatorname\{Var\}\(X\):=\\mathbb\{E\}\\\!\\left\[\\\|X\-\\mathbb\{E\}X\\\|^\{2\}\\right\]\.For probability vectorsp,q∈ℝnp,q\\in\\mathbb\{R\}^\{n\}, we use the total variation distance

TV⁡\(p,q\):=12​∑i=1n\|pi−qi\|=1−∑i=1nmin⁡\{pi,qi\}\.\\operatorname\{TV\}\(p,q\):=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\|p\_\{i\}\-q\_\{i\}\|=1\-\\sum\_\{i=1\}^\{n\}\\min\\\{p\_\{i\},q\_\{i\}\\\}\.We write

Unifn:=\(1n,…,1n\)\\mathrm\{Unif\}\_\{n\}:=\\left\(\\frac\{1\}\{n\},\\dots,\\frac\{1\}\{n\}\\right\)for the uniform distribution onnnpoints\.

For the hard lower bound, define for each pair indexii

pi:=ℙ​\(I​\(Y\)=i\),μi:=𝔼​\[ghard​\(Y\)∣I​\(Y\)=i\]\.p\_\{i\}:=\\mathbb\{P\}\(I\(Y\)=i\),\\qquad\\mu\_\{i\}:=\\mathbb\{E\}\[g\_\{\\mathrm\{hard\}\}\(Y\)\\mid I\(Y\)=i\]\.We also use the two\-index specialization: there exista≠ba\\neq band constantsp∈\(0,1/2\]p\\in\(0,1/2\],m\>0m\>0such that

ℙ​\(I​\(Y\)=a\)≥p,ℙ​\(I​\(Y\)=b\)≥p,‖μa−μb‖≥m\.\\mathbb\{P\}\(I\(Y\)=a\)\\geq p,\\qquad\\mathbb\{P\}\(I\(Y\)=b\)\\geq p,\\qquad\\\|\\mu\_\{a\}\-\\mu\_\{b\}\\\|\\geq m\.
Finally, define

δn​\(τ,R\):=\(n−1\)​\(eτ​R−1\)n​\(eτ​R\+n−1\)\.\\delta\_\{n\}\(\\tau,R\):=\\frac\{\(n\-1\)\(e^\{\\tau R\}\-1\)\}\{n\(e^\{\\tau R\}\+n\-1\)\}\.

#### Assumptions for the Stability Comparison\.

###### Assumption 1\(Bounded branch scores and score gradients\)\.

There exist constantsG\>0G\>0andRw,Rl≥0R\_\{w\},R\_\{l\}\\geq 0such that

‖ui​\(Y\)‖≤G,‖vi​\(Y\)‖≤G\\\|u\_\{i\}\(Y\)\\\|\\leq G,\\qquad\\\|v\_\{i\}\(Y\)\\\|\\leq Gfor every candidate indexiiand every sampled poolYY, and

maxi⁡siw​\(Y\)−mini⁡siw​\(Y\)≤Rw,maxi⁡sil​\(Y\)−mini⁡sil​\(Y\)≤Rl\\max\_\{i\}s\_\{i\}^\{w\}\(Y\)\-\\min\_\{i\}s\_\{i\}^\{w\}\(Y\)\\leq R\_\{w\},\\qquad\\max\_\{i\}s\_\{i\}^\{l\}\(Y\)\-\\min\_\{i\}s\_\{i\}^\{l\}\(Y\)\\leq R\_\{l\}for every sampled poolYY\.

###### Assumption 2\(Bounded within\-pool gradient diameters\)\.

There existDw,Dl≥0D\_\{w\},D\_\{l\}\\geq 0such that

maxi,j⁡‖ui​\(Y\)−uj​\(Y\)‖≤Dw,maxi,j⁡‖vi​\(Y\)−vj​\(Y\)‖≤Dl\\max\_\{i,j\}\\\|u\_\{i\}\(Y\)\-u\_\{j\}\(Y\)\\\|\\leq D\_\{w\},\\qquad\\max\_\{i,j\}\\\|v\_\{i\}\(Y\)\-v\_\{j\}\(Y\)\\\|\\leq D\_\{l\}for every sampled poolYY\.

###### Assumption 3\(Selector\-mixture dominance\)\.

The selector\-mixture variance floor of hard selection dominates the fluctuation budget of smoothing:

p2​m2\>2​Var⁡\(g¯​\(Y\)\)\+2​β2​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)2\.p^\{2\}m^\{2\}\>2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\+2\\beta^\{2\}\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)^\{2\}\.

Assumption[3](https://arxiv.org/html/2607.16431#Thmassumption3)is the explicit comparison condition used in the main theorem\. Its left\-hand side is the irreducible selector\-mixture variance floor induced by hard selection, while the right\-hand side measures the total fluctuation budget of smoothing: the branchwise averaging baselineVar⁡\(g¯​\(Y\)\)\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)plus the additional soft\-weighting term controlled by the diametersDw,DlD\_\{w\},D\_\{l\}and the shrinkage factorδn​\(τ,R\)\\delta\_\{n\}\(\\tau,R\)\. Intuitively, the condition is most plausible when hard selector competition is strong but the softmax weights remain close to a stable branchwise average\.

### A\.3Auxiliary Lemmas for Gradient Stability

###### Lemma 3\(Hard selector induces a variance floor\)\.

Under the notation above,

Vhard​\(x,θ\)≥12​∑i,jpi​pj​‖μi−μj‖2\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\\geq\\frac\{1\}\{2\}\\sum\_\{i,j\}p\_\{i\}p\_\{j\}\\\|\\mu\_\{i\}\-\\mu\_\{j\}\\\|^\{2\}\.In particular, if there exista≠ba\\neq bandp∈\(0,1/2\]p\\in\(0,1/2\]such that

ℙ​\(I​\(Y\)=a\)≥p,ℙ​\(I​\(Y\)=b\)≥p,‖μa−μb‖≥m,\\mathbb\{P\}\(I\(Y\)=a\)\\geq p,\\qquad\\mathbb\{P\}\(I\(Y\)=b\)\\geq p,\\qquad\\\|\\mu\_\{a\}\-\\mu\_\{b\}\\\|\\geq m,then

Vhard​\(x,θ\)≥p2​m2\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\\geq p^\{2\}m^\{2\}\.

###### Proof\.

Let

Z​\(Y\):=ghard​\(Y\),Z\(Y\):=g\_\{\\mathrm\{hard\}\}\(Y\),Applying the law of total variance with respect to the hard\-selected pair index

Var⁡\(Z\)=𝔼​\[Var⁡\(Z∣I​\(Y\)\)\]\+Var⁡\(𝔼​\[Z∣I​\(Y\)\]\),\\operatorname\{Var\}\(Z\)=\\mathbb\{E\}\\\!\\bigl\[\\operatorname\{Var\}\(Z\\mid I\(Y\)\)\\bigr\]\+\\operatorname\{Var\}\\\!\\bigl\(\\mathbb\{E\}\[Z\\mid I\(Y\)\]\\bigr\),and therefore

Vhard​\(x,θ\)=Var⁡\(Z\)≥Var⁡\(𝔼​\[Z∣I​\(Y\)\]\)\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)=\\operatorname\{Var\}\(Z\)\\geq\\operatorname\{Var\}\\\!\\bigl\(\\mathbb\{E\}\[Z\\mid I\(Y\)\]\\bigr\)\.Write

μj:=𝔼​\[Z∣I​\(Y\)=j\],pj:=ℙ​\(I​\(Y\)=j\)\.\\mu\_\{j\}:=\\mathbb\{E\}\[Z\\mid I\(Y\)=j\],\\qquad p\_\{j\}:=\\mathbb\{P\}\(I\(Y\)=j\)\.Then

Var⁡\(𝔼​\[Z∣I​\(Y\)\]\)=∑jpj​‖μj−∑kpk​μk‖2=12​∑i,jpi​pj​‖μi−μj‖2\.\\operatorname\{Var\}\\\!\\bigl\(\\mathbb\{E\}\[Z\\mid I\(Y\)\]\\bigr\)=\\sum\_\{j\}p\_\{j\}\\left\\\|\\mu\_\{j\}\-\\sum\_\{k\}p\_\{k\}\\mu\_\{k\}\\right\\\|^\{2\}=\\frac\{1\}\{2\}\\sum\_\{i,j\}p\_\{i\}p\_\{j\}\\\|\\mu\_\{i\}\-\\mu\_\{j\}\\\|^\{2\}\.This proves the first claim\.

For the second claim, suppose there exist two indicesa≠ba\\neq bsuch thatpa≥pp\_\{a\}\\geq p,pb≥pp\_\{b\}\\geq p, and

‖μa−μb‖≥m\.\\\|\\mu\_\{a\}\-\\mu\_\{b\}\\\|\\geq m\.Keeping only the\(a,b\)\(a,b\)contribution in the pairwise\-variance formula, we obtain

Var⁡\(𝔼​\[Z∣I​\(Y\)\]\)≥pa​pb​‖μa−μb‖2≥p2​m2\.\\operatorname\{Var\}\\\!\\bigl\(\\mathbb\{E\}\[Z\\mid I\(Y\)\]\\bigr\)\\geq p\_\{a\}p\_\{b\}\\\|\\mu\_\{a\}\-\\mu\_\{b\}\\\|^\{2\}\\geq p^\{2\}m^\{2\}\.Hence

Vhard​\(x,θ\)≥p2​m2\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\\geq p^\{2\}m^\{2\}\.∎

###### Lemma 4\(Refined upper bound for smoothed variance\)\.

Under Assumptions[1](https://arxiv.org/html/2607.16431#Thmassumption1)and[2](https://arxiv.org/html/2607.16431#Thmassumption2),

‖gsm​\(Y\)−g¯​\(Y\)‖≤β​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)\\\|g\_\{\\mathrm\{sm\}\}\(Y\)\-\\bar\{g\}\(Y\)\\\|\\leq\\beta\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)for every sampled poolYY\. Consequently,

Vsm​\(x,θ\)≤2​Var⁡\(g¯​\(Y\)\)\+2​β2​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)2\.V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\\leq 2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\+2\\beta^\{2\}\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)^\{2\}\.

###### Proof\.

Write

Ξ​\(Y\):=gsm​\(Y\)−g¯​\(Y\)\.\\Xi\(Y\):=g\_\{\\mathrm\{sm\}\}\(Y\)\-\\bar\{g\}\(Y\)\.By \([8](https://arxiv.org/html/2607.16431#A1.E8)\),

Ξ​\(Y\)=β​ϕ′​\(Δsm​\(Y\)\)​\[\(∑i=1nqw,i​\(Y\)​ui​\(Y\)−u¯​\(Y\)\)−\(∑i=1nql,i​\(Y\)​vi​\(Y\)−v¯​\(Y\)\)\]\.\\Xi\(Y\)=\\beta\\,\\phi^\{\\prime\}\(\\Delta\_\{\\mathrm\{sm\}\}\(Y\)\)\\left\[\\left\(\\sum\_\{i=1\}^\{n\}q\_\{w,i\}\(Y\)u\_\{i\}\(Y\)\-\\bar\{u\}\(Y\)\\right\)\-\\left\(\\sum\_\{i=1\}^\{n\}q\_\{l,i\}\(Y\)v\_\{i\}\(Y\)\-\\bar\{v\}\(Y\)\\right\)\\right\]\.Sinceϕ​\(u\)=−log⁡σ​\(u\)\\phi\(u\)=\-\\log\\sigma\(u\), we have\|ϕ′​\(u\)\|≤1\|\\phi^\{\\prime\}\(u\)\|\\leq 1\.

For the preferred branch, Assumption[1](https://arxiv.org/html/2607.16431#Thmassumption1)implies

maxi⁡e−τ​siw​\(Y\)mini⁡e−τ​siw​\(Y\)≤eτ​Rw\.\\frac\{\\max\_\{i\}e^\{\-\\tau s\_\{i\}^\{w\}\(Y\)\}\}\{\\min\_\{i\}e^\{\-\\tau s\_\{i\}^\{w\}\(Y\)\}\}\\leq e^\{\\tau R\_\{w\}\}\.Hence every soft weight satisfies

1eτ​Rw\+n−1≤qw,i​\(Y\)≤eτ​Rweτ​Rw\+n−1\.\\frac\{1\}\{e^\{\\tau R\_\{w\}\}\+n\-1\}\\leq q\_\{w,i\}\(Y\)\\leq\\frac\{e^\{\\tau R\_\{w\}\}\}\{e^\{\\tau R\_\{w\}\}\+n\-1\}\.Sincemaxi⁡qw,i​\(Y\)≥1/n\\max\_\{i\}q\_\{w,i\}\(Y\)\\geq 1/n, we have

TV⁡\(qw​\(Y\),Unifn\)\\displaystyle\\operatorname\{TV\}\(q\_\{w\}\(Y\),\\mathrm\{Unif\}\_\{n\}\)=1−∑i=1nmin⁡\{qw,i​\(Y\),1/n\}\\displaystyle=1\-\\sum\_\{i=1\}^\{n\}\\min\\\{q\_\{w,i\}\(Y\),1/n\\\}≤1−1n−n−1eτ​Rw\+n−1=δn​\(τ,Rw\)\.\\displaystyle\\leq 1\-\\frac\{1\}\{n\}\-\\frac\{n\-1\}\{e^\{\\tau R\_\{w\}\}\+n\-1\}=\\delta\_\{n\}\(\\tau,R\_\{w\}\)\.\(9\)Now letzzbe any unit vector in the dual norm\. The scalar function

fz​\(i\):=⟨z,ui​\(Y\)⟩f\_\{z\}\(i\):=\\langle z,u\_\{i\}\(Y\)\\ranglehas oscillation at mostDwD\_\{w\}by Assumption[2](https://arxiv.org/html/2607.16431#Thmassumption2)\. Therefore, by the standard total\-variation bound for bounded observables,

\|∑i=1nqw,i​\(Y\)​⟨z,ui​\(Y\)⟩−1n​∑i=1n⟨z,ui​\(Y\)⟩\|≤Dw​TV⁡\(qw​\(Y\),Unifn\)\.\\left\|\\sum\_\{i=1\}^\{n\}q\_\{w,i\}\(Y\)\\langle z,u\_\{i\}\(Y\)\\rangle\-\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\langle z,u\_\{i\}\(Y\)\\rangle\\right\|\\leq D\_\{w\}\\,\\operatorname\{TV\}\(q\_\{w\}\(Y\),\\mathrm\{Unif\}\_\{n\}\)\.Taking the supremum over dual unit vectorszzand using \([9](https://arxiv.org/html/2607.16431#A1.E9)\) gives

‖∑i=1nqw,i​\(Y\)​ui​\(Y\)−u¯​\(Y\)‖≤Dw​δn​\(τ,Rw\)\.\\left\\\|\\sum\_\{i=1\}^\{n\}q\_\{w,i\}\(Y\)u\_\{i\}\(Y\)\-\\bar\{u\}\(Y\)\\right\\\|\\leq D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\.
The non\-preferred branch is identical\. Assumption[1](https://arxiv.org/html/2607.16431#Thmassumption1)implies

TV⁡\(ql​\(Y\),Unifn\)≤δn​\(τ,Rl\),\\operatorname\{TV\}\(q\_\{l\}\(Y\),\\mathrm\{Unif\}\_\{n\}\)\\leq\\delta\_\{n\}\(\\tau,R\_\{l\}\),and Assumption[2](https://arxiv.org/html/2607.16431#Thmassumption2)yields

‖∑i=1nql,i​\(Y\)​vi​\(Y\)−v¯​\(Y\)‖≤Dl​δn​\(τ,Rl\)\.\\left\\\|\\sum\_\{i=1\}^\{n\}q\_\{l,i\}\(Y\)v\_\{i\}\(Y\)\-\\bar\{v\}\(Y\)\\right\\\|\\leq D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\.
Combining \([A\.3](https://arxiv.org/html/2607.16431#A1.Ex63)\) and \([A\.3](https://arxiv.org/html/2607.16431#A1.Ex65)\), we obtain

‖Ξ​\(Y\)‖≤β​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)\.\\\|\\Xi\(Y\)\\\|\\leq\\beta\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)\.This proves the first claim\.

For the variance bound, write

gsm​\(Y\)−𝔼​gsm​\(Y\)=\(g¯​\(Y\)−𝔼​g¯​\(Y\)\)\+\(Ξ​\(Y\)−𝔼​Ξ​\(Y\)\)\.g\_\{\\mathrm\{sm\}\}\(Y\)\-\\mathbb\{E\}g\_\{\\mathrm\{sm\}\}\(Y\)=\\bigl\(\\bar\{g\}\(Y\)\-\\mathbb\{E\}\\bar\{g\}\(Y\)\\bigr\)\+\\bigl\(\\Xi\(Y\)\-\\mathbb\{E\}\\Xi\(Y\)\\bigr\)\.Using‖a\+b‖2≤2​‖a‖2\+2​‖b‖2\\\|a\+b\\\|^\{2\}\\leq 2\\\|a\\\|^\{2\}\+2\\\|b\\\|^\{2\}and

Var⁡\(Ξ\)≤𝔼​‖Ξ‖2,\\operatorname\{Var\}\(\\Xi\)\\leq\\mathbb\{E\}\\\|\\Xi\\\|^\{2\},we get

Vsm​\(x,θ\)≤2​Var⁡\(g¯​\(Y\)\)\+2​𝔼​‖Ξ​\(Y\)‖2\.V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\\leq 2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\+2\\mathbb\{E\}\\\|\\Xi\(Y\)\\\|^\{2\}\.Applying the pointwise bound on‖Ξ​\(Y\)‖\\\|\\Xi\(Y\)\\\|yields

Vsm​\(x,θ\)≤2​Var⁡\(g¯​\(Y\)\)\+2​β2​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)2\.V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\\leq 2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\+2\\beta^\{2\}\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)^\{2\}\.∎

### A\.4Proof of Theorem[2](https://arxiv.org/html/2607.16431#Thmtheorem2)

###### Proof\.

Lemma[3](https://arxiv.org/html/2607.16431#Thmlemma3)gives

Vhard​\(x,θ\)≥p2​m2\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\\geq p^\{2\}m^\{2\}\.Lemma[4](https://arxiv.org/html/2607.16431#Thmlemma4)gives

Vsm​\(x,θ\)≤2​Var⁡\(g¯​\(Y\)\)\+2​β2​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)2\.V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\\leq 2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\+2\\beta^\{2\}\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)^\{2\}\.Subtracting the two bounds yields

Vhard​\(x,θ\)−Vsm​\(x,θ\)≥p2​m2−2​Var⁡\(g¯​\(Y\)\)−2​β2​\(Dw​δn​\(τ,Rw\)\+Dl​δn​\(τ,Rl\)\)2\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\-V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\\geq p^\{2\}m^\{2\}\-2\\operatorname\{Var\}\(\\bar\{g\}\(Y\)\)\-2\\beta^\{2\}\\Big\(D\_\{w\}\\,\\delta\_\{n\}\(\\tau,R\_\{w\}\)\+D\_\{l\}\\,\\delta\_\{n\}\(\\tau,R\_\{l\}\)\\Big\)^\{2\}\.Assumption[3](https://arxiv.org/html/2607.16431#Thmassumption3)therefore implies

Vhard​\(x,θ\)\>Vsm​\(x,θ\)\.V\_\{\\mathrm\{hard\}\}\(x,\\theta\)\>V\_\{\\mathrm\{sm\}\}\(x,\\theta\)\.∎

## Appendix BData Aggregation Strategies

Table[3](https://arxiv.org/html/2607.16431#A2.T3)summarizes the different aggregation strategies used to aggregate multi\-pair preference data for preference optimization\. We present 4 strategies and corresponding formulation to generate pseudo preferred datay~w\\tilde\{y\}\_\{w\}and pseudo non\-preferred datay~l\\tilde\{y\}\_\{l\}\.

Table 3:Different aggregation strategies for multi\-pair preference data to formulatey~w\\tilde\{y\}\_\{w\}andy~l\\tilde\{y\}\_\{l\}\.
## Appendix CBaseline

- •CoT\(Weiet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib18)\)instructs the LLM to generate answers accompanied by explicit reasoning steps\.
- •ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib19)\)integrates reasoning, action, and observation steps, concluding the generation process upon reaching a terminal state\. The action step involves either generating a query to retrieve additional knowledge or finalizing the generation, while the observation step incorporates the retrieved knowledge documents\.
- •SelfAsk\(Presset al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib20)\)involves generating follow\-up questions, retrieving relevant information, and answering these follow\-up questions\. Each retrieval operation is based on the generated follow\-up questions\. When no additional follow\-up questions are produced, the LLM provides the answer to the original question\. We prepend the newly retrieved knowledge to the original question following the approach of Yoran et al\. \(2023\)\.
- •BlendFilter\(Wanget al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib21)\)combines internal and external knowledge to enhance retrieval quality\. Additionally, it employs the LLM to filter out irrelevant information, thereby preventing the model from being misled\.
- •InstructRAG\(Weiet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib22)\)instructs the LLM to generate rationales that analyze the relevance between the query and the retrieved knowledge\.
- •RoseRAG\(Liuet al\.,[2025](https://arxiv.org/html/2607.16431#bib.bib12)\)generates preference data through rejection sampling and optimizes the model using a single hard preference pair selected based on the model’s likelihoods\.

## Appendix DImplementation Details

We evaluate models with two small\-sclae LLMs: Qwen2\.5\-1\.5B\-Instruct\(Team,[2024](https://arxiv.org/html/2607.16431#bib.bib23)\)and Gemma\-2\-2b\-it\(Teamet al\.,[2024](https://arxiv.org/html/2607.16431#bib.bib24)\)\. We employ the state\-of\-the\-art efficient retrieval method ColBERT v2\(Santhanamet al\.,[2022](https://arxiv.org/html/2607.16431#bib.bib25)\)as implemented by\(Khattabet al\.,[2023b](https://arxiv.org/html/2607.16431#bib.bib27)\), which applies quantization to accelerate approximate nearest neighbor search\. Our knowledge base comprises Wikipedia abstracts from the 2017 dump\(Khattabet al\.,[2023a](https://arxiv.org/html/2607.16431#bib.bib26)\)\. For experiments, we use Transformers\(Wolfet al\.,[2020](https://arxiv.org/html/2607.16431#bib.bib30)\)and TRL\(von Werraet al\.,[2020](https://arxiv.org/html/2607.16431#bib.bib31)\)as backbone frameworks for fine\-tuning and vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib32)\)for inference\. For multi\-pair aggregation, we implemented the trainer based on TRL library to apply ourRimsstrategy and comparisons to other data aggregation strategies for three types of preference optimization algorithms \(DPO, CPO, ORPO\)\.

In all experiments, we adopt a 3\-shot in\-context learning setting following the approach of\(Shaoet al\.,[2023](https://arxiv.org/html/2607.16431#bib.bib28)\), with the count of retrieval documents set to 5 for all methods\. The prompts for generating preference data are shown in Figure[5](https://arxiv.org/html/2607.16431#A6.F5), while those for question answering are presented in Figure[6](https://arxiv.org/html/2607.16431#A6.F6)\. We generate preference data using the first 10k training samples from HPQA, 2WMHQA and MSQQA while all data in MHRAGQA\. For all backbones, we apply LoRA\(Huet al\.,[2021](https://arxiv.org/html/2607.16431#bib.bib29)\)with a rank of 16 andlora\_alpha= 32, targeting ”all\-linearmodules\. Experiments are conducted on eight 80G A100 NVIDIA GPUs\. The learning rate is1​e−51e^\{\-5\}and number of training epochs is 2\.

## Appendix EQualitative Comparison of RoseRAG andRims

Figure[4](https://arxiv.org/html/2607.16431#A5.F4)presents a case study comparing the inference outputs of models trained with RoseRAG andRimson a multi\-hop question with noisy retrieved passages\. The RoseRAG\-trained model, which was optimized using only a single hardest preference pair per query, fails to reason correctly through the noisy context and produces an incorrect answer\. In contrast, theRims\-trained model, which leveraged all preference pairs via smoothed aggregation during training, demonstrates stronger robustness to noisy distractors and arrives at the correct answer\. This case illustrates that training with smoothed multi\-pair aggregation yields models with improved evidence discrimination and multi\-hop reasoning ability under noisy retrieval conditions\.

![Refer to caption](https://arxiv.org/html/2607.16431v1/x6.png)Figure 4:Qualitative comparison on a multi\-hop question\. RoseRAG, which relies on hard pair selection, is misled by noisy retrieved passages and produces an incorrect answer\.Rims, using soft aggregation over multiple preference pairs, arrives at the correct answer despite the same noisy context\.
## Appendix FPrompt Details

We show the prompts we used in Fig\.[5](https://arxiv.org/html/2607.16431#A6.F5)and Fig\.[6](https://arxiv.org/html/2607.16431#A6.F6)\.

Rationale GenerationSystem Prompt:You are a useful assistant\.I will provide one question, several pieces of knowledge \(which may be related or unrelated to the question\), and the answer to the question\. Please explain your reasoning process in a single paragraph consisting of no more than four sentences\. If the provided knowledge is insufficient, you may make an informed guess, but do not respond with “Unknown\.”User Prompt:Knowledge:𝒦q\\mathcal\{K\}\_\{q\}Question:qqAnswer:a∗a^\{\*\}Assistant Prompt:Let’s think step by step\.Output:\{rationale​r\}\\\{\\text\{rationale \}r\\\}Figure 5:Prompt template for rationale generationFew\-shot Prompt for Question Answering BenchmarkSystem Prompt:Y ou are a useful assistant\. Y ou should answer the following question quickly and accurately based on given knowledge\. Provide your reasoning in few sentences and answer in one or few words\. Please begin your final answer with ’So the answer is’\.User Prompt:Knowledge:Mr\. Big \(film\) — Mr\. Big is a 2007 documentary directed and produced by Tiffany Burns and edited by Alec MacNeill Richardson\. The documentary examines the ”Mr\. Big” undercover methods used by the Royal Canadian Mounted Police \(RCMP\)\. In these operations, RCMP officers pose as gang criminals and develop a relationship with the target in the hope of eventually determining what, if any, knowledge the target has of the crime being investigated\. ”Mr\. Big” operations have been credited with securing difficult convictions in a large number of cases, but concerns have been raised that they involve a risk of false confessions and wrongful convictions\. It Might Get Loud — It Might Get Loud is a 2008 American documentary film by filmmaker Davis Guggenheim\. It explores the careers and styles of prominent rock musicians Jimmy Page, The Edge, and Jack White\. The film received a wide release on August 14, 2009 in the U\.S\. by Sony Pictures Classics\.Question:Are It Might Get Loud and Mr\. Big both Canadian documentaries?Assistant Prompt:Let’s think step by step\. Mr\. Big is a 2007 documentary which examines the ”Mr\. Big” undercover methods used by the Royal Canadian Mounted Police\. However, Are It Might Get Loud is a 2008 American documentary film\. So the answer is no\.User Prompt:Knowledge:Leslie H\. Martinson — Leslie Herbert ”Les” Martinson \(January 16, 1915 – September 3, 2016\) was an American television and film director\. László Benedek — László Benedek \(\] ; March 5, 1905 – March 11, 1992; sometimes ”Laslo Benedek”\) was a Hungarian\-born film director and cinematographer, most notable for directing ”The Wild One” \(1953\)\.Question:Were László Benedek and Leslie H\. Martinson both film directors?Assistant Prompt:Let’s think step by step\. László Benedek was a Hungarian\-born film director and Leslie H\. Martinson was an American film director\. So the answer is yes\.User Prompt:Knowledge:Lucium — Lucium was the proposed name for an alleged new element found by chemist Prosper Barrière in 1896 in the mineral monazite\. Later, William Crookes confirmed that the new element was actually an impure sample of yttrium\. William Crookes — Sir William Crookes \( ; 17 June 1832 – 4 April 1919\) was an English chemist and physicist who attended the Royal College of Chemistry in London, and worked on spectroscopy\. He was a pioneer of vacuum tubes, inventing the Crookes tube which was made in 1875\. Crookes was the inventor of the Crookes radiometer, which today is made and sold as a novelty item\. Late in life, he became interested in spiritualism, and became the president of the Society for Psychical Research\.Question:Lucium was confimed to be an impure sample of yttrium by an English chemist who became the president of what?Assistant Prompt:Let’s think step by step\. Lucium was confimed to be an impure sample of yttrium by William Crookes\. William Crookes is Sir William Crookes\. Sir William Crookes became the president of the Society for Psychical Research\. So the answer is Society for Psychical Research\.User Prompt:Knowledge:𝒦q\\mathcal\{K\}\_\{q\}Question:qqAssistant Prompt:Let’s think step by step\. \{answeraa\}Figure 6:Few\-shot prompt template for benchmark