IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering

# Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering
Source: [https://arxiv.org/html/2608.13588](https://arxiv.org/html/2608.13588)
Jungmin Yun1and Youngbin Kim1, 2

1Department of Artificial Intelligence, Chung\-Ang University 2Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung\-Ang University \{cocoro357, ybkim85\}@cau\.ac\.kr

###### Abstract

Multi\-hop question answering requires complex reasoning across multiple evidence segments, which often overwhelms retrieval\-augmented generation systems with lengthy and noisy contexts, thereby undermining both efficiency and accuracy\. While existing prompt compression methods attempt to address this issue, they are typically designed for single\-turn queries and fail to capture interdependent reasoning steps\. We propose IterCOMP, a unified, training\-free prompt compression framework that incorporates multi\-hop reasoning within an iterative compression loop\. IterCOMP decomposes documents into evidence segments, evaluates question answerability, and generates targeted follow\-up questions to iteratively integrate essential evidence, producing a compact, reasoning\-oriented prompt\. Experiments on MusiQue, 2WikiMultiHopQA, and HotpotQA demonstrate that IterCOMP achieves substantial improvements in Exact Match and F1 scores while reducing the token budget, outperforming existing baselines and exhibiting robustness as reasoning complexity increases\.

IterCOMP: Reasoning\-aware Adaptive Prompt Compression for Multi\-hop Question Answering

Jungmin Yun1andYoungbin Kim1, 21Department of Artificial Intelligence, Chung\-Ang University2Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung\-Ang University\{cocoro357, ybkim85\}@cau\.ac\.kr

## 1Introduction

Multi\-hop question answering \(QA\) requires reasoning over multiple pieces of evidence to derive the correct answer\. Large Language Models \(LLMs\) have significantly advanced QA performance by enhancing complex query understanding and information integration\. Nevertheless, their reliance on static pre\-training data imposes inherent limitations on knowledge coverage\. Retrieval\-Augmented Generation \(RAG\)Lewis et al\. \([2020](https://arxiv.org/html/2608.13588#bib.bib24)\)mitigates this issue by incorporating dynamic, up\-to\-date information from external sources, thereby enabling LLMs to generate more diverse, accurate, and contextually grounded responsesGao et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib11)\)\.

![Refer to caption](https://arxiv.org/html/2608.13588v1/figure1_1.jpg)Figure 1:Example of multi\-hop reasoning, where answering the question requires integrating evidence from multiple documents through implicit intermediate clues absent from the initial question\.However, RAG systems face challenges in both efficiency and effectiveness due to the long input sequences induced by retrieved documents\. Efficiency declines as input length scales linearly with the number of retrieved documents, leading to higher inference latency and increased computational overheadLi et al\. \([2025a](https://arxiv.org/html/2608.13588#bib.bib25)\); Xu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\)\. For API\-based commercial LLMs, longer inputs also result in elevated operational costsChoi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\)\. Effectiveness is similarly constrained, as lengthy inputs often include irrelevant content that distracts the model and hinders accurate reasoningShi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib35)\)\. Furthermore, LLMs exhibit positional biases, such as the lost\-in\-the\-middle phenomenonLiu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib29)\)\. These challenges are especially pronounced in multi\-hop QA, which requires linking multiple pieces of evidence through sequential and interdependent reasoning steps to derive the final answerTang and Yang \([2024](https://arxiv.org/html/2608.13588#bib.bib37)\)\.

Recently, various prompt compression techniques have been proposed to reduce contextual overhead by eliminating less salient content or condensing information into compact representationsPan et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib31)\); Jiang et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib19)\); Mu et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib30)\)\. However, effective compression requires more than simply reducing input length; it fundamentally depends on selectively retaining query\-relevant information query\-relevant while simultaneously removing irrelevant or distracting contentCao et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib4)\)\.

In this context, query\-focused compression methods enhance contextual relevance by condensing prompts to retain content pertinent to a given queryChoi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\); Liskavets et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib28)\); Hwang et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib16)\)\. However, their predominant reliance on a single\-query paradigm, which primarily leverages surface\-level query\-document relevance, limits their effectiveness in complex scenarios such as multi\-hop QATrivedi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib40)\); Press et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib32)\); Shao et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib34)\)\. As illustrated in Figure[1](https://arxiv.org/html/2608.13588#S1.F1), these tasks require sequential, interdependent reasoning across multiple documents and often depend on implicit intermediate clues absent from the initial querySchnitzler et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib33)\); Geva et al\. \([2021](https://arxiv.org/html/2608.13588#bib.bib13)\); Trivedi et al\. \([2022](https://arxiv.org/html/2608.13588#bib.bib39)\); Ho et al\. \([2020b](https://arxiv.org/html/2608.13588#bib.bib15)\)\. These limitations become more pronounced when queries consist of multiple subcomponents, whose supporting evidence is dispersed across different documentsLevy et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib23)\)\. Consequently, single\-query approaches often fail to capture inter\-document dependencies or to reason effectively over linked information, leading to information loss and incomplete contextual understanding that is inadequate for synthesizing multi\-source answers\.

To this end, we propose IterCOMP, a unified prompt compression framework based on iterative refinement that explicitly incorporates multi\-hop reasoning into the compression loop\. IterCOMP strategically integrates the reasoning capabilities of LLMs into the compression process to directly address the nuanced evidential demands of complex queries\. Specifically, the framework filters relevant evidence segments and employs an LLM to assess whether they are sufficient to answer the question\. When the evidence is insufficient, the LLM identifies the informational gap and formulates a targeted follow\-up question to bridge it\. This initiates a cycle in which new evidence is evaluated against the evolving reasoning path, with only critical segments retained and irrelevant content discarded\. Through progressive accumulation and distillation, IterCOMP incrementally constructs a concise yet comprehensive prompt that enables the original multi\-hop question to be answered using focused, essential information\. Extensive experiments demonstrate that IterCOMP substantially improves both QA performance and efficiency, highlighting the effectiveness of integrating deep reasoning into prompt compression\.

## 2Related Work

### 2\.1Prompt Compression

#### 2\.1\.1Soft Prompt Compression

Soft prompt compression encodes the original prompt into continuous vector representationsGe et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib12)\); Cheng et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib6)\)\. AutoCompressorChevalier et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib7)\)segments long prompts into multiple parts, compresses each segment into a soft prompt representation, and concatenates them to form the final prompt\. GISTMu et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib30)\)generates prefix\-style soft prompts for instructional inputs\. These methods enable parameter\-efficient adaptation of pretrained models while preserving the high\-level semantics of the original input\. However, soft prompts are typically optimized for specific LLMs, which limits their transferability across different modelsLi et al\. \([2025b](https://arxiv.org/html/2608.13588#bib.bib27)\)\. This constraint poses a significant challenge in API\-based environments, where direct access to model internals is restricted\. Consequently, any update to the underlying LLM requires retraining the soft prompts, reducing their practicality in dynamic or cross\-model deployment scenarios\.

#### 2\.1\.2Hard Prompt Compression

Hard prompt compression reduces prompt length by preserving essential information through extractive or abstractive methodsLi et al\. \([2025b](https://arxiv.org/html/2608.13588#bib.bib27)\); Chuang et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib9)\); Jung and Kim \([2024](https://arxiv.org/html/2608.13588#bib.bib21)\)\. This approach improves computational efficiency and enhances the quality of generated outputs by eliminating redundant or uninformative content\.

Query\-Agnostic Compression\.Query\-agnostic methods operate independently of the query, leveraging the statistical or structural properties of the prompt\. Selective\-ContextLi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib26)\)discards low\-information tokens based on self\-information metrics\. LLMLinguaJiang et al\. \([2023b](https://arxiv.org/html/2608.13588#bib.bib18)\)removes tokens with low perplexity, while LLMLingua\-2Pan et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib31)\)formulates compression as a token classification task using knowledge distillation\. Despite their broad applicability, these approaches overlook query\-specific context\. As a result, compressed prompts may retain irrelevant content or omit critical information, thereby degrading retrieval effectiveness and response relevanceCao et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib4)\)\.

Query\-Aware Compression\.Incorporating query information into the compression process is essential for retaining relevant and critical content for reasoning\. LongLLMLinguaJiang et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib19)\)employs a coarse\-to\-fine strategy, first estimating document\-level importance via query\-conditioned perplexity, followed by refining token selection\. COMPACTYoon et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib48)\)compresses context by jointly analyzing previously selected content and newly introduced segments\. RECOMPXu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\)performs sentence\-level compression by measuring the similarity between query and sentence embeddings to identify key content\. R2CChoi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\)encodes the query alongside each context chunk, enabling the decoder to dynamically preserve relevant information\.

Despite their advantages, existing query\-aware methods typically assess relevance by matching a single query against individual documents or sentences\. This one\-to\-one paradigm is insufficient for complex multi\-hop QA, where a query often comprises multiple subcomponents that must be jointly resolved using evidence distributed across documentsTang and Yang \([2024](https://arxiv.org/html/2608.13588#bib.bib37)\)\. Such tasks require capturing inter\-document dependencies and reasoning over linked information segmentsTrivedi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib40)\)\. However, current methods struggle to model these multi\-hop relationships, thereby limiting their effectiveness in synthesizing information from multiple sourcesZhu et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib50)\)\.

### 2\.2Self\-Ask Mechanisms in LLMs

Recent studies highlight the potential of LLMs to generate and respond to follow\-up questions, thereby enhancing performance across various tasks\. Follow\-up questioning improves the coherence and informativeness of document generationTix \([2024](https://arxiv.org/html/2608.13588#bib.bib38)\)and increases user satisfaction in conversational search by supporting deeper explorationKim et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib22)\)\. A prominent approach is the self\-ask method, in which LLMs generate and answer sub\-questions to decompose complex queries, yielding substantial gains in compositional reasoningPress et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib32)\)\. This line of work has been extended to interleave reasoning, retrieval, and self\-reflection, allowing LLMs to dynamically control queries during problem solvingJiang et al\. \([2023c](https://arxiv.org/html/2608.13588#bib.bib20)\); Yao et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib47)\); Asai et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib2)\)\. Domain\-specific adaptations demonstrate that iterative RAG can effectively address complex clinical scenariosXiong et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib44)\)\. In broader contexts, additional work explores anticipating follow\-up questions in information searchWilcock \([2024](https://arxiv.org/html/2608.13588#bib.bib43)\), refining outputs through targeted questioningShridhar et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib36)\), and augmenting LLMs with search engines to improve factualityVu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib41)\)\.

## 3Preliminary Analysis

Our proposed approach is grounded in two key assumptions: \(1\) LLMs can reliably judge whether a question is answerable given a set of evidence; and \(2\) if the question is unanswerable, LLMs can identify the additional information required to derive the correct answer\. To empirically validate these assumptions, we conduct two preliminary analyses guided by the following research questions:RQ1: Can LLMs determine whether the provided evidence is sufficient to answer a given question? RQ2: If the evidence is insufficient, can LLMs identify the missing information needed to derive the correct answer?

### 3\.1Settings

To address these questions, we employ the MuSiQue datasetTrivedi et al\. \([2022](https://arxiv.org/html/2608.13588#bib.bib39)\), which consists of multi\-hop QA pairs requiring two to four reasoning steps\. For each hop length, we randomly sample 400 QA pairs\. Following prior workWang et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib42)\), we useGPT\-4oto generate sub\-question and sub\-answer pairs based on the decomposition annotations provided in the dataset\. Table[7](https://arxiv.org/html/2608.13588#A1.T7)presents examples of the prompts and resulting generated data\.

Using the original question along with the generated sub\-QA pairs, we conduct experiments with several LLMs, includingLLaMA\-3\.1\-8B\-InstructDubey et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib10)\),Mistral\-7B\-InstructJiang et al\. \([2023a](https://arxiv.org/html/2608.13588#bib.bib17)\),GPT\-3\.5\-TurboBrown et al\. \([2020](https://arxiv.org/html/2608.13588#bib.bib3)\), andGPT\-4oAchiam et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib1)\)\. Each model is evaluated on two tasks: \(1\) determining whether the provided evidence alone is sufficient to answer the original question, and \(2\) identifying the additional information required when the evidence is deemed insufficient\.

#### 3\.1\.1Answerability Judgment

To evaluate answerability judgment, we define two experimental conditions based on the completeness of the provided evidence:

- •Full: The evidence includes sub\-answers for all hops required to answer the original question \(e\.g\., all three sub\-answers in a 3\-hop QA pair\) →answerable\.
- •Partial: The evidence includes sub\-answers only up to an intermediate hop \(e\.g\., only the 1st and 2nd hops in a 3\-hop QA pair\) →unanswerable\.

#### 3\.1\.2Missing Information Identification\.

For missing information identification, we adopt apartialevidence setting\. Specifically, the sub\-answers up to a given hop are concatenated and provided as evidence, while the sub\-question at the subsequent hop is treated as the reference missing information that the model is required to identify\.

To evaluate model predictions, we employ two metrics\. BLEU score measures surface\-level similarity between the generated follow\-up question and the reference sub\-question by capturing n\-gram overlap\. Relatedness score \(Rel\.\) is computed through binary classification usingGPT\-4o, assessing whether the predicted and reference questions are topically or conceptually aligned\.

### 3\.2Results

Table 1:Results on answerability judgment for information sufficiency assessment\.![Refer to caption](https://arxiv.org/html/2608.13588v1/fig_final.jpg)Figure 2:Overall pipeline of IterCOMP\. Retrieved documents are decomposed into evidence segments and filtered via dual\-aspect relevance scoring\. An LLM then judges whether the candidate evidence set is sufficient to answer the question\. If answerable, the evidence is finalized as the compressed prompt and passed to the reader model\. If unanswerable, the LLM identifies missing information and generates a targeted follow\-up question, which serves as a new query for the filtering stage to enable iterative evidence accumulation\.Answerability Judgment\.Table[1](https://arxiv.org/html/2608.13588#S3.T1)presents the results of the answerability judgment task across various LLMs under two conditions:Full, where all required evidence \(i\.e\., sub\-answers for each reasoning step\) is provided, andPartial, where only a subset of evidence is available\.

The models exhibit contrasting trends across the two conditions\. In theFullsetting, accuracy consistently declines as the number of hops increases; for example,Mistral\-7Bdrops from 92% at 2\-hop to 78\.5% at 4\-hop\. This indicates that, even when all relevant evidence is available, increasing hop length amplifies both the volume of evidence and the complexity of their interconnections\. Accordingly, answerability judgment requires more than surface\-level binary classification and instead demands advanced reasoning capabilities to integrate multiple evidence segments\. In contrast, under thePartialsetting, all models show consistent improvements as hop length grows\. High\-performing models such asGPT\-4omaintain strong performance with minimal variance even at longer hop lengths\. For a more balanced evaluation, we also report overall F1 scores:Llama\-3\.1\-8B\(70\.51\),Mistral\-7B\(84\.27\),GPT\-3\.5\-Turbo\(73\.12\), andGPT\-4o\(83\.57\)\.

Although these scores reveal an asymmetry in detecting answerable and unanswerable cases, both misclassification types pose distinct risks\. Misclassifying an unanswerable case as answerable causes the compression process to terminate prematurely, leaving the reasoning chain with insufficient clues and thus compromising the reliability of the final answer\. Conversely, misclassifying an answerable case as unanswerable triggers unnecessary iterations; while this allows the reasoning chain to incorporate new evidence, prolonged iterations may introduce irrelevant noise that degrades compression effectiveness\. Given the relative degradation observed in theFullsetting, we bound the maximum number of iterations to prevent excessive refinement\. Building on this analysis, we design IterCOMP’s iterative compression mechanism with an iteration budget calibrated to these tendencies, reliably using answerability as a core signal to guide evidence collection in complex multi\-hop QA\.

Missing Information Identification\. Table[2](https://arxiv.org/html/2608.13588#S3.T2)presents the experimental results for the missing information identification task, which evaluates the quality of follow\-up questions generated by LLMs when the initial evidence is insufficient to answer the original question\. The analysis shows thatGPT\-4oachieves the best performance, with an average relatedness score of 90\.6% and an average BLEU score of 0\.50\. These results highlightGPT\-4o’s strong capability to pinpoint missing information and generate contextually appropriate follow\-up questions within multi\-hop reasoning contexts\. However, all models exhibit a consistent decline in both relatedness and BLEU scores as the number of hops increases\. This trend reflects the growing difficulty of accurately identifying knowledge gaps and formulating effective follow\-up questions as reasoning chains become longer and more complex\. While LLMs clearly demonstrate the capability to generate meaningful follow\-up questions, the observed performance degradation suggests that precisely locating and articulating missing information remains a challenging subtask in multi\-hop QA\.

Table 2:Results on missing information identification\.

## 4Methodology

### 4\.1Problem Formulation

We define the task of prompt compression as follows\. Given an initial questionqqand a large corpus of retrieved documentsD=\{d1,…,dN\}D=\\\{d\_\{1\},\\ldots,d\_\{N\}\\\}, the primary objective is to synthesize a compressed promptPcompP\_\{\\text\{comp\}\}\. The promptPcompP\_\{\\text\{comp\}\}consists of a concise yet information\-sufficient set of evidence fromDD, satisfying the constraintL​\(Pcomp\)≪∑i=1NL​\(di\)L\(P\_\{\\text\{comp\}\}\)\\ll\\sum\_\{i=1\}^\{N\}L\(d\_\{i\}\), whereL​\(⋅\)L\(\\cdot\)denotes the token length\. The principal aim is to construct aPcompP\_\{\\text\{comp\}\}such that a downstream reader modelMMcan generate a high\-fidelity output,y=M​\(Pcomp,q\)y=M\(P\_\{\\text\{comp\}\},q\), while preserving essential reasoning information from corpusDD\.

In multi\-hop QA, where synthesizing information across multiple sources is indispensable, prompt compression plays a pivotal role\. Its utility extends beyond filtering irrelevant content and selecting evidence that directly addresses the initial questionqq\. Multi\-hop reasoning frequently requires resolving latent intermediate questions not explicitly stated inqq\. Consequently, an effective compression strategy must support progressive evidence accumulation, enabling the model to incrementally integrate and build upon relevant information as the reasoning process unfolds\. This capability is fundamental to maintaining logical coherence across complex inferential chains and mitigating performance degradation due to attentional dilution or contextual drift in long inputs\.

### 4\.2Proposed Framework

#### 4\.2\.1Document Decomposition

Each documentdi∈Dd\_\{i\}\\in Dis segmented into a set of smaller units, termedevidence segmentsand denoted byEi=\{ei,1,…,ei,Ri\}E\_\{i\}=\\\{e\_\{i,1\},\\ldots,e\_\{i,R\_\{i\}\}\\\}, whereRiR\_\{i\}indicates the number of segments fordid\_\{i\}\. Although segments can be defined at various levels of granularity, such as paragraphs or sentences, we adopt sentence\-level decomposition in this work\. A sentence serves as a basic unit that preserves the local semantic context of the original input while enabling effective compressionXu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\); Choi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\)\. These evidence segments, along with the initial questionqq, are then passed to the relevant evidence filtering module\.

#### 4\.2\.2Relevant Evidence Filtering

The relevant evidence filtering module systematically identifies and retains salient evidence segments by evaluating the relevance of each candidate segmentei,je\_\{i,j\}with respect to a given questionqq\. To establish a robust foundation for evidence selection, our approach combines semantic and lexical signals in a unified scoring scheme\.

Semantic similaritycaptures contextual alignment between the questionqqand a candidate evidence segmentei,je\_\{i,j\}to encode deeper semantic correspondences\. We transform bothqqandei,je\_\{i,j\}into dense vectors representations via a text encoderE​\(⋅\)E\(\\cdot\)\. The semantic similarity score,Ssem​\(q,ei,j\)S\_\{\\text\{sem\}\}\(q,e\_\{i,j\}\), is then computed as the inner product between the corresponding embeddings:

Ssem​\(q,ei,j\)=E​\(q\)⊤​E​\(ei,j\),S\_\{\\text\{sem\}\}\(q,e\_\{i,j\}\)=E\(q\)^\{\\top\}E\(e\_\{i,j\}\),\(1\)whereE​\(q\)E\(q\)andE​\(ei,j\)E\(e\_\{i,j\}\)denote the embeddings of the questionqqand the evidence segmentei,je\_\{i,j\}, respectively\.

While semantic similarity effectively captures broader contextual relevance, it may overlook critical lexical cues\. In multi\-hop settings, sensitivity to precise keyword matches improves coverage and accuracyZhang et al\. \([2025](https://arxiv.org/html/2608.13588#bib.bib49)\)\. To complement this, we incorporate alexical similaritycomponent that quantifies token\-level relevance based on the importance of overlapping terms betweenqqandei,je\_\{i,j\}\. Following M3\-EmbeddingChen et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib5)\), we define the lexical importance weight of each tokenttin the questionqqas follows:

wtq=ReLU​\(𝐰lex⊤​E​\(t\)\),w\_\{t\}^\{q\}=\\text\{ReLU\}\(\{\\mathbf\{w\}\_\{\\text\{lex\}\}\}^\{\\top\}E\(t\)\),\(2\)where𝐰lex\\mathbf\{w\}\_\{\\text\{lex\}\}is a projection vector that maps a contextualized token representation to a scalar importance score, andE​\(t\)E\(t\)denotes the contextualized embedding of tokentt\. An identical operation is applied to computewtei,jw^\{e\_\{i,j\}\}\_\{t\}for tokens in the evidence segmentei,je\_\{i,j\}\. The lexical similarity score is then defined as:

Slex​\(q,ei,j\)=∑t∈q∩ei,jwtq⋅wtei,j,S\_\{\\text\{lex\}\}\(q,e\_\{i,j\}\)=\\sum\\limits\_\{t\\in q\\cap e\_\{i,j\}\}w^\{q\}\_\{t\}\\cdot w^\{e\_\{i,j\}\}\_\{t\},\(3\)which computes a weighted sparse inner product over co\-occurring tokens, capturing their joint lexical salience under respective contexts\.

For both the encoder and the projection vector𝐰lex\\mathbf\{w\}\_\{\\text\{lex\}\}, we adopt the pretrained bge\-m3 modelChen et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib5)\)without any additional fine\-tuning\. All parameters remain frozen throughout the entire IterCOMP pipeline, in line with our training\-free design principle\.

Finally, we combine the semantic and lexical signals into a singledual\-aspect relevance score:

Sdual​\(q,ei,j\)\\displaystyle S\_\{\\text\{dual\}\}\(q,e\_\{i,j\}\)=λ⋅Ssem​\(q,ei,j\)\\displaystyle=\\lambda\\cdot S\_\{\\text\{sem\}\}\(q,e\_\{i,j\}\)\(4\)\+\(1−λ\)⋅Slex​\(q,ei,j\),\\displaystyle\\quad\+\(1\-\\lambda\)\\cdot S\_\{\\text\{lex\}\}\(q,e\_\{i,j\}\),where the hyperparameterλ∈\[0,1\]\\lambda\\in\[0,1\]balances semantic and lexical relevance\.

To prune less relevant segments, we adopt percentile\-based filtering that adaptively selects evidence according to relative importance within the retrieved context\. Let𝒮=\{Sdual​\(q,ei,j\)∣∀i,j\}\\mathcal\{S\}=\\\{S\_\{\\text\{dual\}\}\(q,e\_\{i,j\}\)\\mid\\forall i,j\\\}denote the multiset of relevance scores for all candidates\. The filtered candidate setEcandE\_\{\\text\{cand\}\}is constructed by retaining segments whose scores exceed thekk\-th percentile threshold over the global score distribution:

Ecand=\{ei,j\|Sdual​\(q,ei,j\)≥Percentile​\(𝒮,k\)\}\.E\_\{\\text\{cand\}\}=\\\{e\_\{i,j\}\\\>\|\\\>S\_\{\\text\{dual\}\}\(q,e\_\{i,j\}\)\\geq\\text\{Percentile\}\(\\mathcal\{S\},k\)\\\}\.\(5\)The segments retained inEcandE\_\{\\text\{cand\}\}form the basis for constructing the final compressed promptPcompP\_\{\\text\{comp\}\}\. During multi\-hop inference, this adaptive filtering is applied at each reasoning step, enforcing a dynamic cutoff that suppresses noise propagation and guides the model to attend to salient evidence throughout the reasoning chain\.

Table 3:Experimental results on three multi\-hop QA benchmark datasets\.Ratiodenotes the compression ratio, defined as the proportion of tokens in the compressed prompt relative to the those in the raw documents\. The best performance is highlighted inbold, and the second\-best is highlighted with anunderline\. The full experimental results are presented in Appendix[B\.3](https://arxiv.org/html/2608.13588#A2.SS3)\.
#### 4\.2\.3Answerability Judgment & Missing Information Identification

The answerability judgment module serves as a reasoning controller, establishing a dynamic feedback loop for iterative evidence accumulation\. At iterationhh, the currently accumulated candidate evidence set is denoted asEcand\(h\)E\_\{\\text\{cand\}\}^\{\(h\)\}\. The module employs an LLM as a binary classifier to determine whetherEcand\(h\)E\_\{\\text\{cand\}\}^\{\(h\)\}contains sufficient information to answer the original questionq\(0\)q^\{\(0\)\}\.

If the judgment isanswerable, the framework triggers anearly termination\. The evidence set is finalized and used as the compressed prompt,Pcomp=Ecand\(h\)P\_\{\\text\{comp\}\}=E\_\{\\text\{cand\}\}^\{\(h\)\}, and passed to the reader modelMMto generate the final answer\. This principled stopping condition prevents unnecessary iterations once sufficient information has been gathered\. Conversely, if the judgment isunanswerable, the framework proceeds withiterative refinement\. The LLM explicitly identifies the missing information that prevents a complete answer to the original questionq\(0\)q^\{\(0\)\}\. To bridge the gap, it formulates a targeted follow\-up questionq\(h\)q^\{\(h\)\}to retrieve complementary evidence\. Relevant evidence filtering is subsequently re\-applied with respect toq\(h\)q^\{\(h\)\}, producing an updated setEcand\(h\+1\)E\_\{\\text\{cand\}\}^\{\(h\+1\)\}progressively accumulated across iterations\. This cycle repeats until the sufficiency condition is satisfied or a predefined maximum hop limit is reached\.

## 5Experiments

### 5\.1Experimental Setup

We evaluate our proposed prompt compression method, IterCOMP, in a zero\-shot setting on three widely used multi\-hop QA datasets: MuSiQueTrivedi et al\. \([2022](https://arxiv.org/html/2608.13588#bib.bib39)\), 2WikiMultiHopQAHo et al\. \([2020a](https://arxiv.org/html/2608.13588#bib.bib14)\), and HotpotQAYang et al\. \([2018](https://arxiv.org/html/2608.13588#bib.bib46)\)\. All experiments are conducted on dev sets of these datasets\. For MuSiQue, we specifically use themusique\_ans\_v1\.0\_devsubset, which contains only answerable questions\. Further detailed statistics and descriptions of the datasets are provided in the Appendix[B\.1](https://arxiv.org/html/2608.13588#A2.SS1)\.

We compare IterCOMP against several baselines\. To ensure a fair and meaningful comparison, we focus our evaluation on hard prompt compression and extractive compression strategies, which are most relevant to our approach\. These include representative prompt compression baselines such as LLMLinguaJiang et al\. \([2023b](https://arxiv.org/html/2608.13588#bib.bib18)\), LongLLMLinguaJiang et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib19)\), LLMLingua\-2Pan et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib31)\), RECOMPXu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\), Selective\-ContextLi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib26)\), and R2CChoi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\)\. We additionally report the performance ofOracle, which serves as an upper bound for document\-level compression by providing the reader model only with the gold supporting documents\. In contrast,Raw Documentconcatenates all documents in datasets without any filtering, representing a no\-compression scenario\. Detailed explanations of these baselines are available in Appendix[B\.2](https://arxiv.org/html/2608.13588#A2.SS2)\.

We employLLaMA\-3\-8BDubey et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib10)\)as the reader model across all evaluated methods and baselines\. For IterCOMP, we bound the maximum number of iterations at 5 and set the hyperparameterλ\\lambda, which balances the dual\-aspect relevance score, to 0\.6\. The percentile thresholdkkfor relevance filtering is empirically determined and aligned with theOraclecompression setting for fair comparison:k=90k=90for MuSiQue and HotpotQA, andk=85k=85for 2WikiMultiHopQA\.

### 5\.2Experimental Results

##### Main Results\.

To evaluate the effectiveness of our proposed IterCOMP, we employ Exact Match \(EM\) and F1 scores as metrics for QA performance, alongside the compression ratio \(Ratio\) to measure efficiency gains\. As shown in Table[3](https://arxiv.org/html/2608.13588#S4.T3), IterCOMP consistently achieves the highest EM and F1 scores across all benchmark datasets, significantly outperforming the baselines\. Notably, it yields substantial improvements over theRaw Documentssetting, in which no compression is applied\. For instance, on the MuSiQue dataset, the F1 score increases from 19\.92 to 27\.36, while on HotpotQA it rises from 43\.63 to 51\.78\. These results demonstrate that our proposed adaptive compression framework enhances downstream QA performance while reducing input length \(e\.g\., a 7×\\timesreduction on MuSiQue\) by effectively filtering out irrelevant content from lengthy raw documents\.

Table 4:Experimental results across different hop lengths\.\# Iterationsindicates the average number of iteration loops per question, and\# Tokensdenotes the token lengths of the compressed prompt\.Compared to existing compression methods, IterCOMP demonstrates notable performance gains\. On MuSiQue, it attains an F1 score of 27\.36, surpassing the second\-best method, R2C, by 4\.92 points\. Importantly, these results are achieved in a training\-free manner, underscoring the generalizability and practicality of our approach\. The proposed framework is designed to bridge missing\-information gaps in multi\-hop QA reasoning\. This iterative refinement design dynamically retains only the most relevant information and ensures high performance even under strict input length constraints\.

To further contextualize the results, we compare performance against theOraclesetting, which serves as an upper bound based on ideal document selection\. On HotpotQA, the performance gap betweenRaw DocumentsandOracleis 15\.32 points\. IterCOMP attains an F1 score of 51\.79, effectively closing 53\.2% of this gap\. These results highlight the effectiveness of our framework in identifying and retaining compact yet highly informative evidence that supports complex multi\-hop reasoning\.

##### Reasoning Complexity\.

To assess the robustness of IterCOMP under varying reasoning complexity, we evaluate its performance on the MuSiQue dataset with respect to hop length\. As shown in Table[4](https://arxiv.org/html/2608.13588#S5.T4), performance declines as reasoning paths become longer\. Specifically, the F1 score drops from 30\.19 at 2\-hop to 20\.72 at 4\-hop\. This decline trend reflects the inherent difficulty of maintaining a coherent evidence chain and further indicates that increasing reasoning complexity imposes additional challenges on the reader model\.

The average number of iterations also increases with hop length, rising from 1\.98 for 2\-hop to 3\.08 for 4\-hop, indicating that IterCOMP dynamically allocates additional refinement steps when confronted with more complex questions\. Similarly, the length of the compressed prompt expands from 228 to 403 tokens, reflecting that the framework adaptively retains a larger evidence base to support longer reasoning chains\. Overall, rather than operating as a static filter, IterCOMP adjusts both its reasoning depth and evidence size in accordance with task complexity, thereby facilitating more effective handling of challenging multi\-hop reasoning scenarios\.

### 5\.3Ablation Study & Analysis

As shown in Table[5\.3](https://arxiv.org/html/2608.13588#S5.SS3), we conduct an ablation study on the MuSiQue dataset to assess the contribution of each component in IterCOMP\. RemovingIterative Refinementleads to the largest performance drop, confirming the necessity of evidence accumulation in multi\-hop reasoning\. ExcludingRelevant Evidence Filtering\(no\-compression\) also results in poor performance \(19\.92 F1\), underscoring the importance of discarding irrelevant context\. The absence of theAnswerability Judgmentmodule degrades performance to 23\.63 F1 and produces less compact prompts, validating the role of early stopping for both accuracy and efficiency\. For the similarity measure, relying solely on semantic or lexical signals is suboptimal, whereas their combination achieves the best performance, demonstrating the synergistic complementarity of semantic and lexical cues\.

Table 5:Ablation study of IterCOMP, evaluating the role of each component and comparing different similarity measures for evidence filtering\.![Refer to caption](https://arxiv.org/html/2608.13588v1/figure_viz.jpg)Figure 3:Comparison of\(left\)QA performance and\(right\)compressed token length across different percentile valueskk\.
We evaluate different percentile valueskkfor relevant evidence filtering on the MuSiQue dataset, as shown in Figure[3](https://arxiv.org/html/2608.13588#S5.F3)\.\(Left\)The F1 score increases steadily with largerkk, peaking at 27\.36 whenk=90k=90\. This indicates that a stricter relevance threshold effectively isolates salient information and reduces noise, thereby enhancing the reader model’s reasoning\. However, performance drops atk=95k=95, suggesting that an overly stringent cutoff discards supplementary evidence necessary for completing the reasoning chain\.\(Right\)Askkincreases, compressed prompts become shorter due to more aggressive filtering\. Notably, whenk≥90k\\geq 90, the compressed prompt is even more concise than that of theOracle\.

Table 6:Efficiency analysis with API\-based LLMs\. The reported cost and latency correspond to the final QA step performed by the reader LLM with the compressed prompt\.
To assess the practical utility and economic viability of our method, we analyze 500 random instances from the MuSiQue dataset using prominent API\-based LLMs\. As shown in Table[6](https://arxiv.org/html/2608.13588#S5.T6), applying our framework for prompt compression substantially improves inference efficiency\. Across all models, compression reduces costs by 75\.5%–79\.4%, making multi\-hop QA with large\-scale commercial LLMs considerably more affordable\. In addition, compressed prompts consistently accelerate inference, highlighting prompt compression as a practical enabler of scalable, time\-efficient, and cost\-effective deployment of advanced complex reasoning in commercial LLMs\.

## 6Conclusion

This paper introduces IterCOMP, a unified prompt compression framework for multi\-hop QA\. IterCOMP embeds multi\-step reasoning into the compression loop through iterative refinement, combining answerability judgment with targeted follow\-up question generation\. This process progressively constructs a concise, information\-rich prompt by retaining only the evidence essential for complex reasoning\. Extensive experiments on three multi\-hop QA benchmarks demonstrate that IterCOMP significantly enhances QA performance while reducing token usage\. Its training\-free and model\-agnostic design ensures broad applicability, including integration with commercial black\-box LLMs for which fine\-tuning is infeasible\. By addressing both the performance degradation in long contexts and the economic burden of large\-scale inference, IterCOMP offers a robust and scalable solution for complex multi\-document LLM applications\.

## Limitations

While IterCOMP demonstrates improvements in multi\-hop QA, several aspects remain open for refinement\. First, its effectiveness depends on the reasoning capability of the underlying LLM for answerability judgment and missing information detection, introducing the risk of error propagation such as premature termination or accumulation of irrelevant evidence\. Second, the iterative design incurs additional compression overhead compared to single\-pass methods, which could be alleviated by lightweight controllers or parallel filtering\. Third, performance is sensitive to hyperparameters such as the relevance percentile and iteration limit; adaptive mechanisms that dynamically adjust these settings based on question complexity or intermediate retrieval quality could further enhance robustness\. Fourth, our lexical\-semantic scoring module adopts a relatively simple combination of signals, and integrating more sophisticated scoring schemes may yield additional performance gains\. Finally, our evaluations focus on general\-domain multi\-hop QA benchmarks to ensure fair comparison with existing baselines; although IterCOMP is modular and domain\-agnostic by design, validating its generalization to diverse reasoning types and non\-Wikipedia domains remains an important direction for future work\.

## Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \[RS\-2021\-II211341, Artificial Intelligence Graduate School Program \(Chung\-Ang University\)\] and by the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \(RS\-2025\-00556246\)\.

## References

- Achiam et al\. \(2023\)Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others\. 2023\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*\.
- Asai et al\. \(2024\)Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi\. 2024\.Self\-rag: Learning to retrieve, generate, and critique through self\-reflection\.In*The Twelfth International Conference on Learning Representations*\.
- Brown et al\. \(2020\)Tom B\. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert\-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M\. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others\. 2020\.[Language models are few\-shot learners](https://arxiv.org/abs/2005.14165)\.*Preprint*, arXiv:2005\.14165\.
- Cao et al\. \(2024\)Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, and Jinsong Su\. 2024\.[Retaining key information under high compression ratios: Query\-guided compressor for LLMs](https://doi.org/10.18653/v1/2024.acl-long.685)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 12685–12695, Bangkok, Thailand\. Association for Computational Linguistics\.
- Chen et al\. \(2024\)Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu\. 2024\.[M3\-embedding: Multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation](https://doi.org/10.18653/v1/2024.findings-acl.137)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, Bangkok, Thailand\. Association for Computational Linguistics\.
- Cheng et al\. \(2024\)Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si\-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao\. 2024\.xrag: Extreme context compression for retrieval\-augmented generation with one token\.*Advances in Neural Information Processing Systems*, 37:109487–109516\.
- Chevalier et al\. \(2023\)Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen\. 2023\.[Adapting language models to compress contexts](https://doi.org/10.18653/v1/2023.emnlp-main.232)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 3829–3846, Singapore\. Association for Computational Linguistics\.
- Choi et al\. \(2024\)Eunseong Choi, Sunkyung Lee, Minjin Choi, Jun Park, and Jongwuk Lee\. 2024\.[From reading to compressing: Exploring the multi\-document reader for prompt compression](https://doi.org/10.18653/v1/2024.findings-emnlp.864)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 14734–14754, Miami, Florida, USA\. Association for Computational Linguistics\.
- Chuang et al\. \(2024\)Yu\-Neng Chuang, Tianwei Xing, Chia\-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu\. 2024\.[Learning to compress prompt in natural language formats](https://doi.org/10.18653/v1/2024.naacl-long.429)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 7756–7767, Mexico City, Mexico\. Association for Computational Linguistics\.
- Dubey et al\. \(2024\)Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others\. 2024\.The llama 3 herd of models\.*arXiv e\-prints*, pages arXiv–2407\.
- Gao et al\. \(2023\)Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, and 1 others\. 2023\.Retrieval\-augmented generation for large language models: A survey\.*arXiv preprint arXiv:2312\.10997*, 2\(1\):32\.
- Ge et al\. \(2024\)Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si\-Qing Chen, and Furu Wei\. 2024\.In\-context autoencoder for context compression in a large language model\.In*The Twelfth International Conference on Learning Representations*\.
- Geva et al\. \(2021\)Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant\. 2021\.Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.*Transactions of the Association for Computational Linguistics*, 9:346–361\.
- Ho et al\. \(2020a\)Xanh Ho, Anh\-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa\. 2020a\.[Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps](https://doi.org/10.18653/v1/2020.coling-main.580)\.In*Proceedings of the 28th International Conference on Computational Linguistics*, pages 6609–6625, Barcelona, Spain \(Online\)\. International Committee on Computational Linguistics\.
- Ho et al\. \(2020b\)Xanh Ho, Anh\-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa\. 2020b\.Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.*arXiv preprint arXiv:2011\.01060*\.
- Hwang et al\. \(2025\)Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C\. Park\. 2025\.[EXIT: Context\-aware extractive compression for enhancing retrieval\-augmented generation](https://doi.org/10.18653/v1/2025.findings-acl.253)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 4895–4924, Vienna, Austria\. Association for Computational Linguistics\.
- Jiang et al\. \(2023a\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\. 2023a\.[Mistral 7b](https://arxiv.org/abs/2310.06825)\.*Preprint*, arXiv:2310\.06825\.
- Jiang et al\. \(2023b\)Huiqiang Jiang, Qianhui Wu, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\. 2023b\.[LLMLingua: Compressing prompts for accelerated inference of large language models](https://doi.org/10.18653/v1/2023.emnlp-main.825)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 13358–13376, Singapore\. Association for Computational Linguistics\.
- Jiang et al\. \(2024\)Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\. 2024\.[LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression](https://doi.org/10.18653/v1/2024.acl-long.91)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1658–1677, Bangkok, Thailand\. Association for Computational Linguistics\.
- Jiang et al\. \(2023c\)Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi\-Yu, Yiming Yang, Jamie Callan, and Graham Neubig\. 2023c\.[Active retrieval augmented generation](https://doi.org/10.18653/v1/2023.emnlp-main.495)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7969–7992, Singapore\. Association for Computational Linguistics\.
- Jung and Kim \(2024\)Hoyoun Jung and Kyung\-Joong Kim\. 2024\.[Discrete prompt compression with reinforcement learning](https://doi.org/10.1109/ACCESS.2024.3403426)\.*IEEE Access*, 12:72578–72587\.
- Kim et al\. \(2024\)Hyunwoo Kim, Yoonseo Choi, Taehyun Yang, Honggu Lee, Chaneon Park, Yongju Lee, Jin Young Kim, and Juho Kim\. 2024\.Using llms to investigate correlations of conversational follow\-up queries with user satisfaction\.*arXiv preprint arXiv:2407\.13166*\.
- Levy et al\. \(2025\)Shahar Levy, Nir Mazor, Lihi Shalmon, Michael Hassid, and Gabriel Stanovsky\. 2025\.[More documents, same length: Isolating the challenge of multiple documents in RAG](https://doi.org/10.18653/v1/2025.findings-emnlp.1064)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 19539–19547, Suzhou, China\. Association for Computational Linguistics\.
- Lewis et al\. \(2020\)Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen\-tau Yih, Tim Rocktäschel, and 1 others\. 2020\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.*Advances in neural information processing systems*, 33:9459–9474\.
- Li et al\. \(2025a\)Yuankai Li, Jia\-Chen Gu, Di Wu, Kai\-Wei Chang, and Nanyun Peng\. 2025a\.[BRIEF: Bridging retrieval and inference for multi\-hop reasoning via compression](https://aclanthology.org/2025.findings-naacl.301/)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 5449–5470, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Li et al\. \(2023\)Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin\. 2023\.[Compressing context to enhance inference efficiency of large language models](https://doi.org/10.18653/v1/2023.emnlp-main.391)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 6342–6353, Singapore\. Association for Computational Linguistics\.
- Li et al\. \(2025b\)Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier\. 2025b\.[Prompt compression for large language models: A survey](https://aclanthology.org/2025.naacl-long.368/)\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 7182–7195, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Liskavets et al\. \(2025\)Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane K Luke\. 2025\.Prompt compression with context\-aware sentence encoding for fast and improved llm inference\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pages 24595–24604\.
- Liu et al\. \(2024\)Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\. 2024\.[Lost in the middle: How language models use long contexts](https://doi.org/10.1162/tacl_a_00638)\.*Transactions of the Association for Computational Linguistics*, 12:157–173\.
- Mu et al\. \(2023\)Jesse Mu, Xiang Li, and Noah Goodman\. 2023\.Learning to compress prompts with gist tokens\.*Advances in Neural Information Processing Systems*, 36:19327–19352\.
- Pan et al\. \(2024\)Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin\-Yew Lin, H\. Vicky Zhao, Lili Qiu, and Dongmei Zhang\. 2024\.[LLMLingua\-2: Data distillation for efficient and faithful task\-agnostic prompt compression](https://doi.org/10.18653/v1/2024.findings-acl.57)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 963–981, Bangkok, Thailand\. Association for Computational Linguistics\.
- Press et al\. \(2023\)Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis\. 2023\.[Measuring and narrowing the compositionality gap in language models](https://doi.org/10.18653/v1/2023.findings-emnlp.378)\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 5687–5711, Singapore\. Association for Computational Linguistics\.
- Schnitzler et al\. \(2024\)Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa\. 2024\.Morehopqa: More than multi\-hop reasoning\.*arXiv preprint arXiv:2406\.13397*\.
- Shao et al\. \(2023\)Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen\. 2023\.[Enhancing retrieval\-augmented large language models with iterative retrieval\-generation synergy](https://doi.org/10.18653/v1/2023.findings-emnlp.620)\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 9248–9274, Singapore\. Association for Computational Linguistics\.
- Shi et al\. \(2023\)Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou\. 2023\.Large language models can be easily distracted by irrelevant context\.In*International Conference on Machine Learning*, pages 31210–31227\. PMLR\.
- Shridhar et al\. \(2024\)Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz\. 2024\.[The ART of LLM refinement: Ask, refine, and trust](https://doi.org/10.18653/v1/2024.naacl-long.327)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 5872–5883, Mexico City, Mexico\. Association for Computational Linguistics\.
- Tang and Yang \(2024\)Yixuan Tang and Yi Yang\. 2024\.Multihop\-rag: Benchmarking retrieval\-augmented generation for multi\-hop queries\.*arXiv preprint arXiv:2401\.15391*\.
- Tix \(2024\)Bernadette J Tix\. 2024\.Follow\-up questions improve documents generated by large language models\.*arXiv preprint arXiv:2407\.12017*\.
- Trivedi et al\. \(2022\)Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\. 2022\.[MuSiQue: Multihop questions via single\-hop question composition](https://doi.org/10.1162/tacl_a_00475)\.*Transactions of the Association for Computational Linguistics*, 10:539–554\.
- Trivedi et al\. \(2023\)Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\. 2023\.[Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions](https://doi.org/10.18653/v1/2023.acl-long.557)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10014–10037, Toronto, Canada\. Association for Computational Linguistics\.
- Vu et al\. \(2024\)Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun\-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong\. 2024\.[FreshLLMs: Refreshing large language models with search engine augmentation](https://doi.org/10.18653/v1/2024.findings-acl.813)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 13697–13720, Bangkok, Thailand\. Association for Computational Linguistics\.
- Wang et al\. \(2025\)Keheng Wang, Feiyu Duan, Peiguang Li, Sirui Wang, and Xunliang Cai\. 2025\.[LLMs know what they need: Leveraging a missing information guided framework to empower retrieval\-augmented generation](https://aclanthology.org/2025.coling-main.163/)\.In*Proceedings of the 31st International Conference on Computational Linguistics*, pages 2379–2400, Abu Dhabi, UAE\. Association for Computational Linguistics\.
- Wilcock \(2024\)Graham Wilcock\. 2024\.[Anticipating follow\-up questions in exploratory information search](https://doi.org/10.18653/v1/2024.sigdial-1.9)\.In*Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 103–109, Kyoto, Japan\. Association for Computational Linguistics\.
- Xiong et al\. \(2024\)Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, and Aidong Zhang\. 2024\.Improving retrieval\-augmented generation in medicine with iterative follow\-up questions\.In*Biocomputing 2025: Proceedings of the Pacific Symposium*, pages 199–214\. World Scientific\.
- Xu et al\. \(2024\)Fangyuan Xu, Weijia Shi, and Eunsol Choi\. 2024\.[RECOMP: Improving retrieval\-augmented LMs with context compression and selective augmentation](https://openreview.net/forum?id=mlJLVigNHp)\.In*The Twelfth International Conference on Learning Representations*\.
- Yang et al\. \(2018\)Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D\. Manning\. 2018\.[HotpotQA: A dataset for diverse, explainable multi\-hop question answering](https://doi.org/10.18653/v1/D18-1259)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2369–2380, Brussels, Belgium\. Association for Computational Linguistics\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\. 2023\.React: Synergizing reasoning and acting in language models\.In*The eleventh international conference on learning representations*\.
- Yoon et al\. \(2024\)Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang\. 2024\.[CompAct: Compressing retrieved documents actively for question answering](https://doi.org/10.18653/v1/2024.emnlp-main.1194)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 21424–21439, Miami, Florida, USA\. Association for Computational Linguistics\.
- Zhang et al\. \(2025\)Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett\-Shapiro, Rui Zhang, Prasenjit Mitra, Caiming Xiong, and Chien\-Sheng Wu\. 2025\.Sirerag: Indexing similar and related information for multihop reasoning\.In*The Thirteenth International Conference on Learning Representations*\.
- Zhu et al\. \(2025\)Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang, and Wei Hu\. 2025\.[Mitigating lost\-in\-retrieval problems in retrieval augmented multi\-hop question answering](https://doi.org/10.18653/v1/2025.acl-long.1089)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 22362–22375, Vienna, Austria\. Association for Computational Linguistics\.

## Appendix APrompts

\\rowcolor\[HTML\]EFEFEFPrompt for Sub\-QA Pair GenerationGiven a question, its final answer, and evidences, please generate a sub\-question and sub\-answer for each evidence\. Each pair must be reflect the content of its corresponding evidence\.Question: When was the creator of The Painter’s Studio born?Answer: 10 June 1819Evidence: \["question": "The Painter’s Studio \>\> creator", "answer": "Gustave Courbet", "question": "The date of birth of \#1 is?", "answer": "10 June 1819"\]\(generated sub\-QA example\)Sub\-question 1: Who is the creator of The Painter’s Studio?Sub\-answer 1: The creator of The Painter’s Studio is Gustave Courbet\.Sub\-question 2: What is the date of birth of Gustave Courbet?Sub\-answer 2: The date of birth of Gustave Courbet is 10 June 1819\.\\rowcolor\[HTML\]EFEFEFAnswerability Judgment & Missing Information IdentificationGiven the following Question and Information, your task is to determine whether the Information alone is sufficient to answer the Question\.If the Information provides a clear and direct answer, of if answer to the Question can be logically derived by combining multiple statements within the Information, then output: "answerable"\.If the Information lacks the necessary content to fully answer the Question, then output: "unanswerable", and generate a follow\-up question that identifies the most specific and essential piece of missing information required to answer the Question\.Your response must strictly follow this format: \["answer": "answerable or unanswerable", "follow\_up\_question": "a specific and detailed question that would help retrieve the missing information"\]Do not include any explanation or reasoning outside of this format\.Table 7:Prompts and generated examples used for sub\-QA pair generation \(top\) and answerability judgment with missing information identification \(bottom\)\.

## Appendix BImplementation Details

### B\.1Datasets

- •MuSiQueTrivedi et al\. \([2022](https://arxiv.org/html/2608.13588#bib.bib39)\)consists of questions constructed by combining multiple single\-hop queries, requiring 2 to 4 reasoning hops, and is specifically designed to prevent answer derivation from superficial clues\.
- •2WikiMultiHopQAHo et al\. \([2020a](https://arxiv.org/html/2608.13588#bib.bib14)\)is a Wikipedia\-based dataset featuring questions that require up to five reasoning steps, with each question annotated with its explicit reasoning path and supporting evidence\.
- •HotpotQAYang et al\. \([2018](https://arxiv.org/html/2608.13588#bib.bib46)\)requires models to retrieve and synthesize evidence from multiple documents, containing explainable questions explicitly designed to encourage transparent and interpretable reasoning paths\.

Table[8](https://arxiv.org/html/2608.13588#A2.T8)presents the statistics of the benchmark multi\-hop QA datasets used in our experiments\. We employ three widely used standard datasets: MusiQue, 2WikiMultiHopQA, and HotpotQA\. For each dataset, we summarize the number of instances in the train, dev, and test splits\.

Table 8:Statistics of multi\-hop QA datasets used in our experiments\.Table 9:Full experimental results on three multi\-hop QA benchmarks, including abstractive RECOMP and CompAct\.
### B\.2Baselines

To ensure reproducibility, all experiments were conducted using officially released codebases and publicly available models\.

- •ForRECOMPXu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\), we implement an extractive compressor based on a dual\-encoder model that identifies and selects relevant sentences, representing a sentence\-level extractive strategy\.
- •Selective\-ContextLi et al\. \([2023](https://arxiv.org/html/2608.13588#bib.bib26)\)is a token\-level compression approach that removes low\-information lexical units using a compact language model\.
- •LLMLinguaJiang et al\. \([2023b](https://arxiv.org/html/2608.13588#bib.bib18)\)leverages a smaller language model to eliminate low\-perplexity tokens from the original prompt to satisfy a predefined compression ratio\.
- •LLMLingua\-2Pan et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib31)\)is an extension of LLMLingua that incorporates a refined budget controller and data distillation mechanisms to more effectively preserve essential information\.
- •LongLLMLinguaJiang et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib19)\)extends LLMLingua, introducing a question\-aware, coarse\-to\-fine compression strategy to retain critical information from lengthy documents\.
- •R2CChoi et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib8)\)compresses prompts by leveraging cross\-attention scores from a Fusion\-in\-Decoder \(FiD\) model to score the relevance of each chunk and sentence, retaining only those identified as most important\.

### B\.3Additional Experiments Results

RECOMPXu et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib45)\)also supports abstractive compression, which generates summaries by synthesizing information across retrieved documents\. Moreover, CompActYoon et al\. \([2024](https://arxiv.org/html/2608.13588#bib.bib48)\)compresses retrieved documents by jointly analyzing previously selected content alongside newly introduced segments, enabling context\-aware compression across the document set\.

However, a fair comparison with these methods is challenging due to inconsistencies in length control\. While neither our method nor most baselines enforce strict token\-level constraints, both abstractive RECOMP and CompAct exhibit substantially larger deviations from the target token budget\. These methods rely on free\-form generation without an explicit length\-control mechanism and fail to reliably regulate the length of the compressed prompt\. Consequently, we exclude these methods from the main comparison and instead report their full results in Table[9](https://arxiv.org/html/2608.13588#A2.T9), explicitly noting the fairness limitations arising from these inconsistent compression budgets\. As shown in Table[9](https://arxiv.org/html/2608.13588#A2.T9), IterCOMP still consistently outperforms all baselines, including abstractive RECOMP and CompAct, across the three benchmarks\.