Selective State-Space Adaptation and Retrieval for Language Model Reasoning

# Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Source: [https://arxiv.org/html/2607.19326](https://arxiv.org/html/2607.19326)
Atahan Dokme Larry Heck AI Virtual Assistant \(AVA\) Lab Georgia Institute of Technology \{adokme3,larryheck\}@gatech\.edu

###### Abstract

Low\-rank adaptation introduces a static learned update applied identically to every input\. The update provides task\-level adaptation but does not explicitly represent token\-level or instance\-level state variation\. A family of adapters is proposed that introduces selective state\-space recurrence at two complementary granularities\. At the token level,MaLoRA\(Mamba\-modulated low\-rank adaptation\) makes the adapter’s scaling factor a dynamic input\-dependent function with recurrent state across tokens, in contrast to the stateless modulators of prior work\. At the context level,MaRA\(Mamba Retrieval Adapter\) tracks cross\-segment state and selects the segments most relevant to the query, before the modulated language model generates its answer\. Across three frozen backbones \(Qwen\-2\.5\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\) and two reasoning benchmarks \(MuSiQue, 2WikiMultihopQA\), the family improves reasoning accuracy on every cell of the3×23\{\\times\}2grid, by\+6\.8\+6\.8F1 \(\+10\.5%\+10\.5\\%relative\) on average and up to\+9\.3\+9\.3F1 \(\+18\.2%\+18\.2\\%relative\) on the hardest cell over the LoRA baseline, and the token\-level gains carry to RULER QA\-2 under length stress\.

Selective State\-Space Adaptation and Retrieval for Language Model Reasoning

Atahan Dokme Larry HeckAI Virtual Assistant \(AVA\) LabGeorgia Institute of Technology\{adokme3,larryheck\}@gatech\.edu

## 1Introduction

Low\-rank adaptation\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)has become the standard approach for adjusting a frozen language model to a downstream task\. The adapter is trained once and applied uniformly to every input\. For tasks with multiple latent states, such as multi\-step reasoning, this static update is shared across all of those states, encouraging reliance on training\-set patterns rather than per\-input adjustment\. The goal of this study is to make adaptation more input\-adaptive and better aligned with the state structure of the query, while leaving the backbone frozen\.

Adaptation can operate at three levels: at the*task*level, where LoRA already operates; at the*token*level, where the strength of adaptation can become a function of the current token; and at the*context*level, where the model can attend to the input segments most relevant to the query\.

The first contribution of this work isMaLoRA\(Mamba\-modulated Low\-Rank Adaptation\), a recurrent token\-level modulator for Low\-Rank Adaptation\. Unlike prior token modulators that use a stateless per\-token scaling factor, MaLoRA uses a Mamba\-based selective state\-space module\(Gu and Dao,[2024](https://arxiv.org/html/2607.19326#bib.bib2)\)to propagate state across tokens\. The second contribution isMaRA\(Mamba Retrieval Adapter\), a*context\-level*adapter that scores each input segment via a state\-space module operating over segment embeddings, and selects the most relevant segments for the adapted model\.

Selective state\-space recurrence contributes at both levels of adaptation\. At the token level, MaLoRA’s recurrent state yields consistent gains over stateless modulation across backbones \(Table[1](https://arxiv.org/html/2607.19326#S3.T1)\)\. At the context level, the state\-space module performs evidence selection, which the token modulator cannot do because it is relevance\-blind at the paragraph level \(Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\)\. The same Mamba block therefore plays two different jobs in the same system\.

The family is evaluated on MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib9)\)and 2WikiMultihopQA\(Hoet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib30)\)across three frozen backbones \(Qwen\-2\.5\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\)\. RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib33)\)measures retention of base\-model capability under length stress\. The token\-level modulator and MaRA compose, and the combined system improves over LoRA on every cell of the3×23\{\\times\}2grid, with the two mechanisms contributing through complementary channels: modulation reshapes the low\-rank update by token role, MaRA selects query\-relevant segments before generation, and the gains accumulate rather than overlap\.

## 2Related Work

#### Parameter\-efficient adaptation\.

Low\-rank adaptation\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)adapts a frozen base model via a learned static update\. Extensions add weight decomposition\(Liuet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib5)\), adaptive rank allocation\(Zhanget al\.,[2023](https://arxiv.org/html/2607.19326#bib.bib4)\), recurrent adapters\(Nguyenet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib7)\), an asymmetric shared\-AAmulti\-head architecture\(Tianet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib26)\), and mixture\-of\-LoRA routing\(Douet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib28); Gaoet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib27)\)\. All share the assumption that the update is a static reparameterization of the base weights, which the input\-dependent variants below relax\.

#### Token\-aware and gated adaptation\.

Making the low\-rank update input\-dependent at the token level has been explored recently\. TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)learns a token\-dependent*diagonal*matrix between the LoRA factors and serves as the stateless prior\-art baseline in this work\. On top of this baseline, a*recurrent*token modulator \(MaLoRA\) propagates state across tokens, and state\-space recurrence is also placed at the*context\-segment*level, where it performs model\-attached evidence selection over the frozen generator’s own hidden states, a setting none of these token\-level methods address\. The token\-level analysis \(Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\) shows that modulation there is*role\-sensitive but relevance\-blind*, which motivates moving evidence\-relevance decisions to the segment level\.

#### State\-space models in language modeling\.

State\-space models provide sub\-quadratic sequence modeling\(Guet al\.,[2021](https://arxiv.org/html/2607.19326#bib.bib3); Smithet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib24); Gu and Dao,[2024](https://arxiv.org/html/2607.19326#bib.bib2); Dao and Gu,[2024](https://arxiv.org/html/2607.19326#bib.bib23)\)\. Two prior adapters apply such dynamics, and both place the recurrence*across layers*of the adapter stack:Nguyenet al\.\([2024](https://arxiv.org/html/2607.19326#bib.bib7)\)\(READ\) propagates state through the adapter stack across layers, andYuet al\.\([2025](https://arxiv.org/html/2607.19326#bib.bib49)\)\(SSMLoRA\) uses an SSM to interconnect LoRA matrices inserted into the same module type at different layers\. MaLoRA differs in*axis*: a selective state\-space module is the*per\-token gate*over the LoRA update, with recurrence*across tokens of the same input*, so that each token’s adapter strength carries memory of the trajectory leading up to it\.

#### Reasoning and evidence selection\.

Multi\-step QA benchmarks\(Hoet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib30); Trivediet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib9)\)expose a gap between the language model’s possession of the answer and its use of the supporting evidence; long\-context evaluation surfaces related shifts\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib33); Baiet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib29); Labanet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib36)\)\.

#### Retrieval baselines\.

Most retrieval baselines used in open\-domain and multi\-hop QA are external modules\. BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.19326#bib.bib44)\)is a sparse retriever, and dense or late\-interaction systems such as DPR\(Karpukhinet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib45)\), Contriever\(Izacardet al\.,[2021](https://arxiv.org/html/2607.19326#bib.bib42)\), E5\(Wanget al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib43)\), ColBERT\(Khattab and Zaharia,[2020](https://arxiv.org/html/2607.19326#bib.bib41)\), and recent embedding encoders\(Zhanget al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib40)\)all train a separate retrieval model rather than reusing the downstream language model’s hidden states\. MaRA, the retrieval adapter introduced here, is instead*model\-attached*: it scores segments from the intermediate hidden states of the same frozen backbone used for generation, then applies a small segment\-level state\-space module on top\. It is therefore closer to a context\-level adapter than to a conventional outsourced retriever\.

## 3MaLoRA: Stateful Token Modulation

![Refer to caption](https://arxiv.org/html/2607.19326v1/figures/malora_fig1.png)Figure 1:From LoRA to MaLoRA\.\(a\) LoRA: a static low\-rank updateB​A​xtBAx\_\{t\}adds to the frozen outputW​xtWx\_\{t\}\.\(b\) TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\): a per\-token diagonal modulatorΛ​\(xt\)=diag​\(exp⁡\(RMSNorm​\(P​xt\)\)\)\\Lambda\(x\_\{t\}\)=\\mathrm\{diag\}\\\!\\bigl\(\\exp\(\\mathrm\{RMSNorm\}\(Px\_\{t\}\)\)\\bigr\)is inserted between the LoRA factors, yieldinght=\(W\+B​Λ​\(xt\)​A\)​xth\_\{t\}=\(W\+B\\,\\Lambda\(x\_\{t\}\)\\,A\)\\,x\_\{t\}\. The modulator is stateless:Λ​\(xt\)\\Lambda\(x\_\{t\}\)depends only on the current token\.\(c\) MaLoRA: the per\-token head is replaced by a selective state\-space module \(Mamba\) over the projected sequence\{P​xt\}\\\{Px\_\{t\}\\\}, and the output is a scalarλ​\(xt\)=softplus⁡\(ϕ​\(Mamba​\(P​xt\)t\)\)\\lambda\(x\_\{t\}\)=\\operatorname\{softplus\}\(\\phi\(\\mathrm\{Mamba\}\(Px\_\{t\}\)\_\{t\}\)\)that scales therr\-dimensional intermediate\.### 3\.1Preliminaries

#### Low\-rank adaptation\.

Low\-rank adaptation\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)reparameterises the update to a frozen weightW0∈ℝdout×dinW\_\{0\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}as a low\-rank product\. For an inputx∈ℝdinx\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\}, the output becomes

h=\(W0\+B​A\)​x,A∈ℝr×din,B∈ℝdout×r,h\\;=\\;\\bigl\(W\_\{0\}\\,\+\\,BA\\bigr\)\\,x,\\quad A\\in\\mathbb\{R\}^\{r\\times d\_\{\\text\{in\}\}\},\\,B\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\},\(1\)wherer≪min⁡\(din,dout\)r\\ll\\min\(d\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\)is the rank\. The base weightW0W\_\{0\}is held frozen; onlyAAandBBare trained\. The same update is applied to every token of every input, so the magnitude of the adaptation is independent of the model’s representation of the current token\.

#### TopLoRA: stateless per\-token modulation\.

TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)relaxes this static assumption by inserting a learned per\-token diagonal modulatorΛ​\(xt\)∈ℝ\>0r×r\\Lambda\(x\_\{t\}\)\\in\\mathbb\{R\}^\{r\\times r\}\_\{\>0\}between the LoRA factors:

ht=\(W0\+B​Λ​\(xt\)​A\)​xt,h\_\{t\}\\;=\\;\\bigl\(W\_\{0\}\\,\+\\,B\\,\\Lambda\(x\_\{t\}\)\\,A\\bigr\)\\,x\_\{t\},\(2\)whereΛ​\(xt\)=diag​\(𝝀​\(xt\)\)\\Lambda\(x\_\{t\}\)=\\mathrm\{diag\}\\bigl\(\\boldsymbol\{\\lambda\}\(x\_\{t\}\)\\bigr\)with𝝀​\(xt\)=exp⁡\(RMSNorm​\(P​xt\)\)∈ℝ\>0r\\boldsymbol\{\\lambda\}\(x\_\{t\}\)=\\exp\\\!\\bigl\(\\mathrm\{RMSNorm\}\(Px\_\{t\}\)\\bigr\)\\in\\mathbb\{R\}^\{r\}\_\{\>0\}andP∈ℝr×dinP\\in\\mathbb\{R\}^\{r\\times d\_\{\\text\{in\}\}\}a learned low\-rank projection \(Fig\.[1](https://arxiv.org/html/2607.19326#S3.F1)b\)\. The modulator is*stateless*:Λ​\(xt\)\\Lambda\(x\_\{t\}\)depends only on the current token, with no propagation across positions\.

### 3\.2MaLoRA

MaLoRA replaces TopLoRA’s stateless per\-token diagonal with a stateful scalar gate\. The static low\-rank update in Eq\.[1](https://arxiv.org/html/2607.19326#S3.E1)is multiplied by a per\-token scalarλ​\(xt\)∈ℝ\>0\\lambda\(x\_\{t\}\)\\in\\mathbb\{R\}\_\{\>0\}produced by a selective state\-space module\(Gu and Dao,[2024](https://arxiv.org/html/2607.19326#bib.bib2)\)over the projected sequence:

ht\\displaystyle h\_\{t\}=\(W0\+B​λ​\(xt\)​A\)​xt,\\displaystyle=\\;\\bigl\(W\_\{0\}\\,\+\\,B\\,\\lambda\(x\_\{t\}\)\\,A\\bigr\)\\,x\_\{t\},\(3\)λ​\(xt\)\\displaystyle\\lambda\(x\_\{t\}\)=softplus⁡\(ϕ​\(ot\)\),\\displaystyle=\\;\\operatorname\{softplus\}\\\!\\bigl\(\\phi\(o\_\{t\}\)\\bigr\),whereP∈ℝr×dinP\\in\\mathbb\{R\}^\{r\\times d\_\{\\text\{in\}\}\}is a learned low\-rank projection sharing dimensions withAA,ϕ\\phiis a small learned head, andoto\_\{t\}is the Mamba output at positiontt\. The Mamba module maintains a hidden statest∈ℝdss\_\{t\}\\in\\mathbb\{R\}^\{d\_\{s\}\}with input\-dependent dynamics:

st=A¯t​st−1\+B¯t​P​xt,ot=C¯t​st,s\_\{t\}\\;=\\;\\bar\{A\}\_\{t\}\\,s\_\{t\-1\}\\;\+\\;\\bar\{B\}\_\{t\}\\,Px\_\{t\},\\quad o\_\{t\}\\;=\\;\\bar\{C\}\_\{t\}\\,s\_\{t\},\(4\)whereA¯t,B¯t,C¯t\\bar\{A\}\_\{t\},\\bar\{B\}\_\{t\},\\bar\{C\}\_\{t\}are computed fromP​xtPx\_\{t\}in the standard Mamba parameterisation \(Fig\.[1](https://arxiv.org/html/2607.19326#S3.F1)c\)\. Eq\.[3](https://arxiv.org/html/2607.19326#S3.E3)reduces to Eq\.[1](https://arxiv.org/html/2607.19326#S3.E1)whenλ​\(xt\)=1\\lambda\(x\_\{t\}\)=1\.

The Mamba block is the central contribution: it brings selective state\-space recurrence inside the parameter\-efficient adapter itself, giving the per\-token gate memory of the trajectoryx1,…,xtx\_\{1\},\\ldots,x\_\{t\}rather than reacting only toxtx\_\{t\}\. The remaining design choices are as follows\.\(i\)The output is a single scalarλ​\(xt\)\\lambda\(x\_\{t\}\)rather than TopLoRA’s diagonalΛ​\(xt\)\\Lambda\(x\_\{t\}\), which exposes one interpretable value per token for the behavioural analyses of Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\.\(ii\)The activation is softplus so the modulator can both suppress \(λ<1\\lambda<1\) and amplify \(λ\>1\\lambda\>1\) the LoRA update; activation alternatives are ablated in Appendix[D](https://arxiv.org/html/2607.19326#A4)\.

MaLoRA is trained jointly with the LoRA matrices, attached to\{q​\_​proj,k​\_​proj,v​\_​proj,up​\_​proj,down​\_​proj\}\\\{q\\\_\\mathrm\{proj\},k\\\_\\mathrm\{proj\},v\\\_\\mathrm\{proj\},\\mathrm\{up\}\\\_\\mathrm\{proj\},\\mathrm\{down\}\\\_\\mathrm\{proj\}\\\}with projection\- and layer\-specific parameters\. The Mamba module adds approximately50%50\\%to the LoRA parameter count; the TopLoRA baseline adds approximately54%54\\%\(its diagonal head is wider than MaLoRA’s scalar output\)\. Exact counts are in Appendix[C](https://arxiv.org/html/2607.19326#A3)\.

### 3\.3Modulator results

Two multi\-hop QA datasets are used: MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib9)\)\(22–44hops over2020candidates\) and 2WikiMultihopQA\(Hoet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib30)\)\(up to44hops over1010candidates\)\. All three frozen backbones \(Qwen\-2\.5\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\) are evaluated at rankr=16r\{=\}16with EM/F1, beam44, over three seeds; full hyperparameters in Appendix[B](https://arxiv.org/html/2607.19326#A2)\. MaLoRA improves over both LoRA and the TopLoRA baseline on every cell of Table[1](https://arxiv.org/html/2607.19326#S3.T1), with the largest gains over LoRA on MuSiQue \(up to\+7\.9\+7\.9F1 on Llama\)\. The recurrent\-state contribution is dissected in Section[3\.5](https://arxiv.org/html/2607.19326#S3.SS5), and the same pattern extends to commonsense reasoning \(Appendix[H](https://arxiv.org/html/2607.19326#A8)\)\.

Table 1:Token modulators across three backbones\.F1 on MuSiQue and 2WikiMultihopQA, and RULER QA\-2 accuracy \(%\) at44k context using the MuSiQue\-trained checkpoints\. Means over three seeds; per\-seed std in Appendix[D](https://arxiv.org/html/2607.19326#A4)\(Table[7](https://arxiv.org/html/2607.19326#A4.T7)\)\.MuSiQue2WikiQA\-2BBMethodF1F1Acc@4kQwen\-7BBase14\.414\.426\.026\.054\.1LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)51\.151\.177\.377\.356\.856\.8TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)55\.655\.677\.977\.959\.859\.8MaLoRA56\.4\\mathbf\{56\.4\}79\.3\\mathbf\{79\.3\}60\.7\\mathbf\{60\.7\}Llama\-8BBase17\.417\.431\.931\.910\.8LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)54\.454\.478\.578\.560\.960\.9TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)61\.761\.780\.380\.363\.063\.0MaLoRA62\.3\\mathbf\{62\.3\}83\.3\\mathbf\{83\.3\}64\.7\\mathbf\{64\.7\}Gemma\-9BBase25\.325\.334\.234\.238\.2LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)59\.859\.877\.277\.264\.464\.4TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)59\.559\.579\.379\.365\.465\.4MaLoRA63\.7\\mathbf\{63\.7\}82\.2\\mathbf\{82\.2\}65\.8\\mathbf\{65\.8\}
### 3\.4Modulation behavior

The trained modulator is suppression\-dominant: over90%90\\%ofλ​\(xt\)\\lambda\(x\_\{t\}\)fall below one, with structural separators and query tokens suppressed most heavily and content tokens retaining higher values \(Fig\.[2](https://arxiv.org/html/2607.19326#S3.F2)\)\. A variance decomposition shows roughly80%80\\%ofλ​\(xt\)\\lambda\(x\_\{t\}\)variance is per\-token rather than per\-projection or per\-layer, so the per\-token component cannot be reproduced by any static rescaling; suppression concentrates inv​\_​projv\\\_\\mathrm\{proj\},q​\_​projq\\\_\\mathrm\{proj\}, and the final transformer layer \(Appendix[E](https://arxiv.org/html/2607.19326#A5)\)\.

#### Mamba state usage\.

A direct check compares the Mamba block’s stateful outputzt=Mamba​\(P​x1:t\)tz\_\{t\}=\\mathrm\{Mamba\}\(Px\_\{1:t\}\)\_\{t\}against a stateless counterfactualztstateless=Mamba​\(P​xt\)z\_\{t\}^\{\\mathrm\{stateless\}\}=\\mathrm\{Mamba\}\(Px\_\{t\}\)that processes each position independently\. The relative discrepancy‖zt−ztstateless‖/‖zt‖\\\|z\_\{t\}\-z\_\{t\}^\{\\mathrm\{stateless\}\}\\\|/\\\|z\_\{t\}\\\|, averaged across the six MaLoRA cells \(Qwen/Llama/Gemma×\\timesMuSiQue/2WikiMultihopQA\), rises from0\.130\.13at position11to0\.610\.61–0\.630\.63across mid\-prompt positions and remains in that range out to the longest positions tested\. The stateless variant therefore diverges from the stateful one by a non\-trivial fraction at every position past the first token: the SSM state is genuinely accumulating information rather than acting as a decorative recurrence\. Per\-cell curves are in Appendix[G](https://arxiv.org/html/2607.19326#A7)\.

Aggregatingλ​\(xt\)\\lambda\(x\_\{t\}\)over prompt regions on the full validation set across the three MaLoRA\-backbone cells \(Table[2](https://arxiv.org/html/2607.19326#S3.T2); extended discussion in Appendix[F](https://arxiv.org/html/2607.19326#A6)\) confirms a clear role\-based pattern:the modulator differentiates tokens by their structural role\. Marker, query, and paragraph\-content regions receive distinctλ\\lambdavalues in every cell, and on Qwen and Llama, structural \(marker\) and query tokens are suppressed more than paragraph content \(Gemma is the exception, amplifying marker and query tokens instead\)\. Within paragraph content, however, the modulator is*relevance*\-blind: supporting and distractor paragraphs receive almost identicalλ\\lambda, with differences below0\.0020\.002in every cell\. Token\-level modulation thus reshapes how strongly the adapter applies to different prompt roles, which is a useful axis of adaptation, but evidence selection requires a separate mechanism at the segment level \(Section[4](https://arxiv.org/html/2607.19326#S4)\)\.

Table 2:Meanλ​\(xt\)\\lambda\(x\_\{t\}\)by token region\(full MuSiQue validation, MaLoRA across three backbones\)\. Mark\. / Ques\. / Supp\. / Dist\. = marker / question / supporting / distractor tokens\. Discussion in Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\.BBMark\.Ques\.Supp\.Dist\.Qwen0\.7920\.7920\.7720\.7720\.9040\.9040\.9040\.904Llama0\.7250\.7250\.6030\.6030\.7080\.7080\.7090\.709Gemma1\.2121\.2121\.7301\.7300\.9290\.9290\.9300\.930![Refer to caption](https://arxiv.org/html/2607.19326v1/x1.png)Figure 2:Per\-token modulationλ​\(xt\)\\lambda\(x\_\{t\}\)for MaLoRA on a MuSiQue example\(Llama\-3\.1\-8B, seed\-4343\)\. Cell colour averagesλ​\(xt\)\\lambda\(x\_\{t\}\)over160160modulated projections; darker = stronger suppression\. Question tokens are suppressed most; supporting and distractor content retain higherλ\\lambda\. Per\-region breakdown in Table[2](https://arxiv.org/html/2607.19326#S3.T2)\.

### 3\.5From input\-dependence to stateful input\-dependence

The gain over LoRA can be decomposed along two axes\.*Input\-dependence*\(TopLoRA, prior art\): replacing LoRA’s static scaling with a learned per\-token modulator already lifts F1 across most cells of Table[1](https://arxiv.org/html/2607.19326#S3.T1)\(up to\+7\.0\+7\.0F1 on MuSiQue Llama\), with no propagation of state across positions\.*Stateful input\-dependence*\(MaLoRA, this work\): a selective state\-space module makesλ​\(xt\)\\lambda\(x\_\{t\}\)depend on the trajectory leading up to positiontt, and adds a further consistent lift over TopLoRA on every cell\.

The same Mamba primitive is used at both granularities of adaptation\. At the token level \(MaLoRA, this section\), it provides a consistent lift over stateless per\-token modulation\. At the segment level \(next\), it performs evidence selection, which the token\-level modulator cannot do because it is relevance\-blind at the paragraph level \(Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\)\.

## 4MaRA: Context\-Level Adaptation

MaRA operates one level above the token modulator: the input is split into context segments, each segment is scored by relevance to the query, and the top\-kkare exposed to the modulated language model for generation\. The adapter is small \(≈3\\approx 3M trainable parameters\) and introduces no separate retrieval encoder: it reads the frozen backbone’s own intermediate hidden states and adds only a segment\-level mixer, with the recurrence running over segment embeddings rather than tokens\. The evidence information already resides in the backbone’s hidden states; the segment\-level state\-space module tracks the reasoning chain across segments and surfaces the supporting ones, in contrast to the token\-level modulator’s paragraph\-level relevance\-blindness \(Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\)\. Full per\-stage exposition is in Appendix[I](https://arxiv.org/html/2607.19326#A9)\.

MaLoRA modifies the frozen model’s weights with a per\-token low\-rank update\. MaRA instead modifies the input’s reasoning state: it adds no weight update and reads the backbone’s hidden states to select which segments the model integrates\. The two are adapters at different targets: one reshapes the weight update, the other reshapes the context and “hops” the model reasons over\.

![Refer to caption](https://arxiv.org/html/2607.19326v1/figures/malora_fig2.png)Figure 3:MaRA architecture and data flow\.Each \(query, segment\) pair\(Q,Ci\)\(Q,C\_\{i\}\)is encoded by the firstK=16K\{=\}16layers of the frozen backbone, yielding hidden statesHiH\_\{i\}\. A learned attention pool reduces eachHiH\_\{i\}to a single embeddingpip\_\{i\}\. A two\-layer Mamba runs over the segment\-level sequence\(p1,…,pN\)\(p\_\{1\},\\ldots,p\_\{N\}\), producing recurrent states\(s1,…,sN\)\(s\_\{1\},\\ldots,s\_\{N\}\)\. A local scoring head maps each recurrent statesis\_\{i\}to a relevance scorerir\_\{i\}; a separate global head reads the final stateg=sNg=s\_\{N\}and provides auxiliary supervision during training\.#### Method\.

Given a queryQQandNNsegments𝒞=\(C1,…,CN\)\\mathcal\{C\}=\(C\_\{1\},\\ldots,C\_\{N\}\), the retrieval adapter proceeds in four stages\.

*\(i\) Segment encoder\.*EachCiC\_\{i\}is concatenated withQQand passed through the firstK=16K\{=\}16layers of the frozen backbone, producing hidden statesHi∈ℝTi×dH\_\{i\}\\in\\mathbb\{R\}^\{T\_\{i\}\\times d\}\. No new encoder parameters are introduced\.

*\(ii\) Attention pool\.*A learned query vectorqaq\_\{a\}attends over each segment’s hidden states with an additive score, and the segment embedding is the attention\-weighted sum of those hidden states\. The hidden states are already query\-conditioned, since each segment is encoded together withQQin stage \(i\)\. WithWk∈ℝdp×dW\_\{k\}\\in\\mathbb\{R\}^\{d\_\{p\}\\times d\},

pi=∑j=1Tisoftmaxj⁡\(qa⊤​tanh⁡\(Wk​Hi,j\)\)​Hi,j\.p\_\{i\}\\;=\\;\\sum\_\{j=1\}^\{T\_\{i\}\}\\operatorname\{softmax\}\_\{j\}\\\!\\big\(q\_\{a\}^\{\\\!\\top\}\\tanh\(W\_\{k\}H\_\{i,j\}\)\\big\)\\,H\_\{i,j\}\.\(5\)
*\(iii\) Mamba over segments\.*The segment embeddings are passed through a two\-layer selective state\-space module\(Gu and Dao,[2024](https://arxiv.org/html/2607.19326#bib.bib2)\), with the recurrence running over*segments*\(not tokens\):

zi\\displaystyle z\_\{i\}=Win​pi,\\displaystyle=\\;W\_\{\\mathrm\{in\}\}\\,p\_\{i\},\(6\)\(s1,…,sN\)\\displaystyle\(s\_\{1\},\\ldots,s\_\{N\}\)=Mamba​\(z1,…,zN\),\\displaystyle=\\;\\mathrm\{Mamba\}\\\!\\big\(z\_\{1\},\\ldots,z\_\{N\}\\big\),whereWinW\_\{\\mathrm\{in\}\}projects each pooled segment embedding to the recurrence width\. The recurrence is causal:sis\_\{i\}depends on segments11throughii\. The unit of state evolution is one context segment, matching the granularity at which contextual transitions occur in multi\-step reasoning \(Section[3\.5](https://arxiv.org/html/2607.19326#S3.SS5)\)\.

*\(iv\) Scoring heads\.*Two linear heads read the recurrent states\. A local head scores each segment from its own statesis\_\{i\}, and a global head reads the final stateg=sNg=s\_\{N\}\(which has integrated allNNsegments through the causal recurrence\) and predicts allNNlabels jointly:

g=sN,ri=𝐰ℓ⊤​si,𝐫~=Wg​g∈ℝN\.g\\;=\\;s\_\{N\},\\qquad r\_\{i\}\\;=\\;\\mathbf\{w\}\_\{\\ell\}^\{\\\!\\top\}\\,s\_\{i\},\\qquad\\tilde\{\\mathbf\{r\}\}\\;=\\;W\_\{g\}\\,g\\;\\in\\;\\mathbb\{R\}^\{N\}\.\(7\)The per\-segment scorerir\_\{i\}is used to rank and select the top\-kksegments, which are forwarded to the modulator\-equipped language model\. The global head𝐫~\\tilde\{\\mathbf\{r\}\}supplies an auxiliary supervision signal that conditions the shared recurrence \(Appendix[I](https://arxiv.org/html/2607.19326#A9)\); it is not used at selection time\.

#### Training objective\.

Binary supporting\-segment labelsyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}supervise the retrieval adapter\. The loss combines weighted BCE on the local scores \(positive class up\-weighted to offset class imbalance\), an intra\-example pairwise margin that pushes supporting scores above non\-supporting scores, and an auxiliary BCE on the global head:

ℒ=\\displaystyle\\mathcal\{L\}\\;=∑iBCE​\(ri,yi\)\\displaystyle\\sum\_\{i\}\\mathrm\{BCE\}\(r\_\{i\},y\_\{i\}\)\(8\)\+β​∑i:yi=1∑j:yj=0max⁡\(0,m−\(ri−rj\)\)\\displaystyle\+\\beta\\sum\_\{i:\\,y\_\{i\}=1\}\\sum\_\{j:\\,y\_\{j\}=0\}\\max\(0,\\,m\-\(r\_\{i\}\-r\_\{j\}\)\)\+γ​∑iBCE​\(r~i,yi\),\\displaystyle\+\\gamma\\sum\_\{i\}\\mathrm\{BCE\}\(\\tilde\{r\}\_\{i\},y\_\{i\}\),wherer~i\\tilde\{r\}\_\{i\}is theii\-th logit of the global head \(Eq\.[7](https://arxiv.org/html/2607.19326#S4.E7)\) andγ=0\.5\\gamma\{=\}0\.5\. Selection uses only the local scoresrir\_\{i\}; the global term is an auxiliary regularizer on the shared recurrence\. The retrieval adapter is trained with the adapter frozen; no end\-to\-end fine\-tuning of the two stages is performed\. Hyperparameters and the canonical training pool are detailed in Appendix[B](https://arxiv.org/html/2607.19326#A2)\.

#### Iterative selection\.

At inference the retrieval adapter is applied overNNpasses\. Each pass re\-scores the candidate segments conditioned on the segments already selected, then appends the highest\-scoring unselected segment to an ordered anchor set\. The final top\-kkplaces these anchors first and fills the remainder by the last pass’s scores\. The number of passes is matched to the reasoning depth of each dataset:N=2N\{=\}2for 2WikiMultihopQA andN=4N\{=\}4for MuSiQue\. The procedure is given in Appendix[J](https://arxiv.org/html/2607.19326#A10)\.

### 4\.1MaRA architecture comparison

MaRA is compared to six alternatives at matched supervision and trainable budget, on MuSiQue and 2WikiMultihopQA with the base language model \(no adapter\)\. The alternatives ablate the per\-segment pool \(MeanPool,LastTok\), the segment\-level mixer \(Transformer,MLP,PoolOnly\), and the recurrence granularity \(TokenMamba, recurrence over concatenated tokens instead of pooled segments\)\. See Appendix[I](https://arxiv.org/html/2607.19326#A9)for definitions\.

Table 3:MaRA architecture comparison\.Recall@44on MuSiQue and 2WikiMultihopQA validation splits, no adapter attached\. Best per column in bold\.MuSiQue2WikiMultihopQAArchitectureQwenLlamaQwenLlamaMLP20\.619\.693\.895\.3LastTok30\.132\.384\.883\.7TokenMamba61\.454\.882\.576\.1PoolOnly57\.358\.290\.094\.7Transformer63\.860\.595\.595\.2MeanPool73\.876\.898\.698\.1Mamba \(ours\)80\.684\.199\.299\.5The Mamba over segments wins every cell\. Two gaps are diagnostic\. The gap toMeanPool\(segment\-level Mamba retained, learned attention pool replaced by a plain mean\) is77pp on MuSiQue, isolating the contribution of the learned within\-segment attention pool\. The gap toTokenMamba\(same Mamba module applied to concatenated tokens\) is1919to2929pp on MuSiQue, isolating the contribution of placing the recurrence at the segment level rather than the token level\. On 2WikiMultihopQA, every architecture beyond MLP and LastTok exceeds90%90\\%R@44and differences compress, but the Mamba still leads\. Per\-architecture analyses are in Appendix[I](https://arxiv.org/html/2607.19326#A9)\.

#### Comparison to classical retrieval baselines\.

BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.19326#bib.bib44)\)reaches R@4=47\.04\{=\}47\.0on MuSiQue and66\.466\.4on 2WikiMultihopQA\.Qwen3\-Embedding\-8B\(Zhanget al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib40)\), an88B\-parameter dense retriever \(larger than the backbone slice used here\), reaches75\.575\.5on MuSiQue via zero\-shot cosine similarity\. MaRA reaches80\.680\.6and99\.299\.2respectively, above both and above every retrieval baseline in Appendix[I](https://arxiv.org/html/2607.19326#A9)\(Table[10](https://arxiv.org/html/2607.19326#A9.T10)\)\. BM25 is an external sparse scoring system and Qwen3\-Embedding\-8B is a separate dense encoder; in contrast, MaRA reuses the frozen generator’s own hidden states and adds only a small segment\-level mixer \(Section[4](https://arxiv.org/html/2607.19326#S4)\)\. The advantage is not encoder size: it comes from training a small \(≈3\\approx 3M\) segment\-level mixer on those hidden states with a supervised objective tied to the downstream task\.

## 5Main Results

The full system combines the token\-level modulator of Section[3](https://arxiv.org/html/2607.19326#S3)\(MaLoRA\) with MaRA of Section[4](https://arxiv.org/html/2607.19326#S4), both attached to a frozen low\-rank adapter\. MaRA restricts the language model’s context to the top\-kkretrieved segments before generation\. Training and evaluation follow the setup of Sections[3](https://arxiv.org/html/2607.19326#S3)and[4](https://arxiv.org/html/2607.19326#S4)on MuSiQue and 2WikiMultihopQA\. All three backbones from Section[3](https://arxiv.org/html/2607.19326#S3)\(Qwen\-2\.5\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\) remain frozen\. All adapters share the canonical training pool described in Appendix[B](https://arxiv.org/html/2607.19326#A2)\. Evaluation uses full\-context decoding for baselines and MaRA top\-kkfor our system\. The number of retrieved segmentskkis fixed once per dataset by a recall criterion on the validation supporting\-paragraph labels, not by end\-task F1, and is then held fixed across all backbones and methods\. For 2WikiMultihopQA \(1010candidates\) the defaultk=4k\{=\}4already attains near\-complete supporting\-paragraph recall \(≈0\.99\\approx 0\.99across backbones\)\. MuSiQue has2020candidates and harder multi\-hop retrieval; MaRA’s recall@kkon the Qwen and Llama validation splits is0\.810\.81/0\.800\.80atk=4k\{=\}4and0\.920\.92/0\.930\.93atk=8k\{=\}8\(both below0\.950\.95\), and we therefore setk=12k\{=\}12, the smallest value exceeding0\.950\.95recall on both \(0\.970\.97\)\. The samekkis held fixed for Gemma, whose router \(safer training schedule, Appendix[B](https://arxiv.org/html/2607.19326#A2)\) yields lower recall than Qwen/Llama; the per\-backbone breakdown is in Appendix[K](https://arxiv.org/html/2607.19326#A11)\. End\-task scores are exact match \(EM\) and F1\.

Table 4:Main results and component decomposition\.EM / F1 across two multi\-hop reasoning datasets and three frozen backbones\. Baselines use the full candidate set; all retrieval\-adapter rows use top\-kkretrieved paragraphs \(k=12k\{=\}12on MuSiQue,k=4k\{=\}4on 2WikiMultihopQA, fixed by validation recall criterion\)\. Best per column in bold\. The LoRA\+\+BM25, LoRA\+\+BGE \(fine\-tuned\), LoRA\+\+MaRA, and TopLoRA\+\+MaRA rows isolate the evidence\-selection contribution under different adapter choices\. The MaLoRA\+\+oracle row uses gold supporting paragraphs as an upper bound\.MuSiQue \(EM / F1\)2WikiMultihopQA \(EM / F1\)MethodQwenLlamaGemmaQwenLlamaGemmaNo adapterBase, full context5\.1/14\.45\.1/14\.46\.0/17\.46\.0/17\.413\.8/25\.313\.8/25\.315\.3/26\.015\.3/26\.020\.3/31\.920\.3/31\.917\.9/34\.217\.9/34\.2Static adapter baselinesLoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)38\.4/51\.138\.4/51\.141\.8/54\.441\.8/54\.445\.4/59\.845\.4/59\.871\.7/77\.371\.7/77\.372\.9/78\.572\.9/78\.571\.5/77\.271\.5/77\.2DoRA\(Liuet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib5)\)37\.4/51\.737\.4/51\.741\.7/56\.041\.7/56\.043\.2/58\.243\.2/58\.270\.7/77\.370\.7/77\.374\.6/80\.474\.6/80\.470\.7/76\.970\.7/76\.9AdaLoRA\(Zhanget al\.,[2023](https://arxiv.org/html/2607.19326#bib.bib4)\)39\.3/53\.939\.3/53\.944\.4/59\.744\.4/59\.743\.5/60\.743\.5/60\.768\.0/73\.768\.0/73\.773\.5/78\.973\.5/78\.975\.3/80\.475\.3/80\.4Token\-level modulated adaptersTopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)41\.5/55\.641\.5/55\.647\.3/61\.447\.3/61\.443\.8/59\.543\.8/59\.573\.0/77\.973\.0/77\.975\.1/80\.475\.1/80\.472\.9/79\.372\.9/79\.3MaLoRA42\.6/56\.4\\mathbf\{42\.6/56\.4\}48\.1/62\.3\\mathbf\{48\.1/62\.3\}48\.7/63\.7\\mathbf\{48\.7/63\.7\}73\.7/79\.3\\mathbf\{73\.7/79\.3\}78\.1/83\.3\\mathbf\{78\.1/83\.3\}77\.1/82\.2\\mathbf\{77\.1/82\.2\}Modulator\+\+Retrieval AdapterLoRA\+\+BM2528\.2/40\.528\.2/40\.529\.6/41\.129\.6/41\.134\.1/45\.834\.1/45\.851\.3/56\.351\.3/56\.351\.5/56\.651\.5/56\.650\.0/55\.450\.0/55\.4LoRA\+\+BGE\-large \(FT\)\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib47)\)38\.1/50\.838\.1/50\.840\.6/54\.040\.6/54\.046\.9/59\.046\.9/59\.075\.2/80\.875\.2/80\.874\.6/80\.474\.6/80\.466\.2/73\.266\.2/73\.2LoRA\+\+MaRA43\.3/57\.143\.3/57\.144\.2/57\.944\.2/57\.946\.2/61\.646\.2/61\.676\.1/81\.776\.1/81\.774\.9/81\.674\.9/81\.674\.0/80\.074\.0/80\.0TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)\+\+MaRA45\.3/59\.945\.3/59\.946\.6/61\.546\.6/61\.544\.8/62\.044\.8/62\.077\.5/83\.077\.5/83\.075\.6/82\.175\.6/82\.174\.4/80\.574\.4/80\.5MaLoRA\+\+MaRA46\.0/60\.4\\mathbf\{46\.0/60\.4\}47\.2/62\.4\\mathbf\{47\.2/62\.4\}47\.6/63\.8\\mathbf\{47\.6/63\.8\}78\.0/83\.5\\mathbf\{78\.0/83\.5\}78\.8/85\.3\\mathbf\{78\.8/85\.3\}77\.5/84\.0\\mathbf\{77\.5/84\.0\}Modulator\+\+OracleMaLoRA\+\+*oracle gold evidence*52\.6/67\.552\.6/67\.557\.0/71\.257\.0/71\.257\.2/72\.857\.2/72\.878\.4/83\.478\.4/83\.480\.9/86\.080\.9/86\.079\.2/84\.379\.2/84\.3Table[4](https://arxiv.org/html/2607.19326#S5.T4)reports the full system against the LoRA baseline across the grid\. The MaLoRA\+\+MaRA system improves over LoRA on every cell, by\+4\.0\+4\.0to\+9\.3\+9\.3F1: the largest margin is on MuSiQue Qwen \(51\.1→60\.451\.1\\to 60\.4,\+9\.3\+9\.3\) and the smallest on MuSiQue Gemma \(59\.8→63\.859\.8\\to 63\.8,\+4\.0\+4\.0\); on 2WikiMultihopQA the gains are\+6\.2\+6\.2/\+6\.8\+6\.8/\+6\.8\+6\.8F1 across Qwen/Llama/Gemma\. The mechanism analyses for the two components are in Section[3](https://arxiv.org/html/2607.19326#S3)\(modulator alone, Table[1](https://arxiv.org/html/2607.19326#S3.T1)\) and Section[4\.1](https://arxiv.org/html/2607.19326#S4.SS1)\(MaRA architecture, Table[3](https://arxiv.org/html/2607.19326#S4.T3)\)\.

#### The two mechanisms are complementary across task difficulty\.

The component\-decomposition rows \(LoRA\+\+MaRA for retrieval alone; the modulator\-only columns of Table[1](https://arxiv.org/html/2607.19326#S3.T1)\) show how the two mechanisms combine\. On MuSiQue, the harder dataset \(2020candidate paragraphs, multi\-hop chains up to length44\), both mechanisms contribute substantially: modulation alone lifts LoRA by\+5\.3\+5\.3F1 \(Qwen,51\.1→56\.451\.1\\to 56\.4\) and retrieval alone by\+6\.0\+6\.0\(→57\.1\\to 57\.1\); the full system \(60\.460\.4,\+9\.3\+9\.3over LoRA\) captures most of the sum but with approximately18%18\\%overlap, indicating the two paths share some signal\. The smallest full\-system gain is on MuSiQue Gemma \(\+4\.0\+4\.0F1\); Gemma’s hidden states yield a less discriminating router than Qwen/Llama on this dataset \(recall0\.930\.93vs\.0\.970\.97atk=12k\{=\}12, Appendix[K](https://arxiv.org/html/2607.19326#A11)\), so retrieval contributes less of the lift\. On 2WikiMultihopQA, the easier dataset \(only1010candidates, recall≈0\.99\\approx 0\.99atk=4k\{=\}4\), retrieval leads: retrieval alone lifts LoRA by\+4\.4\+4\.4F1 \(Qwen,77\.3→81\.777\.3\\to 81\.7\), exceeding modulation alone \(\+2\.0\+2\.0F1,→79\.3\\to 79\.3\)\. MaRA’s effectiveness is bounded by how well the frozen backbone already discriminates evidence, which is easiest precisely when reasoning is shallow; the modulator operates at the weight level and is unaffected by retrieval difficulty\. The full system is robust because the two mechanisms contribute through different channels and their gains largely accumulate, with some overlap \(approximately18%18\\%on the hardest cell\)\. Although TopLoRA uses a more expressive per\-rank diagonal gate and MaLoRA uses a scalar gate, MaLoRA still improves consistently in both standalone \(Table[1](https://arxiv.org/html/2607.19326#S3.T1)\) and combined \(Table[4](https://arxiv.org/html/2607.19326#S5.T4)\) settings, suggesting that recurrent state can compensate for lower per\-token gate dimensionality\.

#### Component decomposition \(MuSiQue Qwen\)\.

On the cell with the most headroom, MaRA beats BM25 by\+16\.6\+16\.6F1 \(57\.157\.1vs40\.540\.5\), confirming a large gain over classical sparse retrieval\. The gap from MaLoRA\+\+MaRA \(60\.460\.4F1\) to MaLoRA\+\+oracle \(67\.567\.5F1,\+7\.1\+7\.1F1\) shows that retrieval remains the main bottleneck on MuSiQue, even after model\-attached retrieval improves over BM25\.

#### Efficiency\.

Parameter overhead, training time, peak memory, and inference latency are detailed in Appendix[C](https://arxiv.org/html/2607.19326#A3)\(Table[5](https://arxiv.org/html/2607.19326#A3.T5)\)\.

## 6Conclusion

A family of selective state\-space adapters was introduced for frozen language models, operating at the token level \(MaLoRA\) and at the context\-segment level \(MaRA\)\. Across three backbones and two multi\-hop reasoning datasets, the combined system improved over LoRA on every cell of the3×23\{\\times\}2grid, with the two components contributing through complementary channels: at the segment level when the backbone can already discriminate evidence, and at the token level when reasoning depth gives the modulator more to act on\.

Three directions extend this work\.\(i\)The hierarchical recipe is expected to transfer to reasoning tasks beyond multi\-hop QA \(math, code\) with a task\-appropriate notion of segment\.\(ii\)Multi\-query settings \(conversational agents, tool\-use loops\) have an explicit per\-turn state and are a natural fit for MaRA, with the turn as the unit of state evolution\.\(iii\)MaRA was used here only for segment selection; a natural extension is to inject its per\-segment state directly into the generator through steering vectors or activation patching at chosen layers\.

## Limitations

Evaluation in this work is restricted to multi\-hop reasoning question answering\. Mathematical reasoning was attempted with a solution\-step variant of the retrieval adapter and did not yield a clean lift; this is reported here as future work rather than a generalisation claim\. Other reasoning settings, including code generation and conversational agents with evolving state, are similarly out of scope of the present empirical evaluation\.

The backbones used here span the77–99B parameter range\. The granularity hypothesis is not tested on substantially smaller backbones \(below33B\) or substantially larger backbones \(above3030B\)\. It is possible that the relative contributions of the modulator and the retrieval adapter change with backbone capacity; this remains to be measured\.

The retrieval adapter is trained with supervised binary labels over the supporting paragraphs for each example\. Settings without such labels, or with noisy labels, are out of scope\. Distantly supervised or self\-supervised variants of the retrieval objective would be needed to extend the method to such settings\.

The training schedule of both the adapter and the retrieval adapter is sensitive to the choice of backbone\. Gemma\-2\-9B required a smaller learning rate and a tighter gradient\-norm clip than Qwen\-2\.5\-7B and Llama\-3\.1\-8B to train stably\. Practitioners applying this method to a new backbone should expect to retune these two values\. Each main\-result cell consumed approximately1010to2020GPU\-hours of training on a single A100\-class device \(Table[5](https://arxiv.org/html/2607.19326#A3.T5)\)\. With five adapter methods across the3×23\\times 2grid trained at three seeds, the full reported set required on the order of1,0001\{,\}000GPU\-hours, a non\-trivial barrier to seed\-replication at larger scales\.

Training is staged rather than end\-to\-end\. The modulated adapter \(LoRA\+\+MaLoRA\) is trained first on the target task, and MaRA is then trained with the adapter and encoder both frozen\. Joint end\-to\-end optimisation of the two stages is left as future work; we expect modest additional gains from co\-adapting MaRA’s segment selection to MaLoRA’s modulated hidden states, but staged training is a clean fit for the model\-attached design \(MaRA reads frozen hidden states, so its training cost stays low\) and we did not chase the joint\-training variant here\.

## Ethics Statement

All datasets used in this work are publicly released benchmarks for multi\-hop question answering\. No human\-subject data is collected or annotated\. The backbones used \(Qwen\-2\.5\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\) are released under their respective licenses and used in accordance with them\. No known dual\-use concerns arise from the contributions of this paper: the modulator and retrieval adapter are general\-purpose adaptation components and do not enable new harmful capabilities beyond those already present in the underlying language models\. The compute footprint is documented in the Limitations section above\.

#### Use of AI assistants\.

AI\-based writing assistants \(large language model chat assistants\) were used for prose polishing, surfacing inconsistencies between sections, drafting LaTeX scaffolding for tables, and code\-assisted error checking during the experimental pipeline \(e\.g\., auditing eval configurations for mismatched checkpoints\)\. All research design, methodological decisions, experiments, analyses, and final wording remain the responsibility of the human authors\.

## References

- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px4.p1.1)\.
- Transformers are SSMs: generalized models and efficient algorithms through structured state space duality\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 10041–10071\.External Links:[Link](https://proceedings.mlr.press/v235/dao24a.html)Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, W\. Shen, L\. Xiong, Y\. Zhou, X\. Wang, Z\. Xi, X\. Fan, S\. Pu, J\. Zhu, R\. Zheng, T\. Gui, Q\. Zhang, and X\. Huang \(2024\)LoRAMoE: alleviating world knowledge forgetting in large language models via MoE\-style plugin\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1932–1945\.External Links:[Link](https://aclanthology.org/2024.acl-long.106/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.106)Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Gao, K\. Chen, J\. Rao, R\. Liu, B\. Sun, Y\. Zhang, D\. Peng, X\. Guo, and V\. Subrahmanian \(2025\)MoLA: MoE LoRA with layer\-wise expert allocation\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5112–5127\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.284/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.284),ISBN 979\-8\-89176\-195\-7Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Gu and T\. Dao \(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[§I\.1](https://arxiv.org/html/2607.19326#A9.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2607.19326#S1.p3.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.19326#S3.SS2.p1.1),[§4](https://arxiv.org/html/2607.19326#S4.SS0.SSS0.Px1.p4.5)\.
- A\. Gu, K\. Goel, and C\. Ré \(2021\)Efficiently modeling long sequences with structured state spaces\.CoRRabs/2111\.00396\.External Links:[Link](https://arxiv.org/abs/2111.00396),2111\.00396Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,D\. Scott, N\. Bel, and C\. Zong \(Eds\.\),Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://aclanthology.org/2020.coling-main.580/),[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580)Cited by:[Appendix B](https://arxiv.org/html/2607.19326#A2.SS0.SSS0.Px1.p1.5),[§1](https://arxiv.org/html/2607.19326#S1.p5.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2607.19326#S3.SS3.p1.8)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.External Links:2404\.06654,[Link](https://arxiv.org/abs/2404.06654)Cited by:[§1](https://arxiv.org/html/2607.19326#S1.p5.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px4.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.International Conference on Learning Representations \(ICLR\)\.Cited by:[Table 5](https://arxiv.org/html/2607.19326#A3.T5.22.6.5),[§1](https://arxiv.org/html/2607.19326#S1.p1.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2607.19326#S3.SS1.SSS0.Px1.p1.2),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.18.16.4),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.29.27.4),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.7.5.4),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.28.12.7)\.
- Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. Lee \(2023\)LLM\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5254–5276\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.319/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.319)Cited by:[Appendix H](https://arxiv.org/html/2607.19326#A8.p1.2)\.
- G\. Izacard, M\. Caron, L\. Hosseini, S\. Riedel, P\. Bojanowski, A\. Joulin, and E\. Grave \(2021\)Unsupervised dense information retrieval with contrastive learning\.arXiv preprint arXiv:2112\.09118\.Cited by:[Table 10](https://arxiv.org/html/2607.19326#A9.T10.18.10.3),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6769–6781\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§I\.2](https://arxiv.org/html/2607.19326#A9.SS2.p1.7),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1)\.
- O\. Khattab and M\. Zaharia \(2020\)Colbert: efficient and effective passage search via contextualized late interaction over bert\.InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval,pp\. 39–48\.Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1)\.
- P\. Laban, H\. Hayashi, Y\. Zhou, and J\. Neville \(2025\)Llms get lost in multi\-turn conversation\.arXiv preprint arXiv:2505\.06120\.Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Li, X\. Luo, H\. Wang, X\. Tang, Z\. Cui, D\. Liu, Y\. Li, X\. He, and R\. Li \(2025\)Beyond higher rank: token\-wise input\-output projections for efficient low\-rank adaptation\.Advances in Neural Information Processing Systems38,pp\. 136414–136435\.Cited by:[Table 5](https://arxiv.org/html/2607.19326#A3.T5.33.17.1),[Table 7](https://arxiv.org/html/2607.19326#A4.T7.14.6.7),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px2.p1.1),[Figure 1](https://arxiv.org/html/2607.19326#S3.F1.19.3),[Figure 1](https://arxiv.org/html/2607.19326#S3.F1.23.3),[§3\.1](https://arxiv.org/html/2607.19326#S3.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.10.8.4),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.21.19.4),[Table 1](https://arxiv.org/html/2607.19326#S3.T1.32.30.4),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.46.30.7),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.75.59.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024\)Dora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[Table 5](https://arxiv.org/html/2607.19326#A3.T5.32.16.6),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.34.18.7)\.
- J\. Nguyen, S\. Wang, K\. Li, and C\. Wu \(2024\)READ: recurrent adaptation of large transformers\.External Links:2305\.15348,[Link](https://arxiv.org/abs/2305.15348)Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Nogueira and K\. Cho \(2020\)Passage re\-ranking with bert\.External Links:1901\.04085,[Link](https://arxiv.org/abs/1901.04085)Cited by:[§I\.2](https://arxiv.org/html/2607.19326#A9.SS2.p1.7)\.
- S\. E\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: bm25 and beyond\.Found\. Trends Inf\. Retr\.3,pp\. 333–389\.External Links:[Link](https://api.semanticscholar.org/CorpusID:207178704)Cited by:[Table 10](https://arxiv.org/html/2607.19326#A9.T10.10.2.3),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1),[§4\.1](https://arxiv.org/html/2607.19326#S4.SS1.SSS0.Px1.p1.7)\.
- J\. T\. Smith, A\. Warrington, and S\. W\. Linderman \(2022\)Simplified state space layers for sequence modeling\.arXiv preprint arXiv:2208\.04933\.Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Tian, Z\. Shi, Z\. Guo, L\. Li, and C\. Xu \(2024\)Hydralora: an asymmetric lora architecture for efficient fine\-tuning\.Advances in Neural Information Processing Systems37,pp\. 9565–9584\.Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475)Cited by:[Appendix B](https://arxiv.org/html/2607.19326#A2.SS0.SSS0.Px1.p1.5),[§1](https://arxiv.org/html/2607.19326#S1.p5.1),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2607.19326#S3.SS3.p1.8)\.
- L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei \(2022\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.Cited by:[Table 10](https://arxiv.org/html/2607.19326#A9.T10.12.4.3),[Table 10](https://arxiv.org/html/2607.19326#A9.T10.16.8.3),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie \(2024\)C\-pack: packed resources for general chinese embeddings\.InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval,pp\. 641–649\.Cited by:[Table 10](https://arxiv.org/html/2607.19326#A9.T10.20.12.3),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.61.45.1)\.
- J\. Yu, Y\. Zhang, B\. Wang, P\. Lin, Y\. Liu, and S\. Feng \(2025\)SSMLoRA: enhancing low\-rank adaptation with state space model\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4493–4506\.External Links:[Link](https://aclanthology.org/2025.naacl-long.230/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.230),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px3.p1.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)Adalora: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[Table 5](https://arxiv.org/html/2607.19326#A3.T5.27.11.6),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2607.19326#S5.T4.40.24.7)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176,[Link](https://arxiv.org/abs/2506.05176)Cited by:[Table 10](https://arxiv.org/html/2607.19326#A9.T10.14.6.3),[§2](https://arxiv.org/html/2607.19326#S2.SS0.SSS0.Px5.p1.1),[§4\.1](https://arxiv.org/html/2607.19326#S4.SS1.SSS0.Px1.p1.7)\.

## Appendix ASystem overview

Figure[4](https://arxiv.org/html/2607.19326#A1.F4)places the two adapters of Sections[3](https://arxiv.org/html/2607.19326#S3)and[4](https://arxiv.org/html/2607.19326#S4)in a single view\. The frozen backbone is shared\. MaLoRA \(left\) operates at the token level: a Mamba block consumes the hidden statextx\_\{t\}and emits a scalarλ​\(xt\)\\lambda\(x\_\{t\}\)that rescales the LoRA updateB​A​xtBAx\_\{t\}inside every adapted projection, as in Eq\.[3](https://arxiv.org/html/2607.19326#S3.E3)\. MaRA \(right\) operates at the context level: a separate Mamba mixer runs over per\-segment summaries obtained from the firstKKlayers of the same backbone and scores each candidate paragraph, with the top\-kkretained for the answer pass\. The two adapters act on different inputs \(token states vs\. segment summaries\) and produce different outputs \(a per\-token modulation vs\. a per\-segment selection\), so their gains compose rather than overlap \(Section[5](https://arxiv.org/html/2607.19326#S5)\)\.

![Refer to caption](https://arxiv.org/html/2607.19326v1/figures/malora_fig3.png)Figure 4:Two\-level selective state\-space adaptation around a frozen LM\.Token\-level adapter \(MaLoRA, left\) modulates the LoRA scaling factor per token; context\-level adapter \(MaRA, right\) selects query\-relevant input segments\. Same frozen backbone, different granularities, different targets\.
## Appendix BHyperparameter and training details

#### Datasets\.

MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib9)\)\(denoted MQ\) has22\-,33\-, and44\-hop compositional questions over2020candidate paragraphs\. 2WikiMultihopQA\(Hoet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib30)\)\(2Wiki\) has compositional, comparison, inference, and bridge\-comparison questions over1010candidate paragraphs\. The fraction of paragraphs that are supporting is small in both\.

#### Canonical training pool\.

A single shared training pool of10,00010\{,\}000examples per dataset is constructed for fair cross\-backbone comparison\. An example is admitted if the prompt length under both the Qwen and Llama tokenisers is at most the dataset’s sequence\-length budget \(20482048for 2Wiki,40964096for MQ\), and the example must satisfy a chunk\-detection check under both tokenisers\. Remaining examples are sampled with stratification by hop count \(MQ\) or by question type \(2Wiki\) under a fixed random seed of4242\. Every model is trained on the same pool, so cross\-backbone differences are attributable to the model, not to the data subset\.

#### Evaluation splits\.

End\-task evaluation uses the released validation splits\. 2WikiMultihopQA takes the first3,0003\{,\}000examples \(dataset\[:3000\]\)\. MuSiQue uses the full answerable subset of its validation split \(2,4172\{,\}417examples after filtering toanswerable=True\); seeeval\_musique\.pyfor the procedure\. All evaluations use beam search of width44\.

#### Adapter training \(LoRA, modulator\)\.

LoRA rankr=16r\{=\}16; the standard LoRA scaling is absorbed into the initialization ofAAandBB\(Eq\.[1](https://arxiv.org/html/2607.19326#S3.E1)\)\. Adapters are attached to\{q​\_​proj,k​\_​proj,v​\_​proj,up​\_​proj,down​\_​proj\}\\\{q\\\_\\mathrm\{proj\},k\\\_\\mathrm\{proj\},v\\\_\\mathrm\{proj\},\\mathrm\{up\}\\\_\\mathrm\{proj\},\\mathrm\{down\}\\\_\\mathrm\{proj\}\\\}\. Training is22epochs with learning rate2×10−42\{\\times\}10^\{\-4\}, batch size44, gradient\-accumulation22, weight decay0,100100warmup steps, bf16 mixed precision\. The modulated variants \(Section[3](https://arxiv.org/html/2607.19326#S3)\) use the same schedule, loss, and module set as LoRA\. Modulator\-only parameters add approximately54%54\\%\(TopLoRA\) or approximately50%50\\%\(MaLoRA\) over the LoRA parameter count; see Appendix[C](https://arxiv.org/html/2607.19326#A3)\.

#### Adapter training on Gemma\.

Gemma\-2\-9B uses a safer schedule: learning rate1×10−41\{\\times\}10^\{\-4\},max⁡‖g‖2=0\.5\\max\\\|g\\\|\_\{2\}=0\.5,300300warmup steps\. All other settings match the table above\. This applies uniformly to LoRA, DoRA, AdaLoRA, and the modulated variants\.

#### Retrieval adapter training\.

MaRA \(Section[4](https://arxiv.org/html/2607.19326#S4)\) is trained for22epochs with batch size11and gradient\-accumulation44\. Qwen and Llama backbones use learning rate5×10−45\{\\times\}10^\{\-4\}andmax⁡‖g‖2=1\.0\\max\\\|g\\\|\_\{2\}=1\.0\. The chunk encoder uses the firstK=16K\{=\}16transformer layers of the backbone with frozen weights\. The recurrence width isD=256D\{=\}256, attention\-pool dimension is256256, scoring\-head MLP hidden dimension is256256, Mamba state dimds=64d\_\{s\}\{=\}64\. Positive\-class BCE weight is8\.08\.0; pairwise margin loss weight0\.50\.5with margin1\.01\.0\. About3\.13\.1M MaRA parameters per cell\.

#### Retrieval adapter training on Gemma\.

MaRA on Gemma uses the safer schedule: learning rate1×10−41\{\\times\}10^\{\-4\},max⁡‖g‖2=0\.5\\max\\\|g\\\|\_\{2\}=0\.5\. Other settings unchanged\.

#### Seeds and checkpoint selection\.

All cells \(modulators and MaRA\) are trained with three random seeds \(4242,4343,4444\)\. All headline numbers in Table[4](https://arxiv.org/html/2607.19326#S5.T4)and Table[1](https://arxiv.org/html/2607.19326#S3.T1)are means over the three seeds, with no best\-seed selection\. For analyses that operate on a single checkpoint \(the state\-usage probe of Appendix[G](https://arxiv.org/html/2607.19326#A7), the per\-projection / per\-layer marginals of Appendix[E](https://arxiv.org/html/2607.19326#A5), and the per\-token region analysis of Appendix[F](https://arxiv.org/html/2607.19326#A6)\), the seed with the highest validation F1 is used per cell; this is a methodological choice \(these analyses require a specific checkpoint\) and does not affect any reported headline metric\.

## Appendix CEfficiency breakdown: parameters, training time, peak memory, inference latency

This appendix combines the four cost dimensions of each adapter into a single view:\(i\)trainable parameter count,\(ii\)end\-to\-end training time,\(iii\)peak GPU memory during training, and\(iv\)inference latency\. Table[5](https://arxiv.org/html/2607.19326#A3.T5)reports each dimension averaged over the six cells of Table[4](https://arxiv.org/html/2607.19326#S5.T4)\(33backbones×\\times22datasets\); per\-cell numbers are in the supplementary CSV\. Configs for every cell are committed underconfigs/canonical\_pool/\.

#### Adapter training \(modulators against static baselines\)\.

DoRA is the most expensive among the static baselines:its weight\-decomposed reparameterisation takes1\.88×1\.88\\timesthe LoRA training time on average \(20\.320\.3h vs\.10\.810\.8h\), and up to30\+30\+h per cell on MuSiQue Gemma\. AdaLoRA is closer to LoRA at1\.03×1\.03\\times\.Both token modulators are cheaper than DoRA at training:TopLoRA trains at0\.95×0\.95\\timesLoRA time, and MaLoRA at1\.07×1\.07\\timesLoRA time despite adding a Mamba module\. Peak GPU memory is within approximately4%4\\%of LoRA for every method: peak training memory is dominated by the frozen backbone and activations, not by the adapter delta at rankr=16r\{=\}16\.

#### Retrieval adapter \(MaRA\) training\.

MaRA is trained as a separate stage with the language model frozen and adds only the segment\-level mixer and scoring heads \(approximately3\.13\.1M parameters,≈10%\\approx 10\\%of the LoRA adapter\)\. Average training time is≈2\.5\\approx 2\.5h per \(backbone, dataset\) cell, peak GPU memory≈12\\approx 12GB\. Because MaRA reuses the firstK=16K\{=\}16frozen layers of the backbone as encoder and trains only the small mixer on top, both its training time and its memory footprint are well below those of the LoRA\-side adapter\.

#### Inference latency\.

At fixed beam width \(44\) and adapter rank \(r=16r\{=\}16\), inference latency has two components\.

*\(a\) LM forward\.*The modulated language model’s forward pass is dominated by the backbone and the dataset’s input length, not by the adapter choice\. Representative measurements on 2WikiMultihopQA \(mean≈1,000\\approx 1\{,\}000input tokens,n=3000n\{=\}3000, beam44\):0\.700\.70s/sample on Qwen\-2\.5\-7B,0\.620\.62s/sample on Llama\-3\.1\-8B,3\.03\.0s/sample on Gemma\-2\-9B \(Gemma is slower due to logit soft\-capping\)\. Across methods at fixed backbone the variance is below5%5\\%: the modulator adds one scalar matmul per token plus, for MaLoRA, ads=16d\_\{s\}\{=\}16\-state Mamba step, which is negligible relative to the backbone’s matmuls\.

*\(b\) MaRA segment scoring \(pre\-pass\)\.*MaRA re\-encodes the candidate segments through the firstK=16K\{=\}16frozen layers of the backbone, once per iterative pass \(Appendix[J](https://arxiv.org/html/2607.19326#A10)\)\. The dataset’s pass count and candidate count determine the cost:N=2N\{=\}2passes overM=10M\{=\}10candidates on 2WikiMultihopQA,N=4N\{=\}4passes overM=20M\{=\}20candidates on MuSiQue\. Empirically, one scoring pass costs approximately0\.40\.4s/sample on MuSiQue \(M=20M\{=\}20candidates\) and0\.20\.2s/sample on 2WikiMultihopQA \(M=10M\{=\}10\), so with the per\-dataset pass counts the pre\-pass adds roughly1515to25%25\\%to inference on 2WikiMultihopQA and4040to60%60\\%on MuSiQue; on Gemma the relative pre\-pass cost is smaller because the LM forward is more expensive to begin with\. This is the principal inference\-time cost of the system, and the trade\-off behind it \(more passes→\\tobetter retrieval recall on hard multi\-hop, more inference compute\) is reflected in the recall\-vs\-kkcurve in Appendix[K](https://arxiv.org/html/2607.19326#A11)\.

Table 5:Efficiency breakdown\.Trainable parameters, training time \(mean over33backbones×\\times22datasets\), peak GPU memory during training, and average F1 gain over LoRA across the same six cells\. “×\\timesLoRA” is the ratio to the LoRA baseline’s training time on the same six cells\. DoRA’s1\.88×1\.88\\timestraining time is the most expensive baseline; MaLoRA stays within1\.07×1\.07\\timesdespite adding a recurrent state\-space module\. MaRA is trained separately with the language model frozen and adds≈3\.1\\approx 3\.1M parameters and≈2\.5\\approx 2\.5h training per cell\.MethodParams \(M\)Δ\\DeltaLoRATrain \(h\)Train×\\timesLoRAPeak GPU \(GB\)Static adapter baselinesLoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib1)\)30\.730\.7–10\.810\.81\.00×1\.00\\times33\.833\.8AdaLoRA\(Zhanget al\.,[2023](https://arxiv.org/html/2607.19326#bib.bib4)\)46\.146\.1\+50%\+50\\%11\.111\.11\.03×1\.03\\times34\.034\.0DoRA\(Liuet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib5)\)31\.631\.6\+3%\+3\\%20\.3\\mathbf\{20\.3\}1\.88×\\mathbf\{1\.88\\times\}33\.833\.8Token modulators\+\+TopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)47\.447\.4\+54%\+54\\%10\.310\.30\.95×0\.95\\times35\.235\.2\+\+MaLoRA46\.246\.2\+50%\+50\\%11\.611\.61\.07×1\.07\\times34\.334\.3Retrieval adapter \(separate stage, language model frozen\)\+\+MaRA \(alone\)3\.13\.1\+10%\+10\\%2\.52\.50\.23×0\.23\\times12\.012\.0The combined picture: among static baselines, MaLoRA matches the training cost of LoRA and AdaLoRA \(1\.07×1\.07\\timesvs\.1\.03×1\.03\\times\) while delivering a larger F1 gain \(Table[4](https://arxiv.org/html/2607.19326#S5.T4)\), and is1\.76×1\.76\\timesfaster to train than DoRA\. Adding MaRA on top costs a separate2\.52\.5h per cell with a small \(1212GB\) memory footprint, well below the main adapter’s profile\. At inference, adapter choice is essentially free at rank1616and beam44\.

## Appendix DStateless modulator head: activation choice

This appendix isolates the per\-token*stateless*modulator head — the scalarλ​\(xt\)=f​\(ϕ​\(P​xt\)\)\\lambda\(x\_\{t\}\)=f\(\\phi\(Px\_\{t\}\)\)family without the Mamba recurrence — under three activation choices \(softplus, sigmoid, tanh\)\. Table[6](https://arxiv.org/html/2607.19326#A4.T6)reports the matrix under the Table[1](https://arxiv.org/html/2607.19326#S3.T1)protocol, full\-context, on MuSiQue and 2WikiMultihopQA across all three backbones\. This ablation does*not*include the Mamba recurrence; MaLoRA in the main paper is the softplus\-scalar variant with a Mamba block added on top \(Section[3\.2](https://arxiv.org/html/2607.19326#S3.SS2)\)\. Its Table[1](https://arxiv.org/html/2607.19326#S3.T1)numbers are higher than the corresponding stateless row on five of six cells; on Llama 2WikiMultihopQA the stateless variant is marginally higher \(84\.184\.1vs\.83\.383\.3\), within the per\-seed standard deviation \(σ≤0\.9\\sigma\\leq 0\.9for Llama 2WikiMultihopQA in Table[7](https://arxiv.org/html/2607.19326#A4.T7)\)\.

The three activations land within approximately22F1 of each other on every cell, with no consistent winner across backbones\. Softplus is preferred over sigmoid because its unbounded positive range admits both suppression and amplification, the latter empirically observed for MaLoRA on Gemma\-2\-9B \(Table[2](https://arxiv.org/html/2607.19326#S3.T2), marker/question tokens withλ\>1\\lambda\>1\); a sigmoid\-bounded modulator\(λ∈\(0,1\)\)\(\\lambda\\in\(0,1\)\)cannot represent this amplification, and its near\-equal end\-task F1 reflects that on these benchmarks the modulator is suppression\-dominated while the capacity to amplify is exercised mainly by the Gemma cell\. A diagonal output form \(𝝀​\(xt\)∈ℝ\>0r\\boldsymbol\{\\lambda\}\(x\_\{t\}\)\\in\\mathbb\{R\}^\{r\}\_\{\>0\}\) was checked on MuSiQue Qwen \(58\.458\.4F1 sigmoid\-diag,54\.354\.3tanh\-diag vs\.54\.254\.2/55\.255\.2scalar\) and shows no clear win, motivating the simpler scalar form for the behavioural analyses of Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\.

Table 6:Stateless modulator\-head activation ablation\(F1, full context,n=3000n\{=\}3000, beam44; scalar output form\)\. The per\-token head is a linearϕ\\phioverP​xtPx\_\{t\}with no recurrence\.†Tanh on MuSiQue Gemma\-2\-9B fails to learn \(F126\.326\.3vs\.≥63\.2\\geq 63\.2for the other activations\); the same tanh head reaches healthy F1 on 2WikiMultihopQA Gemma \(83\.183\.1\) and on Qwen/Llama\.MuSiQue F12Wiki F1ActivationQwLlGeQwLlGesoftplus55\.155\.161\.761\.763\.263\.278\.578\.584\.184\.181\.881\.8sigmoid54\.254\.263\.063\.064\.564\.579\.079\.083\.983\.983\.783\.7tanh55\.255\.261\.861\.826\.3†\\mathbf\{26\.3\}^\{\\dagger\}78\.478\.483\.083\.083\.183\.1#### Per\-seed variance of the three token modulators\.

All headline numbers in Table[4](https://arxiv.org/html/2607.19326#S5.T4)are computed as means over three random seeds \(4242,4343,4444\): the static baselines \(LoRA, DoRA, AdaLoRA\), the token modulators \(TopLoRA, MaLoRA\), and the modulator\-plus\-MaRA rows\. Table[7](https://arxiv.org/html/2607.19326#A4.T7)reports the corresponding per\-seed standard deviations for the token modulators\. Per\-cell variance is small across all three token modulators \(σ≤2\.6\\sigma\\leq 2\.6F1 in every cell\)\.

Table 7:Per\-seed F1 standard deviation\(over33seeds:4242,4343,4444\)\.MuSiQue2WikiMethodQwLlGeQwLlGeTopLoRA\(Liet al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib48)\)2\.52\.52\.42\.42\.62\.60\.80\.80\.90\.92\.22\.2MaLoRA1\.11\.11\.81\.82\.32\.30\.50\.50\.70\.72\.12\.1

## Appendix EPer\-projection and per\-layer modulation marginals

Figure[5](https://arxiv.org/html/2607.19326#A5.F5)reports the mean ofλ​\(xt\)\\lambda\(x\_\{t\}\)broken down by projection type and by transformer layer for the three MaLoRA\-backbone cells: Qwen\-2\.5\-7B, Llama\-3\.1\-8B, and Gemma\-2\-9B\. The figure is referenced from Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\. Suppression concentrates inv​\_​projv\\\_\\mathrm\{proj\},q​\_​projq\\\_\\mathrm\{proj\}, and the final transformer layer across all three cells, consistent across backbones\.

![Refer to caption](https://arxiv.org/html/2607.19326v1/x2.png)Figure 5:Where the modulator suppresses\.Meanλ​\(xt\)\\lambda\(x\_\{t\}\)by projection type \(left\) and transformer layer \(right\), aggregated over the full MuSiQue validation set\. Three cells: MaLoRA on Qwen\-2\.5\-7B, Llama\-3\.1\-8B, and Gemma\-2\-9B\.
## Appendix FModulation by token role: extended discussion

The per\-region means ofλ​\(xt\)\\lambda\(x\_\{t\}\)reported in Table[2](https://arxiv.org/html/2607.19326#S3.T2)\(main body\) are aggregated over the full MuSiQue validation set, across the three MaLoRA\-backbone cells\. Tokens are tagged as*marker*\(format/separator tokens\),*question*\(the query\),*para\_support*/*para\_distractor*\(supporting vs\. non\-supporting paragraph content\), and*context\-other*\.

Two findings are robust\. First,λ\\lambdaon supporting and distractor paragraphs is statistically identical in every cell\(\|Δ\|≤0\.002\|\\Delta\|\\leq 0\.002\): the modulator does not separate relevant from irrelevant evidence\. This is expected, since identifying supporting evidence requires the multi\-hop reasoning that the modulator precedes; evidence selection is instead handled by MaRA \(Section[4](https://arxiv.org/html/2607.19326#S4)\)\. Second, the modulator instead allocates modulation bytoken role: on Qwen and Llama, structural \(marker\) and query tokens are suppressed more than paragraph content, consistent with the per\-token view in Figure[2](https://arxiv.org/html/2607.19326#S3.F2)\. The exception is MaLoRA on Gemma\-2\-9B, which*amplifies*query and marker tokens \(λ≈1\.2\\lambda\\approx 1\.2–1\.71\.7\) rather than suppressing them, a backbone\-specific behavior\. The shared pattern \(relevance\-blind, role\-sensitive\) explains why the modulator and MaRA are non\-redundant: the modulator reshapes the low\-rank update by token role and cannot perform evidence selection, so MaRA contributes an orthogonal capability\.

## Appendix GState\-usage probe: per\-cell results

This appendix expands the state\-usage check from Section[3\.4](https://arxiv.org/html/2607.19326#S3.SS4)\. For each Mamba block in a trained MaLoRA cell, the block’s full stateful outputzt=Mamba​\(P​x1:t\)tz\_\{t\}=\\mathrm\{Mamba\}\(Px\_\{1:t\}\)\_\{t\}is compared against a stateless variantztstatelessz\_\{t\}^\{\\mathrm\{stateless\}\}in which each position is processed independently with the recurrent state zeroed\. The relative discrepancy is

ρt=‖zt−ztstateless‖2‖zt‖2\.\\rho\_\{t\}\\;=\\;\\frac\{\\bigl\\\|z\_\{t\}\-z\_\{t\}^\{\\mathrm\{stateless\}\}\\bigr\\\|\_\{2\}\}\{\\\|z\_\{t\}\\\|\_\{2\}\}\.A value ofρt≈0\\rho\_\{t\}\\approx 0at every position would indicate that the recurrent state contributes nothing and that the Mamba block could be replaced by a stateless head;ρt\>0\\rho\_\{t\}\>0means the state genuinely accumulates information up to positiontt\. The probe is run on55validation samples and1010randomly sampled MambaBlocks per cell, on the six canonical MaLoRA cells, using the locked best\-seed checkpoints from the manifest\. Per\-cell meanρt\\rho\_\{t\}at sampled positionst∈\{1,10,102,103\}t\\in\\\{1,10,10^\{2\},10^\{3\}\\\}is in Table[8](https://arxiv.org/html/2607.19326#A7.T8)\. The aggregate across the six cells rises from0\.130\.13at position11to0\.610\.61–0\.630\.63across the rest of the prompt, with every cell above0\.450\.45fromt=10t\{=\}10onward; the recurrence carries useful trajectory\-level information rather than acting decoratively\.

Table 8:State\-usage probe: meanρt\\rho\_\{t\}by token position\.Averaged over1010sampled Mamba blocks and55validation prompts; largerρt\\rho\_\{t\}= more state contribution\.DSBBt=1t\{=\}1t=10t\{=\}10t=102t\{=\}10^\{2\}t=103t\{=\}10^\{3\}MQQwen0\.180\.180\.550\.550\.560\.560\.620\.62MQLlama0\.100\.100\.600\.600\.610\.610\.650\.65MQGemma0\.090\.090\.760\.760\.690\.690\.610\.612WikiQwen0\.230\.230\.590\.590\.640\.640\.560\.562WikiLlama0\.050\.050\.470\.470\.530\.530\.520\.522WikiGemma0\.140\.140\.710\.710\.780\.780\.780\.78*Mean \(6 cells\)*0\.130\.130\.610\.610\.630\.630\.620\.62
## Appendix HCommonsense reasoning

Commonsense reasoning is a complementary single\-context testbed: each example is a self\-contained multiple\-choice question, with no candidate paragraphs and no multi\-hop chain to retrieve\. MaRA does not apply and is omitted; only the token\-level modulators are compared to LoRA\. Adapters are trained on the standard10,00010\{,\}000\-example Commonsense suite\(Huet al\.,[2023](https://arxiv.org/html/2607.19326#bib.bib39)\)and use the same rankr=16r\{=\}16and schedule as the main experiments\. Table[9](https://arxiv.org/html/2607.19326#A8.T9)reports the mean accuracy over eight multiple\-choice tasks \(BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC\-easy, ARC\-challenge, OpenBookQA\)\.

Table 9:Commonsense accuracy\(%, mean over eight multiple\-choice tasks\)\. Single\-context; MaRA omitted\. Best non\-base per column in bold\.MethodQwenLlamaGemmaBase77\.177\.167\.667\.668\.768\.7LoRA88\.388\.380\.680\.686\.6\\mathbf\{86\.6\}TopLoRA86\.886\.879\.779\.784\.584\.5MaLoRA88\.5\\mathbf\{88\.5\}83\.3\\mathbf\{83\.3\}86\.286\.2On Llama\-3\.1\-8B MaLoRA improves over LoRA \(\+2\.7\+2\.7\) while TopLoRA is slightly below \(−0\.9\-0\.9\); on Qwen\-2\.5\-7B MaLoRA is level with LoRA \(\+0\.2\+0\.2\) and TopLoRA is below \(−1\.5\-1\.5\); on Gemma\-2\-9B all three fine\-tuned variants are tightly clustered \(LoRA86\.686\.6, TopLoRA84\.584\.5, MaLoRA86\.286\.2; all\+15\.8\+15\.8to\+17\.9\+17\.9over the68\.768\.7base\)\. Across backbones MaLoRA wins on average \(86\.086\.0vs\. LoRA85\.285\.2vs\. TopLoRA83\.783\.7\)\. Commonsense reasoning is largely saturated for these backbones once fine\-tuned, so per\-cell differences are small relative to the multi\-hop gains in Table[4](https://arxiv.org/html/2607.19326#S5.T4)and consistent with little headroom for token\-level reshaping in the absence of multi\-paragraph distractor structure\.

## Appendix IMaRA architecture details

This appendix expands the condensed Section[4](https://arxiv.org/html/2607.19326#S4)with per\-stage discussion, the architecture\-ablation deep dive, and the full classical\-retrieval comparison\.

### I\.1Per\-stage notes

#### Segment encoder\.

The frozen encoder is the same backbone used downstream, truncated to its firstK=16K\{=\}16layers\. Each segmentCiC\_\{i\}is concatenated with the queryQQbefore encoding, so the resulting hidden states are already query\-conditioned\. No new encoder parameters are introduced\. Encoding each \(query, segment\) pair independently is the standard cross\-encoder pattern\.

#### QQ\-driven attention pool\.

The attention pool of Eq\.[5](https://arxiv.org/html/2607.19326#S4.E5)reduces each segment’sTiT\_\{i\}\-step hidden\-state sequence to a single embeddingpip\_\{i\}via additive attention with a learned query vectorqaq\_\{a\}\. An ablation replacing the learned attention pool by a plain mean is reported in Table[3](https://arxiv.org/html/2607.19326#S4.T3)as MeanPool\.

#### Mamba over segments\.

The two\-layer selective state\-space module\(Gu and Dao,[2024](https://arxiv.org/html/2607.19326#bib.bib2)\)of Eq\.[6](https://arxiv.org/html/2607.19326#S4.E6)follows the same SSM update as Eq\.[4](https://arxiv.org/html/2607.19326#S3.E4), but the recurrence runs over*segments*rather than tokens\. This is the key architectural choice and is ablated against TokenMamba \(token\-level Mamba\) and Transformer \(segment\-level transformer\) in Table[3](https://arxiv.org/html/2607.19326#S4.T3)\.

#### Scoring heads\.

Eq\.[7](https://arxiv.org/html/2607.19326#S4.E7)uses two linear heads: a local head𝐰ℓ\\mathbf\{w\}\_\{\\ell\}that scores each segment from its own recurrent statesis\_\{i\}\(used for top\-kkselection\), and a global headWgW\_\{g\}that reads the final stateg=sNg=s\_\{N\}and predicts allNNsupporting labels jointly as an auxiliary signal\. There is no per\-segment MLP combiner; global context reaches each score through the causal recurrence \(eachsis\_\{i\}depends on segments11throughii\) and through the shared parameters that the global\-head objective conditions\. The recurrence’s contribution is isolated by the PoolOnly ablation \(mixer removed\) and by the MLP ablation \(per\-segment MLP without recurrence\) in Table[3](https://arxiv.org/html/2607.19326#S4.T3)\.

#### Auxiliary supervision\.

The global\-head term in Eq\.[8](https://arxiv.org/html/2607.19326#S4.E8)\(γ=0\.5\\gamma\{=\}0\.5\) is an auxiliary BCE over allNNparagraphs, predicted jointly from the final stateg=sNg=s\_\{N\}\. It shapes the shared recurrence but does not feed selection, and its contribution to final recall is small\.

#### Composition with the modulator\.

MaRA operates before the modulated language model’s forward pass; the modulator \(Section[3](https://arxiv.org/html/2607.19326#S3)\) operates inside it\. The two share only the language model’s hidden states as a representation substrate\. Training is staged: the modulated adapter is trained first on the target task; MaRA is then trained with the adapter and encoder both frozen\.

### I\.2Architecture ablation \(full analysis\)

The Mamba over segments wins on every cell of Table[3](https://arxiv.org/html/2607.19326#S4.T3)\. The R@44gap to the second\-best architecture \(MeanPool, which keeps segment\-level Mamba but drops the attention pool\) is77pp on both Qwen and Llama for MuSiQue\. TokenMamba uses the same Mamba module as MaRA but applied to the concatenated token sequence; its R@44is1919to2929pp below the segment\-level Mamba on MuSiQue, isolating the contribution of placing the recurrence at the segment level\. The encoder follows the standard cross\-encoder pattern, encoding each \(query, segment\) pair independently, with a trained Mamba mixer on top of the per\-segment embeddings carrying the cross\-segment interaction\(Karpukhinet al\.,[2020](https://arxiv.org/html/2607.19326#bib.bib45); Nogueira and Cho,[2020](https://arxiv.org/html/2607.19326#bib.bib46)\)\. On 2WikiMultihopQA, where the candidate set is smaller and the supporting evidence is less distractor\-rich, every architecture beyond MLP and LastTok reaches≥90%\\geq 90\\%R@44and the differences between architectures are compressed, but the Mamba still leads on every cell\.

### I\.3Scoring\-head ablation: linear vs\. MLP

The local scoring head \(Eq\.[7](https://arxiv.org/html/2607.19326#S4.E7)\) is a single linear map𝐰ℓ\\mathbf\{w\}\_\{\\ell\}\. Replacing it with a two\-layer MLP \(Linear→GELU→Linear\\mathrm\{Linear\}\\\!\\to\\\!\\mathrm\{GELU\}\\\!\\to\\\!\\mathrm\{Linear\}, hidden256256\), with the Mamba mixer and all other settings fixed, does not help: Recall@44is78\.678\.6vs\.80\.680\.6\(MuSiQue Qwen\),83\.283\.2vs\.84\.184\.1\(MuSiQue Llama\),99\.499\.4vs\.99\.299\.2\(2WikiMultihopQA Qwen\), and99\.499\.4vs\.99\.599\.5\(2WikiMultihopQA Llama\)\. The MLP head is marginally worse on MuSiQue and indistinguishable on 2WikiMultihopQA\. The recurrent statesis\_\{i\}already integrates cross\-segment context, so a linear read\-out suffices; the extra head capacity adds parameters without improving retrieval\. The linear head is used throughout\.

### I\.4Retrieval baselines \(full comparison\)

MaRA is compared against a2×22\{\\times\}2grid of alternatives, crossing*outsourced*\(a separate encoder over raw text\) versus*model\-attached*\(reusing the frozen backbone’s own hidden states\), and*untrained*\(off\-the\-shelf or cosine similarity\) versus*trained*\(fine\-tuned on the supporting\-paragraph supervision\)\. Table[10](https://arxiv.org/html/2607.19326#A9.T10)reports Recall@44on the samen=500n\{=\}500validation split and gold labels for every method\.

Table 10:Retrieval comparison\(Recall@44,n=500n\{=\}500validation, identical examples and gold labels\)\. Outsourced methods run a separate encoder over raw text; model\-attached methods score from the frozen backbone’s own hidden states\. Trained = fine\-tuned on the supporting\-paragraph labels\. Best per column in bold\. MaRA beats every outsourced baseline, including the88B Qwen3\-Embedding encoder, while adding only≈3\\approx 3M parameters and no separate encoder\.MethodMuSiQue2WikiMultihopQAOutsourced, off\-the\-shelf \(untrained\)BM25 \(sparse\)\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.19326#bib.bib44)\)47\.047\.066\.466\.4E5\-large\-v2\(Wanget al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib43)\)69\.169\.183\.783\.7Qwen3\-Embedding\-8B\(Zhanget al\.,[2025](https://arxiv.org/html/2607.19326#bib.bib40)\)75\.575\.583\.583\.5Outsourced, trained \(dense bi\-encoder, best of LR sweep\)E5\-large\-v2 \(fine\-tuned\)\(Wanget al\.,[2022](https://arxiv.org/html/2607.19326#bib.bib43)\)63\.363\.397\.197\.1Contriever \(fine\-tuned\)\(Izacardet al\.,[2021](https://arxiv.org/html/2607.19326#bib.bib42)\)68\.368\.396\.996\.9BGE\-large\-en \(fine\-tuned\)\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.19326#bib.bib47)\)68\.268\.297\.797\.7Model\-attached, untrained \(frozen\-LM cosine\)Frozen\-LM cosine \(Qwen\-2\.5\-7B\)37\.737\.754\.954\.9Frozen\-LM cosine \(Llama\-3\.1\-8B\)48\.448\.448\.248\.2Model\-attached, trained \(ours\)MaRA \(Qwen\-2\.5\-7B\)80\.680\.699\.299\.2MaRA \(Llama\-3\.1\-8B\)84\.1\\mathbf\{84\.1\}99\.5\\mathbf\{99\.5\}Three readings\. First, MaRA wins every cell: on MuSiQue it reaches80\.680\.6\(Qwen\) and84\.184\.1\(Llama\), above the strongest outsourced baseline, the88B Qwen3\-Embedding encoder \(75\.575\.5\), despite using no separate encoder and only≈3\\approx 3M new parameters\. Second, training a*separate*dense bi\-encoder on the supporting\-paragraph labels is not enough: across three encoders \(E5, Contriever, BGE\-large\-en\), fine\-tuning lifts 2WikiMultihopQA to96\.996\.9–97\.797\.7but plateaus on MuSiQue at63\.363\.3–68\.368\.3, well below MaRA \(80\.680\.6–84\.184\.1\)\. For E5 the MuSiQue fine\-tune even*degrades*relative to its own zero\-shot \(69\.1→63\.369\.1\\to 63\.3\), a catastrophic\-forgetting effect when a general\-purpose encoder is specialised on a small label set\. The MuSiQue ceiling for text\-only retrieval reflects that the bridge entity is absent from the surface query, which a separate encoder cannot recover but the backbone’s reasoning hidden states can\. Third, the untrained model\-attached baseline \(raw frozen\-LM cosine\) is weak \(37\.737\.7–54\.954\.9\), so the gain is not simply from reusing the backbone’s hidden states; it comes from the trained segment\-level mixer on top of them\. BM25 R@88/R@1212reach60\.860\.8/71\.571\.5\(MuSiQue\) and86\.286\.2/100100\(2WikiMultihopQA\); Qwen3\-Embedding R@88/R@1212reach88\.988\.9/93\.993\.9and96\.496\.4/100100\.

## Appendix JIterative segment selection

MaRA \(Section[4](https://arxiv.org/html/2607.19326#S4)\) is applied iteratively at inference\. Given queryQQ, candidate segments𝒞=\{C1,…,CM\}\\mathcal\{C\}=\\\{C\_\{1\},\\dots,C\_\{M\}\\\}, a pass budgetNN, and a selection budgetkk, an ordered anchor set is built one segment per pass; each pass conditions the segment scoring on the anchors already chosen \(Algorithm[1](https://arxiv.org/html/2607.19326#alg1)\)\. The two\-pass form \(N=2N\{=\}2\) is used for 2WikiMultihopQA and the four\-pass form \(N=4N\{=\}4\) for MuSiQue, matching the deeper multi\-hop chains of the latter\. During training, the intermediate passes1:N−11\{:\}N\{\-\}1receive an auxiliary loss weight and the final pass weight1\.01\.0\.

Algorithm 1Iterative segment selection1:Input:query

QQ, segments

𝒞=\{C1,…,CM\}\\mathcal\{C\}=\\\{C\_\{1\},\\dots,C\_\{M\}\\\}, passes

NN, budget

kk, adapter

RR
2:Output:ordered segment indices, truncated to

kk
3:

A←\[\]A\\leftarrow\[\\,\]\{ordered anchor list\}

4:for

t=1t=1to

NNdo

5:

𝐫←R​\(𝒞,Q∣A\)\\mathbf\{r\}\\leftarrow R\\big\(\\mathcal\{C\},Q\\mid A\\big\)\{score each segment, conditioned on selected anchors\}

6:

at←arg⁡maxi∉A⁡ria\_\{t\}\\leftarrow\\arg\\max\_\{i\\notin A\}\\,r\_\{i\}
7:

A←A∥\[at\]A\\leftarrow A\\mathbin\{\\\|\}\[\\,a\_\{t\}\\,\]
8:endfor

9:

rest←sort​\(\{i:i∉A\}\)\\mathrm\{rest\}\\leftarrow\\mathrm\{sort\}\\big\(\\\{i:i\\notin A\\\}\\big\)by

rir\_\{i\}descending \{

𝐫\\mathbf\{r\}from the final pass\}

10:return

\(A∥rest\)\[:k\]\(A\\mathbin\{\\\|\}\\mathrm\{rest\}\)\[:k\]

## Appendix KTop\-kkselection sweep

The headlinekkper dataset is fixed by a validation recall criterion \(Section[5](https://arxiv.org/html/2607.19326#S5)\): the smallestkkwhose MaRA recall over the validation supporting\-paragraph labels exceeds0\.950\.95on Qwen and Llama\. The samekkis then held fixed for every backbone and method\. Per\-backbone MaRA recall@kkis shown in Table[11](https://arxiv.org/html/2607.19326#A11.T11)\. On MuSiQue, Qwen and Llama cross the threshold atk=12k\{=\}12\(recall0\.970\.97\); Gemma’s router \(safer training schedule, Appendix[B](https://arxiv.org/html/2607.19326#A2)\) is lower across allkkbut the samek=12k\{=\}12is kept for cross\-backbone consistency\. On 2WikiMultihopQA every backbone reaches≈0\.99\\approx 0\.99recall atk=4k\{=\}4\.

Table 11:MaRA supporting\-paragraph recall@kkby backbone\(validation split\)\. Threshold0\.950\.95is satisfied atk=12k\{=\}12on Qwen and Llama for MuSiQue, and atk=4k\{=\}4on every backbone for 2WikiMultihopQA\. The samekkis held fixed for Gemma\.MuSiQue R@kk2Wiki R@kkBackbonek=4k\{=\}4k=8k\{=\}8𝐤=𝟏𝟐\\mathbf\{k\{=\}12\}𝐤=𝟒\\mathbf\{k\{=\}4\}k=8k\{=\}8Qwen0\.810\.810\.920\.920\.97\\mathbf\{0\.97\}0\.99\\mathbf\{0\.99\}1\.001\.00Llama0\.800\.800\.930\.930\.97\\mathbf\{0\.97\}0\.99\\mathbf\{0\.99\}1\.001\.00Gemma0\.680\.680\.840\.840\.93\\mathbf\{0\.93\}0\.99\\mathbf\{0\.99\}1\.001\.00