PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

# Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry
Source: [https://arxiv.org/html/2606.11262](https://arxiv.org/html/2606.11262)
Gowtham Sivaramakrishnan Independent Researcher gowtham\_s@berkeley\.edu&Sarvesha Kumar Kombaiah Seetha Independent Researcher skombaiah@my\.harrisburgu\.eduKishan Gupta Balaji Independent Researcher kbalaji@my\.harrisburgu\.edu&Santhosh Baradwaj Vaduvur Ranganathan Independent Researcher vaduvurr@usc\.edu

###### Abstract

Access control in large language models \(LLMs\) requires modular mechanisms to enable domain\-specific behavior without retraining or cross\-domain interference\. A common hypothesis is that interference during adapter composition arises from overlap in linear parameter updates, suggesting that enforcing orthogonality or directional independence should improve multi\-domain performance\. We test this hypothesis using DoRA\-RBAC, a hierarchical adapter composition framework based on weight\-decomposed low\-rank adaptation\. We compare conventional Euclidean merging with a geometry\-aware Riemannian\-inspired merging strategy that approximates the Fréchet mean via normalized directional averaging across multiple QA benchmarks \(GPQA, PubMedQA, SimpleQA, WMDP\) on LLaMA\-3\.1\-8B and Mistral\-7B\. Our results show that while single\-domain performance matches LoRA, geometry\-aware merging provides no consistent advantage over standard averaging in multi\-domain settings\. Diagnostic analysis further reveals that angular alignment and orthogonality of adapter updates are weak predictors of composition performance\. These findings suggest that adapter interference is not governed primarily by parameter\-space geometry, but is instead consistent with interactions in shared nonlinear representations\.

## 1Introduction

Large language models are increasingly deployed in settings where users have unequal access to domain\-specific knowledge\. Enterprise and scientific systems often combine multiple domains, while individual users are authorized to access only subsets of this information\. This creates a key challenge: enabling modular domain adaptation while supporting controlled composition of multiple domains at inference time\.

Existing access\-control approaches operate at the prompt, routing, or retrieval level\. While effective, they do not enforce modularity at the level of learned representations\. This motivates a parameter\-level perspective, where domain knowledge is stored in separate adapters that can be selectively activated and composed based on user permissions\.

A central question in this setting is what drives interference when multiple adapters are composed\. A common hypothesis is that interference arises from overlap in linear parameter updates, suggesting that enforcing orthogonality or directional separation should improve composition\. Under this view, geometry\-aware merging methods should outperform standard Euclidean averaging\.

We test this hypothesis using DoRA\-RBAC, a hierarchical adapter framework built on weight\-decomposed\(Liu et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib10)\)low\-rank adaptation \(DoRA\), which separates updates into direction and magnitude\. Each domain is represented by a manager adapter on top of a shared baseline, and authorized subsets of domains are composed into a single flattened adapter at inference time\.

This setup enables a controlled comparison between two composition strategies: \(1\) Euclidean merging via averaging and SVD refactorization, and \(2\) a geometry\-aware, Riemannian\-inspired merge that approximates the Fréchet mean\(Karcher,[1977](https://arxiv.org/html/2606.11262#bib.bib7); Shoemake,[1985](https://arxiv.org/html/2606.11262#bib.bib12)\)through normalized directional averaging with separate magnitude aggregation\. If interference is primarily driven by linear overlap, the geometry\-aware approach should provide a consistent advantage\.

Our results do not support this hypothesis\. Across multiple QA benchmarks on Llama\-3\.1\-8B\(Grattafiori et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib2)\)and Mistral\-7B\(Jiang et al\.,[2023](https://arxiv.org/html/2606.11262#bib.bib5)\), geometry\-aware merging provides no consistent improvement over Euclidean merging in multi\-domain settings\. While angular alignment explains when geometry\-aware merging collapses to Euclidean behavior, it does not predict overall composition performance\. These findings indicate that enforcing orthogonality in parameter space is insufficient to resolve adapter interference and are instead consistent with interaction effects in shared nonlinear representations\.

Finally, we evaluate implications for access control\. Domain\-specific adapters improve target\-domain performance \(e\.g\., \+0\.1178 UGI on SimpleQA\(Wei et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib13)\), but distinguishability remains above chance DDI AUC≈\\approx0\.65, indicating that modular composition does not provide strong privacy guarantees\. DoRA\-RBAC should therefore be viewed as a modular composition framework rather than a mechanism for strict isolation\.

## 2Related work

Adapters, prefix tuning\(Li & Liang,[2021](https://arxiv.org/html/2606.11262#bib.bib9)\), and LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2606.11262#bib.bib3)\)enable task\-specific fine\-tuning by updating a small fraction of parameters\. DoRA\(Liu et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib10)\)extends LoRA by decomposing updates into direction and magnitude, enabling finer\-grained analysis of adapter interactions during composition\.

Model soups\(Wortsman et al\.,[2022](https://arxiv.org/html/2606.11262#bib.bib14)\)and TIES\-Merging\(Yadav et al\.,[2023](https://arxiv.org/html/2606.11262#bib.bib15)\)combine fine\-tuned models in a shared parameter space\. Our setting is more constrained: we compose domain\-specific low\-rank updates selected by user access rights into a single adapter at inference time\. Unlike prior merging work, composition is governed by an explicit access\-control policy rather than task similarity\.

PermLLM\(Jayaraman et al\.,[2025](https://arxiv.org/html/2606.11262#bib.bib4)\)formalizes access\-control enforcement over LLM responses; AdapterSwap\(Fleshman et al\.,[2025](https://arxiv.org/html/2606.11262#bib.bib1)\)removes adapters based on access policies\. Differentially private fine\-tuning\(Yu et al\.,[2022](https://arxiv.org/html/2606.11262#bib.bib16)\)bounds training\-data leakage but does not address modular composition\. Our work bridges these areas: we modularize domain adaptations and assess distinguishability via membership\-inference metrics rather than providing formal privacy guarantees\.

## 3Method

### 3\.1Problem Setting and Hierarchical DoRA\-RBAC

We freeze a pretrained backbone and train a shared baseline adapter\. For each domaindd, a manager adapter stores an incremental update to this baseline\. Following DoRA LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2606.11262#bib.bib3)\),we decompose each manager’s update into a unit\-direction vector and scalar magnitude \(Eq\. 1\), rather than the standard LoRA parameterizationΔ​W\{\\Delta W\}=B​ABA\. At inference time, only the manager adapters corresponding to a user’s authorized domains are composed into a single flattened adapter\.

In the final component of our framework DoRA\-RBAC, each domain\-specific manager adapter stores a unique update to the common baseline adapter, while the common adapter serves as the repository for all shared domains\. When a user accesses an authorized domain for inference, only the manager update of this user’s domains will be used to generate an output from that manager\. This composite output will be constructed using a single adapter for all manager outputs\.

V^i=Δ​Wi‖Δ​Wi‖F,Δ​Wi=Bi​Ai,Δ​WiDoRA=mi⋅V^i,mi=‖Δ​Wi‖F\\hat\{V\}\_\{i\}=\\frac\{\\Delta W\_\{i\}\}\{\\\|\\Delta W\_\{i\}\\\|\_\{F\}\},\\qquad\\Delta W\_\{i\}=B\_\{i\}A\_\{i\},\\qquad\\Delta W\_\{i\}^\{\\mathrm\{DoRA\}\}=m\_\{i\}\\cdot\\hat\{V\}\_\{i\},\\quad m\_\{i\}=\\\|\\Delta W\_\{i\}\\\|\_\{F\}\(1\)whereV^​i∈ℝd​out×din\\hat\{V\}i\\in\\mathbb\{R\}^\{d\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}is the normalized direction of the low\-rank update andmi∈ℝ\>0m\_\{i\}\\in\\mathbb\{R\}\_\{\>0\}is the scalar magnitude for domainii

Also, each manager adapterθi\\theta\_\{i\}is trained by minimizing:

ℒ​\(θi\)=ℒtask​\(θi;𝒟i\)\+λ2​∑jFj​\(θi\(j\)−θbase\(j\)\)2\\mathcal\{L\}\(\\theta\_\{i\}\)=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\(\\theta\_\{i\};\\mathcal\{D\}\_\{i\}\)\+\\frac\{\\lambda\}\{2\}\\sum\_\{j\}F\_\{j\}\\bigl\(\\theta\_\{i\}^\{\(j\)\}\-\\theta\_\{\\mathrm\{base\}\}^\{\(j\)\}\\bigr\)^\{2\}\(2\)whereFjF\_\{j\}is the diagonal Fisher information estimated atθbase\\theta\_\{\\mathrm\{base\}\}, andλ\>0\\lambda\>0is the EWC regularization coefficient\. This design makes domain knowledge modular by construction: authorized domains are composed explicitly through manager adapters rather than implicitly through prompts, and inference cost does not scale with the total number of authorized domains\.

### 3\.2Authorized Adapter Composition

A user is permitted to access a specified collection of domains\. Denote the domains the user has access to asD∈\{D​1,…,D​n\}D\\in\\\{D1,\\ldots,Dn\\\}\. We will describe two methods of building the subordinate manager adapters for the user across the domains included in D\.

1. 1\.Euclidean CompositionWe recompute each manager effective weight update from the shared manager and domain\-specific manager and subsequently calculate the average of each effective weight in an Euclidean manner; we then use truncated singular value decomposition to re\-refactor the resulting effective weight into a low number of dimensions\. This approach is a standard method of deriving an acceptable implementation method\. Δ​WDEuc=SVDr⁡\(1\|D\|​∑i∈DΔ​Wi\)\\Delta W\_\{D\}^\{\\text\{Euc\}\}=\\operatorname\{SVD\}\_\{r\}\\\!\\Bigl\(\\tfrac\{1\}\{\|D\|\}\\sum\_\{i\\in D\}\\Delta W\_\{i\}\\Bigr\)\(3\)
2. 2\.Riemannian CompositionSince DoRA differentiates normalized direction from magnitude, we can use a geometric method to compose effective updates\. We take the normalized directional effective updates as points on a unit hypersphere inℝ3\\mathbb\{R\}^\{3\}, compute a normalized directional averaging of these directional effective updates, and compute the average of their magnitudes using a Euclidean metric; we then re\-factor the resulting effective weight using SVD to generate the final merged effective weight\. This construction arises from the idea that normalized direction is a geometrical object; therefore, calculating the mean of the normals directly on the unit hypersphere containing the normals conserves their geometrical structure better\. v^DRiem=∑i∈Dv^i‖∑i∈Dv^i‖,mDRiem=∑i∈Dmi\\hat\{v\}\_\{D\}^\{\\text\{Riem\}\}=\\frac\{\\sum\_\{i\\in D\}\\hat\{v\}\_\{i\}\}\{\\bigl\\\|\\sum\_\{i\\in D\}\\hat\{v\}\_\{i\}\\bigr\\\|\},\\qquad m\_\{D\}^\{\\text\{Riem\}\}=\\sum\_\{i\\in D\}m\_\{i\}\(4\)

### 3\.3Evaluation Metrics

For single domain evaluations, we show both target loss and utility of domains when performing single\-domain evaluations\. Specifically, for SimpleQA, we provide an evaluation of the gains attributable to the domain\-specific manager implemented by summation of all domain\-specific gains into a single measure called the Utility Gap Index \(UGI\)\(Jayaraman et al\.,[2025](https://arxiv.org/html/2606.11262#bib.bib4)\)\. LetfDMf\_\{D\}^\{M\}denote the model composed with domain managerDD, and let𝒟S\\mathcal\{D\}\_\{S\}be the evaluation dataset for domainSS\. Define the accuracy of the model on domainSSas:

Acc​\(f,𝒟S\)=𝔼\(q,y\)∼𝒟S​\[𝟏​\{f​\(q\)=y\}\]\\mathrm\{Acc\}\(f,\\mathcal\{D\}\_\{S\}\)=\\mathbb\{E\}\_\{\(q,y\)\\sim\\mathcal\{D\}\_\{S\}\}\\bigl\[\\mathbf\{1\}\\\{f\(q\)=y\\\}\\bigr\]\(5\)
The Utility Gap Index \(UGI\) for domainSSis then defined as:

UGI​\(S\)=Acc​\(fSM,𝒟S\)−Acc​\(fbase,𝒟S\)\\mathrm\{UGI\}\(S\)=\\mathrm\{Acc\}\(f\_\{S\}^\{M\},\\mathcal\{D\}\_\{S\}\)\-\\mathrm\{Acc\}\(f\_\{\\mathrm\{base\}\},\\mathcal\{D\}\_\{S\}\)\(6\)wherefSMf\_\{S\}^\{M\}denotes the model with the domain\-specific manager forSSactivated,fbasef\_\{\\mathrm\{base\}\}is the baseline model without the domain manager, andUGI​\(S\)∈\[−1,1\]\\mathrm\{UGI\}\(S\)\\in\[\-1,1\]measures the change in task accuracy attributable to the domain\-specific adapter\. Positive values indicate improved domain utility, while negative values indicate degradation\.

With respect to evaluating security we also utilize the Domain Distinguishability Index \(DDI\)\(Jayaraman et al\.,[2025](https://arxiv.org/html/2606.11262#bib.bib4)\)\. The DDI measures how well a model trained using a certain domain can or cannot be distinguished between models that did not train on that domain\. To quantify model distinguishability we compute an inference\-based score for each of the active domains e\.g\. using the Min\-K\+\+ confidence score from our earlier findings in Zhang et al\., 2025 and subsequently compute an area \(AUC\) under the receiver operating curve for the task of classifying “in\-domain” versus “out\-of\-domain” instances\. A DDI AUC equal to 0\.5 denotes chance versus 1\.0 denotes an identifiable fingerprint; meaning that if a DDI is near 0\.5 then there is no identifying characteristics of the behavioral fingerprints of either set of models\. The tables presented in this document will be based upon using Min\-K\+\+\-based DDI AUC values; Log\-based probabilitiy, loss and Zlib versions will all be found in the appendix\. Additionally we will conduct analyses based upon cosine similarity and symmetric orthogonality losses between the Experiment 6 generated data as a basis to determine if the dataset separation is predictive of the quality of the dataset created through domain composition\.

D​D​I​\(m\)=𝔼Si⊆𝕊,Sj⊆𝕊​\[m​\(O\(Si,Sj\)\)\]DDI\(m\)=\\mathbb\{E\}\_\{S\_\{i\}\\subseteq\\mathbb\{S\},S\_\{j\}\\subseteq\\mathbb\{S\}\}\[m\(O^\{\(S\_\{i\},S\_\{j\}\)\}\)\]\(7\)- •D​D​I​\(m\)DDI\(m\): The Domain Distinguishability Index defined for a given membership inference metricmm\.
- •𝔼\\mathbb\{E\}: The expectation taken over all domain sets\.
- •𝕊\\mathbb\{S\}: The domain universe consisting ofnntotal security domains\.
- •Si,Sj⊆𝕊S\_\{i\},S\_\{j\}\\subseteq\\mathbb\{S\}: Ordered pairs of domain sets from the universe𝕊\\mathbb\{S\}that have no overlap \(i\.e\.,Si∩Sj=∅S\_\{i\}\\cap S\_\{j\}=\\emptyset\)\.
- •m​\(⋅\)m\(\\cdot\): A given membership inference metric\.
- •O\(Si,Sj\)O^\{\(S\_\{i\},S\_\{j\}\)\}: The output of a membership inference oracleOO, specifically defined asO\(fDℳ\(q\)\|Si,fDℳ\(q\)\|Sj\)O\(f\_\{D\}^\{\\mathcal\{M\}\}\(q\)\|S\_\{i\},f\_\{D\}^\{\\mathcal\{M\}\}\(q\)\|S\_\{j\}\)for all queriesq∼𝒟Siq\\sim\\mathcal\{D\}\_\{S\_\{i\}\}\.

For the trade\-off analysis we will additionally compute KL divergence statistics for the merged adapter versus using a union\-based average of the scores from those active domains to represent a true utility for each merged adapter created versus a single merged adapter\. The utility represented by the union\-based average is higher than for each individual domain, however, the run time will increase linearly with the number of active domains\.

### 3\.4Hypothesis: Geometry Predicts Equivalence Regimes, Not Performance

We study a narrower and falsifiable hypothesis than the claim that parameter\-space geometry determines composition quality\. Specifically, we test whether angular alignment between adapter updates predicts when Euclidean and geometry\-aware merging should behave similarly\. LetΔ​Wi\\Delta W\_\{i\}andΔ​Wj\\Delta W\_\{j\}denote the effective updates for domainsiiandjj, respectively\. Their cosine similarity is

cos⁡\(Δ​Wi,Δ​Wj\)=⟨Δ​Wi,Δ​Wj⟩F‖Δ​Wi‖F​‖Δ​Wj‖F\.\\cos\(\\Delta W\_\{i\},\\Delta W\_\{j\}\)=\\frac\{\\langle\\Delta W\_\{i\},\\Delta W\_\{j\}\\rangle\_\{F\}\}\{\\\|\\Delta W\_\{i\}\\\|\_\{F\}\\,\\\|\\Delta W\_\{j\}\\\|\_\{F\}\}\.\(8\)
Our hypothesis is that as adapter updates become more collinear, the difference between Euclidean and Riemannian\-inspired merging should shrink:

cos\(ΔWi,ΔWj\)↑⟹\|AccRiem\(D\)−AccEuc\(D\)\|↓\.\\cos\(\\Delta W\_\{i\},\\Delta W\_\{j\}\)\\uparrow\\quad\\Longrightarrow\\quad\\left\|\\mathrm\{Acc\}\{\\mathrm\{Riem\}\}\(D\)\-\\mathrm\{Acc\}\{\\mathrm\{Euc\}\}\(D\)\\right\|\\downarrow\.\(9\)
This is an equivalence\-regime hypothesis rather than a performance hypothesis\. That is, angular alignment may explain when geometry\-aware merging collapses to Euclidean behavior, but it does not by itself imply improved downstream utility or reduced interference\. We test this prediction empirically in Section 5\. \.

## 4Experimental Setup

### 4\.1Datasets and Domains

We evaluate on four benchmarks spanning science, biomedicine, open\-domain factual QA, and safety\-sensitive knowledge\. GPQA\(Rein et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib11)\)is partitioned into biology, chemistry, and physics\. PubMedQA\(Jin et al\.,[2019](https://arxiv.org/html/2606.11262#bib.bib6)\)is clustered into ten semantic groups\. SimpleQA\(Wei et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib13)\)is divided into ten categories including art, geography, history, and science and technology\. WMDP\(Li et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib8)\)is partitioned into bio, chem, and cyber domains\. We use WMDP only for the Llama experiments in the main paper\. For Mistral, we focus the cross\-backbone comparison on GPQA, PubMedQA, and SimpleQA\.

### 4\.2Backbones and Training Protocol

The primary foundation for our study was Llama version 3\.1\-8B\(Grattafiori et al\.,[2024](https://arxiv.org/html/2606.11262#bib.bib2)\)\. Training of LoRA and DoRA adapters was accomplished using rank = 16 and scale = 8; however, a shared baseline adapter was trained and subsequently frozen\. Then, per domain, manager adapters were trained while maintaining the backbone and baseline adapter fixed\.

Also, we utilized Mistral version 7\-B\(Jiang et al\.,[2023](https://arxiv.org/html/2606.11262#bib.bib5)\)as our Cross\-Backbone Validation model\. Results for Mistral were provided in the main paper for the single domain and multidomain merging investigations, however, all deeper ablation, compute\-tradeoff, security and diagnostic analysis was performed centered on Llama producing a cleaner and coherent comparison than would otherwise have been offered through comparisons made using both models\.

### 4\.3Experiments

Our evaluation has six components: \(1\) single\-domain utility of DoRA\-RBAC vs\. LoRA, \(2\) multi\-domain composition on 2\- and 3\-domain SimpleQA combinations, \(3\) cosine\-similarity and orthogonality diagnostics from independent artifacts, \(4\) compute–utility trade\-offs via KL divergence to a union reference, \(5\) domain distinguishability via DDI on Llama SimpleQA, and \(6\) ablation under strong EWC regularization\.

## 5Results

### 5\.1Single\-Domain Utility

Table 1 outlines single domain target loss \(loss\) and UGI values across the various Llama experiments conducted by either DoRA or LoRA on all assessed datasets \(GPQA, PubMedQA, SimpleQA, and WMDP\)\. The average differences between both methods are comparably low \(loss difference<0\.01<0\.01\) across the four respective domains\. However, the analysis of results from the various experimental methodologies leads the researchers to determine that there is no overall performance difference between DoRA and LoRA when running the Mistral method\. Regardless of whether running GPQA, the average difference in loss values was significantly greater when comparing DoRA \(3\.65\) versus LoRA \(3\.33\)\.

Thus, the essence of Table 1 is not that one method clearly dominates over the other nor does it imply that DoRA\-rbac is any less effective than LoRA in the single domain context \(which would call into question their contributions\)\. The original objective was to illustrate that modular decomposition methods developed in the previous chapter would still provide sufficient utility within the single domain; therefore, it creates the opportunity to evaluate the actual impact of decreasing inference caused when compare to coincidental use in the same domain

Table 1:Single‑domain performance \(mean±\\pmstandard deviation across runs\)\. Lower loss and higher UGI are better\. WMDP is reported only for Llama in the main paper\.
### 5\.2Multi\-Domain Composition

In Table 2, we sum all SimpleQA utility and AUC \(Area Under Curve\) for DDIs based upon Min\-K\+\+ for combinations between two domains and three domains of SimpleQA\. For both backbones, Euclidean and Riemannian\-based composition \(composition utilizing Euclidean and Riemaniann geometries\) is almost equivalent\. For the case with Llama, the average SimpleQA score for two\-domain combinations is approximately 0\.0053 under the Merge Rules; three\-domain combinations have an approximate SimpleQA score of 0\.0053 under each Merge Rule\. For the case with Mistral, the average SimpleQA scores for two\-domain combinations is approximately 0\.0123, and for three\-domain combinations is also approximately 0\.0123 with both Merge Rules\. The differences in SimpleQA scores and their inconsistencies are so minor that we cannot reasonably conclude that Geometry\-Aware Composition \(to compose observably from the same geometric properties\) produces a substantial or measurable impact in reducing interference due to linear directional overlap\.

This is the primary empirical finding of this paper\. Had linear directionality overlap been the primary explanation for interference, then a merge rule that honors DoRA’s normalized directions would have produced a stronger empirical finding\. The gain realized from merging is less related to having a more correctly structured geometric composition, and more related to the properties of modularity of composition\.

Table 2:Multi\-domain composition on SimpleQA\. Values are mean±\\pmstandard deviation across all domain combinations\. Accuracy is the target\-domain SimpleQA score; DDI AUC uses the Min–K%\+\+ variant\.a\+ga\+ha\+ma\+og\+hg\+mg\+oh\+mh\+om\+o0\.20\.20\.40\.40\.60\.60\.80\.811⋅10−2\\cdot 10^\{\-2\}Domain CombinationSimpleQA Accuracy2\-Domain Merges \(Llama\)EuclideanRiemanniana\+g\+ha\+g\+ma\+g\+oa\+h\+ma\+h\+oa\+m\+og\+h\+mg\+h\+og\+m\+oh\+m\+o0\.20\.20\.40\.40\.60\.60\.80\.811⋅10−2\\cdot 10^\{\-2\}Domain CombinationSimpleQA Accuracy3\-Domain Merges \(Llama\)Figure 1:SimpleQA accuracy for all 2\-domain \(left\) and 3\-domain \(right\) combinations on Llama\-3\.1\-8B\. Domain abbreviations: a=art, g=geography, h=history, m=music, o=other\. Euclidean and Riemannian merging produce virtually identical results across all combinations, confirming our central empirical finding\.
### 5\.3Diagnostics: Parameter\-Space Orthogonality Is Not Enough

More importantly, these diagnostics line up with the negative composition result: small shifts in cosine alignment or orthogonality do not translate into better downstream utility for Riemannian merging\. We therefore should not frame orthogonality in parameter space as the primary solution to composition interference\. A more plausible interpretation is that interference is dominated by shared nonlinear representations and downstream activation interactions that are not captured by linear angle\-based summaries alone\. The symmetric orthogonality loss for a domain pair\(i,j\)\(i,j\)is:

ℒ⟂​\(i,j\)=12​\(‖Δ​WD​v^j‖F2\+‖Δ​WD​v^i‖F2\)\\mathcal\{L\}\_\{\\perp\}\(i,j\)=\\tfrac\{1\}\{2\}\\Bigl\(\\\|\\Delta W\_\{D\}\\,\\hat\{v\}\_\{j\}\\\|\_\{F\}^\{2\}\+\\\|\\Delta W\_\{D\}\\,\\hat\{v\}\_\{i\}\\\|\_\{F\}^\{2\}\\Bigr\)\(10\)whereΔ​WD\\Delta W\_\{D\}is the composed effective weight for the merged pair\.

0\.10\.10\.110\.110\.120\.120\.130\.130\.140\.140\.150\.150\.160\.160\.170\.170\.180\.180\.190\.190\.20\.2−5\-505510101515Mean Cosine Similarity between AdaptersPerformance GapAngular Alignment Predicts Equivalence Regimes \(Llama\-3\.1\-8B, SimpleQA\)2\-domain pairs3\-domain triplesTrendFigure 2:Cosine similarity between domain adapter updates versus the performance gap between Euclidean and Riemannian merging \(in units of10−410^\{\-4\}SimpleQA accuracy\)\. As cosine similarity increases, the gap between the two merging strategies shrinks toward zero, indicating that angular alignment predicts an equivalence regime in which geometry\-aware merging collapses to Euclidean behavior\. Outliers at very low cosine similarity \(<0\.13<0\.13\) show the only non\-trivial gaps\. This relation does not imply improved downstream composition performance\.
### 5\.4Compute\-Utility Trade\-offs

Table 5\(Appendix\) provides an overview of the results of Llama’s two\-domain and three\-domain merge tradeoff study\. When considering means of Kullback\-Leibler \(KL\) divergence from the union\-style reference, writing using either Euclidean or Riemannian \(for all intents and purposes\) composition, there was an increase in this divergence score \(Euclidean\) of approximately 3\.57 to 3\.95, respectively and from 3\.58 to 3\.97 using the Riemannian method\. SimpleQA utility also showed very little change across these divergent paths, again substantiating the finding that adopting a geometry\-aware merging approach does not change the practical tradeoffs for these merges for domain merge in the current regime\.

The details of the previous paragraph potentially have an important wording distinction\. In general, each of the union\-style references will be a linearly increasing function of how many domains are currently in existence because they represent the combination of all domain logit values for each reference possibility across the total sum of active domains; therefore, The practical benefit of DoRA\-RBAC in relation to replacing multiple serving paths \(i\.e\., the sum of all access request for each reference\) one single adapter that will serve as a composite adapter once the types of access request have been created through authorized merges, however does not result in a total elimination of multiple serving paths for the given number of domains \(i\.e, interference\)\.

### 5\.5Security Audit

Llama’s Security Audit Reports all of the results for three SimpleQA domains using Llama’s \(Domain\-Specific\) Manager’s Modules as shown in Table 3\. By using a Domain\-Specific Manager \(the blue dots on the graph\), the average UGI \(Utility Sensitivity\) across all three SimpleQA domains was improved to 0\.1178\. The average DDI \(Domain Discriminability Index\) AUC \(Area Under Curve\) for the three SimpleQA domains was 0\.6526 \(Chances of success\) — \[one out of two\] which confirms that even though the DDI accurately identifies and tracks each model’s member of its target domain, all domain enabled models’s managers will continue to have distinguishable behavior and produce distinguishable outputs\. The per\-domain spread also supports the idea that these types of \(Domain\-Specific\) Managers’ Modules provide utility and the ability to compose \(i\.e\. modular access control\) or provide protection \(i\.e\. module privacy\) to domain specific information, but the Behavioral Signature of Being a Member of a Domain will always remain\.

Table 3:Llama security audit on selected SimpleQA domains\. Values are mean±\\pmstandard deviation across runs\. DDI AUC uses the Min–K%\+\+ variant\.
### 5\.6Ablation under Strong EWC

Table 6\(Appendix\) presents the results of Llama ablation using a strong EWC regularization method\. The primary assessment measure for this observation should not be taken from which of the two models has the lower final task loss, since DoRA does not outperform LoRA in terms of the lowest final task loss\. To further evaluate this experiment as an additional check for stability, we may find that there were smaller final loss values for each aggregate category associated with the LoRA model than for the DoRA model\. Therefore, we consider that the present aggregate statistic values do not demonstrate an advantage for DoRA over LoRA with respect to the minimum final overall loss \(or total loss\) that can be achieved by both models when only the final loss is calculated\. Consequently, although strong EWC regularization does not compromise the construction of DoRA\-RBAC models, neither does it support the argument that the better parameter space structure provides a complete answer to the problem of composition difficulty\.

### 5\.7Cosine Similarity vs\. Performance Gap

In order to evaluate the hypothesis put forth in Section 3\.4, we need to compare the pairwise cosine similarity among different types of Domain Adapters \(DAs\) and the corresponding performance gap between the different forms of Merger \(viz\., Riemannian and Euclidean\)\. Figure 7\(Appendix\) includes a plot of these variables for our SimpleQA data set, for both pairs and triples of DA’s\. Each individual blue data point represents the case of merging between two DAs or three DAs; the orange line indicates the overall trend\. From the graph it can be seen that, when the cosine similarity value between updates to DA’s is high \(i\.e\. the updates are approximately collinear\), the performance gap between the two forms of merging is essentially zero, that is, the use of geometry\-aware merging does not provide any benefits over the use of Euclidean averaging\. It is not until the cosine similarity value is extremely low between DA’s that a large difference exists between the two forms of merging; however, this occurrence was extremely infrequent in our data set\. This supports the more limited hypothesis that angular alignment predicts when geometry\-aware merging becomes effectively equivalent to Euclidean averaging\. However, this equivalence does not translate into improved downstream composition performance, indicating that angular alignment alone is not sufficient to explain interference\.

## 6Limitations

Our study has several limitations\. First, the experiments are conducted on a limited set of base models, domains, and composition settings, so the conclusions should be interpreted as evidence about these regimes rather than as a universal characterization of all adapter\-based systems\. Second, we study relatively shallow authorization structures and a modest number of simultaneously composed adapters; deeper hierarchies and larger compositions may introduce additional optimization and interference effects that are not captured here\. Third, while our results suggest that geometry\-aware merging does not substantially reduce interference in the studied settings, they do not rule out benefits from other manifold constructions, training objectives, or model families\. Finally, our evaluation focuses on task utility and distinguishability\-style diagnostics, which are informative but do not constitute formal guarantees of privacy, security, or robustness under stronger adversarial settings\.

## 7Conclusions

We introduced DoRA\-RBAC, a hierarchical DoRA\-based framework for modular parameter\-level access control in multi\-domain language models\. Using this framework, we tested whether enforcing cleaner directional structure in parameter space reduces adapter interference during composition\. Our results yield a clear negative finding\. Across Llama\-3\.1\-8B and Mistral\-7B, DoRA\-RBAC matches LoRA in single\-domain utility\. In multi\-domain composition on SimpleQA, Euclidean and Riemannian merging produce virtually identical results for all 2\- and 3\-domain combinations\. Cosine similarity and orthogonality diagnostics do not explain the remaining degradation\. The mechanistic explanation is straightforward: when adapter update directions are nearly collinear — as they are in practice — the normalized directional averaging on the hypersphere reduces to an ordinary arithmetic mean, collapsing any potential advantage of geometry\-aware merging\. The composition bottleneck therefore lies not in linear directional overlap but in how separately trained adapters interact through shared nonlinear representations and downstream activations\. Representation\-level objectives or activation\-space constraints, rather than parameter\-space geometry, are the more promising path toward reliable compositionality\. On the access\-control side, domain\-specific managers provide meaningful utility gains \(\+0\.1178 UGI\), but DDI AUC of 0\.6526 confirms that modular composition does not guarantee privacy\. DoRA\-RBAC is best understood as a modular composition and serving framework; formal privacy defenses such as differentially private adapter training remain necessary for high\-assurance deployments\.

## Acknowledgments

Omitted for double\-blind review\.

## Ethics Statement

This work studies membership inference attacks for the purpose of understanding and mitigating privacy risks in language model deployments\. The attack implementations employed in our evaluation \(log\-probability ratio, loss\-based MIA, Min\-kk%\+\+\(Zhang et al\.,[2025](https://arxiv.org/html/2606.11262#bib.bib17)\)\) are drawn entirely from established literature; we introduce no new attack capabilities\. The finding that Euclidean adapter merging is uniformly vulnerable to membership inference is disclosed to enable practitioners to take protective measures when deploying multi\-adapter systems in sensitive environments\. Riemannian merging is proposed as a protective mechanism, not as a tool for circumventing privacy controls\. All experiments were conducted on publicly available benchmarks \(WMDP, GPQA, SimpleQA, PubMedQA\) using open\-weight models \(Llama\-3\.1\-8B, Mistral\-7B\)\. No personally identifiable information was used in any experiment\.

## Use of Large Language Models\.

The authors previously consults with various large language models \(LLM\) to assist them through all stages of the writing process\. From idea validation to alternative forms of existing hypotheses to drafts and refining parts of the manuscript to developing and implementing code prototypes, LLMs were employed within an iterative process of reviewing, making changes, and integrating generated outputs\.

The main contributions of this research project – formulating the problem, designing the experiments, determining methodology, and interpreting results – were developed and executed by the authors\. While LLMs expedited certain areas of the development process and presentation of findings, they were not relied upon as definitive sources of knowledge\. Therefore, all of the material generated by LLMs \(including code and text\) was extensively reviewed, verified, and validated prior to being included as part of final product\.

The experimental results reported herein are based solely upon the author’s implementations and evaluations\. Any LLM\-assisted code generated by the LLM and utilized in this work was thoroughly reviewed, and corrected where necessary, and subjected to the same validation process required for manually created code\. Any text generated using LLM assistance was revised to ensure accuracy, clarity and direct correlation to the underlying scientific contributions\.

The authors take full responsibility for the accuracy, authenticity, and integrity of this work\. The use of LLMs in this manner is akin to any other tools that enhance productivity, and do not substitute or diminish the importance of human reasoning, judgment, or scientific responsibility throughout the research process\.

## References

- Fleshman et al\. \(2025\)William Fleshman, Aleem Khan, Marc Marone, and Benjamin Van Durme\.AdapterSwap: Continuous training of LLMs with data removal and access\-control guarantees\.*arXiv preprint arXiv:2404\.08417*, 2025\.URL[https://arxiv\.org/abs/2404\.08417](https://arxiv.org/abs/2404.08417)\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.URL[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9)\.
- Jayaraman et al\. \(2025\)Bargav Jayaraman, Virendra J\. Marathe, Hamid Mozaffari, William F\. Shen, and Krishnaram Kenthapadi\.Permissioned LLMs: Enforcing access control in large language models\.*arXiv preprint arXiv:2505\.22860*, 2025\.URL[https://arxiv\.org/abs/2505\.22860](https://arxiv.org/abs/2505.22860)\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\.Mistral 7b\.*arXiv preprint arXiv:2310\.06825*, 2023\.URL[https://arxiv\.org/abs/2310\.06825](https://arxiv.org/abs/2310.06825)\.
- Jin et al\. \(2019\)Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu\.PubMedQA: A dataset for biomedical research question answering\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pp\. 2567–2577\. Association for Computational Linguistics, 2019\.doi:10\.18653/v1/D19\-1259\.URL[https://aclanthology\.org/D19\-1259/](https://aclanthology.org/D19-1259/)\.
- Karcher \(1977\)Hermann Karcher\.Riemannian center of mass and mollifier smoothing\.*Communications on Pure and Applied Mathematics*, 30\(5\):509–541, 1977\.doi:10\.1002/cpa\.3160300502\.
- Li et al\. \(2024\)Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D\. Li, Ann\-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm\-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert\-Voss, Cort B\. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken\-Harder, Kevin Y\. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M\. Esvelt, Alexandr Wang, and Dan Hendrycks\.The WMDP benchmark: Measuring and reducing malicious use with unlearning\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 28525–28550\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/li24bc\.html](https://proceedings.mlr.press/v235/li24bc.html)\.
- Li & Liang \(2021\)Xiang Lisa Li and Percy Liang\.Prefix\-tuning: Optimizing continuous prompts for generation\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pp\. 4582–4597\. Association for Computational Linguistics, 2021\.doi:10\.18653/v1/2021\.acl\-long\.353\.URL[https://aclanthology\.org/2021\.acl\-long\.353/](https://aclanthology.org/2021.acl-long.353/)\.
- Liu et al\. \(2024\)Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.DoRA: Weight\-decomposed low\-rank adaptation\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 32100–32121\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/liu24bn\.html](https://proceedings.mlr.press/v235/liu24bn.html)\.
- Rein et al\. \(2024\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.GPQA: A graduate\-level google\-proof Q&A benchmark\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=Ti67584b98](https://openreview.net/forum?id=Ti67584b98)\.
- Shoemake \(1985\)Ken Shoemake\.Animating rotation with quaternion curves\.In*Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques \(SIGGRAPH ’85\)*, pp\. 245–254\. ACM, 1985\.doi:10\.1145/325334\.325242\.
- Wei et al\. \(2024\)Jason Wei, Karina Nguyen, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus\.Measuring short\-form factuality in large language models\.*arXiv preprint arXiv:2411\.04368*, 2024\.URL[https://arxiv\.org/abs/2411\.04368](https://arxiv.org/abs/2411.04368)\.
- Wortsman et al\. \(2022\)Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo\-Lopes, Ari S\. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt\.Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.In*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pp\. 23965–23998\. PMLR, 2022\.URL[https://proceedings\.mlr\.press/v162/wortsman22a\.html](https://proceedings.mlr.press/v162/wortsman22a.html)\.
- Yadav et al\. \(2023\)Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal\.TIES\-merging: Resolving interference when merging models\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=xtaX3WyCj1](https://openreview.net/forum?id=xtaX3WyCj1)\.
- Yu et al\. \(2022\)Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A\. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang\.Differentially private fine\-tuning of language models\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=Q42f0dfjECO](https://openreview.net/forum?id=Q42f0dfjECO)\.
- Zhang et al\. \(2025\)Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li\.Min\-K%\+\+: Improved baseline for pre\-training data detection from large language models\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=ZGkfoufDaU](https://openreview.net/forum?id=ZGkfoufDaU)\.

## Appendix AFramework overview

PermDoRA operates in three stages, illustrated in Figure[3](https://arxiv.org/html/2606.11262#A1.F3)\.

Stage 1DoRA Trainingper domainStage 2DirectionExtractionStage 3RiemannianMergingDomainData𝒟i\\mathcal\{D\}\_\{i\}RBACAccessPolicyMergedAdapterθ¯\\bar\{\\theta\}θi\\theta\_\{i\}v^i,mi\\hat\{v\}\_\{i\},m\_\{i\}Figure 3:PermDoRA pipeline\. Per\-domain DoRA adapters are trained with EWC regularisation \(Stage 1\), decomposed into unit\-direction and magnitude components \(Stage 2\), and merged via Fréchet mean on𝒮d−1\\mathcal\{S\}^\{d\-1\}subject to an RBAC policy \(Stage 3\), producing a shared merged adapterθ¯\\bar\{\\theta\}\.Stage 1 — Per\-domain adapter training\.For each domaini∈\{1,…,N\}i\\in\\\{1,\\ldots,N\\\}with training set𝒟i\\mathcal\{D\}\_\{i\}, we train a DoRA adapterθi\\theta\_\{i\}on the frozen backbone using task loss plus an EWC penalty \(Section[3\.3](https://arxiv.org/html/2606.11262#S3.SS3)\)\.

Stage 2 — Direction extraction\.DoRA’s weight parameterisation directly provides a unit\-direction vectorv^i∈𝒮d−1\\hat\{v\}\_\{i\}\\in\\mathcal\{S\}^\{d\-1\}and a scalar magnitudemi∈ℝ\>0m\_\{i\}\\in\\mathbb\{R\}\_\{\>0\}for each adapter layer\.

Stage 3 — Riemannian geodesic merging\.Direction components are merged via Fréchet mean on the unit sphere \(Section[3\.2](https://arxiv.org/html/2606.11262#S3.SS2)\)\. Magnitudes are merged by Euclidean averaging\. The RBAC policy determines which domain adapters participate in each merge\.

PermDoRA supports three access\-control operations:Activate\(load a single domain adapter for a role\),Merge\(produce a single merged adapter from an authorised subset of domains\), andUnion\(compute the Fréchet mean over the full authorised set for a role with multi\-domain access\)\.

## Appendix BAdapter Composition Procedure

### B\.1Euclidean / SVD Composition

Recover the effective update for each authorized domain after combining the baseline adapter with the corresponding manager\. Average the effective updates in Euclidean weight space\. Subtract the baseline contribution and refactor the result with truncated SVD to obtain a merged low\-rank adapter\. Use this adapter directly at inference time as the composed authorization\-specific adapter\.

### B\.2Riemannian Composition

Separate each authorized DoRA update into its normalized direction and magnitude\. Compute a normalized directional averaging over the normalized directions on the hypersphere\. Average the magnitudes in Euclidean space\. Recombine the averaged magnitude with the Fréchet\-mean direction and refactor with truncated SVD to obtain the final merged adapter\.

## Appendix CExperimental Setup

To evaluate the efficacy and security of our proposed adapter\-based access control framework against standard baseline methods, we design a comprehensive evaluation matrix encompassing single\-domain isolation, multi\-adapter merging, and membership inference vulnerability\.

### C\.1Hardware and Environment

All fine\-tuning, inference, and diagnostic refactoring experiments were conducted on a dedicated compute node equipped with a single NVIDIA A100 Tensor Core GPU \(80 GB VRAM\)\. The extended memory capacity of the 80 GB A100 was strictly necessary to accommodate the simultaneous memory states required during our dual\-ordered Gram\-Schmidt orthogonalization and Riemannian manifold optimization, without resorting to severe quantization\. The software environment utilized PyTorch and the Hugging Facetransformersandpeftlibraries, with all models loaded natively in bfloat16 precision to optimize memory bandwidth while preserving numerical stability\.

### C\.2Models and Datasets

We benchmark our approach across two prominent, open\-weights foundation models:

- •LLaMA\-3\.1 \(8B\) \(meta\-llama/Meta\-Llama\-3\.1\-8B\)
- •Mistral \(7B\) \(mistralai/Mistral\-7B\-v0\.1\)

To assess domain\-specific isolation and cross\-domain leakage, we utilize four diverse evaluation datasets partitioned into distinct topical domains:

- •SimpleQA:Partitioned into 10 domains \(e\.g\., Art, Geography, History, Music\)\.
- •PubMedQA:Partitioned into 10 medical sub\-domains\.
- •WMDP & GPQA:Partitioned into 3 domains each, representing highly specialized and expert\-level knowledge\.

### C\.3Training Hyperparameters and PEFT Baselines

For our baseline comparisons, we utilize two Parameter\-Efficient Fine\-Tuning \(PEFT\) methodologies: Low\-Rank Adaptation \(LoRA\) and Weight\-Decomposed Low\-Rank Adaptation \(DoRA\)\.

For each specific dataset\-domain pair \(e\.g\., SimpleQA\-Art\), we train dedicated independent adapters\. Unless otherwise specified in ablation studies, the training hyperparameters were standardized across all baseline models as follows:

- •Rank \(rr\) and Alpha \(α\\alpha\):Set tor=16r=16andα=16\\alpha=16to maintain a balanced scaling factor ofαr=1\.0\\frac\{\\alpha\}\{r\}=1\.0\.
- •Epochs:5 epochs per adapter\.
- •Batch Size:2\.
- •Learning Rate:We employed the AdamW optimizer with a peak learning rate of2×10−42\\times 10^\{\-4\}and a cosine decay schedule\.
- •Data Split:A strict 80/20 train\-evaluation split was maintained across all domains to compute out\-of\-sample target loss\.

### C\.4Evaluation Matrix and Diagnostics

Our orchestrator executes a multi\-stage pipeline to rigorously evaluate the adapters:

1. 1\.Utility Gap Index \(UGI\):To measure cross\-domain leakage, we compute the UGI by evaluating an adapter trained on domain A against the evaluation split of domain B, quantifying the absolute utility gap between authorized and unauthorized domain knowledge\.
2. 2\.N\-Way Merging & Tradeoffs:We evaluate multi\-domain synthesis by combining distinct adapters\. We compare standard Euclidean weight averaging against Riemannian manifold merging\. Because standard averaging of low\-rank matrices does not preserve the intrinsic low\-rank geometry, we formulate the merging as an optimization problem on the manifold of rank\-rrmatrices, minimizing the structural degradation caused by conflicting weight updates\. We evaluate this structural interference via ensemble Kullback\-Leibler \(KL\) Divergence backfilling\.
3. 3\.Dual\-Ordered Orthogonal Diagnostics:To understand the topological overlap of independently trained domains, we analyze the adapter weight updates in their true effective update space \(Δ​W=αr​B​A\\Delta W=\\frac\{\\alpha\}\{r\}BA\)\. We calculate the baseline Cosine Similarity between adapter pairs and perform a dual\-ordered Gram\-Schmidt orthogonalization\. For two adapters,AAandBB, we project the flattened effective update ofBBontoAAto find the orthogonal complement: Δ​B⟂A=Δ​B−projΔ​A​\(Δ​B\)\\Delta B\_\{\\perp A\}=\\Delta B\-\\text\{proj\}\_\{\\Delta A\}\(\\Delta B\)We then reconstruct the merged adapter by averaging the primary vector and the orthogonalized secondary vector\. For DoRA adapters, where the weight update is non\-linear due to the learned magnitude vectormm, we implement a 300\-step Adam optimization routine to dynamically refactor the orthogonalized weight space back into the decoupled DoRA format, ensuring the final directional matrices and magnitude vectors precisely map to the orthogonalized target space\. We report the symmetric average loss of theAA\-first andBB\-first orthogonalizations\.
4. 4\.Security Audit \(MIA\):To assess data privacy, we subject the adapters to a comprehensive Membership Inference Attack \(MIA\) suite\. We calculate the Data\-Dependent Information \(DDI\) based on reference log\-probabilities and report the Area Under the Receiver Operating Characteristic Curve \(AUC\-ROC\) to quantify the empirical vulnerability of the adapters to memorization\-based attacks\.

## Appendix DDiscussion

The primary and most widely supported assertion in this article is not merely that DoRA\-RBAC provides a modular means of controlling access; rather, it provides a basis for understanding how adapter interactions that occur during composition are governed by their alignment in angular terms, and that angular alignment cannot be made orthogonal or fit into tidier shapes simply by enforcing orthogonality or other geometric properties\. The value of DoRA\-RBAC for this purpose is that it permits structured experimentation to validate this hypothesis through the direct expression of both direction and magnitude; however, results demonstrate that geometry\-aware merging does not, in a meaningful way, yield better performance than standard Euclidean composition\.

#### Mechanistic explanation: Why geometry\-aware merging collapses\.

To determine how to combine updates of one adapter with other adapters in a way that gives the desired overall update, one can use geometry aware merger to compute a normalized directional averaging\(Karcher,[1977](https://arxiv.org/html/2606.11262#bib.bib7); Shoemake,[1985](https://arxiv.org/html/2606.11262#bib.bib12)\)of the update directions at normalized update values on the hypersphere\. When update directions are nearly collinear \(that is when they are close to being aligned\) and thus have about the same update direction, the resulting normalized directional averaging will be nearly the same as an arithmetic mean\. When the update direction of the adapters is orthogonal to each other \(i\.e\. when they are 90 degrees apart\) or antiparallel \(180 degrees apart\) then using these means on a hypersphere may yield drastically different results than using either of these two approaches in isolation\. Our diagnostics have shown that real\-world adapters/matrices very infrequently are in orthogonals: the pairwise cosine similarities are typically low but positive and the norms of the updates are generally imbalanced\. As such, the effective overall update produced by geometry aware mergers and Euclidean or Riemannian mergers tend to produce essentially the same final results\. This provides mechanistic justification for the fact that geometry cannot help compositionally: the bottleneck doesn’t lie in managing the alignment of update directions but in understanding how these differently trained adapters interact \(through the nonlinear core\) when combined into a single overall update\. Thus, it may be necessary to employ either representation\-level objectives or explicit activation space constraints to achieve the kind of reliable composability of adapter updates than simply making sure that they are aligned potentially\. The implication of that conclusion is therefore informative as well\. The composition bottleneck lies more in understanding how separately trained matrices interact with one another in the shared nonlinear representation space\(s\) than it does in the amounts of linear or other directional overlap that exist between matrices\. Even if two matrices are well separated in parameter spaces, if each has to pass through similar \(or any\!\) features, nonlinearities, or use the same conditioned activations, they may well interfere in terms of the resulting updates\. Thus, in order to mitigate one matrix’s interference with the other’s, it is likely that representation\-level objectives, coordination during training, or explicit constraints in terms of activation space not merely improved angle management—may be needed to maximize reliable compositionality\. That conclusion also keeps the practical contribution of this work in context as well\. DoRA\-RBAC continues to be a very valuable framework for implementing modular parameter level access control or direct serving of single adapters; the security analysis discussed earlier provides insight into where that framework ends\. This system can and will permit the storage of and selective composition of domain and functionality specific behavior, but the performance of modular parameterizations and geometrically aware merger will not be replacements for having complete solutions for privacy or interference\.

## Appendix ESupplemental Results

GPQA BioGPQA ChemGPQA PhysPubMedμ\\muSQA ArtSQA GeoSQA HistWMDP BioWMDP ChemWMDP Cyber05⋅10−25\\cdot 10^\{\-2\}0\.10\.10\.150\.15Dataset / DomainUGISingle\-Domain UGI: DoRA\-RBAC vs\. LoRA \(Llama\-3\.1\-8B\)LoRADoRAFigure 4:Per\-domain UGI for Llama\-3\.1\-8B under single\-domain adaptation\. DoRA and LoRA perform comparably across all datasets \(GPQA, PubMedQA, SimpleQA, WMDP\), confirming that decomposed adaptation does not sacrifice single\-domain utility\.2 Domains3 Domains02244663\.573\.573\.953\.953\.583\.583\.973\.97Number of Domains MergedKL DivergenceKL Divergence: Single Composed Adapter vs\. Union Reference \(Llama\)EuclideanRiemannianFigure 5:KL divergence between the output distributions of a single composed adapter and a union\-style ensemble reference, for 2\- and 3\-domain merges on Llama\. Both merging strategies exhibit nearly identical KL divergence, with only marginal increase as the number of merged domains grows from 2 to 3\.ArtHistorySci&TechAverage05⋅10−25\\cdot 10^\{\-2\}0\.10\.10\.150\.150\.20\.24\.8⋅10−34\.8\\cdot 10^\{\-3\}7\.4⋅10−37\.4\\cdot 10^\{\-3\}7\.6⋅10−37\.6\\cdot 10^\{\-3\}6\.6⋅10−36\.6\\cdot 10^\{\-3\}0\.160\.160\.110\.110\.110\.110\.120\.12SimpleQA AccuracyManager Accuracy vs\. Baseline \(Llama\)BaselineManagerArtHistorySci&TechAverage0\.50\.50\.60\.60\.70\.70\.80\.8Chance0\.680\.680\.70\.70\.580\.580\.650\.65DDI AUCDDI AUC \(Min\-K%\+\+ Variant, Llama\)Figure 6:Left:Per\-domain SimpleQA accuracy with and without the domain\-specific manager adapter \(Llama\-3\.1\-8B, SimpleQA domains\)\. The manager improves average accuracy by0\.11780\.1178UGI\.Right:DDI AUC \(Min\-K%\+\+ variant\) for each domain\. All values exceed the chance level of0\.50\.5\(dashed red line\), confirming that domain membership remains inferrable despite adapter modularisation\.a\+ga\+oa\+ma\+hg\+mg\+hg\+om\+oh\+mh\+o0\.10\.10\.150\.150\.20\.2Domain Combination \(sorted by LoRA cosine similarity\)Average Cosine SimilarityPairwise Cosine Similarity: LoRA vs\. DoRA Directions \(Llama\)LoRADoRAFigure 7:Mean cosine similarity between pairs of domain adapter effective weight updates for LoRA and DoRA across all 10 pairwise domain combinations \(Llama\-3\.1\-8B\)\. DoRA directions are consistently slightly more aligned than LoRA \(Δ≈\+0\.009\\Delta\\approx\+0\.009\), yet this marginal difference does not translate into meaningfully different composition performance, confirming the Orthogonality\-is\-Not\-Enough result of Section 5\.3\.01122334455667788995⋅10−25\\cdot 10^\{\-2\}0\.10\.10\.150\.150\.20\.2Domain Combination Index \(sorted\)Mean Cosine Similarity \(LoRA\)Cross\-Backbone Cosine Similarity: Llama\-3\.1\-8B vs\. Mistral\-7BLlama\-3\.1\-8BMistral\-7BFigure 8:Pairwise LoRA cosine similarity between domain adapters for Llama\-3\.1\-8B and Mistral\-7B\. Mistral shows uniformly lower inter\-adapter cosine similarity \(μ≈0\.097\\mu\\approx 0\.097\) compared to Llama \(μ≈0\.147\\mu\\approx 0\.147\), suggesting that Mistral’s adapters occupy a more spread\-out region of weight space\. Despite this architectural difference, neither model shows composition improvements from Riemannian merging, reinforcing the generality of our finding\.Table 4:Diagnostics from the separate Experiment 6 artifact\.Table 5:Llama compute–utility trade‑offs relative to a union‑style reference that evaluates each active domain adapter separately and averages logits\. Values are reported as mean±\\pmstandard deviation across all two‑ or three‑domain combinations\.Table 6:Llama ablation on SimpleQA with strong EWC\. Values are mean±\\pmstandard deviation; lower is better\.