一次编码，多次作答：通过参数化流内化上下文

# Internalizing Context via Parametric Flow
Source: [https://arxiv.org/html/2610.38356](https://arxiv.org/html/2610.38356)
1\]Tencent2\]UT Austin3\]NTU4\]NUS5\]UCF6\]University of Sydney7\]University of St\. Gallen8\]Technion\\metadata\[ Github\] https://github\.com/Cypher\-Bruce/CLIK\.git\\metadata\[ Correspondence\]Haoyu Cao, Zhangyang Wang, Kai Wang

## Encode Once, Answer Many: Internalizing Context via Parametric Flow

Mengxuan WuChonghua LiaoLinxi XieRuicheng FengYanpei GuoHao TanJiaxin HongJason LiPanpan ZhangZiheng QinQinglin LuYuzhang ShangTongliang LiuDamian BorthHaggai MaronYongxiang HuaZhixiang WeiJian HuYunfei WuXing SunXin LiHaoyu CaoZhangyang WangKai WangAffiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[

###### Abstract

Large language models are typically adapted to a task either by placing knowledge in the input context, which keeps it external and re\-encodes it on every call, or by fine\-tuning, which internalizes it but demands a costly round of optimization per task\. We aim to internalize the knowledge carried by a context directly into model parameters, without per\-task optimization and without keeping the context in the input at inference time\. To this end, we proposeCLIK, a context\-conditioned parameter generator that maps a given context to LoRA parameters for internalizing knowledge in a single forward pass through a Transformer encoder with cross\-attention, requiring no test\-time gradient updates\. Once the generated LoRA is inserted into a frozen base model, the model answers downstream queries from the query alone, having internalized the context into its weights rather than re\-reading it\. Because the adapter depends only on the context, it is generated once and reused for every query on the same document, so the cost of reading that document is amortized across queries\. On document question answering, CLIK substantially improves over the no\-context base model and outperforms existing hypernetwork\-based generators, while remaining behind a base model that reads the context directly, and it carries over to multi\-turn dialogue without any multi\-turn\-specific training\. Analysis further shows that the context\-dependent part of the generated adapter concentrates in a compact and specific substructure\.

††† Equal contribution## 1Introduction

Figure 1:Three ways to supply external knowledge to a frozen LLM\.*Context in Prompt:*re\-injecting the context as text incurs high token cost\.*Context in Parameter:*fine\-tuning each context into a LoRA incurs high computational cost\.*Context\-conditioned Parameter Generator \(ours\):*a generator maps each context to a LoRA in a single forward pass, free of both test\-time optimization and context in the input\.Many uses of large language models \(LLMs\) require the model to act on a specific body of knowledge that is not in its pretrained weights, such as a set of reference documents, a long task instruction, or the history of an ongoing conversation\([Brown et al\., 2020](https://arxiv.org/html/2610.38356#bib.bib1);[Laban et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib12)\)\. The standard way to supply this knowledge is to place it in the model’s input context, as in in\-context learning and retrieval\-augmented prompting, where the material is kept as text and read by attention at generation time; this approach is flexible, requires no training, and has become the default as context windows have grown\. A second route is to write the knowledge into the weights by fine\-tuning, so that the model answers without the context in its input\([Ouyang et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib3);[Zhang et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib9)\): parameter\-efficient methods such as LoRA make each update cheap to store\([Hu et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib8)\), and context distillation trains the model to reproduce, without the context, the outputs it would have produced with it\([Snell et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib10);[Kujanpää et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib11)\)\.

Keeping the knowledge in the input runs into two well\-documented problems\. The first is that quality does not scale with length, because as inputs grow, models increasingly fail to use the relevant information and their accuracy degrades, an effect reported as lost\-in\-the\-middle behavior, weak long in\-context learning, and context rot\([Liu et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib4);[Li et al\., 2025a](https://arxiv.org/html/2610.38356#bib.bib5);[Hong et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib7)\)\. The second is that cost grows with length, because self\-attention is quadratic in the input and the key–value cache that must be stored and re\-read at every decoding step grows linearly with the number of context tokens, so both latency and memory rise with the amount of knowledge supplied\([Ge et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib22)\)\. Prompt\- and cache\-compression methods relieve this overhead by shortening the input or shrinking the cache\([Jiang et al\., 2023](https://arxiv.org/html/2610.38356#bib.bib20);[Pan et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib21);[Ge et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib22)\), but the knowledge still lives outside the parameters and must be re\-supplied and re\-read for every new query, so it is supplied to the model but never internalized by it\.

Writing the knowledge into the weights instead internalizes it directly, but each new context needs its own optimization run, which is slow and hyperparameter\-sensitive and has to be repeated whenever the knowledge changes\. This becomes impractical when contexts are numerous, user\-specific, or arrive continually\([Li et al\., 2025b](https://arxiv.org/html/2610.38356#bib.bib13)\)\. Fine\-tuning therefore internalizes knowledge but cannot adapt on the fly\.

These two routes trade off in opposite directions \([Figure1](https://arxiv.org/html/2610.38356#S1.F1)\)\. Context\-based adaptation is instant but never internalizes its knowledge, while fine\-tuning internalizes the knowledge but is too costly to update for each new context\. Hypernetworks\([Ha et al\., 2017](https://arxiv.org/html/2610.38356#bib.bib14)\)offer a way to keep the advantages of both, since one network can emit the weights of another and thus turn an input into an adapter in a single forward pass, with no test\-time optimization and nothing left in the context window\([Chen et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib15);[Abdalla et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib16);[Liang et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib18)\)\. Recent work brings this idea to LLMs by mapping a task description or a short context to a LoRA adapter\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24);[Liu et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib25)\)\. Turning a long context into a faithful adapter is harder than a naive application of this idea, because the adapter is high\-dimensional and existing generators emit its weights in independent fragments that ignore dependencies across layers and modules, so the longer and richer the context, the harder it is to capture in one coherent set of weights\.

We propose to directly generate context\-conditioned LoRA for internalizing knowledge \(CLIK\), a parameter generator built for this regime that maps a context directly to LoRA parameters in a single forward pass\. CLIK encodes the context with the frozen backbone itself and uses a Transformer that attends from the context to the adapter through cross\-attention, generating the full set of LoRA weights jointly rather than fragment by fragment, so that dependencies across layers and modules are modeled as the knowledge is internalized\. The generated adapter is inserted into the frozen model, which then answers from the query alone, without further access to the original context\.

This design is particularly suited to multi\-turn interaction, the setting where existing paradigms struggle most\([Laban et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib12);[Li et al\., 2025b](https://arxiv.org/html/2610.38356#bib.bib13)\)\. Once a context is internalized into the adapter, it no longer has to occupy the input on later turns, and because the adapter depends only on the context, it is generated once and reused for every subsequent query on the same document, so the cost of reading that document is amortized across queries rather than paid again at each turn\. CLIK therefore combines the immediacy of in\-context learning with the persistence of parameterized knowledge\.

Our main contributions are summarized as follows:

- •A context\-to\-adapter generator that internalizes knowledge jointly\.We propose CLIK, which maps a context to a full LoRA adapter in a single forward pass via cross\-attention over the backbone’s all\-layer representations\. Unlike generators that emit weights as independent fragments, it models dependencies across layers and modules, internalizing a rich context into one coherent adapter with no test\-time optimization\.
- •Strong empirical results\.On single\-hop and multi\-hop document question answering, CLIK substantially outperforms the no\-context base model and prior hypernetwork\-based generators, while remaining behind a base model with full context access, and it transfers to multi\-turn dialogue without multi\-turn\-specific training\.
- •Analysis of how context is internalized\.Through intervention and distributional analysis of the generated adapters, we show that the context\-dependent part of the adapter concentrates in a compact and specific substructure of the LoRA\.

## 2Related Work

### 2\.1Context Internalization

Context internalization transfers the information a model would otherwise read from its context into a form it can use directly, so that downstream queries are answered without re\-processing the original context\. Existing methods divide by whether they require parameter optimization\.*Training\-free*methods keep the weights fixed and act on the input: in\-context learning places the material in the prompt\([Brown et al\., 2020](https://arxiv.org/html/2610.38356#bib.bib1)\), while prompt\- and KV\-cache compression shorten the input or shrink the cached activations to cut test\-time cost\([Jiang et al\., 2023](https://arxiv.org/html/2610.38356#bib.bib20);[Pan et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib21);[Ge et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib22)\)\. In all cases the knowledge stays outside the parameters and must be re\-supplied for every query\.*Optimization\-based*methods instead move the information into the weights: supervised fine\-tuning absorbs a corpus but needs curated data\([Zhang et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib9);[Hu et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib8)\), and context distillation trains the model to reproduce, without the context, the outputs it would have produced with it\([Askell et al\., 2021](https://arxiv.org/html/2610.38356#bib.bib23);[Snell et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib10);[Kujanpää et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib11)\)\. These avoid re\-reading the context at inference, but each new context triggers a separate optimization loop that is too slow for interactive use\. CLIK keeps the benefits of both by amortizing this per\-context optimization into a single learned mapping\.

### 2\.2Hypernetworks

A hypernetwork produces the parameters of another network\([Ha et al\., 2017](https://arxiv.org/html/2610.38356#bib.bib14)\), turning adaptation into a forward pass rather than a gradient search\. For LLMs, this means adapting a frozen model without long prompts or fine\-tuning, typically by generating a low\-rank LoRA adapter\([Hu et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib8)\)from a task description or a context\([Chen et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib15);[Abdalla et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib16)\)\. The central difficulty is the high\-dimensional output: many designs emit weight fragments independently and concatenate them, ignoring dependencies across the adapter\. Closest to us, Doc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24)\)meta\-learns the context\-distillation process into a per\-layer independent hypernetwork that maps a context to a LoRA in a single pass, and SHINE\([Liu et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib25)\)instead inserts and trains an additional LoRA on the base model together with learned memory tokens appended to the context, whose hidden states at every layer form a fixed\-length memory that serves as the input to a subsequent hypernetwork\. CLIK follows this in\-context line but uses a Transformer that attends from the context to the adapter through cross\-attention, generating the full set of LoRA weights jointly in one forward pass\. Beyond text, the same input\-to\-weights mechanism spans modalities and tasks: text\-guided image editing\([Tencent HY Team, 2026](https://arxiv.org/html/2610.38356#bib.bib17)\), large\-scale parameter generation by diffusion\([Wang et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib19);[Wang et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib2)\), reinforcement\-learned parameter adaptation for video\-anomaly detection\([Jacob et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib26)\), internalizing a video or a codebase into LoRA parameters\([Suri et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib28);[Hotsko et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib29)\), and agentic settings where one agent edits another’s weights rather than giving textual advice\([Bao et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib27)\)\. Our context\-to\-LoRA formulation is one instance of this broader trend of treating weight space as a first\-class generative modality\([Wang et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib6)\), specialized to LLM context adaptation\.

Figure 2:Overview of CLIK\.A frozen language model encodes the context, and a learned ELMo\-style weighting of its all\-layer hidden states conditions a trainable generator that produces a context\-specific LoRA adapter in a single forward pass\. The adapter is injected into the same frozen backbone, which then answers the question from a context\-free prompt \(the*student*\)\. We train the parameter generator by distilling an*in\-context teacher*, the identical frozen model that reads context and question together, into this student\. The objective is a temperature\-scaled KL divergence between the teacher and student next\-token distributions over the shared answer tokens\. The backbone is frozen and shared across all paths, and only the generator is trainable\. Gradients \(dashed red\) flow back through the injected adapter to the generator alone\.

## 3Method

We presentCLIK, a framework that internalizes a context passage into the parameters of a frozen language model by generating a low\-rank adapter from that context in a single forward pass\. We study knowledge\-grounded question answering, where each example consists of a context passage, a question, and a target answer\. Placing the context in the prompt re\-encodes it for every query and inflates the input, so we instead internalize the context into the weights of the frozen language model and produce the answer from a context\-free prompt that contains only the question\. We learn a generator network that maps a context to a low\-rank adapter \(LoRA\)\([Hu et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib8)\)for the model’s feed\-forward projections\. The language model stays frozen, and the parameter generator is the only trainable component\.

We design the generator around three choices\. First, to obtain a strong representation of the context without training a separate encoder, we reuse the frozen backbone itself, so the context is encoded by the same model it will adapt and the conditioning signal lives natively in the model’s own representation space\. Second, to capture information ranging from low\-level surface form to high\-level semantics, we draw on the backbone’s hidden states from all of its layers and combine them through a learned weighting\. Third, to produce the high\-dimensional adapter efficiently, we treat the target adapter as a structured sequence of weight tokens and train a neural network transformer that performs message passing among them and predicts the adapter\.

As illustrated in[Figure2](https://arxiv.org/html/2610.38356#S2.F2), CLIK proceeds in three stages:

1. 1\.Context Encoding\.The frozen backbone reads the context in a single forward pass, yielding a sequence of per\-token hidden states at every layer\. We combine these multi\-layer states into a single conditioning sequence through a learnable, ELMo\([Peters et al\., 2018](https://arxiv.org/html/2610.38356#bib.bib30)\)\-style weighted sum, retaining the full token sequence as input to the generator\.
2. 2\.Parameter Generation\.The generator takes the context hidden states as input and predicts the target adapter\. It represents the adapter as a two\-dimensional grid of weight tokens, indexed by backbone layer and by within\-layer position, and applies factored self\-attention along each axis, which lets representations for different layers exchange information \(including from deep layers back to shallow ones\), together with cross\-attention to the context hidden states\.
3. 3\.Downstream Answering\.The generated adapter is injected into the frozen backbone, which then answers the question from a context\-free prompt, one that no longer contains the context\.

### 3\.1Training Data Construction

Since the backbone must answer from the generated adapter alone, the training corpus consists of self\-contained passages to be internalized\. We draw context passages from FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib37)\), a large pretraining corpus\. We prefer a pretraining\-scale, general\-domain corpus over a task\-specific QA dataset for two reasons\. Its diversity drives the generator toward a generic context\-to\-parameter mapping rather than a single benchmark’s distribution, and because it is tied to no benchmark, the generator is never trained on the evaluation tasks themselves, and its long free\-form answers keep every evaluation task out of distribution;[Section4\.4](https://arxiv.org/html/2610.38356#S4.SS4)measures the effect of adding in\-domain training data\. Our ablation confirms this choice, as FineWeb\-Edu surpasses task\-specific or smaller corpora such as SQuAD, MS MARCO, and TransMLA \([Table10](https://arxiv.org/html/2610.38356#S4.T10)\)\.

For each passage we synthesize several question–answer pairs as supervision, which teaches the generator both what to internalize and how it must be recalled while probing different facts within one context\. We generate these pairs with the same frozen backbone under a fixed system prompt, so the supervision follows the model’s native style, and this alignment is what makes the context\-free student a faithful approximation of the in\-context teacher it matches\. Because our goal is to reproduce in\-context behavior from a context\-free prompt, we impose no constraint on answer length or style, following Doc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24)\)\. For the same reason of generality, neither the teacher nor the student prompt contains an answer\-format instruction: the student prompt consists of the question alone and the teacher prompt of the context followed by the question, both wrapped in the backbone’s chat template\.

### 3\.2Context Encoding

The first stage turns the context into a conditioning signal for the generator\. Because the frozen backbone is itself a strong encoder, we run it over the context once and extract all intermediate hidden states\. A backbone withLLlayers producesL\+1L\+1hidden states \(the embedding output and each layer’s output\), denoted𝐇0,…,𝐇L\\mathbf\{H\}\_\{0\},\\dots,\\mathbf\{H\}\_\{L\}, each of shapen×dn\\times dfornncontext tokens and hidden dimensiondd\. The signal useful for adaptation is spread across depth, with shallow layers carrying surface cues and deeper ones more abstract content, so we condition on all layers rather than a single one and collapse theL\+1L\+1states into one representation with an ELMo\-style learned weighting,

𝐇¯=γ​∑ℓ=0Laℓ​𝐇ℓ,aℓ=exp⁡\(wℓ\)∑k=0Lexp⁡\(wk\),\\bar\{\\mathbf\{H\}\}\\;=\\;\\gamma\\sum\_\{\\ell=0\}^\{L\}a\_\{\\ell\}\\,\\mathbf\{H\}\_\{\\ell\},\\qquad a\_\{\\ell\}\\;=\\;\\frac\{\\exp\(w\_\{\\ell\}\)\}\{\\sum\_\{k=0\}^\{L\}\\exp\(w\_\{k\}\)\},\(1\)where the per\-layer weightsw0,…,wLw\_\{0\},\\dots,w\_\{L\}and scalarγ\\gammaare learned jointly with the generator\. Initializing everywℓw\_\{\\ell\}to zero andγ\\gammato one starts the combination as a uniform average, and the model then learns which depths to emphasize\. The aggregated𝐇¯\\bar\{\\mathbf\{H\}\}retains the full length\-nntoken sequence rather than being pooled, which gives the generator position\-specific information, and a linear projection maps each position fromddto the generator hidden dimensiondgd\_\{g\}to form the conditioning sequence\. This learned mixture avoids prematurely committing to where useful information resides, and our ablation confirms its value, as the last layer, a middle layer, or a uniform mean all underperform it \([Table10](https://arxiv.org/html/2610.38356#S4.T10)\)\.

### 3\.3Parameter Generator Pipeline

Our parameter generator follows the Neural Network Transformer introduced in HY\-WU\([Tencent HY Team, 2026](https://arxiv.org/html/2610.38356#bib.bib17)\), which we summarize here and adapt to our setting\. The generator emits a complete LoRA adapter in a single pass\. Because the adapted projections have different input and output dimensions across modules and layers, their factor matrices cannot be produced by one fixed\-size head without either a large parameter count or the loss of their structure\. We instead generate the adapter as a sequence of uniformly shaped weight tokens, refine these tokens with a Transformer that attends to the conditioning sequence from the previous stage, and detokenize them back into factor matrices\.

#### Weight tokenization\.

The rankrris shared by every adapter, whereas the channel dimensions are not, so every factor matrix hasrrtimes a module\-specific number of entries\. Therefore, we flatten each factor matrix and split it into consecutive chunks ofr⋅cr\\cdot centries, padding the last chunk when needed, and reshape each chunk into anr×cr\\times ctoken\. Each factor matrix thus becomes a set of tokens of identical shape\. Gathering the tokens of all adapted projections within a layer and stacking them over theLLlayers represents the adapter as a four\-axis array indexed by layer, within\-layer position, and the two axes of each token\. Detokenization inverts this layout to recover the factor matrices\.

#### Factorized attention\.

Full attention over the flattened array scales quadratically with its length and is impractical for an all\-layer adapter, so each block attends along one axis at a time\. An across\-layer step connects tokens that occupy the same role at different depths, and a within\-layer step connects the tokens of a single layer\. Together they model how each projection varies with depth and how co\-located projections interact\. A cross\-attention step then reads the conditioning sequence, which is the only way context information enters the tokens, since the tokens themselves are initialized from learned layer and within\-layer position embeddings\. We apply a separate rotary position encoding\([Su et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib40)\)to each axis so that every token retains its layer and position identity, while interleaving the steps with normalization and feed\-forward sublayers\.

#### Injection\.

After detokenization, each generated pair is inserted additively into its projection: a projection that computes𝐖𝐱\\mathbf\{W\}\\mathbf\{x\}instead computes

𝐖𝐱\+αr​𝐁𝐀​𝐱,\\mathbf\{W\}\\mathbf\{x\}\\;\+\\;\\frac\{\\alpha\}\{r\}\\,\\mathbf\{B\}\\mathbf\{A\}\\,\\mathbf\{x\},\(2\)with adapter scaleα\\alpha\. Each context yields its own𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, so the modification is applied per input, and gradients from the downstream loss reach the generator only through these terms\.

#### Training objective\.

We train the generator by context distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.38356#bib.bib41)\), matching a context\-free student to an in\-context teacher\. Both are the same frozen backbone: for a triple\(c,q,𝐚\)\(c,q,\\mathbf\{a\}\)with answer𝐚=\(a1,…,aN\)\\mathbf\{a\}=\(a\_\{1\},\\dots,a\_\{N\}\), the teacher reads the context in its prompt while the student carries it through the generated adapterΦ⁡\(c\)\\Phi\(c\)\. At answer positiontttheir next\-token distributions, each a softmax at temperatureτ\\tau, are

pt=pLMτ\(⋅∣𝐚<t,q,c\),qt=pLMτ\(⋅∣𝐚<t,q;Φ\(c\)\),p\_\{t\}=p\_\{\\mathrm\{LM\}\}^\{\\tau\}\(\\cdot\\mid\\mathbf\{a\}\_\{<t\},q,c\),\\qquad q\_\{t\}=p\_\{\\mathrm\{LM\}\}^\{\\tau\}\(\\cdot\\mid\\mathbf\{a\}\_\{<t\},q;\\,\\Phi\(c\)\),\(3\)and since the answer tokens are shared, they align position by position\. Treating the teacher \(computed with gradients disabled\) as a fixed target, we minimize their temperature\-scaled KL divergence over theNNanswer tokens,

ℒ=τ2N∑t=1NDKL\(pt∥qt\),\\mathcal\{L\}=\\frac\{\\tau^\{2\}\}\{N\}\\sum\_\{t=1\}^\{N\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{t\}\\,\\\|\\,q\_\{t\}\\right\),\(4\)where the directionpt\|qtp\_\{t\}\\,\\\|\\,q\_\{t\}draws the student toward the teacher’s full softened distribution rather than its top token alone, andτ2\\tau^\{2\}compensates for the temperature\-induced gradient scaling\. Only the generator is updated, and the gradient reaches it solely throughΦ⁡\(c\)\\Phi\(c\)\. As a simpler alternative that needs no teacher pass, the student can instead be trained with plain next\-token cross\-entropy on𝐚\\mathbf\{a\}, but we adopt distillation by default\. An optional penalty on the generated weights’ magnitude is available but kept disabled in our experiments\.

#### Inference\.

At inference time the context is consumed entirely by the generator and never enters the prompt\. Given a contextcc, we run the frozen backbone once to obtain the aggregated conditioning sequence𝐇¯​\(c\)\\bar\{\\mathbf\{H\}\}\(c\)of[Equation1](https://arxiv.org/html/2610.38356#S3.E1), pass it through the generatorGθG\_\{\\theta\}to produce the full set of adapter factors in a single forward pass,

Φ⁡\(c\)=\{\(𝐀\(m\),𝐁\(m\)\)\}m=Gθ​\(𝐇¯​\(c\)\),\\Phi\(c\)\\;=\\;\\bigl\\\{\(\\mathbf\{A\}^\{\(m\)\},\\mathbf\{B\}^\{\(m\)\}\)\\bigr\\\}\_\{m\}\\;=\\;G\_\{\\theta\}\\\!\\bigl\(\\bar\{\\mathbf\{H\}\}\(c\)\\bigr\),\(5\)and injectΦ⁡\(c\)\\Phi\(c\)into the backbone’s feed\-forward projections through[Equation2](https://arxiv.org/html/2610.38356#S3.E2)\. The adapted model then answers a questionqqfrom a context\-free prompt by greedy decoding from the question alone,

a^t=arg​maxa∈𝒱pLM\(a\|𝐚^<t,q;Φ\(c\)\),t=1,2,…,\\hat\{a\}\_\{t\}\\;=\\;\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{V\}\}\\;p\_\{\\mathrm\{LM\}\}\\\!\\bigl\(a\\,\\big\|\\,\\hat\{\\mathbf\{a\}\}\_\{<t\},\\,q;\\,\\Phi\(c\)\\bigr\),\\qquad t=1,2,\\dots,\(6\)until an end\-of\-sequence token is produced, where𝒱\\mathcal\{V\}is the vocabulary andpLM​\(⋅,Φ​\(c\)\)p\_\{\\mathrm\{LM\}\}\(\\,\\cdot\\,;\\Phi\(c\)\)denotes the frozen backbone with the generated adapter injected\. BecauseΦ⁡\(c\)\\Phi\(c\)depends only on the context, it is produced once and reused for every question about the same context, so the cost of internalization is paid a single time and amortized across queries, and the backbone itself never re\-reads the context\.

## 4Experiments

### 4\.1Experimental Settings

#### Implementation Details\.

We use Qwen3\-4B\-Instruct\([Yang et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib31)\)as the frozen base model throughout our experiments\. CLIK is built from a stack of1212Transformer blocks and produces LoRA adapters of rank1616for the FFN layers of the base model, i\.e\., for the gate, up, and down projections of all3636layers, which amounts to21\.221\.2M generated parameters \(40\.540\.5MiB in BF16\) per context\. The generator itself has1\.011\.01B parameters, of which604604M are in its attention sublayers and403403M in its feed\-forward sublayers \([Table13](https://arxiv.org/html/2610.38356#A1.T13)\); because this is a substantial capacity, we additionally compare against baselines at matched capacity, and report their training compute, in[Section4\.3](https://arxiv.org/html/2610.38356#S4.SS3)\. For training data, we use the FineWeb\-Edu 10B subset\([Penedo et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib37)\)as the context corpus, and for each context we prompt the backbone to generate55question–answer pairs, which together form the final training set\. The context fed to the generator is truncated to a maximum sequence length of20482048tokens during training\. We train CLIK for22epochs with the AdamW optimizer, using a peak learning rate of5×10−55\\times 10^\{\-5\}with linear warmup followed by decay\.

#### Baselines\.

We compare CLIK against the following baselines\.Basedenotes the frozen backbone that receives only the query, without the corresponding context\.Base \+ Contextadditionally provides the full context in the input and serves as a gold reference, since the model has direct access to all relevant information at inference time\. We further consider two methods that internalize the context into parameters\.Gen Adapter\([Chen et al\., 2025](https://arxiv.org/html/2610.38356#bib.bib15)\)internalizes the context into LoRA parameters at test time, andDoc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24)\)is a recent approach that trains a hypernetwork to generate, layer by layer, a separate LoRA module for each layer of the frozen backbone from a given context in a single forward pass, which is the setting closest to ours\. The original Doc\-to\-LoRA generates rank\-88adapters for the down projections only and is trained on a mixture that also includes SQuAD, DROP, and ROPES, so we re\-train it on the same FineWeb\-Edu question–answer pairs as CLIK with its own context\-distillation objective, and let it generate the same rank\-1616adapters on the gate, up, and down projections of all3636layers\.

Recent workSHINE\([Liu et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib25)\)also trains a hypernetwork that generates a LoRA from the backbone’s own context representation and is architecturally the closest method to ours\. The original SHINE uses Qwen3\-8B as the backbone, pretraining on TransMLA data followed by instruction fine\-tuning on question\-answering collections, and rank\-88adapters on both the attention and the feed\-forward projections, so in[Table1](https://arxiv.org/html/2610.38356#S4.T1)we re\-train it with Qwen3\-4B\-Instruct as the backbone and the FineWeb\-Edu question–answer pairs as training data, with its own next\-token prediction objective, and let it generate the same rank\-1616adapters as CLIK and Doc\-to\-LoRA\. Because SHINE reads each memory state directly as a slice of the adapter, the number of memory states per layer must cover the per\-layer adapter size; as589,824589\{,\}824is not a multiple of the hidden size25602560, we use231231memory states per layer and discard the1,5361\{,\}536surplus entries\. All generators in[Table1](https://arxiv.org/html/2610.38356#S4.T1)thus share the backbone, the training data, and the generated adapter, but still differ in generator size, training objective, and training compute, so[Section4\.3](https://arxiv.org/html/2610.38356#S4.SS3)additionally scales both baselines to the parameter count of CLIK and CLIK down to a much smaller generator\. For the cost analysis in[Section4\.6](https://arxiv.org/html/2610.38356#S4.SS6)we further includeprefix\-cached Base \+ Context\([Kwon et al\., 2023](https://arxiv.org/html/2610.38356#bib.bib42);[Zheng et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib43)\), KV\-cache compression withSnapKV\([Li et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib44)\), and prompt compression withLLMLingua\-2\([Pan et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib21)\)\.

#### CLIK\-B\.

The analysis of the generated adapters in[Section4\.9](https://arxiv.org/html/2610.38356#S4.SS9)shows that the gate and up updates and the input factor𝐀down\\mathbf\{A\}^\{\\mathrm\{down\}\}of the down projection vary little across contexts, whereas the output factor𝐁down∈ℝ2560×r\\mathbf\{B\}^\{\\mathrm\{down\}\}\\in\\mathbb\{R\}^\{2560\\times r\}carries most of the context\-dependent variation\. We therefore also evaluateCLIK\-B, a variant in which the gate and up adapters and the down input factors𝐀ℓdown\\mathbf\{A\}^\{\\mathrm\{down\}\}\_\{\\ell\}are ordinary trainable parameters shared across all contexts, and the generator produces only\{𝐁ℓdown​\(c\)\}ℓ=1L\\\{\\mathbf\{B\}^\{\\mathrm\{down\}\}\_\{\\ell\}\(c\)\\\}\_\{\\ell=1\}^\{L\}from𝐇¯​\(c\)\\bar\{\\mathbf\{H\}\}\(c\)\. Because the shared factors are the same for every context,𝐀ℓdown\\mathbf\{A\}^\{\\mathrm\{down\}\}\_\{\\ell\}can no longer be rotated per context, so every context must be expressed in one learned coordinate system\. The shared factors are initialized as in standard LoRA, with random𝐀\\mathbf\{A\}and zero𝐁\\mathbf\{B\}, and together with the zero\-initialized output head of the generator \([AppendixA](https://arxiv.org/html/2610.38356#A1)\) the adapter starts from zero exactly as in CLIK\. The per\-context state shrinks to36×2560×16=1\.4736\\times 2560\\times 16=1\.47M parameters \(2\.82\.8MiB in BF16\), a14\.4×14\.4\\timesreduction\. Withr×c=16×512r\\times c=16\\times 512weight tokens,𝐁ℓdown\\mathbf\{B\}^\{\\mathrm\{down\}\}\_\{\\ell\}fills exactly55tokens, so the generator produces55instead of7272weight tokens per layer \(2020instead of288288generator tokens\) and uses only its𝐁\\mathbf\{B\}output head, with all other components unchanged\. CLIK\-B also simplifies serving: the shared gate and up updates can be merged into the backbone weights once, and in a batch of requests with different contexts the down projection computes𝐖𝐱\+αr​𝐁ℓdown​\(ci\)​\(𝐀ℓdown​𝐱\)\\mathbf\{W\}\\mathbf\{x\}\+\\frac\{\\alpha\}\{r\}\\mathbf\{B\}^\{\\mathrm\{down\}\}\_\{\\ell\}\(c\_\{i\}\)\\,\(\\mathbf\{A\}^\{\\mathrm\{down\}\}\_\{\\ell\}\\mathbf\{x\}\), where𝐀ℓdown​𝐱\\mathbf\{A\}^\{\\mathrm\{down\}\}\_\{\\ell\}\\mathbf\{x\}is a single dense matrix multiplication for the whole batch and only the2560×r2560\\times rmatrix differs per request\. CLIK\-B is trained with the same objective, data, and number of training steps as CLIK\.

#### Evaluation Setup\.

We evaluate all methods on two groups of question\-answering datasets: the single\-hop datasets SQuAD\([Rajpurkar et al\., 2016](https://arxiv.org/html/2610.38356#bib.bib32)\), ROPES\([Lin et al\., 2019](https://arxiv.org/html/2610.38356#bib.bib34)\), and DROP\([Dua et al\., 2019](https://arxiv.org/html/2610.38356#bib.bib33)\), and the multi\-hop datasets HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2610.38356#bib.bib36)\), MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2610.38356#bib.bib38)\), and 2WikiMultihopQA\([Ho et al\., 2020](https://arxiv.org/html/2610.38356#bib.bib35)\)\. Performance is measured with F1 score and ROUGE\-L\. During inference, the context is removed from the input for all methods except the Base \+ Context setting, so that every other method must answer from the query alone, relying solely on the knowledge internalized into the parameters\.

Table 1:Main results on the single\-hop and multi\-hop question\-answering benchmarks with Qwen3\-4B\-Instruct\. We report F1 and ROUGE\-L under two evaluation protocols: without and with an instruction that constrains the output format; the backbone’s chat template is applied in both\. Except forBase \+ Context, which serves as a gold reference with direct access to the context, all methods answer from the query alone after internalizing the context into parameters\. The best result in each column is shown inboldand the second best isunderlined\(Base \+ Context is excluded as a gold reference\)\.Table 2:Decomposition of the single\-hop results on SQuAD, DROP, and ROPES into Precision, Recall, and F1 \(P / R / F1\), reported under both evaluation protocols\. Relative to Doc\-to\-LoRA, CLIK attains substantially higher precision, at comparable recall on SQuAD and lower recall on DROP and ROPES with the format instruction, yielding a markedly better precision–recall balance\.

### 4\.2Main Results

As the reference answers in our evaluation datasets are short, the formatting of a model’s response substantially influences span\-overlap metrics\. We accordingly evaluate under two protocols: one in which the question is given alone, and one in which it is preceded by an instruction that explicitly constrains the output format \(e\.g\.,*“Answer the following question\. Output only the answer and do not output any other words\.”*\)\. In both protocols the prompt is wrapped in the backbone’s chat template\. The two protocols are presented as separate blocks in[Table1](https://arxiv.org/html/2610.38356#S4.T1), and the F1 and ROUGE\-L scores display consistent trends\. CLIK is trained without an answer\-format instruction \([Section3\.1](https://arxiv.org/html/2610.38356#S3.SS1)\), so the first protocol matches its training format, whereas the second is out of distribution for the generator\.

Under the protocol without the format instruction, all methods, including Base \+ Context, produce long free\-form answers, so the scores are low and mostly reflect output length and format\. CLIK surpasses all context\-internalization baselines and, on several datasets, exceeds the Base \+ Context reference, but given how strongly this protocol is dominated by output format, we do not take this as evidence that CLIK matches direct access to the context\. Under the protocol with the format instruction, where the base model can fully exploit it, CLIK is the best context\-free method on all six datasets, with a clear margin in average F1 over both SHINE and Doc\-to\-LoRA, and it improves substantially over Doc\-to\-LoRA across all single\-hop datasets\. It nonetheless remains clearly behind Base \+ Context on every dataset, with the smallest gap on ROPES and the largest on HotpotQA\. Part of this gap stems from the train–test mismatch in prompt format noted above, which affects the generator but not Base \+ Context\. The margin over Doc\-to\-LoRA shows that CLIK preserves the base model’s instruction\-following capability after internalization rather than merely reproducing surface\-level content, even though the output\-format instruction is never seen during training\.

CLIK\-B, which generates only𝐁down\\mathbf\{B\}^\{\\mathrm\{down\}\}and shares all other factors across contexts, retains most of the performance of CLIK\. With the format instruction, it ranks second on every dataset, behind only CLIK and ahead of all baselines, and it stays closest to CLIK on SQuAD and ROPES\. The loss is larger on DROP and on the multi\-hop datasets, where answering requires combining several facts from the context and therefore relies more on the context\-dependent part of the factors that CLIK\-B shares\. Without the format instruction, CLIK\-B is comparable to the strongest baselines\. This makes CLIK\-B an attractive trade\-off when per\-context storage matters, as its per\-context state is more than an order of magnitude smaller than that of CLIK\.

[Table2](https://arxiv.org/html/2610.38356#S4.T2)further characterizes answer quality through precision and recall\. The context\-internalization baselines achieve high recall but low precision, producing outputs that cover the relevant content but remain loosely scoped\. CLIK departs from this pattern and substantially improves precision, at comparable recall on SQuAD and lower recall on DROP and ROPES with the format instruction, and it therefore attains a clearly better precision–recall balance on every dataset\. These results show that the improvements of CLIK do not come from longer or less constrained outputs, but from generating better\-scoped answers\. Whether these answers draw on the context or on the base model’s prior is examined separately in[Section4\.4](https://arxiv.org/html/2610.38356#S4.SS4)\.

### 4\.3Controlled Comparison at Matched Scale

The baselines in[Table1](https://arxiv.org/html/2610.38356#S4.T1)share the backbone, training data, and generated adapter with CLIK, but their generators differ from that of CLIK not only in architecture but also in size, so[Table1](https://arxiv.org/html/2610.38356#S4.T1)alone does not separate the contribution of the architecture from that of scale\. We therefore vary only the scale of the generators and keep everything else as in[Table1](https://arxiv.org/html/2610.38356#S4.T1): the same Qwen3\-4B\-Instruct backbone, the same FineWeb\-Edu question–answer pairs, rank1616adapters on the gate, up, and down projections of all3636layers, the same number of training steps, and the training objective of each method, i\.e\., distillation for Doc\-to\-LoRA and CLIK and next\-token prediction for SHINE \(details in[AppendixD](https://arxiv.org/html/2610.38356#A4)\)\. In[Table3](https://arxiv.org/html/2610.38356#S4.T3), D2L and SHINE denote these native\-size models, which are therefore those of[Table1](https://arxiv.org/html/2610.38356#S4.T1)\. To separate architecture from capacity, we scale each baseline to the parameter count of CLIK, denoted D2L\-L and SHINE\-L, and we evaluate a small CLIK\-S that reduces the generator from1212to66blocks and keeps everything else unchanged, which halves its parameter count and leaves it smaller than D2L\. D2L\-L widens the latent dimension of Doc\-to\-LoRA from512512to576576, and SHINE\-L deepens the M2P Transformer from44to1414layers\.[Table3](https://arxiv.org/html/2610.38356#S4.T3)reports generator size, per\-context adapter size, training compute, and F1 with the format instruction\. Matching parameters does not match compute: at the same number of steps CLIK needs several times the GPU\-hours of Doc\-to\-LoRA, because its generator applies all of its parameters to every weight token of the adapter for every context, whereas Doc\-to\-LoRA concentrates most of its parameters in an output head that is applied to only a few queries per layer, and the M2P Transformer of SHINE is much smaller\. Halving the depth, CLIK\-S also halves the generator compute of CLIK\.

Scaling the baselines to the size of CLIK does not close the gap\. D2L\-L remains essentially unchanged relative to D2L, and SHINE\-L improves on every dataset but stays clearly behind CLIK at the same parameter count\. In the other direction, CLIK\-S, with half the parameters of CLIK, outperforms both the larger D2L and SHINE of comparable size on every dataset, so the advantage of CLIK comes from its architecture rather than from its size\. CLIK\-B further shows that this advantage does not depend on a large training budget: it requires the least training compute of all generators and still achieves a higher average F1 than every baseline, including D2L\-L and SHINE\-L\.

Table 3:Controlled comparison of generator scale under matched backbone \(Qwen3\-4B\-Instruct\), training data \(FineWeb\-Edu\), adapter rank \(1616\), target modules \(gate, up, and down projections of all3636layers\), and number of training steps\. Each method keeps its training objective\. D2L and SHINE are Doc\-to\-LoRA and SHINE at native size, the models of[Table1](https://arxiv.org/html/2610.38356#S4.T1); D2L\-L and SHINE\-L enlarge the generator until it matches the parameter count of CLIK\. \# Gen\. is the number of generator parameters and \# Per\-context the number of generated parameters per context\. We report training GPU\-hours \([AppendixE](https://arxiv.org/html/2610.38356#A5)\) and F1 with the format instruction\.
### 4\.4Context Following of the Generated Adapter

A high score on SQuAD or HotpotQA does not by itself show that the generator internalizes the given context\. The generator could instead have adapted to the evaluation distribution, and the base model already answers part of these questions from its prior \(e\.g\., on ROPES, where Base is already competitive without any context\)\. We address the first concern with in\-domain training data and the second with counterfactual probes\.

#### In\-domain training data\.

The question–answer pairs synthesized from FineWeb\-Edu have long, free\-form answers \([Section3\.1](https://arxiv.org/html/2610.38356#S3.SS1)\), whereas the evaluation datasets have short reference answers, so every evaluation set is out of distribution for the generator\. To measure how much in\-domain supervision changes the results, we follow the data ablation of Doc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24)\)and add the passages and questions of the training splits of SQuAD, DROP, and ROPES to the training data, keeping the distillation objective and all other settings unchanged \([Table4](https://arxiv.org/html/2610.38356#S4.T4)\)\. In\-domain data yields only a modest gain\. The gain is largest on DROP, whose numerical and multi\-span answers differ most from the free\-form answers of FineWeb\-Edu, which matches the observation of[Charakorn et al\. \(2026\)](https://arxiv.org/html/2610.38356#bib.bib24), and it is small on SQuAD and ROPES, whose extractive and short reasoning answers the generator already handles well\. The results of CLIK therefore do not rely on training data that matches the evaluation distribution: training on the general\-domain corpus alone already transfers to the evaluation tasks, and in\-domain supervision mainly adapts the answer format of the task\.

Table 4:Effect of adding the training splits of SQuAD, DROP, and ROPES to the FineWeb\-Edu training data of CLIK\. F1 with the format instruction; all other settings are unchanged\.Table 5:Counterfactual probes of context following \(accuracy in %, with the format instruction\)\. For*Leak*and*Prior*, lower is better; for the wrong\-context swap, the ideal accuracy equals that of Base\.
#### Counterfactual probes\.

Neither benchmark scores nor in\-domain training can separate the internalized context from knowledge that the backbone already holds, so we construct probes whose correct answer can only come from the given context \(construction details in[AppendixF](https://arxiv.org/html/2610.38356#A6)\)\.

- •*Paired counterfactual documents\.*Two copies of a passage differ in a single fact, which is replaced by a nonce value \(e\.g\., a made\-up name or date\), and both are queried with the identical question\. We report the accuracy with respect to each adapter’s own context \(*Own*\), the fraction of pairs where both answers are correct \(*Both*\), and the fraction of answers that give the other document’s value \(*Leak*\)\.
- •*Novel facts\.*Synthetic passages about fictional entities that cannot occur in any pretraining corpus\.
- •*Strong prior conflict\.*Passages that contradict a well\-known fact \(e\.g\., relocating a famous landmark\); we report how often the answer follows the context \(*Ctx*\) and how often it follows the prior \(*Prior*\)\.
- •*Wrong\-context swap\.*The question is asked under the adapter generated from an unrelated passage, which carries no answer, so the ideal accuracy is that of Base, and we also report how often the answer uses content from the unrelated passage \(*Leak*\)\.
- •*Unrelated\-query interference\.*General\-knowledge questions unrelated to the passage are asked under the adapter; we report accuracy relative to Base \(*Retention*\), which measures collateral damage to the backbone\.

[Table5](https://arxiv.org/html/2610.38356#S4.T5)shows that CLIK follows the internalized context rather than the prior of the backbone\. Base stays at chance level on the paired counterfactual documents and on the novel facts, which confirms that these probes can only be solved from the context\. The probes that require writing the context into the adapter reproduce the ranking of[Table1](https://arxiv.org/html/2610.38356#S4.T1)\. On paired counterfactual documents, each adapter of CLIK answers with the nonce value of its own copy, and CLIK answers both copies of a pair correctly far more often than Doc\-to\-LoRA and SHINE, so changing a single fact in the context changes the answer accordingly\. On novel facts, CLIK recovers the answer from the adapter alone with the highest accuracy among the generators\. Under strong prior conflict, where Base always gives the original fact, CLIK follows the edited context more often than the prior, whereas Doc\-to\-LoRA and SHINE follow the prior more often than the context\.

The probes of specificity and collateral effects behave alike for all three generators, since all of them are trained to internalize the facts of a given context\. Their adapters almost never give the value of the other copy of a pair, stay close to the accuracy of Base under a wrong\-context swap and inject less content from the unrelated passage than Base \+ Context, and preserve the general knowledge of the backbone on unrelated queries\. The stronger internalization of CLIK therefore comes without any loss of specificity: its adapter carries the specific facts of its context and leaves the remaining behavior of the backbone intact\.

Figure 3:Multi\-turn evaluation of CLIK on the SQuAD, ROPES, and DROP datasets, comparing the single\-turn setting against two multi\-turn scenarios that differ in how conversational state is propagated\.*Top \(Dynamic KV cache\)*: the base model accumulates a KV cache across turns while the LoRA generated from the initial context is kept fixed\.*Bottom \(Dynamic LoRA\)*: the KV cache is discarded and CLIK regenerates the LoRA parameters at each turn from the updated context\. Both scenarios improve over the single\-turn baseline without any multi\-turn\-specific training, but neither has a per\-turn cost that is independent of the history \([Table7](https://arxiv.org/html/2610.38356#S4.T7)\)\.

### 4\.5Multi\-turn Evaluation

Beyond the single\-turn QA task examined above, we study the behavior of CLIK in realistic multi\-turn conversations to assess its effectiveness under conditions that more closely resemble practical deployment\. To construct such a setting from existing resources, we use the fact that each context in the QA datasets is associated with multiple questions, which we issue sequentially to form a multi\-turn dialogue grounded in a shared initial context\([Bai et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib39)\)\. Within this setup, we consider two complementary scenarios that differ in how information from preceding turns is propagated, namely*dynamic KV cache*and*dynamic LoRA*, and we additionally report an exploratory attempt at a*recurrent update*whose per\-turn cost does not grow with the history\.

#### Dynamic KV cache\.

In this scenario, the base model maintains a KV cache that accumulates across dialogue turns, while CLIK is responsible only for internalizing the initial contextc0c\_\{0\}into LoRA parameters\. At the first turn, the base model is conditioned on the questionq1q\_\{1\}alone and produces the answera1a\_\{1\}under the generated LoRA\. At the second turn, the pair\(q1,a1\)\(q\_\{1\},a\_\{1\}\)is provided together withq2q\_\{2\}, while the LoRA parameters remain fixed\. As shown in[Figure3](https://arxiv.org/html/2610.38356#S4.F3), this scheme yields consistent gains over the single\-turn setting\. We attribute these gains to an emergent transition toward in\-context learning, in which CLIK first enables the model to produce a correct answer and the accumulated dialogue history then acts as reliable in\-context evidence that improves the quality of subsequent responses\. This effect arises without any fine\-tuning on multi\-turn data, which confirms the robustness of CLIK with respect to the base model’s own outputs\. In this scenario the context is encoded only once, but the backbone prompt grows with the dialogue history\.

#### Dynamic LoRA\.

In this scenario, we further dispense with the KV cache entirely and instead allow CLIK to regenerate LoRA parameters dynamically as the conversation unfolds\. The first turn is identical to that of the dynamic KV cache scenario\. At the second turn, however, the initial contextc0c\_\{0\}and the pair\(q1,a1\)\(q\_\{1\},a\_\{1\}\)are recombined into an updated contextc1c\_\{1\}and passed in full through the parameter generator to synthesize new LoRA parameters, while the base model is still conditioned only onq2q\_\{2\}\. As reported in[Figure3](https://arxiv.org/html/2610.38356#S4.F3), this scheme likewise surpasses the single\-turn setting and achieves the strongest results under several configurations\. These results show that CLIK encodes the behavioral regularities that emerge during multi\-turn interaction, such as the output\-format constraints induced by the format instruction, directly into the LoRA parameters, and adapts them dynamically to refine its behavior over successive turns\. This scheme is, however, not “encode once”: the encoder and generator re\-read the original context plus the entire history at every turn, so their input, and hence the per\-turn cost, grows with the number of turns \([Table7](https://arxiv.org/html/2610.38356#S4.T7)\)\.

In summary, the advantages of CLIK extend from the single\-turn setting to multi\-turn dialogue under two distinct mechanisms for propagating conversational state\. Whether preceding turns are carried forward through the KV cache or re\-internalized into newly generated parameters, CLIK consistently improves upon the single\-turn baseline without any multi\-turn\-specific training\. Neither mechanism, however, has a per\-turn cost that is independent of the history, so the “encode once” amortization of CLIK applies to repeated queries against a fixed document rather than to a dialogue whose content keeps changing\.

#### An exploratory attempt: recurrent update\.

To obtain a per\-turn cost that is independent of the history, we explored a recurrent update in which the adapter of turnttis produced from the previous adapter and the new turnxt=\(qt,𝐚^t\)x\_\{t\}=\(q\_\{t\},\\hat\{\\mathbf\{a\}\}\_\{t\}\)only,

Φt=Gθ​\(𝐇¯​\(xt\),Φt−1\),Φ0=Gθ​\(𝐇¯​\(c0\)\)\.\\Phi\_\{t\}\\;=\\;G\_\{\\theta\}\\\!\\bigl\(\\bar\{\\mathbf\{H\}\}\(x\_\{t\}\);\\,\\Phi\_\{t\-1\}\\bigr\),\\qquad\\Phi\_\{0\}=G\_\{\\theta\}\\\!\\bigl\(\\bar\{\\mathbf\{H\}\}\(c\_\{0\}\)\\bigr\)\.\(7\)In CLIK the generator tokens are initialized from learned layer and within\-layer position embeddings \([Section3\.3](https://arxiv.org/html/2610.38356#S3.SS3)\)\. The recurrent update replaces this initialization with the weight tokens of the previous adapter:Φt−1\\Phi\_\{t\-1\}is tokenized as in[Section3\.3](https://arxiv.org/html/2610.38356#S3.SS3), which gives exactly the shape of the position embeddings, and the resulting tokens serve as the initial generator tokens, while the cross\-attention reads only the hidden states of the new turn\. The encoder and generator then process only the\|xt\|\|x\_\{t\}\|tokens of the new turn, regardless of the length of the dialogue, and the previous turns reach the new adapter only throughΦt−1\\Phi\_\{t\-1\}\. The update is applied to the trained CLIK generator as is, without any additional training\.

As shown in[Table6](https://arxiv.org/html/2610.38356#S4.T6), this attempt was not successful: the recurrent update is clearly worse than the single\-turn setting, let alone dynamic LoRA\. We attribute this to the gap between the two kinds of condition\. CLIK learns a text→\\toweight mapping in which the context enters only through cross\-attention to the backbone’s own hidden states, a representation that is already semantically organized, while the initial tokens are a fixed, context\-independent starting point that tells each token which layer and which position it will write\. TakingΦt−1\\Phi\_\{t\-1\}as a condition instead requires a weight→\\toweight mapping that the generator has never learned\. The weight tokens ofΦt−1\\Phi\_\{t\-1\}are consecutive chunks of flattened factor matrices rather than semantic units, they carry no explicit layer or position identity, and the same updateΔ​𝐖=𝐁𝐀\\Delta\\mathbf\{W\}=\\mathbf\{B\}\\mathbf\{A\}can be written by any factor pair\(𝐁𝐑,𝐑−1​𝐀\)\(\\mathbf\{B\}\\mathbf\{R\},\\mathbf\{R\}^\{\-1\}\\mathbf\{A\}\), so they are not a canonical encoding of what has been internalized\. Replacing the position embeddings with them therefore removes the starting point the generator relies on without supplying a condition it can read, which degrades even the internalization of the new turn and explains why the result falls below the single\-turn setting\. The existing text→\\tocondition pathway thus does not transfer directly to a weight→\\tocondition one without training\. Learning both jointly, i\.e\., a multi\-task generator trained to internalize a context from text and to update an existing adapter from new turns, for example with a dedicated input path for the previous adapter or with a gauge\-invariant encoding or a shared coordinate system as in CLIK\-B, in which adapters of different turns are directly comparable, is left to future work\.

Table 6:Exploratory recurrent update \([Equation7](https://arxiv.org/html/2610.38356#S4.E7)\) compared with the multi\-turn schemes above\. “Generator input” is the number of context tokens read by the encoder and generator at turntt\. F1 is averaged over turns11–55with the format instruction\.Table 7:Per\-turn cost and quality of multi\-turn scenarios\.\|c0\|\|c\_\{0\}\|is the length of the initial context,\|xi\|=\|qi\|\+\|ai\|\|x\_\{i\}\|=\|q\_\{i\}\|\+\|a\_\{i\}\|the length of turnii\. “Encoder / generator input” and “Backbone prompt” give the number of tokens processed at turntt\. Latency is the end\-to\-end time of turn55for\|c0\|=2048\|c\_\{0\}\|=2048\([AppendixE](https://arxiv.org/html/2610.38356#A5)\); F1 is averaged over turns11–55with the format instruction\.

### 4\.6Cost and Efficiency

Replacing a context by an adapter only pays off if the adapter is cheaper to keep and serve than the context\. We compare against the strongest context\-based alternatives, namely prefix caching, which prefills the context once and reuses its KV cache for every query, KV\-cache compression, and prompt compression, as well as Doc\-to\-LoRA and SHINE\.

#### Per\-context state\.

[Table8](https://arxiv.org/html/2610.38356#S4.T8)compares the state that must be stored per context\. With the adapter, the backbone answers from the query alone, so inference no longer needs the KV cache of the context, and the adapter takes its place as the per\-context state\. The rank\-1616CLIK adapter is as large as the KV cache of a context of a few hundred tokens, so for the long documents that context internalization targets it is several times smaller than the KV cache it removes, and only for short contexts is it the larger of the two; in every case it has to be loaded and switched per request\. CLIK\-B shrinks the per\-context state by more than an order of magnitude, to the size of the KV cache of only tens of tokens\.

Table 8:Per\-context state in BF16 for Qwen3\-4B, wherennis the number of context tokens\. “KV tokens” expresses each size as the number of tokens whose full KV cache occupies the same memory \(144144KiB per token\)\.
#### Latency, memory, and throughput\.

[Table9](https://arxiv.org/html/2610.38356#S4.T9)reports inference costs for contexts of20482048tokens \(setup in[AppendixE](https://arxiv.org/html/2610.38356#A5)\)\.*Inter\. \(Internalization\)*is the one\-time cost per context \(prefill for context\-based methods, encoder plus generator pass for adapter\-based methods\),*TTFT*is the per\-query time to first token after internalization,*Switch*is the time to load a stored per\-context state from host memory,*TP\. \(Throughput\)*is the decoding throughput for a batch of3232requests with3232distinct contexts, using segmented multi\-adapter kernels\([Chen et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib46);[Sheng et al\., 2024](https://arxiv.org/html/2610.38356#bib.bib45)\)for the adapter\-based methods, and*NQ\. \(New question\)*is the added latency of one further question on the same document: the TTFT when the internalized state is reused, and the internalization time plus the TTFT when SnapKV selects the cache again\.*Memory*refers to this batched run and includes the backbone, any generator or compressor, the3232per\-context states, and the transient memory of one internalization\.

Internalization is more expensive for CLIK than prefilling the context, because the generator adds a pass over all weight tokens of the adapter, whereas for CLIK\-B this extra cost is small\. After internalization, the time to first token of the adapter\-based methods is close to that of prefix caching, since for a single request both are dominated by reading the backbone weights\. The advantages of the adapter lie in memory: because the backbone no longer needs the KV cache of the context, each request holds only its adapter and the KV cache of its own query and answer\. The adapter is several times smaller than the context KV cache it removes and correspondingly faster to load, and a batch of requests with distinct contexts decodes markedly faster with lower peak memory\. SnapKV reaches a similar throughput by reading its retained KV cache, and the internalization time counts that selection once\. A further question selects the cache again, so its added latency is that internalization plus the TTFT\. An adapter is generated once for the document and then reused, so a further question costs only the TTFT\. LLMLingua\-2 compresses the context once and pays its TTFT on every query, which already includes prefilling the compressed prompt\.

Table 9:Inference cost for20482048\-token contexts \([AppendixE](https://arxiv.org/html/2610.38356#A5)\)\.*New question*is the added latency of one further question on the same document, taken from the other columns: the TTFT when the internalized state is reused, and the internalization time plus the TTFT for SnapKV, which selects its cache again\. Avg\. F1 is over the six datasets with the format instruction\.

### 4\.7Ablation Study on Hyper\-parameters

Table 10:Ablation studies on SQuAD, reported as Precision / Recall / F1 \(P / R / F1\)\. We vary three factors independently: the hidden state read by the cross\-attention, the peak learning rate, and the training corpus\. The configuration adopted in our final model is highlighted inbold\.#### Hidden state for cross\-attention\.

We first ablate which hidden representation the cross\-attention reads from when generating the LoRA\. Relying on a single fixed layer is suboptimal, as the last\-layer representation performs poorly, and although the middle layer and a uniform mean over layers are clearly better, both still remain below the best configuration\. Replacing the fixed choice with an ELMo\-style learnable mixture of the hidden states from all layers performs best, which shows that the most useful contextual signal is distributed across depth and that letting the model learn how to combine layers is preferable to committing to any single one\.

#### Learning rate\.

The learning rate controls how quickly the generator converges\. A very small rate underfits within our training budget, and performance improves as the rate increases, peaking at an intermediate value\. A larger rate speeds up convergence, but increasing it further no longer helps and slightly degrades the result, which reflects the instability that accompanies an overly aggressive step size\. We therefore adopt the best\-performing rate, which converges quickly while remaining stable\.

#### Training dataset\.

Finally, we vary the corpus used to train the generator\. A short\-answer dataset such as SQuAD yields the lowest F1, with a particularly low recall, even though it shares the answer format of the evaluation, while larger and more general sources help progressively\. The best result comes from FineWeb\-Edu, showing that increasing the scale and diversity of the training data, rather than matching the answer format of the evaluation, is what truly drives performance\. This holds even though the QA pairs derived from FineWeb\-Edu contain long answers\. We deliberately avoid training on a fixed short\-answer dataset like SQuAD and do not impose any constraint on the base model’s outputs, because our goal is to compress and preserve the base model’s native ability as faithfully as possible rather than to specialize it toward a particular response style\.

### 4\.8Ablation of the Generator Architecture

[Table10](https://arxiv.org/html/2610.38356#S4.T10)studies input and training choices, but the component that distinguishes CLIK from prior generators is the interaction among weight tokens\. We therefore ablate it \([Table11](https://arxiv.org/html/2610.38356#S4.T11)\):*across\-layer attention*and*within\-layer attention*are removed individually and jointly\. Removing both turns CLIK into a generator whose weight tokens interact only through the shared context, i\.e\., independent per\-fragment generation as in prior work\. To keep capacity comparable, we adjust the feed\-forward width of every variant so that its parameter count stays within5%5\\%of the full model \(12,28812\{,\}288when one attention sublayer per block is removed and16,38416\{,\}384when both are removed\)\.

Both forms of interaction contribute to the performance of CLIK\. Removing across\-layer attention lowers every metric, and removing within\-layer attention causes a much larger drop, which shows that the weight tokens of one layer must be generated in coordination to form a usable adapter\. Removing both is far more harmful than removing either one alone, so the two sublayers complement each other\. The resulting independent\-fragment generator falls to the level of Doc\-to\-LoRA in[Table1](https://arxiv.org/html/2610.38356#S4.T1), even though it keeps the cross\-attention and the parameter count of CLIK\. The loss is concentrated in precision and is largest on HotpotQA, whose questions require combining several facts of the context\. The interaction among weight tokens is therefore the key architectural component of CLIK\.

Table 11:Ablation of the weight\-token interaction in the generator\. All variants are trained under the same data and number of steps and are parameter\-matched to within5%5\\%\. We report P / R / F1 on SQuAD, F1 on HotpotQA, and the average F1 over the six datasets, all with the format instruction\.Figure 4:Per\-layer mean cross\-contextcos⁡\(Δ​W\)\\cos\(\\Delta W\)for the gate, up, and down projections, comparing an early \(10k iters\) and a late \(256k iters\) checkpoint of CLIK\. Higher values denote updates that are more consistent across different contexts\.\(a\)Per\-modulecos⁡\(Δ​W\)\\cos\(\\Delta W\)at the early versus late checkpoint, split by module kind\.\(b\)Off\-diagonalcos⁡\(Δ​W\)\\cos\(\\Delta W\)for the down projection over all context pairs, early versus late\.\(c\)Per\-kind change incos⁡\(Δ​W\)\\cos\(\\Delta W\)from early to late \(mean±\\pmstd\)\.
Figure 5:Distributional view of the generated updateΔ​W=\(α/r\)​B​A\\Delta W=\(\\alpha/r\)BA, comparing the early and late checkpoints\. The cross\-context cosinecos⁡\(Δ​W\)\\cos\(\\Delta W\)measures how consistent the update is across contexts: high values indicate a shared, context\-independent component, while low values indicate context\-specific adaptation\. Because it is computed on the productB​ABA, it is invariant to the reparameterization\(B,A\)↦\(B​R,R−1​A\)\(B,A\)\\mapsto\(BR,R^\{\-1\}A\)\.
### 4\.9Analysis of LoRA Patterns

#### Training Dynamics\.

To understand what CLIK learns, we compare the adapters it generates at an early and a late training checkpoint, analyzing the effective updateΔ​W=\(α/r\)​B​A\\Delta W=\(\\alpha/r\)BAover a set of contexts\. Two trends are evident\. First, training increases the magnitude of the adaptation: the mean Frobenius norm∥Δ​W∥F\\lVert\\Delta W\\rVert\_\{F\}grows markedly from the early to the late checkpoint, showing that the generator learns to apply progressively stronger updates rather than converging to a conservative solution\. Second, the updates develop a clear structure across contexts rather than collapsing onto a single shared adapter\. While the input\-side \(gate and up\) projections become increasingly consistent across contexts, the output\-side \(down\) projections move in the opposite direction, and as shown in[Figure5\(b\)](https://arxiv.org/html/2610.38356#S4.F5.sf2), the distribution of the down\-projection cross\-context cosine shifts noticeably toward*lower*values from the early to the late checkpoint\. This shows that CLIK gradually develops a shared, largely context\-independent backbone in its input\-side adapters, while pushing the down projections to carry progressively more context\-specific information\.

This consolidation is not uniform across module types, and instead reveals a clear functional specialization \([Figures4](https://arxiv.org/html/2610.38356#S4.F4),[5\(a\)](https://arxiv.org/html/2610.38356#S4.F5.sf1)and[5\(c\)](https://arxiv.org/html/2610.38356#S4.F5.sf3)\)\. The gate and up projections exhibit high cross\-context cosine throughout the network, and this value increases slightly with training, showing that the generator converges toward a stable, nearly context\-independent transformation for these input\-side modules\. In contrast, the down projections show markedly lower cosine that is strongly layer\-dependent, and, unlike the other two kinds, their cross\-context cosine*decreases*with training \(mirroring the leftward shift of the down\-projection distribution in[Figure5\(b\)](https://arxiv.org/html/2610.38356#S4.F5.sf2)\), with the largest changes concentrated in the deep layers\. The scatter plot mirrors this pattern, as gate and up modules cluster tightly near the upper\-right corner above the identity line, whereas down modules are widely dispersed and frequently fall below it\. Overall, these observations show that CLIK learns a division of labor in which the input\-side projections form a consistent, shared backbone of adaptation, while the output\-side \(down\) projections, particularly in deeper layers, carry the context\-specific information needed to internalize each individual context\. Because all of these quantities are computed onΔ​W\\Delta W, they are invariant to the reparameterization\(𝐁,𝐀\)↦\(𝐁𝐑,𝐑−1​𝐀\)\(\\mathbf\{B\},\\mathbf\{A\}\)\\mapsto\(\\mathbf\{B\}\\mathbf\{R\},\\mathbf\{R\}^\{\-1\}\\mathbf\{A\}\)for any invertible𝐑∈ℝr×r\\mathbf\{R\}\\in\\mathbb\{R\}^\{r\\times r\}and do not depend on how the update is split into factors\.

#### Internalized Knowledge Structure in LoRA\.

To examine how the internalized information is distributed over the factors of the adapter, we intervene on the generated adapter before inference and measure the resulting change on SQuAD \([Table12](https://arxiv.org/html/2610.38356#S4.T12)\)\. SubstitutingdownA\\mathrm\{down\}\_\{A\}with the factor generated for a different context leaves performance essentially unchanged, whereas applying the same substitution todownB\\mathrm\{down\}\_\{B\}causes a near\-complete collapse, and swapping the full adapter is only marginally more damaging than swappingdownB\\mathrm\{down\}\_\{B\}alone\. Substituting the gate and up factors across contexts, by contrast, has only a minor effect, which shows that these input\-side factors are largely shared rather than context\-dependent\. Unlike the analyses above, these single\-factor interventions depend on the coordinate system the generator uses\. ThedownA\\mathrm\{down\}\_\{A\}result is meaningful even so: in the generator’s own coordinates,downA\\mathrm\{down\}\_\{A\}is interchangeable across contexts\. ThedownB\\mathrm\{down\}\_\{B\}result, however, does not by itself show that the context is stored indownB\\mathrm\{down\}\_\{B\}, because pairing thedownB\\mathrm\{down\}\_\{B\}of one context with thedownA\\mathrm\{down\}\_\{A\}of another combines factors expressed in different coordinates\.

Table 12:Intervention study on SQuAD \(Precision / Recall / F1\)\. Each generated adapter is perturbed before inference\.ReplacingdownB\\mathrm\{down\}\_\{B\}with scale\-matched Gaussian noise or randomly permuting its entries is as harmful as removing the adaptation altogether, which shows that the specific learned values matter rather than the overall magnitude or sparsity pattern\. Removing the gate and up factors is likewise detrimental, because although they are not context\-specific, the rest of the adapter was generated jointly with them and is not usable without them\. Finally, ablating groups of layers shows that the internalized context is not confined to one depth band\. The two groups are removed separately, and the layers between them as well as the last layer are kept in both cases\. Removing the deeper layers is slightly more harmful in F1 than removing the shallower layers, and the two removals fail differently: removing the shallower layers mainly lowers recall, whereas removing the deeper layers mainly lowers precision while keeping recall comparatively high\. Overall, these results show that the context\-dependent part of the adapter is compact and specific: the gate and up updates are largely shared across contexts,downA\\mathrm\{down\}\_\{A\}is interchangeable across contexts in the generator’s coordinates, and the context\-dependent variation is expressed through the down projection\.

#### From analysis to architecture\.

Neither the gauge\-invariant analyses nor the factor interventions establish that the knowledge intrinsically resides indownB\\mathrm\{down\}\_\{B\}\. The constructive test is CLIK\-B, which fixes a single learned coordinate system shared by all contexts and generates onlydownB\\mathrm\{down\}\_\{B\}\. Its results in[Tables1](https://arxiv.org/html/2610.38356#S4.T1)and[3](https://arxiv.org/html/2610.38356#S4.T3)show how much of the performance of CLIK a per\-contextdownB\\mathrm\{down\}\_\{B\}alone can retain, at a much smaller per\-context state\. CLIK\-B retains most of the performance of CLIK, outperforms all baselines of[Table1](https://arxiv.org/html/2610.38356#S4.T1)under the format instruction, and achieves a higher average F1 than D2L\-L and SHINE\-L in[Table3](https://arxiv.org/html/2610.38356#S4.T3), which confirms that a single shared coordinate system with a per\-contextdownB\\mathrm\{down\}\_\{B\}captures most of the internalized context\. The remaining gap, which is largest on DROP and the multi\-hop datasets, shows that the gate, up, anddownA\\mathrm\{down\}\_\{A\}factors of CLIK also contribute context\-specific information\.

## 5Conclusion

We presentedCLIK, a parameter generator that adapts a frozen LLM by mapping a context directly to LoRA parameters in a single forward pass\. Rather than keeping task knowledge in the input or optimizing the model separately for each task, CLIK internalizes the context into the parameters with a Transformer that attends from the context to the adapter weights through cross\-attention\. Once the generated adapter is inserted, downstream queries are answered from the query alone, without re\-reading the original context and without any test\-time training\. On document question answering, CLIK improves clearly over the no\-context base model and compares favorably with existing hypernetwork\-based methods under the same setting, while remaining behind the base model that reads the context directly and avoiding the per\-task optimization of fine\-tuning\. Counterfactual probes confirm that the generated adapter answers from the internalized context rather than from the prior of the backbone\. The same mechanism also carries over to multi\-turn dialogue, where CLIK improves over its single\-turn baseline without additional training\.

#### Limitations and future work\.

Our study has limitations that define directions for future work\. We evaluate primarily on document question answering, so the behavior of CLIK on broader task types, longer contexts, and larger backbones is not examined in this work\. The generated adapter also reflects whatever information the context provides, and we do not yet model how to retain or reconcile knowledge across many successive updates\. We view context\-to\-parameter generation as a practical step toward adaptation that keeps the flexibility of in\-context learning while internalizing knowledge as fine\-tuning does, and as a basis for further work on efficient, incremental adaptation of LLMs\.

## References

- Abdallaet al\.\(2025\)M\. H\. I\. Abdalla, Z\. Wang, C\. Frey, S\. Eger, and J\. GrabockaZhyper: factorized hypernetworks for conditioned LLM fine\-tuning\.arXiv preprint arXiv:2510\.19733\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Askellet al\.\(2021\)A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma,et al\.A general language assistant as a laboratory for alignment\.arXiv preprint arXiv:2112\.00861\.Cited by:[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Baiet al\.\(2024\)G\. Bai, J\. Liu, X\. Bu, Y\. He, J\. Liu, Z\. Zhou, Z\. Lin, W\. Su, T\. Ge, B\. Zheng,et al\.MT\-Bench\-101: a fine\-grained benchmark for evaluating large language models in multi\-turn dialogues\.InACL,Cited by:[§4\.5](https://arxiv.org/html/2610.38356#S4.SS5.p1.1)\.
- Baoet al\.\(2026\)W\. Bao, H\. Wang, J\. Wang, Z\. Wang, K\. Wang, and Y\. ShangGood agentic friends do not just give verbal advice: they can update your weights\.arXiv preprint arXiv:2605\.13839\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Charakornet al\.\(2026\)R\. Charakorn, E\. Cetin, S\. Uesaka, and R\. T\. LangeDoc\-to\-LoRA: learning to instantly internalize contexts\.InICML,Cited by:[Appendix C](https://arxiv.org/html/2610.38356#A3.p1.1),[§1](https://arxiv.org/html/2610.38356#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.38356#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2610.38356#S4.SS4.SSS0.Px1.p1.1)\.
- Chenet al\.\(2024\)L\. Chen, Z\. Ye, Y\. Wu, D\. Zhuo, L\. Ceze, and A\. KrishnamurthyPunica: multi\-tenant LoRA serving\.InMLSys,Cited by:[§4\.6](https://arxiv.org/html/2610.38356#S4.SS6.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025\)T\. Chen, H\. Fang, P\. Xia, X\. Liu, B\. Van Durme, L\. Zettlemoyer, J\. Gao, and H\. ChengGenerative adapter: contextualizing language models in parameters with a single forward pass\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p1.1)\.
- Duaet al\.\(2019\)D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. GardnerDROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InNAACL,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Geet al\.\(2024\)S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. GaoModel tells you what to discard: adaptive KV cache compression for LLMs\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Haet al\.\(2017\)D\. Ha, A\. Dai, and Q\. V\. LeHyperNetworks\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§3\.3](https://arxiv.org/html/2610.38356#S3.SS3.SSS0.Px4.p1.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. AizawaConstructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InCOLING,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Honget al\.\(2025\)K\. Hong, A\. Troynikov, and J\. HuberContext rot: how increasing input tokens impacts LLM performance\.Note:[https://research\.trychroma\.com/context\-rot](https://research.trychroma.com/context-rot)Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1)\.
- Hotskoet al\.\(2026\)L\. Hotsko, Y\. Li, Y\. Deng, and P\. NieCode2LoRA: hypernetwork\-generated adapters for code language models under software evolution\.arXiv preprint arXiv:2606\.06492\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1),[§3](https://arxiv.org/html/2610.38356#S3.p1.1)\.
- Jacobet al\.\(2026\)D\. C\. Jacob, X\. Liu, K\. Wang, and P\. HeCOPRA: conditional parameter adaptation with reinforcement learning for video anomaly detection\.arXiv preprint arXiv:2605\.15325\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Jianget al\.\(2023\)H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. QiuLLMLingua: compressing prompts for accelerated inference of large language models\.InEMNLP,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Kujanpääet al\.\(2025\)K\. Kujanpää, P\. Marttinen, H\. Valpola, and A\. IlinEfficient knowledge injection in LLMs via self\-distillation\.TMLR\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InSOSP,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p2.1)\.
- Labanet al\.\(2026\)P\. Laban, H\. Hayashi, Y\. Zhou, and J\. NevilleLLMs get lost in multi\-turn conversation\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§1](https://arxiv.org/html/2610.38356#S1.p6.1)\.
- Liet al\.\(2025a\)T\. Li, G\. Zhang, Q\. D\. Do, X\. Yue, and W\. ChenLong\-context LLMs struggle with long in\-context learning\.TMLR\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1)\.
- Liet al\.\(2025b\)Y\. Li, X\. Shen, Y\. Miao, X\. Yao, X\. Ding, R\. Krishnan, and R\. PadmanBeyond single\-turn: a survey on multi\-turn interactions with large language models\.arXiv preprint arXiv:2504\.04717\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p3.1),[§1](https://arxiv.org/html/2610.38356#S1.p6.1)\.
- Liet al\.\(2024\)Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. ChenSnapKV: LLM knows what you are looking for before generation\.InNeurIPS,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p2.1)\.
- Lianget al\.\(2025\)Z\. Liang, D\. Tang, Y\. Zhou, X\. Zhao, M\. Shi, W\. Zhao, Z\. Li, P\. Wang, K\. Schürholt, D\. Borth,et al\.Drag\-and\-drop LLMs: zero\-shot prompt\-to\-weights\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p4.1)\.
- Linet al\.\(2019\)K\. Lin, O\. Tafjord, P\. Clark, and M\. GardnerReasoning over paragraph effects in situations\.InMRQA Workshop,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.TACL\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1)\.
- Liuet al\.\(2026\)Y\. Liu, X\. Wang, Y\. Mao, Y\. Gelbery, H\. Maron, and M\. ZhangSHINE: a scalable in\-context hypernetwork for mapping context to LoRA in a single pass\.InICML,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1)\.
- Panet al\.\(2024\)Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin,et al\.LLMLingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.InFindings of ACL,Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p2.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, T\. Wolf,et al\.The FineWeb datasets: decanting the web for the finest text data at scale\.InNeurIPS,Cited by:[§3\.1](https://arxiv.org/html/2610.38356#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px1.p1.1)\.
- Peterset al\.\(2018\)M\. E\. Peters, M\. Neumann, M\. Iyyer, M\. Gardner, C\. Clark, K\. Lee, and L\. ZettlemoyerDeep contextualized word representations\.InNAACL,Cited by:[item 1](https://arxiv.org/html/2610.38356#S3.I1.i1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.InEMNLP,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Shenget al\.\(2024\)Y\. Sheng, S\. Cao, D\. Li, C\. Hooper, N\. Lee, S\. Yang, C\. Chou, B\. Zhu, L\. Zheng, K\. Keutzer, J\. E\. Gonzalez, and I\. StoicaS\-LoRA: serving thousands of concurrent LoRA adapters\.InMLSys,Cited by:[§4\.6](https://arxiv.org/html/2610.38356#S4.SS6.SSS0.Px2.p1.1)\.
- Snellet al\.\(2022\)C\. Snell, D\. Klein, and R\. ZhongLearning by distilling context\.arXiv preprint arXiv:2209\.15189\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Suet al\.\(2024\)J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. LiuRoFormer: enhanced transformer with rotary position embedding\.Neurocomputing\.Cited by:[§3\.3](https://arxiv.org/html/2610.38356#S3.SS3.SSS0.Px2.p1.1)\.
- Suriet al\.\(2026\)M\. Suri, S\. Baskar, and D\. ManochaVideo2LoRA: parametric video internalization for vision\-language models\.arXiv preprint arXiv:2606\.04351\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Tencent HY Team \(2026\)Tencent HY TeamHY\-WU \(Part I\): an extensible functional neural memory framework and an instantiation in text\-guided image editing\.arXiv preprint arXiv:2603\.07236\.Cited by:[Appendix G](https://arxiv.org/html/2610.38356#A7.SS0.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2610.38356#S3.SS3.p1.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: multihop questions via single\-hop question composition\.TACL\.Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024\)K\. Wang, D\. Tang, B\. Zeng, Y\. Yin, Z\. Xu, Y\. Zhou, Z\. Zang, T\. Darrell, Z\. Liu, and Y\. YouNeural network diffusion\.arXiv preprint arXiv:2402\.13144\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Wanget al\.\(2025\)K\. Wang, D\. Tang, W\. Zhao, K\. Schürholt, Z\. Wang, and Y\. YouScaling up parameter generation: a recurrent diffusion approach\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, P\. Wang, and K\. WangPosition: weight space should be a first\-class generative AI modality\.arXiv preprint arXiv:2605\.18632\.Cited by:[§2\.2](https://arxiv.org/html/2610.38356#S2.SS2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InEMNLP,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2026\)S\. Zhang, L\. Dong, X\. Li, S\. Zhang, X\. Sun, S\. Wang, J\. Li, R\. Hu, T\. Zhang, G\. Wang,et al\.Instruction tuning for large language models: a survey\.ACM Computing Surveys\.Cited by:[§1](https://arxiv.org/html/2610.38356#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.38356#S2.SS1.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. ShengSGLang: efficient execution of structured language model programs\.InNeurIPS,Cited by:[§4\.1](https://arxiv.org/html/2610.38356#S4.SS1.SSS0.Px2.p2.1)\.

## Appendix AParameter Generator Configuration

We instantiate the parameter generator described in[Section3\.3](https://arxiv.org/html/2610.38356#S3.SS3)for the Qwen3\-4B backbone\.[Table13](https://arxiv.org/html/2610.38356#A1.T13)lists the full configuration; below we relate it to the notation used in the main text\.

#### Adapted modules\.

The generator adapts only the three feed\-forward projections of every transformer layer—the gate, up, and down projections of the MLP block—while the attention projections are left untouched\. With a backbone ofL=36L=36layers, this yields3​L3Ladapted projections\. Each adapter has rankr=16r=16and scaleα=32\\alpha=32\(soα/r=2\\alpha/r=2in[Equation2](https://arxiv.org/html/2610.38356#S3.E2)\)\. Following the backbone’s hidden sized=2560d=2560and feed\-forward intermediate size97289728, the gate and up projections receive factors𝐀∈ℝr×2560\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times 2560\},𝐁∈ℝ9728×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{9728\\times r\}, and the down projection receives𝐀∈ℝr×9728\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times 9728\},𝐁∈ℝ2560×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{2560\\times r\}\. Each projection thus carriesr⁡\(2560\+9728\)=196,608r\\,\(2560\+9728\)=196\{,\}608generated parameters, and the full adapter carries36×3×196,608=21,233,664≈21\.236\\times 3\\times 196\{,\}608=21\{,\}233\{,\}664\\approx 21\.2M parameters per context, i\.e\.,40\.540\.5MiB in BF16\. For CLIK\-B only the down\-projection𝐁\\mathbf\{B\}factors are generated, i\.e\.,36×2560×16=1,474,56036\\times 2560\\times 16=1\{,\}474\{,\}560parameters \(2\.82\.8MiB\), a14\.4×14\.4\\timesreduction\. Rank1616is used in all experiments\.

#### Conditioning\.

The all\-layer backbone hidden states \([Section3\.2](https://arxiv.org/html/2610.38356#S3.SS2)\), of dimensiond=2560d=2560, are combined by the learned ELMo weighting and linearly projected to the generator widthdg=2048d\_\{g\}=2048, which is the dimension at which the generator operates\.

#### Generator transformer\.

The generator is a1212\-block Neural Network Transformer with hidden widthdg=2048d\_\{g\}=2048and attention head dimension128128\(i\.e\.1616heads\), using the factorized across\-layer / within\-layer / cross\-attention pattern of[Section3\.3](https://arxiv.org/html/2610.38356#S3.SS3)\. Weight tokens have sizer×c=16×512r\\times c=16\\times 512, so each layer is covered by7272tokens \(55for each of the three factors with a dimension of25602560and1919for each of the three factors with a dimension of97289728\)\. Inside the generator, every four consecutive rows of a token are concatenated into onedgd\_\{g\}\-dimensional vector \(channel accumulation44\), which yields288288generator tokens per layer\. These tokens are initialized as the sum of a learned within\-layer and a learned layer position embedding, and after the last block two output heads, one for𝐀\\mathbf\{A\}and one for𝐁\\mathbf\{B\}tokens, project each row back to the512512\-dimensional weight space that is detokenized into the factor matrices\. The𝐁\\mathbf\{B\}head is initialized to zero, so that the generated adapter is zero at the start of training, and the output is scaled by10−210^\{\-2\}\.

#### Generator size\.

The generator is large relative to the adapter it produces\. Each block contains an across\-layer, a within\-layer, and a cross\-attention sublayer, which together account for604604M parameters over the1212blocks, and a non\-gated SiLU feed\-forward sublayer of width4​dg=81924d\_\{g\}=8192, which accounts for12×2×2048×8192≈40312\\times 2\\times 2048\\times 8192\\approx 403M parameters\. With the input projection, the position embeddings, the output heads, and the normalization layers, the generator has1\.011\.01B parameters in total \([Table13](https://arxiv.org/html/2610.38356#A1.T13)\), which is the quantity matched against the baselines in[Section4\.3](https://arxiv.org/html/2610.38356#S4.SS3)\.

Table 13:Parameter generator configuration for the Qwen3\-4B backbone\.

## Appendix BData Generation Configuration

To construct the training corpus described in[Section3\.1](https://arxiv.org/html/2610.38356#S3.SS1), we prompt the frozen backbone to synthesize question–answer pairs grounded in each FineWeb\-Edu passage\. The generation is driven by a fixed system prompt \([Listing1](https://arxiv.org/html/2610.38356#listing1)\) that frames the task as reading\-comprehension question generation, together with a user instruction template \([Listing2](https://arxiv.org/html/2610.38356#listing2)\) that injects the passage and specifies the formatting rules for both questions and answers\. We deliberately leave the answer length and response style unconstrained, so that the synthesized pairs follow the model’s native in\-context behavior\.

SYSTEM\_PROMPT=\(

"Youareacreativeandhelpfulassistant\.\\n"

"Youaretaskedwithgeneratingquestionsforreadingcomprehensiontests\.\\n"

"Youwillbegivenacontextandyouneedtogeneratequestionsandcorrespondinganswersfromthegivencontext\.\\n"

"Thequestionsshouldbehighlyspecifictotheinformationprovidedinthecontext,notgeneralquestionsthatsuitanycontext\.\\n"

"\*\*DONOT\*\*hallucinateormakeupinformation\."

\)

Listing 1:System prompt used for question–answer generation\.USER\_PROMPT\_TEMPLATE=\(

"\#\#\#Instructions\#\#\#\\n"

"Generatequestionsandcorrespondinganswersfromthegivencontext\.Thequestionsshouldbehighlyspecifictotheinformationprovidedinthecontext,notgeneralquestionsthatsuitanycontext\.\\n\\n"

"\#\#\#Context\#\#\#\\n"

"\{context\}\\n\\n\\n"

"\#\#\#Rules\#\#\#\\n"

"Rulestofollowwhengeneratingthequestions:\\n"

"1\.Thequestionsmustbespecifictothegivencontextandfullyanswerablefrominformationpresentinthegivencontext\.\\n"

"2\.Askquestionsthatarefact\-seekingbasedontheinformationprovided\.\\n"

"3\.Makesurethequestionsareclearandunambiguous\.\\n"

"4\.Phraseslike’basedontheprovidedcontext’,’accordingtothecontext’,’inthecontext’,etc\.,are\*\*NOTALLOWED\*\*toappearinthequestions\.\\n"

"5\.Thequestionsshouldnotoverlap\.Theyshouldbediverse,coveringmanyaspectsofthecontext\.\\n"

"6\.Donotgiveawaytoomuchinformationinthequestions\.Forexample,ask’WhoisX?’insteadof’WhoisXthatdidY?’whenYisclearfromthecontext\.\\n"

"7\.Ignorethetextformattingofthecontext,e\.g\.,bold,italic,underline\.\\n"

"8\.Ignoretypos,spacing,andgrammaticalerrorsinthecontext\.\\n\\n"

"Rulestofollowwhengeneratingtheanswers:\\n"

"1\.Theanswersmustusethe\(implied\)informationprovidedinthecontext\.\\n"

"2\.Phraseslike’basedontheprovidedcontext’,’accordingtothecontext’,’inthecontext’,etc\.,are\*\*NOTALLOWED\*\*toappearintheanswers\.\\n"

"3\.Donotjustcopywordsfromthecontext\.Answerthequestioninyourownwords\.\\n"

"4\.Theanswersshouldbedetailedandcomprehensive\.Pleaseincludeadditionalspecificdetailsfromthecontext\.\\n\\n"

"Respondwith\{n\_qa\_pairs\}question\-answerpairs\.\\n"

"Alwaysusepropergrammarandpunctuation\.\\n"

"Trytousedifferentquestionformsandstyles\.\\n"

"Usesimplewordsandmakesurethattheanswersareclearandcomprehensive\.\\n\\n"

"Thequestion\-answerpairsshouldbeinthefollowingformat:\\n"

"Question1:\{\{question\_1\}\}\\n"

"Answer1:\{\{answer\_1\}\}\\n"

"Question2:\{\{question\_2\}\}\\n"

"Answer2:\{\{answer\_2\}\}\\n"

"\.\.\."

\)

Listing 2:User instruction template used for question–answer generation\.Applying these prompts to a FineWeb\-Edu passage yields a context paired with several self\-generated question–answer pairs\.[Listing3](https://arxiv.org/html/2610.38356#listing3)shows one representative training instance; at training time each question–answer pair forms a separate supervision target conditioned on the same context\.

Context:

TheHearstranch,SanJosedeBabicora,originallyfoundedinthe18thcenturybytheJesuits,waslocatedinnorthernMexicoandwasinthepathwayoftherevolutionarymovementsofthe1910s\-1920s\.In1907,attheageoftwenty\-four,JimBarkerwenttoMexico,whereheservedasranchmanagerofthemillion\-acreestateinChihuahuaofMrs\.PhoebeHearst\.Describedasa"smallish,wiryIrishmanwithasandycomplexionandsandyhairandsandyeyes,"heservedasacivilianscoutforGeneralJohnJ\.Pershingduringthepunitiveexpedition’ssearchforPanchoVilla\.

ThisinterviewwithMr\.Barker,whodiedatageeighty\-threeinSanAntonio,providesextensiveprimarysourceinformationabouttheMexicanRevolution,hisinvolvementinit,themurderofPeterKeene,andthehistoryoftheBabicoraranch\.PanchoVillaputa$10,000pricetagontheheadofMr\.BarkerinhisattempttocaptureBarker\.ThisledtoBarker’snine\-daytriptoescapebeingcaptured\.BarkerlaterservedinWorldWarIasheadofanextensiveintelligenceoperationinFranceandGermanyandwasabusinessconsultantfortheHearstEnterprisesuntilheretiredin1963\.

Q1:WherewastheHearstranch,SanJosedeBabicora,originallylocated?

A1:TheHearstranch,SanJosedeBabicora,wasoriginallylocatedinnorthernMexico\.

Q2:WhatwasJimBarker’sagewhenhefirstwenttoMexicotoworkforMrs\.PhoebeHearst?

A2:JimBarkerwas24yearsoldwhenhefirstwenttoMexicotoworkforMrs\.PhoebeHearst\.

Q3:HowisJimBarkerphysicallydescribedinthetext?

A3:JimBarkerisdescribedasa"smallish,wiryIrishmanwithasandycomplexionandsandyhairandsandyeyes\."

Q4:WhatwastherewardofferedforJimBarker’scapture,andwhoofferedit?

A4:TherewardofferedforJimBarker’scapturewas$10,000,anditwasofferedbyPanchoVilla\.

Q5:WhatroledidJimBarkerholdduringWorldWarI?

A5:DuringWorldWarI,JimBarkerservedastheheadofanextensiveintelligenceoperationinFranceandGermany\.

Listing 3:A representative FineWeb\-Edu training instance: a context passage paired with self\-generated question–answer pairs\.
## Appendix CEvaluation Configuration

At evaluation time, the backbone never sees the source passage: the knowledge required to answer is supplied entirely by the LoRA adapter that the generator produces from that passage\. Under the protocol with the format instruction, each test question is wrapped in the fixed answer\-format instruction shown in[Listing4](https://arxiv.org/html/2610.38356#listing4), which is the instruction that Doc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2610.38356#bib.bib24)\)uses for SQuAD, DROP, and ROPES; under the protocol without the format instruction, the question is given alone\. In both cases the prompt is wrapped in the backbone’s chat template, passed to the adapted model, and decoded greedily \([Equation6](https://arxiv.org/html/2610.38356#S3.E6)\) with at most256256new tokens\. The answer\-format instruction is used only at evaluation time: training prompts contain the question alone, wrapped in the same chat template \([Section3\.1](https://arxiv.org/html/2610.38356#S3.SS1)\), so the protocol without the format instruction matches the training format, whereas the protocol with it is out of distribution for the generator and tests whether the adapted model still follows an instruction it was not trained with\. We deliberately provide neither the document nor any in\-context exemplar, so that the context can only reach the model through the generated parameters\.

EVAL\_PROMPT\_TEMPLATE=\(

"Answerthefollowingquestion\.Outputonlytheansweranddonotoutputanyotherwords\.\\n\\n"

"Question:\{input\}"

\)

Listing 4:Instruction template used to query the adapted model at evaluation time\.
## Appendix DRe\-implemented Baselines

We re\-train Doc\-to\-LoRA and SHINE from their official implementations\. In[Table1](https://arxiv.org/html/2610.38356#S4.T1), all generators share the backbone \(Qwen3\-4B\-Instruct, frozen\), the training data \(the same FineWeb\-Edu passages and synthesized question–answer pairs\), and the generated adapter \(rank1616,α=32\\alpha=32, gate, up, and down projections of all3636layers\), while each baseline keeps its own architecture and training objective: Doc\-to\-LoRA is trained by context distillation with a KL objective and anℓ1\\ell\_\{1\}penalty on the generated adapter, and SHINE by next\-token prediction on the answers\. Because SHINE maps each memory state of dimensiond=2560d=2560directly to a slice of the adapter, its589,824589\{,\}824generated parameters per layer require⌈589,824/2560⌉=231\\lceil 589\{,\}824/2560\\rceil=231memory states per layer, and the1,5361\{,\}536surplus outputs are discarded\. All other architectural and training hyperparameters follow the official implementations\. The controlled comparison of[Section4\.3](https://arxiv.org/html/2610.38356#S4.SS3)varies only the generator scale and keeps the objective and all other settings of each method as in[Table1](https://arxiv.org/html/2610.38356#S4.T1), so the native\-size models are D2L and SHINE in[Table3](https://arxiv.org/html/2610.38356#S4.T3)\.[Table14](https://arxiv.org/html/2610.38356#A4.T14)lists the architecture\-specific settings\. The parameter count of Doc\-to\-LoRA includes its per\-module output head and the context\-independent LoRA biases, and that of SHINE includes the rank\-128128meta\-LoRA \(0\.260\.26B\), the memory tokens, and the M2P Transformer \(0\.210\.21B\)\. D2L\-L widens the latent dimension of Doc\-to\-LoRA to576576, and SHINE\-L deepens the M2P Transformer to1414layers\.

Table 14:Architecture\-specific settings of the re\-implemented baselines\.
## Appendix ETraining and Inference Cost

Training costs in[Section4\.3](https://arxiv.org/html/2610.38356#S4.SS3)are reported in GPU\-hours\. The backbone has4\.024\.02B parameters \(8\.08\.0GB in BF16\), and its KV cache takes144144KiB per token\. To control the variables across methods and make the different strategies directly comparable, every query and every answer is fixed to3232tokens in all inference measurements, including the multi\-turn latencies\.

#### Training\.

The training set consists of the9\.679\.67M passages of FineWeb\-Edu, each truncated to20482048tokens, with55question–answer pairs per passage and22epochs, i\.e\.,96\.796\.7M training examples\. Training CLIK takes3\.9×1043\.9\\times 10^\{4\}GPU\-hours, i\.e\.,25\.525\.5days\. Doc\-to\-LoRA and SHINE take1\.30×1041\.30\\times 10^\{4\}and1\.36×1041\.36\\times 10^\{4\}GPU\-hours \(8\.58\.5and8\.98\.9days\), and CLIK\-B, whose generator processes only2020tokens per layer, takes1\.03×1041\.03\\times 10^\{4\}GPU\-hours \(6\.76\.7days\)\.

#### Inference\.

Adapters that are applied per request use segmented multi\-adapter kernels\. Internalization of the context\-based methods is one prefill of the context, followed for SnapKV by scoring with the observation window\. That scoring is counted once in the internalization column of[Table9](https://arxiv.org/html/2610.38356#S4.T9)\. The SnapKV accuracy reported there uses a cache selected for the current question, so a new question selects again\. The*New question*column is this added latency: the TTFT when the internalized state is reused, and the internalization time plus the TTFT for SnapKV\. LLMLingua\-2 runs an XLM\-RoBERTa\-large token classifier over the context, including tokenization and word aggregation on the CPU, and then passes the compressed512512\-token prompt to the backbone for every query\. Per\-context states are loaded from host memory\. Peak memory includes the backbone and the generator or compressor weights, the3232per\-context states and the KV cache of the batched run, the workspace, and the transient activations of one internalization\. For the multi\-turn latencies in[Table7](https://arxiv.org/html/2610.38356#S4.T7), Dynamic LoRA re\-internalizes the initial context together with the four previous turns before answering\.

## Appendix FConstruction of the Counterfactual Probes

#### Paired counterfactual documents\.

We sample passages from the evaluation sets and, for each, select one answer\-bearing fact \(an entity, number, or date\)\. We create two copies that differ only in this fact, each replaced by a distinct nonce value of the same type \(e\.g\., invented names or dates that do not occur in the original passage\), and keep the question unchanged\. Each copy is internalized separately, and both adapters are queried with the identical question\.

#### Novel facts\.

We generate short passages about fictional entities \(invented people, organizations, and places with nonce names\) together with fact\-seeking questions, and verify that Base answers them at chance level\.

#### Strong prior conflict\.

We edit passages so that they contradict a fact that Base answers correctly without context \(e\.g\., the location of a well\-known landmark\), and ask for that fact\. An answer is counted as*Ctx*if it matches the edited value and as*Prior*if it matches the original value\.

#### Wrong\-context swap and unrelated queries\.

For the wrong\-context swap, each question is asked under the adapter of a randomly chosen passage from a different evaluation example\. For unrelated\-query interference, we take general\-knowledge questions that Base answers correctly without any adapter, ask them under the adapter of a random passage, and report accuracy relative to Base\.

## Appendix GFrequently Asked Questions

#### What distinguishes CLIK from SHINE?

The across\-layer and within\-layer attention of CLIK is the factorized attention of the Neural Network Transformer\([Tencent HY Team, 2026](https://arxiv.org/html/2610.38356#bib.bib17)\)\. SHINE already attends in both directions:[Table14](https://arxiv.org/html/2610.38356#A4.T14)lists its M2P Transformer as alternating across\-layer and within\-layer attention over a fixed set of memory states\. The difference is the representation those layers read\. SHINE first trains a LoRA that compresses the context into a fixed\-length token sequence, and that sequence is the initial input of the parameter generator\. CLIK leaves the context at its full length and supplies it as the input of cross\-attention, so each weight token reads the context while the adapter is generated\. This is the change that raises accuracy over SHINE\.[Table11](https://arxiv.org/html/2610.38356#S4.T11)shows that the two attentions improve CLIK, and the margin over SHINE is the margin between cross\-attention over the full context representation and generation from a fixed\-length memory\. Per\-layer generation with no interaction across layers is the design of Doc\-to\-LoRA, as recorded in the same table\.

#### Why is CLIK\-B absent from the counterfactual probes?

In CLIK\-B the gate and up factors are shared across contexts and mainly carry the pattern of the question\-answering task, while the per\-context𝐁down\\mathbf\{B\}^\{\\mathrm\{down\}\}mainly compresses the context\. On instruction following, and on tests that compress a context outside question answering, this split lowers accuracy substantially, so CLIK\-B remains a compact variant for the question\-answering setting and is omitted from[Table5](https://arxiv.org/html/2610.38356#S4.T5)\. The published Doc\-to\-LoRA makes the same split, generating only the down\-projection LoRA, and reproducing that version follows the supplied context poorly\. The Doc\-to\-LoRA compared in this paper generates the gate, up, and down factors of every layer, the same adapter CLIK generates, and that full adapter follows the context\.

#### Does the small in\-domain gain rule out overlap with the benchmarks?

Context following and corpus overlap are separate questions\. The probes in[Table5](https://arxiv.org/html/2610.38356#S4.T5)test whether the answer comes from the given context\. Adding the training splits of SQuAD, DROP, and ROPES \([Table4](https://arxiv.org/html/2610.38356#S4.T4)\) measures how much in\-domain supervision changes F1, and a small change there does not test overlap\. The training passages are the FineWeb data provided with Doc\-to\-LoRA, and the question–answer pairs are synthesized again for this paper, so those pairs do not overlap the evaluation questions and answers\. The training mixture in the Doc\-to\-LoRA paper is that cleaned FineWeb together with the training splits of SQuAD, ROPES, and DROP, so the passages do not overlap the evaluation contexts either\.

#### SnapKV decodes faster than CLIK\. Where does a new question change the cost?

The throughput in[Table9](https://arxiv.org/html/2610.38356#S4.T9)already includes attention over the retained state: SnapKV decodes at9\.09\.0k tokens per second and CLIK at8\.38\.3k\. SnapKV keeps tokens according to an observation window at the end of the prompt, and on these datasets that window is the question, so the reported F1 uses a cache selected for the current question\. The internalization column counts that selection once\. A further question on the same document selects the cache again, while CLIK generates the adapter once and reuses it for every question\. The*New question*column is this added latency\. For SnapKV it is the internalization time plus the time to first token; for CLIK it is the time to first token alone\.

#### The data prompt asks for detailed answers\. Is the short\-answer comparison fair?

The generator is trained to reproduce in\-context behavior without being required to emit a short answer, so the supervision is the long question–answer pairs of[Listing2](https://arxiv.org/html/2610.38356#listing2)\. That prompt asks for a detailed and comprehensive answer and for additional details from the context, which is how the long answers are obtained\. When an evaluation requires a short answer, the format instruction of[Listing4](https://arxiv.org/html/2610.38356#listing4)is added at test time\. Every baseline is trained on the same pairs and scored under the same protocol, so the answer length is shared by the comparison\.

#### How much of the gain is the fact, and how much is the answer format?

[Table2](https://arxiv.org/html/2610.38356#S4.T2)separates the two by precision and recall\. Under the short\-answer instruction, precision mainly measures whether the output follows that instruction\. Training uses the same long\-answer instruction on every example and never includes the short\-answer instruction, so on this precision CLIK remains behind Base \+ Context\. Without the short\-answer instruction, the precision and recall of CLIK stay close to those of Base \+ Context\.

## Appendix HQualitative Comparison

To illustrate the behavioral differences between methods, we present a side\-by\-side comparison on a single SQuAD passage\. For every question we report four answers:Base\(the frozen backbone queried with the question only, i\.e\., closed\-book and without the passage\),Base\+Context\(the backbone given the passage in\-context, an upper bound that has direct access to the evidence\),Doc\-to\-LoRA\(the document\-to\-LoRA baseline\), andCLIK\(ours\)\. Both Doc\-to\-LoRA and CLIK answer from the LoRA\-internalized knowledge without seeing the passage at inference time\.[Listing5](https://arxiv.org/html/2610.38356#listing5)shows the resulting transcript\. The closed\-book Base model already answers several questions correctly from its prior, since Super Bowl 50 is well known, but adds spurious or incorrect details \(e\.g\., in Q2\)\. CLIK produces short answers that agree with the in\-context reference on Q1–Q4 but answers Q5 incorrectly \(“Super Bowl 50” instead of “Super Bowl”\)\. Because the backbone already knows much of this passage, this example illustrates output behavior rather than knowledge acquisition, which is tested with counterfactual probes in[Section4\.4](https://arxiv.org/html/2610.38356#S4.SS4)\.

Context:Super Bowl 50 was an American football game to determine the champion of the National Football League \(NFL\) for the 2015 season\. The American Football Conference \(AFC\) champion Denver Broncos defeated the National Football Conference \(NFC\) champion Carolina Panthers 24\-10 to earn their third Super Bowl title\. The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California\. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold\-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals \(under which the game would have been known as "Super Bowl L"\), so that the logo could prominently feature the Arabic numerals 50\.====================================================================Q1: Where did Super Bowl 50 take place?\[Ground Truth\] Santa Clara, California\[Base\] Super Bowl 50 took place at Levi’s Stadium in Santa Clara, California\.\[Base\+Context\] Super Bowl 50 took place at Levi’s Stadium in Santa Clara, California\.\[Doc\-to\-LoRA\] Super Bowl 50 took place at Levi’s Stadium in Santa Clara, California, in the San Francisco Bay Area\.\[CLIK\] Super Bowl 50 took place at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California\.====================================================================Q2: What color was used to emphasize the 50th anniversary of the Super Bowl?\[Ground Truth\] gold\[Base\] The color used to emphasize the 50th anniversary of the Super Bowl was gold\. The 50th Super Bowl \(Super Bowl XLV\) was held in 2011, and while the official anniversary of the first Super Bowl \(1967\) was celebrated in 2017, the 50th Super Bowl \(Super Bowl LVI\) in 2022 marked the 50th anniversary of the first Super Bowl\. For that celebration, gold was prominently used in branding, logos, and promotional materials to highlight the milestone\.\[Base\+Context\] The color used to emphasize the 50th anniversary of the Super Bowl was gold\. This is evident from the context, which states: "the league emphasized the ’golden anniversary’ with various gold\-themed initiatives\."\[Doc\-to\-LoRA\] Gold was used to emphasize the 50th anniversary of the Super Bowl\.\[CLIK\] The color was gold\.====================================================================Q3: If Roman numerals were used, what would Super Bowl 50 have been called?\[Ground Truth\] Super Bowl L\[Base\] To find out what Super Bowl 50 would have been called in Roman numerals: the number 50 is represented as L, so Super Bowl 50 in Roman numerals would be called Super Bowl L\.\[Base\+Context\] If Roman numerals were used, Super Bowl 50 would have been called Super Bowl L\. In Roman numerals, 50 is represented by L\. However, for the 50th Super Bowl the NFL temporarily suspended the use of Roman numerals and instead used the Arabic numeral 50\.\[Doc\-to\-LoRA\] If Roman numerals were used, Super Bowl 50 would have been called "Super Bowl L"\.\[CLIK\] Super Bowl L====================================================================Q4: Where was Super Bowl 50 held?\[Ground Truth\] Santa Clara, California\[Base\] Super Bowl 50 was held at Levi’s Stadium in Santa Clara, California\.\[Base\+Context\] Super Bowl 50 was held at Levi’s Stadium in Santa Clara, California\.\[Doc\-to\-LoRA\] Super Bowl 50 was held at Levi’s Stadium in Santa Clara, California, in the San Francisco Bay Area\.\[CLIK\] Super Bowl 50 was held at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California\.====================================================================Q5: What is the name of the NFL championship game?\[Ground Truth\] Super Bowl\[Base\] The name of the NFL championship game is the Super Bowl, the annual championship game of the National Football League where the winners of the AFC and NFC face off to determine the league champion\.\[Base\+Context\] The name of the NFL championship game is the Super Bowl\.\[Doc\-to\-LoRA\] The name of the NFL championship game is Super Bowl LVI\.\[CLIK\] Super Bowl 50

Listing 5:Qualitative comparison on a SQuAD passage about Super Bowl 50\. Answers are color\-coded by source:Ground Truth,Base\(closed\-book, no passage\),Base\+Context\(passage given in\-context, upper bound\),Doc\-to\-LoRA, andCLIK\(ours\)\. Doc\-to\-LoRA and CLIK answer from the internalized LoRA adapter without the passage\. CLIK agrees with the in\-context reference on Q1–Q4 and errs on Q5, while the closed\-book Base often adds unsupported or incorrect details\.
## Appendix IMulti\-turn Dialogue Example

We illustrate the two multi\-turn mechanisms introduced in[Section4\.5](https://arxiv.org/html/2610.38356#S4.SS5)on a shared initial contextc0c\_\{0\}\(a SQuAD passage about Super Bowl 50\) and the same sequence of five questions\. To keep the listings compact, the passage is shown once and abbreviated asc0c\_\{0\}thereafter; each turn reports the base model’s input and its output\. The two scenarios differ in*what grows*as the dialogue proceeds\. In thedynamic KV cachescenario \([Listing6](https://arxiv.org/html/2610.38356#listing6)\), the adapter is generated once fromc0c\_\{0\}and*kept fixed*; conversational state is carried by the base model’s KV cache, so the model input*accumulates*the previous question–answer pairs turn by turn\. In thedynamic LoRAscenario \([Listing7](https://arxiv.org/html/2610.38356#listing7)\), the KV cache is discarded and the model input always contains only the current question; instead, the context fed to the parameter generator*grows*\(c0→c1→…c\_\{0\}\\\!\\to\\\!c\_\{1\}\\\!\\to\\\!\\dots\) and a fresh adapter is synthesized at every turn\.

Adapter \(fixed\): LoRA = PG\(c0\) \-\- internalized once; passage never enters the inputc0 = "Super Bowl 50 was an American football game \.\.\. so that the logo could prominently feature the Arabic numerals 50\."===== Turn 1 =====Model input :Q1: Which NFL team represented the AFC at Super Bowl 50?Model output:Denver Broncos===== Turn 2 =====\(KV cache from Turn 1 retained\)Model input :Q1: Which NFL team represented the AFC at Super Bowl 50?A1: Denver BroncosQ2: Which NFL team represented the NFC at Super Bowl 50?Model output:Carolina Panthers===== Turn 3 =====\(KV cache from Turns 1\-2 retained\)Model input :Q1: Which NFL team represented the AFC at Super Bowl 50?A1: Denver BroncosQ2: Which NFL team represented the NFC at Super Bowl 50?A2: Carolina PanthersQ3: Where did Super Bowl 50 take place?Model output:Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California===== Turn 4 =====\(KV cache from Turns 1\-3 retained\)Model input :\.\.\. accumulated \(Q1,A1\) \(Q2,A2\) \(Q3,A3\) \.\.\.Q4: Which NFL team won Super Bowl 50?Model output:Denver Broncos===== Turn 5 =====\(KV cache from Turns 1\-4 retained\)Model input :\.\.\. accumulated \(Q1,A1\) \.\.\. \(Q4,A4\) \.\.\.Q5: What color was used to emphasize the 50th anniversary of the Super Bowl?Model output:Gold

Listing 6:Dynamic KV cache\.The adapterLoRA = PG\(c0c\_\{0\}\)is generated once and held fixed\. Conversational state is propagated through the base model’s KV cache, so themodel input*accumulates*the previousquestion–answer historyturn by turn, while themodel outputis produced from the fixed adapter plus this growing history\.c0 = "Super Bowl 50 was an American football game \.\.\. so that the logo could prominently feature the Arabic numerals 50\."===== Turn 1 =====Adapter :LoRA = PG\(c0\)Model input :Q1: Which NFL team represented the AFC at Super Bowl 50?Model output:Denver Broncos===== Turn 2 =====Adapter :LoRA = PG\(c1\), c1 = c0 \+ \(Q1, A1\)Model input :Q2: Which NFL team represented the NFC at Super Bowl 50?Model output:Carolina Panthers===== Turn 3 =====Adapter :LoRA = PG\(c2\), c2 = c1 \+ \(Q2, A2\)Model input :Q3: Where did Super Bowl 50 take place?Model output:Santa Clara, California===== Turn 4 =====Adapter :LoRA = PG\(c3\), c3 = c2 \+ \(Q3, A3\)Model input :Q4: Which NFL team won Super Bowl 50?Model output:Denver Broncos===== Turn 5 =====Adapter :LoRA = PG\(c4\), c4 = c3 \+ \(Q4, A4\)Model input :Q5: What color was used to emphasize the 50th anniversary of the Super Bowl?Model output:The color used to emphasize the 50th anniversary of the Super Bowl was gold\.

Listing 7:Dynamic LoRA\.The KV cache is discarded, so themodel inputalways contains only the current question and never accumulates\. Instead, thecontextcic\_\{i\}fed to the parameter generator grows\(c0→c1→…c\_\{0\}\\\!\\to\\\!c\_\{1\}\\\!\\to\\\!\\dots, each appending the latest\(q,a\)\(\\,q,a\\,\)pair\), and a fresh adapter is regenerated every turn to produce themodel output\.