More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval

# Task-Conditioned Signed FFN Writes in Long-Context Retrieval
Source: [https://arxiv.org/html/2607.16254](https://arxiv.org/html/2607.16254)
\(June 2026\)

###### Abstract

FFNs are often treated as parametric memories\. In long\-context retrieval, however, the sharper question is not only what they store, but whether their native residual writes push the current retrieval state toward or away from the correct answer\. We test this by scaling the model’s own FFN write one layer at a time, without editing weights or injecting external steering vectors\.

Across controlled literal and semantic retrieval suites, native FFN response surfaces are signed, layer\-specific, and task\-conditioned: the final FFN is a suppressor in 7 of 8 model–suite cases, and 60% of layers switch role between retrieval modes \(95% CI \[50%, 69%\]\)\. A local directional derivative along the native write separates the two monotone roles: suppressors have negative derivative in 34/35 cases, and amplifiers have positive derivative in 18/18 cases, so the roles are not reducible to write size\. On a safety\-filtered LongBench retrieval\-QA probe, the same diagnostic predicts attenuation damage with rawR2=0\.796R^\{2\}\{=\}0\.796on Qwen2\.5\-7B and0\.7910\.791on Qwen3\.5\-9B; a held\-out suppressor\-attenuation policy improves retrieval margins over random and norm\-matched controls\. These results show that native FFN scaling exposes a signed, task\-conditioned residual\-write structure in retrieval, and that write\-gradient alignment is a compact diagnostic for the two monotone roles\.

## 1Introduction

A central result of mechanistic interpretability is that Transformer feed\-forward networks \(FFNs\) behave like key\-value memories: they store associations, enrich residual states with factual information, and expose editable factual content\(Gevaet al\.,[2021](https://arxiv.org/html/2607.16254#bib.bib1),[2022](https://arxiv.org/html/2607.16254#bib.bib2); Menget al\.,[2022](https://arxiv.org/html/2607.16254#bib.bib4); Daiet al\.,[2022](https://arxiv.org/html/2607.16254#bib.bib5); Gevaet al\.,[2023](https://arxiv.org/html/2607.16254#bib.bib3)\)\. This view now has engineering consequences\. Recent architectures make memory more explicit, sparse, or modular: Memory Layers add trainable key\-value lookup modules\(Bergeset al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib22)\); DeepSeek Engram introduces conditional memory lookup as a separate sparsity axis\(Chenget al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib23)\); MemoryLLM decouples FFN memory from self\-attention and residual flow\(Jaiswalet al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib24)\); STEM replaces part of FFN computation with embedding modules\(Sadhukhanet al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib25)\); and PEER\-style routing retrieves from many tiny experts\(He,[2024](https://arxiv.org/html/2607.16254#bib.bib26)\)\. The operational premise is that FFN memory can be factored out, sparsified, or replaced without losing the relevant capability\.

The architectural complication, illustrated in Figure[1](https://arxiv.org/html/2607.16254#S1.F1), is that a native FFN is not a side table\. The point is not that FFNs are the only Transformer modules that operate on the residual stream; attention and normalization also participate in residual computation\. The point is narrower and more consequential for the memory view: the memory\-like FFN computation is written inline into the same state that later retrieval computation consumes\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x1.png)Figure 1:Native FFN residual flow without intervention\.Attention first aggregates contextual evidence into the target\-token residual state, the FFN computes a native writeΔ​Fl\\Delta F\_\{l\}, and residual addition writeshl\+1=ul\+Δ​Flh\_\{l\+1\}=u\_\{l\}\+\\Delta F\_\{l\}into the stream consumed by later attention layers and the output head\. No gain scaling or damage metric is applied here\.A Transformer layer writes

hl\+1=ul\+Δ​Fl,ul:=hl\+Attnl​\(hl\),h\_\{l\+1\}=u\_\{l\}\+\\Delta F\_\{l\},\\qquad u\_\{l\}:=h\_\{l\}\+\\mathrm\{Attn\}\_\{l\}\(h\_\{l\}\),into a residual stream that every subsequent layer reads\. In long\-context retrieval, attention first aggregates contextual evidence into the target\-token residual state; the FFN then applies the only token\-wise nonlinear transformation between consecutive attention layers and writesΔ​Fl\\Delta F\_\{l\}back into the same stream; later attention layers and the output head consume that transformed state\. This raises two questions:Q1, does an FFN write affect cross\-layer retrieval information flow?Q2, if it does, in which direction, and is that direction task\-dependent?

We test this view with native FFN scaling \(Figure[2](https://arxiv.org/html/2607.16254#S3.F2)\):

hl\+1←ul\+α​Δ​Fl,α∈\{0,0\.25,…,2\},h\_\{l\+1\}\\leftarrow u\_\{l\}\+\\alpha\\,\\Delta F\_\{l\},\\qquad\\alpha\\in\\\{0,0\.25,\\ldots,2\\\},one layer at a time\. We use this intervention because it isolates the causal role of the native residual write\. It does not edit weights, inject an external steering vector, or replace the attention pattern; it preserves the learned content and direction ofΔ​Fl\\Delta F\_\{l\}while varying only its gain at a single layer\. The identity pointα=1\\alpha\{=\}1is the unmodified model,α<1\\alpha\{<\}1asks whether attenuating or removing the same write helps retrieval, andα\>1\\alpha\{\>\}1asks whether stronger expression of the same write amplifies its effect\. The resulting dose\-response therefore distinguishes two accounts\. If the FFN acted only as passive memory, reducing its write should mainly remove useful stored information\. The signed\-write hypothesis predicts a stronger and more specific pattern: the response should be signed\. Some layers should help retrieval when skipped, some should hurt when skipped, and the same layer can change role when the retrieval mode changes\.

The forward intervention answers whether FFN writes have signed causal effects\. The next question is whether the pattern can be explained from architecture\-internal signals\. Around the native pointα=1\\alpha\{=\}1, the retrieval margin changes along the model’s own FFN write direction\. At layerll, the forward FFN writeΔ​Fl\\Delta F\_\{l\}and the backward retrieval gradient∇M\\nabla Mmeet at the same residual node \(Figure[3](https://arxiv.org/html/2607.16254#S3.F3)\)\. Their inner product

sl=Δ​Fl⊤​∇Ms\_\{l\}=\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla Mis the local directional derivative of retrieval margin along the native write\. We use this quantity as a mechanism anchor, not as a substitute for the causal intervention: the scaling experiment defines the role, and the directional derivative tests whether the role follows the write direction\.

#### Contributions\.

1. 1\.Signed FFN role taxonomy\.Native FFN scaling reveals calibrated, amplifying, and suppressive roles in long\-context retrieval\. These roles are layer\-specific and task\-conditioned: the final layer is a suppressor in 7/8 controlled model–suite cases, and 60% of layers switch role between literal and semantic retrieval\.
2. 2\.Directional mechanism and controls\.The local directional derivative anchors the two monotone roles: suppressor writes point against the retrieval objective in 34/35 cases, while amplifier writes point with it in 18/18 cases\. Seed stability, threshold sweeps, and residual\-size controls show that the taxonomy is not a seed artifact, threshold artifact, or norm\-only effect\.
3. 3\.Naturalistic transfer and diagnostic implication\.A safety\-filtered LongBench retrieval\-QA probe transfers the same diagnostic to natural QA, and a targeted attenuation sanity check shows that suppressor layers are causally useful control points\. This gives a concrete diagnostic for FFN replacement or decoupling studies in retrieval settings: evaluate the write direction, not only lookup accuracy\.

Table 1:Claim–test contract\.The paper is organized as a single chain: signed causal effect, task conditioning, and local directional explanation\.

## 2Related Work

### 2\.1FFN as memory and its engineering consequences

Gevaet al\.\([2021](https://arxiv.org/html/2607.16254#bib.bib1),[2022](https://arxiv.org/html/2607.16254#bib.bib2)\)establish the FFN\-as\-memory view by interpreting FFN sublayers as key\-value memories\. Editing work shows that factual associations are localizable at FFN weights\(Menget al\.,[2022](https://arxiv.org/html/2607.16254#bib.bib4); Daiet al\.,[2022](https://arxiv.org/html/2607.16254#bib.bib5)\), andGevaet al\.\([2023](https://arxiv.org/html/2607.16254#bib.bib3)\)trace factual enrichment to early MLP sublayers with final extraction mediated by attention\. This line explains what FFNs store and how stored associations can be localized or edited\.

Recent architecture work turns this memory view into a design axis\. Memory Layers add explicit trainable lookup modules\(Bergeset al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib22)\); DeepSeek Engram adds conditional memory lookup as a separate sparsity axis\(Chenget al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib23)\); MemoryLLM decouples FFN memory from self\-attention and residual flow\(Jaiswalet al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib24)\); STEM replaces part of FFN computation with embedding modules\(Sadhukhanet al\.,[2026](https://arxiv.org/html/2607.16254#bib.bib25)\); and PEER\-style routing retrieves from many tiny experts\(He,[2024](https://arxiv.org/html/2607.16254#bib.bib26)\)\. These works decouple memory architecturally\. We diagnose a complementary property in conventional Transformers: the memory\-like FFN computation is written inline into the residual stream\. The decoupling literature addresses*what*the module retrieves; we ask*where and in which direction*the retrieved or transformed vector pushes the current retrieval objective\. §[6](https://arxiv.org/html/2607.16254#S6)discusses the narrower design consequence for retrieval settings: FFN replacement studies should check whether the replacement preserves retrieval\-relevant write direction, not only lookup behavior\.

### 2\.2Layer\-wise FFN importance

Pireset al\.\([2023](https://arxiv.org/html/2607.16254#bib.bib14)\)show that FFNs can be redundant and replaceable by one widened FFN;Menet al\.\([2024](https://arxiv.org/html/2607.16254#bib.bib15)\)rank blocks by redundancy; andIkedaet al\.\([2025](https://arxiv.org/html/2607.16254#bib.bib16)\)show that reallocating FFN capacity to middle layers can improve downstream tasks\. These works report FFN importance as a magnitude, redundancy, or relative\-benefit score\. Our role taxonomy is complementary: it measures a*signed*,*task\-dependent*per\-layer response\. A large FFN write is not necessarily helpful, and a small write is not necessarily irrelevant; the direction of the native write relative to the retrieval objective determines the two monotone roles\.

### 2\.3Long\-context retrieval

Long\-context benchmarks show that retrieval is behaviorally subtle\(Kamradt,[2023](https://arxiv.org/html/2607.16254#bib.bib13); Liuet al\.,[2023](https://arxiv.org/html/2607.16254#bib.bib10); Hsiehet al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib9); Baiet al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib11); Zhanget al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib12)\)\. Most relevant isModarressiet al\.\([2025](https://arxiv.org/html/2607.16254#bib.bib8)\), which shows that literal and semantic retrieval are distinct capabilities and attributes the gap primarily to attention\. Our experiments localize part of that divergence to the FFN write direction: the same FFN layer can switch its entire causal role when the retrieval mode changes\. LongBench is used here not as a leaderboard benchmark but as a natural\-language retrieval\-QA probe for testing whether the signed surface and directional\-derivative diagnostics transfer beyond controlled synthetic retrieval\.

### 2\.4Activation steering

Subramaniet al\.\([2022](https://arxiv.org/html/2607.16254#bib.bib19)\); Turneret al\.\([2023](https://arxiv.org/html/2607.16254#bib.bib20)\); Liet al\.\([2023](https://arxiv.org/html/2607.16254#bib.bib17)\); Panicksseryet al\.\([2024](https://arxiv.org/html/2607.16254#bib.bib18)\)steer behavior by injecting residual vectors;Bas and Novak \([2025](https://arxiv.org/html/2607.16254#bib.bib21)\)report an inverted\-U in steering coefficient\. We instead scale the model’s*native FFN residual write*and read the response as a signed role taxonomy\. The contribution is not that residual interventions can change behavior; it is that native FFN writes form a task\-conditioned signed structure, and that the monotone roles are anchored by the local directional derivative of retrieval margin along the native write\.

## 3Native FFN Writes and Local Direction

Figure[1](https://arxiv.org/html/2607.16254#S1.F1)shows where the native FFN write enters the residual stream before intervention\. Figure[2](https://arxiv.org/html/2607.16254#S3.F2)summarizes the forward residual\-write intervention\. Figure[3](https://arxiv.org/html/2607.16254#S3.F3)summarizes the endpoint\-response and directional\-derivative diagnostics used to classify and explain the signed roles\.

![[Uncaptioned image]](https://arxiv.org/html/2607.16254v1/x2.png)

Figure 2:FFN residual\-write intervention\.In a normal forward pass, attention aggregates contextual evidence into the target\-token residual state, and the FFN writes a transformed vectorΔ​Fl\\Delta F\_\{l\}back into the same residual stream that later attention reads\. Native FFN scaling changes only the strength of that write,hl\+1←ul\+α​Δ​Flh\_\{l\+1\}\\leftarrow u\_\{l\}\+\\alpha\\Delta F\_\{l\}, and the readout margin givesDl​\(α\)=Mbase−Ml​\(α\)D\_\{l\}\(\\alpha\)=M\_\{\\mathrm\{base\}\}\-M\_\{l\}\(\\alpha\)\.
![[Uncaptioned image]](https://arxiv.org/html/2607.16254v1/x3.png)

Figure 3:Signed\-role diagnostic\.Endpoint responses classify each layer by the signs ofDl​\(0\)D\_\{l\}\(0\)andDl​\(2\)D\_\{l\}\(2\), yielding calibrated, amplifying, or suppressive roles\. The backward diagnostic measures whether the same native write points with or against the retrieval objective: the forward writeΔ​Fl\\Delta F\_\{l\}and retrieval gradient meet at the same residual node, and the inner productsl=Δ​Fl⊤​∇Ms\_\{l\}=\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla Mis the local directional derivative anchoring the two monotone roles\.
### 3\.1Architectural fact: native FFN writes are consumed downstream

A pre\-norm Transformer layer evolves the residual stream as

hl\+1=hl\+Attnl\+FFNl=ul\+Δ​Fl\.h\_\{l\+1\}=h\_\{l\}\+\\mathrm\{Attn\}\_\{l\}\+\\mathrm\{FFN\}\_\{l\}=u\_\{l\}\+\\Delta F\_\{l\}\.In Llama\-, Qwen\-, and SmolLM2\-style sequential models,FFNl\\mathrm\{FFN\}\_\{l\}readsulu\_\{l\}; in Pythia\- and GPT\-NeoX\-style parallel models, it readshlh\_\{l\}\(§[7](https://arxiv.org/html/2607.16254#S7)\)\. In retrieval, attention aggregates contextual evidence into the target\-token residual state\. The FFN then applies a token\-wise nonlinear transformation and writesΔ​Fl\\Delta F\_\{l\}back into the same residual stream\. Later attention layers and the output head consume this transformed state \(Figure[2](https://arxiv.org/html/2607.16254#S3.F2)\)\. The residual connection itself is not a new observation\. What we test is whether the*direction*ofΔ​Fl\\Delta F\_\{l\}is signed, task\-conditioned, and predictable from the local directional derivative\.

### 3\.2Intervention design: native FFN scaling

We scale the native write

hl\+1←ul\+α​Δ​Fl,α∈\{0,0\.25,…,2\},h\_\{l\+1\}\\leftarrow u\_\{l\}\+\\alpha\\Delta F\_\{l\},\\qquad\\alpha\\in\\\{0,0\.25,\\ldots,2\\\},one layer at a time \(Figure[2](https://arxiv.org/html/2607.16254#S3.F2)\)\. We reportDl​\(α\)=Mbase−Ml​\(α\)D\_\{l\}\(\\alpha\)=M\_\{\\mathrm\{base\}\}\-M\_\{l\}\(\\alpha\), soDl​\(α\)\>0D\_\{l\}\(\\alpha\)\>0means the intervention hurt retrieval andDl​\(α\)<0D\_\{l\}\(\\alpha\)<0means it helped\. We use endpoints rather than AUC or curve slope because the role taxonomy is meant to summarize two explicit causal probes: removing the native write \(α=0\\alpha=0\) and over\-amplifying it \(α=2\\alpha=2\)\. The\|Dl​\(α\)\|<0\.1\|D\_\{l\}\(\\alpha\)\|<0\.1weak band is a pre\-specified margin tolerance for near\-zero endpoint responses; §[5\.5](https://arxiv.org/html/2607.16254#S5.SS5)and Appendix[A](https://arxiv.org/html/2607.16254#A1)report the corresponding threshold sweep\. A layer’s endpoint role is defined by the signs atα=0\\alpha=0andα=2\\alpha=2:

- •calibrated: skip hurts and amplify hurts; the pretrained gain is locally calibrated\.
- •amplifier: skip hurts and amplify helps; more FFN write improves retrieval\.
- •suppressor: skip helps and amplify hurts; more FFN write damages retrieval\.
- •weak/mixed: either endpoint lies inside the\|Dl​\(α\)\|<0\.1\|D\_\{l\}\(\\alpha\)\|<0\.1weak band or the endpoint pattern is not one of the three roles above\.

Native scaling is not intended to preserve residual norm; it is a causal dose intervention on the model’s own FFN write\. Norm\-preserving directional controls are used separately to test whether the signed effect is reducible to update magnitude \(§[5\.5](https://arxiv.org/html/2607.16254#S5.SS5), Appendix[E](https://arxiv.org/html/2607.16254#A5)\)\. This intervention tests two predictions\.P1: native FFN writes produce signed, layer\-structured response surfaces rather than uniform degradation\.P2: the same layer’s role can switch between literal and semantic retrieval because the task changes what counts as useful evidence at the residual state\.

### 3\.3Local directional derivative and backward mechanism

The forward intervention identifies the response surface; the directional criterion explains the two monotone roles\. Let

Ml​\(α\)=M​\(ul\+α​Δ​Fl\)M\_\{l\}\(\\alpha\)=M\(u\_\{l\}\+\\alpha\\Delta F\_\{l\}\)denote the retrieval margin after scaling the native FFN write at layerll\. Around the unmodified model,α=1\\alpha\{=\}1, a first\-order expansion gives

Ml​\(α\)−Ml​\(1\)≈\(α−1\)​Δ​Fl⊤​∇M\.M\_\{l\}\(\\alpha\)\-M\_\{l\}\(1\)\\approx\(\\alpha\-1\)\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla M\.Thus

Dl​\(α\)=Ml​\(1\)−Ml​\(α\)≈\(1−α\)​sl,sl:=Δ​Fl⊤​∇M\.D\_\{l\}\(\\alpha\)=M\_\{l\}\(1\)\-M\_\{l\}\(\\alpha\)\\approx\(1\-\\alpha\)s\_\{l\},\\qquad s\_\{l\}:=\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla M\.We interpretsls\_\{l\}as the local directional derivative of retrieval margin along the native FFN write\. The first\-order formula itself is standard; the empirical claim is that this simple local quantity remains informative across layers, models, retrieval modes, non\-infinitesimal attenuation gains, and LongBench QA transfer, while magnitude\-only controls do not explain the same role structure\. This is a diagnostic, not a second causal intervention: the causal object is still the native writeΔ​Fl\\Delta F\_\{l\}under scaling\. Ifsl\>0s\_\{l\}\>0, the write locally points toward higher retrieval quality and should behave as an amplifier under attenuation; ifsl<0s\_\{l\}<0, the write points against retrieval and should behave as a suppressor\. Calibrated layers are dose\-shape roles and are not claimed to be sign\-determined\.

This local interpretation has a limited scope\. It most directly constrains attenuation\-side signs and ranks near the native point; it need not exactly fit damage amplitude across all gains, task protocols, or depths\.P3: the two monotone roles are predicted by the sign ofsls\_\{l\}\(suppressor⇒sl<0\\Rightarrow s\_\{l\}<0, amplifier⇒sl\>0\\Rightarrow s\_\{l\}\>0\), and the same quantity gives the zero\-parameter local predictionDlpred​\(α\)=\(1−α\)​slD\_\{l\}^\{\\mathrm\{pred\}\}\(\\alpha\)=\(1\-\\alpha\)s\_\{l\}for attenuation damage\.

## 4Experimental Setup

#### Metric: retrieval damageDl​\(α\)D\_\{l\}\(\\alpha\)\.

Throughout, our dependent variable is the*retrieval margin*of the gold answer: the mean logit margin \(gold logit minus the best non\-gold logit\) over the retrieval task’s answer tokens\. The intervention effect is reported asDl​\(α\):=Mbase−Ml​\(α\)D\_\{l\}\(\\alpha\):=M\_\{\\mathrm\{base\}\}\-M\_\{l\}\(\\alpha\)\(raw\_damage\_margin\), soDl​\(α\)\>0D\_\{l\}\(\\alpha\)\>0means the intervention*hurt*retrieval \(lowered the gold margin\), andDl​\(α\)<0D\_\{l\}\(\\alpha\)<0means it*helped*\. Identity \(α=1\\alpha\{=\}1\) givesDl​\(1\)=0D\_\{l\}\(1\)\{=\}0by construction\. Per\-conditionDl​\(α\)D\_\{l\}\(\\alpha\)is averaged over \(i\) three random seeds for the three main\-protocol models and the available SmolLM2 cross\-protocol validation run, and \(ii\) three difficulty levels within each suite:L1L\_\{1\}\(single key\-value\),L2L\_\{2\}\(needle at varying positions\),L3L\_\{3\}\(conflicting distractors\)\. Bootstrap confidence intervals \(over seeds for the main\-protocol end\-layer endpoints; over layers for the role\-switch rate\) are reported in §[5\.5](https://arxiv.org/html/2607.16254#S5.SS5)\.

#### Models\.

Three main\-protocol models span∼\\sim1–9B and both residual\-composition families \(Table[2](https://arxiv.org/html/2607.16254#S4.T2)\): Pythia\-1B\(Biderman and others,[2023](https://arxiv.org/html/2607.16254#bib.bib27)\)\(parallel\), Qwen2\.5\-7B\(Qwen Team,[2024](https://arxiv.org/html/2607.16254#bib.bib28)\)\(sequential\), and Qwen3\.5\-9B\-Base \(sequential\)\. Qwen3\.5\-9B\-Base denotes a Qwen\-family sequential 32\-layer checkpoint used as an additional mechanism\-validation checkpoint in our runs; we report it separately from the standard public Qwen2\.5 release and do not use it as a public benchmark reference\. Results are reported per model rather than merged into a leaderboard\-style score\. SmolLM2\-1\.7B\(Allalet al\.,[2025](https://arxiv.org/html/2607.16254#bib.bib29)\)provides external architecture and data\-protocol validation\. All four have complete literal and semantic suites\. Backward\-gradient diagnostics \(bf16 full\-alpha\) are available for the three main\-protocol models and for sampled SmolLM2 anchor layers; all four are subjects of §[5\.3](https://arxiv.org/html/2607.16254#S5.SS3)\.

#### Cross\-protocol validation design\.

The model set separates estimation from stress testing\. Pythia\-1B, Qwen2\.5\-7B, and Qwen3\.5\-9B form the unified, three\-seed main\-protocol set used for seed aggregation, leave\-one\-seed\-out stability, and the primary Qwen response\-surface figures\. SmolLM2 changes both model family and data protocol relative to those three models, providing a cross\-protocol stress test\. Its literal suite matches the final\-layer suppressor pattern; its semantic suite yields a calibrated role, showing how the same taxonomy records protocol\-sensitive role changes\. We report SmolLM2 side by side for the taxonomy, task\-conditioning, block\-depth pattern, and sampled backward\-anchor role/sign relation, while keeping seed\-LOO summaries restricted to the three main\-protocol models\.

Table 2:Models and residual composition\.
#### Retrieval suites\.

Literal: verbatim needle, question shares vocabulary \(NIAH lineage\(Kamradt,[2023](https://arxiv.org/html/2607.16254#bib.bib13)\)\)\.Semantic: needle and question have minimal lexical overlap, forcing associative retrieval \(NoLiMa\-style\(Modarressiet al\.,[2025](https://arxiv.org/html/2607.16254#bib.bib8)\)\)\. Both at 8K context, three difficulty levels \(L1L\_\{1\}/L2L\_\{2\}/L3L\_\{3\}\)\.

#### Naturalistic LongBench retrieval\-QA probe\.

To evaluate the signed surface under natural\-language long\-context QA, we add a safety\-filtered LongBench retrieval\-QA probe\(Baiet al\.,[2024](https://arxiv.org/html/2607.16254#bib.bib11)\)on Qwen2\.5\-7B and Qwen3\.5\-9B\. We use two candidate\-scored LongBench subsets, Qasper and MultiFieldQA\-en, with 100 examples each \(200 total\), seed 13, bf16, and the same native FFN scaling gridα∈\{0,0\.25,…,2\}\\alpha\\in\\\{0,0\.25,\\ldots,2\\\}\. Qwen2\.5 has a complete28×928\{\\times\}9layer–gain grid plus baseline \(253 summary rows, 50,600 predictions\); Qwen3\.5 has a complete32×932\{\\times\}9grid plus baseline \(289 summary rows, 57,800 predictions\) with tail\-shard merge validation\. We keep this probe separate from the three\-seed controlled taxonomy: it evaluates transfer of the signed surface and directional\-derivative mechanism under a natural QA protocol\.

#### LongBench subset selection\.

LongBench is used here as natural\-language long\-context retrieval validation, not as a leaderboard benchmark\. We therefore select subsets by mechanism\-experiment constraints\. A subset must be QA/retrieval\-shaped, have answers that can be localized or inferred from the context, support forced\-choice logprob scoring with same\-subset distractors, and use relatively short answer spans so that the measured margin is not dominated by generation format, output length, or fluency\. We also prefer English natural\-language contexts, avoid code, summarization, dialogue, classification, and Chinese subsets in this first probe, and avoid strong multi\-hop reasoning so that the task primarily probes retrieval/readout rather than complex reasoning\. We further apply a conservative subtask\-level safety screen and include only QA subtasks with zero strict NSFW hits in the strict audit\. Qasper and MultiFieldQA\-en satisfy these constraints while providing real long\-form natural\-language contexts and enough answer\-pool diversity for length\-matched distractors\.

#### Backward diagnostic filtering\.

Qwen supplementary backward runs include a short\-copy control alongside the semantic tasks\. For semantic directional\-derivative diagnostics, we average only the semantic tasks \(S1S\_\{1\}–S3S\_\{3\}\) and exclude that control\. This affects the directional\-derivative diagnostic but not the endpoint taxonomy, which is computed from the intervention suites\.

#### Provenance\.

All headline numbers are re\-verified from raw per\-run summaries or per\-sample prediction files\. The accompanying project artifacts record the reconstruction scripts, audit notes, LongBench merge checks, and robustness tables used to produce the paper figures and tables\.

#### Sign convention\.

Throughout, positiveDl​\(α\)D\_\{l\}\(\\alpha\)indicates retrieval damage and negativeDl​\(α\)D\_\{l\}\(\\alpha\)indicates retrieval improvement\.

## 5Results

### 5\.1Q1: Native FFN scaling reveals signed causal response surfaces

Figure[4](https://arxiv.org/html/2607.16254#S5.F4)shows the full\-alpha response surface for the two Qwen models: every layer is probed across gains, and the color records signed retrieval damage \(Dl​\(α\)\>0D\_\{l\}\(\\alpha\)\{\>\}0hurts,Dl​\(α\)<0D\_\{l\}\(\\alpha\)\{<\}0helps\)\. The important fact is visible before any summary statistic: FFN response is not a scalar importance curve\. Adjacent layers can have opposite signs under the same gain, and Qwen2\.5\-7B and Qwen3\.5\-9B have visibly different surfaces\. Qwen2\.5 literal retrieval contains many calibrated layers; Qwen3\.5 literal retrieval has a mixed surface with suppressor, amplifier, and weak/mixed regions\. A generic perturbation\-sensitivity account would predict degradation under gain changes; the heatmaps instead show structured, signed variation\. The final\-layer suppressor cases are also matched by negativesls\_\{l\}in the directional\-derivative diagnostic, a property that generic fragility does not predict\.

The endpoint taxonomy in Table[3](https://arxiv.org/html/2607.16254#S5.T3)extends the same conclusion across models and retrieval modes\. In the three main\-protocol models, all six final FFN layers are suppressors: skipping them helps retrieval, while amplifying them hurts\. SmolLM2 then acts as the intended cross\-protocol validation: its literal suite matches the final\-suppressor pattern, while its semantic suite is calibrated\. This is the point of including an external protocol\. The result separates the robust taxonomy claim from the final\-layer specialization: signed roles persist under protocol shift, and the semantic final layer moves into a calibrated role\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x4.png)Figure 4:Qwen FFN response surfaces\.Seed\-averaged literal\-retrieval damageDl​\(α\)D\_\{l\}\(\\alpha\)across layers and gains\. The dashed column isα=1\\alpha\{=\}1; side strips show endpoint roles\.Table 3:Endpoint taxonomy counts\.Roles are computed from seed\-averaged endpoint damageDl​\(0\)D\_\{l\}\(0\)andDl​\(2\)D\_\{l\}\(2\)with threshold\|Dl​\(α\)\|\>0\.1\|D\_\{l\}\(\\alpha\)\|\{\>\}0\.1\. Rare both\-help and below\-threshold endpoint patterns are folded into weak/mixed\.![Refer to caption](https://arxiv.org/html/2607.16254v1/x5.png)Figure 5:Final FFN layers are usually readout\-adjacent suppressors\.In the three main\-protocol models, both literal and semantic final FFNs show the same qualitative pattern:α<1\\alpha\{<\}1helps andα\>1\\alpha\{\>\}1hurts\. Qwen3\.5\-9B literal is the largest case \(α=0→−3\.49\\alpha\{=\}0\\to\{\-\}3\.49,α=2→\+3\.10\\alpha\{=\}2\\to\{\+\}3\.10\)\. SmolLM2 semantic is calibrated, showing that the same taxonomy records protocol\-sensitive final\-layer roles\.
### 5\.2Q2: Signed FFN roles are task\-conditioned

Literal and semantic retrieval differ in the controlled dimension that matters for this paper: lexical overlap\. If an FFN layer were simply a static store for the target fact, its role should not systematically change when the retrieval mode changes\. It does\. Across the 100 layers with both\-suite endpoint data, 60 switch role between literal and semantic retrieval \(60%, 95% CI \[50%, 69%\] by layer bootstrap\)\. At the sharper sign level, using attenuation atα=0\.25\\alpha\{=\}0\.25with\|Dl​\(α\)\|\>0\.1\|D\_\{l\}\(\\alpha\)\|\{\>\}0\.1in both suites, we find 3 opposite\-sign layers in Pythia\-1B, 4 in Qwen2\.5\-7B, 6 in Qwen3\.5\-9B, and 7 in the SmolLM2 cross\-protocol run\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x6.png)Figure 6:Same\-layer FFN effects can flip between retrieval modes\.Top opposite\-sign examples atα=0\.25\\alpha\{=\}0\.25\. Blue bars are literal retrieval; red bars are semantic retrieval\. Positive damage means the intervention hurt retrieval\.These are not small threshold artifacts around zero\. In Qwen2\.5\-7B layer 15, weakening the FFN hurts literal retrieval but helps semantic retrieval; in Qwen3\.5\-9B layer 29, the same pattern appears with comparable magnitude in both directions\. Under the signed\-write view, the same layer’s task gradient can rotate, changing the sign ofsl=Δ​Fl⊤​∇Ms\_\{l\}=\\Delta F\_\{l\}^\{\\top\}\\nabla Mand reassigning the layer’s role\. We verify that mechanism anchor in §[5\.3](https://arxiv.org/html/2607.16254#S5.SS3)\. Semantic retrieval also exposes a propagation/amplitude component, so the role switch analysis combines the local direction signal with the downstream propagation diagnostics summarized in Appendix[B](https://arxiv.org/html/2607.16254#A2)\.

### 5\.3Q3: Local directional derivative explains monotone roles

The mechanism test is targeted\. We computesl=Δ​Fl⊤​∇Ms\_\{l\}=\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla Mfrom backward\-gradient diagnostics, then ask whether the sign ofsls\_\{l\}predicts the behavioral endpoint role\. The target of this test is the two*monotone*roles — suppressor and amplifier — because they are single\-direction effects\. The*calibrated*role is a different dose shape: both endpoints hurt, so it is modeled as a calibrated\-gain role rather than a sign role\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x7.png)Figure 7:Suppressor/amplifier roles are predicted by the sign ofsl=Δ​Fl⊤​∇Ms\_\{l\}=\\Delta F\_\{l\}^\{\\\!\\top\}\\nabla M\. Across the three full\-backward main\-protocol models plus SmolLM2 backward anchor layers, suppressor layers havesl<0s\_\{l\}\{<\}0in 34/35 cases, and amplifier layers havesl\>0s\_\{l\}\{\>\}0in 18/18 cases\. Calibrated layers are shown as the non\-monotone comparison class\. The right panel illustrates that the same layer’ssls\_\{l\}can flip between literal and semantic retrieval, giving a mechanism instance of the role switches\.Figure[7](https://arxiv.org/html/2607.16254#S5.F7)and Table[4](https://arxiv.org/html/2607.16254#S5.T4)give the main mechanism anchor\. Across the three full\-backward main\-protocol models plus SmolLM2 backward anchor layers, suppressor layers havesl<0s\_\{l\}\{<\}0in 34/35 cases \(pooled mean−0\.81\-0\.81\), and amplifier layers havesl\>0s\_\{l\}\{\>\}0in 18/18 cases \(pooled mean\+0\.75\+0\.75\)\. The single suppressor exception is Qwen3\.5 semantic layer 2 after excluding the short\-copy control from the semantic backward diagnostic\. The pattern is otherwise model\-local: Pythia suppressor 8/8, amplifier 2/2; Qwen2\.5 suppressor 8/8, amplifier 7/7; Qwen3\.5 suppressor 16/17, amplifier 8/8; SmolLM2 suppressor 2/2, amplifier 1/1\. Calibrated layers are balanced around sign \(27/61 positive\), matching their interpretation as calibrated\-gain layers rather than monotone direction layers\.

Table 4:Role×\\timessls\_\{l\}sign cross\-tab\.Counts pool the three full\-backward main\-protocol models plus SmolLM2 backward anchor layers\.n=161n\{=\}161layer\-suite pairs: Pythia\-1B 32 \(16 layers×\\times2 suites\), Qwen2\.5\-7B 56, Qwen3\.5\-9B 64, SmolLM2\-1\.7B 9 \(its 9 backward anchor layers, literal suite only\)\. Near\-perfect separation for the two monotone roles; calibrated is sign\-uninformative\.#### First\-order prediction\.

Ifsls\_\{l\}is the signed contribution per unitα−1\\alpha\{\-\}1, thenDl​\(α\)≈−sl​\(α−1\)D\_\{l\}\(\\alpha\)\\approx\-s\_\{l\}\\,\(\\alpha\{\-\}1\)at small\|α−1\|\|\\alpha\{\-\}1\|\. We test this zero\-parameter predictor on each model separately at attenuation alphas\{0\.25,0\.5,0\.75\}\\\{0\.25,0\.5,0\.75\\\}\. In literal retrieval, where attenuation is close to a local write\-direction regime, the predictor is quantitatively strong:R2=0\.888R^\{2\}\{=\}0\.888\(Pythia\),0\.6010\.601\(Qwen2\.5\), and0\.8670\.867\(Qwen3\.5\)\. Semantic retrieval is more revealing when rawR2R^\{2\}is read together with directional diagnostics \(Table[5](https://arxiv.org/html/2607.16254#S5.T5)\): Pythia and Qwen2\.5 have low rawR2R^\{2\}but preserve sign and rank signal, while Qwen3\.5 already has higher rawR2R^\{2\}\. The LongBench retrieval\-QA probe \(§[5\.4](https://arxiv.org/html/2607.16254#S5.SS4)\) recovers rawR2=0\.796R^\{2\}\{=\}0\.796\(Qwen2\.5\) and0\.7910\.791\(Qwen3\.5\) under natural QA, showing that the controlled\-semantic amplitude gap is protocol\-specific rather than a failure of the directional mechanism\.

Table 5:Directional\-derivative diagnostics separate direction from amplitude\.The local directional predictor usesDlpred​\(α\)=\(1−α\)​slD\_\{l\}^\{\\mathrm\{pred\}\}\(\\alpha\)=\(1\-\\alpha\)s\_\{l\}on attenuation gains\. Literal retrieval is close to the zero\-parameter amplitude model\. Semantic retrieval can lower rawR2R^\{2\}while preserving directional signal; majority, balanced accuracy, and MCC show that semantic sign agreement is not a class\-imbalance artifact\.
#### Semantic retrieval separates direction from propagation\.

The zero\-parameter predictor tests two things at once: whethersls\_\{l\}gets the*direction/order*of damage right, and whether its uncalibrated scale matches the observed damage amplitude\. Semantic retrieval often preserves the first while weakening the second\. In Qwen2\.5 semantic retrieval, rawR2R^\{2\}is0\.0740\.074, but Pearson correlation is0\.7700\.770, sign accuracy is0\.8210\.821, balanced sign accuracy is0\.8610\.861, and MCC is0\.6940\.694\. This turns semantic retrieval into a mechanism probe: it changes the task gradient and often flips the same layer’s direction, while the resulting damage magnitude depends more on how that write propagates through later layers and depth structure\. The signal\-graph diagnostic in Appendix[C](https://arxiv.org/html/2607.16254#A3)supports this interpretation: it is used as a propagation check, whilesls\_\{l\}remains the sign anchor for the monotone roles\.

#### Magnitude vs direction\.

Separately, a 33\-dim residual\-magnitude feature model is useful for predicting damage magnitude, but it does not explain the taxonomy\. In three\-class leave\-one\-model\-out role prediction \(calibrated/suppressor/amplifier\), relative norm alone gives balanced accuracy0\.200\.20against0\.330\.33balanced chance; adding depth improves to0\.320\.32, and residual norm/cosine/depth features reach0\.330\.33\. Magnitude carries partial signal, but it is unsigned and cannot tell suppressor from amplifier\. The role is set by the*direction*\(sls\_\{l\}\), which magnitude alone cannot recover\.

#### Propagation and depth diagnostics\.

Block\-level attenuation and the forward\-only signal graph support the same qualitative interpretation: attenuation damage also depends on downstream propagation and depth structure\. These diagnostics are not used to define endpoint roles or thesls\_\{l\}mechanism, so we keep the details in Appendices[B](https://arxiv.org/html/2607.16254#A2)and[C](https://arxiv.org/html/2607.16254#A3)\.

### 5\.4Transfer: LongBench retrieval\-QA preserves the signed surface

The controlled literal/semantic suites isolate retrieval from reasoning and domain knowledge\. We next ask how the same signed\-surface analysis behaves under a more natural QA protocol, and repeat native FFN scaling on the safety\-filtered LongBench retrieval\-QA probe for the two Qwen models\. The grids are complete and directly auditable from predictions: Qwen2\.5\-7B has 253 conditions and 50,600 prediction rows; Qwen3\.5\-9B has 289 conditions and 57,800 prediction rows\. Rebuilding the summaries from predictions recoversDl​\(α\)D\_\{l\}\(\\alpha\)to floating\-point precision \(maximum absolute margin\-damage differences2\.2×10−162\.2\{\\times\}10^\{\-16\}and9\.7×10−179\.7\{\\times\}10^\{\-17\}\)\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x8.png)Figure 8:The safety\-filtered LongBench retrieval\-QA probe also has signed FFN response surfaces\.Heatmaps showDl​\(α\)D\_\{l\}\(\\alpha\)for Qwen2\.5\-7B and Qwen3\.5\-9B across every layer and gain on candidate\-scored Qasper and MultiFieldQA\-en examples\. Signed structure persists under the natural QA probe, while the final\-layer role can shift across protocols\.Figure[8](https://arxiv.org/html/2607.16254#S5.F8)shows that the signed surface persists: gain changes do not produce uniform degradation, and both Qwen models contain multiple endpoint types\. Under the same\|Dl​\(α\)\|\>0\.1\|D\_\{l\}\(\\alpha\)\|\{\>\}0\.1endpoint threshold, Qwen2\.5 has 3 calibrated layers, 8 suppressors, 0 amplifiers, 5 both\-ends\-help layers, and 12 weak/mixed layers; Qwen3\.5 has 3 calibrated layers, 3 suppressors, 1 amplifier, 9 both\-ends\-help layers, and 16 weak/mixed layers \(Table[6](https://arxiv.org/html/2607.16254#S5.T6)\)\. Thus the naturalistic probe does not merely reproduce the controlled taxonomy counts; it shows the same measurement recovering a signed layer\-role surface under a natural QA protocol\.

The directional\-derivative mechanism also transfers\. Using LongBench backward diagnostics and attenuation gainsα∈\{0\.25,0\.5,0\.75\}\\alpha\\in\\\{0\.25,0\.5,0\.75\\\}, the zero\-parameter predictorDlpred​\(α\)=\(1−α\)​slD\_\{l\}^\{\\mathrm\{pred\}\}\(\\alpha\)=\(1\-\\alpha\)s\_\{l\}explains damage with rawR2=0\.796R^\{2\}\{=\}0\.796for Qwen2\.5 and0\.7910\.791for Qwen3\.5; Pearson correlations are0\.9030\.903and0\.9490\.949, and balanced sign accuracies are0\.8570\.857and0\.8020\.802\. The per\-subtask breakdown is similarly strong \(Appendix Table[9](https://arxiv.org/html/2607.16254#A4.T9)\): Qasper and MultiFieldQA\-en each have rawR2\>0\.75R^\{2\}\{\>\}0\.75and Pearson correlation above0\.890\.89\. The weakest sign row is Qwen3\.5/Qasper, where sign accuracy is0\.7080\.708against a0\.5940\.594majority baseline \(balanced accuracy0\.7100\.710\)\. Unlike the lower\-R2R^\{2\}controlled semantic cases, this probe does not show the same direction\-versus\-amplitude split for the Qwen models: the local directional derivative is quantitatively predictive of attenuation damage\.

Table 6:LongBench retrieval\-QA transfer summary\.Pooled Qasper and MultiFieldQA\-en results for the two Qwen models\.Both\-help is reported explicitly here because it is frequent under the natural QA protocol: 5/28 Qwen2\.5 layers and 9/32 Qwen3\.5 layers in the pooled rows\. This is a protocol\-shift role, not a replacement for the monotone mechanism claim; the suppressor/amplifier analysis continues to center on the two single\-direction roles and theirsls\_\{l\}signs\.

The final\-layer shift is informative\. In the controlled retrieval protocol, the final FFN is usually a readout\-adjacent suppressor\. In the LongBench retrieval\-QA probe, Qwen2\.5 retains that aggregate pattern \(Dl​\(0\)=−2\.60D\_\{l\}\(0\)\{=\}\{\-\}2\.60,Dl​\(2\)=\+0\.90D\_\{l\}\(2\)\{=\}\+0\.90\), while Qwen3\.5 moves into a both\-ends\-help role \(Dl​\(0\)=−5\.47D\_\{l\}\(0\)\{=\}\{\-\}5\.47,Dl​\(2\)=−1\.25D\_\{l\}\(2\)\{=\}\{\-\}1\.25\): both skipping and doubling the final FFN improve the margin\. This supports the paper’s main framing\. FFN roles are task\-conditioned signed\-write roles; final\-layer suppression is a strong controlled retrieval phenomenon, and natural QA can move the same architectural position into a different signed role\.

Finally, the signed surface supports a held\-out policy sanity check, reported in full in Appendix[D](https://arxiv.org/html/2607.16254#A4)\. We select top suppressor layers on a calibration split and evaluate only on a disjoint LongBench split\. Withk=3k\{=\}3layers andα=0\.5\\alpha\{=\}0\.5, attenuating the selected suppressors improves held\-out margin in both Qwen models \(mean\+3\.01\+3\.01\), exceeding random layer sets \(\+0\.13\+0\.13\) and relative\-norm\-matched controls \(−0\.14\-0\.14\)\. This is not proposed as an optimizer; it is a compact check that the taxonomy identifies causally useful layer directions\.

### 5\.5Minimal robustness and confound controls

The role taxonomy is defined by signs ofDl​\(0\)D\_\{l\}\(0\)andDl​\(2\)D\_\{l\}\(2\)with a\|Dl​\(α\)\|<0\.1\|D\_\{l\}\(\\alpha\)\|\{<\}0\.1weak band\. We therefore audit its three main degrees of freedom: threshold choice, seed variation, and FFN update size \(scripts/robustness\_checks\_20260621\.py\)\.

#### Threshold sensitivity\.

The exact counts move when the weak band changes, as they should, but the taxonomy does not collapse into one response type\. For Qwen2\.5\-7B literal, the number of calibrated layers is 18 at thresholds 0\.05 and 0\.1, and remains 13 at 0\.25\. For Qwen3\.5\-9B literal, the exact count shifts more \(calibrated 8/6/3 at 0\.05/0\.1/0\.25\), but suppressor and amplifier classes persist\. Thus the threshold affects borderline labels, not the qualitative finding that calibrated, suppressive, amplifying, and weak/mixed roles coexist\.

#### Seed robustness \(leave\-one\-seed\-out\)\.

For the three multi\-seed models we recompute every layer’s role under each of three leave\-one\-seed\-out folds\.136/152136/152layers \(89%89\\%\) retain the same role label across all three folds, and all 6 end layers retain the suppressor label across folds\. The taxonomy is therefore not a single\-seed artifact\.

#### Bootstrap confidence intervals\.

Seed\-bootstrap CIs \(B=2000B\{=\}2000\) on end\-layer damage values are narrow \(Qwen3\.5\-9B literal:Dl​\(0\)=−3\.49​\[−3\.70,−3\.31\]D\_\{l\}\(0\)=\-3\.49\\,\[\-3\.70,\-3\.31\],Dl​\(2\)=\+3\.10​\[\+3\.02,\+3\.15\]D\_\{l\}\(2\)=\+3\.10\\,\[\+3\.02,\+3\.15\]\); the weakest endpoint is Qwen2\.5\-7B semanticα=0\\alpha\{=\}0\(−0\.17​\[−0\.30,−0\.08\]\-0\.17\\,\[\-0\.30,\-0\.08\], close to the weak band\)\. Accordingly, the end\-layer claim is a qualitative monotone\-suppressor pattern, not equal endpoint strength in every suite\. The role\-switch rate is60%60\\%with layer\-bootstrap 95% CI\[50%,69%\]\[50\\%,69\\%\]\.

#### Residual\-size controls\.

Relative FFN norm, depth, and residual geometry do not explain the taxonomy by themselves\. Figure[9](https://arxiv.org/html/2607.16254#S5.F9)gives the minimal control\. Relative FFN norm overlaps across role classes \(median calibrated0\.400\.40, suppressor0\.330\.33, amplifier0\.360\.36; IQRs overlap substantially\)\. In three\-class leave\-one\-model\-out role prediction, relative norm alone gives balanced accuracy0\.200\.20against0\.330\.33balanced chance; adding depth improves to0\.320\.32, and residual norm/cosine/depth features reach0\.330\.33\. Residual size and depth carry partial signal, but they do not determine the signed role\. The direction \(sls\_\{l\}\), which magnitude cannot recover, is what predicts the monotone roles\. Appendix[E](https://arxiv.org/html/2607.16254#A5)adds a fixed\-norm rotation sanity check on Qwen2\.5: rotating writes with or against the local retrieval\-gradient direction while preserving write and residual norms gives true\-gradient win rates of0\.9970\.997\(literal/main\) and1\.0001\.000\(semantic/supplementary\), while random tangent rotations stay near chance \(0\.5130\.513and0\.4860\.486\)\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x9.png)Figure 9:Residual size is a partial signal, not a taxonomy explanation\.\(a\)FFN relative norm overlaps across endpoint taxonomy types\.\(b\)Leave\-one\-model\-out taxonomy prediction from residual size/depth features\. The classifier uses the three signed roles and omits weak/mixed layers; relative norm alone is below balanced chance, while adding depth and residual/cosine features remains weak\.

## 6Discussion

#### Design consequence\.

Memory\-decoupling architectures isolate storage from dense computation; Memory Layers, DeepSeek Engram, MemoryLLM, and STEM are examples\. Our results do not show that all such designs are at risk\. They identify a diagnostic axis for retrieval settings: when an architecture replaces or factors out FFN memory\-like computation, lookup accuracy alone may miss whether the replacement write pushes the residual state in the task\-relevant direction\. The directional\-derivative scoresls\_\{l\}provides a zero\-parameter post\-hoc check: after replacement or decoupling, measure whether the module’s write still aligns with the retrieval gradient\. The LongBench policy sanity check makes the diagnostic actionable within our setting: attenuating top suppressor layers selected on a calibration split improves held\-out margin by\+3\.01\+3\.01on average atk=3,α=0\.5k\{=\}3,\\alpha\{=\}0\.5, compared with\+0\.13\+0\.13for random layer sets and−0\.14\-0\.14for relative\-norm\-matched controls\.

#### What the end\-layer suppressor means\.

In the controlled retrieval protocol, the end layer sits immediately before the output head; there is no later attention for it to shape\. Its suppressor signature is therefore best read as a*readout\-adjacent*effect: the end\-layer FFN write injects or amplifies information that competes with the retrieval decision at output scoring — e\.g\. prior\-token continuations or majority\-class priors — so removing it clarifies the retrieved signal and amplifying it drowns it\. This is the same signed\-write question at the output boundary: non\-final layers can change what later attention selects, while the final layer changes what the output head scores\.

#### What LongBench changes\.

The safety\-filtered LongBench retrieval\-QA probe makes this separation concrete\. It preserves signed FFN response surfaces and strongsls\_\{l\}predictivity in a more natural QA setting, while changing the readout\-adjacent role of Qwen3\.5’s final layer to both\-ends\-help\. This separation is useful: the central claim is a signed, task\-conditioned native\-write taxonomy, and final\-layer suppression is one prominent controlled\-retrieval role within that taxonomy\. Naturalistic QA can change which direction a readout\-adjacent FFN write pushes the retrieval margin, while the signed\-surface and directional\-derivative structure remains visible\.

#### Why the role switches between tasks \(Q2\)\.

Literal retrieval needs attention to surface a verbatim needle; semantic retrieval needs it to surface an associated concept\. The same FFN write can rotate from aiding one to suppressing the other: in Figure[7](https://arxiv.org/html/2607.16254#S5.F7), several layers havesls\_\{l\}of opposite sign across suites\. This is the mechanism behind the 60% role switch and the core evidence that FFN’s role is task\-conditioned rather than a static property of stored knowledge\.

#### Claim calibration\.

The evidence supports a strong but specific claim: in long\-context retrieval, native FFN writes have signed, task\-conditioned effects on retrieval margin\. Direction is the decisive signal for the two monotone roles; residual size, depth, and propagation features shape attenuation magnitude and explain why the full response surface is richer than a single sign\. The mechanism anchor \(Q3\) is verified across both residual\-composition families, with SmolLM2 providing cross\-protocol validation of the Pythia/Qwen protocol\. The role landscape is depth\-structured and mixed in the middle layers, so the paper reports a signed layer\-role distribution rather than a depth\-only rule\.

## 7Limitations

#### Pythia’s parallel residual\.

Pythia computeshl\+1=hl\+Attn\+FFNh\_\{l\+1\}\{=\}h\_\{l\}\{\+\}\\mathrm\{Attn\}\{\+\}\\mathrm\{FFN\}withFFN\\mathrm\{FFN\}readinghlh\_\{l\}\(notulu\_\{l\}\)\. The native\-write intervention is valid in both residual\-composition families, but the “reshapes the next attention’s input” framing is cleanest on the three sequential models\. Pythia is included as a cross\-family check\.

#### Backward scope\.

The mechanism anchor \(Q3\) uses per\-layer backward data \(bf16 full\-alpha\) on the three main\-protocol models and sampled backward anchor layers on SmolLM2\. Qwen semantic backward diagnostics exclude the short\-copy control and average only the semantic tasks\. Thesl↔s\_\{l\}\\leftrightarrowrole mapping is therefore checked across both residual\-composition families and a range of model scales \(1–9B\), but the local directional amplitude model is used as a task\-local diagnostic rather than a universal cross\-model calibration rule\.

#### Training dynamics\.

We show that trained FFN writes have task\-conditioned signed effects on retrieval\. The current experiments characterize the trained mechanism; a checkpoint study would test how these directions form during training\.

#### Vector\-subspace decomposition\.

This paper establishes a signed causal surface with native scaling interventions and robustness controls\. Decomposing individual FFN writes into finer vector subspaces is a complementary analysis, not a prerequisite for the claims here\.

## 8Conclusion

Taken together, the results show that the role of FFNs in LLMs is more than memory\. FFNs do not only store or retrieve associations; in a native Transformer, their outputs are written inline into the residual stream that later attention layers and the output head consume\. In long\-context retrieval, this gives the native FFN write a signed role: it can be calibrated, amplifying, or suppressive depending on layer, task, and direction\.

Our theoretical and empirical analyses support this view\. Native FFN scaling isolates the model’s own residual write and reveals signed, layer\-specific response surfaces\. Across the controlled main protocol and SmolLM2 cross\-protocol validation, final\-layer writes are suppressors in 7 of 8 model–suite pairs, 60% of layers switch role between literal and semantic retrieval \(95% CI \[50%, 69%\]\), and the monotone roles are anchored by the local directional derivative: suppressorsl<0s\_\{l\}\{<\}0in 34/35 cases and amplifiersl\>0s\_\{l\}\{\>\}0in 18/18 cases\. A safety\-filtered LongBench retrieval\-QA probe preserves the signed surface on two Qwen models and gives strong first\-order prediction \(R2=0\.796R^\{2\}\{=\}0\.796/0\.7910\.791\), while also showing that final\-layer roles can change under naturalistic protocol shift\.

For architectures that decouple, sparsify, or replace FFN parameters, the implication is a diagnostic one rather than a universal design rule\. In retrieval settings, lookup accuracy should be complemented by a check on the contextual role of the replacement write: where it enters the residual stream, how later attention and the output head consume it, and whether its direction pushes the retrieval state toward or away from the current task objective\. The storage view explains localization and editing of facts; the signed\-write view adds a directional question to ask when FFN memory\-like computation is factored out of the native Transformer\.

## Appendix ATaxonomy Threshold Sweep

The main taxonomy uses the pre\-specified weak band\|Dl​\(α\)\|<0\.1\|D\_\{l\}\(\\alpha\)\|<0\.1\. Table[7](https://arxiv.org/html/2607.16254#A1.T7)reports the complete threshold sweep used in the robustness audit\. The threshold changes borderline labels, especially for weak/mixed layers, but does not collapse the signed taxonomy into a single response type\.

Table 7:Endpoint\-taxonomy threshold sweep\.Counts are recomputed from the same seed\-averaged endpoints using weak bands 0\.05, 0\.10, and 0\.25\.
## Appendix BFour\-Block Attenuation Diagnostic

The single\-layer taxonomy could be read as isolated per\-layer effects\. As a coarse propagation check, we split each network into four contiguous FFN blocks by depth and scale all FFN writes in one block at a time\. Figure[10](https://arxiv.org/html/2607.16254#A2.F10)shows that the signed depth structure persists at block scale\. This diagnostic is not used to define endpoint roles or to replace the localsls\_\{l\}sign test\.

![Refer to caption](https://arxiv.org/html/2607.16254v1/x10.png)Figure 10:Four\-block attenuation exposes coarse depth roles\.Each network is split into four contiguous FFN blocks\. Cells show damage from scaling all FFN writes in one block toα=0\.25\\alpha\{=\}0\.25orα=0\.5\\alpha\{=\}0\.5\. Positive values mean block attenuation hurts retrieval; negative values mean it helps\.Weakening the*earliest*block atα=0\.25\\alpha\{=\}0\.25is the most damaging block intervention in all six main \(model,suite\) pairs: damage\+2\.80\+2\.80\(Pythia literal\),\+8\.68\+8\.68\(Pythia semantic\),\+4\.02\+4\.02\(Qwen2\.5 literal\),\+11\.26\+11\.26\(Qwen2\.5 semantic\),\+5\.51\+5\.51\(Qwen3\.5 literal\), and\+6\.09\+6\.09\(Qwen3\.5 semantic\)\. Weakening the*last*block has the opposite sign in the same six pairs:−0\.92\-0\.92,−0\.46\-0\.46,−1\.54\-1\.54,−2\.22\-2\.22,−0\.79\-0\.79, and−3\.47\-3\.47\. Because a block contains several FFNs \(including the final suppressor in the last quarter\), this is not a per\-layer attribution; it is a coarse confirmation that signed depth structure survives when contiguous depth segments are perturbed together\. SmolLM2 supplies the cross\-protocol validation: weakening its earliest block atα=0\.25\\alpha\{=\}0\.25hurts literal and semantic retrieval \(\+3\.83\+3\.83and\+7\.57\+7\.57\), while weakening its last block helps literal retrieval \(−1\.46\-1\.46\) and hurts semantic retrieval \(\+1\.26\+1\.26\), again showing signed depth structure rather than one uniform FFN effect\.

## Appendix CSignal\-Graph Propagation Diagnostic

The signal\-graph diagnostic is a forward\-only propagation check\. It predicts attenuation responses from FFN update magnitude and residual\-stream propagation features \(norms, ratios, cosines crossed with dose\), fit with seed leave\-one\-out over the three main\-protocol models\. The source predictions use the delta\-margin sign convention, butR2R^\{2\}is invariant under the global sign convention used forDl​\(α\)D\_\{l\}\(\\alpha\)in the paper\.

Table 8:Signal\-graph propagation diagnostic\.These rows support the claim that attenuation effects follow residual\-stream propagation structure\. They are not used to define endpoint roles and are not a replacement for the localsls\_\{l\}sign test\.
## Appendix DLongBench Policy and Gradient\-Path Sanity

Table 9:Per\-subtask LongBench retrieval\-QA breakdown\.These rows support the pooled LongBench table in the main text and show that the transfer result is not driven by only one subtask\.
## Appendix EFixed\-Norm Directional Rotation Sanity

The residual\-size controls in the main text use scalar diagnostics\. As a more direct sanity check, we also run a Qwen2\.5 fixed\-norm rotation pilot: for selected layers, we rotate the native write in the positive or negative local gradient direction while renormalizing the write and resulting residual state to preserve their original norms\. This is not a replacement for native scaling, because it constructs a tangent perturbation rather than varying the model’s own gain\. It is a targeted test of whether direction alone can move retrieval margin when norm is held fixed\.

Table 10:Fixed\-norm directional rotation sanity check\.Positive gap means the positive\-gradient rotation improves margin relative to the negative\-gradient rotation\. The maximum write/residual norm\-ratio deviation from 1 is below2×10−62\{\\times\}10^\{\-6\}in all rows\.The true\-gradient direction wins almost every paired comparison, while random tangent directions stay near chance\. Because the write and residual norms are effectively fixed, this supports the main claim that the sign of the FFN effect is directional rather than a simple consequence of update size\.

#### Directional\-derivative path sanity\.

We validate the LongBench directional\-derivative path with finite differences on five anchor layers per Qwen model\. The finite\-difference direction and the autograd alpha\-gradient direction agree in sign on all sampled anchors \(Table[11](https://arxiv.org/html/2607.16254#A5.T11)\)\. The direct dot product and the recorded alpha\-gradient are also close, so the sanity check supports the implementation ofsls\_\{l\}rather than adding a separate mechanistic claim\.

Table 11:LongBench finite\-difference sanity check\.Each row uses one Qasper sample and five anchor layers\. FD sign compares finite differences with the recorded alpha\-gradient direction; dot\-alpha sign compares the directΔ​Fl⊤​∇M\\Delta F\_\{l\}^\{\\top\}\\nabla Mdot product with the alpha\-gradient direction\.
#### Held\-out policy controls\.

Table[12](https://arxiv.org/html/2607.16254#A5.T12)gives the full LongBench held\-out policy sanity table\. Top suppressor layers are selected from the single\-layer margin\-damage screen on a calibration split; random and relative\-norm\-matched controls use the samekkand policy alpha\. All values are evaluated on a disjoint 100\-example evaluation split\. The main text reports the conservative attenuation rows \(k=3,α=0\.5k\{=\}3,\\alpha\{=\}0\.5\)\. The ablation rows \(α=0\\alpha\{=\}0\) preserve the same ordering but are not used as the main policy claim because they are a stronger intervention\.

Table 12:Held\-out LongBench FFN\-policy comparison\.Positive values are evaluation\-split margin gains over baseline\.

## References

- L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blazquez, G\. Penedo, L\. Tunstall,et al\.\(2025\)SmolLM2: When Smol Goes Big – Data\-Centric Training of a Small Language Model\.External Links:2502\.02737,[Document](https://dx.doi.org/10.48550/arXiv.2502.02737),[Link](https://arxiv.org/abs/2502.02737)Cited by:[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px2.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang,et al\.\(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of ACL,External Links:[Link](https://arxiv.org/abs/2308.14508)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px5.p1.3)\.
- T\. Bas and K\. Novak \(2025\)What Can We Actually Steer? A Multi\-Behavior Study of Activation Control\.External Links:[Link](https://arxiv.org/abs/2511.18284)Cited by:[§2\.4](https://arxiv.org/html/2607.16254#S2.SS4.p1.1)\.
- V\. Berges, B\. Oguz, D\. Haziza, W\. Yih, L\. Zettlemoyer, and G\. Ghosh \(2024\)Memory Layers at Scale\.External Links:2412\.09764,[Document](https://dx.doi.org/10.48550/arXiv.2412.09764),[Link](https://arxiv.org/abs/2412.09764)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p2.1)\.
- S\. Bidermanet al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.External Links:[Link](https://arxiv.org/abs/2304.01373)Cited by:[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px2.p1.1)\.
- X\. Cheng, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie, K\. Huang, X\. Yu, Z\. Hao, Y\. Li, H\. Zhang, H\. Zhang, D\. Zhao, and W\. Liang \(2026\)Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models\.External Links:2601\.07372,[Document](https://dx.doi.org/10.48550/arXiv.2601.07372),[Link](https://arxiv.org/abs/2601.07372)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p2.1)\.
- D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei \(2022\)Knowledge neurons in pretrained transformers\.InProceedings of ACL,External Links:[Link](https://arxiv.org/abs/2104.08696)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p1.1)\.
- M\. Geva, J\. Bastings, K\. Filippova, and A\. Globerson \(2023\)Dissecting recall of factual associations in auto\-regressive language models\.InProceedings of EMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.751),[Link](https://arxiv.org/abs/2304.14767)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p1.1)\.
- M\. Geva, A\. Caciularu, K\. R\. Wang, and Y\. Goldberg \(2022\)Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space\.InProceedings of EMNLP,External Links:[Link](https://arxiv.org/abs/2203.14680)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of EMNLP,External Links:[Link](https://arxiv.org/abs/2012.14913)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p1.1)\.
- X\. O\. He \(2024\)Mixture of A Million Experts\.External Links:2407\.04153,[Document](https://dx.doi.org/10.48550/arXiv.2407.04153),[Link](https://arxiv.org/abs/2407.04153)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p2.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2404.06654)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1)\.
- W\. Ikeda, K\. Yano, R\. Takahashi, J\. Lee, K\. Shibata, and J\. Suzuki \(2025\)Layerwise Importance Analysis of Feed\-Forward Networks in Transformer\-based Language Models\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2508.17734)Cited by:[§2\.2](https://arxiv.org/html/2607.16254#S2.SS2.p1.1)\.
- A\. Jaiswal, L\. Hannah, H\. Kim, D\. Hoang, A\. Kundu, M\. Farajtabar, and M\. Cho \(2026\)MemoryLLM: Plug\-n\-Play Interpretable Feed\-Forward Memory for Transformers\.External Links:2602\.00398,[Document](https://dx.doi.org/10.48550/arXiv.2602.00398),[Link](https://arxiv.org/abs/2602.00398)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p2.1)\.
- G\. Kamradt \(2023\)Needle in a haystack\.Note:GitHub repositoryExternal Links:[Link](https://github.com/gkamradt/needle-in-a-haystack)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px4.p1.3)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023\)Inference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2306.03341)Cited by:[§2\.4](https://arxiv.org/html/2607.16254#S2.SS4.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2023\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics \(TACL\)\.External Links:[Link](https://arxiv.org/abs/2307.03172)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1)\.
- X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen \(2024\)ShortGPT: layers in large language models are more redundant than you expect\.External Links:[Link](https://arxiv.org/abs/2403.03853)Cited by:[§2\.2](https://arxiv.org/html/2607.16254#S2.SS2.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2202.05262)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p1.1)\.
- A\. Modarressi, H\. Deilamsalehy, F\. Dernoncourt, T\. Bui, R\. A\. Rossi, S\. Yoon, and H\. Schütze \(2025\)NoLiMa: Long\-Context Evaluation Beyond Literal Matching\.InProceedings of ICML,External Links:[Link](https://arxiv.org/abs/2502.05167)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px4.p1.3)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.External Links:[Link](https://arxiv.org/abs/2312.06681)Cited by:[§2\.4](https://arxiv.org/html/2607.16254#S2.SS4.p1.1)\.
- T\. P\. Pires, A\. V\. Lopes, Y\. Assogba, and H\. Setiawan \(2023\)One wide feedforward is all you need\.InProceedings of WMT \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2309.01826)Cited by:[§2\.2](https://arxiv.org/html/2607.16254#S2.SS2.p1.1)\.
- Qwen Team \(2024\)Qwen2\.5 Technical Report\.External Links:2412\.15115,[Document](https://dx.doi.org/10.48550/arXiv.2412.15115),[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4](https://arxiv.org/html/2607.16254#S4.SS0.SSS0.Px2.p1.1)\.
- R\. Sadhukhan, S\. Cao, H\. Dong, C\. Zhao, A\. Purpura\-Pontoniere, Y\. Tian, Z\. Liu, and B\. Chen \(2026\)STEM: Scaling Transformers with Embedding Modules\.External Links:2601\.10639,[Document](https://dx.doi.org/10.48550/arXiv.2601.10639),[Link](https://arxiv.org/abs/2601.10639)Cited by:[§1](https://arxiv.org/html/2607.16254#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16254#S2.SS1.p2.1)\.
- N\. Subramani, N\. Suresh, and M\. E\. Peters \(2022\)Extracting latent steering vectors from pretrained language models\.InFindings of ACL,External Links:[Link](https://arxiv.org/abs/2205.05124)Cited by:[§2\.4](https://arxiv.org/html/2607.16254#S2.SS4.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2023\)Steering language models with activation engineering\.External Links:[Link](https://arxiv.org/abs/2308.10248)Cited by:[§2\.4](https://arxiv.org/html/2607.16254#S2.SS4.p1.1)\.
- X\. Zhang, Y\. Chen, S\. Hu,et al\.\(2024\)∞\\inftyBench: extending long context evaluation beyond 100k tokens\.External Links:[Link](https://arxiv.org/abs/2402.13718)Cited by:[§2\.3](https://arxiv.org/html/2607.16254#S2.SS3.p1.1)\.