HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

# HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
Source: [https://arxiv.org/html/2605.08143](https://arxiv.org/html/2605.08143)
Yuan Fang1Yi Xie2Xuming Ran3 1IXL Learning, Inc2Technical University of Munich3National University of Singapore

###### Abstract

Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment, yet retraining is costly prohibitive, motivating*model editing*in lifelong settings that updates targeted behavior without harming the rest of the model\. One line of work installs new facts by directly modifying base weights through locate\-then\-edit procedures, but accumulated edits progressively disrupt originally preserved knowledge, even with constraint\-based projections\. A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges and its performance degrades at scale\. We proposeHoReN, a codebook\-based parameter\-preserving editor with enhanced routing built on three ideas\. First, HoReN wraps a single MLP layer with a discrete key–value codebook, where each entry is interpreted simultaneously as a knowledge\-memory key and a modern Hopfield stored pattern\. Second, both keys and queries are projected onto the unit hypersphere so retrieval is governed by angular similarity, removing magnitude\-driven mismatches between an edit prompt and its rephrasings\. Third, the query is refined through damped Hopfield attractor dynamics, so paraphrases relax into the correct stored pattern’s basin of attraction while unrelated queries remain undisturbed\. HoReN achieves well\-edited performance with consistent gains across diverse benchmarks spanning standard ZsRE, structured WikiBigEdit, and unstructured UnKE evaluations\. Moreover, HoReN scales to 50K sequential edits on ZsRE with stable overall performance above 0\.9, while prior editors collapse or degrade severely before reaching 10K\. Our code is available at[https://github\.com/ha11ucin8/HoReN](https://github.com/ha11ucin8/HoReN)\.

## 1Introduction

Large language models \(LLMs\)\[[4](https://arxiv.org/html/2605.08143#bib.bib20),[29](https://arxiv.org/html/2605.08143#bib.bib21),[2](https://arxiv.org/html/2605.08143#bib.bib19),[1](https://arxiv.org/html/2605.08143#bib.bib24)\]have become the de facto interface for open\-domain question answering, code, and dialogue, but the world they speak about is not static\. Facts go out of date, deployments uncover incorrect or hallucinated outputs, and downstream operators routinely need to inject corrections that were not present at pre\-training time\. Re\-training a billion\-parameter model end\-to\-end for every such update is prohibitively expensive and risks degrading capabilities acquired from the original corpus—an instance of the classical catastrophic\-forgetting problem\[[11](https://arxiv.org/html/2605.08143#bib.bib22),[24](https://arxiv.org/html/2605.08143#bib.bib7),[28](https://arxiv.org/html/2605.08143#bib.bib6),[17](https://arxiv.org/html/2605.08143#bib.bib5)\]—which has motivated a growing body of work on*model editing*: targeted procedures that revise the behavior of a frozen LLM on a specified prompt while leaving its other behaviors intact\[[18](https://arxiv.org/html/2605.08143#bib.bib1),[7](https://arxiv.org/html/2605.08143#bib.bib15),[26](https://arxiv.org/html/2605.08143#bib.bib11)\]\. In practice the demand is not for a single edit but for a continually arriving stream of them, evaluated jointly on three competing axes:*reliability*on the edited prompt,*generalization*to natural rephrasings, and*locality*on unrelated inputs\[[26](https://arxiv.org/html/2605.08143#bib.bib11),[27](https://arxiv.org/html/2605.08143#bib.bib16),[16](https://arxiv.org/html/2605.08143#bib.bib4)\]\.

![Refer to caption](https://arxiv.org/html/2605.08143v1/x1.png)Figure 1:Overall Performance \(OP, geometric mean of Reliability, Generalization, and Locality\) scaling from100100to50​K50\\text\{K\}sequential edits on ZsRE \(LLaMA\-3\.1\-8B\)\. HoReN sustains OP≥0\.93\\geq 0\.93through 50K edits, while every baseline collapses or plateaus before 10K: ROME fails immediately \(0\.030\.03\), GRACE flatlines at0\.370\.37–0\.390\.39, WISE drifts down to0\.510\.51, AlphaEdit cliffs from0\.820\.82at 2K to0\.080\.08at 10K, and UltraEdit drifts down to0\.670\.67\. Beyond 10K only HoReN remains within the0\.900\.90stability band\. Per\-metric breakdowns and the per\-checkpoint table are in Appendix[E\.1](https://arxiv.org/html/2605.08143#A5.SS1)\.Existing editors fall into two broad paradigms\.Parameter\-modifyingmethods change the weights of the base model\. The dominant locate\-and\-edit line, exemplified by ROME\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]and its successors\[[19](https://arxiv.org/html/2605.08143#bib.bib2),[5](https://arxiv.org/html/2605.08143#bib.bib25)\], identifies a set of MLP weights that the causal\-trace analysis of ROME shows to act as key–value associative memories storing factual knowledge, then applies updates to the corresponding weights so that a target “key” \(the subject representation\) maps to a new “value” \(the desired object\)\. A second branch within this paradigm, represented by UltraEdit\[[6](https://arxiv.org/html/2605.08143#bib.bib26)\], departs from the locate\-then\-edit recipe and instead computes one\-shot parameter shifts directly from the hidden state and its gradient, controlled through a lifelong feature\-statistics normalization that is continually updated across edits\.Parameter\-preservingmethods, in contrast, leave the base weights untouched and route edited behavior through external memory: GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]maintains a discrete codebook at a chosen layer and overrides the layer’s output when an incoming activation falls within a deferral radius of a stored key, while WISE\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\]keeps a separate “side\-memory” copy of the editable region and gates between the side memory and the original weights at inference\.

Each paradigm exposes a structural weakness once edits accumulate\. The parameters of production\-scale LLMs encode the statistics of trillions of pre\-training tokens; updating them with the gradient of a single edit overfits to that one sample and erodes neighboring knowledge\[[11](https://arxiv.org/html/2605.08143#bib.bib22)\]\. Constraint\-based remedies such as AlphaEdit’s null\-space projection\[[5](https://arxiv.org/html/2605.08143#bib.bib25)\]attempt to confine the perturbation to directions that leave a set of*preserved\-knowledge*key vectors unchanged, but the projection has two intrinsic limitations in a sequential regime\. First, the null space is*estimated*from a finite sample of pre\-training\-style activations, so its coverage of what the base model actually represents is only a proxy\. Second, once the editing stream begins the projector itself is not refreshed to include the keys of previously applied edits, so AlphaEdit’s own past edits are no longer protected against the next update—reliability on early edits drifts and locality cliffs as the codebook grows \(Figure[1](https://arxiv.org/html/2605.08143#S1.F1)\)\. Parameter\-preserving methods bypass this catastrophic\-forgetting failure mode by construction but inherit a different one: WISE preserves locality but its side\-memory performance degrades as more edits try to reside in the same side\-memory or a set of side\-memories, while GRACE’s hard\-radius nearest\-neighbor lookup is reliable on exact prompts and fails to recognize natural rephrasings of an edited fact, leaving generalization near zero\.

The routing failure in parameter\-preserving editors traces to a specific geometric mismatch\. Table[1](https://arxiv.org/html/2605.08143#S3.T1)and Figure[5](https://arxiv.org/html/2605.08143#A5.F5)confirm the pattern across all baselines: memory\-based editors preserve reliability but collapse on generalization as edits accumulate\. The root cause is that edit keys are constructed from the original prompt’s hidden states, while a rephrase produces a different representation, creating a non\-trivial angular gap between the stored key and the incoming query\. A diagnostic comparison \(Table[10](https://arxiv.org/html/2605.08143#A7.T10)\) shows that normalization removes magnitude\-driven mismatches yet this angular gap persists—key matching alone cannot bridge it\. The problem compounds with scale: a larger codebook introduces more competing keys, amplifying routing ambiguity even when the correct key exists\. Large\-scale sequential model editing therefore requires a retrieval rule that preserves angular separation for unrelated inputs while actively pulling rephrased queries toward the correct edit\.

We proposeHoReN\(HopfieldRetrieval withNormalized representations\), a parameter\-preserving editor designed to inherit the locality of the codebook paradigm while closing its generalization gap\. HoReN combines three ideas that prior work has kept separate\. \(i\) The*codebook architecture*of GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]: a frozen base model with one MLP layer wrapped by a discrete key–value memory, so each edit is stored once and never re\-touched\. \(ii\)*Normalization*: keys and queries are projected onto the unit hypersphere so retrieval depends on angular similarity rather than activation magnitude—justified by the geometry of post\-activation MLP tokens at the routing layer, where the direction of a hidden\-state vector encodes which fact is recalled while its magnitude reflects prompt\-specific gain, making cosine matching on the unit hypersphere the geometrically natural primitive\. \(iii\)*Hopfield dynamics*\[[23](https://arxiv.org/html/2605.08143#bib.bib13),[12](https://arxiv.org/html/2605.08143#bib.bib14)\]: each codebook entry is read simultaneously as a knowledge\-memory key in the sense of ROME\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]and as a stored pattern of a modern Hopfield network, so the codebook becomes a field of energy basins, and one damped attractor step before matching lets a paraphrase relax into the basin of the correct memory while leaving unrelated queries essentially undisturbed\. The matched edit is realized by a lightweight value adaptor at the next layer, so the entire intervention is concentrated on the routing decision\.

The two components play complementary roles: normalization protects locality by keeping unrelated queries separated on the hypersphere, while one damped Hopfield step closes the generalization gap by allowing rephrased queries to relax into the basin of the correct edit\. We validate HoReN with theoretical analysis and large\-scale experiments, and the two design choices each receive an interpretable empirical correlate\. Diagnostic comparisons against an unnormalized variant confirm the ROME\-aligned hypothesis that for post\-activation MLP tensors it is the*direction*, not the magnitude, of the activation that selects which knowledge is recalled, so cosine matching on the unit sphere is the right primitive on which to build retrieval\. Hopfield refinement supplies the second piece: the codebook becomes a field of energy basins whose attractor dynamics gate which stored pattern is retrieved for a given query, with a principled generalization/locality trade\-off controlled by the number of refinement steps\. The combination produces a sequential editor that scales without catastrophic degradation: on a controlled ZsRE stress test on LLaMA\-3\.1\-8B, HoReN keeps reliability, generalization, and locality all above89%89\\%out to 50K accumulated edits \(Figure[1](https://arxiv.org/html/2605.08143#S1.F1)\), where every baseline we compare against has either cliffed \(AlphaEdit, ROME\), drifted \(UltraEdit, WISE\), or flat\-lined on generalization \(GRACE\) well before 10K\. The same advantage carries over to structured WikiBigEdit\[[25](https://arxiv.org/html/2605.08143#bib.bib30)\]and unstructured UnKE\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]\. Our main contributions are:

- •We characterize a structural divide in lifelong editing: parameter\-modifying editors degrade preserved knowledge as their constraint subspaces fail to cover accumulated edits, while parameter\-preserving editors lose generalization because magnitude\-sensitive nearest\-neighbor routing cannot bridge the representation gap between an edit prompt and its rephrasings\.
- •We propose HoReN, a parameter\-preserving editor that unifies the ROME key–value memory view with the GRACE codebook architecture under a normalized Hopfield retrieval rule: angular matching on the unit hypersphere supplies locality, and one damped Hopfield step supplies paraphrase\-robust generalization\.
- •We provide theoretical analysis showing that iterating standard Hopfield dynamics to convergence attracts all queries—including unrelated ones—toward stored codes, formally motivating HoReN’s single\-step deployment; and we confirm empirically that normalization is the primary driver of the generalization gain by removing magnitude\-driven mismatches from the key comparison\.
- •We demonstrate large\-scale stability on a 50K\-edit ZsRE stress test on LLaMA\-3\.1\-8B—where prior methods cliff, drift, or flat\-line well before 10K edits—and confirm consistent gains on structured WikiBigEdit and unstructured UnKE across seven LLMs spanning four model families\.

![Refer to caption](https://arxiv.org/html/2605.08143v1/x2.png)Figure 2:Overall architecture of HoReN combining normalized representations with Hopfield\-style retrieval\. The pipeline:\(1\)extract hidden states from layerll;\(2\)pool token representations to construct the query;\(3\)normalize query to the unit hypersphere;\(4\)apply one\-step Hopfield\-style refinement over the normalized codebook;\(5\)perform key matching and value adaptation at layerl\+1l\+1\. This design achieves complementary benefits: normalization preserves locality through angular similarity matching, while Hopfield\-style refinement improves paraphrase routing by moving the query toward edit\-key attractors\.
## 2Method

HoReN inherits the architectural skeleton of GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]: the base modelfθ0f\_\{\\theta\_\{0\}\}is frozen and a single MLP layerllis wrapped with a discrete codebook𝒞=\{\(ki,vi,yi\)\}i=1C\\mathcal\{C\}=\\\{\(k\_\{i\},v\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{C\}\(Figure[2](https://arxiv.org/html/2605.08143#S1.F2)\)\. At inference, an incoming query is routed through the codebook; a successful match overrides the layer’s output with the matched value, while an unmatched query passes through the base model unchanged\. Editing reduces to two questions:*what is stored*and*how is a query routed to the right entry*\. HoReN’s answers to both flow from treating each stored key simultaneously as a ROME\-style knowledge\-memory index\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]and as an attractor basin in a modern Hopfield network\[[23](https://arxiv.org/html/2605.08143#bib.bib13)\]: keys and queries are projected onto the unit hypersphere so that retrieval is purely angular \(Section[2\.2](https://arxiv.org/html/2605.08143#S2.SS2)\), and routing proceeds by a single controlled relaxation under the codebook’s attractor field before any matching decision \(Section[2\.3](https://arxiv.org/html/2605.08143#S2.SS3)\)\.

### 2\.1Problem Formulation

Setup\.Following the lifelong editing setup of\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\], letfθ:𝒳→𝒴f\_\{\\theta\}:\\mathcal\{X\}\\rightarrow\\mathcal\{Y\}be an LLM with pre\-edit parametersθ0\\theta\_\{0\}\. Each edit indexttcarries an original promptxtx\_\{t\}, a paraphrasex~t\\tilde\{x\}\_\{t\}, a shared targetyt∗y\_\{t\}^\{\*\}, and an unrelated locality queryxtlocx\_\{t\}^\{\\mathrm\{loc\}\}\. At stepttthe editor produces

fθt=ModelEditing​\(fθt−1,xt,yt∗\),so thatfθt​\(x\)=\{yt∗,x∈\{xt,x~t\},fθ0​\(x\),x=xtloc\.f\_\{\\theta\_\{t\}\}=\\mathrm\{ModelEditing\}\(f\_\{\\theta\_\{t\-1\}\},x\_\{t\},y\_\{t\}^\{\*\}\),\\quad\\text\{so that\}\\quad f\_\{\\theta\_\{t\}\}\(x\)=\\begin\{cases\}y\_\{t\}^\{\*\},&x\\in\\\{x\_\{t\},\\tilde\{x\}\_\{t\}\\\},\\\\ f\_\{\\theta\_\{0\}\}\(x\),&x=x\_\{t\}^\{\\mathrm\{loc\}\}\.\\end\{cases\}\(1\)AfterNNsequential edits, the final model must satisfy this condition for*all*t∈\{1,…,N\}t\\in\\\{1,\\ldots,N\\\}, measured by reliability, generalization, and locality \(definitions and dataset\-specific instantiations in Section[3\.1](https://arxiv.org/html/2605.08143#S3.SS1)\)\. The difficulty of the problem grows withNN: each new edit must coexist with every prior one, so the editor needs a memory that scales gracefully and a routing rule that remains discriminative even as nearby memories accumulate\.

### 2\.2The Codebook: Normalized Keys on the Unit Hypersphere

Each codebook entry\(ki,vi,yi\)\(k\_\{i\},v\_\{i\},y\_\{i\}\)at editing layerllbinds a keyki∈ℝdk\_\{i\}\\in\\mathbb\{R\}^\{d\}to a value vectorvi∈ℝdv\_\{i\}\\in\\mathbb\{R\}^\{d\}and the target labelyiy\_\{i\}that originally created the entry\. Keys are constructed from the layer\-llhidden states of the prompt that triggered the edit, and queries are constructed in the same way at routing time, so that storage and retrieval live in a common representation space\.

Concretely, given an inputxxwe extractHin∈ℝL×dH\_\{\\text\{in\}\}\\in\\mathbb\{R\}^\{L\\times d\}from the frozen base at layerlland pool the lastn%n\\%of token positions into a raw queryq^=select​\(Hin\)∈ℝd\\hat\{q\}=\\mathrm\{select\}\(H\_\{\\text\{in\}\}\)\\in\\mathbb\{R\}^\{d\}\. Pooling a suffix of tokens—rather than a single position—summarises the semantic content of the prompt while remaining robust to surface variation such as padding and minor rephrasing\. We then project to the unit hypersphere,

q0=normalize​\(q^\)∈ℝ1×d,K∈ℝC×d,‖ki‖2=1,q\_\{0\}=\\mathrm\{normalize\}\(\\hat\{q\}\)\\in\\mathbb\{R\}^\{1\\times d\},\\qquad K\\in\\mathbb\{R\}^\{C\\times d\},\\;\\;\\\|k\_\{i\}\\\|\_\{2\}=1,whereC=\|𝒞\|≤NC=\|\\mathcal\{C\}\|\\leq Nis the current codebook size\. The motivation is the geometric fact that, for post\-activation MLP tensors, the direction of the vector is what binds to a particular concept while the magnitude largely reflects prompt\-dependent activation strength; normalising removes this nuisance variation and exposes the angular structure that the keys are actually selecting on\.

Once a query has been routed \(Section[2\.3](https://arxiv.org/html/2605.08143#S2.SS3)\), the matching decision itself is a single inner product: the editor scores keys as𝐚=q​K⊤\\mathbf\{a\}=qK^\{\\top\}, takesi⋆=arg⁡maxi⁡𝐚ii^\{\\star\}=\\arg\\max\_\{i\}\\mathbf\{a\}\_\{i\}, and accepts the match when𝐚i⋆\>c\\mathbf\{a\}\_\{i^\{\\star\}\}\>c\. Because keys and queries lie on the unit sphere,ccis a cosine threshold and admits a precise geometric reading: it is the angular boundary of the attractor basin aroundki⋆k\_\{i^\{\\star\}\}\. This replaces GRACE’s deferral radius—a Euclidean tolerance whose meaning shifts with activation magnitude—with a single, scale\-free notion of what it means for a query to fall “inside” a stored memory\.

### 2\.3Querying via Hopfield Attractor Dynamics

Treating each stored key as an attractor suggests that routing should not be a one\-shot nearest\-neighbour lookup against the raw queryq0q\_\{0\}\. A paraphrasex~t\\tilde\{x\}\_\{t\}produces an activation that is angularly close to, but not identical to, the activation of the original promptxtx\_\{t\}; under a hard argmax this small angular gap can be enough to send the query to the wrong key once the codebook is dense\. The right object to compare against the codebook is therefore notq0q\_\{0\}itself but the result of lettingq0q\_\{0\}relax briefly under the codebook’s attractor field, so that paraphrases drift toward the basin of the correct memory while unrelated queries are left essentially untouched\.

We make this precise using the modern Hopfield framework\[[23](https://arxiv.org/html/2605.08143#bib.bib13)\]\. With energy

E​\(q,K\)=−lseβ​\(q​K⊤\),lseβ​\(𝐳\)=1β​log​∑ieβ​zi,E\(q,K\)\\;=\\;\-\\mathrm\{lse\}\_\{\\beta\}\\\!\\big\(qK^\{\\top\}\\big\),\\qquad\\mathrm\{lse\}\_\{\\beta\}\(\\mathbf\{z\}\)=\\tfrac\{1\}\{\\beta\}\\log\\textstyle\\sum\_\{i\}e^\{\\beta z\_\{i\}\},the standard updateT​\(q\)=softmax​\(β​q​K⊤\)​KT\(q\)=\\mathrm\{softmax\}\(\\beta\\,qK^\{\\top\}\)\\,KdescendsEEand converges to fixed points that lie in the convex hull of the stored patterns, withβ\>0\\beta\>0controlling retrieval sharpness\. HoReN deploys a*normalized, damped*variant of this dynamic\. Starting fromq←q0q\\leftarrow q\_\{0\}, form=1,…,Mm=1,\\ldots,Mwe compute

p\\displaystyle p=softmax​\(β​q​K⊤\)∈ℝ1×C,\\displaystyle=\\mathrm\{softmax\}\(\\beta\\,qK^\{\\top\}\)\\in\\mathbb\{R\}^\{1\\times C\},\(2\)qnew\\displaystyle q\_\{\\text\{new\}\}=normalize​\(p​K\)∈ℝ1×d,\\displaystyle=\\mathrm\{normalize\}\(pK\)\\in\\mathbb\{R\}^\{1\\times d\},and update

q←normalize​\(\(1−γ\)​q\+γ​qnew\),q\\;\\leftarrow\\;\\mathrm\{normalize\}\\\!\\big\(\(1\-\\gamma\)\\,q\+\\gamma\\,q\_\{\\text\{new\}\}\\big\),\(3\)terminating early once‖qnew−q‖2≤ϵ\\\|q\_\{\\text\{new\}\}\-q\\\|\_\{2\}\\leq\\epsilon\. Two deviations from the standard update matter\. The intermediate re\-normalisationqnew=normalize​\(p​K\)q\_\{\\text\{new\}\}=\\mathrm\{normalize\}\(pK\)keeps the iterate on the same hypersphere as the keys, so the dynamics never leave the geometry on which the codebook is defined\. The damping factorγ∈\(0,1\]\\gamma\\in\(0,1\]moves the query only fractionally towardqnewq\_\{\\text\{new\}\}at each step, so unrelated queries—whose attention weightsppare diffuse—are perturbed only slightly, while paraphrases of an existing edit—whoseppconcentrates on a single key—are pulled into the corresponding basin\.

The single damped step is not an empirical convenience but a principled stopping criterion\. IteratingTTto convergence would attract*any*query, including unrelated or genuinely novel ones, into the convex hull of stored patterns, which for largeβ\\betacollapses near the closest existing key and therefore destroys locality\. We characterise this behaviour with two complementary results, stated here and proved in Appendix[C](https://arxiv.org/html/2605.08143#A3)\.

###### Theorem 2\.1\(Asymptotic convergence of standard Hopfield retrieval\)\.

LetK∈ℝC×dK\\in\\mathbb\{R\}^\{C\\times d\}be a normalized codebook \(‖ki‖2=1\\\|k\_\{i\}\\\|\_\{2\}=1\), letq\(0\)∈ℝ1×dq^\{\(0\)\}\\in\\mathbb\{R\}^\{1\\times d\}satisfy‖q\(0\)‖2=1\\\|q^\{\(0\)\}\\\|\_\{2\}=1, and letβ\>0\\beta\>0\. Define the standard Hopfield update mapT​\(q\):=softmax​\(β​q​K⊤\)​KT\(q\):=\\mathrm\{softmax\}\(\\beta\\,qK^\{\\top\}\)\\,Kand generate the infinite sequenceq\(s\+1\)=T​\(q\(s\)\)q^\{\(s\+1\)\}=T\(q^\{\(s\)\}\)\. WithF​\(q\):=1β​log​∑i=1Cexp⁡\(β​q​ki⊤\)F\(q\):=\\tfrac\{1\}\{\\beta\}\\log\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\)and energyE​\(q,K\):=12​‖q‖22−F​\(q\)E\(q,K\):=\\tfrac\{1\}\{2\}\\\|q\\\|\_\{2\}^\{2\}\-F\(q\), the energy is monotonically non\-increasing,

E​\(q\(s\+1\),K\)−E​\(q\(s\),K\)≤−12​‖q\(s\+1\)−q\(s\)‖22,E\(q^\{\(s\+1\)\},K\)\-E\(q^\{\(s\)\},K\)\\;\\leq\\;\-\\tfrac\{1\}\{2\}\\,\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\},and the sequence converges to a fixed point:q\(s\)→q∗q^\{\(s\)\}\\to q^\{\\ast\}withq∗=T​\(q∗\)q^\{\\ast\}=T\(q^\{\\ast\}\)\.

###### Proposition 2\.2\(Finite\-step descent and residual bound\)\.

Under the assumptions of Theorem[2\.1](https://arxiv.org/html/2605.08143#S2.Thmtheorem1), for any finiteN≥1N\\geq 1the truncated iteratesq\(s\+1\)=T​\(q\(s\)\)q^\{\(s\+1\)\}=T\(q^\{\(s\)\}\),s=0,…,N−1s=0,\\ldots,N\-1, satisfy the cumulative descent inequality

∑s=0N−1‖q\(s\+1\)−q\(s\)‖22≤2​\(E​\(q\(0\),K\)−E​\(q\(N\),K\)\),\\sum\_\{s=0\}^\{N\-1\}\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\}\\;\\leq\\;2\\bigl\(E\(q^\{\(0\)\},K\)\-E\(q^\{\(N\)\},K\)\\bigr\),and consequently the iterate with smallest fixed\-point residual obeys the explicit bound

min0≤s<N⁡‖T​\(q\(s\)\)−q\(s\)‖2≤2N\.\\min\_\{0\\leq s<N\}\\,\\\|T\(q^\{\(s\)\}\)\-q^\{\(s\)\}\\\|\_\{2\}\\;\\leq\\;\\frac\{2\}\{\\sqrt\{N\}\}\.The bound applies to the*best*of the firstNNiterates; the last iterateq\(N\)q^\{\(N\)\}is only guaranteed to satisfyE​\(q\(N\),K\)≤E​\(q\(0\),K\)E\(q^\{\(N\)\},K\)\\leq E\(q^\{\(0\)\},K\)and may not itself be an approximate fixed point\.

Theorem[2\.1](https://arxiv.org/html/2605.08143#S2.Thmtheorem1)formalises the over\-attraction risk: in the limit, every query is dragged into the convex hull of stored codes regardless of whether it actually corresponds to an edit\. Proposition[2\.2](https://arxiv.org/html/2605.08143#S2.Thmtheorem2)shows that this contraction is already operative at finite step counts in an averaged sense, so even partial iteration progressively erodes locality\. The normalized, damped single\-step variant deployed by HoReN preserves the generalisation benefit of attractor dynamics—paraphrases are pulled toward the right basin—while controlling exactly this over\-attraction;MMthen exposes a principled generalisation/locality trade\-off whose empirical behaviour we study in Section[3\.4\.2](https://arxiv.org/html/2605.08143#S3.SS4.SSS2)\(Figure[10](https://arxiv.org/html/2605.08143#A7.F10), Table[11](https://arxiv.org/html/2605.08143#A7.T11)\)\.

### 2\.4Updating the Codebook

The codebook grows online as edits arrive\. After routing, three cases can arise\. \(i\) No\-match: the best score𝐚i⋆≤c\\mathbf\{a\}\_\{i^\{\\star\}\}\\leq cfalls outside every existing basin, so the edit is genuinely new and we insert a fresh entry\(q0,vnew,yt∗\)\(q\_\{0\},v\_\{\\text\{new\}\},y\_\{t\}^\{\*\}\), wherevnewv\_\{\\text\{new\}\}is trained by cross\-entropy on the targetyt∗y\_\{t\}^\{\*\}with the base model frozen\. \(ii\) Match with consistent label \(yi⋆=yt∗y\_\{i^\{\\star\}\}=y\_\{t\}^\{\*\}\): the edit is a paraphrase or a reassertion, and the existing\(ki⋆,vi⋆\)\(k\_\{i^\{\\star\}\},v\_\{i^\{\\star\}\}\)is reused without modification\. \(iii\) Match with conflicting label \(yi⋆≠yt∗y\_\{i^\{\\star\}\}\\neq y\_\{t\}^\{\*\}\): the basin is contested and we insert a new entry to preserve the new fact, leaving the prior memory in place\. At inference, a successful match adds the matched value at the next layer on each answer\-token positionjjviahj′=hj\+vi⋆h\_\{j\}^\{\\prime\}=h\_\{j\}\+v\_\{i^\{\\star\}\}; on a no\-match the base model is left unchanged\. An alternative LoRA\-based adaptor is described in Appendix[G\.3](https://arxiv.org/html/2605.08143#A7.SS3), and the full procedure is given in Algorithm[1](https://arxiv.org/html/2605.08143#algorithm1)\(Appendix[B\.1](https://arxiv.org/html/2605.08143#A2.SS1)\)\.

##### Routing mechanism\.

The routing mechanism has a key structural asymmetry: stored keys are always the*pre\-refinement*queryq0q\_\{0\}, while the routing comparison uses the refinedqqoutput by the Hopfield step\. This asymmetry is not a tuning choice but follows directly from the attractor geometry\. If newly stored keys were the output of the Hopfield refinement, each insertion would pull subsequent insertions toward existing attractors, basins would co\-move with the codebook, and the angular separation that retrieval depends on would progressively collapse\. Pinning storage toq0q\_\{0\}anchors the codebook to the model’s native representational geometry, so the attractor field is fixed by what the base model actually produces; the refinement step then moves paraphrases toward the correct basin at routing time without contaminating what is stored\.

## 3Experiments

### 3\.1Experimental Setup

We briefly outline the base LLMs, baselines, datasets, evaluation metrics, and implementation\. Full details are in Appendix[D](https://arxiv.org/html/2605.08143#A4)\.

##### Base LLMs & Baseline Methods\.

Our experiments are conducted on seven LLMs spanning 1\.5B to 32B parameters across four families: DeepSeek\-R1\-Distill\-Qwen\-1\.5B\[[2](https://arxiv.org/html/2605.08143#bib.bib19)\], LLaMA\-3\-8B\-Instruct and LLaMA\-3\.1\-8B\-Instruct\[[4](https://arxiv.org/html/2605.08143#bib.bib20)\], Qwen2\.5\-7B\-Instruct\[[29](https://arxiv.org/html/2605.08143#bib.bib21)\], DeepSeek\-R1\-Distill\-Llama\-8B \(hereafter DeepSeek\-R1\-8B\)\[[2](https://arxiv.org/html/2605.08143#bib.bib19)\], GPT\-OSS\-20B\[[1](https://arxiv.org/html/2605.08143#bib.bib24)\], and Qwen\-SEA\-LION\-v4\-32B\-IT\[[22](https://arxiv.org/html/2605.08143#bib.bib12)\]\. We compare HoReN against representative methods from two paradigms: \(1\)*locate\-and\-edit*methods, including ROME\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\], AlphaEdit\[[5](https://arxiv.org/html/2605.08143#bib.bib25)\], and UltraEdit\[[6](https://arxiv.org/html/2605.08143#bib.bib26)\]; and \(2\)*retrieval\-based lifelong editing*methods, including GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]and WISE\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\]\. For unstructured editing we additionally compare against the domain\-specific UnKE baseline\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]\.

##### Datasets & Evaluation Metrics\.

We evaluate HoReN on three benchmarks that progressively stress its core separation guarantee:ZsRE\[[13](https://arxiv.org/html/2605.08143#bib.bib23)\], the standard lifelong editing benchmark with semantically well\-separated edit and locality queries;WikiBigEdit\[[25](https://arxiv.org/html/2605.08143#bib.bib30)\], a harder regime in which locality queries share subjects and relations with edited facts; andUnKE\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\], an unstructured benchmark with free\-form edit targets\. Following prior work\[[7](https://arxiv.org/html/2605.08143#bib.bib15),[26](https://arxiv.org/html/2605.08143#bib.bib11),[27](https://arxiv.org/html/2605.08143#bib.bib16)\], we report Reliability, Generalization, and Locality, together with their geometric mean \(Overall Performance, OP\)\. For UnKE we follow its original protocol\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]and report ROUGE\-1/2/L and BERTScore\. Full dataset and metric definitions are in Appendices[D\.1](https://arxiv.org/html/2605.08143#A4.SS1)and[D\.3](https://arxiv.org/html/2605.08143#A4.SS3)\.

##### Implementation\.

HoReN usesβ=20\\beta\{=\}20,γ=0\.01\\gamma\{=\}0\.01,M=1M\{=\}1,ϵ=10−5\\epsilon\{=\}10^\{\-5\}across all experiments\. Two model\-specific parameters—matching thresholdccand token pooling ratio—are selected once per model via ablation \(Table[3](https://arxiv.org/html/2605.08143#A4.T3)\)\. The default value adaptor is direct; the LoRA variant usesr=4r\{=\}4\. Per\-edit adaptor optimization uses Adam with learning rate0\.10\.1, up toU=50U\{=\}50steps with early stopping \(Appendix[D\.4\.3](https://arxiv.org/html/2605.08143#A4.SS4.SSS3)\)\. Experiments run on RTX 5090 in fp16 \(GPT\-OSS\-20B on RTX Pro 6000 WS\)\. Code and logs:[https://github\.com/ha11ucin8/HoReN](https://github.com/ha11ucin8/HoReN)\.

### 3\.2ZsRE Baseline Failure Modes and Large\-Scale Stability

AtN=1000N\{=\}1000, every baseline exhibits at least one critical failure mode on ZsRE; Tables[1](https://arxiv.org/html/2605.08143#S3.T1)–[2](https://arxiv.org/html/2605.08143#S3.T2)and Figure[1](https://arxiv.org/html/2605.08143#S1.F1)show that these failures either accelerate \(cliff\) or persist \(slope\) as edits scale to 50K, while HoReN maintains all three metrics throughout\.

#### 3\.2\.1Baseline Comparison on ZsRE

Table 1:Performance comparison on ZsRE benchmark at different editing scales \(N∈\{100,500,1000\}N\\in\\\{100,500,1000\\\}\) on four models\.Fact \(Table[1](https://arxiv.org/html/2605.08143#S3.T1)\)\.AtN=1000N\{=\}1000on LLaMA\-3\.1\-8B, HoReN reaches OP0\.970\.97\(Rel\.0\.990\.99, Gen\.0\.930\.93, Loc\.0\.990\.99\); the strongest parameter\-modifying baseline AlphaEdit reaches OP0\.870\.87\(Loc\.0\.740\.74\); UltraEdit reaches OP0\.740\.74\(Loc\.0\.540\.54\); the parameter\-preserving baselines GRACE and WISE reach OP0\.370\.37\(Gen\.0\.050\.05\) and0\.710\.71respectively; ROME collapses to OP0\.030\.03\. The same pattern holds on LLaMA\-3\-8B, Qwen\-2\.5\-7B, and Qwen\-SEA\-LION\-v4\-32B: HoReN is the only method that keeps Reliability, Generalization, and Locality simultaneously above0\.840\.84on every row\.

These numbers separate cleanly into the two halves\. ROME’s collapse is mechanistic: a closed\-form rank\-one update fitted to a single edit prompt is a point constraint, not a fact constraint, and stacking such updates pulls neighboring representations along with the targeted one—which is why Locality is already0\.010\.01atN=100N\{=\}100and0\.030\.03atN=1000N\{=\}1000\. AlphaEdit and UltraEdit get the mechanistic half right \(OP0\.870\.87,0\.740\.74atN=1000N\{=\}1000\) and break on the statistical half: AlphaEdit’s null\-space projector protects the directions in its proxy preservation sample but cannot include directions added by later edits, so Locality is the metric that moves first \(0\.89→0\.740\.89\\to 0\.74asNNgrows from100100to10001000\); UltraEdit’s running normalization stabilizes the update geometry but cannot manufacture paraphrase coverage from a single sample, so the same pattern is more gradual \(0\.69→0\.540\.69\\to 0\.54\)\. GRACE inverts the failure: Reliability and Locality stay essentially perfect \(≥0\.99\\geq 0\.99on every row\), but Generalization is stuck at0\.010\.01–0\.060\.06at everyNNon every model\. The bottleneck is geometric, not informational—a paraphrase produces a different post\-nonlinearity activation than the prompt that seeded the key, and unnormalized nearest\-neighbor lookup conflates the prompt\-specific gain \(magnitude\) with the fact identity \(direction\),

HoReN’s Rel\.0\.990\.99, Gen\.0\.930\.93, Loc\.0\.990\.99on the LLaMA\-3\.1\-8BN=1000N\{=\}1000row is the empirical signature of fixing exactly the geometric defect that pins GRACE: keys and queries are compared by direction on the unit hypersphere, and a single damped Hopfield refinement closes the residual angular gap between paraphrase and key without contracting unrelated queries\. Section[3\.4\.1](https://arxiv.org/html/2605.08143#S3.SS4.SSS1)isolates each ingredient on its own row of Table[10](https://arxiv.org/html/2605.08143#A7.T10)\.

Takeaway\.Each baseline fails on a different half of the problem: ROME on the mechanistic half via direct weight entanglement; AlphaEdit and UltraEdit on the statistical half via stale preservation samples; GRACE on the statistical half via routing geometry; WISE on a drifting routing rule\. HoReN is the only method whose three metrics are simultaneously≥0\.84\\geq 0\.84on everyN=1000N\{=\}1000row of Table[1](https://arxiv.org/html/2605.08143#S3.T1)\.

#### 3\.2\.2Scaling to 10K and 50K Edits

Table 2:Scaling to 10K edits on ZsRE across four models \(Qwen2\.5\-7B, DeepSeek\-R1\-8B, GPT\-OSS\-20B, and Qwen\-SEA\-LION\-v4\-32B\)\. Full LLaMA\-3\.1\-8B results at large scale, including AlphaEdit and UltraEdit, are in Appendix[E\.1](https://arxiv.org/html/2605.08143#A5.SS1)\.Fact \(Tables[2](https://arxiv.org/html/2605.08143#S3.T2)–[4](https://arxiv.org/html/2605.08143#A5.T4), Figure[1](https://arxiv.org/html/2605.08143#S1.F1)\)\.On LLaMA\-3\.1\-8B \(Table[4](https://arxiv.org/html/2605.08143#A5.T4)\), AlphaEdit’s OP drops from0\.820\.82atN=2000N\{=\}2000to0\.100\.10atN=5000N\{=\}5000as Locality falls from0\.650\.65to0\.030\.03; UltraEdit’s OP drifts from0\.730\.73atN=2000N\{=\}2000to0\.670\.67atN=10000N\{=\}10000with Locality eroding from0\.530\.53to0\.420\.42\. On Qwen2\.5\-7B, DeepSeek\-R1\-8B, GPT\-OSS\-20B, and Qwen\-SEA\-LION\-v4\-32B \(Table[2](https://arxiv.org/html/2605.08143#S3.T2)\), GRACE’s Generalization stays in\[0\.01,0\.03\]\[0\.01,0\.03\]at every scale on every model; WISE’s OP collapses to0\.320\.32–0\.590\.59on Qwen and DeepSeek, falls below0\.130\.13on GPT\-OSS\-20B, and decays from0\.720\.72to0\.680\.68on SEA\-LION\. HoReN keeps Reliability0\.990\.99–1\.001\.00, Generalization0\.660\.66–0\.950\.95, and Locality0\.970\.97–1\.001\.00across all four models and scales; on the LLaMA\-3\.1\-8B5050K stress test \(Figure[1](https://arxiv.org/html/2605.08143#S1.F1)\) all three metrics remain above0\.890\.89\.

The two failure shapes are diagnostic\. AlphaEdit’s behavior is a*cliff*: a fixed null\-space projector is computed once against a proxy preservation sample; every accepted edit becomes new knowledge that later edits must also preserve, and a fixed projector cannot include those directions, so Locality is intact until the edited subspace exceeds the projector’s coverage and then breaks\. UltraEdit’s behavior is a*slope*: the running normalization that replaces explicit preservation stabilizes update geometry but cannot manufacture paraphrase coverage from one sample, so Locality erodes monotonically rather than catastrophically\. The codebook methods isolate the routing component on its own: GRACE’s flat Generalization acrossNNon every model rules out a capacity explanation \(the codebook can absorb10 00010\\,000entries without trouble\) and pins the failure on the geometry of the key\-query comparison; WISE’s monotone OP decay localizes its weakness to the side\-memory routing rule itself, which drifts as the codebook accumulates\.

HoReN’s flatness acrossNNis the empirical complement of these failures: storage is per\-entry \(so a new edit cannot perturb older entries\) and routing depends only on directional geometry of stored keys \(so adding entries does not move existing decision boundaries unless the new key is angularly close to an old one, in which case the matching threshold defers the new edit\)\. The trajectory through5050K edits in Figure[1](https://arxiv.org/html/2605.08143#S1.F1)and the trajectory figures in Appendix[E\.1](https://arxiv.org/html/2605.08143#A5.SS1)\(Figures[6](https://arxiv.org/html/2605.08143#A5.F6)–[8](https://arxiv.org/html/2605.08143#A5.F8)\) show no visible cliff and no monotone slope, which is the operational signature of a method that decouples the cost of a new edit from the behavior on previously stored ones\.

Takeaway\.Cliff \(AlphaEdit\), slope \(UltraEdit\), flat\-line failure \(GRACE Generalization\), and monotone OP decay \(WISE\) all appear in the scaling tables; HoReN’s three metrics are flat through5050K edits at essentially zero additional cost over GRACE \(Appendix[E\.3](https://arxiv.org/html/2605.08143#A5.SS3), Figure[9](https://arxiv.org/html/2605.08143#A5.F9)\)\.

### 3\.3Generalization Across Datasets and Models

ZsRE provides a favorable separation regime; we now test whether HoReN’s advantage holds under structurally different conditions along two orthogonal dimensions: dataset type \(Section[3\.3\.1](https://arxiv.org/html/2605.08143#S3.SS3.SSS1)\) and model family \(Section[3\.3\.2](https://arxiv.org/html/2605.08143#S3.SS3.SSS2)\)\.

#### 3\.3\.1Cross\-Dataset Generalization

![Refer to caption](https://arxiv.org/html/2605.08143v1/x3.png)Figure 3:Cross\-dataset generalization \(LLaMA\-3\.1\-8B\)\.\(a\) WikiBigEdit\(N=1000N\{=\}1000\): HoReN outperforms AlphaEdit on all three metrics; AlphaEdit’s locality collapses to0\.190\.19while HoReN retains0\.650\.65, a gap of\+0\.46\+0\.46\.\(b\) UnKE\(N=500N\{=\}500\): HoReN outperforms the domain\-specific UnKE baseline on all four metrics: ROUGE\-1 \(\+0\.18\+0\.18\), ROUGE\-2 \(\+0\.13\+0\.13\), ROUGE\-L \(\+0\.17\+0\.17\), and BERTScore \(\+0\.50\+0\.50\), with the largest gap on BERTScore as the UnKE baseline degrades toward failure\. Full results across allNNin Tables[6](https://arxiv.org/html/2605.08143#A6.T6)–[9](https://arxiv.org/html/2605.08143#A6.T9)\(Appendix[F\.1](https://arxiv.org/html/2605.08143#A6.SS1)\)\.Fact \(Figure[3](https://arxiv.org/html/2605.08143#S3.F3), Table[9](https://arxiv.org/html/2605.08143#A6.T9)\)\.On WikiBigEdit, where locality queries share subjects and relations with edits, HoReN sustains Reliability0\.9860\.986–0\.9920\.992, Generalization0\.7530\.753–0\.7670\.767, and Locality0\.6310\.631–0\.6820\.682acrossN∈\{500,1000,3000\}N\{\\in\}\\\{500,1000,3000\\\}; AlphaEdit collapses on Locality to0\.1690\.169–0\.2670\.267in the same grid\. On UnKE, whose targets are multi\-sentence free\-form passages, Figure[3](https://arxiv.org/html/2605.08143#S3.F3)b shows HoReN atN=500N\{=\}500leading the domain\-specific UnKE baseline by\+0\.18\+0\.18ROUGE\-1 \(0\.400\.40vs\.0\.220\.22\),\+0\.13\+0\.13ROUGE\-2,\+0\.17\+0\.17ROUGE\-L, and\+0\.50\+0\.50BERTScore\.

The WikiBigEdit gap is diagnostic for the same reason as the AlphaEdit cliff in Section[3\.2\.2](https://arxiv.org/html/2605.08143#S3.SS2.SSS2)\. AlphaEdit’s preservation guarantee is computed against a proxy distribution; on WikiBigEdit the locality queries are exactly the topically proximal queries that proxy is thinnest on, so the projector cannot distinguish them from rephrase queries at inference time\. HoReN’s matching operates on the angular geometry of the queries themselves at the routing layer, so the deferral threshold acts as a hard basin boundary regardless of how close locality and edit queries are in topic—which is why Locality stays above0\.630\.63on the same rows where AlphaEdit’s Locality is below0\.270\.27\.

The UnKE gap is diagnostic in the orthogonal direction\. The routing mechanism in HoReN reads only the hidden\-state geometry at the chosen layer, independently of the format of the target answer; the key construction and the matching rule are identical when the target is a short ZsRE entity string and when it is a multi\-sentence UnKE passage\. The advantage over the domain\-specific UnKE baseline atN=500N\{=\}500is therefore evidence that the routing layer does not need to be re\-designed when the value\-side format changes\.

#### 3\.3\.2Cross\-Model Generalization

Fact \(Table[2](https://arxiv.org/html/2605.08143#S3.T2)\)\.Across Qwen2\.5\-7B, DeepSeek\-R1\-Distill\-Llama\-8B, GPT\-OSS\-20B, and Qwen\-SEA\-LION\-v4\-32B atN∈\{2000,5000,10000\}N\{\\in\}\\\{2000,5000,10000\\\}, HoReN reaches OP0\.860\.86–0\.980\.98on every cell while GRACE is stuck at OP0\.180\.18–0\.310\.31\(Generalization0\.010\.01–0\.030\.03\) and WISE ranges from OP≤0\.13\\leq 0\.13on GPT\-OSS\-20B to0\.680\.68–0\.720\.72on SEA\-LION\-32B\. HoReN’s three metrics on every model and scale combination satisfy Rel\.≥0\.99\\geq 0\.99, Gen\.≥0\.66\\geq 0\.66, Loc\.≥0\.97\\geq 0\.97\. The lower Generalization floor on GPT\-OSS\-20B \(0\.660\.66–0\.690\.69\) reflects a wider angular gap between original and paraphrase representations at the routing layer, attributable to that model’s distinct architectural lineage rather than to a weakness of the retrieval mechanism itself\.

The four backbones are decoder\-only models from independent training pipelines with different tokenizers, pretraining corpora, and post\-training procedures\. The fact that GRACE’s flat\-Generalization failure reproduces identically on all four \(a failure span of0\.020\.02across pipelines\) confirms that the bottleneck is in the key\-query comparison rather than in any model\-specific representation property\. WISE’s behavior varies across families with no consistent relationship to model size: its OP collapses to near\-zero on GPT\-OSS\-20B yet decays only gradually from0\.720\.72to0\.680\.68on the larger SEA\-LION\-32B, localizing its weakness to model\-specific activation structure rather than to scale alone\. HoReN’s recipe is held fixed across backbones atβ=20\\beta\{=\}20,γ=0\.01\\gamma\{=\}0\.01,M=1M\{=\}1; only the matching threshold and the token pooling ratio change, and these are listed once per family in Table[3](https://arxiv.org/html/2605.08143#A4.T3)\. That a single recipe transfers across seven backbones from1\.51\.5B to3232B parameters is itself a fact about the directional structure of post\-nonlinearity activations, revisited in Section[3\.4\.1](https://arxiv.org/html/2605.08143#S3.SS4.SSS1)\.

Takeaway\.The same fixed retrieval recipe transfers across four model families and seven backbones; GRACE’s generalization failure reproduces identically across all four, which localizes it to the key\-query comparison geometry rather than to any model\-specific artefact\.

### 3\.4Analysis and Ablation Studies

#### 3\.4\.1Why HoReN Works: Direction Is the Semantic Key, and One Hopfield Step Is a Direction\-Selective Denoiser

![Refer to caption](https://arxiv.org/html/2605.08143v1/x4.png)Figure 4:Representation gap diagnosis atN=1000N\{=\}1000\(ZsRE, LLaMA\-3\.1\-8B\)\. Generalization improves in two steps: normalization removes the magnitude component \(\+0\.19\+0\.19\); the Hopfield step closes the angular gap \(\+0\.69\+0\.69\)\. Rel\. and Loc\. remain flat, confirming orthogonality\.Fact \(Figure[4](https://arxiv.org/html/2605.08143#S3.F4), Table[10](https://arxiv.org/html/2605.08143#A7.T10), and Figure[10](https://arxiv.org/html/2605.08143#A7.F10)\)\.Holding the codebook location, value adaptor, and matching threshold fixed and varying only the key\-query comparison \(Table[10](https://arxiv.org/html/2605.08143#A7.T10)\) on LLaMA\-3\.1\-8B, ZsRE,N=1000N\{=\}1000: raw activation distance gives Generalization0\.050\.05; unit\-norm cosine matching lifts it to0\.240\.24\(\+0\.19\+0\.19\); adding one damped Hopfield step lifts it to0\.930\.93\(\+0\.69\+0\.69\)\. Reliability and Locality are flat across the three rows \(Loc\.≥0\.99\\geq 0\.99throughout\)\. Holding everything in HoReN fixed and varying only the number of Hopfield steps \(Figure[10](https://arxiv.org/html/2605.08143#A7.F10)\): Generalization is essentially saturated atM=1M\{=\}1\(theM=1M\{=\}1andM=2M\{=\}2values agree to within0\.030\.03\), but Locality decays as1\.00→0\.95–0\.96→0\.55–0\.65→<0\.101\.00\\to 0\.95\\text\{\-\-\}0\.96\\to 0\.55\\text\{\-\-\}0\.65\\to\{<\}0\.10atM=1,2,4,8M\{=\}1,2,4,8respectively\.

Direction, not magnitude, is the semantic signal\.These two facts identify the load\-bearing piece of HoReN’s design\. The first jump in Table[10](https://arxiv.org/html/2605.08143#A7.T10)\(0\.05→0\.240\.05\\to 0\.24\) removes only magnitude from the comparison; nothing about what is stored, where it is stored, or how the value is applied changes\. That stripping magnitude alone multiplies Generalization roughly fivefold without measurable cost to Reliability or Locality means that the magnitude of the post\-nonlinearity activation is not carrying useful semantic discriminability for retrieval\. It behaves like a per\-prompt gain driven by prompt length, surface form, and upstream normalization, while the direction on the unit hypersphere carries the identity of the fact\. Paraphrases of the same fact preserve direction and modulate magnitude; raw nearest\-neighbor matching weights both, conflates fact identity with prompt\-specific gain, and routes paraphrases outside the acceptance radius\. This is the geometric explanation for GRACE’s\[0\.01,0\.06\]\[0\.01,0\.06\]Generalization on every row of Tables[1](https://arxiv.org/html/2605.08143#S3.T1)and[2](https://arxiv.org/html/2605.08143#S3.T2): the failure is geometric, not informational\.

One damped Hopfield step closes the residual angular gap without touching locality\.The second jump \(0\.24→0\.930\.24\\to 0\.93\) is the residual angular gap\. Even with magnitude removed, paraphrase queries sit at non\-trivial angle from their target keys on the sphere; whenever that angle exceeds the matching threshold, one\-shot argmax misroutes\. A single damped softmax\-attention update over the unit\-norm codebook closes the gap by nudging the query toward its nearest stored key\. The Locality column of Figure[10](https://arxiv.org/html/2605.08143#A7.F10)explains why this update can be kept on for paraphrases without dragging unrelated queries with it: at the chosen sharpness, an unrelated query is approximately equidistant from all stored keys, the softmax distribution is near\-uniform, the weighted codebook average has small projection on any single direction, and the damped update barely moves the query; a paraphrase query is close to one key, the softmax is sharply peaked, and the same damped update nudges the query into the basin\. Theorem[2\.1](https://arxiv.org/html/2605.08143#S2.Thmtheorem1)predicts that iterating these dynamics to convergence must contract*every*query to the convex hull of stored keys for largeβ\\beta, and Proposition[2\.2](https://arxiv.org/html/2605.08143#S2.Thmtheorem2)shows that this contraction is already active at finite truncation in an averaged sense, which is exactly the Locality cliff visible atM≥4M\\geq 4in Figure[10](https://arxiv.org/html/2605.08143#A7.F10): the choice ofM=1M\{=\}1is not a tuning convenience but a direct consequence of the theory—one step is enough for paraphrases and too small for unrelated queries\.

Matching thresholds cluster by architecture family, confirming directional stability transfers across backbones\.A third independent check that direction is the right object is the distribution of per\-model parameters in Table[3](https://arxiv.org/html/2605.08143#A4.T3)\. The matching threshold falls into two tight, family\-coherent bands \(LLaMA\-3 / LLaMA\-3\.1 at0\.800\.80–0\.850\.85, Qwen\-2\.5 / SEA\-LION at0\.550\.55–0\.600\.60\) that survive heavy reasoning post\-training: the DeepSeek\-R1 distillations of LLaMA and Qwen each remain inside their parent family’s band\. The pooling ratio is60%60\\%for every model except LLaMA\-3\-8B\-Instruct \(50%50\\%\)\. The threshold therefore tracks a property of the base model’s directional activation structure rather than of the task or the instruction\-tuning recipe, which is why the same recipe withβ=20\\beta\{=\}20,γ=0\.01\\gamma\{=\}0\.01,M=1M\{=\}1transfers across seven backbones from1\.51\.5B to3232B parameters\.

Takeaway\.The two\-step ladder in Table[10](https://arxiv.org/html/2605.08143#A7.T10)attributes0\.190\.19of the Generalization gain to removing magnitude from the comparison and0\.690\.69to a single damped Hopfield refinement; the Locality column of Figure[10](https://arxiv.org/html/2605.08143#A7.F10)confirms the refinement is direction\-selective, active for paraphrases and inert for unrelated queries; the cross\-model parameter bands in Table[3](https://arxiv.org/html/2605.08143#A4.T3)confirm direction is a stable property of the base model\. Together these reads explain Tables[1](https://arxiv.org/html/2605.08143#S3.T1)and[2](https://arxiv.org/html/2605.08143#S3.T2): the mechanistic half of HoReN \(codebook plus value adaptor\) gives Reliability≥0\.99\\geq 0\.99on every row, while the routing layer handles the statistical half by reading only direction\.

#### 3\.4\.2Ablation Studies: Design Choices

Fact \(Figure[11](https://arxiv.org/html/2605.08143#A7.F11), Table[12](https://arxiv.org/html/2605.08143#A7.T12)\)\.On LLaMA\-3\.1\-8B, ZsRE, the last60%60\\%suffix of prompt tokens is Pareto\-optimal at every scale \(OP≥0\.95\\text\{OP\}\\geq 0\.95\); shorter suffixes hurt Generalization and longer ones collapse Locality\. The direct\-vector and LoRA \(r=4r\{=\}4\) value adaptors reach identical OP within0\.010\.01at every scale, with parameter overhead\+4\.09\+4\.09M vs\.\+32\.64\+32\.64M atN=1000N\{=\}1000\(8×8\\timessmaller for the direct variant\)\.

That the60%60\\%ratio is jointly optimal across scales says something specific about the routing key\. Too short a suffix discards tokens that participate in the post\-nonlinearity activation pattern of the fact, hurting Generalization; too long a suffix mixes in early\-context tokens that vary across paraphrases and overlap with locality queries, hurting Locality\. The fact that the ratio is flat acrossNNmatches the cross\-model flatness in Table[3](https://arxiv.org/html/2605.08143#A4.T3): the pooling decision is governed by the geometry of the routing layer, not by edit\-stream size\.

That direct\-vector and LoRA value adaptors reach identical OP matters as a confirmation of the decoupling claim from Section[3\.4\.1](https://arxiv.org/html/2605.08143#S3.SS4.SSS1)\. If the routing layer correctly selects the right entry, either adaptor has enough expressivity to write the target answer at these scales; the residual signal in OP is dominated by routing quality rather than value\-payload capacity\. The direct variant is therefore the default for its8×8\\timessmaller parameter cost\.

## 4Conclusion

We studied lifelong model editing through the lens of memory\-based editors and identified paraphrase routing under accumulated edits as the bottleneck of this paradigm: existing methods can store edits faithfully but cannot reliably activate the right one when the codebook grows\. We proposedHoReN, a routing\-layer mechanism that combines L2\-normalized angular matching with a single damped Hopfield\-style query refinement, motivated by an attractor\-basin reading in which the deferral threshold serves as the basin boundary\. Theoretically, we showed that standard Hopfield retrieval admits convergent attractor dynamics—monotone energy descent with convergence to fixed points—which would attract all queries, including unrelated ones, toward stored codes if iterated to convergence, justifying HoReN’s single\-step deployment\. Empirically, HoReN sustains all three editing metrics above0\.890\.89on a controlled5050K\-edit ZsRE stress test, reaches OP0\.940\.94–0\.980\.98at1,0001\{,\}000edits across LLaMA and Qwen backbones, generalizes to structured WikiBigEdit and unstructured UnKE, and matches GRACE on edit/inference latency and parameter overhead\. Limitations include the linear growth of codebook memory with the number of edits and the gap between the multi\-step convergence theory and the deployed single\-step variant\. Extending HoReN to temporal conflicts among edits, multi\-hop reasoning across edits, and adaptive matching thresholds are natural future directions\.

## References

- \[1\]O\. S\. Agarwal, L\. Ahmad, J\. Ai,et al\.\(2025\)Gpt\-oss\-120b&gpt\-oss\-20b model card\.External Links:[Link](https://api.semanticscholar.org/CorpusID:280671456)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[2\]DeepSeek\-AI, D\. Guo, D\. Yang,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645,pp\. 633 – 638\.External Links:[Link](https://api.semanticscholar.org/CorpusID:275789950)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[3\]J\. Deng, Z\. Wei, L\. Pang, H\. Ding, H\. Shen, and X\. Cheng\(2024\)UnKE: unstructured knowledge editing in large language models\.ArXivabs/2405\.15349\.External Links:[Link](https://api.semanticscholar.org/CorpusID:282065712)Cited by:[3rd item](https://arxiv.org/html/2605.08143#A4.I1.i3.p1.1),[4th item](https://arxiv.org/html/2605.08143#A4.I2.i4.p1.1),[§D\.3\.2](https://arxiv.org/html/2605.08143#A4.SS3.SSS2.p1.3),[§F\.1](https://arxiv.org/html/2605.08143#A6.SS1.p2.5.1),[Table 8](https://arxiv.org/html/2605.08143#A6.T8),[Table 8](https://arxiv.org/html/2605.08143#A6.T8.4.2),[§1](https://arxiv.org/html/2605.08143#S1.p6.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[4\]A\. Dubey, A\. Jauhri, A\. Pandey,et al\.\(2024\)The llama 3 herd of models\.External Links:[Link](https://api.semanticscholar.org/CorpusID:271571434)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[5\]J\. Fang, H\. Jiang, K\. Wang, Y\. Ma, X\. Wang, X\. He, and T\. Chua\(2024\)AlphaEdit: null\-space constrained knowledge editing for language models\.ArXivabs/2410\.02355\.External Links:[Link](https://api.semanticscholar.org/CorpusID:273098148)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2605.08143#A4.I2.i2.p1.1),[§D\.4\.1](https://arxiv.org/html/2605.08143#A4.SS4.SSS1.Px2.p1.17),[§1](https://arxiv.org/html/2605.08143#S1.p2.1),[§1](https://arxiv.org/html/2605.08143#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[6\]X\. Gu, Z\. Huang, J\. Gu, and K\. Zhang\(2025\)UltraEdit: training\-, subject\-, and memory\-free lifelong editing in language models\.External Links:[Link](https://api.semanticscholar.org/CorpusID:281659488)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2605.08143#A4.I2.i3.p1.1),[§D\.4\.1](https://arxiv.org/html/2605.08143#A4.SS4.SSS1.Px5.p1.2),[§1](https://arxiv.org/html/2605.08143#S1.p2.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[7\]T\. Hartvigsen, S\. Sankaranarayanan, H\. Palangi, Y\. Kim, and M\. Ghassemi\(2023\)Aging with grace: lifelong model editing with discrete key\-value adaptors\.Advances in Neural Information Processing Systems36,pp\. 47934–47959\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2605.08143#A4.I1.i1.p1.2),[1st item](https://arxiv.org/html/2605.08143#A4.I3.i1.p1.1),[§D\.3\.1](https://arxiv.org/html/2605.08143#A4.SS3.SSS1.p1.2),[§D\.4\.1](https://arxiv.org/html/2605.08143#A4.SS4.SSS1.Px3.p1.4),[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p2.1),[§1](https://arxiv.org/html/2605.08143#S1.p5.1),[§2](https://arxiv.org/html/2605.08143#S2.p1.3),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[8\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for nlp\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:59599816)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[9\]J\. E\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, and W\. Chen\(2021\)LoRA: low\-rank adaptation of large language models\.ArXivabs/2106\.09685\.External Links:[Link](https://api.semanticscholar.org/CorpusID:235458009)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[10\]U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis\(2020\)Generalization through memorization: nearest neighbor language models\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[11\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p3.1)\.
- \[12\]D\. Krotov\(2023\)A new frontier for hopfield networks\.Nature Reviews Physics5,pp\. 366–367\.External Links:[Link](https://api.semanticscholar.org/CorpusID:258812300)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p5.1)\.
- \[13\]O\. Levy, M\. Seo, E\. Choi, and L\. Zettlemoyer\(2017\)Zero\-shot relation extraction via reading comprehension\.arXiv preprint arXiv:1706\.04115\.Cited by:[1st item](https://arxiv.org/html/2605.08143#A4.I1.i1.p1.2),[§D\.4\.4](https://arxiv.org/html/2605.08143#A4.SS4.SSS4.p1.2),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[14\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[15\]X\. L\. Li and P\. Liang\(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4582–4597\.External Links:[Link](https://api.semanticscholar.org/CorpusID:230433941)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[16\]H\. Luo, X\. Ran, T\. J\. Karkkainen, Z\. Chen, J\. Shen, Q\. Xu, and F\. Cong\(2026\)Reversible lifelong model editing via semantic routing\-based lora\.External Links:[Link](https://api.semanticscholar.org/CorpusID:286489211)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1)\.
- \[17\]H\. Luo, X\. Ran, Z\. Li, H\. Xue, T\. Jiang, J\. Shen, T\. J\. Karkkainen, Q\. Xu, and F\. Cong\(2026\)Key\-value pair\-free continual learner via task\-specific prompt\-prototype\.Neural networks : the official journal of the International Neural Network Society198,pp\. 108576\.External Links:[Link](https://api.semanticscholar.org/CorpusID:284544134)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1)\.
- \[18\]K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov\(2022\)Locating and editing factual associations in gpt\.Advances in neural information processing systems35,pp\. 17359–17372\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2605.08143#A4.I2.i1.p1.1),[§D\.4\.1](https://arxiv.org/html/2605.08143#A4.SS4.SSS1.Px1.p1.9),[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p2.1),[§1](https://arxiv.org/html/2605.08143#S1.p5.1),[§2](https://arxiv.org/html/2605.08143#S2.p1.3),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[19\]K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau\(2022\)Mass\-editing memory in a transformer\.arXiv preprint arXiv:2210\.07229\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p2.1)\.
- \[20\]E\. Mitchell, C\. Lin, A\. Bosselut, C\. Finn, and C\. D\. Manning\(2021\)Fast model editing at scale\.arXiv preprint arXiv:2110\.11309\.Cited by:[1st item](https://arxiv.org/html/2605.08143#A4.I1.i1.p1.2),[§D\.3\.1](https://arxiv.org/html/2605.08143#A4.SS3.SSS1.p1.2),[§D\.3](https://arxiv.org/html/2605.08143#A4.SS3.p1.11),[§D\.4\.4](https://arxiv.org/html/2605.08143#A4.SS4.SSS4.p1.2)\.
- \[21\]E\. Mitchell, C\. Lin, A\. Bosselut, C\. D\. Manning, and C\. Finn\(2022\)Memory\-based model editing at scale\.InInternational Conference on Machine Learning,pp\. 15817–15831\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.
- \[22\]R\. Ng, T\. N\. Nguyen, Y\. Huang, N\. Chia, W\. Leong,et al\.\(2025\)SEA\-lion: southeast asian languages in one network\.InIJCNLP\-AACL,External Links:[Link](https://api.semanticscholar.org/CorpusID:277450612)Cited by:[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[23\]H\. Ramsauer, B\. Schafl, J\. Lehner, P\. Seidl, M\. Widrich, L\. Gruber, M\. Holzleitner, M\. Pavlovi’c, G\. K\. F\. Sandve, V\. Greiff, D\. P\. Kreil, M\. Kopp, G\. Klambauer, J\. Brandstetter, and S\. Hochreiter\(2020\)Hopfield networks is all you need\.ArXivabs/2008\.02217\.External Links:[Link](https://api.semanticscholar.org/CorpusID:220968978)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p5.1),[§2\.3](https://arxiv.org/html/2605.08143#S2.SS3.p2.12),[§2](https://arxiv.org/html/2605.08143#S2.p1.3)\.
- \[24\]X\. Ran, J\. Yao, Y\. Wang, M\. Xu, and D\. Liu\(2024\)Brain\-inspired continual pre\-trained learner via silent synaptic consolidation\.ArXivabs/2410\.05899\.External Links:[Link](https://api.semanticscholar.org/CorpusID:273228983)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1)\.
- \[25\]L\. Thede, K\. Roth, M\. Bethge, and Z\. Akata\(2025\)Understanding the limits of lifelong knowledge editing in LLMs\.InInternational Conference on Machine Learning,Cited by:[2nd item](https://arxiv.org/html/2605.08143#A4.I1.i2.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p6.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[26\]P\. Wang, Z\. Li, N\. Zhang, Z\. Xu, Y\. Yao, Y\. Jiang, P\. Xie, F\. Huang, and H\. Chen\(2024\)WISE: rethinking the knowledge memory for lifelong model editing of large language models\.ArXivabs/2405\.14768\.External Links:[Link](https://api.semanticscholar.org/CorpusID:269982715)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2605.08143#A4.I3.i2.p1.1),[§D\.3\.1](https://arxiv.org/html/2605.08143#A4.SS3.SSS1.p1.2),[§D\.4\.1](https://arxiv.org/html/2605.08143#A4.SS4.SSS1.Px4.p1.9),[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.08143#S2.SS1.p1.8),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[27\]Y\. Wang, M\. Wang, H\. Song, W\. Huang, C\. Wang, Y\. Xie, and X\. Ran\(2025\)REPAIR: robust editing via progressive adaptive intervention and reintegration\.arXiv preprint arXiv:2510\.01879\.Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px2.p1.1)\.
- \[28\]H\. Xue, X\. Ran, Y\. Li, Q\. Xu, E\. Li, Y\. Xu, and Q\. Zhang\(2025\)Distillation\-guided structural transfer for continual learning beyond sparse distributed memory\.InAAAI Conference on Artificial Intelligence,External Links:[Link](https://api.semanticscholar.org/CorpusID:283920124)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1)\.
- \[29\]Q\. A\. Yang, B\. Yang, B\. Zhang,et al\.\(2024\)Qwen2\.5 technical report\.ArXivabs/2412\.15115\.External Links:[Link](https://api.semanticscholar.org/CorpusID:274859421)Cited by:[§1](https://arxiv.org/html/2605.08143#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08143#S3.SS1.SSS0.Px1.p1.1)\.
- \[30\]N\. Zhang, Y\. Yao, B\. Tian, P\. Wang, S\. Deng, M\. Wang, Z\. Xi, S\. Mao, J\. Zhang, Y\. Ni, S\. Cheng, Z\. Xu, X\. Xu, J\. Gu, Y\. Jiang, P\. Xie, F\. Huang, L\. Liang, Z\. Zhang, X\. Zhu, J\. Zhou, and H\. Chen\(2024\)A comprehensive study of knowledge editing for large language models\.ArXivabs/2401\.01286\.External Links:[Link](https://api.semanticscholar.org/CorpusID:266725300)Cited by:[Appendix A](https://arxiv.org/html/2605.08143#A1.SS0.SSS0.Px2.p1.1)\.

## Appendix ARelated Work

##### Parameter\-modifying editors\.

Locate\-and\-edit methods such as ROME\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]and MEMIT\[[19](https://arxiv.org/html/2605.08143#bib.bib2)\]modify internal weights via causal tracing or rank\-constrained updates but suffer interference under sequential application—a variant of the catastrophic forgetting problem long studied in continual learning\[[11](https://arxiv.org/html/2605.08143#bib.bib22)\]\. AlphaEdit\[[5](https://arxiv.org/html/2605.08143#bib.bib25)\]mitigates this with null\-space projection, and UltraEdit\[[6](https://arxiv.org/html/2605.08143#bib.bib26)\]studies training\-free unstructured editing\. These methods concern how edits are*written*into the base weights; HoReN concerns how accumulated edits are*routed*at inference time\.

##### Parameter\-preserving / memory\-based editors\.

Memory\-based editors store updates externally and activate them on matching queries, with each method innovating on a different axis: DEFER\[[21](https://arxiv.org/html/2605.08143#bib.bib17)\]learns a refinement network for retrieval, GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]maintains a discrete codebook keyed on hidden states with a fixedε\\varepsilon\-radius match, WISE\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\]adds activation\-based importance weighting, and REPAIR\[[27](https://arxiv.org/html/2605.08143#bib.bib16)\]enforces cross\-layer consistency\. Value payloads are typically lightweight modules—LoRA\[[9](https://arxiv.org/html/2605.08143#bib.bib3)\], adapters\[[8](https://arxiv.org/html/2605.08143#bib.bib8)\], or prefix tuning\[[15](https://arxiv.org/html/2605.08143#bib.bib9)\]—recently surveyed byZhanget al\.\[[30](https://arxiv.org/html/2605.08143#bib.bib10)\]\. Conceptually this family is close to retrieval\-augmented language modeling\[[10](https://arxiv.org/html/2605.08143#bib.bib28),[14](https://arxiv.org/html/2605.08143#bib.bib29)\], but with keys and values constructed per edit\. HoReN stays in this family and differs from GRACE along two specific axes: \(i\) it L2\-normalizes keys and queries so matching depends on angular similarity rather than magnitude, and \(ii\) it applies a single damped Hopfield\-style refinement step on the query before the final argmax; value adaptors are kept modular \(direct vector or LoRA\)\.

##### Associative retrieval with Hopfield networks\.

Modern Hopfield networks\[[23](https://arxiv.org/html/2605.08143#bib.bib13)\]extend classical associative memory with continuous energy functions and high storage capacity, retrieving patterns through attractor dynamics that pull noisy queries toward stored memories, with connections to attention and few\-shot learning\[[12](https://arxiv.org/html/2605.08143#bib.bib14)\]\. To our knowledge, this perspective has not been brought to large\-scale sequential model editing; HoReN borrows the energy/attractor view to motivate a lightweight retrieval refinement rather than to deploy full Hopfield dynamics\.

## Appendix BMethod Details

### B\.1Algorithm Pseudocode

Input:Codebook

𝒞=\{\(ki,vi,yi\)\}i=1N\\mathcal\{C\}=\\\{\(k\_\{i\},v\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, possibly empty\.

Input:Edit sample

\(𝐱t,𝐲t∗\)\(\\mathbf\{x\}\_\{t\},\\mathbf\{y\}\_\{t\}^\{\*\}\)
Input:Hopfield params

\(γ,β,ϵ\)\(\\gamma,\\beta,\\epsilon\), iterations

MM, match threshold

cc, adaptor steps

UU
Output:Updated codebook

𝒞\\mathcal\{C\}
\(Hin,Hout\)←fθ​\(𝐱t\)\(H\_\{\\mathrm\{in\}\},\\,H\_\{\\mathrm\{out\}\}\)\\leftarrow f\_\{\\theta\}\(\\mathbf\{x\}\_\{t\}\);

//

HinH\_\{\\mathrm\{in\}\}at layerll,HoutH\_\{\\mathrm\{out\}\}at layerl\+1l\{\+\}1

q0←normalize⁡\(select⁡\(Hin\)\)q\_\{0\}\\leftarrow\\operatorname\{normalize\}\(\\operatorname\{select\}\(H\_\{\\mathrm\{in\}\}\)\);

q←q0q\\leftarrow q\_\{0\};

if*𝒞=∅\\mathcal\{C\}=\\emptyset*then

vnew←InitValueAdaptor​\(\)v\_\{\\text\{new\}\}\\leftarrow\\textsc\{InitValueAdaptor\}\(\);

for*u←1u\\leftarrow 1toUU*do

Apply

vnewv\_\{\\text\{new\}\}to

HoutH\_\{\\mathrm\{out\}\}and compute

ℒCE​\(fθ​\(𝐱t\),𝐲t∗\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(f\_\{\\theta\}\(\\mathbf\{x\}\_\{t\}\),\\mathbf\{y\}\_\{t\}^\{\*\}\);

Update only

vnewv\_\{\\text\{new\}\}by gradient descent;

if*early stopping criterion is met*then

break;

𝒞←𝒞∪\{\(q0,vnew,𝐲t∗\)\}\\mathcal\{C\}\\leftarrow\\mathcal\{C\}\\cup\\\{\(q\_\{0\},v\_\{\\text\{new\}\},\\mathbf\{y\}\_\{t\}^\{\*\}\)\\\};

return

𝒞\\mathcal\{C\};

K←stack⁡\(\{ki:\(ki,vi,yi\)∈𝒞\}\)K\\leftarrow\\operatorname\{stack\}\(\\\{k\_\{i\}:\(k\_\{i\},v\_\{i\},y\_\{i\}\)\\in\\mathcal\{C\}\\\}\);

for*m←1m\\leftarrow 1toMM*do

p←softmax⁡\(β​q​K⊤\)p\\leftarrow\\operatorname\{softmax\}\(\\beta\\,qK^\{\\top\}\);

qnew←normalize⁡\(p​K\)q\_\{\\text\{new\}\}\\leftarrow\\operatorname\{normalize\}\(pK\);

if*‖q*new*−q‖2≤ϵ\|\|q\_\{\\text\{new\}\}\-q\|\|\_\{2\}\\leq\\epsilon*then

//Early Stop

break;

q←normalize⁡\(\(1−γ\)​q\+γ​qnew\)q\\leftarrow\\operatorname\{normalize\}\\\!\\big\(\(1\-\\gamma\)q\+\\gamma q\_\{\\text\{new\}\}\);

𝐚←q​K⊤\\mathbf\{a\}\\leftarrow qK^\{\\top\};

i⋆←arg⁡maxi⁡𝐚ii^\{\\star\}\\leftarrow\\arg\\max\_\{i\}\\mathbf\{a\}\_\{i\};

if*𝐚i⋆≤c\\mathbf\{a\}\_\{i^\{\\star\}\}\\leq coryi⋆≠𝐲t∗y\_\{i^\{\\star\}\}\\neq\\mathbf\{y\}\_\{t\}^\{\*\}*then

//No match or label conflict: add a new entry

knew←q0k\_\{\\text\{new\}\}\\leftarrow q\_\{0\};

vnew←InitValueAdaptor​\(\)v\_\{\\text\{new\}\}\\leftarrow\\textsc\{InitValueAdaptor\}\(\);

for*u←1u\\leftarrow 1toUU*do

Apply

vnewv\_\{\\text\{new\}\}to

HoutH\_\{\\mathrm\{out\}\}and compute

ℒCE​\(fθ​\(𝐱t\),𝐲t∗\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(f\_\{\\theta\}\(\\mathbf\{x\}\_\{t\}\),\\mathbf\{y\}\_\{t\}^\{\*\}\);

Update only

vnewv\_\{\\text\{new\}\}by gradient descent;

if*early stopping criterion is met*then

break;

𝒞←𝒞∪\{\(knew,vnew,𝐲t∗\)\}\\mathcal\{C\}\\leftarrow\\mathcal\{C\}\\cup\\\{\(k\_\{\\text\{new\}\},v\_\{\\text\{new\}\},\\mathbf\{y\}\_\{t\}^\{\*\}\)\\\};

else

//Match with consistent label: use existing entry

Apply value adaptor

vi⋆v\_\{i^\{\\star\}\}to

HoutH\_\{\\mathrm\{out\}\};

return

𝒞\\mathcal\{C\}

Algorithm 1Codebook Update at Layerll

## Appendix CProofs of Theoretical Results

### C\.1Setup and Descent Inequality

The two results below share a common Lyapunov construction\. We isolate the per\-step descent inequality first; the asymptotic theorem and finite\-step proposition then follow as two complementary readings of it\.

###### Lemma C\.1\(Per\-step descent of standard Hopfield retrieval\)\.

LetK∈ℝC×dK\\in\\mathbb\{R\}^\{C\\times d\}be a codebook with rows satisfying‖ki‖2=1\\\|k\_\{i\}\\\|\_\{2\}=1, letq\(0\)∈ℝ1×dq^\{\(0\)\}\\in\\mathbb\{R\}^\{1\\times d\}satisfy‖q\(0\)‖2=1\\\|q^\{\(0\)\}\\\|\_\{2\}=1, and letβ\>0\\beta\>0\. Define

T​\(q\):=softmax⁡\(β​q​K⊤\)​K,F​\(q\):=1β​log​∑i=1Cexp⁡\(β​q​ki⊤\),E​\(q,K\):=12​‖q‖22−F​\(q\)\.T\(q\):=\\operatorname\{softmax\}\(\\beta qK^\{\\top\}\)\\,K,\\quad F\(q\):=\\tfrac\{1\}\{\\beta\}\\log\\\!\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\),\\quad E\(q,K\):=\\tfrac\{1\}\{2\}\\\|q\\\|\_\{2\}^\{2\}\-F\(q\)\.Then for everys≥0s\\geq 0the iterateq\(s\+1\)=T​\(q\(s\)\)q^\{\(s\+1\)\}=T\(q^\{\(s\)\}\)satisfies

E​\(q\(s\+1\),K\)−E​\(q\(s\),K\)≤−12​‖q\(s\+1\)−q\(s\)‖22\.E\(q^\{\(s\+1\)\},K\)\-E\(q^\{\(s\)\},K\)\\;\\leq\\;\-\\tfrac\{1\}\{2\}\\,\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\}\.

###### Proof\.

We construct a Lyapunov function\. Forq∈ℝ1×dq\\in\\mathbb\{R\}^\{1\\times d\}, the potentialFFis the log\-sum\-exp of the similarities,

F​\(q\):=1β​log⁡\(∑i=1Cexp⁡\(β​q​ki⊤\)\)=1β​lseβ​\(q​K⊤\),F\(q\):=\\frac\{1\}\{\\beta\}\\log\\left\(\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\)\\right\)=\\frac\{1\}\{\\beta\}\\text\{lse\}\_\{\\beta\}\(qK^\{\\top\}\),\(4\)whereq​ki⊤qk\_\{i\}^\{\\top\}denotes the dot product between the query and theii\-th key\. The functionF​\(q\)F\(q\)is convex inqq\. Its gradient with respect toqqis:

∇qF​\(q\)=∑i=1Cki​exp⁡\(β​q​ki⊤\)∑i=1Cexp⁡\(β​q​ki⊤\)=softmax​\(β​q​K⊤\)​K=p​K,\\nabla\_\{q\}F\(q\)=\\frac\{\\sum\_\{i=1\}^\{C\}k\_\{i\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\)\}\{\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\)\}=\\text\{softmax\}\(\\beta\\,qK^\{\\top\}\)K=pK,\(5\)wherep=softmax​\(β​q​K⊤\)∈ℝ1×Cp=\\text\{softmax\}\(\\beta\\,qK^\{\\top\}\)\\in\\mathbb\{R\}^\{1\\times C\}\. Consequently, the standard Hopfield update corresponds exactly to the gradient mapping:

q\(s\+1\)=∇qF​\(q\(s\)\)\.q^\{\(s\+1\)\}=\\nabla\_\{q\}F\(q^\{\(s\)\}\)\.\(6\)
Define the energy function asE​\(q,K\):=12​‖q‖22−F​\(q\)E\(q,K\):=\\frac\{1\}\{2\}\\\|q\\\|\_\{2\}^\{2\}\-F\(q\)\. SinceFFis convex, it satisfies the first\-order inequality:

F​\(y\)≥F​\(x\)\+⟨∇qF​\(x\),\(y−x\)⊤⟩F\(y\)\\geq F\(x\)\+\\langle\\nabla\_\{q\}F\(x\),\(y\-x\)^\{\\top\}\\rangle\(7\)for allx,y∈ℝ1×dx,y\\in\\mathbb\{R\}^\{1\\times d\}\. Lettingx=q\(s\)x=q^\{\(s\)\}andy=q\(s\+1\)y=q^\{\(s\+1\)\}, and using \([6](https://arxiv.org/html/2605.08143#A3.E6)\), we obtain:

F​\(q\(s\+1\)\)−F​\(q\(s\)\)≥⟨q\(s\+1\),\(q\(s\+1\)−q\(s\)\)⊤⟩=q\(s\+1\)​\(q\(s\+1\)−q\(s\)\)⊤\.F\(q^\{\(s\+1\)\}\)\-F\(q^\{\(s\)\}\)\\geq\\langle q^\{\(s\+1\)\},\(q^\{\(s\+1\)\}\-q^\{\(s\)\}\)^\{\\top\}\\rangle=q^\{\(s\+1\)\}\(q^\{\(s\+1\)\}\-q^\{\(s\)\}\)^\{\\top\}\.\(8\)
We now evaluate the change in energy between consecutive iterations:

E​\(q\(s\+1\),K\)−E​\(q\(s\),K\)\\displaystyle E\(q^\{\(s\+1\)\},K\)\-E\(q^\{\(s\)\},K\)=12​‖q\(s\+1\)‖22−12​‖q\(s\)‖22−\(F​\(q\(s\+1\)\)−F​\(q\(s\)\)\)\\displaystyle=\\frac\{1\}\{2\}\\\|q^\{\(s\+1\)\}\\\|\_\{2\}^\{2\}\-\\frac\{1\}\{2\}\\\|q^\{\(s\)\}\\\|\_\{2\}^\{2\}\-\\Big\(F\(q^\{\(s\+1\)\}\)\-F\(q^\{\(s\)\}\)\\Big\)\(9\)≤12​‖q\(s\+1\)‖22−12​‖q\(s\)‖22−q\(s\+1\)​\(q\(s\+1\)−q\(s\)\)⊤\\displaystyle\\leq\\frac\{1\}\{2\}\\\|q^\{\(s\+1\)\}\\\|\_\{2\}^\{2\}\-\\frac\{1\}\{2\}\\\|q^\{\(s\)\}\\\|\_\{2\}^\{2\}\-q^\{\(s\+1\)\}\(q^\{\(s\+1\)\}\-q^\{\(s\)\}\)^\{\\top\}=−12​‖q\(s\+1\)−q\(s\)‖22≤0\.\\displaystyle=\-\\frac\{1\}\{2\}\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\}\\leq 0\.
This establishes the descent inequality\.∎∎

The descent inequality has two complementary readings\.*Asymptotically*\(s→∞s\\to\\infty\), it forces‖q\(s\+1\)−q\(s\)‖2→0\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}\\to 0and, combined with real\-analyticity ofEE, gives single\-limit convergence to a fixed point ofTT\.*Finitely*\(anyN≥1N\\geq 1\), it gives a cumulative bound on the squared increments, hence a1/N1/\\sqrt\{N\}residual rate for the best of the firstNNiterates \(Proposition[C\.3](https://arxiv.org/html/2605.08143#A3.Thmtheorem3)\)\. We state and prove the two results separately below\.

### C\.2Proof of Theorem[2\.1](https://arxiv.org/html/2605.08143#S2.Thmtheorem1): Asymptotic Convergence

###### Theorem C\.2\(Asymptotic convergence of standard Hopfield retrieval\)\.

Under the assumptions above, the sequence\{q\(s\)\}s=0∞\\\{q^\{\(s\)\}\\\}\_\{s=0\}^\{\\infty\}generated byq\(s\+1\)=T​\(q\(s\)\)q^\{\(s\+1\)\}=T\(q^\{\(s\)\}\)converges to a fixed pointq∗q^\{\\ast\}, i\.e\.,q\(s\)→q∗q^\{\(s\)\}\\to q^\{\\ast\}ass→∞s\\to\\infty, whereq∗=T​\(q∗\)=softmax⁡\(β​q∗​K⊤\)​Kq^\{\\ast\}=T\(q^\{\\ast\}\)=\\operatorname\{softmax\}\(\\beta q^\{\\ast\}K^\{\\top\}\)\\,K\.

###### Proof\.

*Step 1: Boundedness of iterates and lower\-boundedness ofEE\.*Sinceq\(s\+1\)=softmax⁡\(β​q\(s\)​K⊤\)​Kq^\{\(s\+1\)\}=\\operatorname\{softmax\}\(\\beta q^\{\(s\)\}K^\{\\top\}\)Kis a convex combination of unit\-norm rows ofKK,q\(s\+1\)∈conv⁡\{k1,…,kC\}q^\{\(s\+1\)\}\\in\\operatorname\{conv\}\\\{k\_\{1\},\\ldots,k\_\{C\}\\\}and‖q\(s\+1\)‖2≤1\\\|q^\{\(s\+1\)\}\\\|\_\{2\}\\leq 1\. Together with‖q\(0\)‖2=1\\\|q^\{\(0\)\}\\\|\_\{2\}=1, the full sequence is bounded\. For anyqq,

F​\(q\)=1β​log​∑i=1Cexp⁡\(β​q​ki⊤\)≤maxi⁡q​ki⊤\+log⁡Cβ≤‖q‖2\+log⁡Cβ,F\(q\)=\\tfrac\{1\}\{\\beta\}\\log\\\!\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,qk\_\{i\}^\{\\top\}\)\\leq\\max\_\{i\}qk\_\{i\}^\{\\top\}\+\\frac\{\\log C\}\{\\beta\}\\leq\\\|q\\\|\_\{2\}\+\\frac\{\\log C\}\{\\beta\},soE​\(q,K\)≥12​‖q‖22−‖q‖2−log⁡Cβ≥−12−log⁡CβE\(q,K\)\\geq\\tfrac\{1\}\{2\}\\\|q\\\|\_\{2\}^\{2\}\-\\\|q\\\|\_\{2\}\-\\tfrac\{\\log C\}\{\\beta\}\\geq\-\\tfrac\{1\}\{2\}\-\\tfrac\{\\log C\}\{\\beta\}\. ThusEEis bounded below\.

*Step 2: Square\-summability of increments\.*SetEs:=E​\(q\(s\),K\)E\_\{s\}:=E\(q^\{\(s\)\},K\)andds:=q\(s\+1\)−q\(s\)d\_\{s\}:=q^\{\(s\+1\)\}\-q^\{\(s\)\}\. Lemma[C\.1](https://arxiv.org/html/2605.08143#A3.Thmtheorem1)givesEs−Es\+1≥12​‖ds‖22E\_\{s\}\-E\_\{s\+1\}\\geq\\tfrac\{1\}\{2\}\\\|d\_\{s\}\\\|\_\{2\}^\{2\}\. Since\{Es\}\\\{E\_\{s\}\\\}is monotone and bounded below, it converges to someE∞E\_\{\\infty\}, and

∑s=0∞‖ds‖22≤2​\(E0−E∞\)<∞,\\sum\_\{s=0\}^\{\\infty\}\\\|d\_\{s\}\\\|\_\{2\}^\{2\}\\leq 2\(E\_\{0\}\-E\_\{\\infty\}\)<\\infty,so‖ds‖2→0\\\|d\_\{s\}\\\|\_\{2\}\\to 0\.

*Step 3: Accumulation points are fixed points\.*LetΩ\\Omegabe the set of accumulation points of\{q\(s\)\}\\\{q^\{\(s\)\}\\\}; boundedness givesΩ≠∅\\Omega\\neq\\emptyset\. Forq∗∈Ωq^\{\\ast\}\\in\\Omega, pick a subsequenceq\(sj\)→q∗q^\{\(s\_\{j\}\)\}\\to q^\{\\ast\}\. Since‖dsj‖2→0\\\|d\_\{s\_\{j\}\}\\\|\_\{2\}\\to 0,q\(sj\+1\)→q∗q^\{\(s\_\{j\}\+1\)\}\\to q^\{\\ast\}, andq\(sj\+1\)=T​\(q\(sj\)\)→T​\(q∗\)q^\{\(s\_\{j\}\+1\)\}=T\(q^\{\(s\_\{j\}\)\}\)\\to T\(q^\{\\ast\}\)by continuity ofTT\. Henceq∗=T​\(q∗\)q^\{\\ast\}=T\(q^\{\\ast\}\)\.

*Step 4: Single\-limit convergence via Kurdyka\-Łojasiewicz\.*SinceFFis real analytic, so isEE, andEEsatisfies the Kurdyka\-Łojasiewicz \(KL\) property\. Note that∇E​\(q\)=q−∇F​\(q\)=q−T​\(q\)\\nabla E\(q\)=q\-\\nabla F\(q\)=q\-T\(q\), so∇E​\(q\(s\)\)=q\(s\)−q\(s\+1\)=−ds\\nabla E\(q^\{\(s\)\}\)=q^\{\(s\)\}\-q^\{\(s\+1\)\}=\-d\_\{s\}\. SinceEs→E∞E\_\{s\}\\to E\_\{\\infty\}, everyq∗∈Ωq^\{\\ast\}\\in\\OmegasatisfiesE​\(q∗,K\)=E∞E\(q^\{\\ast\},K\)=E\_\{\\infty\}\. By the uniformized KL property on the compact setΩ\\Omega, there existη\>0\\eta\>0,ε\>0\\varepsilon\>0, and a concave desingularizingφ\\varphiwithφ​\(0\)=0\\varphi\(0\)=0,φ′\>0\\varphi^\{\\prime\}\>0, such that for all sufficiently largesswithEs\>E∞E\_\{s\}\>E\_\{\\infty\},

φ′​\(Es−E∞\)​‖∇E​\(q\(s\)\)‖2≥1\.\\varphi^\{\\prime\}\(E\_\{s\}\-E\_\{\\infty\}\)\\,\\\|\\nabla E\(q^\{\(s\)\}\)\\\|\_\{2\}\\geq 1\.\(IfEs=E∞E\_\{s\}=E\_\{\\infty\}for somess, the descent inequality forcesds=0d\_\{s\}=0and the sequence is constant from that point on\.\) By concavity ofφ\\varphi,

φ​\(Es−E∞\)−φ​\(Es\+1−E∞\)\\displaystyle\\varphi\(E\_\{s\}\-E\_\{\\infty\}\)\-\\varphi\(E\_\{s\+1\}\-E\_\{\\infty\}\)≥φ′​\(Es−E∞\)​\(Es−Es\+1\)\\displaystyle\\geq\\varphi^\{\\prime\}\(E\_\{s\}\-E\_\{\\infty\}\)\(E\_\{s\}\-E\_\{s\+1\}\)≥Es−Es\+1‖∇E​\(q\(s\)\)‖2=Es−Es\+1‖ds‖2≥12​‖ds‖2\.\\displaystyle\\geq\\frac\{E\_\{s\}\-E\_\{s\+1\}\}\{\\\|\\nabla E\(q^\{\(s\)\}\)\\\|\_\{2\}\}=\\frac\{E\_\{s\}\-E\_\{s\+1\}\}\{\\\|d\_\{s\}\\\|\_\{2\}\}\\geq\\tfrac\{1\}\{2\}\\\|d\_\{s\}\\\|\_\{2\}\.Hence‖ds‖2≤2​\(φ​\(Es−E∞\)−φ​\(Es\+1−E∞\)\)\\\|d\_\{s\}\\\|\_\{2\}\\leq 2\\bigl\(\\varphi\(E\_\{s\}\-E\_\{\\infty\}\)\-\\varphi\(E\_\{s\+1\}\-E\_\{\\infty\}\)\\bigr\), and summing gives∑s=0∞‖ds‖2<∞\\sum\_\{s=0\}^\{\\infty\}\\\|d\_\{s\}\\\|\_\{2\}<\\infty\. Therefore\{q\(s\)\}\\\{q^\{\(s\)\}\\\}has finite length, is Cauchy, and converges to someq∗q^\{\\ast\}\. Combined with Step 3,q∗=T​\(q∗\)=softmax⁡\(β​q∗​K⊤\)​Kq^\{\\ast\}=T\(q^\{\\ast\}\)=\\operatorname\{softmax\}\(\\beta q^\{\\ast\}K^\{\\top\}\)\\,K\. ∎∎

### C\.3Proof of Proposition[2\.2](https://arxiv.org/html/2605.08143#S2.Thmtheorem2): Finite\-Step Descent and Residual Bound

###### Proposition C\.3\(Finite\-step descent and residual bound\)\.

Under the assumptions of Theorem[C\.2](https://arxiv.org/html/2605.08143#A3.Thmtheorem2), letN≥1N\\geq 1be a finite integer and generateq\(s\+1\)=T​\(q\(s\)\)q^\{\(s\+1\)\}=T\(q^\{\(s\)\}\),s=0,1,…,N−1s=0,1,\\ldots,N\-1\. Then, for everyss,

E​\(q\(s\+1\),K\)≤E​\(q\(s\),K\)−12​‖q\(s\+1\)−q\(s\)‖22,E\(q^\{\(s\+1\)\},K\)\\leq E\(q^\{\(s\)\},K\)\-\\tfrac\{1\}\{2\}\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\},and consequently

∑s=0N−1‖q\(s\+1\)−q\(s\)‖22≤2​\(E​\(q\(0\),K\)−E​\(q\(N\),K\)\)\.\\sum\_\{s=0\}^\{N\-1\}\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\}\\;\\leq\\;2\\bigl\(E\(q^\{\(0\)\},K\)\-E\(q^\{\(N\)\},K\)\\bigr\)\.Letrs:=‖T​\(q\(s\)\)−q\(s\)‖2=‖q\(s\+1\)−q\(s\)‖2r\_\{s\}:=\\\|T\(q^\{\(s\)\}\)\-q^\{\(s\)\}\\\|\_\{2\}=\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}be the fixed\-point residual at stepss\. For any lower boundEinfE\_\{\\inf\}ofEE,

1N​∑s=0N−1rs2≤2​\(E​\(q\(0\),K\)−Einf\)N,min0≤s<N⁡rs≤2​\(E​\(q\(0\),K\)−Einf\)N\.\\frac\{1\}\{N\}\\sum\_\{s=0\}^\{N\-1\}r\_\{s\}^\{2\}\\;\\leq\\;\\frac\{2\\bigl\(E\(q^\{\(0\)\},K\)\-E\_\{\\inf\}\\bigr\)\}\{N\},\\qquad\\min\_\{0\\leq s<N\}r\_\{s\}\\;\\leq\\;\\sqrt\{\\frac\{2\\bigl\(E\(q^\{\(0\)\},K\)\-E\_\{\\inf\}\\bigr\)\}\{N\}\}\.With the explicit constantsEinf=−12−log⁡CβE\_\{\\inf\}=\-\\tfrac\{1\}\{2\}\-\\frac\{\\log C\}\{\\beta\}and the normalization‖q\(0\)‖2=1\\\|q^\{\(0\)\}\\\|\_\{2\}=1,‖ki‖2=1\\\|k\_\{i\}\\\|\_\{2\}=1, this yields

min0≤s<N⁡‖T​\(q\(s\)\)−q\(s\)‖2≤2N\.\\min\_\{0\\leq s<N\}\\,\\\|T\(q^\{\(s\)\}\)\-q^\{\(s\)\}\\\|\_\{2\}\\;\\leq\\;\\frac\{2\}\{\\sqrt\{N\}\}\.

###### Proof\.

The per\-step descent inequality is exactly Lemma[C\.1](https://arxiv.org/html/2605.08143#A3.Thmtheorem1)\. Summing it froms=0s=0toN−1N\-1gives

E​\(q\(N\),K\)−E​\(q\(0\),K\)≤−12​∑s=0N−1‖q\(s\+1\)−q\(s\)‖22,E\(q^\{\(N\)\},K\)\-E\(q^\{\(0\)\},K\)\\leq\-\\tfrac\{1\}\{2\}\\sum\_\{s=0\}^\{N\-1\}\\\|q^\{\(s\+1\)\}\-q^\{\(s\)\}\\\|\_\{2\}^\{2\},or equivalently∑s=0N−1rs2≤2​\(E​\(q\(0\),K\)−E​\(q\(N\),K\)\)\\sum\_\{s=0\}^\{N\-1\}r\_\{s\}^\{2\}\\leq 2\\bigl\(E\(q^\{\(0\)\},K\)\-E\(q^\{\(N\)\},K\)\\bigr\)\.

IfEinf≤E​\(q,K\)E\_\{\\inf\}\\leq E\(q,K\)for allqq, thenE​\(q\(N\),K\)≥EinfE\(q^\{\(N\)\},K\)\\geq E\_\{\\inf\}, so

∑s=0N−1rs2≤2​\(E​\(q\(0\),K\)−Einf\)\.\\sum\_\{s=0\}^\{N\-1\}r\_\{s\}^\{2\}\\;\\leq\\;2\\bigl\(E\(q^\{\(0\)\},K\)\-E\_\{\\inf\}\\bigr\)\.Dividing byNNand usingmins⁡rs2≤1N​∑srs2\\min\_\{s\}r\_\{s\}^\{2\}\\leq\\tfrac\{1\}\{N\}\\sum\_\{s\}r\_\{s\}^\{2\}yields the residual bound\.

For the explicit constant, Step 1 of the proof of Theorem[C\.2](https://arxiv.org/html/2605.08143#A3.Thmtheorem2)givesE​\(q,K\)≥−12−log⁡CβE\(q,K\)\\geq\-\\tfrac\{1\}\{2\}\-\\tfrac\{\\log C\}\{\\beta\}, so we may takeEinf=−12−log⁡CβE\_\{\\inf\}=\-\\tfrac\{1\}\{2\}\-\\tfrac\{\\log C\}\{\\beta\}\. Under the normalization‖q\(0\)‖2=1\\\|q^\{\(0\)\}\\\|\_\{2\}=1,‖ki‖2=1\\\|k\_\{i\}\\\|\_\{2\}=1,q\(0\)​ki⊤≥−1q^\{\(0\)\}k\_\{i\}^\{\\top\}\\geq\-1for everyii, hence

∑i=1Cexp⁡\(β​q\(0\)​ki⊤\)≥C​e−β,F​\(q\(0\)\)≥−1\+log⁡Cβ,\\sum\_\{i=1\}^\{C\}\\exp\(\\beta\\,q^\{\(0\)\}k\_\{i\}^\{\\top\}\)\\geq Ce^\{\-\\beta\},\\qquad F\(q^\{\(0\)\}\)\\geq\-1\+\\frac\{\\log C\}\{\\beta\},and thus

E​\(q\(0\),K\)=12−F​\(q\(0\)\)≤32−log⁡Cβ\.E\(q^\{\(0\)\},K\)=\\tfrac\{1\}\{2\}\-F\(q^\{\(0\)\}\)\\leq\\tfrac\{3\}\{2\}\-\\frac\{\\log C\}\{\\beta\}\.Combining withEinf=−12−log⁡CβE\_\{\\inf\}=\-\\tfrac\{1\}\{2\}\-\\frac\{\\log C\}\{\\beta\}givesE​\(q\(0\),K\)−Einf≤2E\(q^\{\(0\)\},K\)\-E\_\{\\inf\}\\leq 2, and therefore

min0≤s<N⁡rs≤2⋅2N=2N\.∎\\min\_\{0\\leq s<N\}r\_\{s\}\\;\\leq\\;\\sqrt\{\\frac\{2\\cdot 2\}\{N\}\}=\\frac\{2\}\{\\sqrt\{N\}\}\.\\qquad\\qed∎

##### Interpretation of the finite\-step bound\.

Proposition[C\.3](https://arxiv.org/html/2605.08143#A3.Thmtheorem3)controls the*best*of the firstNNiterates: there existss¯∈arg⁡min0≤s<N⁡‖T​\(q\(s\)\)−q\(s\)‖2\\bar\{s\}\\in\\arg\\min\_\{0\\leq s<N\}\\\|T\(q^\{\(s\)\}\)\-q^\{\(s\)\}\\\|\_\{2\}with residual at most2/N2/\\sqrt\{N\}\. It does*not*certify that the last iterateq\(N\)=TN​\(q\(0\)\)q^\{\(N\)\}=T^\{N\}\(q^\{\(0\)\}\)is itself an approximate fixed point—onlyE​\(q\(N\),K\)≤E​\(q\(0\),K\)E\(q^\{\(N\)\},K\)\\leq E\(q^\{\(0\)\},K\)is guaranteed\. Whether‖T​\(q\(N\)\)−q\(N\)‖2\\\|T\(q^\{\(N\)\}\)\-q^\{\(N\)\}\\\|\_\{2\}is small at the last step must be checked separately\. This distinction matters for HoReN: the algorithm fixesM=1M=1and outputsq\(1\)q^\{\(1\)\}rather than the residual\-minimizing iterate, so the design relies on the asymptotic\-attraction picture of Theorem[C\.2](https://arxiv.org/html/2605.08143#A3.Thmtheorem2)as a warning—namely, that runningM≫1M\\gg 1steps would contract*every*query, including unrelated ones—rather than as a route to a tighter approximate fixed point\.

### C\.4Empiricalβ\\betaCalibration

We observe no performance degradation up to 50K edits \(Figure[1](https://arxiv.org/html/2605.08143#S1.F1)\), indicating that normalized representations retain sufficient angular separation empirically as the codebook grows\. Our fixedβ=20\\beta=20is therefore sufficient across all scales studied in this paper, with no saturation observed in Table[2](https://arxiv.org/html/2605.08143#S3.T2)\.

## Appendix DSupplementary Experimental Setup Details

In this appendix, we provide a detailed description of the experimental configuration, including an introduction to the datasets, a comprehensive explanation of the evaluation metrics, the implementation details, and a discussion of the baselines\.

### D\.1Datasets

Here, we provide a detailed introduction to the datasets used in this paper:

- •ZsRE\[[13](https://arxiv.org/html/2605.08143#bib.bib23)\]is a question\-answering dataset that has become the standard benchmark for lifelong knowledge editing\. Edit prompts are factual questions paired with target answers, and rephrased prompts are produced via back\-translation to evaluate generalization\. Following prior work\[[20](https://arxiv.org/html/2605.08143#bib.bib18),[7](https://arxiv.org/html/2605.08143#bib.bib15)\], natural questions unrelated to the edited fact are used as out\-of\-scope queries to evaluate locality\. Each sample contains \(i\) an editing prompt, \(ii\) the target answer, \(iii\) a paraphrased prompt, and \(iv\) a locality prompt with its own ground\-truth answer\. We use the split fromMitchellet al\.\[[20](https://arxiv.org/html/2605.08143#bib.bib18)\]\(\>15\{\>\}15K samples\); the5050K stress test is constructed from the same source distribution \(Appendix[D\.4\.4](https://arxiv.org/html/2605.08143#A4.SS4.SSS4)\)\.
- •WikiBigEdit\[[25](https://arxiv.org/html/2605.08143#bib.bib30)\]is a large\-scale benchmark constructed from Wikidata triples that introduces a substantially harder locality regime: locality queries are sampled from triples sharing the same subject and related relations as the edited facts, placing them semantically close to edit queries\. This setting tests whether an editor can preserve clean angular separation between edits and unrelated knowledge even when the two are topically adjacent, a property that is trivially satisfied on ZsRE but fails for many existing methods on WikiBigEdit\.
- •UnKE\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]is an unstructured knowledge editing benchmark in which edits are free\-form passages rather than triples and target answers are multi\-sentence generations rather than short tokens\. UnKE serves as a probe into the robustness of an editor beyond the structured \(subject, relation, object\) regime\. Because targets are free\-form, evaluation uses surface\- and semantic\-similarity metrics \(ROUGE\-1/2/L and BERTScore\) instead of token\-level exact match \(see Appendix[D\.3\.2](https://arxiv.org/html/2605.08143#A4.SS3.SSS2)\)\.

### D\.2Baselines

We describe the baselines used across the experiments\. The descriptions are grouped by the binary categorization introduced in Section[3\.1](https://arxiv.org/html/2605.08143#S3.SS1): \(I\) parameter\-modifying methods, which write closed\-form updates into the base model’s feed\-forward weights, and \(II\) parameter\-preserving methods, which leave every base feed\-forward weight frozen and store edits in an external/side module consulted at inference\. The UnKE baseline is included only for the unstructured editing benchmark, as stated in Section[3\.1](https://arxiv.org/html/2605.08143#S3.SS1)\.

##### \(I\) Parameter\-modifying baselines\.

- •ROME\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]: a single\-fact editor for structured \(subject, relation, object\) triples\. ROME first runs a causal\-tracing analysis to identify the single mid\-layer feed\-forward block most responsible for recalling a given fact, then treats that block as an associative key–value memory in which the “key” is the internal representation of the subject and the “value” is the representation that triggers the object\. To install a new fact it computes a closed\-form, rank\-one adjustment to that block’s output projection so that the subject’s key is mapped to the new object’s representation while the responses for all other previously cached keys are left as unchanged as possible\. ROME edits one fact at a time and modifies the base model’s weights in place\.
- •AlphaEdit\[[5](https://arxiv.org/html/2605.08143#bib.bib25)\]: a closed\-form batch editor designed for sequential editing\. AlphaEdit performs a MEMIT\-style update to a feed\-forward block, but before applying the update it first projects the proposed weight change onto the subspace orthogonal to the internal representations of a sample of preserved facts drawn from a general corpus\. Because the update is forced to lie in this “do\-not\-disturb” subspace, applying it to the model leaves the outputs for those preserved representations exactly unchanged, which lets many edits be stacked sequentially with reduced interference on previously stored knowledge\.
- •UltraEdit\[[6](https://arxiv.org/html/2605.08143#bib.bib26)\]: a training\-free, subject\-free, and memory\-free editor\. UltraEdit requires no optimizer training, no identification of a subject token, and stores no record of past edits\. From a single forward pass it reads the hidden representation at a chosen layer and one gradient signal that points in the direction of the desired output, and combines them in closed form into a one\-step adjustment of the layer’s weights\. As successive edits arrive it maintains a running normalization of the statistics of those hidden representations so that later edits remain well\-scaled relative to earlier ones, enabling long edit streams\.
- •UnKE\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]: a domain\-specific baseline for unstructured edits whose targets are free\-form multi\-sentence passages rather than short entity strings\. UnKE extends the locate\-and\-edit recipe by working at the granularity of an entire layer’s input/output rather than a single subject token: the “key” becomes the chosen layer’s input representation of the whole question and the “value” becomes the layer\-output representation that, if produced, would cause the rest of the network to generate the desired passage\. UnKE first solves a causal optimization to determine that target layer\-output, then updates the feed\-forward weights of that layer so the question’s input representation produces it\.

##### \(II\) Parameter\-preserving baselines\.

- •GRACE\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]: a lifelong editor that leaves all model weights frozen and instead maintains an external lookup table whose entries are pairs of \(a stored hidden representation, the replacement hidden representation that should be produced at one chosen layer\)\. Each entry also owns a small “acceptance radius” around its stored representation\. At inference, the current activation at that layer is compared against every stored entry; if it falls inside any entry’s acceptance radius, the activation is overwritten by that entry’s replacement, otherwise the model runs unchanged\. Entries are added, shrunk, or merged as new edits arrive, so the lookup table grows monotonically with the edit stream\.
- •WISE\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\]: a lifelong editor that picks one mid\-to\-late feed\-forward block of the base model and never modifies its weights\. WISE allocates an extra copy of just that block’s value\-projection matrix as a “side memory”, initialized identically to the original\. All edits are written into the side copy only\. To prevent edits from interfering with each other, the edit history is split into shards: each shard is trained on a different random subset of the side copy’s parameters, and the resulting shards are then combined into one side copy via the Ties\-merging procedure – this merging happens across the side shards only and the side copy is never folded back into the original block\. At inference, a routing rule measures how strongly each input token activates the difference between the side copy and the original block; tokens whose activation crosses a learned threshold are sent through the side copy, all others go through the untouched original block\.

### D\.3Metrics

We evaluate knowledge editing methods along three axes—reliability, generalization, and locality—following the evaluation framework ofMitchellet al\.\[[20](https://arxiv.org/html/2605.08143#bib.bib18)\]\. All metrics are computed over a set ofNNedit instances\{\(xi,yi\)\}i=1N\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, wherexix\_\{i\}is the edit prompt andyiy\_\{i\}is the desired target answer\. We denote the original unedited model byfθ0f\_\{\\theta\_\{0\}\}and the model after sequentially applying allNNedits byfθNf\_\{\\theta\_\{N\}\}\. Let𝒱\\mathcal\{V\}denote the vocabulary\. Given a modelffand promptxx, we obtain a predicted sequencey^=\(t^1,…,t^L\)\\hat\{y\}=\(\\hat\{t\}\_\{1\},\\dots,\\hat\{t\}\_\{L\}\)via greedy autoregressive decoding,

t^j=arg​maxv∈𝒱⁡Pf​\(v∣x,t^1,…,t^j−1\),\\hat\{t\}\_\{j\}=\\operatorname\*\{arg\\,max\}\_\{v\\in\\mathcal\{V\}\}\\;P\_\{f\}\\\!\\left\(v\\mid x,\\,\\hat\{t\}\_\{1\},\\dots,\\hat\{t\}\_\{j\-1\}\\right\),\(10\)whereLLis fixed to the number of tokens in the targetyiy\_\{i\}\. We define the per\-instance task measure

m​\(y,y^\)=1L​∑j=1L𝟙​\[t^j=tj\],m\(y,\\hat\{y\}\)=\\frac\{1\}\{L\}\\sum\_\{j=1\}^\{L\}\\mathbb\{1\}\[\\hat\{t\}\_\{j\}=t\_\{j\}\],\(11\)which computes the fraction of correctly predicted tokens\.

#### D\.3\.1Structured Metrics \(ZsRE & WikiBigEdit\)

Following the previous work\[[20](https://arxiv.org/html/2605.08143#bib.bib18),[7](https://arxiv.org/html/2605.08143#bib.bib15),[26](https://arxiv.org/html/2605.08143#bib.bib11)\], the three metrics on structured benchmarks are defined as follows\.

- •Reliabilitymeasures whether the edited model produces the target answer for the original edit prompt: Reliability=1N​∑i=1Nm​\(yi,y^ifθN,xi\)\.\\mathrm\{Reliability\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}m\\\!\\left\(y\_\{i\},\\;\\hat\{y\}\_\{i\}^\{\\,f\_\{\\theta\_\{N\}\},\\,x\_\{i\}\}\\right\)\.\(12\)
- •Generalizationevaluates whether the edit transfers to semantically equivalent rephrasingsxi′x\_\{i\}^\{\\prime\}of the edit prompt: Generalization=1N​∑i=1Nm​\(yi,y^ifθN,xi′\)\.\\mathrm\{Generalization\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}m\\\!\\left\(y\_\{i\},\\;\\hat\{y\}\_\{i\}^\{\\,f\_\{\\theta\_\{N\}\},\\,x\_\{i\}^\{\\prime\}\}\\right\)\.\(13\)
- •Localityquantifies the extent to which the edit preserves the model’s behavior on unrelated inputs\. For each locality promptxiℓx\_\{i\}^\{\\ell\}with associated ground\-truth lengthLiℓL\_\{i\}^\{\\ell\}, we extract the lastLiℓL\_\{i\}^\{\\ell\}tokens from the full output sequence of both the original and edited models, denotedzifθ0z\_\{i\}^\{\\,f\_\{\\theta\_\{0\}\}\}andzifθNz\_\{i\}^\{\\,f\_\{\\theta\_\{N\}\}\}respectively, and compute Locality=1N​∑i=1N1Liℓ​∑j=1Liℓ𝟙​\[zi,jfθ0=zi,jfθN\]\.\\mathrm\{Locality\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{1\}\{L\_\{i\}^\{\\ell\}\}\\sum\_\{j=1\}^\{L\_\{i\}^\{\\ell\}\}\\mathbb\{1\}\\\!\\left\[z^\{\\,f\_\{\\theta\_\{0\}\}\}\_\{i,j\}=z^\{\\,f\_\{\\theta\_\{N\}\}\}\_\{i,j\}\\right\]\.\(14\)Notably, locality compares the original and edited model outputs to each other rather than against a ground\-truth label, directly measuring behavioral preservation\.

The Overall Performance \(OP\) score reported alongside the three axes is their geometric mean,OP=\(Reliability⋅Generalization⋅Locality\)1/3\\mathrm\{OP\}=\(\\mathrm\{Reliability\}\\cdot\\mathrm\{Generalization\}\\cdot\\mathrm\{Locality\}\)^\{1/3\}\.

#### D\.3\.2Unstructured Metrics \(UnKE\)

Because UnKE targets are multi\-sentence free\-form passages, the per\-token measurem​\(y,y^\)m\(y,\\hat\{y\}\)is replaced by surface\-level and semantic\-level similarity scores between the generated answery^i\\hat\{y\}\_\{i\}and the referenceyiy\_\{i\}\. Following the original UnKE protocol\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\], the main text reports ROUGE\-1, ROUGE\-2, ROUGE\-L, and BERTScore\. Appendix[F\.1](https://arxiv.org/html/2605.08143#A6.SS1)additionally includes the full UnKEBench view with Original, Paraphrase, and Sub\-question splits; these correspond to reliability, generalization, and factual coverage for unstructured passages, rather than to the structured locality metric used for ZsRE and WikiBigEdit\.

### D\.4Implementation Details

#### D\.4\.1Baseline Hyperparameters

For every baseline we adopt the original hyperparameters published by the corresponding authors and only re\-map architecture\-specific module names \(e\.g\.,mlp\.down\_projfor LLaMA/Qwen,self\_attn\.o\_projfor GPT\-OSS\) when porting a recipe to a model that the original paper did not study\. Random seeds are fixed to4242across all experiments for all baselines and HoReN\.

##### ROME\.

We use theMenget al\.\[[18](https://arxiv.org/html/2605.08143#bib.bib1)\]recipe verbatim: rewrite layerℓ=5\\ell\{=\}5,subject\_lastfact\-token,2525value\-optimization steps with learning rate0\.50\.5, weight decay10−310^\{\-3\}, KL factor0\.06250\.0625, clamp norm factor44, and second\-moment adjustment withK0K\_\{0\}statistics estimated from10510^\{5\}Wikipedia samples in float32\. The value\-loss layer is set to the last transformer block, i\.e\.,vloss=nlayers−1v\_\{\\text\{loss\}\}=n\_\{\\text\{layers\}\}\-1for every LLM\.

##### AlphaEdit\.

We follow theFanget al\.\[[5](https://arxiv.org/html/2605.08143#bib.bib25)\]LLaMA\-3 configuration: a 5\-layer rewrite window\{4,5,6,7,8\}\\\{4,5,6,7,8\\\},λ=15,000\\lambda\{=\}15\{,\}000forK0K\_\{0\}adjustment,2525value\-optimization steps with learning rate0\.10\.1, weight decay0\.50\.5, clamp norm0\.750\.75, KL factor0\.06250\.0625,L2L\_\{2\}regularization1010, and null\-space threshold2×10−22\{\\times\}10^\{\-2\}\. The null\-space projectionPPis precomputed once per model from the same10510^\{5\}\-sample Wikipedia covariance\. The value\-loss layer is againnlayers−1n\_\{\\text\{layers\}\}\-1\(3131for the 8B LLaMAs,2727for both 28\-layer Qwen\-based models,6363for the 64\-layer SEA\-LION\-32B\)\.

##### GRACE\.

We use theHartvigsenet al\.\[[7](https://arxiv.org/html/2605.08143#bib.bib15)\]defaults: codebook learning rate1\.01\.0\(SGD\) with100100inner steps, Euclidean distance, cold value initialization,replace\_lasttoken replacement, coverage\-basedϵ\\epsilon\-expansion with initialϵ=1\\epsilon\{=\}1and early\-stop regularization\. Edits are applied to a single MLPdown\_projlayer per LLM \(see the cross\-method layer alignment below\)\.

##### WISE\.

We adopt theWanget al\.\[[26](https://arxiv.org/html/2605.08143#bib.bib11)\]Tab\. 10 recipe: edit learning rate1\.01\.0\(SGD\),7070inner iterations, mask ratioρ=0\.2\\rho\{=\}0\.2, activation margins\(α,β,γ\)=\(5,20,10\)\(\\alpha,\\beta,\\gamma\)\{=\}\(5,20,10\), activation ratio0\.880\.88, side\-memory merge frequency10001000with TIES merging at density0\.530\.53and weight1\.01\.0, and retrieval enabled\. Norm constraint is1\.01\.0\. Edits are applied to a single MLPdown\_projlayer per LLM \(see the cross\-method layer alignment below\)\.

##### UltraEdit\.

We use theGuet al\.\[[6](https://arxiv.org/html/2605.08143#bib.bib26)\]recipe: lifelong learning rate10−610^\{\-6\},token=mask, editor batch size10241024, with the publishedgate\_proj\+up\_projlayer ranges for LLaMA\-3\-8B \(layers 11–15 / 18–24\) and Qwen2\.5\-7B \(layers 18–26\)\. For DeepSeek\-R1\-Distill\-Qwen\-1\.5B \(28 layers\) and Qwen\-SEA\-LION\-v4\-32B \(64 layers\), where no published recipe exists, we transfer the Qwen recipe \(layers 16–20 and 18–26 respectively\)\.

#### D\.4\.2HoReN Hyperparameters

HoReN usesβ=20\\beta\{=\}20,γ=0\.01\\gamma\{=\}0\.01,M=1M\{=\}1, andϵ=10−5\\epsilon\{=\}10^\{\-5\}across all experiments for the Hopfield\-style refinement\.

When a new entry is created \(Algorithm[1](https://arxiv.org/html/2605.08143#algorithm1)\), the base model is frozen and only the codebook value adaptor parameters are optimized by Adam on the per\-token cross\-entropy loss of the target answer\. Per\-edit adaptor optimization uses learning rate0\.10\.1, up toU=50U\{=\}50optimization steps per edit, and early stopping when the per\-token loss falls below10−210^\{\-2\}or fails to improve for 3 consecutive steps\. No gradient flows back into the base model or into stored codebook keys; each codebook value adaptor is trained in isolation\. The choice of layer where the HoReN codebook resides is specified in Appendix[D\.4\.3](https://arxiv.org/html/2605.08143#A4.SS4.SSS3)\.

Two other model\-specific parameters are required: the matching thresholdccand the token pooling ratio, both selected once per model via ablation; per\-model values are listed in Table[3](https://arxiv.org/html/2605.08143#A4.T3)\. HoReN uses tensor values as the default codebook value adaptor; the optional LoRA variant uses rankr=4r\{=\}4\. Experiments run on RTX 5090 in fp16, with GPT\-OSS\-20B run on RTX Pro 6000 WS\.

Table[3](https://arxiv.org/html/2605.08143#A4.T3)lists the two model\-specific parameters—matching thresholdccand token pooling ratio—for each backbone\. Both are selected once per model via the ablation in Section[3\.4\.2](https://arxiv.org/html/2605.08143#S3.SS4.SSS2)and held fixed across all datasets and edit scales\.

Table 3:Per\-model hyperparameters used in all experiments\.
#### D\.4\.3Cross\-method alignment and Hardware

##### Cross\-method layer alignment\.

To avoid confounding HoReN’s gains with a more favorable layer choice, GRACE, WISE, and HoReN are configured to edit the same layer per LLM: layer2929\(LLaMA\-3 / 3\.1 / DeepSeek\-R1\-Distill\-LLaMA\-8B\), layer2424\(Qwen2\.5\-7B\-Instruct\), layer2727\(DeepSeek\-R1\-Distill\-Qwen\-1\.5B\), layer5656\(SEA\-LION\-32B\), and layer2020ofself\_attn\.o\_proj\(GPT\-OSS\-20B\)\. These choices follow the published WISE/GRACE convention of editing a late MLPdown\_projblock \(≈nlayers−3\\approx n\_\{\\text\{layers\}\}\-3tonlayers−8n\_\{\\text\{layers\}\}\-8\)\. ROME and AlphaEdit are excluded from this alignment because their original methodology prescribes different layer locations \(causal\-tracing layer55for ROME; the rewrite window\{4,…,8\}\\\{4,\\ldots,8\\\}for AlphaEdit\) and changing them would deviate from those papers\.

##### Hardware\.

Experiments run on RTX 5090 in fp16, with GPT\-OSS\-20B run on RTX Pro 6000 WS, matching the implementation summary in Section[3\.1](https://arxiv.org/html/2605.08143#S3.SS1)\.

#### D\.4\.450K ZsRE Stress\-Test Construction

The 50K stress test in Figure[1](https://arxiv.org/html/2605.08143#S1.F1)is drawn from a pool of 75,658 ZsRE\-style samples aggregated from the standard ZsRE train split released byMitchellet al\.\[[20](https://arxiv.org/html/2605.08143#bib.bib18)\]together with additional entries drawn from the original ZsRE source corpus\[[13](https://arxiv.org/html/2605.08143#bib.bib23)\]that do not overlap with the standard split, yielding a pool large enough to support a stream of 50K non\-overlapping edits\. Each pool entry carries \(i\) an original prompt, \(ii\) a target answer, \(iii\) a paraphrased query, and \(iv\) a locality query, so the three evaluation sets \(𝒟edit,𝒟rephrase,𝒟locality\\mathcal\{D\}\_\{\\mathrm\{edit\}\},\\mathcal\{D\}\_\{\\mathrm\{rephrase\}\},\\mathcal\{D\}\_\{\\mathrm\{locality\}\}\) are aligned one\-to\-one with each edit, matching the Section[2\.1](https://arxiv.org/html/2605.08143#S2.SS1)formulation\. For the stress test we sample 50K entries from this pool \(with a fixed seed\) and apply them as a single uninterrupted sequential stream on LLaMA\-3\.1\-8B, chosen as a representative 8B\-scale model; results on other backbones at smaller scales are consistent \(Table[2](https://arxiv.org/html/2605.08143#S3.T2)\)\. Evaluation is carried out at checkpointsN∈\{1​K,10​K,15​K,20​K,30​K,40​K,50​K\}N\\in\\\{1\\text\{K\},10\\text\{K\},15\\text\{K\},20\\text\{K\},30\\text\{K\},40\\text\{K\},50\\text\{K\}\\\}over the matching prefix of each dataset\.

#### D\.4\.5Portability Guidelines

Model\-specific parameters are limited to two choices: \(1\) the matching thresholdccand \(2\) the token pooling ratio\. Both are selected once per model and then held fixed across datasets and edit scales\. The resulting values \(Table[3](https://arxiv.org/html/2605.08143#A4.T3)\) cluster tightly within each family while shifting cleanly between families, indicating that they reflect genuine embedding\-geometry differences rather than dataset\- or benchmark\-specific tuning\.

Concretely, the calibrated thresholds fall into three distinct, family\-coherent bands:

- •LLaMA\-3 / 3\.1 family\(c∈\[0\.80,0\.85\]c\\in\[0\.80,0\.85\], span0\.050\.05\): LLaMA\-3\-8B\-Instruct uses0\.800\.80, while LLaMA\-3\.1\-8B\-Instruct and DeepSeek\-R1\-Distill\-Llama\-8B both use0\.850\.85\. The fact that DeepSeek’s distilled LLaMA inherits the LLaMA\-3\.1 threshold despite having undergone substantial reasoning\-oriented post\-training is direct evidence thatcctracks the underlying representation geometry of the base model rather than the downstream task\.
- •Qwen\-2\.5 / 3 family\(c∈\[0\.55,0\.60\]c\\in\[0\.55,0\.60\], span0\.050\.05\): Qwen2\.5\-7B\-Instruct and the 32B Qwen\-Sea\-Lion\-v4 both use0\.550\.55, and DeepSeek\-R1\-Distill\-Qwen\-1\.5B uses0\.600\.60– again, the distilled variant remains within the family band\. The consistent∼0\.25\\sim\\\!0\.25gap from the LLaMA band reflects the smaller cosine spread of Qwen hidden states at the edited layer, not a per\-experiment correction\.
- •GPT\-OSS\(c=0\.80c=0\.80\): the only model in its family, sitting in the LLaMA range, consistent with its larger residual stream norms\.

Pooling ratio is even more stable:60%60\\%for every model except LLaMA\-3\-8B\-Instruct \(50%50\\%\), and chosen by a single ablation \(Figure[11](https://arxiv.org/html/2605.08143#A7.F11)\) rather than re\-tuned per benchmark or scale\.

This level of calibration is consistent with existing methods: GRACE requires per\-modelε\\varepsilon, WISE requires per\-model importance weights, and REPAIR requires per\-model cross\-layer settings – in each case the per\-model knob also reflects representation\-level properties rather than dataset\-specific tuning\. For a new model we recommend: \(i\) identify the family of the base model and initializeccto the family’s central value \(0\.850\.85for LLaMA\-derived models,0\.550\.55for Qwen\-derived models\); \(ii\) run the token\-selection ablation at moderate scale \(N=100N\{=\}100\) to confirm the60%60\\%pooling default; and \(iii\) if needed, refineccwith a short sweep over\{0\.55,0\.60,0\.80,0\.85\}\\\{0\.55,0\.60,0\.80,0\.85\\\}on3030edits, anchored at the family\-default value\.

## Appendix EAdditional ZsRE Results

### E\.1LLaMA\-3\.1\-8B Large\-Scale Stability

Table[4](https://arxiv.org/html/2605.08143#A5.T4)reports full LLaMA\-3\.1\-8B results atN=2000N\{=\}2000,50005000, and1000010000edits, including AlphaEdit and UltraEdit which are omitted from Table[2](https://arxiv.org/html/2605.08143#S3.T2)due to prohibitive runtime on other models\. AlphaEdit collapses betweenN=2000N\{=\}2000andN=5000N\{=\}5000\(OP drops from0\.820\.82to0\.100\.10\) due to near\-zero locality; UltraEdit degrades more gradually \(OP from0\.730\.73to0\.670\.67\)\. HoReN maintains OP above0\.950\.95throughout all scales\.

Table 4:LLaMA\-3\.1\-8B results at large scale on ZsRE \(N=2000N\{=\}2000,50005000,1000010000\)\. AlphaEdit and UltraEdit are included here to document their scaling behavior; they are excluded from Table[2](https://arxiv.org/html/2605.08143#S3.T2)due to prohibitive runtime on other models\.Figure[5](https://arxiv.org/html/2605.08143#A5.F5)tracks the full sequential trajectory up to 10K edits: HoReN’s generalization stays flat while GRACE and WISE decay monotonically, and HoReN retains99%99\\%accuracy on early edits after the full 10K sequence with no catastrophic forgetting\.

![Refer to caption](https://arxiv.org/html/2605.08143v1/x5.png)Figure 5:Stability of editing performance over sequential edits on ZsRE with LLaMA\-3\.1\-8B \(up to 10K edits\)\. HoReN maintains stable generalization while GRACE and WISE degrade\.Figures[6](https://arxiv.org/html/2605.08143#A5.F6)–[8](https://arxiv.org/html/2605.08143#A5.F8)decompose the OP curves from Figure[1](https://arxiv.org/html/2605.08143#S1.F1)into the three constituent metrics for all methods on ZsRE \(LLaMA\-3\.1\-8B\)\. HoReN maintains Reliability above 0\.99 and Generalization above 0\.91 throughout 50K edits, while Locality declines gradually from 1\.00 to 0\.89—the primary source of the slight OP drop\. GRACE sustains near\-perfect Reliability and Locality but has effectively zero Generalization at all scales, confirming the routing bottleneck rather than a weight\-editing limitation\. AlphaEdit’s Locality collapse between 2K and 5K drives its OP cliff\.

![Refer to caption](https://arxiv.org/html/2605.08143v1/x6.png)Figure 6:Reliability scaling to 50K sequential edits on ZsRE \(LLaMA\-3\.1\-8B\)\.![Refer to caption](https://arxiv.org/html/2605.08143v1/x7.png)Figure 7:Generalization scaling to 50K sequential edits on ZsRE \(LLaMA\-3\.1\-8B\)\.![Refer to caption](https://arxiv.org/html/2605.08143v1/x8.png)Figure 8:Locality scaling to 50K sequential edits on ZsRE \(LLaMA\-3\.1\-8B\)\.
### E\.2Efficiency Scaling AcrossNN

![Refer to caption](https://arxiv.org/html/2605.08143v1/x9.png)Figure 9:Efficiency scaling on ZsRE \(LLaMA\-3\.1\-8B\)\.\(a\)Edit latency is constant inNN\(∼1670\{\\sim\}1670ms\) for all methods, dominated by per\-edit adaptor training\.\(b\)Inference latency scales linearly inNN; HoReN adds only∼5%\{\\sim\}5\\%overhead over GRACE from the single Hopfield step\.\(c\)Parameter overhead also scales linearly; HoReN and GRACE are nearly identical \(\+4\.09\+4\.09M vs\.\+4\.10\+4\.10M per 1K edits\), while WISE costs14×14\{\\times\}more\.All three methods scale linearly in inference latency withNN, as expected for nearest\-neighbor retrieval over a growing codebook\. HoReN’s single damped Hopfield step adds only∼5%\{\\sim\}5\\%overhead over GRACE at every scale, including 50K edits, because the step is a fixed\-cost softmax/matmul/normalization independent of codebook size\. Edit time is constant inNNfor all methods \(∼1670\{\\sim\}1670ms\), confirming that per\-edit adaptor training dominates and does not accumulate\. WISE achieves lower inference latency by routing through a smaller dual\-memory structure, but at the cost of\+58\.72\+58\.72M additional parameters versus\+4\.09\+4\.09M for HoReN\.

### E\.3Efficiency Details atN=1000N\{=\}1000

Table[5](https://arxiv.org/html/2605.08143#A5.T5)reports edit latency, inference latency, and parameter overhead atN=1000N\{=\}1000edits on ZsRE for LLaMA\-3\.1\-8B and Qwen\-SEA\-LION\-v4\-32B\. HoReN matches GRACE on all three axes at both scales; AlphaEdit costs2\.4×2\.4\\times–4\.7×4\.7\\timesmore edit time and71×71\\times–129×129\\timesmore parameters\.

Table 5:Efficiency comparison atN=1000N\{=\}1000edits on ZsRE\. Edit latency is dominated by per\-edit adaptor training and is constant inNNfor all methods\.
### E\.4Efficiency: Single\-Pass Latency and Memory Scaling

Single\-pass latency\.HoReN uses exactlyM=1M\{=\}1iteration with a damped update \(γ=0\.01\\gamma\{=\}0\.01\), not multiple recursive passes\. The Hopfield refinement consists of one softmax, one matrix multiplication, and one normalization—essentially one additional attention\-like operation\. AtN=10N\{=\}10K the softmax is computed over a larger key set, but remains a single\-pass operation\.

Memory scaling\.Codebook memory scales linearly with edits\. AtN=1000N\{=\}1000on LLaMA\-3\.1\-8B \(d=4096d\{=\}4096\): HoReN \(Value\) adds\+4\.09\+4\.09M parameters \(∼16\{\\sim\}16MB in fp16\), comparable to GRACE \(\+4\.10\+4\.10M\); HoReN \(LoRA\) adds\+32\.64\+32\.64M \(∼65\{\\sim\}65MB in fp16\)\. Since LoRA and direct value editing achieve identical performance \(Table[12](https://arxiv.org/html/2605.08143#A7.T12)\), practitioners can choose the Value variant for memory efficiency with no performance penalty\.

## Appendix FCross\-Dataset Generalization

### F\.1Full Numerical Results

Tables[6](https://arxiv.org/html/2605.08143#A6.T6)–[9](https://arxiv.org/html/2605.08143#A6.T9)report the complete numerical results underlying Figure[3](https://arxiv.org/html/2605.08143#S3.F3)in Section[3\.3\.1](https://arxiv.org/html/2605.08143#S3.SS3.SSS1)\.

Table 6:Sequential editing on WikiBigEdit \(LLaMA\-3\.1\-8B\)\. AlphaEdit’s locality degrades sharply under long edit sequences while HoReN remains stable\.Table 7:Unstructured knowledge editing on UnKE \(LLaMA\-3\.1\-8B\)\. HoReN outperforms the domain\-specific UnKE baseline across all metrics at all three scales; the gap widens dramatically atN=500N\{=\}500\.Table 8:Unstructured knowledge editing on UnKE \(LLaMA\-3\.1\-8B\) broken down by question type\. Rows: three UnKEBench question types \(*Original*,*Paraphrase*,*Sub\-question*\) crossed with all applicable metrics\. Columns: edit countsN∈\{10,100,500\}N\\\!\\in\\\!\\\{10,100,500\\\}, each split between UnKE and HoReN\. Best per \(row,NN\) in bold; BLEU and BERTScore are not reported for the Sub\-question split in the original UnKEBench protocol\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]\.Column / row semantics \(UnKEBench\[[3](https://arxiv.org/html/2605.08143#bib.bib27)\]\)\.For each unstructured edit textAA, UnKEBench provides three question types probing complementary aspects of the edited model’s behavior\. TheOriginalblock measures*reliability*: whether the edit is faithfully recalled on the original questionQQ\. TheParaphraseblock measures*generalization*: whether the edit transfers to a semantically equivalent rewordingQpQ\_\{p\}ofQQ\. TheSub\-questionblock measures*factual coverage*: whether individual knowledge entities within the passage are correctly installed\.BLEUandROUGE\-1/2/Lare word\-levelnn\-gram overlap scores;BERTScorecaptures semantic similarity beyond lexical match\. BLEU and BERTScore are not reported for the Sub\-question split in the original UnKEBench protocol\.

Table 9:Cross\-dataset generalization on WikiBigEdit and ZsRE atN=1000N\{=\}1000edits \(DeepSeek\-R1\-Distill\-Qwen\-1\.5B\)\. Gray superscripts show HoReN’s absolute improvements over each baseline\.

## Appendix GAnalysis and Ablation Details

### G\.1Routing Ingredients: GRACE→\\toGRACE\+Norm\.→\\toHoReN

Table[10](https://arxiv.org/html/2605.08143#A7.T10)isolates the contribution of each routing ingredient\. Table[11](https://arxiv.org/html/2605.08143#A7.T11)confirms that applying Hopfield updates*without*normalization collapses locality, motivating the joint design\.

Table 10:Routing ingredient ablation: unnormalized GRACE vs\. GRACE\+Norm\. vs\. HoReN \(ZsRE, LLaMA\-3\.1\-8B\)\.Table 11:Effect of normalization on Hopfield retrieval \(ZsRE, LLaMA\-3\.1\-8B\)\. Without normalization, the Hopfield step collapses locality\.
### G\.2Hopfield Iteration Steps

![Refer to caption](https://arxiv.org/html/2605.08143v1/x10.png)Figure 10:Effect of Hopfield refinement stepsMMon the four editing metrics across edit countsN∈\{100,500,1000\}N\\\!\\in\\\!\\\{100,500,1000\\\}\(LLaMA\-3\.1\-8B, ZsRE\)\. Reliability is essentially flat inMM; Generalization peaks atM∈\[1,4\]M\\\!\\in\\\!\[1,4\]; Locality cliffs from∼0\.95\{\\sim\}0\.95to∼0\.6\{\\sim\}0\.6betweenM=2M\{=\}2andM=4M\{=\}4and bottoms out near0\.070\.07forM≥8M\\\!\\geq\\\!8, dragging OP with it\. The single damped step \(M=1M\{=\}1, gold star\) is the only value ofMMthat keeps all three metrics jointly stable across all three scales\.
### G\.3Value Adaptor: Direct Vector vs\. LoRA

Direct\-vector and LoRA value adaptors achieve identical editing performance; the direct variant is preferred for its8×8\{\\times\}lower parameter cost\.The default adaptor addshj′=hj\+vi⋆h\_\{j\}^\{\\prime\}=h\_\{j\}\+v\_\{i^\{\\star\}\}\(one vector of dimensionddper edit\); the LoRA variant applieshj′=hj\+hj​Ai⋆​Bi⋆⊤h\_\{j\}^\{\\prime\}=h\_\{j\}\+h\_\{j\}A\_\{i^\{\\star\}\}B\_\{i^\{\\star\}\}^\{\\top\}withr=4r\{=\}4\(2​r​d2rdparameters per edit\)\. Table[12](https://arxiv.org/html/2605.08143#A7.T12)compares the two acrossN∈\{100,500,1000\}N\{\\in\}\\\{100,500,1000\\\}\.

Both variants reach OP0\.950\.95–0\.980\.98at every scale, with differences of at most0\.010\.01\. The equivalence is expected: HoReN’s retrieval accuracy depends entirely on the key\-matching mechanism \(normalization \+ Hopfield step\), not on the capacity of the value payload\. Once the correct edit is retrieved, either adaptor has sufficient expressivity to represent the target answer at these scales\. The direct variant adds\+4\.09\+4\.09M parameters atN=1000N\{=\}1000versus\+32\.64\+32\.64M for LoRA—an8×8\{\\times\}gap with no performance benefit\. We therefore recommend direct\-vector as the default; LoRA remains available for deployments requiring strict parameter isolation between edits \(e\.g\., federated or multi\-tenant settings\)\.

Table 12:Direct value vs\. LoRA \(r=4r\{=\}4\) on ZsRE, LLaMA\-3\.1\-8B\. Performance is identical; direct variant adds8×8\\timesfewer parameters\.
### G\.4Key/Query Representation: Token Pooling Ratio

The last 60% suffix pooling ratio is Pareto\-optimal across all scales; performance is non\-monotonic, with both short and long suffixes introducing distinct failure modes\.Figure[11](https://arxiv.org/html/2605.08143#A7.F11)sweeps ten ratios from 10% to 100% on LLaMA\-3\.1\-8B, ZsRE, atN∈\{100,500,1000\}N\{\\in\}\\\{100,500,1000\\\}\.

*Short suffixes \(≤\\leq30%\) — generalization failure\.*Very short suffixes pool too few tokens to capture the semantic content of the prompt, producing a query representation that reflects surface position rather than meaning\. Gen\. stays at0\.470\.47–0\.670\.67atN=1000N\{=\}1000regardless ofNN: the key constructed from the original prompt’s last few tokens is insufficiently similar to the key from the paraphrase’s last few tokens, so the Hopfield step cannot bridge the residual angular gap\.

*Long suffixes \(≥\\geq80%\) — locality failure\.*Very long suffixes include tokens specific to the prompt’s preamble or instruction format, making the query representation sensitive to surface\-level prompt differences rather than semantic content\. This causes locality collapse: atN=1000N\{=\}1000, Loc\. drops to0\.740\.74–0\.840\.84as unrelated locality queries—which share similar preambles with edit queries—begin triggering false matches\.

*Last 60% — Pareto\-optimal\.*The 60% suffix captures enough semantic context for robust paraphrase matching \(Gen\.0\.880\.88–0\.930\.93\) while excluding enough of the prompt\-specific prefix to keep locality queries well\-separated \(Loc\.0\.990\.99–1\.001\.00\)\. The5050–70%70\\%range all reach OP≥0\.95\{\\geq\}0\.95, indicating the choice is not sensitive to exact calibration\. We treat the 60% default as an empirical heuristic; learned token selection \(e\.g\., attention\-weighted pooling\) is a concrete future direction\.

![Refer to caption](https://arxiv.org/html/2605.08143v1/x11.png)Figure 11:Effect of suffix pooling ratio on the four editing metrics acrossN∈\{100,500,1000\}N\\\!\\in\\\!\\\{100,500,1000\\\}\(LLaMA\-3\.1\-8B, ZsRE\)\. Reliability is essentially flat at1\.001\.00; Generalization grows monotonically from∼0\.50\{\\sim\}0\.50at10%10\\%to∼0\.93\{\\sim\}0\.93from60%60\\%onward; Locality is held at1\.001\.00up to60%60\\%then decays as longer suffixes start incorporating prompt\-specific tokens, dropping to0\.74−0\.840\.74\\\!\-\\\!0\.84at≥80%\\geq 80\\%forN=1000N\{=\}1000\. The50%−70%50\\%\\\!\-\\\!70\\%Pareto band \(green\) all reach OP≥0\.94\\geq 0\.94; the≥80%\\geq 80\\%locality\-decay zone \(red\) cuts OP\. The chosen60%60\\%setting \(gold star\) is the only ratio that simultaneously achieves Generalization≥0\.88\\geq 0\.88and Locality≥0\.99\\geq 0\.99at every scale\.