EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Hugging Face Daily Papers Papers

Summary

EngramEdit 提出一种通过条件记忆(conditional memory)实现 LLM 解耦知识更新的方法,联合更新共享 n-gram 嵌入以实现近乎完美的编辑成功率,在多跳推理中达到最强基线近三倍的准确率,同时保留无关知识与通用能力。

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:13 PM

Paper page - EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Source: https://huggingface.co/papers/2610.10533

Abstract

ConditionalmemoryarchitecturessuchasDeepSeekEngramuseinputn-gramstolookuplearnedembeddings,expandingthecapacityoflargelanguagemodels(LLMs)withlimitedadditionalcomputation.Beyondmodelscaling,thisarchitecturehasdemonstratedthepotentialtodecouplefactualknowledgestoragefromgeneral-purposecomputation,offeringapromisingroutetoupdatingfactualknowledgewhilekeepingtheTransformerbackbonefixed.Realizingthispotentialischallengingbecausedifferentexpressionsofafactmayactivatedifferentn-gramembeddings,whileupdatingsharedembeddingscanunintentionallychangethemodel’spredictionsaboutotherfacts.WeproposeEngramEditfordecoupledknowledgeupdatesthroughconditionalmemory.EngramEditfirstcomputestargetmemoryrepresentationsthatmakethemodelpredicttheupdatedfactacrossmultipleexpressions.Itthenjointlyupdatesthesharedn-gramembeddingstomatchthesetargetsacrossexpressionsandedits,penalizingupdatestofrequentlyreusedembeddingsmorestronglytopreserveunrelatedknowledge.ExperimentsshowthatEngramEditenablesindependentfactualknowledgeupdatesthroughconditionalmemory,achievingnear-perfecteditingsuccess.Revisedknowledgeisusableacrossunseenexpressionsandinmulti-hopreasoning,withnearlythreetimesthestrongestbaseline’saccuracyunderchain-of-thought(CoT)prompting.Unrelatedknowledgeandgeneralcapabilitiesarelargelypreservedevenasfactualupdatesaccumulate.ThesefindingsshowthatEngramEditturnsconditionalmemoryintoaneditableknowledgeinterface,extendingitsrolebeyondmodelscalingtosupportdecoupledknowledgeupdates.

View arXiv pageView PDFProject pageGitHub1Add to collection

Similar Articles