The Hard Part of AI Memory Is the Write, Not the Read

I was reading through a reddit post now deleted, around AI memory, and it pushed me to write down something I have been thinking about for a long time. Not which tools to use. Just, in simple terms, how memory should actually function. Anyone working with AI has some version of this in their head. This is mine. Would love to hear where you agree or disagree. These are just my thoughts on this, I am more than happy to refine my thought process, below is the article. Any useful AI agent needs memory to work well. Claude Code working on a large codebase needs to know what it has already built, why certain architectural decisions were made, which bugs have already been investigated and what is still unfinished. A personal agent needs to know what you prefer, what you are working on, what you decided last week and what has changed since then. An enterprise agent may need to understand customers, projects, processes and decisions across hundreds of things happening in an organisation. So memory is obviously important. But the more I think about it, the less convinced I am that the hard part is simply storing more information and retrieving the right information later. That is how a lot of AI memory is framed today. Something happens. We store it. Later, when the model needs context, we find the pieces that look relevant and put them back into the context window. Vector search, recency, summaries, graphs, larger context windows, memory tiers. All of these are useful. But before deciding how to store memory, I think there is a more basic question. What is memory actually supposed to do? Because having access to the past is not the same thing as understanding the past. And that difference becomes obvious as soon as an agent starts doing real work. Imagine I am using an agent to help sell to a company. There isn't just one conversation. We research the company. We understand their business. We identify the people we want to speak with. Then outreach, a discovery call, perhaps a proposal, then follow-ups. Now suppose I hand the agent the transcript of that discovery call. Somewhere in forty minutes of conversation, the customer tells us their data team spends a lot of time answering ad-hoc questions. We learn they use Snowflake. Someone says they don't want another dashboard. We agree to run a six-week pilot. Security approval is still pending. These are all things worth remembering. But they aren't the same kind of thing. "They use Snowflake" is a fact. "The data team seems overloaded" is an observation. "We don't want another dashboard" is a preference expressed by somebody. "We'll run a six-week pilot" is a decision. "Security approval is pending" is unresolved work. The agent's job is not to store the transcript. It is to break the transcript down into these pieces and file each one where it belongs, at the level of this account. The pilot decision belongs to this deal. The Snowflake fact belongs to this company. The preference belongs to the person who said it. But this account is not the only thing that call tells us about. We are targeting a certain kind of company, with a certain kind of problem, and we have a theory about what we should hear if that theory is right. So the same call also answers a question one level up: Are companies like this giving us the signals we expected? Did the overloaded data team show up again? Did the "no more dashboards" objection come up, and from which role? That second layer only exists if the first one was filed properly. You cannot ask "what are companies like this telling us" against a pool of five hundred transcript chunks. You can ask it against observations and preferences that were tagged to accounts, roles and segments at the moment they were captured. If all of this just becomes text sitting in memory, both layers are lost. The system doesn't just need to remember what was said. It needs some understanding of what that information means, and what it belongs to. And I think there is an even earlier piece we usually throw away. At the moment the agent encountered that information, it already knew why. I am working on sales. I am working on this particular company. I am currently trying to understand their problem. This came from a discovery call. That structure already exists when the memory is created. So why throw it away and later ask a retrieval system to reconstruct it? This is where I have started thinking that memory may need to be organised around work, not just conversations. Something like: Type of work → Task → Subtask → Action If I am researching a customer, the things I learn belong to that research. If I am debugging something, the things I learn belong to that debugging. If I am planning a trip, the decisions and preferences belong to that trip. The exact hierarchy will vary. But the principle is simple. When something happens, the system should already know what it belongs to. But even that only solves part of the problem. Because memory also needs to understand change. Suppose a customer initially tells us: Cost is our biggest concern. Three months later they say: Right now speed matters much more than cost. A simple memory system keeps both statements. A retrieval system returns whichever one is more similar to the current question. A recency-based system prefers the second. But what actually happened is more interesting. Their priorities changed. The useful memory is not "cost matters" and "speed matters." It is closer to: Cost used to be the priority. Now speed is. Something changed. That distinction matters everywhere. A company changes CEO. A decision gets reversed. An unresolved issue gets resolved. The system needs to know not only that a new piece of information appeared. It needs to understand whether that information added something, changed something, contradicted something, replaced something, or simply recorded an event. That starts looking very different from search. Every time something new happens, the system almost needs to ask: What am I doing? What did I just learn? What did I already know about this? Does this change anything? What should I believe now? At the same time, we don't want to erase the past. If a company changed CEOs, I want the agent to know who the CEO is today. But if I ask who was running the company when a particular decision was made two years ago, the old information matters. So there are really two things an agent needs to maintain. The current state of the world. And: The history of how we got there. They are not competing versions of the truth. They answer different questions. This is why I increasingly think the conversation about AI memory starts too quickly with infrastructure. Vector databases may be useful. Knowledge graphs may be useful. Relational databases may be useful. Huge context windows may be useful. But those are decisions about how we implement memory. There is another layer before that. The logical system itself. At a very simple level, I imagine it as: we are doing something. Something happens while we are doing it. We understand what happened and what kind of information it created. We figure out whether it changes something we already know. We update our understanding of the present. We preserve the history. And when we do something later, we bring back the parts that matter. Or more simply: Work → Event → Interpretation → Change → State → History → Recall That is very different from: Conversation → Store → Search → Retrieve Retrieval is still important. There will always be information that becomes useful unexpectedly, and the agent won't always know what it will need six months from now. So we absolutely need ways of searching the past. But perhaps retrieval should operate over a memory system rather than being the memory system itself. The other thing this changes for me is where the hard engineering problem actually sits. A lot of the focus today is on the read side. How do we retrieve the right memory at the right time? But maybe the write side deserves just as much attention. At the exact moment something happens, the agent has enormous context. It knows what task it is working on. It knows what led to this point. It knows the people involved, what was just said, what decision is being made, what existed before. That may be the richest point in time for deciding what should actually become memory. If we throw all of that away, save some text, and try to reconstruct everything months later, perhaps we are creating a much harder problem than necessary. Maybe the first question shouldn't be: How do I retrieve this later? Maybe it should be: What exactly happened here, and what did it change? I don't know what the final architecture for AI memory looks like. It will probably involve multiple systems. This is the problem I keep running into while building MindPalace, and the more I work on it, the more I think we should separate the architecture of memory itself from the infrastructure used to implement it. Because when I say I want an AI system to remember, I don't mean: Give it access to every conversation I have ever had. I mean: Know what we have been working on. Remember what we learned. Remember the decisions we made. Know which of those decisions later changed. Know what is unresolved. Understand what used to be true. Understand what is true now. And use that understanding when we continue the work. That feels much closer to what we actually mean by memory. And if that is the goal, then maybe the problem is not how much an AI can store, or how accurately it can retrieve it. The real problem is whether it can maintain an understanding of what happened, what changed, and what matters now.