I built agent memory the textbook way (agent retrieves on demand). Watching it run made me invert the whole design. Architecture + the failure mode that scared me off write-back.

My first version of agent memory was the textbook approach: give the agent retrieval tools, let it decide when to fetch. Sounded right. In practice it had two failure modes. It either spent a tool call fetching memory on every trivial turn (latency for nothing), or it skipped retrieval and answered confidently from an empty context (much worse). Letting the model decide when it needs memory assumes it knows what it doesn't know. It doesn't. So I inverted it to inject-first. The architecture: * **Storage:** each turn is a graph node tagged `{role, conversation_id, kind}`. Per conversation there's a meta summary record and a user-preferences record, both as structured data, so memory is addressable state I can inspect, not an opaque vector blob. * **Prompt assembly:** before inference, the layer builds a block with a distilled user summary, standing preferences, and the last N raw turns, in tagged sections (`<start_of_user_context>...`, `<start_of_previous_messages>...`). The agent starts oriented with zero round-trip on the common path. * **Bounded growth:** a background job keeps a rolling summary. While it's under a budget (\~20k chars in my config) it accumulates; past that, an updater agent compresses older turns into observation paragraphs so the prompt never balloons. This is the practical fix for long-context degradation: distill, don't dump. * **Depth on demand:** it still gets tools for multi-hop traversal over a knowledge graph of the user's data, reserved for when the injected summary isn't enough. So it's pin-the-essentials, fetch-the-long-tail, except the pinned part is a generated summary of the user and the long tail is a real graph you can walk. Net effect: far fewer tool calls, and it stops answering from nothing because the basics are always present. **The failure mode that scared me (the gotcha):** I let the updater write new edges back into the graph from the conversation itself. Within a day it had invented relationships (two people connected purely because their names co-occurred near a keyword) and asserted a "decision" I had explicitly argued *against*, because the words appeared in a thread where I was rejecting them. Then it read its own bad edges back as ground truth, so the error compounded every turn. Self-poisoning memory is strictly worse than no memory. I disabled write-back, the agent now reads a graph I curate and doesn't get to rewrite it from chatter. Turning unstructured conversation into reliable graph structure is the genuinely open problem, and it's what the RL-traversal work (Graph-R1) is chasing. Two things I'd actually like input on: how do you decide an extracted edge is trustworthy enough to persist (confidence threshold, human review queue, nothing)? And how are you splitting inject-vs-retrieve in your own agents?