No .md files until Series B

A few weeks ago I was talking with a founder of an SF startup building vertical agents. They had one customer in production, and the agent’s memory for that single customer was already** 1,000+ markdown files**. Cracked engineers, shipping fast, but they hadn’t really thought about how to scale the architecture as the company grew. No blame; we have all been there. You write hacky code to hit PMF, then you rewrite it properly.

Long story short, after a couple more customers the workarounds started piling up. A YAML schema for permissions. A script to enforce it at retrieval. A merge resolver for when two agents wrote to the same file. A Postgres table for traces. A graph of cross-references between files, maintained by hand. Each one made sense on its own, and together they had quietly become the system slowing down every change to the actual product.

So they redesigned the whole memory layer, this time to build a moat around their workflows.

I have seen plenty of founders end up on this same path.

If you are building agents you expect to run in production, across teams, across time, you will hit it too. And if you are already there, you have probably started patching around it without realizing what you are rebuilding.

I have had this conversation enough times to know how it goes. Here are the five questions I get most often, and the answers I usually hear back.

Q1. How do you do permissions with memory.md?

A1: One .md file per topic, each with filesystem ACLs.

Permissions in real systems aren’t per-topic. They’re per-entity and per-attribute. User A can see customer X’s contact info but not their contract terms. You’d need a file per (entity, sensitivity-level) pair. That combinatorially explodes.

A2: Encode permissions as frontmatter metadata, filter at retrieval.

Now your retrieval layer parses, respects, and enforces ACLs before the LLM sees content. That’s an access-control system embedded in  a file store. Get it wrong once and you have leaked data into an LLM context window. And even worse,  “sleep” refinement merges content across files. Now your refiner needs to understand permissions too, or it cross-contaminates sensitive content into shared summaries.

A3: Separate memory store per permission boundary.

You have lost the cross-cutting reasoning that made memory valuable. The agent can’t answer “across everything I’m allowed to see, what patterns emerge?” because you have physically partitioned the knowledge.

Q2. How do multiple agents interact with memory.md?

A1: File locking: only one agent writes at a time.

You have serialized your entire memory layer. Latency scales with agent count. And locks don’t solve semantic conflicts: when two agents write about the same customer, you don’t get a merge. You get the last write winning, or two contradictory paragraphs sitting next to each other.

A2: Append-only logs per agent, with a periodic merge job (basically claude’s “dreams”at scale).

Merge is now fully non-deterministic. And even if you trust the LLM to merge correctly, how do you version it? Which merge produced which memory? What did it overwrite?

A3: One shared md, let the LLM resolve conflicts at read time.

If the LLM resolves conflicts at read time, you don’t have a memory. You have a log and an LLM guessing what to believe. Every query pays the cost of re-deriving the truth from scratch, and every query can derive it differently.

Q3. How do you handle temporal queries?

A1: Timestamp every entry.

A timestamp tells you when a note was written, and not when the fact was true in the world. A customer telling you in March about an October renewal, and in June pushing it to December, gives you two notes with two written-at timestamps and no way to know which renewal date was current on any given day.

A2: Use Auto Dream to maintain a “current state” section and an “archive” section.

You have built a snapshot system that quietly corrupts itself. Every merge pass, the LLM has to guess which old note each new note replaces, with nothing telling it that “customer’s renewal” and “their renewal date” are the same thing. It will guess wrong. You won’t know when. And the archive is too unstructured to go back and check what you used to believe.

A3: Include all dates in the .md and let the LLM reason temporally.

You have turned a storage problem into an inference problem. Every time-based question now costs a model call, returns a different answer each run, and has no ground truth to test against.

Q4. How do you include traces in your memory so they can be used to self-improve your agents later?

A1: Append a “lessons learned” section after each session.

Lessons accumulate, and md has no notion of which ones are still valid. After a hundred sessions you have a hundred lessons. Some might be contradicting each other, or some might have code paths that you have deleted.They all sit at the same retrieval priority, and the agent has no signal for which to weigh. You have built a write-only log and called it learning.

A2: Use Auto Dream to distill traces into rules.

Distillation is one-directional. Once a rule is written, you can’t query “which traces produced this rule, and what were the outcomes when it got executed?”. The traces that would answer that have been compressed away. You have made inference cheaper and evaluation impossible.

A3: Keep raw traces in a separate store and the distilled rules in md.

You now have two memory systems with no link between them. The rule in md can’t point back to the traces that produced it. The traces can’t be queried for “which produced rules that are still active.” You’re back to needing a relational or graph model.

Q5.. How are you going to find the right information across thousands of md files?

A1: Chunk and embed, retrieve top-k.

You have built a vector store on top of a file store, and retrieval ignores the file structure entirely. Chunks lose entity context too;  a paragraph about “their pricing” comes back without telling you whose. Top-k optimizes for semantic similarity, not for the entity the query is actually about.

A2: Add metadata filtering (tags for entity, topic, date)

Now you are maintaining a schema md doesn’t enforce. Tags drift, taxonomies fight reality, one mis-tagged file silently degrades retrieval forever. You have built a database without the constraints that make databases trustworthy.

A3: Build a graph-based navigation system across thousands of md files.

At this point you are building a graph database where the storage layer happens to be markdown files. You will need an index (md isn’t queryable), a way to keep that index in sync as files change (md doesn’t emit events), typed edges (links in md are just strings), and a query engine on top. Every one of those is something graph databases and memory layer providers already solve. You have reinvented the system and made the storage layer worse on purpose.

Using .md files for memory is not wrong or bad, do not take me wrong. It is just a key-value store with LLM-driven compaction, and for one user, one agent, one timeline, that is probably fine. But the moment your system grows along any of those axes, you need something else.

And, trust me, we have only scratched the surface above. What about provenance? How do you separate short-term from long-term memory without losing the link between them? What happens when the shape of what you remember changes six months in? How do you actually delete something, when Dream has already absorbed and rephrased it into a dozen summaries? How do you do multi-hop reasoning over notes that don’t know they are connected? Each one is its own a post :)

These are all the questions we had in mind when designing cognee.

Check us out here:

https://github.com/topoteretes/cognee** ⭐**

https://www.cognee.ai/

Join us on Discord for support and discussion: