Build a Knowledge Brain Like Karpathy in 30 Minutes — Let LLM Maintain Your Notes

On April 2nd, Karpathy posted a thread about how he’s been using LLM to build a personal knowledge base. No launch event, no new model — just a 600-word plain text thread. Result: 21 million views, 108k bookmarks.

Lex Fridman commented that he’s been running this setup for a year, even using it as an interactive podcast while running; Obsidian’s CEO jumped in to discuss how to prevent AI from polluting personal note vaults; HubSpot founder Dharmesh directly said he wants to buy secondbrain.com for a cloud version.

Two days later, Karpathy organized the method into a gist named llm-wiki. 5000+ stars, 5000+ forks. Over the next month, HN argued about it twice, 7-8 open-source implementations popped up, Obsidian got a companion plugin, the Chinese tech circle was flooded with “second brain” tutorials, and course sellers and product builders all jumped on the bandwagon. By the end of June, 36Kr was still publishing interpretations.

I read through everything I could find from this wave: his original thread and gist, hundreds of replies, two HN debates, production environment test reports in the gist comments, plus a dozen tutorials in both Chinese and English. My takeaway: the parts worth copying are over-mystified, while the real pitfalls are almost never mentioned. This article clarifies both sides: what it actually is, how to set it up step by step, and when it becomes a trap.

1. What Exactly Is This Viral Thing?

Let’s pin down the definition first, because almost half the tutorials online explain it wrong — some even confuse it with Karpathy’s other project, llm.c, and teach you to compile C code with great seriousness.

llm-wiki describes something very simple: you collect raw material, LLM “compiles” it into a continuously maintained wiki, and you only read and never write.

His original words break it into six steps:

• Collect. Articles, papers, codebases, images — all dumped into a raw/ directory. Web pages use Obsidian Web Clipper to convert to Markdown in one click, images downloaded locally with a shortcut.

• Compile. LLM incrementally processes the raw/ content into a bunch of interconnected .md files: a summary page for each source, a separate page for each concept, pages linked bidirectionally, index auto-maintained.

• View. Obsidian is used only as a browser — view the graph, click links, read pages. His analogy: Obsidian is the IDE, LLM is the programmer, wiki is the codebase.

• Ask. Once the library reaches a certain scale (his own: 100 sources, 400k words), ask complex questions directly against the wiki. LLM reads the index first, drills into specific pages, and synthesizes the answer.

• Archive. Answers can be Markdown, Marp slides, matplotlib charts. Good answers get saved back into the wiki as new pages. Every question you ask also adds brick to the knowledge base.

• Health Check. Periodically let LLM scan the entire library: which pages contradict each other, which conclusions have been overturned by new sources, which concepts are mentioned without their own pages.

One sentence many retellings missed: “You rarely ever write or edit the wiki manually, it’s the domain of the LLM.” The wiki belongs to LLM. Humans do not touch it. Your job is only two things: decide what to read, and ask good questions.

There’s also a background that helps understand why he designed it this way. As early as March 2025, Karpathy wrote a blog post about his note-taking method called “append-and-review”: a single Apple Notes text note used for years, any thought appended to the top, no tags, no folders, every so often scroll down, keep what’s worth it by copying back to the top, let the rest sink. He called multiple notes and folders “cognitive burden” — one note with Ctrl+F is enough.

Put the two together, his knowledge system is actually two layers. The capture layer is laughably bad: one crappy text file. The compilation layer is absurdly heavy: an LLM-managed wiki. This combination itself is a judgment: any ritual in the capture step is a waste; don’t do a single structural task yourself. Most note-taking courses do the exact opposite — they teach you to tag, template, process during capture, then maintain structure manually. Wrong direction entirely.

2. Why He Dares to Say “No RAG”

The most provocative line in that thread: “I thought I’d need RAG, but turns out I don’t.”

For the past two years, the standard answer for “personal knowledge base + AI” has been RAG: chunk documents, build vector index, retrieve similar segments when asking, feed to the model. NotebookLM, ChatGPT file uploads, “chat with your documents” products — all this architecture.

The problem with RAG can be explained with an analogy. You write a piece of code; you don’t have the machine re-read the source every time you run it. You compile it once, then use the compiled artifact for every run. RAG is like “interpreting” all your notes every time you ask: retrieve, piece together, answer, forget. Ask 100 questions, your knowledge base is still that pile of raw files — it never grows.

The compile model works the opposite way. Each time a new source is added, LLM integrates it into the existing structure: update related pages, revise summaries, highlight where new data conflicts with old conclusions. By the time you ask a question, cross-references are already built, contradictions already flagged. The core sentence from the gist is worth memorizing: “the wiki is a persistent, compounding artifact.”

In daily use, RAG has three incurable diseases that the compile model fully solves:

• RAG only sees local context. It can tell you that note #5 mentions A, but cannot tell you what all 500 notes collectively point to, because “collective pointing” doesn’t exist in any single text chunk — it exists in the relationships between chunks. The wiki’s concept pages are specifically designed for this.

• RAG is schizophrenic. Your note from six months ago says A is correct; last month’s note overturns A. Retrieval will grab both chunks and stitch them together, logic contradictory. The compile model flags the contradiction the moment the new source comes in.

• RAG has no maintenance concept. Vector indexes won’t tell you which knowledge is outdated. The wiki has lint — outdatedness gets caught.

But I have to be thorough here, because this is the part the hype train collectively omits: “No RAG” is a conclusion with a scope — specifically the scope Karpathy himself reported: 400k words. Someone on HN did the math: at this scale, LLM navigating via the index file is completely sufficient. At 4 million words, the same approach breaks, and you’ll eventually have to go back to retrieval.

A team that ran this in production for 6 months with 4000+ cross-linked concepts confirmed the line: flat index works up to a few hundred pages; after that, they added local hybrid search. A more accurate statement: for personal scale, no RAG needed; this scale covers 90% of people, but it has a boundary. Don’t take a personal solution to an enterprise job.

3. Three Layers, Three Operations — Architecture Explained in 10 Minutes

The gist abstracts the system into three layers.

raw/ layer: raw material, immutable. All original sources live here. LLM reads only, never writes. This is the foundation: every conclusion in the wiki must be traceable back to a file in raw/. Later when we talk about the hallucination problem, you’ll see why this “immutable” rule is the only safety belt.

wiki/ layer: compiled output, fully managed by LLM. Summary pages, concept pages, entity pages, comparison pages, plus two special files: index.md is a content table — one line per page with a link and a one-sentence summary, LLM reads it first to find its way before answering; log.md is an append-only operation log — each ingestion, each health check is recorded, so the next session knows what was done.

schema layer: one CLAUDE.md (or AGENTS.md) — the soul of the entire system. It clearly states directory structure, page format, naming conventions, and the steps for each of the three operations. Karpathy’s original words: this file determines whether the LLM is a “disciplined wiki administrator” or a generic chatbot. The team that ran it for 6 months in production put it even more bluntly: the schema file is the single most important file in the entire repository. I fully agree, and I’ll add one sentence: eight out of ten failures for beginners happen because they didn’t write the schema, or wrote it too vaguely. Without a schema, every LLM compile produces a different format; pages that should be merged aren’t merged, pages that should be split aren’t split. A month later, you have a pile of AI-scented fragments, then you abandon it, concluding “this method doesn’t work.” The method is fine — you skipped the most important step.

There are three daily operations, each with a corresponding prompt:

• Ingest: drop a new source into raw/, say “compile it into the wiki.” LLM reads the original, writes a summary page, updates all related pages, updates index, logs. It’s common for one source to touch a dozen pages.

• Query: ask a direct question. LLM reads index.md to locate, then reads 2-3 relevant pages, synthesizes an answer with citations. Good answers are saved back into the wiki.

• Lint: periodically run “scan the entire library for contradictions, outdated conclusions, orphan pages, missing concepts” — output a health check report.

One easily overlooked operation detail comes from Karpathy’s own answer in the replies. Someone asked about incremental compilation and batch size. He said: there’s no automation right now; I add each source manually, one at a time. I’m present the whole time, especially early on. Once LLM learns the pattern, the cost per later source drops dramatically, to the point where “file this new doc to our wiki” is enough. This line is more important than the gist body.

Every tutorial on the web teaches you “batch compile 50 files with one prompt,” while the inventor himself feeds them one by one. The time saved by batch ingestion comes back double in the form of page quality loss. The first 10 sources must be done one at a time — correct the format and detail as you go, and write those corrections back into CLAUDE.md. You’re training your administrator; only after training can you scale.

4. Step-by-Step: Build a Minimum Viable Version

Now that the theory is clear, let’s get hands-on. Following this section exactly takes 30 minutes. The cost is your existing subscription (use your existing Claude subscription). Prerequisite: install Claude Code (Codex, OpenCode, WorkBuddy, any similar tool works — the process is the same).

Step 1: Create directory structure.

my-brain/
├── raw/          # Raw materials, input only, never modify
├── wiki/         # LLM's territory
│   ├── index.md  # Empty file
│   └── log.md    # Empty file
└── CLAUDE.md     # Write this now

Also run git init on the spot. The wiki is just text files — Git gives you version history for free. If a compile goes wrong, just roll back. Log.md is one layer of audit; Git is another.

Step 2: Write CLAUDE.md.

This template synthesizes the gist original, several production testers’ versions, and my own edits. You can copy it directly, then change the topic to yours:

# Knowledge Base: [Your Topic]

## Structure
- raw/: Raw materials, never modify any file here
- wiki/: Markdown pages you generate and maintain
- wiki/index.md: Master table of contents, update after each operation
- wiki/log.md: Append-only operation log

## Page Rules
Every wiki page must have frontmatter:
---
title: Page Title
type: concept | entity | source-summary | comparison
sources: [paths to source files in raw/]
related: ["[[Related Pages]]"]
updated: YYYY-MM-DD
confidence: high | medium | low
---
- File names use lowercase-hyphen (attention-mechanism.md)
- Internal references always use [[wikilink]]
- Every conclusion must trace back to a source in raw/; no unattributed conclusions

## Ingest Workflow
1. Read the specified new file in raw/
2. First, report key points to me; wait for my confirmation before writing
3. Write a summary page in wiki/, update or create related concept pages
4. Update index.md, append to log.md

## Query Workflow
1. Read index.md to find relevant pages, read 2-3 pages before answering
2. In the answer, cite [[Page]], and give raw/ path when tracing back to original
3. If the answer has long-term value, ask if I want to save it as a new page

## Lint Workflow
Check for: contradictions between pages / old conclusions overturned by new sources / orphan pages with no incoming links / concepts mentioned but without pages. Output a report. Do not delete anything without my approval.

## Hard Rules
- NEVER delete any file
- NEVER modify raw/
- New page vs update: if this thing will be linked to by other pages, create a new page; if it's just supplemental info for an existing page, update in place

The last heuristic about “new page vs update” comes from production testing: once the schema clearly lists page types, the model gets it right in 90% of cases. Don’t delete the NEVER rules either — the most dangerous thing about AI tools is their over-enthusiasm.

Step 3: Feed the first three sources.

Pick three articles on the same topic that you’ve actually read. Drop them into raw/ (Obsidian Web Clipper is easiest for web pages). Then, in the directory, launch Claude Code with:

Ingest the three articles in raw/ into the wiki one by one, strictly following the CLAUDE.md workflow.
Process one article and show me the result; I'll confirm before you proceed to the next.

After the first one comes out, check three things: page granularity (one page per concept — if it’s mixed, tell it to split), link density (do concept pages reference each other?), detail level (summary pages should be much shorter than the original; concept pages should have cross-source synthesis). If you’re unsatisfied, say so directly, and have it write the improvement rules back into CLAUDE.md.

Step 4: Open wiki/ with Obsidian and take a look. Install Obsidian, “Open folder as vault” — select the wiki directory, press Cmd+G to open the graph. Three articles will likely produce 10-20 nodes. The value of this step is psychological: the graph grows visibly as you feed materials, and that feedback is the fuel that keeps you going.

Step 5: Ask a question you already know the answer to. For example, “Is there any conflict between these three articles on concept X?” Check if the pages it cites are correct and if the conclusions are fabricated. This is your acceptance test.

At this point, the minimum viable version is working. Two things NOT to do now: (1) Don’t install plugins — Dataview, Templater, etc., wait until the library hits 100 pages. Toolchain complexity will kill your motivation before you’ve built the habit. (2) Don’t dump all 500 of your old bookmarks at once — the reason was explained in the previous section.

5. Advanced: From Toy to System

After using the minimum version smoothly for a week or two (note: smoothly — adding features to a system that’s not working well is like putting sails on a leaky boat), add these components. Each comes from someone who hit a real pitfall and corrected it.

• Token three-tier budget. As the library grows, cost becomes the first thing to spiral, and the biggest money goes to an unexpected place: repeated reading. One person building a course knowledge base measured that the main token consumption is the model re-reading AGENTS.md, index, and original files again and again. The reasoning itself is cheap. Solution: hardcode a three-tier budget in CLAUDE.md — when answering, first read index.md (few KB, light), then 2-3 relevant wiki pages (medium), only touch raw/ if truly necessary (heavy). Prohibit “read the entire library before answering.”

• Mandatory dual output to prevent knowledge evaporation. Every query should require the model to output two things: the answer itself, plus one wiki update (even just appending two lines to a concept page). Write this rule into CLAUDE.md; do not allow choosing only one. Without this constraint, your most valuable thoughts evaporate into chat logs, and the knowledge base degenerates into a pile of files with no output. Compounding comes from backflow, and backflow requires enforcement.

• Model tiering to save money. Ingest is repetitive grunt work that doesn’t require high reasoning ability — use cheap models (Haiku, DeepSeek, local Ollama). Query, especially cross-page synthesis, is worth using a strong model. Also set a rule for single file: anything over 50KB, only extract the summary, don’t compile the full text. In production tests, a medium project’s full initialization costs under $1 USD; daily updates are 5-15 cents each; serious monthly use tops at $10-20. If you’re spending more, you probably skipped tiering.

• Source file granularity determines compile quality. Someone ran a controlled experiment with three business books (155k words). Feeding each entire book as one file produced typical AI sludge. The same book split by chapter into 68 files, then compiled, produced 210 concept pages, 4600 cross-references. The model spontaneously synthesized across the three books and even found a viewpoint contradiction between two books that neither explicitly stated. The model and prompt were identical — the only variable was split granularity. Before ingesting a long source, split it: book by chapter, long report by section, podcast transcript by topic segment. This 10-minute manual effort is the highest ROI step in the entire workflow.

• Lint on a timer — no negotiation. The 4000-concept production team reported the biggest failure mode called “drift”: when ingesting, cross-reference updates were missed; pages quietly became outdated; errors started being cited by other pages; your library became “an organized misinformation base.” Their conclusion: lint must be scheduled. For personal use, don’t need to be that heavy — run it every 20 new pages, or fixed once a month, with:

  Audit the entire wiki/: find contradictory pages, conclusions overturned by new sources,
  orphan pages with no incoming links, unattributed claims.
  Output a report, and suggest 3 concepts worth creating new pages for.
  

  The last half is a bonus: lint also helps you discover what to read next.

• Two vaults isolated to protect your own voice. This comes from Obsidian CEO: AI-generated content and your own handwritten notes should be in two separate vaults. His reasoning is worth reading verbatim: the value of a personal note vault lies in high signal-to-noise, known provenance. Once mixed with AI products, search, backlinks, and the graph no longer represent “your” thoughts. Useful things from the AI vault, manually pick and move to the main vault. The friction of this step is deliberately preserved.

• Search tool — you probably don’t need it. Shopify CEO’s qmd is the standard search engine in this ecosystem: local, hybrid retrieval. But the consensus from production testing: under 100 pages, the model’s own grep plus reading the index is completely sufficient. Add search only after a few hundred pages. Those who start by configuring search, MCP, automation hooks are mostly using the thrill of building a system as a substitute for the boredom of using it.

6. When NOT to Build — The Opposition Deserves a Full Chapter

The most nutritious content from this viral wave is all in the opposition. Two HN discussions totaled 130 comments; the gist comments also contained controlled experiments with data. I pick the truly solid ones, along with my position.

Knife #1: For small scale, this is wasted effort, with experimental data. Someone built a 28-page wiki for their small codebase, then ran an A/B test: same four questions, one group had the agent grep code directly, the other looked up the wiki. Token consumption was nearly identical (27.3k vs 27.6k), correctness was perfect for both. The heartbreaking reason: in a repo where one concept maps to one file of 100 lines, the entity page is longer than the source file itself — compression ratio <= 1. Worse, the page says “derived content, verify against source,” and the agent reads both wiki and source, doubling cost. Their fix was to collapse 28 pages into one dense single-page map. Query cost immediately dropped to half of grep. Their boundary condition is, in my opinion, the most valuable sentence in the entire discussion: “This model only makes money if one page compresses facts scattered across multiple sources. Mirroring something you can see at a glance is negative value.” They also had another sharp line: “Either trust the wiki unconditionally when querying, or it shouldn’t exist.”

First judgment: If you don’t have 20-30+ sources that you reference repeatedly, don’t build. For a single source, having the LLM read it directly is the optimal solution. The entire value of a knowledge base lies in cross-file relationships. Not enough files, no relationships.

Knife #2: Outsource organization, outsource understanding. This was the loudest opposition on HN, from an Obsidian veteran with 4100 hand-written notes. Their argument: archiving, linking, summarizing — these “chores” are exactly when new ideas emerge. When you create a link between two notes, you’re forced to think about their relationship. That forcing is the learning itself. An AI-built knowledge base has “nothing personal about it — it’s AI’s database; you’re just asking questions on the side.” Another user in the same thread added an analogy: using AI to generate memory cards is absurd because making the cards is the learning; reviewing the cards is just consolidation. Even more convincing was a real user’s testimony: after burning out and heavily relying on the wiki workflow, they found themselves accumulating a sort of “continuous mental deficit” — more deliverables, thinner understanding, and the workflow was addictive to the point they couldn’t stop.

My position: This knife cuts halfway. It’s largely irrelevant for research-style use (reading papers, due diligence, tracking a domain). In those scenarios, Karpathy’s division of labor holds: you still read the original, you still make judgments. LLM only takes over the “accounting” that you never could sustain anyway. But it’s fatal for learning-style use. If you want to understand something, don’t use this system. Write your own notes, the old-fashioned way. Use this system only to manage things you already understand. Those who can’t distinguish these two scenarios will end up with a beautiful wiki and an increasingly empty head.

Knife #3: Productivity pornography. A colder observation: the knowledge base category perfectly simulates the feeling of “harvesting” but doesn’t produce output. Collecting ≠ owning. Highlighting ≠ understanding. Similarly, an AI-tidied wiki is not the same as having produced anything. Someone gave the ultimate example, using Karpathy himself: his 400k-word wiki is only seen by himself, while his thread had 21 million viewers. The world has always quoted his output, not his warehouse.

I accept this entirely, and I consider it the last piece of the puzzle for the whole method: the knowledge base is middleware; the endpoint must be output. Build it into the workflow — after each lint, also ask “based on the current library, what are the three most worth writing topics?” Let the library directly spit out production tasks. Use “how many things did this library help you produce this month” to measure its health, not a graph.

Knife #4: A satirical warning. Later, the gist comments turned into a large-scale crash scene. Hundreds of people fed this document directly to their agent: “implement this gist, make no mistakes.” Then they came back to post their implementation as an ad. The comments section was drowned in AI-generated marketing copy. Someone commented that the comment section itself was the best example of what not to do. Even Karpathy himself posted a rebuttal written by Claude on HN, got downvoted, and deleted it. The lesson is tool-agnostic: the only step in this system that cannot be outsourced is judgment. Once you let AI decide “what to put in the library” and “whether to trust its output,” you become one of the slop accounts in that comment section.

7. Three Starting Routes for Different People

Finally, collapsing by audience — three routes with increasing difficulty.

Route 1: Non-technical background, just want a taste. Don’t touch the terminal. Go to Claude.ai web, upload 5 PDFs on the same topic, one prompt: “Generate a Markdown page for each key concept in these documents, with summary, [[wikilink]] cross-references, and highlight contradictions between documents.” Save the generated pages manually into a local folder, open with Obsidian, view the graph. Experience the core of the compile model in half an hour, then decide if you want the full setup.

Route 2: Comfortable with command line, personal knowledge management. Follow the complete workflow from sections 4 and 5: three-layer directory, CLAUDE.md, ingest one by one, monthly lint. First two weeks: only ingestion and query. Third week: add dual output and model tiering. The sign you’re on the right track: one day you ask a complex question, and it cites three pages from sources you fed at different times, and you yourself hadn’t realized that connection. At that point, compounding has begun.

Route 3: Engineer, want to integrate into workflow. On top of route 2, add automation: SessionStart hook to automatically inject the first 60 lines of index and last 15 lines of log into each session; post-commit hook to trigger wiki incremental update on code commit, with --max-budget-usd 0.5 as a fuse; scheduled lint every Sunday. Running six projects this way costs $10-20/month in API. But remember the knife from section 6: first confirm that your project’s knowledge is truly scattered across multiple sources (meeting notes, old PR discussions, deployment verbal conventions). If everything is in the code, grep is your knowledge base — don’t build.

Writing to this point, I’ll end with the sentence I consider the most undervalued from Karpathy’s two pieces.

He said: the most exhausting part of maintaining a knowledge base has always been the accounting. People quit because maintenance cost always grows faster than value. Eighty years ago, Memex died on this. Now LLM pushes maintenance cost to near zero.

Reading that sentence backward is the key: if the machine takes over accounting, you no longer have any excuse to treat “organizing” as output. Screening what’s worth reading, asking questions no one else asks, turning understanding into published work — these three things have never been done by a tool, and they still aren’t today.

No matter how beautiful your knowledge brain is, it’s just a warehouse. And warehouses don’t produce grain.

About the Author

Kyrie — Former R&D engineer at a large Chinese tech company, now based in Bangkok, doing BD for Chinese tech companies going overseas. Continually sharing real frontline records of overseas expansion, practical uses of AI in business, and occasionally stock investing and life abroad.

• X: @KyrieCheungYep