I'm building an ambient memory agent that watches my screen all day, and its memory lives in SQLite instead of the model. Come tell me where it breaks.
The author describes building an ambient memory agent that runs locally on a Mac, watching the screen and optionally audio, OCRs/transcribes everything into structured Markdown in an Obsidian vault, with SQLite as the source of truth. They invite feedback on design choices like using SQLite over vector/graph DB, cost, and durability of such systems.
Every "second brain" I've set up turns into a graveyard. Thousands of notes, gorgeous graph view, not one decision I actually made better. The problem was always me. I have to stop and write the thing down, and I never do. So I'm building something that grabs it whether I bother or not. Runs on my Mac, watches what's on my screen (and optionally mic + system audio), OCRs/transcribes all of it, and turns it into structured Markdown in an Obsidian vault. The whole point is catching the stuff I'd never jot down myself: the little decisions I make without noticing, the context that's gone by tomorrow morning. Kind of like what Rewind used to do before Meta bought it, except instead of a search box you get a wiki that maintains itself, and nothing leaves my machine unless I plug in an API key I own. Here's the stuff I actually want you to roast: The database is the source of truth, the Markdown is throwaway. All the real state (identity, dedup, search, cost tracking, the redaction guard) lives in SQLite. The Obsidian vault, meaning daily notes, a timeline narrative, and wiki pages per person/project/topic, is just a projection I regenerate from the DB. Nuke the vault and I can rebuild it. The model never owns memory, it just reasons over whatever the store hands it. Went this way after watching too many agents "remember" stuff from chat history and completely fall apart across sessions. Redaction is fail-closed. Screen text gets scrubbed for secrets before anything goes to an LLM, and every send is logged to an append-only file. If the scrubber can't run, nothing goes out. Period. Bring your own key. Anthropic, OpenAI, OpenRouter, or fully local with Ollama. No accounts, no managed cloud. Every file has "hands off" and "go nuts" zones. The LLM only rewrites its own sections between markers. Anything I type outside those stays exactly how I left it, so we're not constantly clobbering each other. Where I'm honestly not sure: Is "DB is truth, Markdown is a cache" smart, or am I gonna wish I'd used a vector/graph DB once I want real retrieval? Right now it's just SQLite + full text search, no embeddings. Always-on capture spits out a mountain of near-identical text. I collapse the near-dupes before the LLM sees them, but I'm definitely throwing away signal somewhere. Cost gets scary fast over a full day of capture. How are you keeping always-on agents cheap? More local-model triage before you call the expensive one? The real one: what ever made a memory system actually stick for you past month two instead of going write-only? Building it solo, happy to go deep on any part in the comments. Mostly just want to hear where the design is naive.
The author reflects on the limitations of using flat markdown files for long-term agent memory, which leads to prompt debt as the memory grows, and advocates for graph-based memory representations that retrieve relevant context dynamically.
A comprehensive review of local-first AI agent memory systems, comparing options like mem0, Hindsight, and mnem, ultimately recommending Engram for its unique combination of local storage and human-readable Markdown files.
The author introduces Noosphere, an open-source memory and wiki layer for AI agents and humans, seeking feedback on its structured Postgres-backed memory, Redis-cached recall, and human-editable wiki pages.
Hermes Agent v0.14 integrates with Obsidian vault to provide persistent, inspectable memory for AI agents, using plain markdown files instead of vector databases.