Every news-reading agent I built refetched the same articles forever. Here is the memory layer I settled on.

Reddit r/AI_Agents Tools

Summary

The author describes building a memory layer for news-reading AI agents to handle deduplication and improve continuity across sessions, using SQLite and vector storage with a Python API, CLI, and MCP server.

Every agent I built that touched news had the same hole in it. It calls a news source, gets a set of articles, uses them, forgets. Next run it refetches, gets a slightly different set back, and has no idea it already saw two thirds of them. There was never a "what have I already read about this company" and that turned out to be the thing I actually wanted. So I built the memory layer instead of writing that glue for a fifth time. The design decisions were less obvious than I expected, so here they are in case anyone else is stuck on the same thing. Dedup is the entire problem. Google News hands back four or five URL variants for one article. Matching on URL fails on all of them. What worked was hashing the normalised title together with the normalised publisher. Note that Reuters and BBC covering the same event stay as two separate rows on purpose. Two outlets carrying a story is information, and collapsing them throws it away. Title normalisation also has to handle unicode apostrophe variants or you get "duplicates" that look identical to a human and different to a hash. Store the embedding model on every row. Learned this the annoying way. Swap your embedding model six months in and the old vectors are silently in a different space, and retrieval just gets quietly worse with no error anywhere. Now every row carries the model name and dimension, so a mismatch fails loudly instead. Recency belongs in ranking, not filtering. First version filtered to the last N days, which meant an agent asking about a company could not see the thing from five weeks ago that explained everything happening now. Now similarity gets blended with an exponential decay on age, three day half life. Old but very relevant still surfaces. Same operations everywhere. ingest, search, timeline, brief, sentiment, stats, all exposed identically through a Python API, a CLI, and an MCP server. The MCP part changed my usage more than any prompt engineering did. Handing Claude a news memory it can query across sessions is a different experience from pasting articles into context every time. The thing I have not solved: article revisions. Publishers rewrite headlines and bodies within the first hour, and my store keeps whatever version it caught first with no record that anything changed. Someone pointed this out to me last week and I have not found a clean answer for agent memory specifically. If you have handled it, I would like to hear how. Storage is SQLite plus a vector store, all local. The LLM steps are optional and point at whatever provider you want including Ollama, so the memory itself runs with no keys at all. It is open source. Putting the link in the comments per rule 3.
Original Article

Similar Articles

rohitg00/agentmemory

GitHub Trending (daily)

agentmemory is an open-source persistent memory layer for AI coding agents (Claude Code, Cursor, Gemini CLI, Codex CLI, etc.) that uses knowledge graphs, confidence scoring, and hybrid search to give agents long-term memory across sessions via MCP, hooks, or REST API. Built on the iii engine, it requires no external databases and exposes 51 MCP tools.