Dumping your whole site into the agent is making it worse and everyone keeps doing it anyway

Reddit r/AI_Agents News
rag retrieval web-crawling agents vector-store indexing best-practices

Summary

作者批评把整站页面粗暴灌入向量库的做法，主张先爬取发现 URL、再用简单路径过滤大幅缩减索引规模，因为智能体有任务和 token 预算，无法容忍大量噪声内容。

I watched someone point a website crawler API at an entire domain last week, dump 3000 pages into a vector store, and then ask me why retrieval had got worse than when they only had 40 pages in there. It got worse because 3000 pages of website is not 3000 pages of content, it is nav and legal and 400 blog posts from 2019, and an ai web crawler that does not filter is just an expensive way to pour noise into your index. What has worked for me is crawl to discover urls, filter hard on path before I fetch a single page, then pull content only on what survives. and my filter is always embarrassingly simple, one prefix and two exclusions takes 3000 down to roughly 250 that anyone will ever need. The pushback i always get is that you cannot know what matters in advance, so index everything and let retrieval sort it out, which i think is right for search and wrong for agents, because an agent has a task and a token budget and no patience whatsoever. I have built my entire pipeline on this opinion so please come and tell me where it falls apart.
Original Article

Similar Articles

@CycleDecoded: For those building AI Agents and automated scrapers, look no further—this thing is basically a godsend that feeds the entire web to LLMs. Previously, to feed dynamic webpages to AI or scrape data, you had to wrestle with Puppeteer, set up dynamic proxies, deal with JavaScript, and tune API tokens…

X AI KOLs Timeline

Introducing the open-source project Firecrawl: a web data API that converts any URL into clean Markdown/JSON, supports AI interactions and whole-site crawling, designed specifically for LLMs and Agents, with 25k+ GitHub stars.

@laowangbabababa: This is absolutely incredible. Seriously, absolutely incredible. It can truly crawl anything, and it's blazing fast! Previously, I would ask Claude Code to search for something online, but the Twitter API costs money, web scraping requires subscriptions, and Bilibili and Xiaohongshu directly block you. Every platform has its own barrier — such a hassle. Agent-Reach, a free and open-source repository...

X AI KOLs Timeline

Agent-Reach is a free and open-source repository that, with a single command, gives AI agents access to 14 platforms, including web pages, YouTube, RSS, Bilibili, and Xiaohongshu, without complex configuration, significantly lowering the barrier for agents to go online.

@make_claude: I built a WEB CRAWLER that tries to avoid calling LLMs when it doesn't need to. In the video, you can see the bot crawling through a website, opening up new branches, and then progressively narrowing the search. What's truly interesting to me isn't the crawler itself, but what happens between each step...

X AI KOLs Timeline

The author introduces a web crawler that minimizes LLM calls: by placing a small decision layer (Jev) in the middle of the Agent pipeline, it decides whether a link hop is worth making, only invoking Claude when full context and reasoning are truly required — significantly reducing the cost of using an LLM as a general-purpose if statement.

we used a file index instead of a vector database for agent retrieval. here's the tradeoff.

Reddit r/AI_Agents

作者分享用 INDEX.md 文件索引替代向量数据库做 agent 文档检索的实践：由 summarizer 生成每行描述，doc-searcher 子代理挑选并读取文件，把中间读取隔离在子代理上下文中。作者指出这节省的是上下文而非总 token，小规模文档集可行，但上千文档时索引本身会变重。