Would agents use a website's own semantic search endpoint?

Reddit r/AI_Agents Tools

Summary

The article discusses a prototype open-source package called Agentize that exposes semantic search endpoints for AI agents on websites, and questions whether this approach solves retrieval problems or duplicates existing methods.

When an agent needs an answer from a website, it often searches, fetches one or more pages, strips the HTML, and guesses which page is authoritative. I've been prototyping a different approach in a small open-source package called Agentize. The site owner chooses the public content, builds a semantic index locally, and exposes search plus Markdown resources under /agents/*. Results keep canonical URLs so an agent can still inspect the source. I built the package, so I have a stake in the idea. I'm leaving the link out of this post because I want to test the premise rather than drive installs. Would you teach an agent to check for an endpoint like this before crawling? The weaknesses I see are publisher bias, incomplete indexes, and the need for agents to verify claims independently. Adoption also seems hard unless agent runtimes agree on discovery. Does this solve a real retrieval problem, or duplicate llms.txt, MCP, and normal browsing without enough benefit? I would especially value objections from anyone building browsing or research agents.
Original Article

Similar Articles

Is traditional SEO slowly dying because of AI agents?

Reddit r/AI_Agents

Discussion on how AI agents like ChatGPT, Perplexity, and Claude are shifting user search behavior away from traditional Google searches, potentially making SEO less about ranking articles and more about brand authority and structured data.