Better semantic search will not fix an agent that never verifies context

A common answer to weak agent memory is better semantic search or a larger context window. I disagree. More recall does not make an agent more reliable if it never checks what it retrieved. I initially treated the memory layer as a context-loading problem: find the most relevant snippets and put them in the prompt. The failure in that model is subtle. A repository fragment, old ticket, database row, document paragraph, or previous message can look perfectly relevant while being stale, incomplete, or outside the agent's authorization boundary. The model then turns a plausible pointer into a confident answer. The step-by-step implementation workflow I trust more is search, then browse, then verify. I would map every authorized source into a stable namespace, use Milvus Lite as the local vector tool for narrowing candidates, and require the agent to open the original file, row, ticket, or message slice before it answers or acts. Search decides where to look; the source itself decides what can be claimed. I would also keep ingestion, deletion cleanup, credentials, and permission rules outside the reasoning loop. That is less flexible, but I think the smaller action space is a feature. It makes failures easier to inspect and prevents the model from silently expanding its own access. My rule is simple: if the original context cannot be reopened and verified, the agent should say it does not know. If the requested action exceeds the source's authorization scope, it should stop. I would rather have an agent refuse occasionally than produce an answer that only looks grounded. What evidence do you require before an agent can move from answering questions to changing an external system?