For agents using knowledge bases, maybe clean the data first

A lot of agent projects eventually run into the same problem: the agent needs access to knowledge. That knowledge may come from docs, PDFs, support tickets, databases, meeting notes, webpages, code repos, product manuals, or internal processes. Once the agent starts using tools, answering user questions, or making decisions based on retrieved context, the quality of that knowledge base matters a lot. The issue is that raw data is rarely ready for agents. If we index messy data directly, the agent may retrieve incomplete chunks, duplicated content, broken tables, outdated information, missing metadata, or context with no source trace. The agent may look capable, but its answers become unstable because the knowledge layer is weak. A better workflow might be to prepare the knowledge base before connecting it to the agent: parse different source formats into structured text preserve headings, tables, source IDs, and metadata clean noisy content without changing facts remove duplicates and low-value chunks split content based on source type, not one fixed chunk size anonymize sensitive information when needed generate QA pairs or eval sets for retrieval testing keep raw and cleaned versions for debugging make every chunk traceable back to its source For agent systems, this is especially important because retrieval is often only one step in a longer workflow. Bad context can affect tool choice, reasoning, planning, and final actions. So I’m starting to think that “agent + knowledge base” should not only mean plugging a vector DB into an agent framework. There should be a real data preparation layer before the agent touches the knowledge. This is the design direction we are exploring in OpenDCAI/DataFlow, and I’d love to hear feedback from people building or using agent systems in practice.