Built an open-source fact-checker for AI agents, it won't let a claim through unless it can actually back it up

Reddit r/AI_Agents Tools

Summary

Built an open-source fact-checker for AI agents that verifies claims by fetching real-time sources and providing truth and confidence scores, useful for both public and internal documents.

The thing that kept bugging me: ask an AI "how sure are you?" and it says 95% almost every time, whether it's right or not. Fine for a chat. Not fine when an agent is about to act on that info — write it into a report, save it to a knowledge base, make a decision. So we built the thing that checks first. You give it a claim, it goes and fetches real sources right now (never pulls from the model's memory), and comes back with two separate numbers: a truth score (is it actually true) and a confidence score (how much you should trust that answer — drops when sources are thin or disagree with each other). Plus the sources and a plain-language "why." The part that ended up being the most useful for us: it's not limited to checking against the public web. It can also verify a document's claims against the sources it cites — which matters for internal stuff (policies, metric definitions, schema docs) that the web has no opinion on. We built a demo that plants four errors into a fake finance-policy bundle and it catches all four while leaving the untouched control file alone. Fully open source, MIT licensed, works standalone in a browser, as an MCP tool for your agent, or over a whole knowledge base. Two free API keys and you're running it locally in a couple minutes. Would genuinely love feedback/issues if you try it — link's in the first comment (sub rule). If it's useful to you, a star on the repo helps more than you'd think.
Original Article

Similar Articles

FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact

arXiv cs.CL

Introduces factwash, an open-source write-time gate that deterministically catches AI rewrites that turn hearsay into fact by stripping attribution, hedging, or temporal context. The paper analyzes when simple checks suffice versus when an LLM witness is needed, and evaluates the approach on large annotated corpora.