@ianchanning: If your RAG system is no better than just https://google.com/ai you've wasted a ton of money (my suspicion is that ther…
Summary
Ian Channing criticizes companies that waste money on RAG systems that perform no better than Google AI, arguing that access to a corpus doesn't equate to deep expertise and that reasoning cannot be cleanly separated from knowledge.
View Cached Full Text
Cached at: 07/05/26, 12:33 PM
If your RAG system is no better than just https://google.com/ai you’ve wasted a ton of money (my suspicion is that there’s quite a few companies who’ve done this).
From the glorious Machine Studying:
“But this conflates having access to the corpus with developing deep expertise: you wouldn’t hire any of us as a lawyer just because we can Google the legal literature very intelligently. At minimum, what makes a lawyer a good lawyer is knowing what to look for, where to look, and what to do with a passage after they find it.
You could say that reasoning and search are not separable from knowledge (see below). An agent deciding what to grep for or which file to open is acting from its current weights, and those weights may actively conflict with the world encoded in the corpus.“ – https://jacobxli.com/blog/2026/machine-studying/#2-cant-the-agent-just-search-the-corpus…
Omar Khattab (@lateinteraction): You cannot separate reasoning and knowledge as cleanly as you think.
If you’d asked me what I care about in 2020/2021, I’d have said it was “decoupling the capacity that language models have for understanding text from how they store knowledge” (quote from link below this
Similar Articles
Evaluated a RAG chatbot and the most expensive model was the worst performer. Notes on what actually moved the needle.
A detailed evaluation of a RAG customer support chatbot reveals that retrieval issues often masquerade as LLM problems, heuristic evaluators are misleading, deduplication improves quality, stricter grounding trades helpfulness for accuracy, and model sweeping can dramatically reduce cost while improving performance.
"Most RAG benchmarks lie about real-world corpora." Test data from 3 production websites.
This article argues that most RAG benchmarks are misleading because they assume uniform corpus quality, while real-world corpora vary significantly in content density. Using data from three production websites, it shows that a tiered approach and a 'yield score' can better predict retrieval effectiveness.
@DivyanshT91162: The AI industry spent billions solving a problem that a 52-year-old terminal command had already solved. For the last t…
A Twitter thread argues that the AI industry's expensive investment in vector retrieval systems for RAG may be unnecessary, as the 52-year-old terminal command 'grep' outperforms modern semantic search for exact matches in AI agent contexts.
@GoogleResearch: Introducing our new agentic RAG framework. A collab with Google Cloud, our multi-agent workflow goes beyond standard RA…
Google Research introduces an agentic RAG framework hosted on the Gemini Enterprise Agent Platform that uses multi-agent workflows to break down complex enterprise queries and iteratively search for sufficient context, achieving up to 34% increased accuracy over standard RAG.
@NainsiDwiv50980: RAG might already be becoming obsolete. A month ago, Andrej Karpathy dropped a simple GitHub gist called “LLM Wiki.” No…
Andrej Karpathy's 'LLM Wiki' concept is sparking a rapid developer ecosystem focused on persistent AI memory and self-maintaining knowledge bases, potentially making traditional RAG obsolete.