@ando_w: Today I interviewed a candidate whose resume said "Proficient in RAG". I asked about the recall rate, he said "It's pretty good." I asked how it was evaluated, he said "I asked a few questions and it felt fine." This is the current state of the industry: 90% of RAG projects have no evaluation set, all relying on gut feeling. Those who can say Recall and Precision are already...
Summary
An interviewer complained that a candidate claiming to be "proficient in RAG" could not state the recall rate and precision, pointing out that 90% of RAG projects in the industry lack standardized evaluation sets and rely entirely on subjective judgment.
View Cached Full Text
Cached at: 07/09/26, 09:40 AM
Today I interviewed a candidate whose resume said “Proficient in RAG.”
I asked about the recall rate, and they said, “Pretty good.” I asked how they evaluated it, and they replied, “I asked a few questions and it felt fine.”
This is the state of the industry: 90% of RAG projects have no evaluation set—it’s all based on gut feeling. Those who can name Recall and Precision are already in the top 10%.
Similar Articles
@shrav_10: Interviewed another candidate today. I asked him what RAG is. He replied: RAG is a technique that allows an LLM to answ…
A hiring manager shares a candidate's correct definitions of RAG and fine-tuning, then asks followers to explain when to use one over the other.
Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations
This paper presents an empirical study evaluating RAG evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) by comparing them to human judgments and standard recall metrics, using a question-answering dataset created from business data.
@sitinme: GitHub 30k stars, do RAG without vector databases and with higher accuracy! Anyone doing RAG has probably experienced this: the vector database returns content that "looks relevant" but isn't the answer you're looking for. Especially with long documents like contracts, financial reports, technical manuals, when you ask "What was Q3 revenue?", it returns a paragraph about "company business overview." Similarity ≠ relevance—this is the fundamental problem with vector retrieval. PageIndex's solution is straightforward and brute-force: skip vectors, use reasoning.
Introduces an open-source project with 30k stars on GitHub that achieves RAG through reasoning instead of vector databases, claiming higher accuracy and solving the problem of similarity not equating to relevance.
@lidangzzz: I told you last year that using RAG and vector databases is a dead end. The correct approach is: 1. Use memory correctly; 2. Properly chunk content, index it well, and summarize it; 3. Give the agent proper search tools...
The author criticizes the RAG and vector database approach, proposing that the correct methods include using memory correctly, chunking and indexing, summarizing, providing search tools for agents, and using SRAM-only inference services such as Groq and Cerebras.
"Most RAG benchmarks lie about real-world corpora." Test data from 3 production websites.
This article argues that most RAG benchmarks are misleading because they assume uniform corpus quality, while real-world corpora vary significantly in content density. Using data from three production websites, it shows that a tiered approach and a 'yield score' can better predict retrieval effectiveness.