Your RAG is hallucinating because of garbage retrieval — here's the 3-line fix (with real scores)

Reddit r/AI_Agents Tools
rag hallucination retrieval cross-encoder reranking llm

Summary

A practical fix for RAG hallucination caused by noisy retrieval: use cross-encoder re-ranking to filter chunks with a score > 1.5, improving relevance from -0.28 to +3.80 on average.

My RAG agent hallucinated. Not because the LLM was bad — because the retrieval was feeding it noise. Query: "What are Python decorators?" What my retriever returned (before fix): | Rank | Score | Content | Relevant? | |---|---|---|---| | 1 | +5.80 | Decorator definition | Yes | | 2 | +1.40 | Acknowledgments page | No | | 3 | +1.13 | u/staticmethod example | Yes | | 4 | -4.69 | Class exercises | No | | 5 | -11.0 | Monty Python reference | No | The LLM received all 5 chunks. It hallucinated because it trusted the noise. The fix — cross-encoder re-ranking (3 lines): scores = cross\_encoder.score(pairs) ranked = sorted(zip(scores, candidates), reverse=True) filtered = \[doc for score, doc in ranked if score > 1.5\] After fix: only chunks with score > 1.5 reach the LLM. Overall results (10 queries): avg relevance went from -0.28 to +3.80. 80% win rate. Model: cross-encoder/ms-marco-MiniLM-L-6-v2 (free, local, HuggingFace). If your chatbot hallucinates, check your retrieval before blaming the LLM. What threshold are you using for your re-ranker?
Original Article

Similar Articles

RAG hallucinations are annoying AF

Reddit r/AI_Agents

The team discovered that 80% of RAG hallucinations were caused by poor retrieval, not the generation model, emphasizing the need to evaluate retrieval and generation separately to effectively debug bad answers.

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration

arXiv cs.CL

RAGognizer introduces a hallucination-aware fine-tuning approach that integrates a lightweight detection head into LLMs for joint optimization of language modeling and hallucination detection in RAG systems. The paper presents RAGognize, a dataset of naturally occurring closed-domain hallucinations with token-level annotations, and demonstrates state-of-the-art hallucination detection while reducing hallucination rates without degrading language quality.

Most agent RAG problems I see are retrieval problems, not model problems

Reddit r/AI_Agents

The author argues that most agent RAG failures are due to retrieval problems—specifically chunking errors, lack of freshness signals, and reliance on pure vector search—rather than the LLM, and recommends structural chunking, decay-based ranking, and hybrid BM25+vector search.