I benchmarked RAG techniques on a synthetic healthcare database. The biggest gains came from document shape, not model tweaks.
Summary
A benchmarking study on RAG techniques using a synthetic healthcare database found that optimizing document shape provided larger performance gains than adjusting model parameters.
Similar Articles
When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
A large-scale study across 5 models (7B–72B), 10 biomedical QA datasets, 4 retrieval methods, and 4 corpora finds that RAG yields only small and inconsistent gains (1–2 points) over no-retrieval baselines in biomedical question answering. The study concludes that the main bottleneck is not retrieval quality but models' limited ability to effectively use retrieved evidence.
Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations
This paper presents an empirical study evaluating RAG evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) by comparing them to human judgments and standard recall metrics, using a question-answering dataset created from business data.
"Most RAG benchmarks lie about real-world corpora." Test data from 3 production websites.
This article argues that most RAG benchmarks are misleading because they assume uniform corpus quality, while real-world corpora vary significantly in content density. Using data from three production websites, it shows that a tiered approach and a 'yield score' can better predict retrieval effectiveness.
How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
This paper introduces HieraRAG, a hierarchical framework for determining optimal granularity in RAG benchmarks. It generates 5,872 synthetic QA pairs across three dimensions and finds that ideal granularity varies by dimension, offering a portable procedure for practitioners.
@oliviscusAI: The entire RAG industry is about to get cooked. Researchers developed a new RAG approach that bypasses almost everythin…
Researchers introduced PageIndex, a novel RAG approach that eliminates reliance on vector databases, embeddings, and chunking, achieving 98.7% accuracy on FinanceBench and outperforming existing methods while being free and open source.