The author benchmarks classic vector RAG against Google's new Open Knowledge Format (OKF) and a hybrid approach using a local stack (Ollama, ChromaDB), finding hybrid retrieval answers more questions correctly but at higher token cost, while highlighting specific failure modes.
Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each other, with an index.md for progressive disclosure. The only required field is `type`. I wanted to know whether it actually fixes anything, so I built a test corpus and measured. Everything runs locally: qwen3:8b + nomic-embed-text + ChromaDB, no external APIs. SETUP - Corpus: 60 markdown files of fake-but-realistic company docs (wiki, table schemas, ADRs, 40 support tickets). 85 chunks at 800/100. - OKF bundle: 9 curated concepts covering the same ground. - 7 questions, each designed to trigger a different retrieval failure mode. RESULTS (7 questions) RAG OKF OKF+RAG correct 2 3 4 tokens 6341 8625 8435 Nothing passes. The combined layer gets twice what plain RAG does, at ~33% more tokens. THE ONE THAT SURPRISED ME Question: "how do we calculate revenue?" The corpus has a 2023 doc (deprecated, verbose, 4000 chars) and the current 2026 spec (terse, 500 chars). The deprecated doc splits into 7 chunks, the current one into 1. Three of the top-5 retrieved chunks came from the deprecated doc. The correct document ranked 15th out of 85 — behind a glossary, a customer table schema, and a support ticket about shipping costs to the Canary Islands. Raising k to 15 doesn't help: you'd pull in 6 chunks saying the wrong thing against 1 saying the right thing. A reranker can't fix it either — there's nothing in the chunk text indicating which is current. The date isn't in the chunk. OTHER FAILURE MODES THAT FIRED - Chunker split an 18-column schema table. The right file WAS in context; the table wasn't. Model said "I don't know" at both k=3 and k=5. - Composition: a metric definition needs 3 rules living in 3 separate files. RAG retrieved 2 of 3 and answered confidently, citing sources, never hinting anything might be missing. - Interesting pattern: it said "I don't know" when it had almost nothing, and said nothing when it had almost everything. It goes quiet exactly when it's most expensive. WHERE OKF LOSES Long-tail questions. "Was there an incident with duplicate orders in March?" — plain RAG nailed it over 40 messy, unreviewed tickets. Curating those by hand would be absurd. OKF alone failed it. TERMINOLOGY CAVEAT I'm using "RAG" as shorthand for the classic vector implementation. Strictly, an agent navigating an OKF index is also a RAG pipeline — just with structured retrieval instead of vector retrieval. The precise framing is "classic vector RAG vs structured retrieval over OKF". Someone rightly called me out on this. Full code, corpus, bundle and the raw results.txt: https://github.com/JoaquinRuiz/rag-vs-okf git clone + uv sync and you can reproduce it. Curious whether anyone gets different numbers with a bigger model — question 4 was unstable across runs for me.
This article provides a deep-dive into Retrieval-Augmented Generation (RAG) and vector search, with measured benchmarks on 100,000 documents showing the trade-offs between exact search and IVF index for speed and recall.
This paper presents a controlled scaling study comparing lexical, dense, graph-based, and agentic RAG paradigms across corpus sizes from 1,000 to 512,000 documents, finding that BM25 provides the best accuracy-cost tradeoff, while graph-based RAG faces high construction costs that limit scalability.
Microsoft Research's LazyGraphRAG outperformed vector RAG on data-local questions across 8k, 120k, and million-token contexts, winning 92/90/91% at a tenth of the cost, and is now open-sourced on GitHub.
This paper benchmarks Google Embeddings 2 against five open-source models for multilingual dense retrieval and RAG, finding GE2 top in accuracy but slower, with mE5-L as a competitive low-latency alternative.
Researchers introduced PageIndex, a novel RAG approach that eliminates reliance on vector databases, embeddings, and chunking, achieving 98.7% accuracy on FinanceBench and outperforming existing methods while being free and open source.