How RAG Finds the Right Context

Every number in this article is printed by a real run over 100,000 generated documents. Nothing is invented. Replay everything with python3 demo.py.

Full demo code, tests, and run logs: github.com/vikasgupta4190/rag-vector-search-demo

📬 Enjoying this kind of deep-dive? I write **ByteBuilder **, a weekly newsletter unpacking the design decisions behind the tools developers actually use. Free, no spam, one email a week. Subscribe here →

TL;DR

• Exact vector search scores 100% of the corpus per query. Measured: 8.858 ms for 100,000 documents.

• An IVF index probes only the 4 nearest centroid lists. Measured: 0.725 ms, a 12.2x speedup.

• The cost: recall@10 drops to 0.883. Only 6.71% of the corpus is ever scored.

• The knobs are nlist and nprobe. Our 7-by-4 sweep tells you exactly what each turn costs.

1. Why retrieval is the hard half of RAG

Retrieval-augmented generation lives or dies on one question: given a query, which documents deserve a seat in the context window?

Scoring every document against every query answers it perfectly. It is also a bill that grows linearly with the corpus. Our demo makes the bill concrete:

• 100,000 documents, TF-IDF vectors of 418 dimensions, cosine similarity.

• Exact top-10 per query: 8.858 ms, scoring 100.0% of the corpus.

• Quality check: 98.3% of exact top-10 hits land in the query’s own topic cluster.

A deliberate note on honesty: we use TF-IDF, not a neural embedding model. Every weight is inspectable, every distance is exact, and the structure it reveals (98.3% cluster purity) is the same structure production embeddings exploit.

2. The exact baseline: truth, and the bill

Exact kNN is the recall reference. Its recall@10 is 1.000 by definition, because every approximate index is graded against the exact top ten.

plaintextscores = docs @ q # all 100000 docs, every query top10 = argpartition(scores)[:10]

At 100k documents it costs 8.858 ms. At 10 million it would cost about a second. At a billion, never. Retrieval at scale is impossible without skipping work.

3. The IVF trick: file first, probe few

Production vector databases skip work with an inverted file index (IVF):

• Train k centroids over the vector cloud (we use 64).

• File every document under its nearest centroid.

• Probe only the nprobe closest lists at query time (we use 4).

• Re-score exactly, but only inside those lists.

Our index stores the lists in one contiguous array with offsets, the same CSR layout a real engine uses. Measured outcome, 100,000 documents:

• Index build: 294.7 ms, one time.

• Candidates scored: 6.71% of the corpus.

• Latency: 0.725 ms per query. 12.2x faster than exact.

• Recall@10 vs exact: 0.883.

• Cluster purity of the top-10: 0.992.

That last number deserves attention: 93.3% of the corpus is never touched. Recall only falls by 11.7 points

4. The knobs, swept and measured

Every vector database exposes the same two knobs. We swept all 28 combinations:

• Best cell: nlist 32, nprobe 8 -> recall 1.000.

• Worst cell: nlist 192, nprobe 1 -> recall 0.429.

• More probes: higher recall, higher latency.

• More centroids: smaller lists, less work per probe.

And the latency follows recall up. At nprobe 8 and nlist 32 you get perfect recall; you also score far more lists. There is no best config, only a chosen point on a measured curve.

5. What the retriever writes down

A retriever that never logs can never be tuned. The demo logs the whole chain:

• DOCUMENT links to its EMBEDDING rows, one per embedder version.

• INDEX_VERSION records nlist, nprobe, and the measured build_ms.

• QUERY_LOG stores every query with its latency and the index that served it.

• HIT stores each ranked result per query.

Every claim in this article traces back to those rows, then to workspace/runs.

6. Failure modes, read before shipping

• Stale index. New documents are invisible until a rebuild. Our measured build is 294.7 ms; schedule accordingly.

• Recall collapse. nprobe 1 at nlist 192 measured 0.429. Wrong knobs are silent failures.

• Latency lie. Exact feels instant on a laptop corpus and dies at production scale.

• No rerank. Probing finds candidates; exactly re-scoring them is what protects quality.

• No query log. Without QUERY_LOG there is nothing to tune against.

• Embedding drift. Swap the embedder and the whole index is orphaned. Re-embed everything.

7. Sizing it for production (planning figures, not measurements)

• Vector databases commonly default nlist near 1024 for million-scale corpora. (example)

• Typical nprobe operating points: 8 to 64 lists per query. (example)

• Query budgets often target single-digit milliseconds p50. (example)

• Reranking adds a second pass over 20 to 200 candidates. (example)

Treat these as design prompts. Measure your own stack before quoting them.

8. Key takeaways

• Exact search is both the truth and the bill: 8.858 ms for 100% of the corpus.

• IVF skips 93.3% of the work and keeps 88.3% of the answers.

• nlist and nprobe are a measured tradeoff, not a mystery: see the sweep.

• Log every query. Tuning without a log is guessing.

• Rerank candidates exactly. Probing finds them; scoring defends them.

Sources and replay

• Full demo code, test suite, and archived run logs: github.com/vikasgupta4190/rag-vector-search-demo

• Terminal figures: screenshots/, regenerated by render_evidence.py.

• Charts: images/demo-*.png, regenerated by render_charts.py.

• Metrics dump: metrics.json.

Subscribe to ByteBuilders and stay ahead in AI