I benchmarked my reasoning-based retrieval system against FAISS and BM25 on 700 queries, running everything on local Qwen. Results + where it loses

Disclosure up front: this is my own project (ClawIndex), one-person shop. Not selling anything here, the writeup is free to read and I’m mostly after criticism from people who do retrieval seriously. The setup: a retrieval approach that does a reasoning pass over an index instead of pure vector similarity. No embeddings, no vector DB. The entire benchmark ran on self-hosted Qwen, nothing left the machine. Benchmarked against FAISS and BM25 across 700 queries: 500 HotpotQA + 100 BEIR ArguAna + 100 BEIR SciDocs. What it won: • NDCG@10 on all five dataset splits (0.934 on HotpotQA full set) • Biggest gap on multi-hop bridge questions: 0.920 vs FAISS 0.837 • 51/500 HotpotQA queries hit a fallback path; all still resolved to valid traced results Where it loses / caveats I want to be honest about: • FAISS beats it on MRR@10 on both BEIR datasets • SciDocs margin (0.975 vs 0.972) is within noise at n=100, no confidence intervals yet, so I'm calling it directional not a win • HotpotQA was the distractor setting, not fullwiki. My comparison to a published system (PRISM) may be apples-to-oranges since I haven't confirmed their corpus setting • ~27 seconds per query. FAISS is 7ms. This is the real cost and it's not small Honest take: it’s slow and it’s built for a narrow job, async work where a traceable, fully-local answer beats a fast one (contract review, compliance, anything that can’t touch an external API). For real-time search, FAISS wins, no contest. Full tables and method in the link. Genuinely want to know what I’m measuring wrong or what else I should test!