bm25

Ratel is an open-source tool that reduces input tokens by 79% and improves tool selection accuracy for AI agents by loading only needed tools using a BM25 index, instead of all available tools.

AutoIndex is a framework that learns executable representation programs to transform documents before indexing, improving retrieval quality over static BM25 baselines by average 8.4% Recall@100 and 8.3% nDCG@10.

A reasoning-based retrieval system (ClawIndex) is benchmarked against FAISS and BM25 on 700 queries using local Qwen, achieving higher NDCG@10 but slower speed and lower MRR on some datasets, with the author seeking feedback.

Describes DR-DCI, an optimization that combines RAG with bash commands on a virtual filesystem to enable agents to perform precise corpus retrieval, and discusses scaling to distributed systems for inference providers.

Technical deep dive into Lume, a Rust hybrid search engine that combines BM25, dense vectors, and entity graphs for auditable, local-first retrieval.

This paper introduces a self-evolving framework that uses an LLM-based agent to iteratively create and refine query rewriting rules for BM25 in legal case retrieval, outperforming non-evolutionary baselines on the LeCaRD-v2 benchmark without any parameter training.

Archex is a new open-source code RAG tool that improves retrieval by combining hybrid search (BM25F + dense embeddings), cross-encoder reranking, and dependency-graph expansion, achieving much higher recall and token efficiency than pure embeddings-based approaches.

The author shares their experience switching from semantic embeddings to BM25 for tool selection in agents, finding that BM25 achieves 81% top-1 accuracy vs. 64% for embeddings on a corpus of 200 query-tool pairs, because tool descriptions are short and keyword-driven rather than semantically rich like documents.

A developer shares that the real problem in a RAG app was the retrieval step failing on version numbers and codes, fixed by hybrid search (vector + BM25 + reciprocal rank fusion), not the model.

RISE framework constructs bounded interaction spaces for agentic search by combining BM25 retrieval with preprocessed document indexing, enabling efficient corpus exploration while maintaining high accuracy at scale.

This paper proposes a training-free, CPU-only retrieval method that fuses BM25 lexical scores with late-interaction dense scores for conversational memory retrieval, achieving up to +17.2 points improvement on LoCoMo Hit@1 over late interaction alone across six encoders. The study provides controlled ablations on pooling operators, reranker effects, and benchmark robustness, framing the gain as a division of labor between dense and lexical signals.

This post questions whether combining BM25 and vector search with RRF improves hit rates in agentic memory retrieval, suggesting BM25 alone may suffice.

Single-vector embedding models can be used to extract sparse latent terms, and BM25 can turn this vocabulary into a strong retriever.

A developer recounts debugging RAG systems, discovering that fixed-size chunking breaks sentence boundaries, vector search fails for exact identifiers (solved with BM25), and stale indexes cause confident wrong answers.

Researchers extract indexable, BM25-ready sparse features from frozen dense retrievers using reconstruction-trained sparse autoencoders.

The paper proposes Latent Terms, a method using Sparse Autoencoders to extract BM25-ready sparse features from frozen dense retrievers, achieving competitive performance without retrieval-specific training.

Jerry Liu asks about the current state-of-the-art for file search and retrieval, listing options from grep to hybrid search over a database.

Context Mode is a tool that solves AI agent context problems by sandboxing tool outputs and persisting sessions, achieving up to 98% compression of Playwright snapshots and using BM25 retrieval to reduce context window usage. It supports 15 platforms including Claude Code, Gemini CLI, VS Code Copilot, and is used by major tech companies.

This paper introduces Pi-Serini, a BM25-based agentic search system that demonstrates lexical retrieval can suffice for deep search when agents refine queries, achieving high accuracy and reducing costs compared to default settings.

The article argues against overusing vector search, highlighting BM25's effectiveness for exact keyword matching and its role in hybrid search systems.