Tag
The author shares their experience switching from semantic embeddings to BM25 for tool selection in agents, finding that BM25 achieves 81% top-1 accuracy vs. 64% for embeddings on a corpus of 200 query-tool pairs, because tool descriptions are short and keyword-driven rather than semantically rich like documents.
This paper proposes Evidence-Calibrated Query Clustering (ECC), an algorithm that aligns semantic embeddings with latent LLM capability demands using posterior model comparisons and Bradley-Terry modeling, significantly improving capability ranking quality for LLM evaluation.