Q2D-Web: Evaluating First-Stage Retrievers at Scale (11 minute read)
Summary
Q2D-Web is a large-scale benchmark and leaderboard for evaluating retrieval models on web search, featuring 190 million documents and queries in ten languages with methods to minimize bias and reduce evaluation costs.
Similar Articles
QontoFAQ: A better Information Retrieval Benchmark [R]
QontoFAQ introduces a new benchmark and metric for information retrieval, focusing on improving relevance measurement for answering product questions, with associated code and dataset released.
I benchmarked my reasoning-based retrieval system against FAISS and BM25 on 700 queries, running everything on local Qwen. Results + where it loses
A reasoning-based retrieval system (ClawIndex) is benchmarked against FAISS and BM25 on 700 queries using local Qwen, achieving higher NDCG@10 but slower speed and lower MRR on some datasets, with the author seeking feedback.
@dianetc_: We set out to build a better retriever, so we looked for the hardest IR benchmarks. For each, we asked how much headroo…
The authors introduce OBLIQ-Bench, a new benchmark designed to evaluate information retrieval systems on significantly harder search queries where previous benchmarks showed little remaining headroom.
Benchmarking Hybrid Deep Research Across Database Querying and Web Search
This paper introduces HybridDeepResearch, a benchmark for evaluating AI agents on tasks that require integrating web search and SQL querying, revealing that even state-of-the-art models struggle with maintaining constraints across structured and unstructured data.
Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval
This paper benchmarks 17 deep learning models for first-stage recall in large-scale code-to-code retrieval, evaluating their precision, efficiency, and scalability across multiple programming languages and datasets. It introduces LLM-based code normalization and query rewriting schemes that improve precision for lower-performing models.