QontoFAQ: A better Information Retrieval Benchmark [R]
Summary
QontoFAQ introduces a new benchmark and metric for information retrieval, focusing on improving relevance measurement for answering product questions, with associated code and dataset released.
Similar Articles
I benchmarked my reasoning-based retrieval system against FAISS and BM25 on 700 queries, running everything on local Qwen. Results + where it loses
A reasoning-based retrieval system (ClawIndex) is benchmarked against FAISS and BM25 on 700 queries using local Qwen, achieving higher NDCG@10 but slower speed and lower MRR on some datasets, with the author seeking feedback.
@dianetc_: We set out to build a better retriever, so we looked for the hardest IR benchmarks. For each, we asked how much headroo…
The authors introduce OBLIQ-Bench, a new benchmark designed to evaluate information retrieval systems on significantly harder search queries where previous benchmarks showed little remaining headroom.
RAGless: Q-Q retrieval with score aggregation for closed-domain FAQ [P]
RAGless is a semantic retrieval system that matches user questions to pre-generated question variants for closed-domain FAQ, eliminating the LLM generation step in standard RAG for improved precision.
Q2D-Web: Evaluating First-Stage Retrievers at Scale (11 minute read)
Q2D-Web is a large-scale benchmark and leaderboard for evaluating retrieval models on web search, featuring 190 million documents and queries in ten languages with methods to minimize bias and reduce evaluation costs.
FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
FinFIRST introduces a benchmark for evaluating financial search agents by jointly assessing answers and supporting evidence through atomic rubrics, with 123 expert-authored tasks spanning difficulty levels.