@dianetc_: We set out to build a better retriever, so we looked for the hardest IR benchmarks. For each, we asked how much headroo…
Summary
The authors introduce OBLIQ-Bench, a new benchmark designed to evaluate information retrieval systems on significantly harder search queries where previous benchmarks showed little remaining headroom.
View Cached Full Text
Cached at: 05/08/26, 10:48 AM
We set out to build a better retriever, so we looked for the hardest IR benchmarks.
For each, we asked how much headroom remained by running oracle reranking with a frontier LLM. Most had little room left!
So we built OBLIQ-Bench to study much harder search queries than before. https://t.co/IIWIbk98nN
Similar Articles
@_reachsumit: OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries @dianetc_ et al pres…
OBLIQ-Bench is a new benchmark that exposes weaknesses in current retrieval systems when handling oblique queries requiring latent or implicit reasoning, showing that even sophisticated retrieval pipelines fail to surface relevant documents that reasoning LLMs can easily verify.
I benchmarked my reasoning-based retrieval system against FAISS and BM25 on 700 queries, running everything on local Qwen. Results + where it loses
A reasoning-based retrieval system (ClawIndex) is benchmarked against FAISS and BM25 on 700 queries using local Qwen, achieving higher NDCG@10 but slower speed and lower MRR on some datasets, with the author seeking feedback.
@lu__jasper: Some early results from playing around with search on a subsampled version of OBLIQ-bench. Mixedbread's reranker is a b…
Early results from testing search on a subsampled OBLIQ-bench show that Mixedbread's reranker achieves strong MRR, sometimes outperforming GPT 5.5 on certain metrics with faster speed, though the benchmark remains challenging.
QontoFAQ: A better Information Retrieval Benchmark [R]
QontoFAQ introduces a new benchmark and metric for information retrieval, focusing on improving relevance measurement for answering product questions, with associated code and dataset released.
@bclavie: This might be the best IR release of the year. Text benchmarking is (was) broken, DL19/DL20/BEIR no longer hold valuabl…
A new IR benchmark release addresses broken text benchmarking in DL19/DL20/BEIR, enabling meaningful measurement of improvements in current era training methods.