@bclavie: This might be the best IR release of the year. Text benchmarking is (was) broken, DL19/DL20/BEIR no longer hold valuabl…
Summary
A new IR benchmark release addresses broken text benchmarking in DL19/DL20/BEIR, enabling meaningful measurement of improvements in current era training methods.
View Cached Full Text
Cached at: 05/08/26, 07:37 PM
This might be the best IR release of the year.
Text benchmarking is (was) broken, DL19/DL20/BEIR no longer hold valuable signal for the current era and training methods.
… but now we can meaningfully measure improvements again, thanks to @dianetc_ and @lateinteraction
Similar Articles
New benchmark dropped
A new benchmark has been released, likely for evaluating AI or software performance.
@dianetc_: We set out to build a better retriever, so we looked for the hardest IR benchmarks. For each, we asked how much headroo…
The authors introduce OBLIQ-Bench, a new benchmark designed to evaluate information retrieval systems on significantly harder search queries where previous benchmarks showed little remaining headroom.
@Lyubh22: Coding benchmarks are saturating. AI4Research is the next frontier. Thrilled to see our MLS-Bench (https://mls-bench.co…
Announcing MLS-Bench, the first AI4Research benchmark to gain broad community adoption, testing AI agents on 140 executable tasks across 12 domains to propose modular ML improvements. The post includes leaderboard scores for models like Claude Opus 4.6 and GPT-5.4.
New local model reaching near frontier on PII removal at 9 ms CPU inference
Introduces ScreenLeak, a benchmark for measuring PII redaction in computer-use AI data, and presents two local models (v45_phase3 for text and rfdetr_v8 for images) achieving near-frontier performance at low latency.
@jerryjliu0: ParseBench is the first benchmark to include VLM chart understanding over enterprise documents. Existing benchmarks (Ch…
ParseBench introduces the first benchmark evaluating vision-language models on chart comprehension within full enterprise documents, addressing gaps in prior chart-only benchmarks.