@lateinteraction: At this point in time, two of the extremely few long-context benchmarks I'd assign any weight at all to are OBLIQ-Bench…
Summary
A commentator highlights OBLIQ-Bench (recall@k) and StudyBench (expertise) as two of the few reliable long-context benchmarks.
Similar Articles
@_reachsumit: OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries @dianetc_ et al pres…
OBLIQ-Bench is a new benchmark that exposes weaknesses in current retrieval systems when handling oblique queries requiring latent or implicit reasoning, showing that even sophisticated retrieval pipelines fail to surface relevant documents that reasoning LLMs can easily verify.
SWE Context Bench just proved something I think a lot of coding agent users already feel
A new benchmark paper 'SWE Context Bench' tests whether coding agents can reuse knowledge across tasks, highlighting a gap in existing benchmarks that only evaluate isolated problem-solving. The author discusses solutions like external memory and mentions tools such as langmem, mem0, supermemory, and Greplica.
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
This paper identifies a blind spot in long-context LLM reasoning benchmarks: they fail to control task position within the context, allowing positional failures to go undetected. The authors propose Context Rot Evaluation (CRE) to systematically vary task position, filler content, and context length, revealing severe accuracy drops for some models when reasoning tasks are placed in the middle of long contexts.
@dianetc_: We set out to build a better retriever, so we looked for the hardest IR benchmarks. For each, we asked how much headroo…
The authors introduce OBLIQ-Bench, a new benchmark designed to evaluate information retrieval systems on significantly harder search queries where previous benchmarks showed little remaining headroom.
@omarsar0: // Continual Learning Bench // One of the research areas with lots of investments is continual learning. While there ar…
CL-Bench is a new expert-validated benchmark across six domains that evaluates whether LLM-based agents genuinely learn from sequential experience. It finds that naive in-context learning often outperforms dedicated memory systems, indicating current architectures add overhead rather than genuine learning.