LLM rankings are not a ladder: experimental results from a transitive benchmark graph [D]

Reddit r/MachineLearning Tools
llm-evaluation benchmarking transitivity model-ranking data-visualization

Summary

The author introduces LLM Win, a tool that visualizes LLM benchmark results as a directed graph to analyze transitive relationships and ranking reversals. Experimental findings suggest that LLM rankings function more like a capability graph with high weak-to-strong reachability rather than a linear ladder.

I built a small website called **LLM Win**: https://llm-win.com It turns LLM benchmark results into a directed graph: ```text If model A beats model B on benchmark X, add an edge A -> B. ``` Then it searches for the shortest transitive chain between two models. The meme version is: ```text Can LLaMA 2 7B beat Claude Opus 4.7? ``` In an absurd transitive benchmark sense, sometimes yes. But I added a Report tab because the structure itself seems useful for model evaluation. Some experimental findings from the current Artificial Analysis data snapshot: 1. **Weak-to-strong reachability is high.** I checked `126,937` pairs where the source model has lower Intelligence Index than the target model. `119,514` of them are reachable through benchmark win chains, for a reachable rate of `94.2%`. 2. **Most paths are short.** Among reachable weak-to-strong pairs: `2-3 hop` paths account for `91.4%`. So this is not mostly long-chain cherry-picking. 3. **Direct reversal triples are abundant.** After treating non-positive benchmark values as missing, there are still about `119k` direct weak-over-strong triples of the form: `(source model, target model, benchmark)`, where the source has lower Intelligence Index but higher score on that benchmark. 4. **Some benchmarks create more reversals than others.** Current high-reversal / useful-signal candidates include: Humanity's Last Exam, IFBench, AIME 2025, TAU2, SciCode 5. **Different benchmarks have different interpretations.** For example, IFBench has roughly: reversal rate: ~17.5%, coverage: ~80.0%, correlation with Intelligence Index: r≈0.82. This suggests it may provide an independent skill signal rather than simply duplicating the overall ranking. My current interpretation: LLM rankings are better represented as a benchmark-specific capability graph than as a single ladder. Some reversals probably reflect real specialization; some reflect benchmark coverage limits, volatility, or measurement noise. The next question is whether reversal structure can help build better evaluation metrics: - identify specialist models; - identify volatile benchmarks; - build robust generalist scores; - select complementary benchmark sets; - decompose models into capability fingerprints. Curious what people think: Is benchmark reversal structure a useful evaluation signal, or mostly an artifact of noisy benchmarks?
Original Article

Similar Articles

GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory

arXiv cs.AI

The paper introduces GTBench, a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, containing 63 problems across three difficulty levels. It evaluates five frontier models and finds that performance degrades with difficulty, with GPT-5 achieving near-perfect results on basic problems but only 82% on graduate-level proofs.

GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs

arXiv cs.LG

Introduces GraphInfer-Bench, a benchmark to evaluate whether LLMs can perform graph inference—producing open-ended answers about a node and its neighborhood that cannot be retrieved from a single node or path. Experiments show that even frontier LLMs lag behind plain GNNs on these tasks, revealing a capability gap.

Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Hugging Face Daily Papers

This paper argues that aggregate-score leaderboards for LLM agent benchmarks fail to capture deployment-relevant dimensions and show rank instability. It proposes ranking configurations by predictive validity—the correlation between in-sample and out-of-sample rank—and introduces a twelve-tier measurement apparatus along with falsifiable out-of-distribution criteria.

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

arXiv cs.CL

DLawBench is a new benchmark for evaluating large language models in multi-turn legal consultation, covering Chinese and US law with four client types. Experiments show significant room for improvement, with the best model achieving only 0.562 on legal reasoning.