I collected every single LLM coding benchmark, and computed their Intelligence Density
Summary
The article presents the Agentic Coding Index, a method to compute the intelligence density of LLMs by aggregating various coding benchmarks and normalizing for parameter count.
Similar Articles
Open-source LLM benchmark runs 147 coding tasks every 4 hours, 5-trial median with 95% CI, and uses CUSUM for change-point detection. Curious what people think of the methodology
An open-source LLM benchmark with 147 coding tasks runs every 4 hours, using 5-trial median with 95% confidence intervals and CUSUM for change-point detection, sparking discussion on its methodology.
Benchmarking LLM Competence on Logical Inference over Probability Operators
This paper introduces a benchmark of 14,320 procedurally-generated prompts for evaluating LLMs on logical inference over probability operators like 'probably', 'might', and 'must'. Testing 29 models, the authors find systematic answer biases and show that only 9 exceed random chance.
Benchmarking Different Methods of LLM Confidence Estimation
This article benchmarks various blackbox and whitebox methods for LLM confidence estimation, including verbalized confidence, linguistic uncertainty, reasoning-length, P(Answer), P(True), and self-consistency, comparing their effectiveness for tasks like active learning and safety classification.
AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable
This paper evaluates LLM-based coding agents (Claude Code and Codex) in social science analysis, finding they match or exceed human methodological diversity while remaining vulnerable to interpretation bias through verdict-layer manipulation.
The gap between open weights LLMs and closed source LLMs
Analyzes the gap between open weights and closed source LLMs using the Artificial Analysis Intelligence Index and other benchmarks, finding that the gap is shrinking on some metrics but stable on others.