Predicting Task Difficulty Without Rollouts

# Predicting Task Difficulty Without Rollouts
Source: [https://arxiv.org/html/2608.05797](https://arxiv.org/html/2608.05797)
###### Abstract

Task difficulty dictates an agent’s likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments\. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula\. This becomes increasingly important as agents move into long\-horizon domains, where empirical trial\-and\-error is a severe computational bottleneck\. Prior work on early prediction is limited to static tasks or isolated coding environments, often relying on narrow features and inaccurate evaluation metrics\. We studyex antedifficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains\. We show that AUC can mask poor difficulty estimates, identify token\-level entropy as a useful predictive signal, and show how residuals between expected and observed difficulty can expose hidden environment flaws such as contamination and infeasibility\.

## 1Introduction

As LLM agents rapidly improve, the performance frontier is pushed toward domains that are increasingly hard to design, validate, and simulate\. This shift creates a financial and logistical bottleneck for environment designers\. Evaluating frontier systems on complex, long\-horizon tasks can require hours of simulated interaction and many model generations per attempt\(Kwaet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib10)\)\. Consequently, the field faces a paradox: progress requires carefully calibrated environments, but verifying their difficulty empirically is increasingly prohibitive\. This paper argues for a shift towardex antedifficulty prediction, with the goal of estimating whether a task is likely to be hard before spending large amounts of compute on evaluation or training\.

Recent work has used a model’s uncertainty to detect data contamination in text\-based benchmarks like question\-answering\(Li,[2023](https://arxiv.org/html/2608.05797#bib.bib27); Xuet al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib49); Donget al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib26)\)\. Yet contamination can be viewed as an instance of a more fundamental variable, namely task difficulty\. If a model has effectively memorized an item, the task becomes easier for reasons unrelated to the intended capability\. In interactive environments, difficulty can affect both success rates and the strategies agents use\. In human settings, high task complexity can push people to alter their strategies, such as resorting to dishonesty\(Mazaret al\.,[2008](https://arxiv.org/html/2608.05797#bib.bib6)\)\. Similarly, when language\-model agents operate near the edge of their competence, they may exploit shortcuts\(Skalseet al\.,[2022](https://arxiv.org/html/2608.05797#bib.bib7); Helffet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib73)\)\. Additionally, difficulty can reveal when an evaluation task is misaligned with its intended purpose\. A task may register as difficult because it exceeds an agent’s capabilities, or simply because the environment precludes a successful solution\. Conversely, a task may appear easy because it is familiar or prone to shortcuts\. We therefore posit that difficulty serves as a common variable connecting task design and agent behavior\.

If difficulty could be estimated before rollouts, environment designers could construct calibrated evaluations without first spending the cost of full empirical validation\. Training pipelines could also build curricula of progressively harder tasks\(Bengioet al\.,[2009](https://arxiv.org/html/2608.05797#bib.bib13)\), while evaluations could better contextualize success and failure\(Kwaet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib10)\)\.

Despite this motivation, pre\-rollout difficulty prediction remains underdeveloped\. To the best of our knowledge, only one prior paper directly studies this setting\(Geet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib11)\), specifically in coding benchmarks\. This leaves two open questions: how should such estimates be evaluated, and do they transfer across benchmark families? We show that response AUC can be misleading, while task\-level rank metrics give a more cautious picture\. In this work, we broaden the setting beyond coding tasks to seven additional benchmark types and take a diagnostic view of difficulty prediction\. Our core contributions are:

- •Evaluation of difficulty prediction, which shows that response AUC can remain high even when task difficulties are poorly estimated\. Under rank correlation, entropy is the strongest single feature family, while the combined feature set performs best overall, reaching Spearmanρ=0\.399\\rho=0\.399under K\-fold evaluation \(p<0\.001p<0\.001vs\. baseline\) andρ=0\.225\\rho=0\.225out of distribution \(§[4](https://arxiv.org/html/2608.05797#S4), §[5](https://arxiv.org/html/2608.05797#S5)\)\.
- •Residual diagnostics for hidden task factors, showing how gaps between predicted and observed difficulty can guide follow\-up analysis\. Positive residuals point to unexpectedly hard tasks, such as feasibility or evaluator failures, while negative residuals point to unexpectedly easy tasks, such as familiarity \(§[5\.3](https://arxiv.org/html/2608.05797#S5.SS3)\)\.
- •An open multi\-benchmark corpuswith 415,470 agent\-task outcomes across 17 benchmarks, including task metadata, verifier summaries, textual descriptions, and IRT\-estimated difficulties for reuse beyond this paper in behavioral analysis, environment design, and training \(§[3\.1](https://arxiv.org/html/2608.05797#S3.SS1)\)\.

## 2Related work

Taking the rollout as the point of reference, we distinguish three sets of methods by when the difficulty signal is observed: after, during, and before execution\.

#### After rollout\.

Agentic benchmarks often use empirical success rates, rewards, or other trajectory\-level quantities such as number of actions, steps, and elapsed time to estimate difficulty, with human completion time serving as a gold standard\(Kwaet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib10); Liuet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib12); Hoet al\.,[2025](https://arxiv.org/html/2608.05797#bib.bib74)\)\. These methods make difficulty explicit, but their signal is observed only after agents or humans interact with the task\. Our work instead studies whether difficulty can be predicted from information available before running agents\.

#### Adaptive methods\.

Curriculum learning relies on difficulty to order training examples or environments for greater sample efficiency\(Bengioet al\.,[2009](https://arxiv.org/html/2608.05797#bib.bib13); Vygotsky,[1978](https://arxiv.org/html/2608.05797#bib.bib29); Portelaset al\.,[2020](https://arxiv.org/html/2608.05797#bib.bib46)\)\. Many online variants define difficulty relative to the current learner\. Some methods, such as self\-paced and automated curriculum learning, use loss, prediction error, or temporal\-difference \(TD\) error as the difficulty signal\(Kumaret al\.,[2010](https://arxiv.org/html/2608.05797#bib.bib14); Graveset al\.,[2017](https://arxiv.org/html/2608.05797#bib.bib15); Jianget al\.,[2021](https://arxiv.org/html/2608.05797#bib.bib44)\)\. Others, such as teacher\-student curricula and unsupervised environment design, use reward, regret or success to select tasks that are neither too easy nor too hard\(Florensaet al\.,[2018](https://arxiv.org/html/2608.05797#bib.bib43); Denniset al\.,[2020](https://arxiv.org/html/2608.05797#bib.bib22); Parker\-Holderet al\.,[2022](https://arxiv.org/html/2608.05797#bib.bib45); Beukmanet al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib23); Tioet al\.,[2023](https://arxiv.org/html/2608.05797#bib.bib48); Mahrooghiet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib18); Matiisenet al\.,[2019](https://arxiv.org/html/2608.05797#bib.bib19); Baeet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib20)\)\. Nonetheless, these methods require executing tasks, often inside an additional teacher\-student loop, which can further compound the rollout overhead we aim to circumvent\.

#### Before rollout\.

More distant work on static benchmarks motivates using uncertainty signals from language models, including perplexity and entropy\(Gonenet al\.,[2023](https://arxiv.org/html/2608.05797#bib.bib24); Li,[2023](https://arxiv.org/html/2608.05797#bib.bib27); Xuet al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib49); Donget al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib26); Spiesbergeret al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib50); Zhuet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib28); Ostmeieret al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib51); Catalàet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib53); Fenget al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib52); Truonget al\.,[2025](https://arxiv.org/html/2608.05797#bib.bib76)\)\. These settings are not interactive and do not cover agentic benchmarks\. To the best of our knowledge, Agent Psychometrics is the only prior work that directly studiesex antedifficulty prediction for agentic tasks, focusing on coding benchmarks\(Geet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib11)\)\. We test broader generalization by evaluating predictors across seven additional benchmark families\. We also evaluate a broader feature set, including token\-level entropy, cross\-scorer disagreement, structural features, and embeddings\.

## 3Methods

We decompose the prediction pipeline into two steps: constructing a difficulty target from observed agent\-task outcomes, then training a predictor to estimate this target from task descriptions and other pre\-rollout features \(Figure[1](https://arxiv.org/html/2608.05797#S3.F1)\)\.

### 3\.1Dataset

Table 1:Summary of the dataset\.Our dataset consists of agent\-task outcomes from 17 agentic benchmarks, listed in Appendix[A\.1](https://arxiv.org/html/2608.05797#A1.SS1)\. Each instance records one trial of one agent configuration on one task, labeled by success under the benchmark’s scoring rules\. We use*agent*to mean a model \(LLM\) together with the scaffold through which it acts\.

Table[1](https://arxiv.org/html/2608.05797#S3.T1)summarizes the main analysis subset used throughout the main\-text results\. It contains 5,230 tasks from 17 benchmarks, evaluated by 497 agent configurations built from 216 models and 90 scaffolds\. For each task, the task, environment, and action\-space fields are stored as string descriptions derived from the benchmark, either directly or via an LLM when missing\. These benchmarks span coding and software engineering, function calling, web navigation, computer use, terminal tasks, math, ML, cybersecurity, and general\-purpose task suites\. Task dates denote the task release or source date when available and range from 2010 to 2026\. Agent dates denote the model\-weight release date when available, otherwise the first public availability date, and range from 2023 to 2026\. Each trial is treated as an independent Bernoulli observation under the benchmark scoring rules, with 1\.2 trials per agent\-task pair on average\.

### 3\.2Difficulty target

difficulty targettask featuresResponsesYa,tY\_\{a,t\}IRT fitθa,βt\\theta\_\{a\},\\beta\_\{t\}Fitted diff\.βt\\beta\_\{t\}Task signalsxtx\_\{t\}Predictorf​\(xt\)f\(x\_\{t\}\)Pred\. diff\.β^t\\hat\{\\beta\}\_\{t\}MetricM​\(𝜷^,𝜷\)M\(\\hat\{\\boldsymbol\{\\beta\}\},\\boldsymbol\{\\beta\}\)Figure 1:Overview of the difficulty prediction pipeline\.Upper branch: binary agent success and failure outcomes are used to fit IRT and estimate each task’s difficulty,βt\\beta\_\{t\}\. Lower branch: task features are used to predict difficulty,β^t\\hat\{\\beta\}\_\{t\}, and evaluation compares the predicted and IRT\-estimated difficulty vectors\.IRT is a psychometric framework that models the relationship between an individual’s latent trait \(e\.g\., knowledge, ability, anxiety\) and their responses to specific test items\(Embretson and Reise,[2025](https://arxiv.org/html/2608.05797#bib.bib36)\)\. In our case, the “individual” is the agent and the “test items” are the tasks\. Let𝒜=a1,…,aN\\mathcal\{A\}=\{a\_\{1\},\\ldots,a\_\{N\}\}denote the set of \(NN\) agent configurations and𝒯=t1,…,tM\\mathcal\{T\}=\{t\_\{1\},\\ldots,t\_\{M\}\}the set of \(MM\) tasks\. The data form a partially observed binary response matrix

Ya,t=t1t2t3⋯tMa1a2⋮aN\[10x⋯x010⋯x⋮⋮⋮⋱⋮xx1⋯1\]Y\_\{a,t\}=\\begin\{array\}\[\]\{c@\{\\;\}c\}&\\begin\{array\}\[\]\{ccccc\}t\_\{1\}&t\_\{2\}&t\_\{3\}&\\cdots&t\_\{M\}\\end\{array\}\\\\\[0\.0pt\] \\begin\{array\}\[\]\{c\}a\_\{1\}\\\\ a\_\{2\}\\\\ \\vdots\\\\ a\_\{N\}\\end\{array\}&\\left\[\\begin\{array\}\[\]\{ccccc\}1&0&\\mathrm\{x\}&\\cdots&\\mathrm\{x\}\\\\ 0&1&0&\\cdots&\\mathrm\{x\}\\\\ \\vdots&\\vdots&\\vdots&\\ddots&\\vdots\\\\ \\mathrm\{x\}&\\mathrm\{x\}&1&\\cdots&1\\end\{array\}\\right\]\\end\{array\}
where11denotes success,0denotes failure,x\\mathrm\{x\}denotes a missing evaluation, andΩ\\Omegais the set of observed agent\-task pairs\. Each row is a unique agent configuration, such as a base model with a scaffold, and each column is a unique task\. The goal is to model the probability of success by learning one scalarθa\\theta\_\{a\}for each agent and one scalarβt\\beta\_\{t\}for each task\. In the one\-parameter logistic IRT model \(1PL; Rasch model\(Rasch,[1960](https://arxiv.org/html/2608.05797#bib.bib30)\)\), this is written as

P​\(Ya,t=1∣θa,βt\)=σ​\(θa−βt\),P\(Y\_\{a,t\}=1\\mid\\theta\_\{a\},\\beta\_\{t\}\)=\\sigma\(\\theta\_\{a\}\-\\beta\_\{t\}\),\(1\)
whereθa\\theta\_\{a\}represents agent capability,βt\\beta\_\{t\}represents task difficulty andσ​\(z\)=1/\(1\+exp⁡\(−z\)\)\\sigma\(z\)=1/\(1\+\\exp\(\-z\)\)is the sigmoid function\. This form gives the desired inductive bias because success depends on the relative position of capability and difficulty on a shared scale\. Ifθa=βt\\theta\_\{a\}=\\beta\_\{t\}, capability and difficulty are balanced; ifθa\>βt\\theta\_\{a\}\>\\beta\_\{t\}, capability exceeds difficulty; and ifθa<βt\\theta\_\{a\}<\\beta\_\{t\}, difficulty exceeds capability\. The resultingβt\\beta\_\{t\}is therefore a scalar summary of apparent task difficulty as inferred from observed outcomes, and may absorb task\-side covariates such as feasibility, strictness, familiarity, and noise, just asθa\\theta\_\{a\}may absorb agent\-side covariates such as scaffold effectiveness, tool affordances, and noise\. With an interpretable predictor and an appropriate setup, this lets us ask which features account for different parts of the fitted difficulty\.

The model is trained on observed pairs\(a,t\)∈Ω\(a,t\)\\in\\Omegaby optimizing the evidence lower bound \(ELBO\) via stochastic variational inference \(SVI\)\(Hoffmanet al\.,[2013](https://arxiv.org/html/2608.05797#bib.bib33)\)and the reparameterization trick\(Kingma and Welling,[2013](https://arxiv.org/html/2608.05797#bib.bib32)\)\. We place zero\-centered Gaussian priors over all parameters and approximate the posterior distribution over these parameters with independent Gaussian variational distributions:

q​\(θa\)=𝒩​\(μθa,σθa2\),q​\(βt\)=𝒩​\(μβt,σβt2\)\.q\(\\theta\_\{a\}\)=\\mathcal\{N\}\(\\mu\_\{\\theta\_\{a\}\},\\sigma\_\{\\theta\_\{a\}\}^\{2\}\),\\qquad q\(\\beta\_\{t\}\)=\\mathcal\{N\}\(\\mu\_\{\\beta\_\{t\}\},\\sigma\_\{\\beta\_\{t\}\}^\{2\}\)\.Because the model depends strictly on the differenceθa−βt\\theta\_\{a\}\-\\beta\_\{t\}, the parameters can shift arbitrarily without changing the probabilities\. To resolve this, we anchor the scale by centering the abilitiesθa\\theta\_\{a\}at zero\. We then use the resulting task difficulty estimates \(β\\beta\) as the target for downstream prediction\. The fitted scores achieve a response AUC of0\.9370\.937on observed pairs, indicating that the 1PL model captures much of the observed response structure\. Finally, as difficulties vary across different benchmarks, we compute a standardized version of these values within each benchmark to obtainz​\(βt\)z\(\\beta\_\{t\}\)\. More details are provided in Appendix[A\.2](https://arxiv.org/html/2608.05797#A1.SS2)\.

### 3\.3Task features

The target is the IRT\-estimated difficultyβt\\beta\_\{t\}, which we standardize within\-benchmark toz​\(βt\)z\(\\beta\_\{t\}\)for our main experiments, as described above\. The practical constraint is that every feature must be less expensive to compute than running rollouts on the task\. LetCtC\_\{t\}denote the full task\-side context for tasktt, consisting of the environment description, action\-space description, and task description\. Feature families useCtC\_\{t\}either directly or through a reasoning traceRtR\_\{t\}generated fromCtC\_\{t\}by Claude Sonnet 4\.6 with a 4k\-token cap\.

We use five feature sets throughout the experiments\. The*random*control samplesxt∼𝒩​\(0,1\)x\_\{t\}\\sim\\mathcal\{N\}\(0,1\)independently from the task\. The*baseline*setting uses only the context length,xtlen=log⁡\(1\+\|Ct\|\)x\_\{t\}^\{\\text\{len\}\}=\\log\(1\+\|C\_\{t\}\|\), to control for the amount of task text\. The*embedding*family maps the reasoning trace toet=ϕ​\(Rt\)e\_\{t\}=\\phi\(R\_\{t\}\), whereϕ\\phiisbge\-base\-en\-v1\.5\. We reduce this representation with fold\-aware PCA to avoid overfitting\.

The*entropy*setting uses an open\-weight scorer model\. GivenRt=\(r1,…,rL\)R\_\{t\}=\(r\_\{1\},\\ldots,r\_\{L\}\), the scorer definespℓ​\(v\)=p​\(v∣Ct,r<ℓ\)p\_\{\\ell\}\(v\)=p\(v\\mid C\_\{t\},r\_\{<\\ell\}\)at each positionℓ\\ell, wherevvis a token from the model’s vocabulary\. We compute Shannon entropy\(Shannon,[1948](https://arxiv.org/html/2608.05797#bib.bib54)\)as

Hℓ=−∑vpℓ​\(v\)​log⁡pℓ​\(v\),H\_\{\\ell\}=\-\\sum\_\{v\}p\_\{\\ell\}\(v\)\\log p\_\{\\ell\}\(v\),using the renormalized top\-100 next\-token distribution for faster computation\. The primary entropy configuration aggregates\(H1,…,HL\)\(H\_\{1\},\\ldots,H\_\{L\}\)from Qwen3\-8B\-Base over the bounded reasoning traceRtR\_\{t\}\. The*full*setting concatenates entropy summaries from a five\-model scorer panel, cross\-scorer disagreement features, benchmark metadata, structural properties ofCtC\_\{t\}, and the embedding features defined above\. Appendix[A\.3](https://arxiv.org/html/2608.05797#A1.SS3)gives the scorer panel\.

To see that these features satisfy the cost constraint from above, consider a 100\-task benchmark\. Establishing empirical difficulty at a density of 10 outcomes per task requires 1,000 full executions\. Assuming a representative 100,000\-token context per execution\(Baiet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib75)\), this process demands approximately1\.0×1081\.0\\times 10^\{8\}input tokens\. In contrast, estimating difficultyex anteusing the features proposed here \(accounting for the reasoning trace\) consumes roughly4\.0×1064\.0\\times 10^\{6\}tokens for the same 100 tasks\. Under these assumptions, the feature pass uses25×25\\timesfewer tokens than empirical difficulty estimation, and the efficiency grows with the number of outcomes required per task\.

## 4Evaluation setup

### 4\.1Prediction metrics

The preceding sections define a target task difficultyβt\\beta\_\{t\}, obtained from observed agent\-task outcomes, and a predicted difficultyβ^t=f​\(xt\)\\hat\{\\beta\}\_\{t\}=f\(x\_\{t\}\), obtained from task features available before running rollouts\. The evaluation asks how wellβ^t\\hat\{\\beta\}\_\{t\}recoversβt\\beta\_\{t\}\.

We report Spearman’sρ\\rhobetween predicted and IRT\-estimated task difficulty\. We useρβ\\rho\_\{\\beta\}for raw fitted difficulty andρz​\(β\)\\rho\_\{z\(\\beta\)\}for within\-benchmark standardized difficulty\. We also report within\-benchmark pairwise accuracy \(pAcc\\mathrm\{pAcc\}\) as a complementary ordering metric\. Letqtq\_\{t\}denote the target being evaluated, eitherβt\\beta\_\{t\}orz​\(βt\)z\(\\beta\_\{t\}\), and letq^t\\hat\{q\}\_\{t\}denote its prediction\. ThenpAcc\\mathrm\{pAcc\}is the fraction of within\-benchmark task pairs for which the predicted ordering matches the target ordering:

pAcc=1\|𝒫\|​∑\(i,j\)∈𝒫𝟙​\[\(q^i−q^j\)​\(qi−qj\)\>0\],\\mathrm\{pAcc\}=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}\}\\mathbb\{1\}\\left\[\(\\hat\{q\}\_\{i\}\-\\hat\{q\}\_\{j\}\)\(q\_\{i\}\-q\_\{j\}\)\>0\\right\],\(2\)
where𝒫\\mathcal\{P\}is the set of task pairs within the same benchmark\. K\-fold evaluation \(KF\) measures in\-distribution performance, where tasks from the target distribution are available during training\. Leave\-one\-benchmark\-out \(LOBO\) measures out\-of\-distribution performance, where the target benchmark is unseen during training\.

### 4\.2AUC can be misleading

Prior work\(Geet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib11)\)evaluates IRT\-style predictors through AUC\. Using its probabilistic interpretation, AUC measures the probability that a randomly chosen successful attempt receives a higher score than a randomly chosen unsuccessful attempt\. For a successful attempt by agentaaon tasktt, and an unsuccessful attempt by agenta′a^\{\\prime\}on taskt′t^\{\\prime\}, the IRT score gives

AUC\\displaystyle\\mathrm\{AUC\}=P​\(θa−β^t\>θa′−β^t′\)\\displaystyle=P\(\\theta\_\{a\}\-\\hat\{\\beta\}\_\{t\}\>\\theta\_\{a^\{\\prime\}\}\-\\hat\{\\beta\}\_\{t^\{\\prime\}\}\)\(3\)=P​\(θa−θa′\>β^t−β^t′\)\.\\displaystyle=P\(\\theta\_\{a\}\-\\theta\_\{a^\{\\prime\}\}\>\\hat\{\\beta\}\_\{t\}\-\\hat\{\\beta\}\_\{t^\{\\prime\}\}\)\.
This expression shows that AUC here is affected by both the difference in agent abilities and the difference in predicted task difficulties\. Consider a degenerate case where the predictor simply assigns the same prediction to all tasks\. This forcesβ^t=β^t′\\hat\{\\beta\}\_\{t\}=\\hat\{\\beta\}\_\{t^\{\\prime\}\}for every pair of sampled attempts, meaning

AUC=P​\(θa\>θa′\)\.\\mathrm\{AUC\}=P\(\\theta\_\{a\}\>\\theta\_\{a^\{\\prime\}\}\)\.\(4\)
Thus, AUC can remain high even when the predictor contains no task\-specific difficulty information\. We demonstrate this failure mode empirically on the observed response matrix by comparing fixed difficulty assignments against the fitted IRT target and observed responses, shown in Table[2](https://arxiv.org/html/2608.05797#S4.T2)\.

Table 2:Response AUC remains high for poor difficulty estimates\.Benchmark avg\. assigns each task its benchmark\-average\.ρβ\\rho\_\{\\beta\}andρz​\(β\)\\rho\_\{z\(\\beta\)\}compare against raw and within\-benchmark\-standardizedβ\\beta\. pAcc is within\-benchmark pairwise accuracy\. Dashes indicate undefined\. Note that this is illustrative, all later predictions targetz​\(β\)z\(\\beta\)instead ofβ\\beta\.The oracle row uses the IRT\-estimatedβ\\beta, establishing an upper bound of 0\.937 AUC for this response matrix\. Predicting a single constant difficulty for every task still yields an AUC of 0\.715, confirming the constant difficulty case in Equation[4](https://arxiv.org/html/2608.05797#S4.E4)\. This inflation is further exaggerated by the benchmark\-average estimator\. Simply assigning every task the mean difficulty of its parent benchmark achieves an AUC of 0\.806 and a non\-trivial rank correlation \(ρβ\\rho\_\{\\beta\}of 0\.519\)\. This estimator captures broad benchmark\-level shifts, but it cannot differentiate taskswithina benchmark, giving a strict pAcc of 0\.000\. Because the practical goal is to help benchmark builders rank and select candidate tasks within their specific evaluation settings, recovering this intra\-benchmark signal is essential\. To remove the benchmark\-identity confound and evaluate task\-level difficulty more directly, all subsequent training and prediction target within\-benchmark standardized difficulty,z​\(β\)z\(\\beta\)\.

## 5Results

### 5\.1Difficulty is predictable from task features

![Refer to caption](https://arxiv.org/html/2608.05797v1/x1.png)Figure 2:Difficulty prediction performance\.The target isz​\(β\)z\(\\beta\)\. Spearmanρ\\rhois shown under 50\-fold KF and LOBO evaluation\. Error bars show split standard deviations\. With 50 folds, each test fold contains roughly 100 tasks, leading to substantial uncertainty\. Stars mark one\-sided paired Wilcoxon tests against the baseline at significance levelsα∗=0\.05\{\}^\{\*\}\\alpha=0\.05,α∗∗=0\.01\{\}^\{\*\*\}\\alpha=0\.01, andα∗⁣∗∗=0\.001\{\}^\{\*\*\*\}\\alpha=0\.001\.Figure[2](https://arxiv.org/html/2608.05797#S5.F2)reports rank recovery across the 17\-benchmark corpus using Spearmanρ\\rhoand ridge regression \(Appendix[A\.4](https://arxiv.org/html/2608.05797#A1.SS4)compares alternative estimators\)\. KF measures in\-distribution performance, where similar tasks are available during training\. LOBO measures out\-of\-distribution performance on an unseen target benchmark\. The prediction target isz​\(β\)z\(\\beta\)\.

The random control stays near zero, while the length baseline captures only weak signal withρ=0\.086\\rho=0\.086under KF andρ=0\.101\\rho=0\.101under LOBO\. The entropy features improve over this baseline, reachingρ=0\.193\\rho=0\.193under KF andρ=0\.137\\rho=0\.137under LOBO\. Embedding features are weaker in this setup, withρ=0\.107\\rho=0\.107under KF andρ=0\.017\\rho=0\.017under LOBO, close to the length baseline\. This suggests that these semantic representations capture little additional difficulty signal here, although stronger embedding models may change this conclusion\. Combining feature families gives the strongest KF result,ρ=0\.399\\rho=0\.399\. Under LOBO, the same full model reachesρ=0\.225\\rho=0\.225\. Thus, task features encode real difficulty signal, but transfer to unseen benchmarks remains limited\. Taken together, the results point to a practical near\-term setting in which an evaluated task suite already exists \(with executed rollouts\) and new candidate tasks can be ranked with moderate confidence\.

### 5\.2Entropy ablations

![Refer to caption](https://arxiv.org/html/2608.05797v1/x2.png)

Task \(a\)\.Consider an axis\-aligned44dimensional cube with side length44, subdivided into444^\{4\}unit cubes\. Cubey starts at a random unit cube\. A fish is independently placed in a uniformly random unit cube,possibly the sameas Cubey’s starting position\. Compute theexpected timefor Cubey to catch the fish, assuming hemoves optimally\.

Figure 3:Entropy profiles and feature ablations\.\(a\)An example task from MathArena, showing the entropy profile over its span\. Colored lines are different models, black is their average, and the textual excerpt is given in the blue box\.\(b\)Scorer\-count ablation using one fixed order\.\(c\)Shows an ablation of feature derivation from the entropy profile\.As illustrated in Figure[3](https://arxiv.org/html/2608.05797#S5.F3)\(a\), entropy can be considered a trajectory rather than a static metric\. Following the principle that conditioning reduces entropy\(Cover,[1999](https://arxiv.org/html/2608.05797#bib.bib55)\), additional context should reduce expected next\-token uncertainty\. Consequently, global statistics such as the mean and variance may miss local structure in the entropy profile\. For instance, a highly unexpected, difficulty\-inducing condition that appears late in the context may have a lower entropy magnitude than the initial tokens\. Thus, any aggregate \(compression\) is susceptible to losing this information\.

The critical information may therefore lie in local discrepancies and structural anomalies\. Figure[3](https://arxiv.org/html/2608.05797#S5.F3)\(c\)supports this intuition\. Beyond the mean,*spread*adds distributional summaries such as standard deviation, quantiles, and top\-kkmeans,*shape*adds trajectory summaries such as slope, first\- and last\-decile means, and total variation, and*gaps*adds pairwise absolute differences between scorer summaries\. These trajectory summaries improve in\-distribution performance over mean\-only entropy while preserving or improving LOBO performance\. This suggests that the shape of the entropy trajectory carries useful information\. Future work could model entropy explicitly as a time series, for example with frequency\-domain or learned sequence features\. With sufficient data, deep learning architectures such as 1D\-CNNs or RNNs could also learn such features directly\.

At the same time, ensembling models introduces a counterintuitive dynamic\. Figure[3](https://arxiv.org/html/2608.05797#S5.F3)\(b\)shows that increasing the number of scorers strengthens the in\-distribution performance, while LOBO performance is non\-monotonic and does not improve in the same way\. One plausible explanation is benchmark\-specific overfitting\. Averaging across multiple models reduces scorer\-specific noise, but it may also reinforce shared sensitivity to benchmark\-specific phrasing and formatting, limiting out\-of\-distribution transfer\.

### 5\.3Contamination and feasibility

One diagnostic use ofex antedifficulty prediction is to compare predicted and observed difficulty after rollouts\. Letz​\(βt\)z\(\\beta\_\{t\}\)be the within\-benchmark standardized difficulty implied by rollouts, and letz^t\\hat\{z\}\_\{t\}be the predicted task difficulty from the full ridge model in Section[5\.1](https://arxiv.org/html/2608.05797#S5.SS1)\. Their difference

rt=z​\(βt\)−z^tr\_\{t\}=z\(\\beta\_\{t\}\)\-\\hat\{z\}\_\{t\}\(5\)
is the unexplained difficulty\. Ifrt\>0r\_\{t\}\>0, agents found the task harder than the model predicted\. Ifrt<0r\_\{t\}<0, agents found the task easier than the model predicted\. The second case is the direction expected under familiarity or contamination, where agents succeed more often than the predicted difficulty would suggest\. The first case is the direction expected when some missing factor, such as feasibility, makes the rollout harder than the task description suggests\.

To illustrate this idea, we perform a small behavioral case study\. An OpenAI audit reports that59\.4%59\.4\\%of 138 frequently failed SWE\-bench Verified tasks had material test\-design or problem\-description issues, and it also reports evidence of contamination for several tasks\. Two of these audited examples appear in our dataset\. For*django\_\_django\-14725*, flagged for missing prompt information, the residual equation yieldsrt=1\.46−0\.05=\+1\.41r\_\{t\}=1\.46\-0\.05=\+1\.41\. The task has success rate0\.0900\.090and IRT estimated difficulty ofβ=2\.58\\beta=2\.58, above the9090th percentile within SWE\-bench Verified\. For*django\_\_django\-11451*, discussed as a contamination example, the residual equation givesrt=−0\.51−0\.05=−0\.56r\_\{t\}=\-0\.51\-0\.05=\-0\.56\. The task has success rate0\.7180\.718and IRT\-estimated difficultyβ=−2\.35\\beta=\-2\.35, below the benchmark median\. Both residual signs match the intended interpretation\. Unexpectedly hard tasks point toward feasibility or evaluator mismatch, while unexpectedly easy tasks point toward familiarity or contamination\.

Because this residual uses empirical difficultyβt\\beta\_\{t\}from full agent rollouts, it supports detection rather than pure pre\-rollout prediction\. It remains an open question how far this mechanism can be pushed to automatically audit benchmark integrity and whether such anomalies can eventually be predicted without requiring execution at all\.

## 6Conclusion

We investigate whether the difficulty of agentic tasks can be estimated prior to execution\. Our results show both promise and clear limits\. Task features, particularly the sequential structure of token\-level entropy, capture a meaningful difficulty signal, yet the results still reveal a clear divide between in\- and out\-of\-distribution performance\. While in\-distribution rank recovery proves reliable, transferring these predictions to unseen benchmarks remains an open problem\. We also show that AUC is unreliable for this setting\. Because it conflates agent ability with task difficulty, it can remain artificially high even when estimates are degenerate, underscoring the need for rank\-based evaluation\.

Overall, we believe this work provides a foundation for more proactive environment design\. If refined,ex anteprediction could serve as a diagnostic filter by using the residual gap between expected and observed difficulty to audit tasks for anomalies, including data contamination or structural infeasibility, before committing to exhaustive rollouts\. Future work should focus on improving cross\-benchmark generalization, for example by modeling entropy as a time series or by developing more mechanistic predictors of task familiarity and feasibility\. As the cost of simulating frontier agents grows, such predictive diagnostics will become increasingly important for both training and evaluation\.

## References

- S\. Bae, J\. Hong, M\. Y\. Lee, H\. Kim, J\. Nam, and D\. Kwak \(2026\)Online difficulty filtering for reasoning oriented reinforcement learning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 700–719\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Bai, Z\. Huang, X\. Wang, J\. Sun, R\. Mihalcea, E\. Brynjolfsson, A\. Pentland, and J\. Pei \(2026\)How do ai agents spend your money? analyzing and predicting token consumption in agentic coding tasks\.arXiv preprint arXiv:2604\.22750\.Cited by:[§3\.3](https://arxiv.org/html/2608.05797#S3.SS3.p4.3)\.
- M\. Balunovic, J\. Dekoninck, I\. Petrov, N\. Jovanović, and M\. Vechev \(2026\)Matharena: evaluating llms on uncontaminated math competitions\.Advances in Neural Information Processing Systems38\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.10.9.1)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InProceedings of the 26th annual international conference on machine learning,pp\. 41–48\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p3.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Beukman, S\. Coward, M\. Matthews, M\. Fellows, M\. Jiang, M\. Dennis, and J\. Foerster \(2024\)Refining minimax regret for unsupervised environment design\.arXiv preprint arXiv:2402\.12284\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- M\. G\. I\. Català, H\. S\. d\. O\. Borde, G\. D\. Montañez, and P\. Liò \(2026\)The stepwise informativeness assumption: why are entropy dynamics and reasoning correlated in llms?\.arXiv preprint arXiv:2604\.06192\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan,et al\.\(2025\)Mle\-bench: evaluating machine learning agents on machine learning engineering\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 50466–50494\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.11.10.1)\.
- T\. M\. Cover \(1999\)Elements of information theory\.John Wiley & Sons\.Cited by:[§5\.2](https://arxiv.org/html/2608.05797#S5.SS2.p1.1)\.
- X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane,et al\.\(2025\)Swe\-bench pro: can ai agents solve long\-horizon software engineering tasks?\.arXiv preprint arXiv:2509\.16941\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.15.14.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2web: towards a generalist agent for the web\.Advances in Neural Information Processing Systems36,pp\. 28091–28114\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.12.11.1)\.
- M\. Dennis, N\. Jaques, E\. Vinitsky, A\. Bayen, S\. Russell, A\. Critch, and S\. Levine \(2020\)Emergent complexity and zero\-shot transfer via unsupervised environment design\.Advances in neural information processing systems33,pp\. 13049–13061\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Dong, X\. Jiang, H\. Liu, Z\. Jin, B\. Gu, M\. Yang, and G\. Li \(2024\)Generalization or memorization: data contamination and trustworthy evaluation for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12039–12050\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- S\. E\. Embretson and S\. P\. Reise \(2025\)Item response theory: foundations for psychologists and social scientists\.Routledge\.Cited by:[§3\.2](https://arxiv.org/html/2608.05797#S3.SS2.p1.4)\.
- X\. Feng, V\. Veeriah, M\. Chiam, M\. Dennis, F\. Barbero, J\. Obando Ceron, J\. Shi, S\. Singh, S\. Hou, N\. Tomasev,et al\.\(2026\)Generating creative chess puzzles\.Advances in Neural Information Processing Systems38,pp\. 49539–49605\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Florensa, D\. Held, X\. Geng, and P\. Abbeel \(2018\)Automatic goal generation for reinforcement learning agents\.InInternational conference on machine learning,pp\. 1515–1528\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Ge, D\. Kryvosheieva, D\. Fried, U\. Girit, and K\. Hariharan \(2026\)Agent psychometrics: task\-level performance prediction in agentic coding benchmarks\.arXiv preprint arXiv:2604\.00594\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p4.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2608.05797#S4.SS2.p1.4)\.
- H\. Gonen, S\. Iyer, T\. Blevins, N\. A\. Smith, and L\. Zettlemoyer \(2023\)Demystifying prompts in language models via perplexity estimation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10136–10148\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§A\.3](https://arxiv.org/html/2608.05797#A1.SS3.p1.1)\.
- A\. Graves, M\. G\. Bellemare, J\. Menick, R\. Munos, and K\. Kavukcuoglu \(2017\)Automated curriculum learning for neural networks\.Ininternational conference on machine learning,pp\. 1311–1320\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Helff, Q\. Delfosse, D\. Steinmann, R\. Härle, H\. Shindo, P\. Schramowski, W\. Stammer, K\. Kersting, and F\. Friedrich \(2026\)LLMs gaming verifiers: rlvr can lead to reward hacking\.arXiv preprint arXiv:2604\.15149\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1)\.
- A\. Ho, J\. Denain, D\. Atanasov, S\. Albanie, and R\. Shah \(2025\)A rosetta stone for ai benchmarks\.arXiv preprint arXiv:2512\.00193\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px1.p1.1)\.
- M\. D\. Hoffman, D\. M\. Blei, C\. Wang, and J\. Paisley \(2013\)Stochastic variational inference\.Journal of machine learning research\.Cited by:[§3\.2](https://arxiv.org/html/2608.05797#S3.SS2.p4.1)\.
- N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)Livecodebench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.9.8.1)\.
- M\. Jiang, E\. Grefenstette, and T\. Rocktäschel \(2021\)Prioritized level replay\.InInternational Conference on Machine Learning,pp\. 4940–4950\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.16.15.1)\.
- D\. P\. Kingma and M\. Welling \(2013\)Auto\-encoding variational bayes\.arXiv preprint arXiv:1312\.6114\.Cited by:[§3\.2](https://arxiv.org/html/2608.05797#S3.SS2.p4.1)\.
- Y\. Koren, R\. Bell, and C\. Volinsky \(2009\)Matrix factorization techniques for recommender systems\.Computer42\(8\),pp\. 30–37\.Cited by:[§A\.2](https://arxiv.org/html/2608.05797#A1.SS2.SSS0.Px1.p1.3)\.
- M\. Kumar, B\. Packer, and D\. Koller \(2010\)Self\-paced learning for latent variable models\.Advances in neural information processing systems23\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Kwa, B\. West, J\. Becker, A\. Deng, K\. Garcia, M\. Hasin, S\. Jawhar, M\. Kinniment, N\. Rush, S\. Von Arx,et al\.\(2026\)Measuring ai ability to complete long software tasks\.Advances in Neural Information Processing Systems38,pp\. 92213–92266\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.14.13.1),[§1](https://arxiv.org/html/2608.05797#S1.p1.1),[§1](https://arxiv.org/html/2608.05797#S1.p3.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li \(2023\)Estimating contamination via perplexity: quantifying memorisation in language model evaluation\.URL https://arxiv\. org/abs/2309\.10677\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- F\. Liu, J\. Gala, D\. Bahdanau, S\. Reddy, H\. Larochelle,et al\.\(2026\)Bridge: predicting human task completion time from model performance\.arXiv preprint arXiv:2602\.07267\.Cited by:[§A\.5](https://arxiv.org/html/2608.05797#A1.SS5.p1.4),[Table 5](https://arxiv.org/html/2608.05797#A1.T5),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Mahrooghi, A\. Lotfi, and E\. Abbe \(2026\)Goldilocks rl: tuning task difficulty to escape sparse rewards for reasoning\.arXiv preprint arXiv:2602\.14868\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Matiisen, A\. Oliver, T\. Cohen, and J\. Schulman \(2019\)Teacher–student curriculum learning\.IEEE transactions on neural networks and learning systems31\(9\),pp\. 3732–3740\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Mazar, O\. Amir, and D\. Ariely \(2008\)The dishonesty of honest people: a theory of self\-concept maintenance\.Journal of marketing research45\(6\),pp\. 633–644\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.17.16.1)\.
- T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.\(2025\)Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§A\.3](https://arxiv.org/html/2608.05797#A1.SS3.p1.1)\.
- S\. Ostmeier, B\. Axelrod, M\. Varma, A\. Aali, Y\. Zhang, M\. Paschali, S\. Koyejo, C\. Langlotz, and A\. Chaudhari \(2026\)Attention head entropy of llms predicts answer correctness\.arXiv preprint arXiv:2602\.13699\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Parker\-Holder, M\. Jiang, M\. Dennis, M\. Samvelyan, J\. Foerster, E\. Grefenstette, and T\. Rocktäschel \(2022\)Evolving curricula with regret\-based environment design\.InInternational Conference on Machine Learning,pp\. 17473–17498\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.2.1.1),[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.3.2.1)\.
- T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. P\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet,et al\.\(2025\)Gdpval: evaluating ai model performance on real\-world economically valuable tasks\.arXiv preprint arXiv:2510\.04374\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.5.4.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.8.7.1)\.
- R\. Portelas, C\. Colas, L\. Weng, K\. Hofmann, and P\. Oudeyer \(2020\)Automatic curriculum learning for deep rl: a short survey\.arXiv preprint arXiv:2003\.04664\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Rasch \(1960\)Probabilistic models for some intelligence and attainment tests\.Copenhagen: Institute of Education Research,Copenhagen\.Cited by:[§3\.2](https://arxiv.org/html/2608.05797#S3.SS2.p2.6)\.
- D\. Rein, J\. Becker, A\. Deng, S\. Nix, C\. Canal, D\. O’Connel, P\. Arnott, R\. Bloom, T\. Broadley, K\. Garcia,et al\.\(2025\)Hcast: human\-calibrated autonomy software tasks\.arXiv preprint arXiv:2503\.17354\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.7.6.1)\.
- C\. E\. Shannon \(1948\)A mathematical theory of communication\.The Bell system technical journal27\(3\),pp\. 379–423\.Cited by:[§3\.3](https://arxiv.org/html/2608.05797#S3.SS3.p3.4)\.
- M\. Shetty, N\. Jain, J\. Liu, V\. Kethanaboyina, K\. Sen, and I\. Stoica \(2026\)Gso: challenging software optimization tasks for evaluating swe\-agents\.Advances in Neural Information Processing Systems38\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.6.5.1)\.
- J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward gaming\.Advances in Neural Information Processing Systems35,pp\. 9460–9471\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1)\.
- A\. Spiesberger, J\. J\. Vazquez, N\. Pochinkov, T\. Gavenčiak, P\. Grietzer, G\. Leech, and N\. Schoots \(2026\)Soft contamination means benchmarks test shallow generalization\.arXiv preprint arXiv:2602\.12413\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Tio, J\. Ho, and P\. Varakantham \(2023\)Training reinforcement learning agents and humans with difficulty\-conditioned generators\.arXiv preprint arXiv:2312\.02309\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Truong, Y\. Tu, P\. Liang, B\. Li, and S\. Koyejo \(2025\)Reliable and efficient amortized model\-based evaluation\.arXiv preprint arXiv:2503\.13335\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- L\. S\. Vygotsky \(1978\)Mind in society: the development of higher psychological processes\.Harvard University Press,Cambridge, MA\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Wijk, T\. Lin, J\. Becker, S\. Jawhar, N\. Parikh, T\. Broadley, L\. Chan, M\. Chen, J\. Clymer, J\. Dhyani,et al\.\(2024\)Re\-bench: evaluating frontier ai r&d capabilities of language model agents against human experts\.arXiv preprint arXiv:2411\.15114\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.13.12.1)\.
- C\. Xu, S\. Guan, D\. Greene, M\. Kechadi,et al\.\(2024\)Benchmark data contamination of large language models: a survey\.arXiv preprint arXiv:2406\.04244\.Cited by:[§1](https://arxiv.org/html/2608.05797#S1.p2.1),[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.
- F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao,et al\.\(2026\)Theagentcompany: benchmarking llm agents on consequential real world tasks\.Advances in Neural Information Processing Systems38\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.18.17.1)\.
- T\. Xue, W\. Qi, T\. Shi, C\. H\. Song, B\. Gou, D\. Song, H\. Sun, and Y\. Su \(2025\)An illusion of progress? assessing the current state of web agents\.arXiv preprint arXiv:2504\.01382\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.12.11.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§A\.3](https://arxiv.org/html/2608.05797#A1.SS3.p1.1)\.
- A\. K\. Zhang, N\. Perry, R\. Dulepet, J\. Ji, C\. Menders, J\. Lin, E\. Jones, G\. Hussein, S\. Liu, D\. Jasper,et al\.\(2025\)Cybench: a framework for evaluating cybersecurity capabilities and risks of language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 25094–25243\.Cited by:[Table 3](https://arxiv.org/html/2608.05797#A1.T3.1.4.3.1)\.
- C\. Zhu, S\. Wu, X\. Zeng, Z\. Xu, Z\. Kang, Y\. Guo, Y\. Lu, J\. Huang, and G\. Zhou \(2026\)EDIS: diagnosing llm reasoning via entropy dynamics\.arXiv preprint arXiv:2602\.01288\.Cited by:[§2](https://arxiv.org/html/2608.05797#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix ASupplementary material

### A\.1Benchmark list

Table 3:Benchmarks and primary domains\.
### A\.2IRT fitting details

The IRT model is fit on the observed binary entries of the agent\-task response matrix\. We place zero\-centered Gaussian priors with learned global scales on each agent abilityθa\\theta\_\{a\}and task difficultyβt\\beta\_\{t\}, approximate the posterior for each scalar with a normal variational factor, and optimize the evidence lower bound with stochastic variational inference\. The posterior means are then used as the operational ability and difficulty estimates\. The fitted scoresθa−βt\\theta\_\{a\}\-\\beta\_\{t\}achieve response AUC0\.9370\.937on observed pairs, indicating that the fitted scale captures the response matrix well enough to define the difficulty target used in the paper\.

#### Fitting rationale\.

As in matrix factorization\(Korenet al\.,[2009](https://arxiv.org/html/2608.05797#bib.bib31)\), only observed response\-matrix entries contribute to the fit\. A direct joint maximum\-likelihood \(JML\) fit would initialize all abilities and difficulties and minimize binary cross\-entropy over observed trials\. This point\-estimate fit is fragile under imbalance\. Parameters with few observations become noisy, and the loss sees only the error inθa−βt\\theta\_\{a\}\-\\beta\_\{t\}\. Ifθa\\theta\_\{a\}is poorly estimated, a mispredicted trial can still push the pairedβt\\beta\_\{t\}in the opposite direction, transferring noise between ability and difficulty\.

The natural correction is to avoid committing to a single latent value too early\. In the standard ability\-marginalization view, for fixed task difficulties, each agent contribution is obtained by averaging over plausible abilities\. LetΩa=\{t:\(a,t\)∈Ω\}\\Omega\_\{a\}=\\\{t:\(a,t\)\\in\\Omega\\\}be the observed tasks for agentaa\. Then

p​\(Ya,Ωa∣𝜷\)=∫p​\(θa\)​∏t∈Ωap​\(Ya,t∣θa,βt\)​d​θa\.p\(Y\_\{a,\\Omega\_\{a\}\}\\mid\\boldsymbol\{\\beta\}\)=\\int p\(\\theta\_\{a\}\)\\prod\_\{t\\in\\Omega\_\{a\}\}p\(Y\_\{a,t\}\\mid\\theta\_\{a\},\\beta\_\{t\}\)\\,d\\theta\_\{a\}\.A grid approximation withKKsupport points requiresO​\(K​\|Ω\|\)O\(K\|\\Omega\|\)likelihood evaluations per iteration, about4\.15×1064\.15\\times 10^\{6\}evaluations forK=10K=10at our scale, before repeated iterations\. The variational fit replaces this grid enumeration with learned Gaussian approximate posteriors\. The posterior means provide point estimates, while the posterior scales keep an uncertainty estimate for each ability and difficulty\.

#### Implementation details\.

The fit uses a 1PL variational IRT implementation with Trace\_ELBO optimized by ClippedAdam for 2,000 epochs, learning rate0\.010\.01, gradient clipping at norm5\.05\.0, and random seed4242\. The saved state containsθ\\theta,β\\beta, and their posterior scales for 502 agents and 7,266 tasks before filtering to the main analysis subset\.

![Refer to caption](https://arxiv.org/html/2608.05797v1/x3.png)Figure 4:Fitted IRT ability and difficulty distributions\.Kernel\-density estimates are shown for item difficultyβ\\betaand agent abilityθ\\thetabefore filtering to the main analysis subset\.

### A\.3Entropy scorer panel

The single\-scorer*entropy*setting uses Qwen3\-8B\-Base\. The*full*setting uses the following five\-scorer panel, in this fixed order for the scorer\-count ablation: Qwen3\-8B\-Base; Qwen3\-8B\-Instruct, abbreviated as Qwen3\-8B in Figure[3](https://arxiv.org/html/2608.05797#S5.F3)\(Yanget al\.,[2025](https://arxiv.org/html/2608.05797#bib.bib56)\); Gemma 4 E4B, abbreviated as Gemma\-E4B\-it in Figure[3](https://arxiv.org/html/2608.05797#S5.F3); OLMo 3 7B Instruct, abbreviated as OLMo\-3\-7B\-Inst\(Olmoet al\.,[2025](https://arxiv.org/html/2608.05797#bib.bib58)\); and Llama 3\.1 8B Instruct, abbreviated as Llama\-3\.1\-8B\-Inst\(Grattafioriet al\.,[2024](https://arxiv.org/html/2608.05797#bib.bib57)\)\.

### A\.4Estimator ablation

Figure[2](https://arxiv.org/html/2608.05797#S5.F2)fixes the prediction protocol to compare feature families\. Table[4](https://arxiv.org/html/2608.05797#A1.T4)instead fixes the full feature representation and varies the regressor\. Random forests improve KF performance slightly, reachingρ=0\.475\\rho=0\.475and pairwise accuracy0\.6830\.683\. Linear regression gives the strongest LOBO result, withρ=0\.295\\rho=0\.295and pairwise accuracy0\.6020\.602\. The ablation therefore leaves the main conclusion unchanged\. Estimator choice affects interpolation within the pooled dataset, while benchmark transfer remains the limiting case\.

Table 4:Estimator choice does not remove the held\-out\-benchmark gap\. Lin\.=linear regression, RF=random forest, XGB=XGBoost\. KF uses 10 folds; LOBO leaves out one benchmark\. pAcc denotes within\-benchmark pairwise accuracy\. Entries report mean±\\pmstandard deviation across folds or held\-out benchmarks\.
### A\.5Human\-time calibration

Table[5](https://arxiv.org/html/2608.05797#A1.T5)calibrates fitted IRT difficulty tolog\\logmedian human completion time on the subset with human\-time annotations, following the BRIDGE framing\(Liuet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib12)\)\. Linear and ridge regression perform nearly identically, with KFρ=0\.422\\rho=0\.422and MAE0\.8570\.857\. XGBoost gives the lowest KF MAE,0\.8490\.849, but weaker LOBO rank correlation\. These results are auxiliary to the main paper, but they indicate that the fitted difficulty scale can be connected to human\-interpretable task time when such annotations are available\. Practically, accurate pre\-rollout difficulty prediction could help researchers estimate human\-time requirements more efficiently\.

Table 5:Human\-time calibration of fitted IRT difficulty, following the BRIDGE framing\(Liuet al\.,[2026](https://arxiv.org/html/2608.05797#bib.bib12)\)\. Lin\.=linear regression, RF=random forest, XGB=XGBoost\. KF uses 5 folds\. LOBO leaves out one benchmark\. Uncertainties are standard deviations across folds\. MAE is measured on thelog\\logmedian\-human\-minutes target\.
### A\.6OLMo surprisal

To sanity\-check the use of language\-model likelihood as a lightweight surprise signal, we scored a small set of prompt categories withallenai/OLMo\-1B\-hf\. The experiment uses only short text snippets and should be read as a qualitative diagnostic rather than a benchmark result\. For each prompt, we compute mean token surprisal,−log⁡p\-\\log p, in nats per token under the model\. Higher bars, therefore, indicate text that is more surprising to the scorer model\.

![Refer to caption](https://arxiv.org/html/2608.05797v1/x4.png)Figure 5:Mean token surprisal under OLMo\-1B across prompt categories, measured in nats per token\. Bars show means over five prompts per category; error bars show standard deviations\. Random\-token strings are included as a high\-surprise control\.