Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

arXiv cs.AI Papers

Summary

A Tsinghua/Delft paper presents a fast empirical-Bayes Beta-Bernoulli predictor for sparse AI benchmark evaluation, achieving a 24.6% Brier score reduction over a constant baseline with a 7.6 KiB program and microsecond-scale predictions.

arXiv:2611.24159v1 Announce Type: new Abstract: Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.
Original Article
View Cached Full Text

Cached at: 11/20/26, 09:14 AM

# Fast Bayesian Methods for Uncertainty Quantificationin Sparse AI Evaluation
Source: [https://arxiv.org/html/2611.24159](https://arxiv.org/html/2611.24159)
Dario Deniz Ergun1Haohua Wang1Brendon Tao1 Jonah Kollenberg2Chen Ye Gan1 1Institute of Data and Information, Tsinghua University 2Delft Institute of Applied Mathematics, Delft University of Technology

###### Abstract

Estimating how an AI system will perform on a new benchmark need not require another large model\. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes\. The predictor uses no item semantics, pretrained weights, or inference API\. Its single online evaluation in the AIMS competition returned a team\-reported Brier ALC of0\.188477, a24\.6%reduction from the constant\-0\.50\.5reference\. The complete submission is7,754 bytes; a server microbenchmark measures12\.71μ\\musper prediction with 31 matching labels\. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction\. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference\. A public evaluation covers 79 system–benchmark pairs and eight seeds, reaching Brier ALC 0\.200250 for the submitted method\. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories\. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost\.

## 1Introduction

Evaluating an AI system requires running it on tasks, but a large evaluation can be expensive\. Before collecting many responses, we would like to estimate the probability that the system succeeds on a new benchmark\. Public results give an initial expectation\. A few observed outcomes then tell us how much that expectation should change\. The question is how much useful prediction can be obtained from these two sources of information with a small computational cost\.

The AIMS competition makes this question concrete\([AIMS Organizers, 2026](https://arxiv.org/html/2611.24159#bib.bib3)\)\. A program receives the attributes of an evaluated system and a benchmark item, together with a small number of revealed binary outcomes\. It predicts the probability of a positive response without running the evaluated system\. We therefore estimate*another system’s behavior*\. The predictor does not need to answer the question itself\.

We begin with one unknown success rate for each system–benchmark pair\. Public evaluation counts determine the mean of a Beta prior, while its concentration is fixed at two\. This gives source knowledge the weight of only two observations: enough to specify a starting probability, but little enough for the target outcomes to replace it quickly\. Beta–Bernoulli conjugacy then gives the prediction directly,

p^n=2​m\+s2\+n,\\widehat\{p\}\_\{n\}=\\frac\{2m\+s\}\{2\+n\},\(1\)wheremmis the source prior mean,nnis the number of matching acquired labels, andssis their number of successes\.

The calculation uses a prior mean and two counts\. It needs no representation of the item text and no numerical posterior approximation\. In our single online evaluation, submission1039683returned Brier ALC 0\.188477\. Relative to the constant\-0\.50\.5reference, this is a 24\.6% reduction in probability error, obtained with a 7\.6 KiB program and microsecond\-scale measured prediction time\. The small runtime information state should not be confused with a small source collection: fitting the prior uses millions of permitted public responses\.

We will describe how source counts determine the prior, derive how observed labels change it, and then test what this simple construction gains and loses\. The contributions are:

- •A complete empirical\-Bayes competition predictor with a benchmark\-balanced source mean and exact adaptation from sparse target feedback\.
- •An explicit account of prior sensitivity, posterior uncertainty, and the computational work required by the tested implementation\.
- •A reproducible evaluation and prior\-sensitivity study over 79 public pairs and eight seeds, separating the single online result from controlled development evidence\.

Estimating a prior from related observations follows the empirical\-Bayes principle\([Robbins, 1956](https://arxiv.org/html/2611.24159#bib.bib6)\); conjugacy itself is established methodology\. Our contribution is the implementation and analysis\. The analysis isolates the influence of the prior while keeping the submitted update and observed evidence fixed\.

## 2Prediction task and evaluation

Letx=\(u,i\)x=\(u,i\)contain a complete supplied system configurationuuand an itemii\. A pair is identified by that full configuration and the item’s anonymous benchmark identifier\. Different harnesses, reasoning settings, or other configuration fields define different evidence groups, even when the model name is the same\. The target is the recorded binary outcomeY∈\{0,1\}Y\\in\\\{0,1\\\}under the evaluation harness\. Its semantics are inherited from the source evaluation; in some tasks a positive event measures attack success rather than desirable behavior\.

At budgetsℬ=\{0,1,3,7,15,31\}\\mathcal\{B\}=\\\{0,1,3,7,15,31\\\}, the predictor receives nested sets of acquired outcomesDnD\_\{n\}\. Acquisition and evaluation use disjoint items, and repeated responses for an item remain together\. All evaluation responses contribute to the Brier loss\(p^−Y\)2\(\\widehat\{p\}\-Y\)^\{2\}\([Brier, 1950](https://arxiv.org/html/2611.24159#bib.bib4)\)\. The normalized area under the curve on thelog2⁡\(n\+1\)\\log\_\{2\}\(n\+1\)axis is

ALC=0\.1​B0\+0\.2​B1\+0\.2​B3\+0\.2​B7\+0\.2​B15\+0\.1​B31,\\operatorname\{ALC\}=0\.1B\_\{0\}\+0\.2B\_\{1\}\+0\.2B\_\{3\}\+0\.2B\_\{7\}\+0\.2B\_\{15\}\+0\.1B\_\{31\},\(2\)computed within each pair and then averaged equally across pairs\. Smaller values are better\. Constant prediction0\.50\.5gives0\.250\.25at every budget\.

Brier loss is a proper scoring rule\([Gneiting and Raftery, 2007](https://arxiv.org/html/2611.24159#bib.bib5)\)\. If the true conditional positive probability isp∗p\_\{\*\}, then

𝔼\[\(q−Y\)2∣x,Dn\]=p∗\(1−p∗\)\+\(q−p∗\)2\.\\mathbb\{E\}\[\(q\-Y\)^\{2\}\\mid x,D\_\{n\}\]=p\_\{\*\}\(1\-p\_\{\*\}\)\+\(q\-p\_\{\*\}\)^\{2\}\.\(3\)Thus probability estimation, rather than drawing a binary prediction, is the relevant decision\. Under a Bayesian model, the posterior predictive mean minimizes the corresponding posterior expected loss\.

## 3A weak source prior with exact adaptation

### 3\.1Estimating the prior from public evaluations

For each eligible source system–benchmark recordrr, letSr,FrS\_\{r\},F\_\{r\}denote the counts of positive and negative responses\. We first form a smoothed rate

qr=Sr\+1/2Sr\+Fr\+1\.q\_\{r\}=\\frac\{S\_\{r\}\+1/2\}\{S\_\{r\}\+F\_\{r\}\+1\}\.\(4\)The half\-count smoothing keeps finite records away from the boundary\. It is a source\-rate estimator, separate from the concentration used for adaptation on a new benchmark\.

For a subset of source recordsℛ\\mathcal\{R\}, letcb​\(ℛ\)c\_\{b\}\(\\mathcal\{R\}\)be the number of its records from benchmarkbb\. We estimate

m⁡\(ℛ\)=∑r∈ℛqr/cb⁡\(r\)​\(ℛ\)∑r∈ℛ1/cb⁡\(r\)​\(ℛ\)\.m\(\\mathcal\{R\}\)=\\frac\{\\sum\_\{r\\in\\mathcal\{R\}\}q\_\{r\}/c\_\{b\(r\)\}\(\\mathcal\{R\}\)\}\{\\sum\_\{r\\in\\mathcal\{R\}\}1/c\_\{b\(r\)\}\(\\mathcal\{R\}\)\}\.\(5\)Each represented benchmark therefore contributes equally, regardless of how many source configurations it contains\. This avoids allowing a large evaluation table to determine the prior merely through its size\. Counts within a record still use all eligible repeated outcomes; no claim of independent trials is required to compute the statistic\.

We retain source records with at least 20 distinct items\. A model name is normalized by trimming whitespace and applying Unicode case folding\. Its matched subset is used only when it contains at least two source families; otherwise the predictor uses the global eligible subset\. The exported asset contains 301 supported model identities and global meanm0=0\.5067429803m\_\{0\}=0\.5067429803\. The final export uses all eligible source records\. In offline evaluation, the complete target family is excluded before computing either the model\-specific or fallback mean\.

### 3\.2Posterior prediction and prior sensitivity

For a target pair, the working model is

θ∼Beta\(κm,κ\(1−m\)\),Yj∣θ∼Bernoulli\(θ\),κ=2\.\\theta\\sim\\operatorname\{Beta\}\(\\kappa m,\\kappa\(1\-m\)\),\\qquad Y\_\{j\}\\mid\\theta\\sim\\operatorname\{Bernoulli\}\(\\theta\),\\qquad\\kappa=2\.\(6\)Aftersssuccesses andfffailures, conjugacy gives

θ∣Dn∼Beta\(an,bn\),an=2m\+s,bn=2\(1−m\)\+f,\\theta\\mid D\_\{n\}\\sim\\operatorname\{Beta\}\(a\_\{n\},b\_\{n\}\),\\qquad a\_\{n\}=2m\+s,\\quad b\_\{n\}=2\(1\-m\)\+f,\(7\)withn=s\+fn=s\+f\. Equation \([1](https://arxiv.org/html/2611.24159#S1.E1)\) is its predictive mean\. No numerical integration, optimization, or MCMC is needed to implement this update\.

Forn\>0n\>0, the same expression is a convex combination,

p^n=2n\+2⏟prior weight​m\+nn\+2⏟evidence weight​sn\.\\widehat\{p\}\_\{n\}=\\underbrace\{\\frac\{2\}\{n\+2\}\}\_\{\\text\{prior weight\}\}m\+\\underbrace\{\\frac\{n\}\{n\+2\}\}\_\{\\text\{evidence weight\}\}\\frac\{s\}\{n\}\.\(8\)At 31 labels, only2/33≃6\.1%2/33\\simeq 6\.1\\%of the weight remains on the prior\. More generally, changing the source mean frommmtom′m^\{\\prime\}changes the prediction by exactly

\|p^n​\(m\)−p^n​\(m′\)\|=2n\+2​\|m−m′\|\.\|\\widehat\{p\}\_\{n\}\(m\)\-\\widehat\{p\}\_\{n\}\(m^\{\\prime\}\)\|=\\frac\{2\}\{n\+2\}\|m\-m^\{\\prime\}\|\.\(9\)This is an algebraic robustness property with respect to the prior mean\. It does not protect against a nonrepresentative acquired sample or an inadequate pair\-level model\. Concentration two was a candidate in our development sweep; it was not inferred from a posterior over concentration values\.

Figure 1:How the selected model learns\.Left: illustrative Beta densities withm=0\.6m=0\.6and\(n,s\)=\(0,0\),\(3,2\),\(15,10\),\(31,22\)\(n,s\)=\(0,0\),\(3,2\),\(15,10\),\(31,22\)\. These are synthetic counts explaining the mechanism, not benchmark results\. Right: the exact prior weightκ/\(κ\+n\)\\kappa/\(\\kappa\+n\); the selectedκ=2\\kappa=2yields rapidly to observations\. All plotted coordinates are supplied as CSV files\.
### 3\.3What the uncertainty means

The posterior describes uncertainty about a*pair\-level rate*\. Writingμn=p^n\\mu\_\{n\}=\\widehat\{p\}\_\{n\}, its variance and the next\-outcome variance are

Var⁡\(θ∣Dn\)\\displaystyle\\operatorname\{Var\}\(\\theta\\mid D\_\{n\}\)=μn​\(1−μn\)n\+3,\\displaystyle=\\frac\{\\mu\_\{n\}\(1\-\\mu\_\{n\}\)\}\{n\+3\},\(10\)Var⁡\(Y∗∣Dn\)\\displaystyle\\operatorname\{Var\}\(Y\_\{\*\}\\mid D\_\{n\}\)=𝔼⁡\[θ⁡\(1−θ\)∣Dn\]⏟outcome variability\+Var⁡\(θ∣Dn\)⏟uncertainty about the rate=μn​\(1−μn\)\.\\displaystyle=\\underbrace\{\\mathbb\{E\}\[\\theta\(1\-\\theta\)\\mid D\_\{n\}\]\}\_\{\\text\{outcome variability\}\}\+\\underbrace\{\\operatorname\{Var\}\(\\theta\\mid D\_\{n\}\)\}\_\{\\text\{uncertainty about the rate\}\}=\\mu\_\{n\}\(1\-\\mu\_\{n\}\)\.\(11\)The epistemic term is bounded by1/\[4​\(n\+3\)\]1/\[4\(n\+3\)\], but it need not decrease after every realized label: an unexpected outcome can move the posterior mean toward1/21/2\. The expected posterior variance decreases under the working model\. Equal\-tailed intervals forθ\\thetacan be computed from Beta quantiles, although the submission returns only the predictive mean required by the interface\.

These uncertainty statements are conditional on the exchangeable Bernoulli model and a fitted source mean\. Item heterogeneity, dependence among outcomes, and uncertainty in the estimated source prior are not fully represented\. In particular, a narrow posterior interval is not a validated guarantee of item\-level calibration or real\-world coverage\.

### 3\.4Implementation and acquisition

The runtime loads a small JSON prior table once and reconstructs counts from the evidence supplied to each call\. Exact equality of the complete subject dictionary and benchmark identifier prevents information from unrelated pairs from entering the update\. Reconstruction also makes rollback to a smaller budget safe: no later labels remain in mutable posterior state\. The core computation is summarized below; the bundled submission preserves the tested implementation exactly\.

name=str\(subject\.get\("normalized\_name",""\)\)\.strip\(\)\.casefold\(\)

m=model\_means\.get\(name,global\_mean\)

matched=\[\(x,y\)forx,yinlabeled

ifx\[0\]==subject

andx\[1\]\["benchmark\_id"\]==item\["benchmark\_id"\]\]

s=sum\(yfor\_,yinmatched\)

returnfloat\(\(2\*m\+s\)/\(2\+len\(matched\)\)\)

The archive omits the optional acquisition file and uses the evaluator’s random policy\. Within this model, every item in a pair has the same latent rate, so information\-based utilities cannot distinguish candidates without introducing additional structure\. Random acquisition also matches the offline protocol used to evaluate the submitted predictor\.

## 4Data and experimental design

All competition\-specific fitting uses the permitted publicmeasurement\-dbrelease\([AIMS Foundations, 2026b](https://arxiv.org/html/2611.24159#bib.bib2)\), pinned to revision54074453da7b9911be0f48e4c560be547bc84d21\. We processed 86 source wrappers under a conservative quarantine policy\. After binary\-outcome, condition\-ambiguity, and exact\-text filtering, the intermediate bank contains 11,330 pairs and 17,694,172 responses\. Applying the selected model’s 20\-item threshold leaves11,251 records from 84 wrappers and 83 families, containing 17,693,497 responses\. Appendix[C](https://arxiv.org/html/2611.24159#A3)lists every wrapper; the package contains the exact aggregate counts used for fitting\.

Source construction excludes nonempty item texts that exactly match the public development panel, recording 48,178 source\-item exclusions\. This is a sum across wrappers, not a count of unique original questions\. Known related benchmark families are quarantined, and every offline prior additionally excludes its target family\. These safeguards do not certify independence of paraphrases, translations, shared original question banks, or foundation\-model pretraining\.

The public development panel contains 79 system–benchmark pairs, 42 benchmarks, and 19,520 recorded responses\. We use eight seeds,11,23,47,59,71,101,137,18111,23,47,59,71,101,137,181, with the same acquired label stream across predictors for a given pair and seed\. Each split assigns half the distinct items to acquisition and half to evaluation; repeated outcomes stay with their item\. We average over seeds within a pair and then equally across pairs\. All six budgets are evaluated, giving 632 pair–seed trajectories per method\. This panel has been repeatedly inspected and tuned against; it is an exploratory development set\.

For Source Beta, 46 of the 79 pairs use the global fallback after target\-family exclusion\. Sparse identity coverage therefore limits how often the method actually exploits a model\-specific mean\.

## 5Results: learning from sparse feedback

### 5\.1The online submission

The team developed and tested the predictor locally, then evaluated the selected archive once online as Codabench submission1039683\. That run returned a team\-reported Brier ALC of0\.188477\. A predictor that always returns 0\.5 has Brier ALC 0\.25 on any binary\-outcome sample\. The submitted model therefore reduces this reference error by0\.061523, or24\.6%\. It obtains this result by looking up a prior mean and counting matching labels, without interpreting the item text or running a neural model\.

This online result concerns the all\-source exported prior\. The public evaluation below excludes each target family when fitting its prior\. The two scores describe their respective evaluation settings; they are not a before–after comparison\. The original detailed online scoring log is not bundled\.

### 5\.2Adaptation and calibration

On the public panel, the submitted update reaches Brier ALC0\.200250\. Its Brier score falls from 0\.253063 at zero labels to0\.174150at 31 labels\. The progression is visible in Figure[2](https://arxiv.org/html/2611.24159#S5.F2): each additional budget gives the target evidence more influence over the starting expectation\. At 31 labels, the source prior contributes only 6\.1% of the posterior mean\.

The calibration diagnostics improve at the same time\. Macro 10\-bin expected calibration error decreases from 0\.240556 to0\.060802, and negative log likelihood decreases from 0\.699249 to0\.518072\. These diagnostics answer a complementary question: whether the assigned probabilities agree with observed response frequencies\. Appendix[B](https://arxiv.org/html/2611.24159#A2)supplies the full per\-budget table and reliability curves\.

Figure 2:The submitted predictor learns from target feedback\.Brier score and macro calibration error are evaluated on the same 79 public pairs and eight seeds\. Both improve as outcomes replace the initial prior expectation\. These curves are public development results; the single online score is reported separately\.

## 6How much does the prior matter?

### 6\.1Exact resistance to the initial mean

The update makes its prior sensitivity explicit\. If two priors have the same concentrationκ\\kappaand meansm,m′m,m^\{\\prime\}, then after the samennlabels,

\|p^n​\(m\)−p^n​\(m′\)\|=κκ\+n​\|m−m′\|\.\|\\widehat\{p\}\_\{n\}\(m\)\-\\widehat\{p\}\_\{n\}\(m^\{\\prime\}\)\|=\\frac\{\\kappa\}\{\\kappa\+n\}\|m\-m^\{\\prime\}\|\.\(12\)With the submittedκ=2\\kappa=2, the multiplier falls from one at zero labels to2/332/33at 31 labels\. Thus93\.9% of the initial difference is removed\. Even changing the starting mean from 0\.1 to 0\.9 changes the 31\-label prediction by only0\.8​\(2/33\)=0\.0484850\.8\(2/33\)=0\.048485\. The property follows from the update itself; it does not require that the source mean be close to the target rate\.

The same calculation explains why the concentration matters\. A large concentration preserves a source assumption for longer\. A weak concentration lets observed outcomes determine the prediction sooner\. We chose two pseudo\-observations so that source knowledge supplies a starting point while target evidence rapidly takes over\.

### 6\.2Replaying alternative priors on identical evidence

We test this mechanism directly by replaying the supplied public sufficient statistics\. We change only the prior mean or concentration; the acquired labels, evaluation outcomes, budgets, pairs, and seeds stay fixed\. Constant means0\.1,0\.25,0\.5,0\.75,0\.90\.1,0\.25,0\.5,0\.75,0\.9test widely separated starting beliefs at concentration two\. A second sweep keeps the fitted source means and varies concentration over0\.5,1,2,8,320\.5,1,2,8,32\. These are sensitivity settings of the same Beta predictor, not different prediction methods\.

Figure 3:Target evidence reduces sensitivity to prior choice\.Left: alternative prior means with the submitted concentration two, alongside the fitted source mean\. Right: the source means with alternative concentrations\. Every curve replays the same public evidence\. Weak priors converge rapidly toward predictions determined by the acquired outcomes\.Table[1](https://arxiv.org/html/2611.24159#S6.T1)gives the exact replay scores\. Among the tested concentrations with the same source mean, the submittedκ=2\\kappa=2has the lowest ALC\. Across the source mean and constant means from 0\.1 to 0\.9, the 31\-label Brier scores occupy the narrow range0\.174150–0\.175078, despite widely different starting beliefs\. At concentration two, the uniform mean 0\.5 gives ALC 0\.199570 andB31=0\.174163B\_\{31\}=0\.174163, compared with 0\.200250 and 0\.174150 for the source mean\. Their final\-budget difference is about1\.3×10−51\.3\\times 10^\{\-5\}\. The source mean does not improve average ALC on this panel; the robust behavior comes from revising the weak starting assumption rapidly\. Early budgets still contribute to ALC, so resistance at 31 labels does not make the initial prior irrelevant to the integrated score\.

Table 1:Prior sensitivity within the submitted Beta update\.Exact replay of 632 public pair–seed trajectories\. Source means are fitted with the target family excluded\. All settings use the same outcomes\. Lower scores are better\.

## 7Microsecond prediction and evidence scaling

The submitted predictor performs no transformer forward pass, matrix multiplication, or GPU computation\. It scans the supplied evidence, counts matching outcomes, and evaluates Equation \([1](https://arxiv.org/html/2611.24159#S1.E1)\)\. WithMMsupplied records andn≤31n\\leq 31matches, the implementation usesO⁡\(M\+n\)O\(M\+n\)time andO⁡\(n\)O\(n\)temporary storage, plus the fixed prior dictionary\. Exact configuration matching prevents unrelated evidence from entering the posterior\. Reconstructing counts at each call also permits returning to a smaller label budget safely\.

We measured the frozen implementation on a shared server with two Intel Xeon Platinum 8358P CPUs\. The process was restricted to two logical cores\. Each budget uses nine blocks of 1,000 predictions; quoted medians summarize block\-average per\-call times\. At 31 matching labels, the median is12\.71μ\\mus\. With 2,048 supplied records but still 31 matches, it is0\.372 ms\. Figure[4](https://arxiv.org/html/2611.24159#S7.F4)shows that the evidence scan determines scaling\.

Figure 4:Runtime of the submitted predictor\.Left: measured warm prediction time as matching evidence grows\. Right: packet\-size stress with 31 matches and additional unrelated records\. Measurements use the frozen CPU implementation\.The complete tested archive is7,754 bytes\. Fresh\-process readiness has a median of 25\.05 ms over three recorded trials\. Fitting the prior table from loaded aggregate counts took 15\.32 ms in one recorded server run\. These timings exclude the earlier source\-data curation\. The selected prior uses millions of public source responses; its compact deployment state results from aggregating them into a small table\.

ForQQpredictions at each budget, only\-matching packets require57​Q57Qrecord inspections across all six budgets because0\+1\+3\+7\+15\+31=570\+1\+3\+7\+15\+31=57\. The implementation performs scalar arithmetic, string comparisons, and memory accesses, with zero transformer or dense\-matrix multiply–accumulates\. The measured 31\-label workload therefore captures a small and explicit computation\.

## 8Scope and limitations

The predictor estimates a pair\-level rate and does not distinguish item difficulty\. Its posterior uncertainty is conditional on the exchangeable Bernoulli model and fitted source mean\. The public panel was reused during development, and interval coverage is not evaluated\. The single online run gives no repeated\-sample uncertainty estimate\. These boundaries motivate testing the same weak\-prior update on fresh target collections\.

## 9Conclusion

A prior mean and two counts are enough to construct a useful predictive evaluator\. Our single online run reaches Brier ALC 0\.188477, 24\.6% below the constant\-0\.50\.5reference, using a 7\.6 KiB archive and 12\.71μ\\mus measured prediction time at 31 labels\. A weak prior turns public knowledge into a starting expectation and then yields to target evidence: after 31 labels, only 6\.1% of its initial influence remains\. Exact sensitivity formulas and matched\-evidence replays make this behavior visible\. The result shows how much predictive value an explicit Bayesian model can obtain from sparse feedback with very little online computation\.

#### Data, code, and assistant disclosure\.

The companion release supplies the tested submission, prior\-fitting and replay code, public aggregate data, figure inputs, and LaTeX sources\. All competition\-specific data originate from the permitted public release\. The predictor uses no pretrained model or inference API\. Codex and local Qwen models through OpenCode assisted with implementation, experiment supervision, analysis, and drafting\. The authors remain responsible for the claims\. Selected Codabench ID:1039683\. Code and permitted derived statistics:[https://github\.com/theNewRootUser/aims\-fast\-bayesian\-evaluation](https://github.com/theNewRootUser/aims-fast-bayesian-evaluation)

## References

- AIMS Foundations \[2026a\]AIMS Foundations\.PAIEC predictor baselines, 2026a\.URL[https://github\.com/aims\-foundations/paiec\_baseline](https://github.com/aims-foundations/paiec_baseline)\.
- AIMS Foundations \[2026b\]AIMS Foundations\.measurement\-db: Public training data for predictive AI evaluation, 2026b\.URL[https://huggingface\.co/datasets/aims\-foundations/measurement\-db](https://huggingface.co/datasets/aims-foundations/measurement-db)\.Release revision 54074453da7b9911be0f48e4c560be547bc84d21\.
- AIMS Organizers \[2026\]AIMS Organizers\.The predictive AI evaluation competition at NeurIPS 2026, 2026\.URL[https://aims\.stanford\.edu/competition](https://aims.stanford.edu/competition)\.Accessed 7 November 2026\.
- Brier \[1950\]Glenn W\. Brier\.Verification of forecasts expressed in terms of probability\.*Monthly Weather Review*, 78\(1\):1–3, 1950\.URL[https://journals\.ametsoc\.org/view/journals/mwre/78/1/1520\-0493\_1950\_078\_0001\_vofeit\_2\_0\_co\_2\.xml](https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml)\.
- Gneiting and Raftery \[2007\]Tilmann Gneiting and Adrian E\. Raftery\.Strictly proper scoring rules, prediction, and estimation\.*Journal of the American Statistical Association*, 102\(477\):359–378, 2007\.doi:10\.1198/016214506000001437\.
- Robbins \[1956\]Herbert Robbins\.An empirical bayes approach to statistics\.In*Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability*, volume 1, pages 157–163\. University of California Press, 1956\.URL[https://digicoll\.lib\.berkeley\.edu/record/112828](https://digicoll.lib.berkeley.edu/record/112828)\.

## Appendix APosterior uncertainty and expected information value

The Beta variance formula follows from𝔼⁡\[θ\]=a/\(a\+b\)\\mathbb\{E\}\[\\theta\]=a/\(a\+b\)and𝔼⁡\[θ2\]=a⁡\(a\+1\)/\[\(a\+b\)​\(a\+b\+1\)\]\\mathbb\{E\}\[\\theta^\{2\}\]=a\(a\+1\)/\[\(a\+b\)\(a\+b\+1\)\]\. Witht=a\+b=n\+2t=a\+b=n\+2, it yields

Var⁡\(θ∣Dn\)=a​bt2​\(t\+1\),𝔼⁡\[θ⁡\(1−θ\)∣Dn\]=a​bt⁡\(t\+1\)\.\\operatorname\{Var\}\(\\theta\\mid D\_\{n\}\)=\\frac\{ab\}\{t^\{2\}\(t\+1\)\},\\qquad\\mathbb\{E\}\[\\theta\(1\-\\theta\)\\mid D\_\{n\}\]=\\frac\{ab\}\{t\(t\+1\)\}\.These two terms sum to the predictive Bernoulli variance\. Conditional on the model, averaging over the next outcome gives

𝔼Yn\+1\|Dn​\[Var⁡\(θ∣Dn,Yn\+1\)\]=tt\+1​Var⁡\(θ∣Dn\)\.\\mathbb\{E\}\_\{Y\_\{n\+1\}\\mid D\_\{n\}\}\[\\operatorname\{Var\}\(\\theta\\mid D\_\{n\},Y\_\{n\+1\}\)\]=\\frac\{t\}\{t\+1\}\\operatorname\{Var\}\(\\theta\\mid D\_\{n\}\)\.\(13\)The expected contraction is exact, although realized posterior variance can increase\. This distinction matters when describing uncertainty as “decreasing with more information\.”

For distinct future outcomes that are independent conditional onθ\\theta, the expected reduction in optimal predictive Brier risk after acquiring one additional label is

VarYn\+1\|Dn\(𝔼\[Y∗∣Dn,Yn\+1\]\)=μn​\(1−μn\)\(n\+3\)2≥0\.\\operatorname\{Var\}\_\{Y\_\{n\+1\}\\mid D\_\{n\}\}\\\!\\left\(\\mathbb\{E\}\[Y\_\{\*\}\\mid D\_\{n\},Y\_\{n\+1\}\]\\right\)=\\frac\{\\mu\_\{n\}\(1\-\\mu\_\{n\}\)\}\{\(n\+3\)^\{2\}\}\\geq 0\.\(14\)All candidate items have this same value in the selected exchangeable model\. A more informative item\-selection rule requires a model in which candidates differ\.

## Appendix BAdditional calibration and computational details

Figure 5:Pooled reliability on the public development panel\.Bins have equal width, with contributions weighted to preserve equal pair and seed weighting\. Empty bins are omitted\. These pooled curves differ from averaging ECE within each pair and can conceal opposing pair\-level errors\. They are descriptive diagnostics and do not establish posterior\-interval coverage\.Table 2:Selected source Beta: complete offline diagnostic curve\.Negative log likelihood clips predictions to\[10−12,1−10−12\]\[10^\{\-12\},1\-10^\{\-12\}\]\. ECE uses ten equal\-width bins within each pair and seed before averaging\. The package retains both these metrics and the pooled bin masses so that aggregation choices are explicit\.

The selected method’s 12\.71μ\\mus timing is the median of nine block means, each computed over 1,000 calls\. The reported upper percentiles in the raw evidence are percentiles of block means, not individual\-call tail latencies\. Separate process\-startup trials use a minimal standard\-library harness and warm operating\-system file caches\. They exclude first\-time internet setup and the original data curation\. A Linux/proc/self/statusaudit of a fresh selected\-model process, with a 31\-label synthetic fixture, records 12,197,888 resident/high\-water bytes \(11\.63 MiB\)\. This isolated measurement is more informative for deployment than the earlier harness\-levelgetrusagehigh\-water statistic\. It is one fixture measurement, not a certified memory limit\.

## Appendix CData disclosure and source inventory

The source snapshot is the permittedaims\-foundations/measurement\-dbrelease\. The selected predictor uses only aggregate success/failure counts and supplied model identities; it does not use private Codabench responses, web retrieval, external inference APIs, or pretrained weights\. Missing and nonbinary outcomes are excluded\. Source item–subject combinations with ambiguous observed test conditions are excluded\. Retained repeated outcomes contribute to aggregate rates\. The full processed bank and the final eligible bank are distinct stages, as shown below\.

The processed inventory contains 86 wrappers\. Of these, 84 contribute eligible records to the selected prior\. Table[3](https://arxiv.org/html/2611.24159#A3.T3)lists all processed wrappers; an asterisk marks one with no eligible record\. The conservative family map merges selected related wrappers, leaving 83 eligible families\. The exact counts, configuration, exclusion manifest, and policy files are supplied with the report\. The final asset is reconstructed from the compressed aggregate\-count file, whose decompressed SHA256 is

c4bc1ce814669e829212df29f56062026b031c5f101a8c76d645aea6b7260eaf

The frozen public development panel is identified by SHA256

ffe4849e7a665fff08e0d9affb9ebb7a385c29ff68a755c0c9e8cb882e40f832\.

The package does not redistribute raw task texts\. It supplies the public aggregate counts actually used to fit the selected prior, together with 3,792 pair–seed–budget sufficient\-statistic rows for exact score replay\. For each such row, the prediction and evaluation mean suffice because the constant\-prediction Brier loss equalsp2−2​p​y¯\+y¯p^\{2\}\-2p\\bar\{y\}\+\\bar\{y\}\. These are public development aggregates, not hidden online test data\. All plotted experimental coordinates derive from included evidence; the mechanism illustration is explicitly synthetic\.

Table 3:Processed source wrappers and eligible record counts\. Asterisks mark zero eligible records\.WrapperRecordsWrapperRecordsabstract\_reason1judgetuning7adaptivestep2kernelbench10aegis1legal\_rag\_bench6afrieval948legalbench31afrimedqa32lingoly11ai2d\_test254liveaopsbench13algotune18llm\_bp\_tag1alignment\_faking4278llmail\_inject22alpha\_sql1mathvista\_mini263appworld18mind2web1averimatec\*0mj\_bench55babilong71mmbench\_v11251beavertails4mtbbench25benger29naturebench23bertaqa6nester3bird\_sql120openbiorq11brace28os\_harm7causal\_axioms2perfcodebench20cellverse1perumedqa11ceval11pxplore14clasheval6real\_webagents33crow34realtimeqa92dave8reliancescope24devbench19risebench23disco4robust\_reasoning\_benchmark31editbench44safeagentbench6edu\_circuit\_hw6sgrades65egoschema11sib2003308emoji\_attack10sketchjudge38engdesign12statqa38enginemt\_qa\*0subjective\_qa5erbench8sugarcrepe1exploitgym4sustainable\_food6faithcot4tensortrust3felm1truthfulqa\_mc150fewshot\_ttt\_bbh11tulu\_human\_eval4hallusionbench253visual\_riddles8healthadminbench2visualwebarena3helm\_thaiexam42wikihow\_agent12hle16wmt\_mqm84igakuqa5workarena2igakuqa1191workbench\_revisited24ikp170worldcentralbanks10Upstream source licenses and attribution conditions remain applicable\. Only compact derived statistics and the prior asset are redistributed; the package does not relicense raw benchmark texts\.

## Appendix DReproducing and identifying the submission

The selected archive isaims\_beta\_modelprior\_k2\.zip\. It containsmodel\.py, its standard\-library runtime, the prior JSON, and the original README and manifest\. Its SHA256 is

681db0f9cdaea6b0ac062c63f521a5117605509fb089dfdc9d6d697b7ca0af54\.

The original archive passed the official submission checker and actual prediction smoke\[[AIMS Foundations, 2026a](https://arxiv.org/html/2611.24159#bib.bib1)\]\. The report\-building audit uses explicit compensated summation to reconstruct the prior file byte for byte from the source counts, recomputes family\-held\-out means, and recovers ALC 0\.2002499321275406 over 632 trajectories to floating\-point precision\. The publication bundle has a separate file manifest; modifying the paper does not alter the tested submission ZIP\.

pythoncode/reproduce\_selected\.py

python\-mpipinstall\-rrequirements\-figures\.txt

pythoncode/make\_selected\_figures\.py

tectonicmain\.tex

The release separates executable reproduction code from historical development snapshots\. The former is self\-contained for the selected prior and score replay\. The latter records the original filtering and evaluation procedures and their server paths; rebuilding the entire raw\-data campaign additionally requires the pinned public release and historical manifests\. The full public source collection remains on the lab server\. Data used by every figure are included as both evidence JSON and small CSV tables, with PDF, SVG, and PNG outputs\.

The report uses the official NeurIPS 2026 style in preprint mode, without modifying its margins or typography\. AIMS permits nonanonymous reports of approximately four to eight main\-text pages, with unlimited references and appendices\. The companionREQUIREMENTS\.mdmaps those requirements to this release\. The selected Codabench submission ID is1039683\. Code and permitted derived statistics:[https://github\.com/theNewRootUser/aims\-fast\-bayesian\-evaluation](https://github.com/theNewRootUser/aims-fast-bayesian-evaluation)No additional online model evaluation was performed while preparing the report\.

Similar Articles

You Don't Need to Run Every Eval

arXiv cs.LG

This research paper demonstrates that the scores of frontier AI models across 133 benchmarks are approximately rank-2, meaning only two latent factors explain over 90% of variation. The authors introduce BenchPress, a logit-space matrix completion method that predicts a model's full scorecard from just a few benchmarks, significantly reducing the cost of evaluation.

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

arXiv cs.LG

Introduces an efficient Bayesian deep ensemble method for predictive regression that combines low-dimensional ensemble representation, closed-form Bayesian aggregation, and independent ensemble training to achieve calibrated uncertainty estimates with computational efficiency.